[Fri Jan 23 14:56:53 UTC 2026] respawn: 0: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace /data/bpftrace/sys_write_delay.bt -v -p 9 osmo-bts-trx Attaching 'begin' probe Trying to attach probe: tracepoint:syscalls:sys_enter_write Trying to attach probe: tracepoint:syscalls:sys_exit_write Trying to attach probe: fexit:vmlinux:vfs_write Trying to attach probe: fexit:ext4:ext4_file_write_iter Trying to attach probe: fexit:vmlinux:generic_perform_write Trying to attach probe: fentry:vmlinux:generic_perform_write Trying to attach probe: fentry:ext4:ext4_file_write_iter Trying to attach probe: fentry:vmlinux:vfs_write Attached 9 probes Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 18 root cwd DIR 9,3 4096 13773864 /data sys_write 18 root rtd DIR 0,75 4096 8435560 / sys_write 18 root txt REG 0,75 129736 7647243 /usr/bin/dash sys_write 18 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,75) sys_write 18 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,75) sys_write 18 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,75) sys_write 18 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 18 root 1w FIFO W 0,13 0t0 26719 pipe sys_write 18 root 2w FIFO W 0,13 0t0 26720 pipe sys_write 18 root 10r REG LG,CX 9,3 98 13773903 /data/bpftrace/sys_write_delay.sh maps output: 563117b5b000-563117b5f000 r--p 00000000 09:03 7647243 /usr/bin/dash 563117b5f000-563117b73000 r-xp 00004000 09:03 7647243 /usr/bin/dash 563117b73000-563117b79000 r--p 00018000 09:03 7647243 /usr/bin/dash 563117b79000-563117b7b000 r--p 0001d000 09:03 7647243 /usr/bin/dash 563117b7b000-563117b7c000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 563117b7c000-563117b7e000 rw-p 00000000 00:00 0 56314530b000-56314532c000 rw-p 00000000 00:00 0 [heap] 7ff18a01e000-7ff18a021000 rw-p 00000000 00:00 0 7ff18a021000-7ff18a049000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff18a049000-7ff18a1ad000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff18a1ad000-7ff18a203000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff18a203000-7ff18a207000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff18a207000-7ff18a209000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff18a209000-7ff18a216000 rw-p 00000000 00:00 0 7ff18a224000-7ff18a226000 rw-p 00000000 00:00 0 7ff18a226000-7ff18a227000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff18a227000-7ff18a24f000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff18a24f000-7ff18a25a000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff18a25a000-7ff18a25c000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff18a25c000-7ff18a25d000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff18a25d000-7ff18a25e000 rw-p 00000000 00:00 0 7ffe5b40b000-7ffe5b42c000 rw-p 00000000 00:00 0 [stack] 7ffe5b502000-7ffe5b506000 r--p 00000000 00:00 0 [vvar] 7ffe5b506000-7ffe5b508000 r-xp 00000000 00:00 0 [vdso] @latency[2]: [1] 175 |@@@@@@@@@@@@@ | [2, 4) 410 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [4, 8) 454 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 685 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 224 |@@@@@@@@@@@@@@@@@ | [32, 64) 118 |@@@@@@@@ | [64, 128) 1 | | [128, 256) 0 | | [256, 512) 0 | | [512, 1K) 0 | | [1K, 2K) 0 | | [2K, 4K) 0 | | [4K, 8K) 0 | | [8K, 16K) 1 | | @latency[6]: [0] 509 | | [1] 1437 | | [2, 4) 127020 |@@@@@@@@@@@@ | [4, 8) 33664 |@@@ | [8, 16) 514979 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 11923 |@ | [32, 64) 754 | | [64, 128) 458 | | [Fri Jan 23 15:00:17 UTC 2026] respawn: 0: stopped pid 18 with status 0 [Fri Jan 23 15:00:17 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 23 15:00:18 UTC 2026] respawn: 1: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace /data/bpftrace/sys_write_delay.bt -v -p 40 osmo-bts-trx Attaching 'begin' probe Trying to attach probe: tracepoint:syscalls:sys_enter_write Trying to attach probe: tracepoint:syscalls:sys_exit_write Trying to attach probe: fexit:vmlinux:vfs_write Trying to attach probe: fexit:ext4:ext4_file_write_iter Trying to attach probe: fexit:vmlinux:generic_perform_write Trying to attach probe: fentry:vmlinux:generic_perform_write Trying to attach probe: fentry:ext4:ext4_file_write_iter Trying to attach probe: fentry:vmlinux:vfs_write Attached 9 probes Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 47 root cwd DIR 9,3 4096 13773864 /data sys_write 47 root rtd DIR 0,75 4096 8435560 / sys_write 47 root txt REG 0,75 129736 7647243 /usr/bin/dash sys_write 47 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,75) sys_write 47 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,75) sys_write 47 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,75) sys_write 47 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 47 root 1w FIFO W 0,13 0t0 26719 pipe sys_write 47 root 2w FIFO W 0,13 0t0 26720 pipe sys_write 47 root 10r REG LG,CX 9,3 98 13773903 /data/bpftrace/sys_write_delay.sh maps output: 556b55f3b000-556b55f3f000 r--p 00000000 09:03 7647243 /usr/bin/dash 556b55f3f000-556b55f53000 r-xp 00004000 09:03 7647243 /usr/bin/dash 556b55f53000-556b55f59000 r--p 00018000 09:03 7647243 /usr/bin/dash 556b55f59000-556b55f5b000 r--p 0001d000 09:03 7647243 /usr/bin/dash 556b55f5b000-556b55f5c000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 556b55f5c000-556b55f5e000 rw-p 00000000 00:00 0 556b7faea000-556b7fb0b000 rw-p 00000000 00:00 0 [heap] 7f549d4e7000-7f549d4ea000 rw-p 00000000 00:00 0 7f549d4ea000-7f549d512000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f549d512000-7f549d676000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f549d676000-7f549d6cc000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f549d6cc000-7f549d6d0000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f549d6d0000-7f549d6d2000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f549d6d2000-7f549d6df000 rw-p 00000000 00:00 0 7f549d6ed000-7f549d6ef000 rw-p 00000000 00:00 0 7f549d6ef000-7f549d6f0000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f549d6f0000-7f549d718000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f549d718000-7f549d723000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f549d723000-7f549d725000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f549d725000-7f549d726000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f549d726000-7f549d727000 rw-p 00000000 00:00 0 7fff04d48000-7fff04d69000 rw-p 00000000 00:00 0 [stack] 7fff04d99000-7fff04d9d000 r--p 00000000 00:00 0 [vvar] 7fff04d9d000-7fff04d9f000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 40 detected long write(fd=2, count=114, ret=114): 416831 us PESPIN: vfs_write delta=416823 us, file->f_flags=0x8c01 PESPIN: ext4_file_write_iter delta=416815 us PESPIN: inode_lock delta=416792 us PESPIN: generic_perform_write delta=17 us PESPIN: sys_write content: 20260123151012942 DL1P NOTICE (bts=0,trx=0,ts=5,ss=1) SACCH no SI available (lchan.c:284) stack trace of write(): 0x7f914ce01687 0 @[ 0x7f914ce01687 0 ]: 1 @latency[22]: [32, 64) 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| @latency[2]: [1] 4240 |@@@@@@@@@@@@@@ | [2, 4) 14879 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [4, 8) 10881 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 15526 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 8731 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [32, 64) 4246 |@@@@@@@@@@@@@@ | [64, 128) 35 | | [128, 256) 7 | | [256, 512) 5 | | [512, 1K) 1 | | [1K, 2K) 0 | | [2K, 4K) 2 | | [4K, 8K) 6 | | [8K, 16K) 7 | | [16K, 32K) 11 | | [32K, 64K) 0 | | [64K, 128K) 0 | | [128K, 256K) 4 | | [256K, 512K) 2 | | @latency[6]: [0] 10991 | | [1] 22170 | | [2, 4) 1156874 |@@@@@@@@@ | [4, 8) 1195872 |@@@@@@@@@ | [8, 16) 6615627 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 847850 |@@@@@@ | [32, 64) 37986 | | [64, 128) 66197 | | [128, 256) 3 | | [Fri Jan 23 16:01:52 UTC 2026] respawn: 1: stopped pid 47 with status 0 [Fri Jan 23 16:01:52 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 23 16:01:53 UTC 2026] respawn: 2: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace /data/bpftrace/sys_write_delay.bt -v -p 40 osmo-bts-trx Attaching 'begin' probe Trying to attach probe: tracepoint:syscalls:sys_enter_write Trying to attach probe: tracepoint:syscalls:sys_exit_write Trying to attach probe: fexit:vmlinux:vfs_write Trying to attach probe: fexit:ext4:ext4_file_write_iter Trying to attach probe: fexit:vmlinux:generic_perform_write Trying to attach probe: fentry:vmlinux:generic_perform_write Trying to attach probe: fentry:ext4:ext4_file_write_iter Trying to attach probe: fentry:vmlinux:vfs_write Attached 9 probes Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 76 root cwd DIR 9,3 4096 13773864 /data sys_write 76 root rtd DIR 0,75 4096 8435560 / sys_write 76 root txt REG 0,75 129736 7647243 /usr/bin/dash sys_write 76 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,75) sys_write 76 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,75) sys_write 76 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,75) sys_write 76 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 76 root 1w FIFO W 0,13 0t0 26719 pipe sys_write 76 root 2w FIFO W 0,13 0t0 26720 pipe sys_write 76 root 10r REG LG,CX 9,3 98 13773903 /data/bpftrace/sys_write_delay.sh maps output: 55b209ab8000-55b209abc000 r--p 00000000 09:03 7647243 /usr/bin/dash 55b209abc000-55b209ad0000 r-xp 00004000 09:03 7647243 /usr/bin/dash 55b209ad0000-55b209ad6000 r--p 00018000 09:03 7647243 /usr/bin/dash 55b209ad6000-55b209ad8000 r--p 0001d000 09:03 7647243 /usr/bin/dash 55b209ad8000-55b209ad9000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 55b209ad9000-55b209adb000 rw-p 00000000 00:00 0 55b2347da000-55b2347fb000 rw-p 00000000 00:00 0 [heap] 7f0ebf4d0000-7f0ebf4d3000 rw-p 00000000 00:00 0 7f0ebf4d3000-7f0ebf4fb000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0ebf4fb000-7f0ebf65f000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0ebf65f000-7f0ebf6b5000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0ebf6b5000-7f0ebf6b9000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0ebf6b9000-7f0ebf6bb000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0ebf6bb000-7f0ebf6c8000 rw-p 00000000 00:00 0 7f0ebf6d6000-7f0ebf6d8000 rw-p 00000000 00:00 0 7f0ebf6d8000-7f0ebf6d9000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0ebf6d9000-7f0ebf701000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0ebf701000-7f0ebf70c000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0ebf70c000-7f0ebf70e000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0ebf70e000-7f0ebf70f000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0ebf70f000-7f0ebf710000 rw-p 00000000 00:00 0 7ffc9db91000-7ffc9dbb2000 rw-p 00000000 00:00 0 [stack] 7ffc9dbba000-7ffc9dbbe000 r--p 00000000 00:00 0 [vvar] 7ffc9dbbe000-7ffc9dbc0000 r-xp 00000000 00:00 0 [vdso]