[Tue Jan 20 15:26:24 UTC 2026] respawn: 0: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 9 osmo-bts-trx Attached 5 probes Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 27 root cwd DIR 9,3 4096 13773903 /data sys_write 27 root rtd DIR 0,75 4096 18112779 / sys_write 27 root txt REG 0,75 129736 7647243 /usr/bin/dash sys_write 27 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,75) sys_write 27 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,75) sys_write 27 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,75) sys_write 27 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 27 root 1w FIFO W 0,13 0t0 4120204 pipe sys_write 27 root 2w FIFO W 0,13 0t0 4120205 pipe sys_write 27 root 10r REG LG,CX 9,3 109 13773935 /data/bpftrace/sys_write_delay.sh maps output: 55f063173000-55f063177000 r--p 00000000 09:03 7647243 /usr/bin/dash 55f063177000-55f06318b000 r-xp 00004000 09:03 7647243 /usr/bin/dash 55f06318b000-55f063191000 r--p 00018000 09:03 7647243 /usr/bin/dash 55f063191000-55f063193000 r--p 0001d000 09:03 7647243 /usr/bin/dash 55f063193000-55f063194000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 55f063194000-55f063196000 rw-p 00000000 00:00 0 55f098960000-55f098981000 rw-p 00000000 00:00 0 [heap] 7f1344d91000-7f1344d94000 rw-p 00000000 00:00 0 7f1344d94000-7f1344dbc000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f1344dbc000-7f1344f20000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f1344f20000-7f1344f76000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f1344f76000-7f1344f7a000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f1344f7a000-7f1344f7c000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f1344f7c000-7f1344f89000 rw-p 00000000 00:00 0 7f1344f97000-7f1344f99000 rw-p 00000000 00:00 0 7f1344f99000-7f1344f9a000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f1344f9a000-7f1344fc2000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f1344fc2000-7f1344fcd000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f1344fcd000-7f1344fcf000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f1344fcf000-7f1344fd0000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f1344fd0000-7f1344fd1000 rw-p 00000000 00:00 0 7ffc22614000-7ffc22635000 rw-p 00000000 00:00 0 [stack] 7ffc22755000-7ffc22759000 r--p 00000000 00:00 0 [vvar] 7ffc22759000-7ffc2275b000 r-xp 00000000 00:00 0 [vdso] @latency[2]: [0] 3 | | [1] 18466 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 7851 |@@@@@@@@@@@@@@@@@@@@@@ | [4, 8) 13294 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 4243 |@@@@@@@@@@@ | [16, 32) 1467 |@@@@ | [32, 64) 165 | | [64, 128) 23 | | [128, 256) 26 | | [256, 512) 6 | | [512, 1K) 0 | | [1K, 2K) 1 | | [2K, 4K) 1 | | [4K, 8K) 0 | | [8K, 16K) 0 | | [16K, 32K) 2 | | [32K, 64K) 2 | | [64K, 128K) 3 | | [128K, 256K) 2 | | [256K, 512K) 4 | | @latency[6]: [1] 2393 | | [2, 4) 440825 |@@@@@@@@@@@@@@@@@ | [4, 8) 236918 |@@@@@@@@@ | [8, 16) 210790 |@@@@@@@@ | [16, 32) 1345895 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [32, 64) 12191 | | [64, 128) 1781 | | [128, 256) 9 | | [256, 512) 2 | | [512, 1K) 1 | | [Tue Jan 20 15:37:27 UTC 2026] respawn: 0: stopped pid 27 with status 0 [Tue Jan 20 15:37:27 UTC 2026] respawn: sleeping 1 seconds... [Tue Jan 20 15:37:28 UTC 2026] respawn: 1: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 64 osmo-bts-trx Attached 5 probes Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 84 root cwd DIR 9,3 4096 13773903 /data sys_write 84 root rtd DIR 0,75 4096 18112779 / sys_write 84 root txt REG 0,75 129736 7647243 /usr/bin/dash sys_write 84 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,75) sys_write 84 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,75) sys_write 84 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,75) sys_write 84 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 84 root 1w FIFO W 0,13 0t0 4120204 pipe sys_write 84 root 2w FIFO W 0,13 0t0 4120205 pipe sys_write 84 root 10r REG LG,CX 9,3 109 13773935 /data/bpftrace/sys_write_delay.sh maps output: 560f4b1cb000-560f4b1cf000 r--p 00000000 09:03 7647243 /usr/bin/dash 560f4b1cf000-560f4b1e3000 r-xp 00004000 09:03 7647243 /usr/bin/dash 560f4b1e3000-560f4b1e9000 r--p 00018000 09:03 7647243 /usr/bin/dash 560f4b1e9000-560f4b1eb000 r--p 0001d000 09:03 7647243 /usr/bin/dash 560f4b1eb000-560f4b1ec000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 560f4b1ec000-560f4b1ee000 rw-p 00000000 00:00 0 560f799a8000-560f799c9000 rw-p 00000000 00:00 0 [heap] 7fba43e2d000-7fba43e30000 rw-p 00000000 00:00 0 7fba43e30000-7fba43e58000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fba43e58000-7fba43fbc000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fba43fbc000-7fba44012000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fba44012000-7fba44016000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fba44016000-7fba44018000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fba44018000-7fba44025000 rw-p 00000000 00:00 0 7fba44033000-7fba44035000 rw-p 00000000 00:00 0 7fba44035000-7fba44036000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fba44036000-7fba4405e000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fba4405e000-7fba44069000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fba44069000-7fba4406b000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fba4406b000-7fba4406c000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fba4406c000-7fba4406d000 rw-p 00000000 00:00 0 7ffdb5ec0000-7ffdb5ee1000 rw-p 00000000 00:00 0 [stack] 7ffdb5f41000-7ffdb5f45000 r--p 00000000 00:00 0 [vvar] 7ffdb5f45000-7ffdb5f47000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 64 detected long write(fd=2, count=114, ret=114): 410768 us PESPIN: vfs_write delta=410764 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120154040027 DL1P NOTICE (bts=0,trx=0,ts=6,ss=7) SACCH no SI available (lchan.c:284) stack trace of write(): 0x7f1f6ad3b687 0 PESPIN: PID 64 detected long write(fd=2, count=136, ret=136): 2013098 us PESPIN: vfs_write delta=2013092 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120154102890 DABIS NOTICE A-bis connection establishment to BSC (172.18.14.11) in progress... (abis.c:170) stack trace of write(): 0x7f1f6ad3b687 0 PESPIN: PID 64 detected long write(fd=2, count=184, ret=184): 645826 us PESPIN: vfs_write delta=645818 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120154111066 DL1P NOTICE 000244/00/10/40/36 (bts=0,trx=0,ts=5) TCH/H(0): Received bad data (rc=-1, BER 0/211) ending at fn=36/104 (sched_lchan_tchh.c:350) stack trace of write(): 0x7f1f6ad3b687 0 PESPIN: PID 64 detected long write(fd=2, count=128, ret=128): 2697675 us PESPIN: vfs_write delta=2697672 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120154312007 DTRX INFO phy0.0: Clock indication: fn=10608 (trx_if.c:103) stack trace of write(): 0x7f1f6ad3b687 0 PESPIN: PID 64 detected long write(fd=2, count=128, ret=128): 418867 us PESPIN: vfs_write delta=418866 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120154652352 DPAG INFO Removed paging record, queue_len=2 (paging.c:825) stack trace of write(): 0x7f1f6ad3b687 0 PESPIN: PID 64 detected long write(fd=2, count=127, ret=127): 503157 us PESPIN: vfs_write delta=503150 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120154740977 DTRX INFO phy0.0: Clock indication: fn=2448 (trx_if.c:103) stack trace of write(): 0x7f1f6ad3b687 0 PESPIN: PID 64 detected long write(fd=2, count=136, ret=136): 1454239 us PESPIN: vfs_write delta=1454231 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120154808097 DABIS NOTICE A-bis connection establishment to BSC (172.18.14.11) in progress... (abis.c:170) stack trace of write(): 0x7f1f6ad3b687 0 PESPIN: PID 64 detected long write(fd=2, count=130, ret=130): 703422 us PESPIN: vfs_write delta=703418 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120154826593 DPAG INFO Removed paging record, queue_len=199 (paging.c:825) stack trace of write(): 0x7f1f6ad3b687 0 PESPIN: PID 64 detected long write(fd=2, count=126, ret=126): 1356519 us PESPIN: vfs_write delta=1356510 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120155055482 DTRX INFO phy0.0: Clock indication: fn=612 (trx_if.c:103) stack trace of write(): 0x7f1f6ad3b687 0 @[ 0x7f1f6ad3b687 0 ]: 9 @latency[2]: [0] 5 | | [1] 46135 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 7769 |@@@@@@@@ | [4, 8) 5537 |@@@@@@ | [8, 16) 2363 |@@ | [16, 32) 513 | | [32, 64) 56 | | [64, 128) 42 | | [128, 256) 19 | | [256, 512) 8 | | [512, 1K) 2 | | [1K, 2K) 1 | | [2K, 4K) 1 | | [4K, 8K) 5 | | [8K, 16K) 9 | | [16K, 32K) 11 | | [32K, 64K) 8 | | [64K, 128K) 12 | | [128K, 256K) 14 | | [256K, 512K) 4 | | [512K, 1M) 2 | | [1M, 2M) 3 | | [2M, 4M) 1 | | @latency[6]: [1] 14689 | | [2, 4) 1164025 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 1043054 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 148556 |@@@@@@ | [16, 32) 151895 |@@@@@@ | [32, 64) 33725 |@ | [64, 128) 1352 | | [128, 256) 145 | | [256, 512) 0 | | [512, 1K) 0 | | [1K, 2K) 2 | | [Tue Jan 20 15:50:57 UTC 2026] respawn: 1: stopped pid 84 with status 0 [Tue Jan 20 15:50:57 UTC 2026] respawn: sleeping 1 seconds... [Tue Jan 20 15:50:58 UTC 2026] respawn: 2: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 112 osmo-bts-trx Attached 5 probes Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 132 root cwd DIR 9,3 4096 13773903 /data sys_write 132 root rtd DIR 0,75 4096 18112779 / sys_write 132 root txt REG 0,75 129736 7647243 /usr/bin/dash sys_write 132 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,75) sys_write 132 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,75) sys_write 132 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,75) sys_write 132 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 132 root 1w FIFO W 0,13 0t0 4120204 pipe sys_write 132 root 2w FIFO W 0,13 0t0 4120205 pipe sys_write 132 root 10r REG LG,CX 9,3 109 13773935 /data/bpftrace/sys_write_delay.sh maps output: 5622289e8000-5622289ec000 r--p 00000000 09:03 7647243 /usr/bin/dash 5622289ec000-562228a00000 r-xp 00004000 09:03 7647243 /usr/bin/dash 562228a00000-562228a06000 r--p 00018000 09:03 7647243 /usr/bin/dash 562228a06000-562228a08000 r--p 0001d000 09:03 7647243 /usr/bin/dash 562228a08000-562228a09000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 562228a09000-562228a0b000 rw-p 00000000 00:00 0 5622455dd000-5622455fe000 rw-p 00000000 00:00 0 [heap] 7f64ffa56000-7f64ffa59000 rw-p 00000000 00:00 0 7f64ffa59000-7f64ffa81000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f64ffa81000-7f64ffbe5000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f64ffbe5000-7f64ffc3b000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f64ffc3b000-7f64ffc3f000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f64ffc3f000-7f64ffc41000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f64ffc41000-7f64ffc4e000 rw-p 00000000 00:00 0 7f64ffc5c000-7f64ffc5e000 rw-p 00000000 00:00 0 7f64ffc5e000-7f64ffc5f000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f64ffc5f000-7f64ffc87000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f64ffc87000-7f64ffc92000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f64ffc92000-7f64ffc94000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f64ffc94000-7f64ffc95000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f64ffc95000-7f64ffc96000 rw-p 00000000 00:00 0 7ffe957ae000-7ffe957cf000 rw-p 00000000 00:00 0 [stack] 7ffe957e7000-7ffe957eb000 r--p 00000000 00:00 0 [vvar] 7ffe957eb000-7ffe957ed000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 112 detected long write(fd=2, count=136, ret=136): 1796347 us PESPIN: vfs_write delta=1796340 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120155545333 DABIS NOTICE A-bis connection establishment to BSC (172.18.14.11) in progress... (abis.c:170) stack trace of write(): 0x7f8ef71b3687 0 @[ 0x7f8ef71b3687 0 ]: 1 @latency[2]: [1] 23088 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 4170 |@@@@@@@@@ | [4, 8) 2271 |@@@@@ | [8, 16) 363 | | [16, 32) 128 | | [32, 64) 36 | | [64, 128) 18 | | [128, 256) 10 | | [256, 512) 2 | | [512, 1K) 1 | | [1K, 2K) 0 | | [2K, 4K) 0 | | [4K, 8K) 2 | | [8K, 16K) 1 | | [16K, 32K) 3 | | [32K, 64K) 4 | | [64K, 128K) 2 | | [128K, 256K) 1 | | [256K, 512K) 0 | | [512K, 1M) 0 | | [1M, 2M) 1 | | @latency[6]: [1] 1589 | | [2, 4) 550130 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 117180 |@@@@@@@@@@@ | [8, 16) 32176 |@@@ | [16, 32) 41525 |@@@ | [32, 64) 2707 | | [64, 128) 209 | | [128, 256) 2 | | [256, 512) 1 | | [Tue Jan 20 15:57:23 UTC 2026] respawn: 2: stopped pid 132 with status 0 [Tue Jan 20 15:57:23 UTC 2026] respawn: sleeping 1 seconds... [Tue Jan 20 15:57:24 UTC 2026] respawn: 3: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 160 osmo-bts-trx Attached 5 probes Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 180 root cwd DIR 9,3 4096 13773903 /data sys_write 180 root rtd DIR 0,75 4096 18112779 / sys_write 180 root txt REG 0,75 129736 7647243 /usr/bin/dash sys_write 180 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,75) sys_write 180 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,75) sys_write 180 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,75) sys_write 180 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 180 root 1w FIFO W 0,13 0t0 4120204 pipe sys_write 180 root 2w FIFO W 0,13 0t0 4120205 pipe sys_write 180 root 10r REG LG,CX 9,3 109 13773935 /data/bpftrace/sys_write_delay.sh maps output: 563340d16000-563340d1a000 r--p 00000000 09:03 7647243 /usr/bin/dash 563340d1a000-563340d2e000 r-xp 00004000 09:03 7647243 /usr/bin/dash 563340d2e000-563340d34000 r--p 00018000 09:03 7647243 /usr/bin/dash 563340d34000-563340d36000 r--p 0001d000 09:03 7647243 /usr/bin/dash 563340d36000-563340d37000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 563340d37000-563340d39000 rw-p 00000000 00:00 0 5633746dd000-5633746fe000 rw-p 00000000 00:00 0 [heap] 7f25f0e73000-7f25f0e76000 rw-p 00000000 00:00 0 7f25f0e76000-7f25f0e9e000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f25f0e9e000-7f25f1002000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f25f1002000-7f25f1058000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f25f1058000-7f25f105c000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f25f105c000-7f25f105e000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f25f105e000-7f25f106b000 rw-p 00000000 00:00 0 7f25f1079000-7f25f107b000 rw-p 00000000 00:00 0 7f25f107b000-7f25f107c000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f25f107c000-7f25f10a4000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f25f10a4000-7f25f10af000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f25f10af000-7f25f10b1000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f25f10b1000-7f25f10b2000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f25f10b2000-7f25f10b3000 rw-p 00000000 00:00 0 7ffc54577000-7ffc54598000 rw-p 00000000 00:00 0 [stack] 7ffc545af000-7ffc545b3000 r--p 00000000 00:00 0 [vvar] 7ffc545b3000-7ffc545b5000 r-xp 00000000 00:00 0 [vdso] @latency[22]: [8, 16) 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| @latency[2]: [0] 3 | | [1] 69871 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 25837 |@@@@@@@@@@@@@@@@@@@ | [4, 8) 22655 |@@@@@@@@@@@@@@@@ | [8, 16) 7667 |@@@@@ | [16, 32) 3065 |@@ | [32, 64) 513 | | [64, 128) 47 | | [128, 256) 21 | | [256, 512) 10 | | [512, 1K) 4 | | [1K, 2K) 2 | | [2K, 4K) 6 | | [4K, 8K) 3 | | [8K, 16K) 8 | | [16K, 32K) 6 | | [32K, 64K) 3 | | [64K, 128K) 5 | | [128K, 256K) 3 | | [256K, 512K) 1 | | @latency[6]: [1] 11410 | | [2, 4) 1353708 |@@@@@@@@@@@@@@@@@@@@@ | [4, 8) 543596 |@@@@@@@@ | [8, 16) 252182 |@@@@ | [16, 32) 3237055 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [32, 64) 18822 | | [64, 128) 19665 | | [128, 256) 69 | | [256, 512) 4 | | [512, 1K) 1 | | [Tue Jan 20 16:33:13 UTC 2026] respawn: 3: stopped pid 180 with status 0 [Tue Jan 20 16:33:13 UTC 2026] respawn: sleeping 1 seconds... [Tue Jan 20 16:33:15 UTC 2026] respawn: 0: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 9 osmo-bts-trx Attached 5 probes Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 25 root cwd DIR 9,3 4096 13773903 /data sys_write 25 root rtd DIR 0,73 4096 18112753 / sys_write 25 root txt REG 0,73 129736 7647243 /usr/bin/dash sys_write 25 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,73) sys_write 25 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,73) sys_write 25 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,73) sys_write 25 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 25 root 1w FIFO W 0,13 0t0 5540917 pipe sys_write 25 root 2w FIFO W 0,13 0t0 5540918 pipe sys_write 25 root 10r REG LG,CX 9,3 109 13773946 /data/bpftrace/sys_write_delay.sh maps output: 55ed78898000-55ed7889c000 r--p 00000000 09:03 7647243 /usr/bin/dash 55ed7889c000-55ed788b0000 r-xp 00004000 09:03 7647243 /usr/bin/dash 55ed788b0000-55ed788b6000 r--p 00018000 09:03 7647243 /usr/bin/dash 55ed788b6000-55ed788b8000 r--p 0001d000 09:03 7647243 /usr/bin/dash 55ed788b8000-55ed788b9000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 55ed788b9000-55ed788bb000 rw-p 00000000 00:00 0 55edb4116000-55edb4137000 rw-p 00000000 00:00 0 [heap] 7f72fd8ee000-7f72fd8f1000 rw-p 00000000 00:00 0 7f72fd8f1000-7f72fd919000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f72fd919000-7f72fda7d000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f72fda7d000-7f72fdad3000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f72fdad3000-7f72fdad7000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f72fdad7000-7f72fdad9000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7f72fdad9000-7f72fdae6000 rw-p 00000000 00:00 0 7f72fdaf4000-7f72fdaf6000 rw-p 00000000 00:00 0 7f72fdaf6000-7f72fdaf7000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f72fdaf7000-7f72fdb1f000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f72fdb1f000-7f72fdb2a000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f72fdb2a000-7f72fdb2c000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f72fdb2c000-7f72fdb2d000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f72fdb2d000-7f72fdb2e000 rw-p 00000000 00:00 0 7fffc5f3a000-7fffc5f5b000 rw-p 00000000 00:00 0 [stack] 7fffc5ff1000-7fffc5ff5000 r--p 00000000 00:00 0 [vvar] 7fffc5ff5000-7fffc5ff7000 r-xp 00000000 00:00 0 [vdso] @latency[2]: [1] 602 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 124 |@@@@@@@@@@ | [4, 8) 222 |@@@@@@@@@@@@@@@@@@@ | [8, 16) 38 |@@@ | [16, 32) 28 |@@ | [32, 64) 12 |@ | @latency[6]: [1] 1490 |@@@@@@ | [2, 4) 11278 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 2632 |@@@@@@@@@@@@ | [8, 16) 2300 |@@@@@@@@@@ | [16, 32) 2478 |@@@@@@@@@@@ | [32, 64) 36 | | [64, 128) 35 | | [128, 256) 1 | | [Tue Jan 20 16:34:53 UTC 2026] respawn: 0: stopped pid 25 with status 0 [Tue Jan 20 16:34:53 UTC 2026] respawn: sleeping 1 seconds... [Tue Jan 20 16:34:54 UTC 2026] respawn: 0: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 9 osmo-bts-trx Attached 5 probes Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 27 root cwd DIR 9,3 4096 13773903 /data sys_write 27 root rtd DIR 0,75 4096 18112819 / sys_write 27 root txt REG 0,75 129736 7647243 /usr/bin/dash sys_write 27 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,75) sys_write 27 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,75) sys_write 27 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,75) sys_write 27 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 27 root 1w FIFO W 0,13 0t0 5505875 pipe sys_write 27 root 2w FIFO W 0,13 0t0 5505876 pipe sys_write 27 root 10r REG LG,CX 9,3 109 13773923 /data/bpftrace/sys_write_delay.sh maps output: 55a451072000-55a451076000 r--p 00000000 09:03 7647243 /usr/bin/dash 55a451076000-55a45108a000 r-xp 00004000 09:03 7647243 /usr/bin/dash 55a45108a000-55a451090000 r--p 00018000 09:03 7647243 /usr/bin/dash 55a451090000-55a451092000 r--p 0001d000 09:03 7647243 /usr/bin/dash 55a451092000-55a451093000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 55a451093000-55a451095000 rw-p 00000000 00:00 0 55a45249b000-55a4524bc000 rw-p 00000000 00:00 0 [heap] 7fe860926000-7fe860929000 rw-p 00000000 00:00 0 7fe860929000-7fe860951000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe860951000-7fe860ab5000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe860ab5000-7fe860b0b000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe860b0b000-7fe860b0f000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe860b0f000-7fe860b11000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe860b11000-7fe860b1e000 rw-p 00000000 00:00 0 7fe860b2c000-7fe860b2e000 rw-p 00000000 00:00 0 7fe860b2e000-7fe860b2f000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe860b2f000-7fe860b57000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe860b57000-7fe860b62000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe860b62000-7fe860b64000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe860b64000-7fe860b65000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe860b65000-7fe860b66000 rw-p 00000000 00:00 0 7ffcf457d000-7ffcf459e000 rw-p 00000000 00:00 0 [stack] 7ffcf45f0000-7ffcf45f4000 r--p 00000000 00:00 0 [vvar] 7ffcf45f4000-7ffcf45f6000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 9 detected long write(fd=2, count=128, ret=128): 647610 us PESPIN: vfs_write delta=647601 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260120170440496 DL1C INFO (bts=0,trx=0) ramping TRX board output power to 8000 mdBm. (tx_power.c:181) stack trace of write(): 0x7ff86e12e687 0 @[ 0x7ff86e12e687 0 ]: 1 @latency[2]: [0] 2 | | [1] 49050 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 21068 |@@@@@@@@@@@@@@@@@@@@@@ | [4, 8) 35098 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 14711 |@@@@@@@@@@@@@@@ | [16, 32) 4995 |@@@@@ | [32, 64) 741 | | [64, 128) 28 | | [128, 256) 13 | | [256, 512) 16 | | [512, 1K) 2 | | [1K, 2K) 1 | | [2K, 4K) 0 | | [4K, 8K) 2 | | [8K, 16K) 3 | | [16K, 32K) 1 | | [32K, 64K) 1 | | [64K, 128K) 3 | | [128K, 256K) 2 | | [256K, 512K) 0 | | [512K, 1M) 1 | | @latency[6]: [1] 11267 | | [2, 4) 782391 |@@@@@@@@@@@@ | [4, 8) 472649 |@@@@@@@ | [8, 16) 386217 |@@@@@@ | [16, 32) 3343091 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [32, 64) 36361 | | [64, 128) 30475 | | [128, 256) 107 | | [256, 512) 5 | | [Tue Jan 20 17:04:41 UTC 2026] respawn: 0: stopped pid 27 with status 0 [Tue Jan 20 17:04:41 UTC 2026] respawn: sleeping 1 seconds... [Tue Jan 20 17:04:42 UTC 2026] respawn: 1: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 67 osmo-bts-trx Tracing sys_write() latency... Hit Ctrl-C to end. Attached 5 probes lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 71 root cwd DIR 9,3 4096 13773903 /data sys_write 71 root rtd DIR 0,75 4096 18112819 / sys_write 71 root txt REG 0,75 129736 7647243 /usr/bin/dash sys_write 71 root mem REG 9,3 7647243 /usr/bin/dash (path dev=0,75) sys_write 71 root mem REG 9,3 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,75) sys_write 71 root mem REG 9,3 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,75) sys_write 71 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 71 root 1w FIFO W 0,13 0t0 5505875 pipe sys_write 71 root 2w FIFO W 0,13 0t0 5505876 pipe sys_write 71 root 10r REG LG,CX 9,3 109 13773923 /data/bpftrace/sys_write_delay.sh maps output: 564a07ea6000-564a07eaa000 r--p 00000000 09:03 7647243 /usr/bin/dash 564a07eaa000-564a07ebe000 r-xp 00004000 09:03 7647243 /usr/bin/dash 564a07ebe000-564a07ec4000 r--p 00018000 09:03 7647243 /usr/bin/dash 564a07ec4000-564a07ec6000 r--p 0001d000 09:03 7647243 /usr/bin/dash 564a07ec6000-564a07ec7000 rw-p 0001f000 09:03 7647243 /usr/bin/dash 564a07ec7000-564a07ec9000 rw-p 00000000 00:00 0 564a1cf0e000-564a1cf2f000 rw-p 00000000 00:00 0 [heap] 7fe70f290000-7fe70f293000 rw-p 00000000 00:00 0 7fe70f293000-7fe70f2bb000 r--p 00000000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe70f2bb000-7fe70f41f000 r-xp 00028000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe70f41f000-7fe70f475000 r--p 0018c000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe70f475000-7fe70f479000 r--p 001e1000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe70f479000-7fe70f47b000 rw-p 001e5000 09:03 7647837 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe70f47b000-7fe70f488000 rw-p 00000000 00:00 0 7fe70f496000-7fe70f498000 rw-p 00000000 00:00 0 7fe70f498000-7fe70f499000 r--p 00000000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe70f499000-7fe70f4c1000 r-xp 00001000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe70f4c1000-7fe70f4cc000 r--p 00029000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe70f4cc000-7fe70f4ce000 r--p 00034000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe70f4ce000-7fe70f4cf000 rw-p 00036000 09:03 7647817 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe70f4cf000-7fe70f4d0000 rw-p 00000000 00:00 0 7ffde02ac000-7ffde02cd000 rw-p 00000000 00:00 0 [stack] 7ffde02ff000-7ffde0303000 r--p 00000000 00:00 0 [vvar] 7ffde0303000-7ffde0305000 r-xp 00000000 00:00 0 [vdso] @latency[22]: [32, 64) 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| @latency[2]: [0] 2 | | [1] 52775 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 27769 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [4, 8) 30950 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 11773 |@@@@@@@@@@@ | [16, 32) 8684 |@@@@@@@@ | [32, 64) 1459 |@ | [64, 128) 24 | | [128, 256) 15 | | [256, 512) 11 | | [512, 1K) 1 | | [1K, 2K) 1 | | [2K, 4K) 1 | | [4K, 8K) 5 | | [8K, 16K) 1 | | @latency[6]: [1] 20392 | | [2, 4) 426405 |@@@@@ | [4, 8) 609126 |@@@@@@@@ | [8, 16) 367194 |@@@@@ | [16, 32) 3699511 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [32, 64) 46613 | | [64, 128) 67858 | | [128, 256) 71 | | [256, 512) 1 | | [Tue Jan 20 17:40:56 UTC 2026] respawn: 1: stopped pid 71 with status 0 [Tue Jan 20 17:40:56 UTC 2026] respawn: sleeping 1 seconds...