[Fri Jan 16 12:51:58 UTC 2026] respawn: 0: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 9 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 25 root cwd DIR 9,3 4096 25887107 /data sys_write 25 root rtd DIR 0,146 4096 25527869 / sys_write 25 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 25 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 25 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 25 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 25 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 25 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 25 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 25 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55becafd3000-55becafd7000 r--p 00000000 09:03 25259300 /usr/bin/dash 55becafd7000-55becafeb000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55becafeb000-55becaff1000 r--p 00018000 09:03 25259300 /usr/bin/dash 55becaff1000-55becaff3000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55becaff3000-55becaff4000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55becaff4000-55becaff6000 rw-p 00000000 00:00 0 55becbf2f000-55becbf50000 rw-p 00000000 00:00 0 [heap] 7f43674e6000-7f43674e9000 rw-p 00000000 00:00 0 7f43674e9000-7f4367511000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f4367511000-7f4367675000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f4367675000-7f43676cb000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f43676cb000-7f43676cf000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f43676cf000-7f43676d1000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f43676d1000-7f43676de000 rw-p 00000000 00:00 0 7f43676ec000-7f43676ee000 rw-p 00000000 00:00 0 7f43676ee000-7f43676ef000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f43676ef000-7f4367717000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f4367717000-7f4367722000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f4367722000-7f4367724000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f4367724000-7f4367725000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f4367725000-7f4367726000 rw-p 00000000 00:00 0 7fff1e35f000-7fff1e380000 rw-p 00000000 00:00 0 [stack] 7fff1e3f5000-7fff1e3f9000 r--p 00000000 00:00 0 [vvar] 7fff1e3f9000-7fff1e3fb000 r-xp 00000000 00:00 0 [vdso] @latency[2]: [1] 14169 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 1332 |@@@@ | [4, 8) 809 |@@ | [8, 16) 194 | | [16, 32) 24 | | [32, 64) 5 | | [64, 128) 8 | | [128, 256) 4 | | [256, 512) 2 | | [512, 1K) 2 | | [1K, 2K) 0 | | [2K, 4K) 2 | | [4K, 8K) 0 | | [8K, 16K) 1 | | [16K, 32K) 1 | | [32K, 64K) 0 | | [64K, 128K) 0 | | [128K, 256K) 1 | | @latency[6]: [1] 285 | | [2, 4) 86439 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 20152 |@@@@@@@@@@@@ | [8, 16) 5342 |@@@ | [16, 32) 3330 |@@ | [32, 64) 911 | | [64, 128) 23 | | [Fri Jan 16 12:52:53 UTC 2026] respawn: 0: stopped pid 25 with status 0 [Fri Jan 16 12:52:53 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 12:52:54 UTC 2026] respawn: 1: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 60 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. PESPIN: PID 60 detected long write(fd=2, count=137, ret=137): 626585 us PESPIN: vfs_write delta=1171 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116125258787 DABIS NOTICE A-bis connection establishment to BSC (172.18.186.11) in progress... (abis.c:170) stack trace of write(): 0x7f7d98675687 0 lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 70 root cwd DIR 9,3 4096 25887107 /data sys_write 70 root rtd DIR 0,146 4096 25527869 / sys_write 70 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 70 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 70 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 70 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 70 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 70 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 70 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 70 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 562f3fa91000-562f3fa95000 r--p 00000000 09:03 25259300 /usr/bin/dash 562f3fa95000-562f3faa9000 r-xp 00004000 09:03 25259300 /usr/bin/dash 562f3faa9000-562f3faaf000 r--p 00018000 09:03 25259300 /usr/bin/dash 562f3faaf000-562f3fab1000 r--p 0001d000 09:03 25259300 /usr/bin/dash 562f3fab1000-562f3fab2000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 562f3fab2000-562f3fab4000 rw-p 00000000 00:00 0 562f401bc000-562f401dd000 rw-p 00000000 00:00 0 [heap] 7ff71aeb9000-7ff71aebc000 rw-p 00000000 00:00 0 7ff71aebc000-7ff71aee4000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff71aee4000-7ff71b048000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff71b048000-7ff71b09e000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff71b09e000-7ff71b0a2000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff71b0a2000-7ff71b0a4000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff71b0a4000-7ff71b0b1000 rw-p 00000000 00:00 0 7ff71b0bf000-7ff71b0c1000 rw-p 00000000 00:00 0 7ff71b0c1000-7ff71b0c2000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff71b0c2000-7ff71b0ea000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff71b0ea000-7ff71b0f5000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff71b0f5000-7ff71b0f7000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff71b0f7000-7ff71b0f8000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff71b0f8000-7ff71b0f9000 rw-p 00000000 00:00 0 7ffc2e678000-7ffc2e699000 rw-p 00000000 00:00 0 [stack] 7ffc2e750000-7ffc2e754000 r--p 00000000 00:00 0 [vvar] 7ffc2e754000-7ffc2e756000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 60 detected long write(fd=2, count=229, ret=229): 645355 us PESPIN: vfs_write delta=645350 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116125311395 DMEAS INFO (bts=0,trx=0,ts=6,ss=1) Computed TA256( 512), BER-FULL(33.33%), RSSI-FULL(- 60dBm), C/I-FULL( 60 cB), BER-SUB(33.33%), RSSI-SUB(- 60dBm), C/I-SUB( 60 cB) (measurement.c:732) stack trace of write(): 0x7f7d98675687 0 PESPIN: PID 60 detected long write(fd=2, count=229, ret=229): 625483 us PESPIN: vfs_write delta=625477 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116125647300 DMEAS INFO (bts=0,trx=0,ts=6,ss=3) Computed TA256( 512), BER-FULL( 0.00%), RSSI-FULL(- 60dBm), C/I-FULL( 60 cB), BER-SUB( 0.00%), RSSI-SUB(- 60dBm), C/I-SUB( 60 cB) (measurement.c:732) stack trace of write(): 0x7f7d98675687 0 PESPIN: PID 60 detected long write(fd=2, count=120, ret=120): 1595063 us PESPIN: vfs_write delta=1595059 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116125705144 DRSL INFO Tx RSL RF RESource INDication (rsl.c:519) stack trace of write(): 0x7f7d98675687 0 @[ 0x7f7d98675687 0 ]: 4 @latency[2]: [1] 16424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 5141 |@@@@@@@@@@@@@@@@ | [4, 8) 4474 |@@@@@@@@@@@@@@ | [8, 16) 2255 |@@@@@@@ | [16, 32) 1023 |@@@ | [32, 64) 131 | | [64, 128) 18 | | [128, 256) 13 | | [256, 512) 11 | | [512, 1K) 3 | | [1K, 2K) 1 | | [2K, 4K) 1 | | [4K, 8K) 1 | | [8K, 16K) 3 | | [16K, 32K) 4 | | [32K, 64K) 3 | | [64K, 128K) 6 | | [128K, 256K) 2 | | [256K, 512K) 1 | | [512K, 1M) 3 | | [1M, 2M) 1 | | @latency[6]: [1] 12162 | | [2, 4) 819854 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 337684 |@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 203797 |@@@@@@@@@@@@ | [16, 32) 244041 |@@@@@@@@@@@@@@@ | [32, 64) 10579 | | [64, 128) 1118 | | [128, 256) 25 | | [Fri Jan 16 13:00:22 UTC 2026] respawn: 1: stopped pid 70 with status 0 [Fri Jan 16 13:00:22 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:00:23 UTC 2026] respawn: 2: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 101 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 105 root cwd DIR 9,3 4096 25887107 /data sys_write 105 root rtd DIR 0,146 4096 25527869 / sys_write 105 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 105 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 105 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 105 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 105 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 105 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 105 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 105 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 562305425000-562305429000 r--p 00000000 09:03 25259300 /usr/bin/dash 562305429000-56230543d000 r-xp 00004000 09:03 25259300 /usr/bin/dash 56230543d000-562305443000 r--p 00018000 09:03 25259300 /usr/bin/dash 562305443000-562305445000 r--p 0001d000 09:03 25259300 /usr/bin/dash 562305445000-562305446000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 562305446000-562305448000 rw-p 00000000 00:00 0 56230676a000-56230678b000 rw-p 00000000 00:00 0 [heap] 7f693720e000-7f6937211000 rw-p 00000000 00:00 0 7f6937211000-7f6937239000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f6937239000-7f693739d000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f693739d000-7f69373f3000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f69373f3000-7f69373f7000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f69373f7000-7f69373f9000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f69373f9000-7f6937406000 rw-p 00000000 00:00 0 7f6937414000-7f6937416000 rw-p 00000000 00:00 0 7f6937416000-7f6937417000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f6937417000-7f693743f000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f693743f000-7f693744a000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f693744a000-7f693744c000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f693744c000-7f693744d000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f693744d000-7f693744e000 rw-p 00000000 00:00 0 7ffeaaddb000-7ffeaadfc000 rw-p 00000000 00:00 0 [stack] 7ffeaafaa000-7ffeaafae000 r--p 00000000 00:00 0 [vvar] 7ffeaafae000-7ffeaafb0000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 101 detected long write(fd=2, count=127, ret=127): 1983465 us PESPIN: vfs_write delta=1983460 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116130125959 DTRX INFO phy0.0: Clock indication: fn=2958 (trx_if.c:103) stack trace of write(): 0x7ff7af033687 0 PESPIN: PID 101 detected long write(fd=2, count=127, ret=127): 816306 us PESPIN: vfs_write delta=816298 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116130316264 DTRX INFO phy0.0: Clock indication: fn=5916 (trx_if.c:103) stack trace of write(): 0x7ff7af033687 0 PESPIN: PID 101 detected long write(fd=2, count=127, ret=127): 647887 us PESPIN: vfs_write delta=647875 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116130338551 DTRX INFO phy0.0: Clock indication: fn=3570 (trx_if.c:103) stack trace of write(): 0x7ff7af033687 0 @[ 0x7ff7af033687 0 ]: 3 @latency[2]: [1] 9136 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 2344 |@@@@@@@@@@@@@ | [4, 8) 2382 |@@@@@@@@@@@@@ | [8, 16) 490 |@@ | [16, 32) 89 | | [32, 64) 12 | | [64, 128) 13 | | [128, 256) 1 | | [256, 512) 3 | | [512, 1K) 0 | | [1K, 2K) 1 | | [2K, 4K) 1 | | [4K, 8K) 4 | | [8K, 16K) 4 | | [16K, 32K) 4 | | [32K, 64K) 1 | | [64K, 128K) 2 | | [128K, 256K) 1 | | [256K, 512K) 2 | | [512K, 1M) 2 | | [1M, 2M) 1 | | @latency[6]: [1] 1694 | | [2, 4) 464620 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 60852 |@@@@@@ | [8, 16) 23435 |@@ | [16, 32) 26737 |@@ | [32, 64) 2231 | | [64, 128) 60 | | [128, 256) 0 | | [256, 512) 0 | | [512, 1K) 1 | | [Fri Jan 16 13:03:39 UTC 2026] respawn: 2: stopped pid 105 with status 0 [Fri Jan 16 13:03:39 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:03:40 UTC 2026] respawn: 3: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 142 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 152 root cwd DIR 9,3 4096 25887107 /data sys_write 152 root rtd DIR 0,146 4096 25527869 / sys_write 152 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 152 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 152 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 152 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 152 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 152 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 152 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 152 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55a8b9713000-55a8b9717000 r--p 00000000 09:03 25259300 /usr/bin/dash 55a8b9717000-55a8b972b000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55a8b972b000-55a8b9731000 r--p 00018000 09:03 25259300 /usr/bin/dash 55a8b9731000-55a8b9733000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55a8b9733000-55a8b9734000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55a8b9734000-55a8b9736000 rw-p 00000000 00:00 0 55a8b9da0000-55a8b9dc1000 rw-p 00000000 00:00 0 [heap] 7fe0d772e000-7fe0d7731000 rw-p 00000000 00:00 0 7fe0d7731000-7fe0d7759000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe0d7759000-7fe0d78bd000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe0d78bd000-7fe0d7913000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe0d7913000-7fe0d7917000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe0d7917000-7fe0d7919000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fe0d7919000-7fe0d7926000 rw-p 00000000 00:00 0 7fe0d7934000-7fe0d7936000 rw-p 00000000 00:00 0 7fe0d7936000-7fe0d7937000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe0d7937000-7fe0d795f000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe0d795f000-7fe0d796a000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe0d796a000-7fe0d796c000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe0d796c000-7fe0d796d000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fe0d796d000-7fe0d796e000 rw-p 00000000 00:00 0 7ffe27274000-7ffe27295000 rw-p 00000000 00:00 0 [stack] 7ffe272a2000-7ffe272a6000 r--p 00000000 00:00 0 [vvar] 7ffe272a6000-7ffe272a8000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 142 detected long write(fd=2, count=127, ret=127): 434306 us PESPIN: vfs_write delta=434303 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116130542074 DTRX INFO phy0.0: Clock indication: fn=1020 (trx_if.c:103) stack trace of write(): 0x7f5be9d01687 0 PESPIN: PID 142 detected long write(fd=2, count=127, ret=127): 482369 us PESPIN: vfs_write delta=482362 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116130602456 DTRX INFO phy0.0: Clock indication: fn=2142 (trx_if.c:103) stack trace of write(): 0x7f5be9d01687 0 @[ 0x7f5be9d01687 0 ]: 2 @latency[2]: [1] 6058 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 1597 |@@@@@@@@@@@@@ | [4, 8) 1469 |@@@@@@@@@@@@ | [8, 16) 445 |@@@ | [16, 32) 85 | | [32, 64) 26 | | [64, 128) 8 | | [128, 256) 3 | | [256, 512) 2 | | [512, 1K) 2 | | [1K, 2K) 1 | | [2K, 4K) 0 | | [4K, 8K) 1 | | [8K, 16K) 1 | | [16K, 32K) 2 | | [32K, 64K) 3 | | [64K, 128K) 0 | | [128K, 256K) 1 | | [256K, 512K) 4 | | @latency[6]: [1] 1786 | | [2, 4) 398673 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 132324 |@@@@@@@@@@@@@@@@@ | [8, 16) 26525 |@@@ | [16, 32) 37713 |@@@@ | [32, 64) 2676 | | [64, 128) 254 | | [128, 256) 1 | | [256, 512) 0 | | [512, 1K) 1 | | [1K, 2K) 1 | | [Fri Jan 16 13:06:58 UTC 2026] respawn: 3: stopped pid 152 with status 0 [Fri Jan 16 13:06:58 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:06:59 UTC 2026] respawn: 4: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 183 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 187 root cwd DIR 9,3 4096 25887107 /data sys_write 187 root rtd DIR 0,146 4096 25527869 / sys_write 187 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 187 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 187 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 187 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 187 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 187 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 187 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 187 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55f067444000-55f067448000 r--p 00000000 09:03 25259300 /usr/bin/dash 55f067448000-55f06745c000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55f06745c000-55f067462000 r--p 00018000 09:03 25259300 /usr/bin/dash 55f067462000-55f067464000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55f067464000-55f067465000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55f067465000-55f067467000 rw-p 00000000 00:00 0 55f068bf7000-55f068c18000 rw-p 00000000 00:00 0 [heap] 7f0d6d2d7000-7f0d6d2da000 rw-p 00000000 00:00 0 7f0d6d2da000-7f0d6d302000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0d6d302000-7f0d6d466000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0d6d466000-7f0d6d4bc000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0d6d4bc000-7f0d6d4c0000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0d6d4c0000-7f0d6d4c2000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f0d6d4c2000-7f0d6d4cf000 rw-p 00000000 00:00 0 7f0d6d4dd000-7f0d6d4df000 rw-p 00000000 00:00 0 7f0d6d4df000-7f0d6d4e0000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0d6d4e0000-7f0d6d508000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0d6d508000-7f0d6d513000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0d6d513000-7f0d6d515000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0d6d515000-7f0d6d516000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f0d6d516000-7f0d6d517000 rw-p 00000000 00:00 0 7ffc9ad1a000-7ffc9ad3b000 rw-p 00000000 00:00 0 [stack] 7ffc9ad8e000-7ffc9ad92000 r--p 00000000 00:00 0 [vvar] 7ffc9ad92000-7ffc9ad94000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 183 detected long write(fd=2, count=173, ret=173): 1468501 us PESPIN: vfs_write delta=1468470 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116130706303 DL1P NOTICE PH-RTS-IND: Unable to determine actual BS_AG_BLKS_RES value as SI3 is not available yet, fallback to 1 (sysinfo.c:162) stack trace of write(): 0x7f945954b687 0 PESPIN: PID 183 detected long write(fd=2, count=114, ret=114): 1372361 us PESPIN: vfs_write delta=1372350 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116130820449 DL1P NOTICE (bts=0,trx=0,ts=1,ss=0) SACCH no SI available (lchan.c:284) stack trace of write(): 0x7f945954b687 0 @[ 0x7f945954b687 0 ]: 2 @latency[2]: [1] 7890 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 768 |@@@@@ | [4, 8) 574 |@@@ | [8, 16) 239 |@ | [16, 32) 101 | | [32, 64) 18 | | [64, 128) 4 | | [128, 256) 2 | | [256, 512) 1 | | [512, 1K) 0 | | [1K, 2K) 0 | | [2K, 4K) 0 | | [4K, 8K) 0 | | [8K, 16K) 0 | | [16K, 32K) 0 | | [32K, 64K) 1 | | [64K, 128K) 1 | | [128K, 256K) 1 | | [256K, 512K) 0 | | [512K, 1M) 0 | | [1M, 2M) 2 | | @latency[6]: [1] 9497 |@@@ | [2, 4) 152900 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 44140 |@@@@@@@@@@@@@@@ | [8, 16) 30348 |@@@@@@@@@@ | [16, 32) 36718 |@@@@@@@@@@@@ | [32, 64) 2321 | | [64, 128) 339 | | [128, 256) 2 | | [Fri Jan 16 13:08:22 UTC 2026] respawn: 4: stopped pid 187 with status 0 [Fri Jan 16 13:08:22 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:08:23 UTC 2026] respawn: 5: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 224 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 228 root cwd DIR 9,3 4096 25887107 /data sys_write 228 root rtd DIR 0,146 4096 25527869 / sys_write 228 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 228 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 228 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 228 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 228 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 228 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 228 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 228 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 5562bcc6d000-5562bcc71000 r--p 00000000 09:03 25259300 /usr/bin/dash 5562bcc71000-5562bcc85000 r-xp 00004000 09:03 25259300 /usr/bin/dash 5562bcc85000-5562bcc8b000 r--p 00018000 09:03 25259300 /usr/bin/dash 5562bcc8b000-5562bcc8d000 r--p 0001d000 09:03 25259300 /usr/bin/dash 5562bcc8d000-5562bcc8e000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 5562bcc8e000-5562bcc90000 rw-p 00000000 00:00 0 5562be245000-5562be266000 rw-p 00000000 00:00 0 [heap] 7f81f9908000-7f81f990b000 rw-p 00000000 00:00 0 7f81f990b000-7f81f9933000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f81f9933000-7f81f9a97000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f81f9a97000-7f81f9aed000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f81f9aed000-7f81f9af1000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f81f9af1000-7f81f9af3000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f81f9af3000-7f81f9b00000 rw-p 00000000 00:00 0 7f81f9b0e000-7f81f9b10000 rw-p 00000000 00:00 0 7f81f9b10000-7f81f9b11000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f81f9b11000-7f81f9b39000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f81f9b39000-7f81f9b44000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f81f9b44000-7f81f9b46000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f81f9b46000-7f81f9b47000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f81f9b47000-7f81f9b48000 rw-p 00000000 00:00 0 7ffd89653000-7ffd89674000 rw-p 00000000 00:00 0 [stack] 7ffd896fc000-7ffd89700000 r--p 00000000 00:00 0 [vvar] 7ffd89700000-7ffd89702000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 224 detected long write(fd=2, count=128, ret=128): 2237061 us PESPIN: vfs_write delta=2237059 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116131026483 DPAG INFO Removed paging record, queue_len=3 (paging.c:825) stack trace of write(): 0x7fcc7b34d687 0 @[ 0x7fcc7b34d687 0 ]: 1 @latency[2]: [1] 5310 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 696 |@@@@@@ | [4, 8) 1063 |@@@@@@@@@@ | [8, 16) 361 |@@@ | [16, 32) 143 |@ | [32, 64) 22 | | [64, 128) 4 | | [128, 256) 1 | | [256, 512) 4 | | [512, 1K) 3 | | [1K, 2K) 1 | | [2K, 4K) 0 | | [4K, 8K) 1 | | [8K, 16K) 1 | | [16K, 32K) 2 | | [32K, 64K) 1 | | [64K, 128K) 0 | | [128K, 256K) 2 | | [256K, 512K) 1 | | [512K, 1M) 0 | | [1M, 2M) 0 | | [2M, 4M) 1 | | @latency[6]: [1] 840 | | [2, 4) 241624 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 37842 |@@@@@@@@ | [8, 16) 29582 |@@@@@@ | [16, 32) 50144 |@@@@@@@@@@ | [32, 64) 1031 | | [64, 128) 209 | | [128, 256) 16 | | [Fri Jan 16 13:10:32 UTC 2026] respawn: 5: stopped pid 228 with status 0 [Fri Jan 16 13:10:32 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:10:33 UTC 2026] respawn: 6: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 265 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 275 root cwd DIR 9,3 4096 25887107 /data sys_write 275 root rtd DIR 0,146 4096 25527869 / sys_write 275 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 275 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 275 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 275 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 275 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 275 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 275 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 275 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55e472820000-55e472824000 r--p 00000000 09:03 25259300 /usr/bin/dash 55e472824000-55e472838000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55e472838000-55e47283e000 r--p 00018000 09:03 25259300 /usr/bin/dash 55e47283e000-55e472840000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55e472840000-55e472841000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55e472841000-55e472843000 rw-p 00000000 00:00 0 55e473e51000-55e473e72000 rw-p 00000000 00:00 0 [heap] 7f675a820000-7f675a823000 rw-p 00000000 00:00 0 7f675a823000-7f675a84b000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f675a84b000-7f675a9af000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f675a9af000-7f675aa05000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f675aa05000-7f675aa09000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f675aa09000-7f675aa0b000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f675aa0b000-7f675aa18000 rw-p 00000000 00:00 0 7f675aa26000-7f675aa28000 rw-p 00000000 00:00 0 7f675aa28000-7f675aa29000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f675aa29000-7f675aa51000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f675aa51000-7f675aa5c000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f675aa5c000-7f675aa5e000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f675aa5e000-7f675aa5f000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f675aa5f000-7f675aa60000 rw-p 00000000 00:00 0 7ffe0d80f000-7ffe0d830000 rw-p 00000000 00:00 0 [stack] 7ffe0d9b6000-7ffe0d9ba000 r--p 00000000 00:00 0 [vvar] 7ffe0d9ba000-7ffe0d9bc000 r-xp 00000000 00:00 0 [vdso] @latency[2]: [1] 1259 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 261 |@@@@@@@@@@ | [4, 8) 272 |@@@@@@@@@@@ | [8, 16) 85 |@@@ | [16, 32) 20 | | [32, 64) 0 | | [64, 128) 1 | | [128, 256) 1 | | [256, 512) 2 | | [512, 1K) 0 | | [1K, 2K) 0 | | [2K, 4K) 1 | | [4K, 8K) 0 | | [8K, 16K) 0 | | [16K, 32K) 0 | | [32K, 64K) 0 | | [64K, 128K) 0 | | [128K, 256K) 0 | | [256K, 512K) 1 | | @latency[6]: [1] 121 | | [2, 4) 50290 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 9692 |@@@@@@@@@@ | [8, 16) 4475 |@@@@ | [16, 32) 3938 |@@@@ | [32, 64) 221 | | [64, 128) 9 | | [Fri Jan 16 13:11:01 UTC 2026] respawn: 6: stopped pid 275 with status 0 [Fri Jan 16 13:11:01 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:11:02 UTC 2026] respawn: 7: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 306 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 310 root cwd DIR 9,3 4096 25887107 /data sys_write 310 root rtd DIR 0,146 4096 25527869 / sys_write 310 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 310 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 310 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 310 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 310 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 310 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 310 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 310 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 56459d308000-56459d30c000 r--p 00000000 09:03 25259300 /usr/bin/dash 56459d30c000-56459d320000 r-xp 00004000 09:03 25259300 /usr/bin/dash 56459d320000-56459d326000 r--p 00018000 09:03 25259300 /usr/bin/dash 56459d326000-56459d328000 r--p 0001d000 09:03 25259300 /usr/bin/dash 56459d328000-56459d329000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 56459d329000-56459d32b000 rw-p 00000000 00:00 0 56459eb92000-56459ebb3000 rw-p 00000000 00:00 0 [heap] 7f964397e000-7f9643981000 rw-p 00000000 00:00 0 7f9643981000-7f96439a9000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f96439a9000-7f9643b0d000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f9643b0d000-7f9643b63000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f9643b63000-7f9643b67000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f9643b67000-7f9643b69000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f9643b69000-7f9643b76000 rw-p 00000000 00:00 0 7f9643b84000-7f9643b86000 rw-p 00000000 00:00 0 7f9643b86000-7f9643b87000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9643b87000-7f9643baf000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9643baf000-7f9643bba000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9643bba000-7f9643bbc000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9643bbc000-7f9643bbd000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9643bbd000-7f9643bbe000 rw-p 00000000 00:00 0 7ffce0a2e000-7ffce0a4f000 rw-p 00000000 00:00 0 [stack] 7ffce0af5000-7ffce0af9000 r--p 00000000 00:00 0 [vvar] 7ffce0af9000-7ffce0afb000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 306 detected long write(fd=2, count=169, ret=169): 1076308 us PESPIN: vfs_write delta=1076300 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116131125049 DL1P NOTICE GSMTAP: Unable to determine actual BS_AG_BLKS_RES value as SI3 is not available yet, fallback to 1 (sysinfo.c:162) stack trace of write(): 0x7f1a1f359687 0 PESPIN: PID 306 detected long write(fd=2, count=127, ret=127): 495257 us PESPIN: vfs_write delta=495247 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116131325863 DTRX INFO phy0.0: Clock indication: fn=1224 (trx_if.c:103) stack trace of write(): 0x7f1a1f359687 0 PESPIN: PID 306 detected long write(fd=2, count=120, ret=120): 831698 us PESPIN: vfs_write delta=831689 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116131433944 DRSL INFO Tx RSL RF RESource INDication (rsl.c:519) stack trace of write(): 0x7f1a1f359687 0 PESPIN: PID 306 detected long write(fd=2, count=127, ret=127): 661358 us PESPIN: vfs_write delta=661355 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116131435714 DTRX INFO phy0.0: Clock indication: fn=3366 (trx_if.c:103) stack trace of write(): 0x7f1a1f359687 0 PESPIN: PID 306 detected long write(fd=2, count=127, ret=127): 547955 us PESPIN: vfs_write delta=547952 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116131725603 DTRX INFO phy0.0: Clock indication: fn=1734 (trx_if.c:103) stack trace of write(): 0x7f1a1f359687 0 @[ 0x7f1a1f359687 0 ]: 5 @latency[2]: [1] 32464 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 4203 |@@@@@@ | [4, 8) 5116 |@@@@@@@@ | [8, 16) 1637 |@@ | [16, 32) 461 | | [32, 64) 65 | | [64, 128) 15 | | [128, 256) 14 | | [256, 512) 8 | | [512, 1K) 4 | | [1K, 2K) 0 | | [2K, 4K) 1 | | [4K, 8K) 2 | | [8K, 16K) 8 | | [16K, 32K) 6 | | [32K, 64K) 8 | | [64K, 128K) 9 | | [128K, 256K) 5 | | [256K, 512K) 1 | | [512K, 1M) 3 | | [1M, 2M) 1 | | @latency[6]: [1] 2733 | | [2, 4) 799579 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 183692 |@@@@@@@@@@@ | [8, 16) 84276 |@@@@@ | [16, 32) 104245 |@@@@@@ | [32, 64) 9466 | | [64, 128) 2281 | | [128, 256) 149 | | [256, 512) 1 | | [Fri Jan 16 13:19:20 UTC 2026] respawn: 7: stopped pid 310 with status 0 [Fri Jan 16 13:19:20 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:19:21 UTC 2026] respawn: 8: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 347 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 351 root cwd DIR 9,3 4096 25887107 /data sys_write 351 root rtd DIR 0,146 4096 25527869 / sys_write 351 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 351 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 351 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 351 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 351 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 351 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 351 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 351 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55a163f74000-55a163f78000 r--p 00000000 09:03 25259300 /usr/bin/dash 55a163f78000-55a163f8c000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55a163f8c000-55a163f92000 r--p 00018000 09:03 25259300 /usr/bin/dash 55a163f92000-55a163f94000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55a163f94000-55a163f95000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55a163f95000-55a163f97000 rw-p 00000000 00:00 0 55a164425000-55a164446000 rw-p 00000000 00:00 0 [heap] 7fb14ee9d000-7fb14eea0000 rw-p 00000000 00:00 0 7fb14eea0000-7fb14eec8000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fb14eec8000-7fb14f02c000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fb14f02c000-7fb14f082000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fb14f082000-7fb14f086000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fb14f086000-7fb14f088000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fb14f088000-7fb14f095000 rw-p 00000000 00:00 0 7fb14f0a3000-7fb14f0a5000 rw-p 00000000 00:00 0 7fb14f0a5000-7fb14f0a6000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fb14f0a6000-7fb14f0ce000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fb14f0ce000-7fb14f0d9000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fb14f0d9000-7fb14f0db000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fb14f0db000-7fb14f0dc000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fb14f0dc000-7fb14f0dd000 rw-p 00000000 00:00 0 7ffeca4a9000-7ffeca4ca000 rw-p 00000000 00:00 0 [stack] 7ffeca5a4000-7ffeca5a8000 r--p 00000000 00:00 0 [vvar] 7ffeca5a8000-7ffeca5aa000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 347 detected long write(fd=2, count=127, ret=127): 3555811 us PESPIN: vfs_write delta=3555802 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116132102915 DTRX INFO phy0.0: Clock indication: fn=2142 (trx_if.c:103) stack trace of write(): 0x7f78e8237687 0 @[ 0x7f78e8237687 0 ]: 1 @latency[2]: [1] 2418 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [2, 4) 4204 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 1378 |@@@@@@@@@@@@@@@@@ | [8, 16) 676 |@@@@@@@@ | [16, 32) 50 | | [32, 64) 7 | | [64, 128) 13 | | [128, 256) 7 | | [256, 512) 2 | | [512, 1K) 2 | | [1K, 2K) 2 | | [2K, 4K) 3 | | [4K, 8K) 1 | | [8K, 16K) 2 | | [16K, 32K) 1 | | [32K, 64K) 2 | | [64K, 128K) 0 | | [128K, 256K) 0 | | [256K, 512K) 2 | | [512K, 1M) 0 | | [1M, 2M) 0 | | [2M, 4M) 1 | | @latency[6]: [1] 16 | | [2, 4) 45475 |@@@@@@@@@@@@@@@@@ | [4, 8) 134776 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [8, 16) 9498 |@@@ | [16, 32) 6938 |@@ | [32, 64) 1631 | | [64, 128) 92 | | [128, 256) 1 | | [Fri Jan 16 13:21:07 UTC 2026] respawn: 8: stopped pid 351 with status 0 [Fri Jan 16 13:21:07 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:21:08 UTC 2026] respawn: 9: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 388 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 398 root cwd DIR 9,3 4096 25887107 /data sys_write 398 root rtd DIR 0,146 4096 25527869 / sys_write 398 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 398 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 398 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 398 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 398 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 398 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 398 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 398 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 56302ff4c000-56302ff50000 r--p 00000000 09:03 25259300 /usr/bin/dash 56302ff50000-56302ff64000 r-xp 00004000 09:03 25259300 /usr/bin/dash 56302ff64000-56302ff6a000 r--p 00018000 09:03 25259300 /usr/bin/dash 56302ff6a000-56302ff6c000 r--p 0001d000 09:03 25259300 /usr/bin/dash 56302ff6c000-56302ff6d000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 56302ff6d000-56302ff6f000 rw-p 00000000 00:00 0 563030971000-563030992000 rw-p 00000000 00:00 0 [heap] 7fc998b1a000-7fc998b1d000 rw-p 00000000 00:00 0 7fc998b1d000-7fc998b45000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fc998b45000-7fc998ca9000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fc998ca9000-7fc998cff000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fc998cff000-7fc998d03000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fc998d03000-7fc998d05000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fc998d05000-7fc998d12000 rw-p 00000000 00:00 0 7fc998d20000-7fc998d22000 rw-p 00000000 00:00 0 7fc998d22000-7fc998d23000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fc998d23000-7fc998d4b000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fc998d4b000-7fc998d56000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fc998d56000-7fc998d58000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fc998d58000-7fc998d59000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fc998d59000-7fc998d5a000 rw-p 00000000 00:00 0 7ffdfa9bc000-7ffdfa9dd000 rw-p 00000000 00:00 0 [stack] 7ffdfa9f3000-7ffdfa9f7000 r--p 00000000 00:00 0 [vvar] 7ffdfa9f7000-7ffdfa9f9000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 388 detected long write(fd=2, count=137, ret=137): 5605898 us PESPIN: vfs_write delta=21 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116132112501 DABIS NOTICE A-bis connection establishment to BSC (172.18.186.11) in progress... (abis.c:170) stack trace of write(): 0x7fbcc76b6687 0 PESPIN: PID 388 detected long write(fd=2, count=137, ret=137): 1060655 us PESPIN: vfs_write delta=1060649 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116132143291 DABIS NOTICE A-bis connection establishment to BSC (172.18.186.11) in progress... (abis.c:170) stack trace of write(): 0x7fbcc76b6687 0 PESPIN: PID 388 detected long write(fd=2, count=169, ret=169): 451388 us PESPIN: vfs_write delta=451377 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116132219451 DL1P NOTICE GSMTAP: Unable to determine actual BS_AG_BLKS_RES value as SI3 is not available yet, fallback to 1 (sysinfo.c:162) stack trace of write(): 0x7fbcc76b6687 0 @[ 0x7fbcc76b6687 0 ]: 3 @latency[2]: [1] 5986 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 587 |@@@@@ | [4, 8) 468 |@@@@ | [8, 16) 139 |@ | [16, 32) 56 | | [32, 64) 17 | | [64, 128) 0 | | [128, 256) 0 | | [256, 512) 1 | | [512, 1K) 0 | | [1K, 2K) 1 | | [2K, 4K) 0 | | [4K, 8K) 0 | | [8K, 16K) 1 | | [16K, 32K) 1 | | [32K, 64K) 0 | | [64K, 128K) 1 | | [128K, 256K) 3 | | [256K, 512K) 1 | | [512K, 1M) 0 | | [1M, 2M) 1 | | [2M, 4M) 0 | | [4M, 8M) 1 | | @latency[6]: [1] 550 | | [2, 4) 90600 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 11187 |@@@@@@ | [8, 16) 4419 |@@ | [16, 32) 9280 |@@@@@ | [32, 64) 269 | | [64, 128) 455 | | [128, 256) 1 | | [Fri Jan 16 13:22:43 UTC 2026] respawn: 9: stopped pid 398 with status 0 [Fri Jan 16 13:22:43 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:22:44 UTC 2026] respawn: 10: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 429 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 433 root cwd DIR 9,3 4096 25887107 /data sys_write 433 root rtd DIR 0,146 4096 25527869 / sys_write 433 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 433 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 433 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 433 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 433 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 433 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 433 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 433 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 564aa6444000-564aa6448000 r--p 00000000 09:03 25259300 /usr/bin/dash 564aa6448000-564aa645c000 r-xp 00004000 09:03 25259300 /usr/bin/dash 564aa645c000-564aa6462000 r--p 00018000 09:03 25259300 /usr/bin/dash 564aa6462000-564aa6464000 r--p 0001d000 09:03 25259300 /usr/bin/dash 564aa6464000-564aa6465000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 564aa6465000-564aa6467000 rw-p 00000000 00:00 0 564aa72e8000-564aa7309000 rw-p 00000000 00:00 0 [heap] 7f45d3c15000-7f45d3c18000 rw-p 00000000 00:00 0 7f45d3c18000-7f45d3c40000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f45d3c40000-7f45d3da4000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f45d3da4000-7f45d3dfa000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f45d3dfa000-7f45d3dfe000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f45d3dfe000-7f45d3e00000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f45d3e00000-7f45d3e0d000 rw-p 00000000 00:00 0 7f45d3e1b000-7f45d3e1d000 rw-p 00000000 00:00 0 7f45d3e1d000-7f45d3e1e000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f45d3e1e000-7f45d3e46000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f45d3e46000-7f45d3e51000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f45d3e51000-7f45d3e53000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f45d3e53000-7f45d3e54000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f45d3e54000-7f45d3e55000 rw-p 00000000 00:00 0 7ffce9161000-7ffce9182000 rw-p 00000000 00:00 0 [stack] 7ffce9187000-7ffce918b000 r--p 00000000 00:00 0 [vvar] 7ffce918b000-7ffce918d000 r-xp 00000000 00:00 0 [vdso] @latency[2]: [1] 6123 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 714 |@@@@@@ | [4, 8) 354 |@@@ | [8, 16) 100 | | [16, 32) 8 | | [32, 64) 3 | | [64, 128) 3 | | [128, 256) 4 | | [256, 512) 1 | | [512, 1K) 1 | | [1K, 2K) 1 | | [2K, 4K) 1 | | [4K, 8K) 1 | | [8K, 16K) 0 | | [16K, 32K) 0 | | [32K, 64K) 0 | | [64K, 128K) 0 | | [128K, 256K) 0 | | [256K, 512K) 1 | | @latency[6]: [1] 524 | | [2, 4) 49101 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 11221 |@@@@@@@@@@@ | [8, 16) 1815 |@ | [16, 32) 1239 |@ | [32, 64) 419 | | [64, 128) 21 | | [Fri Jan 16 13:23:56 UTC 2026] respawn: 10: stopped pid 433 with status 0 [Fri Jan 16 13:23:56 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:23:57 UTC 2026] respawn: 11: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 470 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 474 root cwd DIR 9,3 4096 25887107 /data sys_write 474 root rtd DIR 0,146 4096 25527869 / sys_write 474 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 474 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 474 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 474 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 474 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 474 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 474 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 474 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55692201f000-556922023000 r--p 00000000 09:03 25259300 /usr/bin/dash 556922023000-556922037000 r-xp 00004000 09:03 25259300 /usr/bin/dash 556922037000-55692203d000 r--p 00018000 09:03 25259300 /usr/bin/dash 55692203d000-55692203f000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55692203f000-556922040000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 556922040000-556922042000 rw-p 00000000 00:00 0 55692322f000-556923250000 rw-p 00000000 00:00 0 [heap] 7f273e6e1000-7f273e6e4000 rw-p 00000000 00:00 0 7f273e6e4000-7f273e70c000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f273e70c000-7f273e870000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f273e870000-7f273e8c6000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f273e8c6000-7f273e8ca000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f273e8ca000-7f273e8cc000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f273e8cc000-7f273e8d9000 rw-p 00000000 00:00 0 7f273e8e7000-7f273e8e9000 rw-p 00000000 00:00 0 7f273e8e9000-7f273e8ea000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f273e8ea000-7f273e912000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f273e912000-7f273e91d000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f273e91d000-7f273e91f000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f273e91f000-7f273e920000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f273e920000-7f273e921000 rw-p 00000000 00:00 0 7fffbedca000-7fffbedeb000 rw-p 00000000 00:00 0 [stack] 7fffbedf5000-7fffbedf9000 r--p 00000000 00:00 0 [vvar] 7fffbedf9000-7fffbedfb000 r-xp 00000000 00:00 0 [vdso] @latency[2]: [2, 4) 5491 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 1114 |@@@@@@@@@@ | [8, 16) 450 |@@@@ | [16, 32) 53 | | [32, 64) 14 | | [64, 128) 12 | | [128, 256) 5 | | [256, 512) 1 | | [512, 1K) 0 | | [1K, 2K) 0 | | [2K, 4K) 0 | | [4K, 8K) 0 | | [8K, 16K) 1 | | [16K, 32K) 0 | | [32K, 64K) 0 | | [64K, 128K) 2 | | [128K, 256K) 0 | | [256K, 512K) 1 | | @latency[6]: [2, 4) 860 | | [4, 8) 82066 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [8, 16) 36331 |@@@@@@@@@@@@@@@@@@@@@@@ | [16, 32) 7415 |@@@@ | [32, 64) 3718 |@@ | [64, 128) 182 | | [128, 256) 4 | | [Fri Jan 16 13:25:13 UTC 2026] respawn: 11: stopped pid 474 with status 0 [Fri Jan 16 13:25:13 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:25:14 UTC 2026] respawn: 12: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 511 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 515 root cwd DIR 9,3 4096 25887107 /data sys_write 515 root rtd DIR 0,146 4096 25527869 / sys_write 515 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 515 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 515 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 515 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 515 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 515 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 515 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 515 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 556e1a523000-556e1a527000 r--p 00000000 09:03 25259300 /usr/bin/dash 556e1a527000-556e1a53b000 r-xp 00004000 09:03 25259300 /usr/bin/dash 556e1a53b000-556e1a541000 r--p 00018000 09:03 25259300 /usr/bin/dash 556e1a541000-556e1a543000 r--p 0001d000 09:03 25259300 /usr/bin/dash 556e1a543000-556e1a544000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 556e1a544000-556e1a546000 rw-p 00000000 00:00 0 556e1aee5000-556e1af06000 rw-p 00000000 00:00 0 [heap] 7f9ee8502000-7f9ee8505000 rw-p 00000000 00:00 0 7f9ee8505000-7f9ee852d000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f9ee852d000-7f9ee8691000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f9ee8691000-7f9ee86e7000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f9ee86e7000-7f9ee86eb000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f9ee86eb000-7f9ee86ed000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f9ee86ed000-7f9ee86fa000 rw-p 00000000 00:00 0 7f9ee8708000-7f9ee870a000 rw-p 00000000 00:00 0 7f9ee870a000-7f9ee870b000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9ee870b000-7f9ee8733000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9ee8733000-7f9ee873e000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9ee873e000-7f9ee8740000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9ee8740000-7f9ee8741000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f9ee8741000-7f9ee8742000 rw-p 00000000 00:00 0 7ffd0a163000-7ffd0a184000 rw-p 00000000 00:00 0 [stack] 7ffd0a1a9000-7ffd0a1ad000 r--p 00000000 00:00 0 [vvar] 7ffd0a1ad000-7ffd0a1af000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 511 detected long write(fd=2, count=126, ret=126): 1333666 us PESPIN: vfs_write delta=1333657 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116132621487 DTRX INFO phy0.0: Clock indication: fn=918 (trx_if.c:103) stack trace of write(): 0x7f445b942687 0 @[ 0x7f445b942687 0 ]: 1 @latency[2]: [1] 4341 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 522 |@@@@@@ | [4, 8) 454 |@@@@@ | [8, 16) 171 |@@ | [16, 32) 10 | | [32, 64) 8 | | [64, 128) 2 | | [128, 256) 2 | | [256, 512) 0 | | [512, 1K) 0 | | [1K, 2K) 1 | | [2K, 4K) 1 | | [4K, 8K) 0 | | [8K, 16K) 1 | | [16K, 32K) 2 | | [32K, 64K) 0 | | [64K, 128K) 2 | | [128K, 256K) 0 | | [256K, 512K) 0 | | [512K, 1M) 0 | | [1M, 2M) 1 | | @latency[6]: [1] 261 | | [2, 4) 99484 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 23769 |@@@@@@@@@@@@ | [8, 16) 6385 |@@@ | [16, 32) 6381 |@@@ | [32, 64) 779 | | [64, 128) 30 | | [128, 256) 1 | | [Fri Jan 16 13:26:23 UTC 2026] respawn: 12: stopped pid 515 with status 0 [Fri Jan 16 13:26:23 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:26:24 UTC 2026] respawn: 13: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 552 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 562 root cwd DIR 9,3 4096 25887107 /data sys_write 562 root rtd DIR 0,146 4096 25527869 / sys_write 562 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 562 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 562 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 562 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 562 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 562 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 562 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 562 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55a7b8a16000-55a7b8a1a000 r--p 00000000 09:03 25259300 /usr/bin/dash 55a7b8a1a000-55a7b8a2e000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55a7b8a2e000-55a7b8a34000 r--p 00018000 09:03 25259300 /usr/bin/dash 55a7b8a34000-55a7b8a36000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55a7b8a36000-55a7b8a37000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55a7b8a37000-55a7b8a39000 rw-p 00000000 00:00 0 55a7ba605000-55a7ba626000 rw-p 00000000 00:00 0 [heap] 7fadbee7f000-7fadbee82000 rw-p 00000000 00:00 0 7fadbee82000-7fadbeeaa000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fadbeeaa000-7fadbf00e000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fadbf00e000-7fadbf064000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fadbf064000-7fadbf068000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fadbf068000-7fadbf06a000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fadbf06a000-7fadbf077000 rw-p 00000000 00:00 0 7fadbf085000-7fadbf087000 rw-p 00000000 00:00 0 7fadbf087000-7fadbf088000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fadbf088000-7fadbf0b0000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fadbf0b0000-7fadbf0bb000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fadbf0bb000-7fadbf0bd000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fadbf0bd000-7fadbf0be000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fadbf0be000-7fadbf0bf000 rw-p 00000000 00:00 0 7ffef1555000-7ffef1576000 rw-p 00000000 00:00 0 [stack] 7ffef15f6000-7ffef15fa000 r--p 00000000 00:00 0 [vvar] 7ffef15fa000-7ffef15fc000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 552 detected long write(fd=2, count=114, ret=114): 445731 us PESPIN: vfs_write delta=445724 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116132658334 DL1P NOTICE (bts=0,trx=0,ts=1,ss=0) SACCH no SI available (lchan.c:284) stack trace of write(): 0x7f24c0b26687 0 @[ 0x7f24c0b26687 0 ]: 1 @latency[2]: [1] 11184 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 1133 |@@@@@ | [4, 8) 900 |@@@@ | [8, 16) 432 |@@ | [16, 32) 53 | | [32, 64) 11 | | [64, 128) 2 | | [128, 256) 7 | | [256, 512) 1 | | [512, 1K) 1 | | [1K, 2K) 3 | | [2K, 4K) 0 | | [4K, 8K) 2 | | [8K, 16K) 1 | | [16K, 32K) 2 | | [32K, 64K) 0 | | [64K, 128K) 2 | | [128K, 256K) 4 | | [256K, 512K) 2 | | @latency[6]: [1] 2483 | | [2, 4) 196559 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 42482 |@@@@@@@@@@@ | [8, 16) 9749 |@@ | [16, 32) 11922 |@@@ | [32, 64) 1682 | | [64, 128) 139 | | [128, 256) 1 | | [Fri Jan 16 13:28:57 UTC 2026] respawn: 13: stopped pid 562 with status 0 [Fri Jan 16 13:28:57 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:28:58 UTC 2026] respawn: 14: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 593 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 603 root cwd DIR 9,3 4096 25887107 /data sys_write 603 root rtd DIR 0,146 4096 25527869 / sys_write 603 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 603 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 603 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 603 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 603 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 603 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 603 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 603 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55ae7ce3d000-55ae7ce41000 r--p 00000000 09:03 25259300 /usr/bin/dash 55ae7ce41000-55ae7ce55000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55ae7ce55000-55ae7ce5b000 r--p 00018000 09:03 25259300 /usr/bin/dash 55ae7ce5b000-55ae7ce5d000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55ae7ce5d000-55ae7ce5e000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55ae7ce5e000-55ae7ce60000 rw-p 00000000 00:00 0 55ae7cf87000-55ae7cfa8000 rw-p 00000000 00:00 0 [heap] 7fa361524000-7fa361527000 rw-p 00000000 00:00 0 7fa361527000-7fa36154f000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fa36154f000-7fa3616b3000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fa3616b3000-7fa361709000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fa361709000-7fa36170d000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fa36170d000-7fa36170f000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7fa36170f000-7fa36171c000 rw-p 00000000 00:00 0 7fa36172a000-7fa36172c000 rw-p 00000000 00:00 0 7fa36172c000-7fa36172d000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fa36172d000-7fa361755000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fa361755000-7fa361760000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fa361760000-7fa361762000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fa361762000-7fa361763000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7fa361763000-7fa361764000 rw-p 00000000 00:00 0 7ffdb6007000-7ffdb6028000 rw-p 00000000 00:00 0 [stack] 7ffdb612a000-7ffdb612e000 r--p 00000000 00:00 0 [vvar] 7ffdb612e000-7ffdb6130000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 593 detected long write(fd=2, count=120, ret=120): 759632 us PESPIN: vfs_write delta=759629 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116133045004 DRSL INFO Tx RSL RF RESource INDication (rsl.c:519) stack trace of write(): 0x7fe20bed3687 0 PESPIN: PID 593 detected long write(fd=2, count=126, ret=126): 625379 us PESPIN: vfs_write delta=625369 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116133058357 DTRX INFO phy0.0: Clock indication: fn=816 (trx_if.c:103) stack trace of write(): 0x7fe20bed3687 0 @[ 0x7fe20bed3687 0 ]: 2 @latency[22]: [16, 32) 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| @latency[2]: [2, 4) 6038 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 1854 |@@@@@@@@@@@@@@@ | [8, 16) 1198 |@@@@@@@@@@ | [16, 32) 341 |@@ | [32, 64) 24 | | [64, 128) 5 | | [128, 256) 5 | | [256, 512) 4 | | [512, 1K) 2 | | [1K, 2K) 1 | | [2K, 4K) 5 | | [4K, 8K) 3 | | [8K, 16K) 4 | | [16K, 32K) 3 | | [32K, 64K) 5 | | [64K, 128K) 5 | | [128K, 256K) 5 | | [256K, 512K) 0 | | [512K, 1M) 2 | | @latency[6]: [2, 4) 2043 | | [4, 8) 112040 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 123719 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16, 32) 12377 |@@@@@ | [32, 64) 12098 |@@@@@ | [64, 128) 1356 | | [128, 256) 12 | | [Fri Jan 16 13:30:59 UTC 2026] respawn: 14: stopped pid 603 with status 0 [Fri Jan 16 13:30:59 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:31:00 UTC 2026] respawn: 15: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 634 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 644 root cwd DIR 9,3 4096 25887107 /data sys_write 644 root rtd DIR 0,146 4096 25527869 / sys_write 644 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 644 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 644 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 644 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 644 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 644 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 644 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 644 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55cda5d63000-55cda5d67000 r--p 00000000 09:03 25259300 /usr/bin/dash 55cda5d67000-55cda5d7b000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55cda5d7b000-55cda5d81000 r--p 00018000 09:03 25259300 /usr/bin/dash 55cda5d81000-55cda5d83000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55cda5d83000-55cda5d84000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55cda5d84000-55cda5d86000 rw-p 00000000 00:00 0 55cda6ca7000-55cda6cc8000 rw-p 00000000 00:00 0 [heap] 7f41fb7ac000-7f41fb7af000 rw-p 00000000 00:00 0 7f41fb7af000-7f41fb7d7000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f41fb7d7000-7f41fb93b000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f41fb93b000-7f41fb991000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f41fb991000-7f41fb995000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f41fb995000-7f41fb997000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f41fb997000-7f41fb9a4000 rw-p 00000000 00:00 0 7f41fb9b2000-7f41fb9b4000 rw-p 00000000 00:00 0 7f41fb9b4000-7f41fb9b5000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f41fb9b5000-7f41fb9dd000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f41fb9dd000-7f41fb9e8000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f41fb9e8000-7f41fb9ea000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f41fb9ea000-7f41fb9eb000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f41fb9eb000-7f41fb9ec000 rw-p 00000000 00:00 0 7ffe76586000-7ffe765a7000 rw-p 00000000 00:00 0 [stack] 7ffe765b6000-7ffe765ba000 r--p 00000000 00:00 0 [vvar] 7ffe765ba000-7ffe765bc000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 634 detected long write(fd=2, count=173, ret=173): 2252862 us PESPIN: vfs_write delta=2252852 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116134850017 DL1P NOTICE PH-RTS-IND: Unable to determine actual BS_AG_BLKS_RES value as SI3 is not available yet, fallback to 1 (sysinfo.c:162) stack trace of write(): 0x7fabd89f7687 0 @[ 0x7fabd89f7687 0 ]: 1 @latency[2]: [1] 21527 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [2, 4) 24374 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 8733 |@@@@@@@@@@@@@@@@@@ | [8, 16) 6454 |@@@@@@@@@@@@@ | [16, 32) 2432 |@@@@@ | [32, 64) 498 |@ | [64, 128) 37 | | [128, 256) 17 | | [256, 512) 8 | | [512, 1K) 7 | | [1K, 2K) 2 | | [2K, 4K) 6 | | [4K, 8K) 14 | | [8K, 16K) 14 | | [16K, 32K) 8 | | [32K, 64K) 9 | | [64K, 128K) 6 | | [128K, 256K) 5 | | [256K, 512K) 1 | | [512K, 1M) 0 | | [1M, 2M) 0 | | [2M, 4M) 1 | | @latency[6]: [1] 2878 | | [2, 4) 748667 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [4, 8) 747508 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 278384 |@@@@@@@@@@@@ | [16, 32) 1152275 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [32, 64) 30053 |@ | [64, 128) 10662 | | [128, 256) 50 | | [256, 512) 1 | | [Fri Jan 16 13:49:42 UTC 2026] respawn: 15: stopped pid 644 with status 0 [Fri Jan 16 13:49:42 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:49:43 UTC 2026] respawn: 16: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 675 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 679 root cwd DIR 9,3 4096 25887107 /data sys_write 679 root rtd DIR 0,146 4096 25527869 / sys_write 679 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 679 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 679 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 679 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 679 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 679 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 679 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 679 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55965e1cd000-55965e1d1000 r--p 00000000 09:03 25259300 /usr/bin/dash 55965e1d1000-55965e1e5000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55965e1e5000-55965e1eb000 r--p 00018000 09:03 25259300 /usr/bin/dash 55965e1eb000-55965e1ed000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55965e1ed000-55965e1ee000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55965e1ee000-55965e1f0000 rw-p 00000000 00:00 0 55965e4d3000-55965e4f4000 rw-p 00000000 00:00 0 [heap] 7ff2add45000-7ff2add48000 rw-p 00000000 00:00 0 7ff2add48000-7ff2add70000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff2add70000-7ff2aded4000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff2aded4000-7ff2adf2a000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff2adf2a000-7ff2adf2e000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff2adf2e000-7ff2adf30000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7ff2adf30000-7ff2adf3d000 rw-p 00000000 00:00 0 7ff2adf4b000-7ff2adf4d000 rw-p 00000000 00:00 0 7ff2adf4d000-7ff2adf4e000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff2adf4e000-7ff2adf76000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff2adf76000-7ff2adf81000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff2adf81000-7ff2adf83000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff2adf83000-7ff2adf84000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7ff2adf84000-7ff2adf85000 rw-p 00000000 00:00 0 7ffe6f372000-7ffe6f393000 rw-p 00000000 00:00 0 [stack] 7ffe6f3dc000-7ffe6f3e0000 r--p 00000000 00:00 0 [vvar] 7ffe6f3e0000-7ffe6f3e2000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 675 detected long write(fd=2, count=114, ret=114): 1080605 us PESPIN: vfs_write delta=1080592 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116134959229 DL1P NOTICE (bts=0,trx=0,ts=0,ss=2) SACCH no SI available (lchan.c:284) stack trace of write(): 0x7f033e10b687 0 PESPIN: PID 675 detected long write(fd=2, count=120, ret=120): 916423 us PESPIN: vfs_write delta=916411 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116135003319 DRSL INFO Tx RSL RF RESource INDication (rsl.c:519) stack trace of write(): 0x7f033e10b687 0 PESPIN: PID 675 detected long write(fd=2, count=114, ret=114): 3327303 us PESPIN: vfs_write delta=3327291 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116135017315 DL1P NOTICE (bts=0,trx=0,ts=0,ss=2) SACCH no SI available (lchan.c:284) stack trace of write(): 0x7f033e10b687 0 @[ 0x7f033e10b687 0 ]: 3 @latency[2]: [2, 4) 17967 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 1127 |@@@ | [8, 16) 337 | | [16, 32) 110 | | [32, 64) 40 | | [64, 128) 6 | | [128, 256) 1 | | [256, 512) 1 | | [512, 1K) 0 | | [1K, 2K) 0 | | [2K, 4K) 0 | | [4K, 8K) 1 | | [8K, 16K) 0 | | [16K, 32K) 0 | | [32K, 64K) 0 | | [64K, 128K) 0 | | [128K, 256K) 0 | | [256K, 512K) 0 | | [512K, 1M) 1 | | [1M, 2M) 1 | | [2M, 4M) 1 | | @latency[6]: [2, 4) 9554 |@@@@@@@@@ | [4, 8) 50265 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [8, 16) 30771 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [16, 32) 15394 |@@@@@@@@@@@@@@@ | [32, 64) 23913 |@@@@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 1073 |@ | [128, 256) 5 | | [Fri Jan 16 13:50:22 UTC 2026] respawn: 16: stopped pid 679 with status 0 [Fri Jan 16 13:50:22 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:50:23 UTC 2026] respawn: 17: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p osmo-bts-trx ERROR: Failed to parse pid: pid 'osmo-bts-trx' is not a valid decimal number lsof output: maps output: cat: /proc/724/maps: No such file or directory [Fri Jan 16 13:50:28 UTC 2026] respawn: 17: stopped pid 724 with status 1 [Fri Jan 16 13:50:28 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:50:29 UTC 2026] respawn: 18: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 728 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 753 root cwd DIR 9,3 4096 25887107 /data sys_write 753 root rtd DIR 0,146 4096 25527869 / sys_write 753 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 753 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 753 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 753 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 753 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 753 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 753 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 753 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 55d2d7fdd000-55d2d7fe1000 r--p 00000000 09:03 25259300 /usr/bin/dash 55d2d7fe1000-55d2d7ff5000 r-xp 00004000 09:03 25259300 /usr/bin/dash 55d2d7ff5000-55d2d7ffb000 r--p 00018000 09:03 25259300 /usr/bin/dash 55d2d7ffb000-55d2d7ffd000 r--p 0001d000 09:03 25259300 /usr/bin/dash 55d2d7ffd000-55d2d7ffe000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 55d2d7ffe000-55d2d8000000 rw-p 00000000 00:00 0 55d2d9e8c000-55d2d9ead000 rw-p 00000000 00:00 0 [heap] 7f3ca16dd000-7f3ca16e0000 rw-p 00000000 00:00 0 7f3ca16e0000-7f3ca1708000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f3ca1708000-7f3ca186c000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f3ca186c000-7f3ca18c2000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f3ca18c2000-7f3ca18c6000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f3ca18c6000-7f3ca18c8000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f3ca18c8000-7f3ca18d5000 rw-p 00000000 00:00 0 7f3ca18e3000-7f3ca18e5000 rw-p 00000000 00:00 0 7f3ca18e5000-7f3ca18e6000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f3ca18e6000-7f3ca190e000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f3ca190e000-7f3ca1919000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f3ca1919000-7f3ca191b000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f3ca191b000-7f3ca191c000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f3ca191c000-7f3ca191d000 rw-p 00000000 00:00 0 7ffefac42000-7ffefac63000 rw-p 00000000 00:00 0 [stack] 7ffefadae000-7ffefadb2000 r--p 00000000 00:00 0 [vvar] 7ffefadb2000-7ffefadb4000 r-xp 00000000 00:00 0 [vdso] @latency[2]: [1] 523 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 36 |@@@ | [4, 8) 63 |@@@@@@ | [8, 16) 21 |@@ | [16, 32) 4 | | [32, 64) 1 | | [64, 128) 1 | | [128, 256) 0 | | [256, 512) 0 | | [512, 1K) 0 | | [1K, 2K) 0 | | [2K, 4K) 0 | | [4K, 8K) 0 | | [8K, 16K) 0 | | [16K, 32K) 0 | | [32K, 64K) 0 | | [64K, 128K) 0 | | [128K, 256K) 1 | | @latency[6]: [1] 79 | | [2, 4) 17672 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 3671 |@@@@@@@@@@ | [8, 16) 644 |@ | [16, 32) 800 |@@ | [32, 64) 178 | | [64, 128) 1 | | [Fri Jan 16 13:50:38 UTC 2026] respawn: 18: stopped pid 753 with status 0 [Fri Jan 16 13:50:38 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:50:39 UTC 2026] respawn: 19: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 781 osmo-bts-trx Attaching 5 probes... Tracing sys_write() latency... Hit Ctrl-C to end. lsof output: COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME sys_write 785 root cwd DIR 9,3 4096 25887107 /data sys_write 785 root rtd DIR 0,146 4096 25527869 / sys_write 785 root txt REG 0,146 129736 25259300 /usr/bin/dash sys_write 785 root mem REG 9,3 25259300 /usr/bin/dash (path dev=0,146) sys_write 785 root mem REG 9,3 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 (path dev=0,146) sys_write 785 root mem REG 9,3 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (path dev=0,146) sys_write 785 root 0r CHR LG 1,3 0t0 15 /dev/null sys_write 785 root 1w FIFO W 0,13 0t0 395667142 pipe sys_write 785 root 2w FIFO W 0,13 0t0 395667143 pipe sys_write 785 root 10r REG LG,CX 9,3 109 25872172 /data/bpftrace/sys_write_delay.sh maps output: 564a5e087000-564a5e08b000 r--p 00000000 09:03 25259300 /usr/bin/dash 564a5e08b000-564a5e09f000 r-xp 00004000 09:03 25259300 /usr/bin/dash 564a5e09f000-564a5e0a5000 r--p 00018000 09:03 25259300 /usr/bin/dash 564a5e0a5000-564a5e0a7000 r--p 0001d000 09:03 25259300 /usr/bin/dash 564a5e0a7000-564a5e0a8000 rw-p 0001f000 09:03 25259300 /usr/bin/dash 564a5e0a8000-564a5e0aa000 rw-p 00000000 00:00 0 564a5e805000-564a5e826000 rw-p 00000000 00:00 0 [heap] 7f7198bdd000-7f7198be0000 rw-p 00000000 00:00 0 7f7198be0000-7f7198c08000 r--p 00000000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f7198c08000-7f7198d6c000 r-xp 00028000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f7198d6c000-7f7198dc2000 r--p 0018c000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f7198dc2000-7f7198dc6000 r--p 001e1000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f7198dc6000-7f7198dc8000 rw-p 001e5000 09:03 25259853 /usr/lib/x86_64-linux-gnu/libc.so.6 7f7198dc8000-7f7198dd5000 rw-p 00000000 00:00 0 7f7198de3000-7f7198de5000 rw-p 00000000 00:00 0 7f7198de5000-7f7198de6000 r--p 00000000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f7198de6000-7f7198e0e000 r-xp 00001000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f7198e0e000-7f7198e19000 r--p 00029000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f7198e19000-7f7198e1b000 r--p 00034000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f7198e1b000-7f7198e1c000 rw-p 00036000 09:03 25259833 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 7f7198e1c000-7f7198e1d000 rw-p 00000000 00:00 0 7ffff0124000-7ffff0145000 rw-p 00000000 00:00 0 [stack] 7ffff018c000-7ffff0190000 r--p 00000000 00:00 0 [vvar] 7ffff0190000-7ffff0192000 r-xp 00000000 00:00 0 [vdso] PESPIN: PID 781 detected long write(fd=2, count=114, ret=114): 1099650 us PESPIN: vfs_write delta=1099641 us, file->f_flags=0x8c01 PESPIN: sys_write content: 20260116135248676 DL1P NOTICE (bts=0,trx=0,ts=0,ss=2) SACCH no SI available (lchan.c:284) stack trace of write(): 0x7fce34167687 0 @[ 0x7fce34167687 0 ]: 1 @latency[2]: [1] 21619 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [2, 4) 2651 |@@@@@@ | [4, 8) 3004 |@@@@@@@ | [8, 16) 1258 |@@@ | [16, 32) 252 | | [32, 64) 56 | | [64, 128) 25 | | [128, 256) 10 | | [256, 512) 3 | | [512, 1K) 1 | | [1K, 2K) 5 | | [2K, 4K) 2 | | [4K, 8K) 7 | | [8K, 16K) 8 | | [16K, 32K) 6 | | [32K, 64K) 8 | | [64K, 128K) 3 | | [128K, 256K) 5 | | [256K, 512K) 1 | | [512K, 1M) 0 | | [1M, 2M) 1 | | @latency[6]: [1] 4356 | | [2, 4) 547558 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 122265 |@@@@@@@@@@@ | [8, 16) 57204 |@@@@@ | [16, 32) 71666 |@@@@@@ | [32, 64) 5288 | | [64, 128) 626 | | [128, 256) 5 | | [Fri Jan 16 13:56:00 UTC 2026] respawn: 19: stopped pid 785 with status 0 [Fri Jan 16 13:56:00 UTC 2026] respawn: sleeping 1 seconds... [Fri Jan 16 13:56:01 UTC 2026] respawn: 20: starting: /data/bpftrace/sys_write_delay.sh + pidof osmo-bts-trx + bpftrace --no-warnings /data/bpftrace/sys_write_delay.bt -p 781 osmo-bts-trx