![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3m4.py (from gprs_sysmo)
Failing for the past 9 builds
(Since Failed )
Stacktrace
rm-remote-dir(pid=31626): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31626)]
Standard Output
----------------------------------------------
trial-2071 gprs:sysmo iperf3m4.py
----------------------------------------------
21:59:58.259978 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:58.271656 tst gprs:sysmo: Using 1 x bts (candidates: 1)
21:59:58.403711 tst iperf3m4.py: using LAC 41718
21:59:58.533729 tst iperf3m4.py: using RAC 153
21:59:58.661335 tst iperf3m4.py: using CellId 41718
21:59:58.789291 tst iperf3m4.py: using BVCI 41719
21:59:58.814419 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:58.825473 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:58.836460 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:58.847338 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:58.858191 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:58.869070 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:58.880078 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:58.890984 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:58.902079 tst gprs:sysmo: Using 1 x modem (candidates: 4)
21:59:59.021255 tst gprs:sysmo: Using 1 x modem (candidates: 4)
21:59:59.145333 tst gprs:sysmo: Using 1 x modem (candidates: 4)
21:59:59.265580 tst gprs:sysmo: Using 1 x modem (candidates: 4)
21:59:59.548997 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
21:59:59.677000 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:59:59.996438 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31576): Launched
22:00:00.261516 run iperf3-srv_10.42.42.10(pid=31577): Launched
22:00:00.449066 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
22:00:00.577388 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:00:00.896329 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31578): Launched
22:00:01.160243 run iperf3-srv_10.42.42.10(pid=31579): Launched
22:00:01.346969 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
22:00:01.474550 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:00:01.793426 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31583): Launched
22:00:02.056988 run iperf3-srv_10.42.42.10(pid=31584): Launched
22:00:02.243526 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
22:00:02.371030 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:00:02.688444 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31585): Launched
22:00:02.952029 run iperf3-srv_10.42.42.10(pid=31586): Launched
22:00:03.139392 tst iperf3m4.py:8: start network...
22:00:03.267170 run osmo-hlr_10.42.42.2: Starting osmo-hlr
22:00:03.751622 run create_hlr_db(pid=31587): Launched
22:00:03.928095 bus /gobi_6: Setting Powered False
22:00:04.949754 run create_hlr_db(pid=31587): Terminated: ok {rc=0}
22:00:05.357969 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31589): Launched
22:00:05.665775 run osmo-hlr_10.42.42.2(pid=31590): Launched
22:00:05.794056 run osmo-stp_10.42.42.5: Starting osmo-stp
22:00:06.369053 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31591): Launched
22:00:06.799316 run osmo-stp_10.42.42.5(pid=31592): Launched
22:00:06.943288 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
22:00:07.530723 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31593): Launched
22:00:07.648138 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
22:00:07.935387 run patchelf(pid=31594): Launched
22:00:09.131211 run patchelf(pid=31594): Terminated: ok {rc=0}
22:00:09.251847 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
22:00:09.549536 run osmo-gsm-tester_setcap_net_admin.sh(pid=31595): Launched
22:00:10.744997 run osmo-gsm-tester_setcap_net_admin.sh(pid=31595): Terminated: ok {rc=0}
22:00:11.076131 run osmo-ggsn_10.42.42.6(pid=31598): Launched
22:00:11.343936 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
22:00:11.917082 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31616): Launched
22:00:12.245554 run osmo-sgsn_10.42.42.7(pid=31617): Launched
22:00:12.365751 run osmo-msc_10.42.42.8: Starting osmo-msc
22:00:12.967353 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31618): Launched
22:00:13.295990 run osmo-msc_10.42.42.8(pid=31619): Launched
22:00:16.461066 run osmo-mgw_10.42.42.3: Starting osmo-mgw
22:00:16.991286 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31620): Launched
22:00:17.306437 run osmo-mgw_10.42.42.3(pid=31621): Launched
22:00:17.424611 run osmo-mgw_10.42.42.4: Starting osmo-mgw
22:00:17.982905 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31622): Launched
22:00:18.310736 run osmo-mgw_10.42.42.4(pid=31623): Launched
22:00:18.429076 run osmo-bsc_10.42.42.9: Starting osmo-bsc
22:00:19.123834 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31624): Launched
22:00:19.433814 run osmo-bsc_10.42.42.9(pid=31625): Launched
22:00:25.695862 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
22:00:26.240432 run rm-remote-dir(pid=31626): Launched
22:00:29.434583 run rm-remote-dir(pid=31626): ERR: Terminated: ERROR {rc=255} [trial-2071↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31626)]
22:00:29.570142 run rm-remote-dir(pid=31626): stdout:
| (launched: 2023-01-03_22:00:26.083485)
22:00:29.705403 run rm-remote-dir(pid=31626): stderr:
| (launched: 2023-01-03_22:00:26.132259)
| ssh: connect to host 10.42.42.114 port 22: No route to host
22:00:29.761564 tst iperf3m4.py:8: ERR: Error: rm-remote-dir(pid=31626): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31626)] [trial-2071↪gprs:sysmo↪iperf3m4.py:8]
22:00:29.769298 tst iperf3m4.py:8: Test FAILED (31.5 sec)
22:00:29.805658 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31576): Terminating (SIGTERM)
22:00:29.846827 run iperf3-srv_10.42.42.10(pid=31577): Terminating (SIGTERM)
22:00:29.887522 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31578): Terminating (SIGTERM)
22:00:29.928083 run iperf3-srv_10.42.42.10(pid=31579): Terminating (SIGTERM)
22:00:29.968910 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31583): Terminating (SIGTERM)
22:00:30.009400 run iperf3-srv_10.42.42.10(pid=31584): Terminating (SIGTERM)
22:00:30.049989 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31585): Terminating (SIGTERM)
22:00:30.090095 run iperf3-srv_10.42.42.10(pid=31586): Terminating (SIGTERM)
22:00:30.130880 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31589): Terminating (SIGTERM)
22:00:30.171530 run osmo-hlr_10.42.42.2(pid=31590): Terminating (SIGTERM)
22:00:30.213417 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31591): Terminating (SIGTERM)
22:00:30.255440 run osmo-stp_10.42.42.5(pid=31592): Terminating (SIGTERM)
22:00:30.297680 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31593): Terminating (SIGTERM)
22:00:30.342354 run osmo-ggsn_10.42.42.6(pid=31598): Terminating (SIGTERM)
22:00:30.405835 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31616): Terminating (SIGTERM)
22:00:30.452274 run osmo-sgsn_10.42.42.7(pid=31617): Terminating (SIGTERM)
22:00:30.498374 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31618): Terminating (SIGTERM)
22:00:30.547601 run osmo-msc_10.42.42.8(pid=31619): Terminating (SIGTERM)
22:00:30.590299 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31620): Terminating (SIGTERM)
22:00:30.636634 run osmo-mgw_10.42.42.3(pid=31621): Terminating (SIGTERM)
22:00:30.679970 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31622): Terminating (SIGTERM)
22:00:30.722368 run osmo-mgw_10.42.42.4(pid=31623): Terminating (SIGTERM)
22:00:30.765771 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31624): Terminating (SIGTERM)
22:00:30.808378 run osmo-bsc_10.42.42.9(pid=31625): Terminating (SIGTERM)
22:00:30.818336 --- ParallelTerminationStrategy: PID 31576 died...
22:00:30.875182 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31576): Terminated: ok {rc=0}
22:00:30.885529 --- ParallelTerminationStrategy: PID 31577 died...
22:00:30.942004 run iperf3-srv_10.42.42.10(pid=31577): Terminated {rc=256}
22:00:30.952371 --- ParallelTerminationStrategy: PID 31578 died...
22:00:31.008738 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31578): Terminated: ok {rc=0}
22:00:31.019086 --- ParallelTerminationStrategy: PID 31579 died...
22:00:31.075732 run iperf3-srv_10.42.42.10(pid=31579): Terminated {rc=256}
22:00:31.085927 --- ParallelTerminationStrategy: PID 31583 died...
22:00:31.142530 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31583): Terminated: ok {rc=0}
22:00:31.152895 --- ParallelTerminationStrategy: PID 31584 died...
22:00:31.208978 run iperf3-srv_10.42.42.10(pid=31584): Terminated {rc=256}
22:00:31.219140 --- ParallelTerminationStrategy: PID 31585 died...
22:00:31.275414 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31585): Terminated: ok {rc=0}
22:00:31.285789 --- ParallelTerminationStrategy: PID 31586 died...
22:00:31.345542 run iperf3-srv_10.42.42.10(pid=31586): Terminated {rc=256}
22:00:31.356437 --- ParallelTerminationStrategy: PID 31589 died...
22:00:31.414257 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31589): Terminated: ok {rc=0}
22:00:31.425533 --- ParallelTerminationStrategy: PID 31590 died...
22:00:31.511629 run osmo-hlr_10.42.42.2(pid=31590): Terminated: ok {rc=0}
22:00:31.525187 --- ParallelTerminationStrategy: PID 31591 died...
22:00:31.618258 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31591): Terminated: ok {rc=0}
22:00:31.645999 --- ParallelTerminationStrategy: PID 31592 died...
22:00:31.757847 run osmo-stp_10.42.42.5(pid=31592): Terminated: ok {rc=0}
22:00:31.783664 --- ParallelTerminationStrategy: PID 31593 died...
22:00:31.843528 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31593): Terminated: ok {rc=0}
22:00:31.854957 --- ParallelTerminationStrategy: PID 31598 died...
22:00:31.914345 run osmo-ggsn_10.42.42.6(pid=31598): Terminated: ok {rc=0}
22:00:31.925514 --- ParallelTerminationStrategy: PID 31616 died...
22:00:31.985450 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31616): Terminated: ok {rc=0}
22:00:31.995932 --- ParallelTerminationStrategy: PID 31617 died...
22:00:32.052264 run osmo-sgsn_10.42.42.7(pid=31617): Terminated: ok {rc=0}
22:00:32.062824 --- ParallelTerminationStrategy: PID 31618 died...
22:00:32.119372 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31618): Terminated: ok {rc=0}
22:00:32.129856 --- ParallelTerminationStrategy: PID 31619 died...
22:00:32.186528 run osmo-msc_10.42.42.8(pid=31619): Terminated: ok {rc=0}
22:00:32.196911 --- ParallelTerminationStrategy: PID 31620 died...
22:00:32.253206 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31620): Terminated: ok {rc=0}
22:00:32.263511 --- ParallelTerminationStrategy: PID 31621 died...
22:00:32.319920 run osmo-mgw_10.42.42.3(pid=31621): Terminated {rc=15}
22:00:32.330250 --- ParallelTerminationStrategy: PID 31622 died...
22:00:32.386846 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31622): Terminated: ok {rc=0}
22:00:32.397348 --- ParallelTerminationStrategy: PID 31623 died...
22:00:32.453865 run osmo-mgw_10.42.42.4(pid=31623): Terminated {rc=15}
22:00:32.464225 --- ParallelTerminationStrategy: PID 31624 died...
22:00:32.520550 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31624): Terminated: ok {rc=0}
22:00:32.530862 --- ParallelTerminationStrategy: PID 31625 died...
22:00:32.587039 run osmo-bsc_10.42.42.9(pid=31625): Terminated: ok {rc=0}
22:00:32.674017 bus /gobi_4: Setting Powered False
22:00:33.780648 bus /gobi_4: Setting Powered False
22:00:34.888897 bus /gobi_1: Setting Powered False
22:00:35.993556 bus /gobi_1: Setting Powered False
22:00:37.111252 bus /gobi_0: Setting Powered False
22:00:38.217151 bus /gobi_0: Setting Powered False
22:00:39.325646 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
bts.start()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
rem_host.recreate_remote_dir(self.remote_inst)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
self.rm_remote_dir(remote_dir)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=31626): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31626)]