![[Jenkins]](/jenkins/static/433051ca/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3m4.py (from gprs_sysmo)
Failing for the past 8 builds
(Since Failed )
Stacktrace
rm-remote-dir(pid=9913): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9913)]
Standard Output
----------------------------------------------
trial-2070 gprs:sysmo iperf3m4.py
----------------------------------------------
15:43:40.164043 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:40.175414 tst gprs:sysmo: Using 1 x bts (candidates: 1)
15:43:40.298568 tst iperf3m4.py: using LAC 41711
15:43:40.419695 tst iperf3m4.py: using RAC 146
15:43:40.550632 tst iperf3m4.py: using CellId 41711
15:43:40.672346 tst iperf3m4.py: using BVCI 41712
15:43:40.695492 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:40.706210 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:40.716727 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:40.727388 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:40.737961 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:40.748500 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:40.759200 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:40.769916 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:40.780748 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:40.898721 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:41.015308 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:41.134394 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:41.428496 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
15:43:41.564330 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:41.898929 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9863): Launched
15:43:42.176632 run iperf3-srv_10.42.42.10(pid=9864): Launched
15:43:42.375561 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
15:43:42.512027 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:42.846326 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9865): Launched
15:43:43.124252 run iperf3-srv_10.42.42.10(pid=9866): Launched
15:43:43.323164 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
15:43:43.459482 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:43.794128 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9867): Launched
15:43:44.070939 run iperf3-srv_10.42.42.10(pid=9868): Launched
15:43:44.269012 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
15:43:44.404349 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:44.739806 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9869): Launched
15:43:45.017608 run iperf3-srv_10.42.42.10(pid=9870): Launched
15:43:45.216438 tst iperf3m4.py:8: start network...
15:43:45.351983 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:43:45.862581 run create_hlr_db(pid=9871): Launched
15:43:46.050865 bus /gobi_6: Setting Powered False
15:43:47.072847 run create_hlr_db(pid=9871): Terminated: ok {rc=0}
15:43:47.504367 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9873): Launched
15:43:47.828890 run osmo-hlr_10.42.42.2(pid=9874): Launched
15:43:47.945231 run osmo-stp_10.42.42.5: Starting osmo-stp
15:43:48.503941 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9875): Launched
15:43:48.827062 run osmo-stp_10.42.42.5(pid=9876): Launched
15:43:48.942943 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:43:49.506211 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9877): Launched
15:43:49.622134 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:43:49.904137 run patchelf(pid=9878): Launched
15:43:51.099368 run patchelf(pid=9878): Terminated: ok {rc=0}
15:43:51.224014 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
15:43:51.537049 run osmo-gsm-tester_setcap_net_admin.sh(pid=9879): Launched
15:43:52.730631 run osmo-gsm-tester_setcap_net_admin.sh(pid=9879): Terminated: ok {rc=0}
15:43:53.060789 run osmo-ggsn_10.42.42.6(pid=9882): Launched
15:43:53.283877 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
15:43:53.890463 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9903): Launched
15:43:54.214534 run osmo-sgsn_10.42.42.7(pid=9904): Launched
15:43:54.333556 run osmo-msc_10.42.42.8: Starting osmo-msc
15:43:54.926590 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9905): Launched
15:43:55.250318 run osmo-msc_10.42.42.8(pid=9906): Launched
15:43:58.425231 run osmo-mgw_10.42.42.3: Starting osmo-mgw
15:43:58.986158 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9907): Launched
15:43:59.310531 run osmo-mgw_10.42.42.3(pid=9908): Launched
15:43:59.427413 run osmo-mgw_10.42.42.4: Starting osmo-mgw
15:43:59.977138 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9909): Launched
15:44:00.300726 run osmo-mgw_10.42.42.4(pid=9910): Launched
15:44:00.417677 run osmo-bsc_10.42.42.9: Starting osmo-bsc
15:44:01.112410 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9911): Launched
15:44:01.439684 run osmo-bsc_10.42.42.9(pid=9912): Launched
15:44:07.716975 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
15:44:08.292517 run rm-remote-dir(pid=9913): Launched
15:44:11.497050 run rm-remote-dir(pid=9913): ERR: Terminated: ERROR {rc=255} [trial-2070↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9913)]
15:44:11.639710 run rm-remote-dir(pid=9913): stdout:
| (launched: 2022-12-23_15:44:08.127336)
15:44:11.785654 run rm-remote-dir(pid=9913): stderr:
| (launched: 2022-12-23_15:44:08.178659)
| ssh: connect to host 10.42.42.114 port 22: No route to host
15:44:11.845030 tst iperf3m4.py:8: ERR: Error: rm-remote-dir(pid=9913): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9913)] [trial-2070↪gprs:sysmo↪iperf3m4.py:8]
15:44:11.853492 tst iperf3m4.py:8: Test FAILED (31.7 sec)
15:44:11.890929 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9863): Terminating (SIGTERM)
15:44:11.933259 run iperf3-srv_10.42.42.10(pid=9864): Terminating (SIGTERM)
15:44:11.975119 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9865): Terminating (SIGTERM)
15:44:12.016870 run iperf3-srv_10.42.42.10(pid=9866): Terminating (SIGTERM)
15:44:12.058511 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9867): Terminating (SIGTERM)
15:44:12.100330 run iperf3-srv_10.42.42.10(pid=9868): Terminating (SIGTERM)
15:44:12.142165 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9869): Terminating (SIGTERM)
15:44:12.184090 run iperf3-srv_10.42.42.10(pid=9870): Terminating (SIGTERM)
15:44:12.225761 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9873): Terminating (SIGTERM)
15:44:12.267797 run osmo-hlr_10.42.42.2(pid=9874): Terminating (SIGTERM)
15:44:12.310627 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9875): Terminating (SIGTERM)
15:44:12.353266 run osmo-stp_10.42.42.5(pid=9876): Terminating (SIGTERM)
15:44:12.416885 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9877): Terminating (SIGTERM)
15:44:12.468048 run osmo-ggsn_10.42.42.6(pid=9882): Terminating (SIGTERM)
15:44:12.563337 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9903): Terminating (SIGTERM)
15:44:12.608079 run osmo-sgsn_10.42.42.7(pid=9904): Terminating (SIGTERM)
15:44:12.652653 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9905): Terminating (SIGTERM)
15:44:12.698057 run osmo-msc_10.42.42.8(pid=9906): Terminating (SIGTERM)
15:44:12.741893 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9907): Terminating (SIGTERM)
15:44:12.785649 run osmo-mgw_10.42.42.3(pid=9908): Terminating (SIGTERM)
15:44:12.828829 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9909): Terminating (SIGTERM)
15:44:12.870589 run osmo-mgw_10.42.42.4(pid=9910): Terminating (SIGTERM)
15:44:12.913311 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9911): Terminating (SIGTERM)
15:44:12.954415 run osmo-bsc_10.42.42.9(pid=9912): Terminating (SIGTERM)
15:44:12.964310 --- ParallelTerminationStrategy: PID 9863 died...
15:44:13.020081 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9863): Terminated: ok {rc=0}
15:44:13.030122 --- ParallelTerminationStrategy: PID 9864 died...
15:44:13.085963 run iperf3-srv_10.42.42.10(pid=9864): Terminated {rc=256}
15:44:13.096032 --- ParallelTerminationStrategy: PID 9865 died...
15:44:13.151669 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9865): Terminated: ok {rc=0}
15:44:13.161882 --- ParallelTerminationStrategy: PID 9866 died...
15:44:13.217598 run iperf3-srv_10.42.42.10(pid=9866): Terminated {rc=256}
15:44:13.227648 --- ParallelTerminationStrategy: PID 9867 died...
15:44:13.283484 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9867): Terminated: ok {rc=0}
15:44:13.293565 --- ParallelTerminationStrategy: PID 9868 died...
15:44:13.349687 run iperf3-srv_10.42.42.10(pid=9868): Terminated {rc=256}
15:44:13.360580 --- ParallelTerminationStrategy: PID 9869 died...
15:44:13.418309 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9869): Terminated: ok {rc=0}
15:44:13.429259 --- ParallelTerminationStrategy: PID 9870 died...
15:44:13.486285 run iperf3-srv_10.42.42.10(pid=9870): Terminated {rc=256}
15:44:13.497245 --- ParallelTerminationStrategy: PID 9873 died...
15:44:13.556575 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9873): Terminated: ok {rc=0}
15:44:13.567633 --- ParallelTerminationStrategy: PID 9874 died...
15:44:13.690818 run osmo-hlr_10.42.42.2(pid=9874): Terminated: ok {rc=0}
15:44:13.708976 --- ParallelTerminationStrategy: PID 9875 died...
15:44:13.818543 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9875): Terminated: ok {rc=0}
15:44:13.829698 --- ParallelTerminationStrategy: PID 9876 died...
15:44:13.889203 run osmo-stp_10.42.42.5(pid=9876): Terminated: ok {rc=0}
15:44:13.900433 --- ParallelTerminationStrategy: PID 9877 died...
15:44:13.959691 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9877): Terminated: ok {rc=0}
15:44:13.970764 --- ParallelTerminationStrategy: PID 9882 died...
15:44:14.030164 run osmo-ggsn_10.42.42.6(pid=9882): Terminated: ok {rc=0}
15:44:14.041341 --- ParallelTerminationStrategy: PID 9903 died...
15:44:14.100452 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9903): Terminated: ok {rc=0}
15:44:14.111556 --- ParallelTerminationStrategy: PID 9905 died...
15:44:14.170843 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9905): Terminated: ok {rc=0}
15:44:14.182013 --- ParallelTerminationStrategy: PID 9906 died...
15:44:14.241026 run osmo-msc_10.42.42.8(pid=9906): Terminated: ok {rc=0}
15:44:14.252128 --- ParallelTerminationStrategy: PID 9907 died...
15:44:14.311485 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9907): Terminated: ok {rc=0}
15:44:14.322682 --- ParallelTerminationStrategy: PID 9908 died...
15:44:14.381994 run osmo-mgw_10.42.42.3(pid=9908): Terminated {rc=15}
15:44:14.392427 --- ParallelTerminationStrategy: PID 9904 died...
15:44:14.449012 run osmo-sgsn_10.42.42.7(pid=9904): Terminated: ok {rc=0}
15:44:14.459218 --- ParallelTerminationStrategy: PID 9909 died...
15:44:14.515497 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9909): Terminated: ok {rc=0}
15:44:14.525854 --- ParallelTerminationStrategy: PID 9910 died...
15:44:14.582308 run osmo-mgw_10.42.42.4(pid=9910): Terminated {rc=15}
15:44:14.592612 --- ParallelTerminationStrategy: PID 9911 died...
15:44:14.648786 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9911): Terminated: ok {rc=0}
15:44:14.659128 --- ParallelTerminationStrategy: PID 9912 died...
15:44:14.714906 run osmo-bsc_10.42.42.9(pid=9912): Terminated: ok {rc=0}
15:44:14.801993 bus /gobi_4: Setting Powered False
15:44:15.908481 bus /gobi_4: Setting Powered False
15:44:17.016006 bus /gobi_1: Setting Powered False
15:44:18.122836 bus /gobi_1: Setting Powered False
15:44:19.230669 bus /gobi_0: Setting Powered False
15:44:20.336274 bus /gobi_0: Setting Powered False
15:44:21.454818 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
bts.start()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
rem_host.recreate_remote_dir(self.remote_inst)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
self.rm_remote_dir(remote_dir)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=9913): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9913)]