Stack Trace
rm-remote-dir(pid=13869): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13869)]Standard Output
----------------------------------------------
trial-2072 gprs:sysmo iperf3m4.py
----------------------------------------------
16:55:49.204549 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:49.216284 tst gprs:sysmo: Using 1 x bts (candidates: 1)
16:55:49.345585 tst iperf3m4.py: using LAC 41725
16:55:49.473148 tst iperf3m4.py: using RAC 160
16:55:49.600777 tst iperf3m4.py: using CellId 41725
16:55:49.729104 tst iperf3m4.py: using BVCI 41726
16:55:49.753824 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:49.764783 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:49.775658 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:49.786717 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:49.797632 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:49.808647 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:49.819691 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:49.830794 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:49.842014 tst gprs:sysmo: Using 1 x modem (candidates: 4)
16:55:49.962207 tst gprs:sysmo: Using 1 x modem (candidates: 4)
16:55:50.079545 tst gprs:sysmo: Using 1 x modem (candidates: 4)
16:55:50.199706 tst gprs:sysmo: Using 1 x modem (candidates: 4)
16:55:50.496231 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
16:55:50.631638 run iperf3-srv_10.42.42.10: Starting iperf3-srv
16:55:50.985507 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13551): Launched
16:55:51.256002 run iperf3-srv_10.42.42.10(pid=13552): Launched
16:55:51.455203 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
16:55:51.600455 run iperf3-srv_10.42.42.10: Starting iperf3-srv
16:55:52.030469 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13589): Launched
16:55:52.330442 run iperf3-srv_10.42.42.10(pid=13602): Launched
16:55:52.532054 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
16:55:52.680395 run iperf3-srv_10.42.42.10: Starting iperf3-srv
16:55:53.049883 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13628): Launched
16:55:53.384541 run iperf3-srv_10.42.42.10(pid=13647): Launched
16:55:53.618320 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
16:55:53.775809 run iperf3-srv_10.42.42.10: Starting iperf3-srv
16:55:54.175287 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13753): Launched
16:55:54.498138 run iperf3-srv_10.42.42.10(pid=13799): Launched
16:55:54.714890 tst iperf3m4.py:8: start network...
16:55:54.845254 run osmo-hlr_10.42.42.2: Starting osmo-hlr
16:55:55.390481 run create_hlr_db(pid=13829): Launched
16:55:55.584698 bus /gobi_6: Setting Powered False
16:55:56.604444 run create_hlr_db(pid=13829): Terminated: ok {rc=0}
16:55:57.035946 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13832): Launched
16:55:57.446464 run osmo-hlr_10.42.42.2(pid=13834): Launched
16:55:57.614972 run osmo-stp_10.42.42.5: Starting osmo-stp
16:55:58.193972 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13835): Launched
16:55:58.606910 run osmo-stp_10.42.42.5(pid=13836): Launched
16:55:58.768869 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
16:55:59.314930 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13837): Launched
16:55:59.424002 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
16:55:59.712293 run patchelf(pid=13838): Launched
16:56:00.901114 run patchelf(pid=13838): Terminated: ok {rc=0}
16:56:01.018212 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
16:56:01.326233 run osmo-gsm-tester_setcap_net_admin.sh(pid=13839): Launched
16:56:02.516111 run osmo-gsm-tester_setcap_net_admin.sh(pid=13839): Terminated: ok {rc=0}
16:56:02.928622 run osmo-ggsn_10.42.42.6(pid=13842): Launched
16:56:03.167036 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
16:56:03.890515 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13859): Launched
16:56:04.315757 run osmo-sgsn_10.42.42.7(pid=13860): Launched
16:56:04.562647 run osmo-msc_10.42.42.8: Starting osmo-msc
16:56:05.246184 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13861): Launched
16:56:05.675189 run osmo-msc_10.42.42.8(pid=13862): Launched
16:56:08.872418 run osmo-mgw_10.42.42.3: Starting osmo-mgw
16:56:09.429851 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13863): Launched
16:56:09.840034 run osmo-mgw_10.42.42.3(pid=13864): Launched
16:56:10.014097 run osmo-mgw_10.42.42.4: Starting osmo-mgw
16:56:10.558603 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13865): Launched
16:56:10.976446 run osmo-mgw_10.42.42.4(pid=13866): Launched
16:56:11.141323 run osmo-bsc_10.42.42.9: Starting osmo-bsc
16:56:11.827975 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13867): Launched
16:56:12.244935 run osmo-bsc_10.42.42.9(pid=13868): Launched
16:56:18.526382 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
16:56:19.065704 run rm-remote-dir(pid=13869): Launched
16:56:22.295517 run rm-remote-dir(pid=13869): ERR: Terminated: ERROR {rc=255} [trial-2072↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13869)]
16:56:22.429527 run rm-remote-dir(pid=13869): stdout:
| (launched: 2023-03-07_16:56:18.910430)
16:56:22.563300 run rm-remote-dir(pid=13869): stderr:
| (launched: 2023-03-07_16:56:18.958435)
| ssh: connect to host 10.42.42.114 port 22: No route to host
16:56:22.618816 tst iperf3m4.py:8: ERR: Error: rm-remote-dir(pid=13869): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13869)] [trial-2072↪gprs:sysmo↪iperf3m4.py:8]
16:56:22.626578 tst iperf3m4.py:8: Test FAILED (33.5 sec)
16:56:22.661841 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13551): Terminating (SIGTERM)
16:56:22.701960 run iperf3-srv_10.42.42.10(pid=13552): Terminating (SIGTERM)
16:56:22.741991 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13589): Terminating (SIGTERM)
16:56:22.781976 run iperf3-srv_10.42.42.10(pid=13602): Terminating (SIGTERM)
16:56:22.821946 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13628): Terminating (SIGTERM)
16:56:22.861776 run iperf3-srv_10.42.42.10(pid=13647): Terminating (SIGTERM)
16:56:22.901521 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13753): Terminating (SIGTERM)
16:56:22.941261 run iperf3-srv_10.42.42.10(pid=13799): Terminating (SIGTERM)
16:56:22.981234 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13832): Terminating (SIGTERM)
16:56:23.021163 run osmo-hlr_10.42.42.2(pid=13834): Terminating (SIGTERM)
16:56:23.062052 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13835): Terminating (SIGTERM)
16:56:23.105520 run osmo-stp_10.42.42.5(pid=13836): Terminating (SIGTERM)
16:56:23.148688 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13837): Terminating (SIGTERM)
16:56:23.192837 run osmo-ggsn_10.42.42.6(pid=13842): Terminating (SIGTERM)
16:56:23.252083 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13859): Terminating (SIGTERM)
16:56:23.308481 run osmo-sgsn_10.42.42.7(pid=13860): Terminating (SIGTERM)
16:56:23.350413 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13861): Terminating (SIGTERM)
16:56:23.391782 run osmo-msc_10.42.42.8(pid=13862): Terminating (SIGTERM)
16:56:23.433035 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13863): Terminating (SIGTERM)
16:56:23.473191 run osmo-mgw_10.42.42.3(pid=13864): Terminating (SIGTERM)
16:56:23.513805 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13865): Terminating (SIGTERM)
16:56:23.553866 run osmo-mgw_10.42.42.4(pid=13866): Terminating (SIGTERM)
16:56:23.594507 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13867): Terminating (SIGTERM)
16:56:23.634453 run osmo-bsc_10.42.42.9(pid=13868): Terminating (SIGTERM)
16:56:23.643534 --- ParallelTerminationStrategy: PID 13551 died...
16:56:23.697170 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13551): Terminated: ok {rc=0}
16:56:23.706616 --- ParallelTerminationStrategy: PID 13552 died...
16:56:23.759835 run iperf3-srv_10.42.42.10(pid=13552): Terminated {rc=256}
16:56:23.769344 --- ParallelTerminationStrategy: PID 13589 died...
16:56:23.824015 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13589): Terminated: ok {rc=0}
16:56:23.833965 --- ParallelTerminationStrategy: PID 13602 died...
16:56:23.888668 run iperf3-srv_10.42.42.10(pid=13602): Terminated {rc=256}
16:56:23.898859 --- ParallelTerminationStrategy: PID 13628 died...
16:56:23.953062 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13628): Terminated: ok {rc=0}
16:56:23.962558 --- ParallelTerminationStrategy: PID 13647 died...
16:56:24.015786 run iperf3-srv_10.42.42.10(pid=13647): Terminated {rc=256}
16:56:24.025258 --- ParallelTerminationStrategy: PID 13753 died...
16:56:24.078697 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13753): Terminated: ok {rc=0}
16:56:24.088225 --- ParallelTerminationStrategy: PID 13799 died...
16:56:24.141522 run iperf3-srv_10.42.42.10(pid=13799): Terminated {rc=256}
16:56:24.150947 --- ParallelTerminationStrategy: PID 13832 died...
16:56:24.204387 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13832): Terminated: ok {rc=0}
16:56:24.213925 --- ParallelTerminationStrategy: PID 13834 died...
16:56:24.267045 run osmo-hlr_10.42.42.2(pid=13834): Terminated: ok {rc=0}
16:56:24.276552 --- ParallelTerminationStrategy: PID 13835 died...
16:56:24.330242 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13835): Terminated: ok {rc=0}
16:56:24.340219 --- ParallelTerminationStrategy: PID 13836 died...
16:56:24.394725 run osmo-stp_10.42.42.5(pid=13836): Terminated: ok {rc=0}
16:56:24.421726 --- ParallelTerminationStrategy: PID 13837 died...
16:56:24.541848 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13837): Terminated: ok {rc=0}
16:56:24.560720 --- ParallelTerminationStrategy: PID 13842 died...
16:56:24.675288 run osmo-ggsn_10.42.42.6(pid=13842): Terminated: ok {rc=0}
16:56:24.686206 --- ParallelTerminationStrategy: PID 13859 died...
16:56:24.743750 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13859): Terminated: ok {rc=0}
16:56:24.758374 --- ParallelTerminationStrategy: PID 13861 died...
16:56:24.811386 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13861): Terminated: ok {rc=0}
16:56:24.821187 --- ParallelTerminationStrategy: PID 13860 died...
16:56:24.874021 run osmo-sgsn_10.42.42.7(pid=13860): Terminated: ok {rc=0}
16:56:24.883756 --- ParallelTerminationStrategy: PID 13862 died...
16:56:24.937001 run osmo-msc_10.42.42.8(pid=13862): Terminated: ok {rc=0}
16:56:24.946746 --- ParallelTerminationStrategy: PID 13863 died...
16:56:24.999742 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13863): Terminated: ok {rc=0}
16:56:25.009530 --- ParallelTerminationStrategy: PID 13864 died...
16:56:25.062948 run osmo-mgw_10.42.42.3(pid=13864): Terminated {rc=15}
16:56:25.072837 --- ParallelTerminationStrategy: PID 13865 died...
16:56:25.125942 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13865): Terminated: ok {rc=0}
16:56:25.136092 --- ParallelTerminationStrategy: PID 13866 died...
16:56:25.189620 run osmo-mgw_10.42.42.4(pid=13866): Terminated {rc=15}
16:56:25.199425 --- ParallelTerminationStrategy: PID 13867 died...
16:56:25.252772 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13867): Terminated: ok {rc=0}
16:56:25.262634 --- ParallelTerminationStrategy: PID 13868 died...
16:56:25.315782 run osmo-bsc_10.42.42.9(pid=13868): Terminated: ok {rc=0}
16:56:25.404668 bus /gobi_4: Setting Powered False
16:56:26.515800 bus /gobi_4: Setting Powered False
16:56:27.624860 bus /gobi_1: Setting Powered False
16:56:28.732623 bus /gobi_1: Setting Powered False
16:56:29.841440 bus /gobi_0: Setting Powered False
16:56:30.948659 bus /gobi_0: Setting Powered False
16:56:32.056687 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
bts.start()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
rem_host.recreate_remote_dir(self.remote_inst)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
self.rm_remote_dir(remote_dir)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=13869): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13869)]