![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3.py (from gprs_sysmo)
Failing for the past 8 builds
(Since Failed )
Stacktrace
rm-remote-dir(pid=9847): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9847)]
Standard Output
----------------------------------------------
trial-2070 gprs:sysmo iperf3.py
----------------------------------------------
15:43:08.191970 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.203027 tst gprs:sysmo: Using 1 x bts (candidates: 1)
15:43:08.326500 tst iperf3.py: using LAC 41710
15:43:08.446927 tst iperf3.py: using RAC 145
15:43:08.567460 tst iperf3.py: using CellId 41710
15:43:08.689450 tst iperf3.py: using BVCI 41711
15:43:08.712488 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.722814 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.733150 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.743335 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.753554 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.763676 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.773992 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.784073 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.794427 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:09.087821 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
15:43:09.222898 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:09.557092 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9805): Launched
15:43:09.836478 run iperf3-srv_10.42.42.10(pid=9806): Launched
15:43:10.034119 tst iperf3.py:8: start network...
15:43:10.168929 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:43:10.677719 run create_hlr_db(pid=9807): Launched
15:43:10.867728 bus /gobi_6: Setting Powered False
15:43:11.886235 run create_hlr_db(pid=9807): Terminated: ok {rc=0}
15:43:12.289150 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9809): Launched
15:43:12.589930 run osmo-hlr_10.42.42.2(pid=9810): Launched
15:43:12.698993 run osmo-stp_10.42.42.5: Starting osmo-stp
15:43:13.218834 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9811): Launched
15:43:13.519292 run osmo-stp_10.42.42.5(pid=9812): Launched
15:43:13.626649 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:43:14.149466 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9813): Launched
15:43:14.256605 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:43:14.518013 run patchelf(pid=9814): Launched
15:43:15.697175 run patchelf(pid=9814): Terminated: ok {rc=0}
15:43:15.807645 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
15:43:16.080106 run osmo-gsm-tester_setcap_net_admin.sh(pid=9815): Launched
15:43:17.258805 run osmo-gsm-tester_setcap_net_admin.sh(pid=9815): Terminated: ok {rc=0}
15:43:17.563818 run osmo-ggsn_10.42.42.6(pid=9818): Launched
15:43:17.762009 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
15:43:18.401454 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9837): Launched
15:43:18.724576 run osmo-sgsn_10.42.42.7(pid=9838): Launched
15:43:18.842748 run osmo-msc_10.42.42.8: Starting osmo-msc
15:43:19.436137 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9839): Launched
15:43:19.760089 run osmo-msc_10.42.42.8(pid=9840): Launched
15:43:22.923554 run osmo-mgw_10.42.42.3: Starting osmo-mgw
15:43:23.445076 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9841): Launched
15:43:23.746354 run osmo-mgw_10.42.42.3(pid=9842): Launched
15:43:23.853577 run osmo-mgw_10.42.42.4: Starting osmo-mgw
15:43:24.364912 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9843): Launched
15:43:24.664878 run osmo-mgw_10.42.42.4(pid=9844): Launched
15:43:24.771709 run osmo-bsc_10.42.42.9: Starting osmo-bsc
15:43:25.401038 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9845): Launched
15:43:25.702702 run osmo-bsc_10.42.42.9(pid=9846): Launched
15:43:31.969761 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
15:43:32.543020 run rm-remote-dir(pid=9847): Launched
15:43:35.745472 run rm-remote-dir(pid=9847): ERR: Terminated: ERROR {rc=255} [trial-2070↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9847)]
15:43:35.888697 run rm-remote-dir(pid=9847): stdout:
| (launched: 2022-12-23_15:43:32.378595)
15:43:36.045316 run rm-remote-dir(pid=9847): stderr:
| (launched: 2022-12-23_15:43:32.429827)
| ssh: connect to host 10.42.42.114 port 22: No route to host
15:43:36.112380 tst iperf3.py:8: ERR: Error: rm-remote-dir(pid=9847): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9847)] [trial-2070↪gprs:sysmo↪iperf3.py:8]
15:43:36.124491 tst iperf3.py:8: Test FAILED (28.0 sec)
15:43:36.168864 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9805): Terminating (SIGTERM)
15:43:36.221304 run iperf3-srv_10.42.42.10(pid=9806): Terminating (SIGTERM)
15:43:36.273678 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9809): Terminating (SIGTERM)
15:43:36.324554 run osmo-hlr_10.42.42.2(pid=9810): Terminating (SIGTERM)
15:43:36.398190 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9811): Terminating (SIGTERM)
15:43:36.493940 run osmo-stp_10.42.42.5(pid=9812): Terminating (SIGTERM)
15:43:36.618939 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9813): Terminating (SIGTERM)
15:43:36.731422 run osmo-ggsn_10.42.42.6(pid=9818): Terminating (SIGTERM)
15:43:36.855648 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9837): Terminating (SIGTERM)
15:43:36.943212 run osmo-sgsn_10.42.42.7(pid=9838): Terminating (SIGTERM)
15:43:36.990839 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9839): Terminating (SIGTERM)
15:43:37.038384 run osmo-msc_10.42.42.8(pid=9840): Terminating (SIGTERM)
15:43:37.087765 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9841): Terminating (SIGTERM)
15:43:37.137303 run osmo-mgw_10.42.42.3(pid=9842): Terminating (SIGTERM)
15:43:37.203779 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9843): Terminating (SIGTERM)
15:43:37.253151 run osmo-mgw_10.42.42.4(pid=9844): Terminating (SIGTERM)
15:43:37.295981 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9845): Terminating (SIGTERM)
15:43:37.339997 run osmo-bsc_10.42.42.9(pid=9846): Terminating (SIGTERM)
15:43:37.352556 --- ParallelTerminationStrategy: PID 9805 died...
15:43:37.410908 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9805): Terminated: ok {rc=0}
15:43:37.421794 --- ParallelTerminationStrategy: PID 9806 died...
15:43:37.479864 run iperf3-srv_10.42.42.10(pid=9806): Terminated {rc=256}
15:43:37.490566 --- ParallelTerminationStrategy: PID 9809 died...
15:43:37.549774 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9809): Terminated: ok {rc=0}
15:43:37.560481 --- ParallelTerminationStrategy: PID 9810 died...
15:43:37.620730 run osmo-hlr_10.42.42.2(pid=9810): Terminated: ok {rc=0}
15:43:37.634072 --- ParallelTerminationStrategy: PID 9811 died...
15:43:37.710565 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9811): Terminated: ok {rc=0}
15:43:37.721397 --- ParallelTerminationStrategy: PID 9812 died...
15:43:37.850754 run osmo-stp_10.42.42.5(pid=9812): Terminated: ok {rc=0}
15:43:37.861620 --- ParallelTerminationStrategy: PID 9813 died...
15:43:37.920123 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9813): Terminated: ok {rc=0}
15:43:37.930963 --- ParallelTerminationStrategy: PID 9818 died...
15:43:38.002490 run osmo-ggsn_10.42.42.6(pid=9818): Terminated: ok {rc=0}
15:43:38.018565 --- ParallelTerminationStrategy: PID 9837 died...
15:43:38.095450 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9837): Terminated: ok {rc=0}
15:43:38.118578 --- ParallelTerminationStrategy: PID 9839 died...
15:43:38.190126 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9839): Terminated: ok {rc=0}
15:43:38.201873 --- ParallelTerminationStrategy: PID 9840 died...
15:43:38.259473 run osmo-msc_10.42.42.8(pid=9840): Terminated: ok {rc=0}
15:43:38.271120 --- ParallelTerminationStrategy: PID 9841 died...
15:43:38.328961 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9841): Terminated: ok {rc=0}
15:43:38.340594 --- ParallelTerminationStrategy: PID 9842 died...
15:43:38.399220 run osmo-mgw_10.42.42.3(pid=9842): Terminated {rc=15}
15:43:38.411068 --- ParallelTerminationStrategy: PID 9843 died...
15:43:38.469995 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9843): Terminated: ok {rc=0}
15:43:38.480867 --- ParallelTerminationStrategy: PID 9844 died...
15:43:38.539395 run osmo-mgw_10.42.42.4(pid=9844): Terminated {rc=15}
15:43:38.551286 --- ParallelTerminationStrategy: PID 9845 died...
15:43:38.610022 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9845): Terminated: ok {rc=0}
15:43:38.621021 --- ParallelTerminationStrategy: PID 9846 died...
15:43:38.677903 run osmo-bsc_10.42.42.9(pid=9846): Terminated: ok {rc=0}
15:43:38.856616 --- ParallelTerminationStrategy: PID 9838 died...
15:43:38.915410 run osmo-sgsn_10.42.42.7(pid=9838): Terminated: ok {rc=0}
15:43:39.002982 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
bts.start()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
rem_host.recreate_remote_dir(self.remote_inst)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
self.rm_remote_dir(remote_dir)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=9847): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9847)]