rm-remote-dir(pid=9847): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9847)]

Standard Output

----------------------------------------------
trial-2070 gprs:sysmo iperf3.py
----------------------------------------------
15:43:08.191970 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.203027 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
15:43:08.326500 tst                        iperf3.py: using LAC 41710
15:43:08.446927 tst                        iperf3.py: using RAC 145
15:43:08.567460 tst                        iperf3.py: using CellId 41710
15:43:08.689450 tst                        iperf3.py: using BVCI 41711
15:43:08.712488 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.722814 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.733150 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.743335 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.753554 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.763676 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.773992 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.784073 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:08.794427 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:09.087821 tst                      iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
15:43:09.222898 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:09.557092 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9805): Launched
15:43:09.836478 run iperf3-srv_10.42.42.10(pid=9806): Launched
15:43:10.034119 tst                      iperf3.py:8: start network...
15:43:10.168929 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
15:43:10.677719 run          create_hlr_db(pid=9807): Launched
15:43:10.867728 bus                          /gobi_6: Setting Powered False
15:43:11.886235 run          create_hlr_db(pid=9807): Terminated: ok {rc=0}
15:43:12.289150 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9809): Launched
15:43:12.589930 run    osmo-hlr_10.42.42.2(pid=9810): Launched
15:43:12.698993 run              osmo-stp_10.42.42.5: Starting osmo-stp
15:43:13.218834 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9811): Launched
15:43:13.519292 run    osmo-stp_10.42.42.5(pid=9812): Launched
15:43:13.626649 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:43:14.149466 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9813): Launched
15:43:14.256605 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:43:14.518013 run               patchelf(pid=9814): Launched
15:43:15.697175 run               patchelf(pid=9814): Terminated: ok {rc=0}
15:43:15.807645 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
15:43:16.080106 run osmo-gsm-tester_setcap_net_admin.sh(pid=9815): Launched
15:43:17.258805 run osmo-gsm-tester_setcap_net_admin.sh(pid=9815): Terminated: ok {rc=0}
15:43:17.563818 run   osmo-ggsn_10.42.42.6(pid=9818): Launched
15:43:17.762009 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
15:43:18.401454 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9837): Launched
15:43:18.724576 run   osmo-sgsn_10.42.42.7(pid=9838): Launched
15:43:18.842748 run              osmo-msc_10.42.42.8: Starting osmo-msc
15:43:19.436137 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9839): Launched
15:43:19.760089 run    osmo-msc_10.42.42.8(pid=9840): Launched
15:43:22.923554 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
15:43:23.445076 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9841): Launched
15:43:23.746354 run    osmo-mgw_10.42.42.3(pid=9842): Launched
15:43:23.853577 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
15:43:24.364912 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9843): Launched
15:43:24.664878 run    osmo-mgw_10.42.42.4(pid=9844): Launched
15:43:24.771709 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
15:43:25.401038 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9845): Launched
15:43:25.702702 run    osmo-bsc_10.42.42.9(pid=9846): Launched
15:43:31.969761 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
15:43:32.543020 run          rm-remote-dir(pid=9847): Launched
15:43:35.745472 run          rm-remote-dir(pid=9847): ERR: Terminated: ERROR {rc=255}  [trial-2070↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9847)]
15:43:35.888697 run          rm-remote-dir(pid=9847): stdout: 
| (launched: 2022-12-23_15:43:32.378595) 
15:43:36.045316 run          rm-remote-dir(pid=9847): stderr: 
| (launched: 2022-12-23_15:43:32.429827)
| ssh: connect to host 10.42.42.114 port 22: No route to host 
15:43:36.112380 tst                      iperf3.py:8: ERR: Error: rm-remote-dir(pid=9847): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9847)]  [trial-2070↪gprs:sysmo↪iperf3.py:8]
15:43:36.124491 tst                      iperf3.py:8: Test FAILED (28.0 sec)
15:43:36.168864 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9805): Terminating (SIGTERM)
15:43:36.221304 run iperf3-srv_10.42.42.10(pid=9806): Terminating (SIGTERM)
15:43:36.273678 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9809): Terminating (SIGTERM)
15:43:36.324554 run    osmo-hlr_10.42.42.2(pid=9810): Terminating (SIGTERM)
15:43:36.398190 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9811): Terminating (SIGTERM)
15:43:36.493940 run    osmo-stp_10.42.42.5(pid=9812): Terminating (SIGTERM)
15:43:36.618939 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9813): Terminating (SIGTERM)
15:43:36.731422 run   osmo-ggsn_10.42.42.6(pid=9818): Terminating (SIGTERM)
15:43:36.855648 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9837): Terminating (SIGTERM)
15:43:36.943212 run   osmo-sgsn_10.42.42.7(pid=9838): Terminating (SIGTERM)
15:43:36.990839 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9839): Terminating (SIGTERM)
15:43:37.038384 run    osmo-msc_10.42.42.8(pid=9840): Terminating (SIGTERM)
15:43:37.087765 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9841): Terminating (SIGTERM)
15:43:37.137303 run    osmo-mgw_10.42.42.3(pid=9842): Terminating (SIGTERM)
15:43:37.203779 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9843): Terminating (SIGTERM)
15:43:37.253151 run    osmo-mgw_10.42.42.4(pid=9844): Terminating (SIGTERM)
15:43:37.295981 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9845): Terminating (SIGTERM)
15:43:37.339997 run    osmo-bsc_10.42.42.9(pid=9846): Terminating (SIGTERM)
15:43:37.352556 ---      ParallelTerminationStrategy: PID 9805 died...
15:43:37.410908 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9805): Terminated: ok {rc=0}
15:43:37.421794 ---      ParallelTerminationStrategy: PID 9806 died...
15:43:37.479864 run iperf3-srv_10.42.42.10(pid=9806): Terminated {rc=256}
15:43:37.490566 ---      ParallelTerminationStrategy: PID 9809 died...
15:43:37.549774 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9809): Terminated: ok {rc=0}
15:43:37.560481 ---      ParallelTerminationStrategy: PID 9810 died...
15:43:37.620730 run    osmo-hlr_10.42.42.2(pid=9810): Terminated: ok {rc=0}
15:43:37.634072 ---      ParallelTerminationStrategy: PID 9811 died...
15:43:37.710565 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9811): Terminated: ok {rc=0}
15:43:37.721397 ---      ParallelTerminationStrategy: PID 9812 died...
15:43:37.850754 run    osmo-stp_10.42.42.5(pid=9812): Terminated: ok {rc=0}
15:43:37.861620 ---      ParallelTerminationStrategy: PID 9813 died...
15:43:37.920123 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9813): Terminated: ok {rc=0}
15:43:37.930963 ---      ParallelTerminationStrategy: PID 9818 died...
15:43:38.002490 run   osmo-ggsn_10.42.42.6(pid=9818): Terminated: ok {rc=0}
15:43:38.018565 ---      ParallelTerminationStrategy: PID 9837 died...
15:43:38.095450 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9837): Terminated: ok {rc=0}
15:43:38.118578 ---      ParallelTerminationStrategy: PID 9839 died...
15:43:38.190126 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9839): Terminated: ok {rc=0}
15:43:38.201873 ---      ParallelTerminationStrategy: PID 9840 died...
15:43:38.259473 run    osmo-msc_10.42.42.8(pid=9840): Terminated: ok {rc=0}
15:43:38.271120 ---      ParallelTerminationStrategy: PID 9841 died...
15:43:38.328961 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9841): Terminated: ok {rc=0}
15:43:38.340594 ---      ParallelTerminationStrategy: PID 9842 died...
15:43:38.399220 run    osmo-mgw_10.42.42.3(pid=9842): Terminated {rc=15}
15:43:38.411068 ---      ParallelTerminationStrategy: PID 9843 died...
15:43:38.469995 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9843): Terminated: ok {rc=0}
15:43:38.480867 ---      ParallelTerminationStrategy: PID 9844 died...
15:43:38.539395 run    osmo-mgw_10.42.42.4(pid=9844): Terminated {rc=15}
15:43:38.551286 ---      ParallelTerminationStrategy: PID 9845 died...
15:43:38.610022 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9845): Terminated: ok {rc=0}
15:43:38.621021 ---      ParallelTerminationStrategy: PID 9846 died...
15:43:38.677903 run    osmo-bsc_10.42.42.9(pid=9846): Terminated: ok {rc=0}
15:43:38.856616 ---      ParallelTerminationStrategy: PID 9838 died...
15:43:38.915410 run   osmo-sgsn_10.42.42.7(pid=9838): Terminated: ok {rc=0}
15:43:39.002982 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
    setup_run_iperf3_test_parallel(1)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
    rem_host.recreate_remote_dir(self.remote_inst)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
    self.rm_remote_dir(remote_dir)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
    self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=9847): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9847)]