rm-remote-dir(pid=31563): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31563)]

Standard Output

----------------------------------------------
trial-2071 gprs:sysmo iperf3.py
----------------------------------------------
21:59:26.692580 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:26.704327 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
21:59:26.839554 tst                        iperf3.py: using LAC 41717
21:59:26.969825 tst                        iperf3.py: using RAC 152
21:59:27.099797 tst                        iperf3.py: using CellId 41717
21:59:27.240908 tst                        iperf3.py: using BVCI 41718
21:59:27.267017 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:27.278062 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:27.289034 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:27.299914 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:27.310807 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:27.321644 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:27.332665 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:27.343474 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:59:27.354558 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
21:59:27.651289 tst                      iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
21:59:27.788135 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
21:59:28.126617 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31521): Launched
21:59:28.408229 run iperf3-srv_10.42.42.10(pid=31522): Launched
21:59:28.609276 tst                      iperf3.py:8: start network...
21:59:28.745870 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
21:59:29.270460 run         create_hlr_db(pid=31523): Launched
21:59:29.463266 bus                          /gobi_6: Setting Powered False
21:59:30.482174 run         create_hlr_db(pid=31523): Terminated: ok {rc=0}
21:59:30.893254 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31525): Launched
21:59:31.201238 run   osmo-hlr_10.42.42.2(pid=31526): Launched
21:59:31.312316 run              osmo-stp_10.42.42.5: Starting osmo-stp
21:59:31.842909 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31527): Launched
21:59:32.149299 run   osmo-stp_10.42.42.5(pid=31528): Launched
21:59:32.258157 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:59:32.818638 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31529): Launched
21:59:32.927857 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:59:33.194867 run              patchelf(pid=31530): Launched
21:59:34.375830 run              patchelf(pid=31530): Terminated: ok {rc=0}
21:59:34.488872 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
21:59:34.764875 run osmo-gsm-tester_setcap_net_admin.sh(pid=31531): Launched
21:59:35.980850 run osmo-gsm-tester_setcap_net_admin.sh(pid=31531): Terminated: ok {rc=0}
21:59:36.519317 run  osmo-ggsn_10.42.42.6(pid=31534): Launched
21:59:36.738793 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
21:59:37.356470 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31553): Launched
21:59:37.667391 run  osmo-sgsn_10.42.42.7(pid=31554): Launched
21:59:37.782510 run              osmo-msc_10.42.42.8: Starting osmo-msc
21:59:38.351039 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31555): Launched
21:59:38.659125 run   osmo-msc_10.42.42.8(pid=31556): Launched
21:59:41.822549 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
21:59:42.351542 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31557): Launched
21:59:42.659572 run   osmo-mgw_10.42.42.3(pid=31558): Launched
21:59:42.769021 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
21:59:43.288047 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31559): Launched
21:59:43.594766 run   osmo-mgw_10.42.42.4(pid=31560): Launched
21:59:43.703834 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
21:59:44.343055 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31561): Launched
21:59:44.649951 run   osmo-bsc_10.42.42.9(pid=31562): Launched
21:59:50.923507 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
21:59:51.504267 run         rm-remote-dir(pid=31563): Launched
21:59:54.697986 run         rm-remote-dir(pid=31563): ERR: Terminated: ERROR {rc=255}  [trial-2071↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31563)]
21:59:54.832272 run         rm-remote-dir(pid=31563): stdout: 
| (launched: 2023-01-03_21:59:51.321004) 
21:59:54.969796 run         rm-remote-dir(pid=31563): stderr: 
| (launched: 2023-01-03_21:59:51.370310)
| ssh: connect to host 10.42.42.114 port 22: No route to host 
21:59:55.025595 tst                      iperf3.py:8: ERR: Error: rm-remote-dir(pid=31563): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31563)]  [trial-2071↪gprs:sysmo↪iperf3.py:8]
21:59:55.033588 tst                      iperf3.py:8: Test FAILED (28.4 sec)
21:59:55.069311 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31521): Terminating (SIGTERM)
21:59:55.109395 run iperf3-srv_10.42.42.10(pid=31522): Terminating (SIGTERM)
21:59:55.150092 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31525): Terminating (SIGTERM)
21:59:55.190085 run   osmo-hlr_10.42.42.2(pid=31526): Terminating (SIGTERM)
21:59:55.232365 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31527): Terminating (SIGTERM)
21:59:55.275054 run   osmo-stp_10.42.42.5(pid=31528): Terminating (SIGTERM)
21:59:55.317627 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31529): Terminating (SIGTERM)
21:59:55.360380 run  osmo-ggsn_10.42.42.6(pid=31534): Terminating (SIGTERM)
21:59:55.406439 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31553): Terminating (SIGTERM)
21:59:55.458817 run  osmo-sgsn_10.42.42.7(pid=31554): Terminating (SIGTERM)
21:59:55.503588 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31555): Terminating (SIGTERM)
21:59:55.548545 run   osmo-msc_10.42.42.8(pid=31556): Terminating (SIGTERM)
21:59:55.590989 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31557): Terminating (SIGTERM)
21:59:55.634344 run   osmo-mgw_10.42.42.3(pid=31558): Terminating (SIGTERM)
21:59:55.676274 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31559): Terminating (SIGTERM)
21:59:55.716747 run   osmo-mgw_10.42.42.4(pid=31560): Terminating (SIGTERM)
21:59:55.758262 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31561): Terminating (SIGTERM)
21:59:55.800854 run   osmo-bsc_10.42.42.9(pid=31562): Terminating (SIGTERM)
21:59:55.811333 ---      ParallelTerminationStrategy: PID 31521 died...
21:59:55.876406 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31521): Terminated: ok {rc=0}
21:59:55.895621 ---      ParallelTerminationStrategy: PID 31522 died...
21:59:55.953190 run iperf3-srv_10.42.42.10(pid=31522): Terminated {rc=256}
21:59:55.963765 ---      ParallelTerminationStrategy: PID 31525 died...
21:59:56.022863 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31525): Terminated: ok {rc=0}
21:59:56.033957 ---      ParallelTerminationStrategy: PID 31526 died...
21:59:56.097302 run   osmo-hlr_10.42.42.2(pid=31526): Terminated: ok {rc=0}
21:59:56.107023 ---      ParallelTerminationStrategy: PID 31527 died...
21:59:56.160849 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31527): Terminated: ok {rc=0}
21:59:56.170604 ---      ParallelTerminationStrategy: PID 31528 died...
21:59:56.224405 run   osmo-stp_10.42.42.5(pid=31528): Terminated: ok {rc=0}
21:59:56.233945 ---      ParallelTerminationStrategy: PID 31529 died...
21:59:56.287198 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31529): Terminated: ok {rc=0}
21:59:56.297112 ---      ParallelTerminationStrategy: PID 31534 died...
21:59:56.350477 run  osmo-ggsn_10.42.42.6(pid=31534): Terminated: ok {rc=0}
21:59:56.360312 ---      ParallelTerminationStrategy: PID 31553 died...
21:59:56.413834 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31553): Terminated: ok {rc=0}
21:59:56.423542 ---      ParallelTerminationStrategy: PID 31555 died...
21:59:56.477822 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31555): Terminated: ok {rc=0}
21:59:56.488530 ---      ParallelTerminationStrategy: PID 31556 died...
21:59:56.545178 run   osmo-msc_10.42.42.8(pid=31556): Terminated: ok {rc=0}
21:59:56.556008 ---      ParallelTerminationStrategy: PID 31557 died...
21:59:56.613916 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31557): Terminated: ok {rc=0}
21:59:56.624937 ---      ParallelTerminationStrategy: PID 31558 died...
21:59:56.682797 run   osmo-mgw_10.42.42.3(pid=31558): Terminated {rc=15}
21:59:56.693839 ---      ParallelTerminationStrategy: PID 31559 died...
21:59:56.751363 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31559): Terminated: ok {rc=0}
21:59:56.762177 ---      ParallelTerminationStrategy: PID 31560 died...
21:59:56.816980 run   osmo-mgw_10.42.42.4(pid=31560): Terminated {rc=15}
21:59:56.826829 ---      ParallelTerminationStrategy: PID 31554 died...
21:59:56.880463 run  osmo-sgsn_10.42.42.7(pid=31554): Terminated: ok {rc=0}
21:59:56.890505 ---      ParallelTerminationStrategy: PID 31561 died...
21:59:56.944393 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31561): Terminated: ok {rc=0}
21:59:56.954265 ---      ParallelTerminationStrategy: PID 31562 died...
21:59:57.008067 run   osmo-bsc_10.42.42.9(pid=31562): Terminated: ok {rc=0}
21:59:57.096620 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
    setup_run_iperf3_test_parallel(1)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
    rem_host.recreate_remote_dir(self.remote_inst)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
    self.rm_remote_dir(remote_dir)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
    self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=31563): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31563)]