rm-remote-dir(pid=13538): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13538)]

Standard Output

----------------------------------------------
trial-2072 gprs:sysmo iperf3.py
----------------------------------------------
16:55:18.092130 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:18.103799 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
16:55:18.233783 tst                        iperf3.py: using LAC 41724
16:55:18.386259 tst                        iperf3.py: using RAC 159
16:55:18.506296 tst                        iperf3.py: using CellId 41724
16:55:18.631608 tst                        iperf3.py: using BVCI 41725
16:55:18.656527 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:18.667479 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:18.678413 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:18.689539 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:18.700586 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:18.711466 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:18.722732 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:18.733948 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
16:55:18.745114 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
16:55:19.022854 tst                      iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
16:55:19.148529 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
16:55:19.462423 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13495): Launched
16:55:19.721363 run iperf3-srv_10.42.42.10(pid=13496): Launched
16:55:19.905020 tst                      iperf3.py:8: start network...
16:55:20.030201 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
16:55:20.512291 run         create_hlr_db(pid=13497): Launched
16:55:20.690213 bus                          /gobi_6: Setting Powered False
16:55:21.752046 run         create_hlr_db(pid=13497): Terminated: ok {rc=0}
16:55:22.202457 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13499): Launched
16:55:22.533200 run   osmo-hlr_10.42.42.2(pid=13500): Launched
16:55:22.650772 run              osmo-stp_10.42.42.5: Starting osmo-stp
16:55:23.211321 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13501): Launched
16:55:23.589515 run   osmo-stp_10.42.42.5(pid=13502): Launched
16:55:23.706513 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
16:55:24.233526 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13503): Launched
16:55:24.340932 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
16:55:24.604761 run              patchelf(pid=13504): Launched
16:55:25.786169 run              patchelf(pid=13504): Terminated: ok {rc=0}
16:55:25.897189 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
16:55:26.168536 run osmo-gsm-tester_setcap_net_admin.sh(pid=13505): Launched
16:55:27.350021 run osmo-gsm-tester_setcap_net_admin.sh(pid=13505): Terminated: ok {rc=0}
16:55:27.663067 run  osmo-ggsn_10.42.42.6(pid=13508): Launched
16:55:27.879243 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
16:55:28.452077 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13528): Launched
16:55:28.758367 run  osmo-sgsn_10.42.42.7(pid=13529): Launched
16:55:28.868875 run              osmo-msc_10.42.42.8: Starting osmo-msc
16:55:29.423986 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13530): Launched
16:55:29.726137 run   osmo-msc_10.42.42.8(pid=13531): Launched
16:55:32.887011 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
16:55:33.411599 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13532): Launched
16:55:33.712954 run   osmo-mgw_10.42.42.3(pid=13533): Launched
16:55:33.820389 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
16:55:34.333924 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13534): Launched
16:55:34.634027 run   osmo-mgw_10.42.42.4(pid=13535): Launched
16:55:34.741507 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
16:55:35.373812 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13536): Launched
16:55:35.675545 run   osmo-bsc_10.42.42.9(pid=13537): Launched
16:55:41.933205 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
16:55:42.471955 run         rm-remote-dir(pid=13538): Launched
16:55:45.662550 run         rm-remote-dir(pid=13538): ERR: Terminated: ERROR {rc=255}  [trial-2072↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13538)]
16:55:45.795859 run         rm-remote-dir(pid=13538): stdout: 
| (launched: 2023-03-07_16:55:42.317281) 
16:55:45.929947 run         rm-remote-dir(pid=13538): stderr: 
| (launched: 2023-03-07_16:55:42.365389)
| ssh: connect to host 10.42.42.114 port 22: No route to host 
16:55:45.985329 tst                      iperf3.py:8: ERR: Error: rm-remote-dir(pid=13538): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13538)]  [trial-2072↪gprs:sysmo↪iperf3.py:8]
16:55:45.993106 tst                      iperf3.py:8: Test FAILED (27.9 sec)
16:55:46.027919 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13495): Terminating (SIGTERM)
16:55:46.067685 run iperf3-srv_10.42.42.10(pid=13496): Terminating (SIGTERM)
16:55:46.107624 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13499): Terminating (SIGTERM)
16:55:46.147419 run   osmo-hlr_10.42.42.2(pid=13500): Terminating (SIGTERM)
16:55:46.187810 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13501): Terminating (SIGTERM)
16:55:46.229351 run   osmo-stp_10.42.42.5(pid=13502): Terminating (SIGTERM)
16:55:46.271447 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13503): Terminating (SIGTERM)
16:55:46.311890 run  osmo-ggsn_10.42.42.6(pid=13508): Terminating (SIGTERM)
16:55:46.373542 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13528): Terminating (SIGTERM)
16:55:46.427214 run  osmo-sgsn_10.42.42.7(pid=13529): Terminating (SIGTERM)
16:55:46.468219 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13530): Terminating (SIGTERM)
16:55:46.516297 run   osmo-msc_10.42.42.8(pid=13531): Terminating (SIGTERM)
16:55:46.561652 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13532): Terminating (SIGTERM)
16:55:46.609392 run   osmo-mgw_10.42.42.3(pid=13533): Terminating (SIGTERM)
16:55:46.649145 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13534): Terminating (SIGTERM)
16:55:46.688556 run   osmo-mgw_10.42.42.4(pid=13535): Terminating (SIGTERM)
16:55:46.728696 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13536): Terminating (SIGTERM)
16:55:46.768071 run   osmo-bsc_10.42.42.9(pid=13537): Terminating (SIGTERM)
16:55:46.777213 ---      ParallelTerminationStrategy: PID 13495 died...
16:55:46.832085 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13495): Terminated: ok {rc=0}
16:55:46.842217 ---      ParallelTerminationStrategy: PID 13496 died...
16:55:46.898021 run iperf3-srv_10.42.42.10(pid=13496): Terminated {rc=256}
16:55:46.925463 ---      ParallelTerminationStrategy: PID 13499 died...
16:55:46.994099 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13499): Terminated: ok {rc=0}
16:55:47.004617 ---      ParallelTerminationStrategy: PID 13500 died...
16:55:47.059472 run   osmo-hlr_10.42.42.2(pid=13500): Terminated: ok {rc=0}
16:55:47.070289 ---      ParallelTerminationStrategy: PID 13501 died...
16:55:47.123964 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13501): Terminated: ok {rc=0}
16:55:47.133593 ---      ParallelTerminationStrategy: PID 13502 died...
16:55:47.186503 run   osmo-stp_10.42.42.5(pid=13502): Terminated: ok {rc=0}
16:55:47.195970 ---      ParallelTerminationStrategy: PID 13503 died...
16:55:47.248809 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13503): Terminated: ok {rc=0}
16:55:47.258435 ---      ParallelTerminationStrategy: PID 13508 died...
16:55:47.311259 run  osmo-ggsn_10.42.42.6(pid=13508): Terminated: ok {rc=0}
16:55:47.320985 ---      ParallelTerminationStrategy: PID 13528 died...
16:55:47.373820 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13528): Terminated: ok {rc=0}
16:55:47.383380 ---      ParallelTerminationStrategy: PID 13530 died...
16:55:47.436214 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13530): Terminated: ok {rc=0}
16:55:47.446455 ---      ParallelTerminationStrategy: PID 13531 died...
16:55:47.500788 run   osmo-msc_10.42.42.8(pid=13531): Terminated: ok {rc=0}
16:55:47.510834 ---      ParallelTerminationStrategy: PID 13532 died...
16:55:47.566087 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13532): Terminated: ok {rc=0}
16:55:47.576438 ---      ParallelTerminationStrategy: PID 13533 died...
16:55:47.631735 run   osmo-mgw_10.42.42.3(pid=13533): Terminated {rc=15}
16:55:47.642054 ---      ParallelTerminationStrategy: PID 13534 died...
16:55:47.697670 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13534): Terminated: ok {rc=0}
16:55:47.707895 ---      ParallelTerminationStrategy: PID 13535 died...
16:55:47.762966 run   osmo-mgw_10.42.42.4(pid=13535): Terminated {rc=15}
16:55:47.772693 ---      ParallelTerminationStrategy: PID 13529 died...
16:55:47.825826 run  osmo-sgsn_10.42.42.7(pid=13529): Terminated: ok {rc=0}
16:55:47.835445 ---      ParallelTerminationStrategy: PID 13536 died...
16:55:47.888363 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13536): Terminated: ok {rc=0}
16:55:47.898084 ---      ParallelTerminationStrategy: PID 13537 died...
16:55:47.951166 run   osmo-bsc_10.42.42.9(pid=13537): Terminated: ok {rc=0}
16:55:48.035081 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
    setup_run_iperf3_test_parallel(1)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
    rem_host.recreate_remote_dir(self.remote_inst)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
    self.rm_remote_dir(remote_dir)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
    self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=13538): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13538)]