rm-remote-dir(pid=9508): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2069↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9508)]

Standard Output

----------------------------------------------
trial-2069 gprs:sysmo iperf3.py
----------------------------------------------
18:29:01.328078 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.339899 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
18:29:01.478275 tst                        iperf3.py: using LAC 41703
18:29:01.599605 tst                        iperf3.py: using RAC 138
18:29:01.721723 tst                        iperf3.py: using CellId 41703
18:29:01.841755 tst                        iperf3.py: using BVCI 41704
18:29:01.864845 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.875432 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.886001 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.896476 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.906794 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.917102 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.927610 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.938047 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.949312 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
18:29:02.241929 tst                      iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
18:29:02.377366 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
18:29:02.712109 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9466): Launched
18:29:02.989703 run iperf3-srv_10.42.42.10(pid=9467): Launched
18:29:03.187870 tst                      iperf3.py:8: start network...
18:29:03.323005 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
18:29:03.835008 run          create_hlr_db(pid=9468): Launched
18:29:04.022935 bus                          /gobi_6: Setting Powered False
18:29:05.043205 run          create_hlr_db(pid=9468): Terminated: ok {rc=0}
18:29:05.448615 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9470): Launched
18:29:05.751549 run    osmo-hlr_10.42.42.2(pid=9471): Launched
18:29:05.915369 run              osmo-stp_10.42.42.5: Starting osmo-stp
18:29:06.469444 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9472): Launched
18:29:06.894837 run    osmo-stp_10.42.42.5(pid=9473): Launched
18:29:07.046214 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:29:07.592450 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9474): Launched
18:29:07.699788 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:29:07.962859 run               patchelf(pid=9475): Launched
18:29:09.164291 run               patchelf(pid=9475): Terminated: ok {rc=0}
18:29:09.285946 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
18:29:09.554877 run osmo-gsm-tester_setcap_net_admin.sh(pid=9476): Launched
18:29:10.735170 run osmo-gsm-tester_setcap_net_admin.sh(pid=9476): Terminated: ok {rc=0}
18:29:11.040409 run   osmo-ggsn_10.42.42.6(pid=9479): Launched
18:29:11.304791 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
18:29:11.852518 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9498): Launched
18:29:12.154883 run   osmo-sgsn_10.42.42.7(pid=9499): Launched
18:29:12.265485 run              osmo-msc_10.42.42.8: Starting osmo-msc
18:29:12.818010 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9500): Launched
18:29:13.120499 run    osmo-msc_10.42.42.8(pid=9501): Launched
18:29:16.282026 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
18:29:16.806950 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9502): Launched
18:29:17.109013 run    osmo-mgw_10.42.42.3(pid=9503): Launched
18:29:17.216499 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
18:29:17.729953 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9504): Launched
18:29:18.031519 run    osmo-mgw_10.42.42.4(pid=9505): Launched
18:29:18.139235 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
18:29:18.773801 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9506): Launched
18:29:19.075670 run    osmo-bsc_10.42.42.9(pid=9507): Launched
18:29:25.333155 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
18:29:25.872269 run          rm-remote-dir(pid=9508): Launched
18:29:29.063418 run          rm-remote-dir(pid=9508): ERR: Terminated: ERROR {rc=255}  [trial-2069↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9508)]
18:29:29.196636 run          rm-remote-dir(pid=9508): stdout: 
| (launched: 2022-12-20_18:29:25.717831) 
18:29:29.330055 run          rm-remote-dir(pid=9508): stderr: 
| (launched: 2022-12-20_18:29:25.765791)
| ssh: connect to host 10.42.42.114 port 22: No route to host 
18:29:29.385115 tst                      iperf3.py:8: ERR: Error: rm-remote-dir(pid=9508): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2069↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9508)]  [trial-2069↪gprs:sysmo↪iperf3.py:8]
18:29:29.392699 tst                      iperf3.py:8: Test FAILED (28.1 sec)
18:29:29.427755 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9466): Terminating (SIGTERM)
18:29:29.468419 run iperf3-srv_10.42.42.10(pid=9467): Terminating (SIGTERM)
18:29:29.508618 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9470): Terminating (SIGTERM)
18:29:29.548553 run    osmo-hlr_10.42.42.2(pid=9471): Terminating (SIGTERM)
18:29:29.589723 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9472): Terminating (SIGTERM)
18:29:29.631775 run    osmo-stp_10.42.42.5(pid=9473): Terminating (SIGTERM)
18:29:29.674567 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9474): Terminating (SIGTERM)
18:29:29.717921 run   osmo-ggsn_10.42.42.6(pid=9479): Terminating (SIGTERM)
18:29:29.765567 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9498): Terminating (SIGTERM)
18:29:29.818607 run   osmo-sgsn_10.42.42.7(pid=9499): Terminating (SIGTERM)
18:29:29.866823 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9500): Terminating (SIGTERM)
18:29:29.915230 run    osmo-msc_10.42.42.8(pid=9501): Terminating (SIGTERM)
18:29:29.961149 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9502): Terminating (SIGTERM)
18:29:30.009367 run    osmo-mgw_10.42.42.3(pid=9503): Terminating (SIGTERM)
18:29:30.061927 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9504): Terminating (SIGTERM)
18:29:30.105077 run    osmo-mgw_10.42.42.4(pid=9505): Terminating (SIGTERM)
18:29:30.150381 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9506): Terminating (SIGTERM)
18:29:30.195001 run    osmo-bsc_10.42.42.9(pid=9507): Terminating (SIGTERM)
18:29:30.205964 ---      ParallelTerminationStrategy: PID 9466 died...
18:29:30.280370 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9466): Terminated: ok {rc=0}
18:29:30.291423 ---      ParallelTerminationStrategy: PID 9467 died...
18:29:30.349424 run iperf3-srv_10.42.42.10(pid=9467): Terminated {rc=256}
18:29:30.361037 ---      ParallelTerminationStrategy: PID 9470 died...
18:29:30.420347 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9470): Terminated: ok {rc=0}
18:29:30.432961 ---      ParallelTerminationStrategy: PID 9471 died...
18:29:30.513500 run    osmo-hlr_10.42.42.2(pid=9471): Terminated: ok {rc=0}
18:29:30.525216 ---      ParallelTerminationStrategy: PID 9472 died...
18:29:30.586674 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9472): Terminated: ok {rc=0}
18:29:30.597857 ---      ParallelTerminationStrategy: PID 9473 died...
18:29:30.651578 run    osmo-stp_10.42.42.5(pid=9473): Terminated: ok {rc=0}
18:29:30.661552 ---      ParallelTerminationStrategy: PID 9474 died...
18:29:30.715223 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9474): Terminated: ok {rc=0}
18:29:30.725288 ---      ParallelTerminationStrategy: PID 9479 died...
18:29:30.779014 run   osmo-ggsn_10.42.42.6(pid=9479): Terminated: ok {rc=0}
18:29:30.788990 ---      ParallelTerminationStrategy: PID 9498 died...
18:29:30.842842 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9498): Terminated: ok {rc=0}
18:29:30.853253 ---      ParallelTerminationStrategy: PID 9500 died...
18:29:30.908872 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9500): Terminated: ok {rc=0}
18:29:30.919552 ---      ParallelTerminationStrategy: PID 9501 died...
18:29:30.976112 run    osmo-msc_10.42.42.8(pid=9501): Terminated: ok {rc=0}
18:29:30.986706 ---      ParallelTerminationStrategy: PID 9502 died...
18:29:31.042956 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9502): Terminated: ok {rc=0}
18:29:31.053730 ---      ParallelTerminationStrategy: PID 9503 died...
18:29:31.110106 run    osmo-mgw_10.42.42.3(pid=9503): Terminated {rc=15}
18:29:31.120611 ---      ParallelTerminationStrategy: PID 9504 died...
18:29:31.175033 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9504): Terminated: ok {rc=0}
18:29:31.185105 ---      ParallelTerminationStrategy: PID 9499 died...
18:29:31.239152 run   osmo-sgsn_10.42.42.7(pid=9499): Terminated: ok {rc=0}
18:29:31.249090 ---      ParallelTerminationStrategy: PID 9505 died...
18:29:31.302845 run    osmo-mgw_10.42.42.4(pid=9505): Terminated {rc=15}
18:29:31.313200 ---      ParallelTerminationStrategy: PID 9506 died...
18:29:31.366956 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9506): Terminated: ok {rc=0}
18:29:31.377088 ---      ParallelTerminationStrategy: PID 9507 died...
18:29:31.430560 run    osmo-bsc_10.42.42.9(pid=9507): Terminated: ok {rc=0}
18:29:31.520823 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
    setup_run_iperf3_test_parallel(1)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
    rem_host.recreate_remote_dir(self.remote_inst)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
    self.rm_remote_dir(remote_dir)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
    self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=9508): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2069↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9508)]