rm-remote-dir(pid=9791): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9791)]

Standard Output

----------------------------------------------
trial-2070 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
15:42:28.104920 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:28.115669 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
15:42:28.242706 tst         cs_paging_gprs_active.py: using LAC 41709
15:42:28.374016 tst         cs_paging_gprs_active.py: using RAC 144
15:42:28.503187 tst         cs_paging_gprs_active.py: using CellId 41709
15:42:28.626929 tst         cs_paging_gprs_active.py: using BVCI 41710
15:42:28.651618 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:28.662556 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:28.673387 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:28.685162 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:28.695757 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:28.706691 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:28.717822 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:28.728746 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:28.739923 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
15:42:28.857642 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
15:42:29.133415 tst      cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
15:42:29.259168 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
15:42:29.568601 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9742): Launched
15:42:29.825569 run iperf3-srv_10.42.42.10(pid=9743): Launched
15:42:30.021532 tst      cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
15:42:30.156438 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
15:42:30.488575 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9744): Launched
15:42:30.764976 run iperf3-srv_10.42.42.10(pid=9745): Launched
15:42:30.962471 tst      cs_paging_gprs_active.py:38: start network...
15:42:31.097057 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
15:42:31.603844 run          create_hlr_db(pid=9746): Launched
15:42:31.790247 bus                        /sierra_2: Setting Powered False
15:42:32.811542 run          create_hlr_db(pid=9746): Terminated: ok {rc=0}
15:42:33.213247 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9748): Launched
15:42:33.514054 run    osmo-hlr_10.42.42.2(pid=9749): Launched
15:42:33.623368 run              osmo-stp_10.42.42.5: Starting osmo-stp
15:42:34.142138 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9750): Launched
15:42:34.442215 run    osmo-stp_10.42.42.5(pid=9751): Launched
15:42:34.549743 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:42:36.059100 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9752): Launched
15:42:36.171908 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:42:36.470633 run               patchelf(pid=9753): Launched
15:42:37.650886 run               patchelf(pid=9753): Terminated: ok {rc=0}
15:42:37.760889 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
15:42:38.031924 run osmo-gsm-tester_setcap_net_admin.sh(pid=9754): Launched
15:42:39.217490 run osmo-gsm-tester_setcap_net_admin.sh(pid=9754): Terminated: ok {rc=0}
15:42:39.521844 run   osmo-ggsn_10.42.42.6(pid=9757): Launched
15:42:39.786371 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
15:42:44.485835 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9781): Launched
15:42:44.786863 run   osmo-sgsn_10.42.42.7(pid=9782): Launched
15:42:44.897466 run              osmo-msc_10.42.42.8: Starting osmo-msc
15:42:45.449033 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9783): Launched
15:42:45.749360 run    osmo-msc_10.42.42.8(pid=9784): Launched
15:42:48.909681 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
15:42:49.450309 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9785): Launched
15:42:49.754257 run    osmo-mgw_10.42.42.3(pid=9786): Launched
15:42:49.881295 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
15:42:50.394484 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9787): Launched
15:42:50.696130 run    osmo-mgw_10.42.42.4(pid=9788): Launched
15:42:50.823268 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
15:42:51.549004 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9789): Launched
15:42:51.976872 run    osmo-bsc_10.42.42.9(pid=9790): Launched
15:42:58.286774 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
15:42:58.861279 run          rm-remote-dir(pid=9791): Launched
15:43:02.064880 run          rm-remote-dir(pid=9791): ERR: Terminated: ERROR {rc=255}  [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9791)]
15:43:02.207258 run          rm-remote-dir(pid=9791): stdout: 
| (launched: 2022-12-23_15:42:58.695771) 
15:43:02.352796 run          rm-remote-dir(pid=9791): stderr: 
| (launched: 2022-12-23_15:42:58.747034)
| ssh: connect to host 10.42.42.114 port 22: No route to host 
15:43:02.412320 tst      cs_paging_gprs_active.py:38: ERR: Error: rm-remote-dir(pid=9791): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9791)]  [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38]
15:43:02.420934 tst      cs_paging_gprs_active.py:38: Test FAILED (34.4 sec)
15:43:02.458426 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9742): Terminating (SIGTERM)
15:43:02.500361 run iperf3-srv_10.42.42.10(pid=9743): Terminating (SIGTERM)
15:43:02.542441 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9744): Terminating (SIGTERM)
15:43:02.584565 run iperf3-srv_10.42.42.10(pid=9745): Terminating (SIGTERM)
15:43:02.626573 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9748): Terminating (SIGTERM)
15:43:02.668427 run    osmo-hlr_10.42.42.2(pid=9749): Terminating (SIGTERM)
15:43:02.710679 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9750): Terminating (SIGTERM)
15:43:02.753955 run    osmo-stp_10.42.42.5(pid=9751): Terminating (SIGTERM)
15:43:02.830210 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9752): Terminating (SIGTERM)
15:43:02.890447 run   osmo-ggsn_10.42.42.6(pid=9757): Terminating (SIGTERM)
15:43:02.938612 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9781): Terminating (SIGTERM)
15:43:02.983410 run   osmo-sgsn_10.42.42.7(pid=9782): Terminating (SIGTERM)
15:43:03.026020 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9783): Terminating (SIGTERM)
15:43:03.067163 run    osmo-msc_10.42.42.8(pid=9784): Terminating (SIGTERM)
15:43:03.109152 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9785): Terminating (SIGTERM)
15:43:03.150919 run    osmo-mgw_10.42.42.3(pid=9786): Terminating (SIGTERM)
15:43:03.191071 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9787): Terminating (SIGTERM)
15:43:03.230262 run    osmo-mgw_10.42.42.4(pid=9788): Terminating (SIGTERM)
15:43:03.270126 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9789): Terminating (SIGTERM)
15:43:03.310082 run    osmo-bsc_10.42.42.9(pid=9790): Terminating (SIGTERM)
15:43:03.319144 ---      ParallelTerminationStrategy: PID 9742 died...
15:43:03.372103 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9742): Terminated: ok {rc=0}
15:43:03.381575 ---      ParallelTerminationStrategy: PID 9743 died...
15:43:03.433826 run iperf3-srv_10.42.42.10(pid=9743): Terminated {rc=256}
15:43:03.443171 ---      ParallelTerminationStrategy: PID 9744 died...
15:43:03.496055 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9744): Terminated: ok {rc=0}
15:43:03.505484 ---      ParallelTerminationStrategy: PID 9745 died...
15:43:03.557790 run iperf3-srv_10.42.42.10(pid=9745): Terminated {rc=256}
15:43:03.567161 ---      ParallelTerminationStrategy: PID 9748 died...
15:43:03.619969 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9748): Terminated: ok {rc=0}
15:43:03.629429 ---      ParallelTerminationStrategy: PID 9749 died...
15:43:03.681749 run    osmo-hlr_10.42.42.2(pid=9749): Terminated: ok {rc=0}
15:43:03.691086 ---      ParallelTerminationStrategy: PID 9750 died...
15:43:03.743541 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9750): Terminated: ok {rc=0}
15:43:03.752987 ---      ParallelTerminationStrategy: PID 9751 died...
15:43:03.805236 run    osmo-stp_10.42.42.5(pid=9751): Terminated: ok {rc=0}
15:43:03.814622 ---      ParallelTerminationStrategy: PID 9752 died...
15:43:03.869382 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9752): Terminated: ok {rc=0}
15:43:03.879963 ---      ParallelTerminationStrategy: PID 9757 died...
15:43:03.934633 run   osmo-ggsn_10.42.42.6(pid=9757): Terminated: ok {rc=0}
15:43:03.944983 ---      ParallelTerminationStrategy: PID 9781 died...
15:43:03.998050 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9781): Terminated: ok {rc=0}
15:43:04.007700 ---      ParallelTerminationStrategy: PID 9783 died...
15:43:04.062267 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9783): Terminated: ok {rc=0}
15:43:04.073009 ---      ParallelTerminationStrategy: PID 9784 died...
15:43:04.155559 run    osmo-msc_10.42.42.8(pid=9784): Terminated: ok {rc=0}
15:43:04.182590 ---      ParallelTerminationStrategy: PID 9785 died...
15:43:04.304368 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9785): Terminated: ok {rc=0}
15:43:04.316309 ---      ParallelTerminationStrategy: PID 9786 died...
15:43:04.372319 run    osmo-mgw_10.42.42.3(pid=9786): Terminated {rc=15}
15:43:04.382675 ---      ParallelTerminationStrategy: PID 9787 died...
15:43:04.437984 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9787): Terminated: ok {rc=0}
15:43:04.448070 ---      ParallelTerminationStrategy: PID 9788 died...
15:43:04.500934 run    osmo-mgw_10.42.42.4(pid=9788): Terminated {rc=15}
15:43:04.510483 ---      ParallelTerminationStrategy: PID 9782 died...
15:43:04.562897 run   osmo-sgsn_10.42.42.7(pid=9782): Terminated: ok {rc=0}
15:43:04.572471 ---      ParallelTerminationStrategy: PID 9789 died...
15:43:04.625018 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9789): Terminated: ok {rc=0}
15:43:04.634545 ---      ParallelTerminationStrategy: PID 9790 died...
15:43:04.686923 run    osmo-bsc_10.42.42.9(pid=9790): Terminated: ok {rc=0}
15:43:04.769775 bus                          /gobi_0: Setting Powered False
15:43:05.889065 bus                          /gobi_0: Setting Powered False
15:43:07.029953 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
    setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
    rem_host.recreate_remote_dir(self.remote_inst)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
    self.rm_remote_dir(remote_dir)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
    self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=9791): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9791)]