rm-remote-dir(pid=31507): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31507)]

Standard Output

----------------------------------------------
trial-2071 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
21:58:45.595324 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:58:45.606504 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
21:58:45.742867 tst         cs_paging_gprs_active.py: using LAC 41716
21:58:45.954066 tst         cs_paging_gprs_active.py: using RAC 151
21:58:46.074657 tst         cs_paging_gprs_active.py: using CellId 41716
21:58:46.194317 tst         cs_paging_gprs_active.py: using BVCI 41717
21:58:46.217870 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:58:46.228652 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:58:46.239259 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:58:46.250192 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:58:46.260949 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:58:46.271664 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:58:46.282673 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:58:46.293554 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
21:58:46.304711 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
21:58:46.427159 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
21:58:46.707444 tst      cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
21:58:46.835506 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:47.153247 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31461): Launched
21:58:47.419627 run iperf3-srv_10.42.42.10(pid=31462): Launched
21:58:47.606262 tst      cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
21:58:47.733651 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:48.050248 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31463): Launched
21:58:48.313866 run iperf3-srv_10.42.42.10(pid=31464): Launched
21:58:48.500494 tst      cs_paging_gprs_active.py:38: start network...
21:58:48.627864 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:49.108280 run         create_hlr_db(pid=31465): Launched
21:58:49.285322 bus                        /sierra_2: Setting Powered False
21:58:50.305549 run         create_hlr_db(pid=31465): Terminated: ok {rc=0}
21:58:50.712977 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31467): Launched
21:58:51.160948 run   osmo-hlr_10.42.42.2(pid=31468): Launched
21:58:51.359295 run              osmo-stp_10.42.42.5: Starting osmo-stp
21:58:51.938524 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31469): Launched
21:58:52.276016 run   osmo-stp_10.42.42.5(pid=31470): Launched
21:58:52.389307 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:58:53.870859 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31471): Launched
21:58:53.979286 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:58:54.247505 run              patchelf(pid=31472): Launched
21:58:55.430382 run              patchelf(pid=31472): Terminated: ok {rc=0}
21:58:55.542608 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
21:58:55.831077 run osmo-gsm-tester_setcap_net_admin.sh(pid=31473): Launched
21:58:57.027355 run osmo-gsm-tester_setcap_net_admin.sh(pid=31473): Terminated: ok {rc=0}
21:58:57.358411 run  osmo-ggsn_10.42.42.6(pid=31476): Launched
21:58:57.593148 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
21:59:02.485596 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31497): Launched
21:59:02.792607 run  osmo-sgsn_10.42.42.7(pid=31498): Launched
21:59:02.907373 run              osmo-msc_10.42.42.8: Starting osmo-msc
21:59:03.469605 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31499): Launched
21:59:03.777268 run   osmo-msc_10.42.42.8(pid=31500): Launched
21:59:06.967481 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
21:59:07.518210 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31501): Launched
21:59:07.826571 run   osmo-mgw_10.42.42.3(pid=31502): Launched
21:59:07.938563 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
21:59:08.463106 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31503): Launched
21:59:08.769931 run   osmo-mgw_10.42.42.4(pid=31504): Launched
21:59:08.895961 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
21:59:09.587591 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31505): Launched
21:59:09.916084 run   osmo-bsc_10.42.42.9(pid=31506): Launched
21:59:16.196893 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
21:59:16.779525 run         rm-remote-dir(pid=31507): Launched
21:59:19.984803 run         rm-remote-dir(pid=31507): ERR: Terminated: ERROR {rc=255}  [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31507)]
21:59:20.128992 run         rm-remote-dir(pid=31507): stdout: 
| (launched: 2023-01-03_21:59:16.612399) 
21:59:20.276520 run         rm-remote-dir(pid=31507): stderr: 
| (launched: 2023-01-03_21:59:16.664320)
| ssh: connect to host 10.42.42.114 port 22: No route to host 
21:59:20.336593 tst      cs_paging_gprs_active.py:38: ERR: Error: rm-remote-dir(pid=31507): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31507)]  [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38]
21:59:20.345255 tst      cs_paging_gprs_active.py:38: Test FAILED (34.8 sec)
21:59:20.382439 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31461): Terminating (SIGTERM)
21:59:20.425066 run iperf3-srv_10.42.42.10(pid=31462): Terminating (SIGTERM)
21:59:20.467576 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31463): Terminating (SIGTERM)
21:59:20.509940 run iperf3-srv_10.42.42.10(pid=31464): Terminating (SIGTERM)
21:59:20.552443 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31467): Terminating (SIGTERM)
21:59:20.594585 run   osmo-hlr_10.42.42.2(pid=31468): Terminating (SIGTERM)
21:59:20.638070 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31469): Terminating (SIGTERM)
21:59:20.683904 run   osmo-stp_10.42.42.5(pid=31470): Terminating (SIGTERM)
21:59:20.797299 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31471): Terminating (SIGTERM)
21:59:20.882783 run  osmo-ggsn_10.42.42.6(pid=31476): Terminating (SIGTERM)
21:59:20.979009 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31497): Terminating (SIGTERM)
21:59:21.080542 run  osmo-sgsn_10.42.42.7(pid=31498): Terminating (SIGTERM)
21:59:21.146060 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31499): Terminating (SIGTERM)
21:59:21.239544 run   osmo-msc_10.42.42.8(pid=31500): Terminating (SIGTERM)
21:59:21.312819 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31501): Terminating (SIGTERM)
21:59:21.358874 run   osmo-mgw_10.42.42.3(pid=31502): Terminating (SIGTERM)
21:59:21.404331 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31503): Terminating (SIGTERM)
21:59:21.450717 run   osmo-mgw_10.42.42.4(pid=31504): Terminating (SIGTERM)
21:59:21.495026 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31505): Terminating (SIGTERM)
21:59:21.539213 run   osmo-bsc_10.42.42.9(pid=31506): Terminating (SIGTERM)
21:59:21.549346 ---      ParallelTerminationStrategy: PID 31461 died...
21:59:21.607769 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31461): Terminated: ok {rc=0}
21:59:21.618505 ---      ParallelTerminationStrategy: PID 31462 died...
21:59:21.676624 run iperf3-srv_10.42.42.10(pid=31462): Terminated {rc=256}
21:59:21.687321 ---      ParallelTerminationStrategy: PID 31463 died...
21:59:21.745678 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31463): Terminated: ok {rc=0}
21:59:21.756471 ---      ParallelTerminationStrategy: PID 31464 died...
21:59:21.814421 run iperf3-srv_10.42.42.10(pid=31464): Terminated {rc=256}
21:59:21.825152 ---      ParallelTerminationStrategy: PID 31467 died...
21:59:21.884446 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31467): Terminated: ok {rc=0}
21:59:21.895343 ---      ParallelTerminationStrategy: PID 31468 died...
21:59:21.953880 run   osmo-hlr_10.42.42.2(pid=31468): Terminated: ok {rc=0}
21:59:21.964701 ---      ParallelTerminationStrategy: PID 31469 died...
21:59:22.118935 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31469): Terminated: ok {rc=0}
21:59:22.160943 ---      ParallelTerminationStrategy: PID 31470 died...
21:59:22.360199 run   osmo-stp_10.42.42.5(pid=31470): Terminated: ok {rc=0}
21:59:22.370652 ---      ParallelTerminationStrategy: PID 31471 died...
21:59:22.449727 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31471): Terminated: ok {rc=0}
21:59:22.468688 ---      ParallelTerminationStrategy: PID 31476 died...
21:59:22.539973 run  osmo-ggsn_10.42.42.6(pid=31476): Terminated: ok {rc=0}
21:59:22.550722 ---      ParallelTerminationStrategy: PID 31497 died...
21:59:22.610347 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31497): Terminated: ok {rc=0}
21:59:22.621167 ---      ParallelTerminationStrategy: PID 31499 died...
21:59:22.680358 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31499): Terminated: ok {rc=0}
21:59:22.691484 ---      ParallelTerminationStrategy: PID 31500 died...
21:59:22.748027 run   osmo-msc_10.42.42.8(pid=31500): Terminated: ok {rc=0}
21:59:22.758214 ---      ParallelTerminationStrategy: PID 31498 died...
21:59:22.814520 run  osmo-sgsn_10.42.42.7(pid=31498): Terminated: ok {rc=0}
21:59:22.825136 ---      ParallelTerminationStrategy: PID 31501 died...
21:59:22.881818 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31501): Terminated: ok {rc=0}
21:59:22.892032 ---      ParallelTerminationStrategy: PID 31502 died...
21:59:22.948299 run   osmo-mgw_10.42.42.3(pid=31502): Terminated {rc=15}
21:59:22.958473 ---      ParallelTerminationStrategy: PID 31503 died...
21:59:23.014932 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31503): Terminated: ok {rc=0}
21:59:23.025577 ---      ParallelTerminationStrategy: PID 31504 died...
21:59:23.082097 run   osmo-mgw_10.42.42.4(pid=31504): Terminated {rc=15}
21:59:23.092399 ---      ParallelTerminationStrategy: PID 31505 died...
21:59:23.148753 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31505): Terminated: ok {rc=0}
21:59:23.159032 ---      ParallelTerminationStrategy: PID 31506 died...
21:59:23.215415 run   osmo-bsc_10.42.42.9(pid=31506): Terminated: ok {rc=0}
21:59:23.303956 bus                          /gobi_0: Setting Powered False
21:59:24.412482 bus                          /gobi_0: Setting Powered False
21:59:25.521964 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
    setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
    rem_host.recreate_remote_dir(self.remote_inst)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
    self.rm_remote_dir(remote_dir)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
    self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=31507): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31507)]