229.0false70010nitb_sms:sysmo6.0osmo-nitb_10.42.42.2(pid=13402): Process ended prematurely: osmo-nitb_10.42.42.2(pid=13402) [trial-2072↪nitb_sms:sysmo↪mo_mt_sms.py:11↪mo_mt_sms.py↪osmo-nitb_10.42.42.2↪osmo-nitb_10.42.42.2(pid=13402)]2063mo_mt_sms.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/nitb_sms/mo_mt_sms.py", line 11, in <module> nitb.start() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/nitb_osmo.py", line 67, in start self.ctrl.connect() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/osmo_ctrl.py", line 92, in connect MainLoop.wait(self.try_connect, timestep=3, timeout=timeout) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise self.poll(may_block=True) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll self.deferred_handling.handle_queue() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue handler(*args, **kwargs) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll raise log_module.Error('Process ended prematurely: %s' % proc.name()) osmo_gsm_tester.core.log.Error: osmo-nitb_10.42.42.2(pid=13402): Process ended prematurely: osmo-nitb_10.42.42.2(pid=13402) [trial-2072↪nitb_sms:sysmo↪mo_mt_sms.py:11↪mo_mt_sms.py↪osmo-nitb_10.42.42.2↪osmo-nitb_10.42.42.2(pid=13402)] ---------------------------------------------- trial-2072 nitb_sms:sysmo mo_mt_sms.py ---------------------------------------------- 16:53:47.069702 tst nitb_sms:sysmo: Using 1 x ip_address (candidates: 1) 16:53:47.080767 tst nitb_sms:sysmo: Using 1 x bts (candidates: 1) 16:53:47.663473 tst mo_mt_sms.py: using LAC 41721 16:53:47.776268 tst mo_mt_sms.py: using RAC 156 16:53:47.903089 tst mo_mt_sms.py: using CellId 41721 16:53:48.020138 tst mo_mt_sms.py: using BVCI 41722 16:53:48.041957 tst nitb_sms:sysmo: Using 1 x modem (candidates: 2) 16:53:48.222308 tst nitb_sms:sysmo: Using 1 x modem (candidates: 2) 16:53:48.509041 tst mo_mt_sms.py:9: start nitb and bts... 16:53:48.640515 run osmo-nitb_10.42.42.2: Starting osmo-nitb 16:53:51.062898 run pcap-recorder_any(filters='host 10.42.42.2 and port not 22')(pid=13401): Launched 16:53:51.416922 run osmo-nitb_10.42.42.2(pid=13402): Launched 16:53:51.589920 run osmo-nitb_10.42.42.2(pid=13402): ERR: Terminated: ERROR {rc=2} [trial-2072↪nitb_sms:sysmo↪mo_mt_sms.py:11↪mo_mt_sms.py↪osmo-nitb_10.42.42.2↪osmo-nitb_10.42.42.2(pid=13402)] 16:53:51.716369 run osmo-nitb_10.42.42.2(pid=13402): stdout: | (launched: 2023-03-07_16:53:51.248928) 16:53:51.842565 run osmo-nitb_10.42.42.2(pid=13402): stderr: | (launched: 2023-03-07_16:53:51.293502) | ====================================================================== | ERROR: You should not run osmo-nitb. It is ancient, unmaintained and | obsolete for several years. If you really want to ignore years of hard work on | the new post-NITB stack (osmo-bsc, osmo-mgw, osmo-msc, osmo-hlr, osmo-stp), | you may ues the '--yes-i-really-want-to-run-prehistoric-software' command line | argument. DON'T EVER BOTHER THE DEVELOPERS IF YOU ENCOUNTER PROBLEMS WITH IT!!! | Use https://osmocom.org/projects/cellular-infrastructure/wiki/Osmocom_Network_In_The_Box | ====================================================================== 16:53:51.964435 run osmo-nitb_10.42.42.2(pid=13402): stdout: | (launched: 2023-03-07_16:53:51.248928) 16:53:52.085913 run osmo-nitb_10.42.42.2(pid=13402): stderr: | (launched: 2023-03-07_16:53:51.293502) | ====================================================================== | ERROR: You should not run osmo-nitb. It is ancient, unmaintained and | obsolete for several years. If you really want to ignore years of hard work on | the new post-NITB stack (osmo-bsc, osmo-mgw, osmo-msc, osmo-hlr, osmo-stp), | you may ues the '--yes-i-really-want-to-run-prehistoric-software' command line | argument. DON'T EVER BOTHER THE DEVELOPERS IF YOU ENCOUNTER PROBLEMS WITH IT!!! | Use https://osmocom.org/projects/cellular-infrastructure/wiki/Osmocom_Network_In_The_Box | ====================================================================== 16:53:52.138784 tst mo_mt_sms.py:11: ERR: Error: osmo-nitb_10.42.42.2(pid=13402): Process ended prematurely: osmo-nitb_10.42.42.2(pid=13402) [trial-2072↪nitb_sms:sysmo↪mo_mt_sms.py:11↪mo_mt_sms.py↪osmo-nitb_10.42.42.2↪osmo-nitb_10.42.42.2(pid=13402)] [trial-2072↪nitb_sms:sysmo↪mo_mt_sms.py:11] 16:53:52.147179 tst mo_mt_sms.py:11: Test FAILED (5.1 sec) 16:53:52.182456 run pcap-recorder_any(filters='host 10.42.42.2 and port not 22')(pid=13401): Terminating (SIGTERM) 16:53:52.360396 --- ParallelTerminationStrategy: PID 13401 died... 16:53:52.414926 run pcap-recorder_any(filters='host 10.42.42.2 and port not 22')(pid=13401): Terminated: ok {rc=0} 16:53:52.499290 bus /sierra_3: Setting Powered False 16:53:53.609757 bus /sierra_3: Setting Powered False 16:53:54.721582 bus /sierra_2: Setting Powered False 10.00nitb_sms_sysmo2023-03-07T16:53:464sms:sysmo36.0rm-remote-dir(pid=13420): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪sms:sysmo↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13420)]2069mo_mt_sms.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/sms/mo_mt_sms.py", line 23, in <module> bts.start() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start rem_host.recreate_remote_dir(self.remote_inst) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir self.rm_remote_dir(remote_dir) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str)) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync proc.launch_sync() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync raise self.RunError('launch_sync()') osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=13420): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪sms:sysmo↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13420)] ---------------------------------------------- trial-2072 sms:sysmo mo_mt_sms.py ---------------------------------------------- 16:53:56.564387 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 16:53:56.574761 tst sms:sysmo: Using 1 x bts (candidates: 1) 16:53:56.692461 tst mo_mt_sms.py: using LAC 41722 16:53:56.806260 tst mo_mt_sms.py: using RAC 157 16:53:56.920051 tst mo_mt_sms.py: using CellId 41722 16:53:57.035543 tst mo_mt_sms.py: using BVCI 41723 16:53:57.057846 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 16:53:57.068251 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 16:53:57.078567 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 16:53:57.088858 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 16:53:57.099022 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 16:53:57.109268 tst sms:sysmo: Using 1 x modem (candidates: 2) 16:53:57.224962 tst sms:sysmo: Using 1 x modem (candidates: 2) 16:53:57.420928 run osmo-hlr_10.42.42.2: Starting osmo-hlr 16:53:59.073270 run create_hlr_db(pid=13406): Launched 16:53:59.242882 bus /sierra_2: Setting Powered False 16:54:00.264316 run create_hlr_db(pid=13406): Terminated: ok {rc=0} 16:54:00.683917 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13408): Launched 16:54:00.998096 run osmo-hlr_10.42.42.2(pid=13409): Launched 16:54:01.112787 run osmo-stp_10.42.42.5: Starting osmo-stp 16:54:03.726868 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13410): Launched 16:54:04.039792 run osmo-stp_10.42.42.5(pid=13411): Launched 16:54:04.152946 run osmo-msc_10.42.42.6: Starting osmo-msc 16:54:09.588905 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13412): Launched 16:54:09.903980 run osmo-msc_10.42.42.6(pid=13413): Launched 16:54:13.073015 run osmo-mgw_10.42.42.3: Starting osmo-mgw 16:54:14.858107 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13414): Launched 16:54:15.175557 run osmo-mgw_10.42.42.3(pid=13415): Launched 16:54:15.290779 run osmo-mgw_10.42.42.4: Starting osmo-mgw 16:54:15.829331 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13416): Launched 16:54:16.191934 run osmo-mgw_10.42.42.4(pid=13417): Launched 16:54:16.335139 run osmo-bsc_10.42.42.7: Starting osmo-bsc 16:54:20.805487 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13418): Launched 16:54:21.236654 run osmo-bsc_10.42.42.7(pid=13419): Launched 16:54:27.502511 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7 16:54:28.420439 run rm-remote-dir(pid=13420): Launched 16:54:31.607409 run rm-remote-dir(pid=13420): ERR: Terminated: ERROR {rc=255} [trial-2072↪sms:sysmo↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13420)] 16:54:31.737782 run rm-remote-dir(pid=13420): stdout: | (launched: 2023-03-07_16:54:28.269292) 16:54:31.868016 run rm-remote-dir(pid=13420): stderr: | (launched: 2023-03-07_16:54:28.316438) | ssh: connect to host 10.42.42.114 port 22: No route to host 16:54:31.921774 tst mo_mt_sms.py:23: ERR: Error: rm-remote-dir(pid=13420): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪sms:sysmo↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13420)] [trial-2072↪sms:sysmo↪mo_mt_sms.py:23] 16:54:31.928993 tst mo_mt_sms.py:23: Test FAILED (35.4 sec) 16:54:31.964084 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13408): Terminating (SIGTERM) 16:54:32.004420 run osmo-hlr_10.42.42.2(pid=13409): Terminating (SIGTERM) 16:54:32.045420 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13410): Terminating (SIGTERM) 16:54:32.087405 run osmo-stp_10.42.42.5(pid=13411): Terminating (SIGTERM) 16:54:32.129768 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13412): Terminating (SIGTERM) 16:54:32.172916 run osmo-msc_10.42.42.6(pid=13413): Terminating (SIGTERM) 16:54:32.214823 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13414): Terminating (SIGTERM) 16:54:32.256769 run osmo-mgw_10.42.42.3(pid=13415): Terminating (SIGTERM) 16:54:32.298889 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13416): Terminating (SIGTERM) 16:54:32.341716 run osmo-mgw_10.42.42.4(pid=13417): Terminating (SIGTERM) 16:54:32.382483 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13418): Terminating (SIGTERM) 16:54:32.422057 run osmo-bsc_10.42.42.7(pid=13419): Terminating (SIGTERM) 16:54:32.430837 --- ParallelTerminationStrategy: PID 13408 died... 16:54:32.484013 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13408): Terminated: ok {rc=0} 16:54:32.493810 --- ParallelTerminationStrategy: PID 13409 died... 16:54:32.547423 run osmo-hlr_10.42.42.2(pid=13409): Terminated: ok {rc=0} 16:54:32.557145 --- ParallelTerminationStrategy: PID 13410 died... 16:54:32.610704 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13410): Terminated: ok {rc=0} 16:54:32.620438 --- ParallelTerminationStrategy: PID 13411 died... 16:54:32.673659 run osmo-stp_10.42.42.5(pid=13411): Terminated: ok {rc=0} 16:54:32.683163 --- ParallelTerminationStrategy: PID 13412 died... 16:54:32.736281 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13412): Terminated: ok {rc=0} 16:54:32.745738 --- ParallelTerminationStrategy: PID 13413 died... 16:54:32.798658 run osmo-msc_10.42.42.6(pid=13413): Terminated: ok {rc=0} 16:54:32.808095 --- ParallelTerminationStrategy: PID 13414 died... 16:54:32.861485 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13414): Terminated: ok {rc=0} 16:54:32.871014 --- ParallelTerminationStrategy: PID 13415 died... 16:54:32.924047 run osmo-mgw_10.42.42.3(pid=13415): Terminated {rc=15} 16:54:32.933619 --- ParallelTerminationStrategy: PID 13416 died... 16:54:32.986642 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13416): Terminated: ok {rc=0} 16:54:32.996255 --- ParallelTerminationStrategy: PID 13417 died... 16:54:33.049557 run osmo-mgw_10.42.42.4(pid=13417): Terminated {rc=15} 16:54:33.059107 --- ParallelTerminationStrategy: PID 13418 died... 16:54:33.112184 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13418): Terminated: ok {rc=0} 16:54:33.121736 --- ParallelTerminationStrategy: PID 13419 died... 16:54:33.174804 run osmo-bsc_10.42.42.7(pid=13419): Terminated: ok {rc=0} 16:54:33.263386 bus /sierra_3: Setting Powered False 16:54:34.367005 bus /sierra_3: Setting Powered False 16:54:35.479955 bus /sierra_2: Setting Powered False 41.01sms_sysmo2023-03-07T16:53:5610gprs:sysmo35.0rm-remote-dir(pid=13482): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13482)]2063cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module> setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel bts.start() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start rem_host.recreate_remote_dir(self.remote_inst) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir self.rm_remote_dir(remote_dir) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str)) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync proc.launch_sync() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync raise self.RunError('launch_sync()') osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=13482): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13482)] ---------------------------------------------- trial-2072 gprs:sysmo cs_paging_gprs_active.py ---------------------------------------------- 16:54:37.576394 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:54:37.587252 tst gprs:sysmo: Using 1 x bts (candidates: 1) 16:54:37.719362 tst cs_paging_gprs_active.py: using LAC 41723 16:54:37.844477 tst cs_paging_gprs_active.py: using RAC 158 16:54:37.969362 tst cs_paging_gprs_active.py: using CellId 41723 16:54:38.094905 tst cs_paging_gprs_active.py: using BVCI 41724 16:54:38.119578 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:54:38.130651 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:54:38.141624 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:54:38.152802 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:54:38.163677 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:54:38.174717 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:54:38.185963 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:54:38.196978 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:54:38.208325 tst gprs:sysmo: Using 1 x modem (candidates: 4) 16:54:38.324817 tst gprs:sysmo: Using 1 x modem (candidates: 4) 16:54:38.602055 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003... 16:54:38.728543 run iperf3-srv_10.42.42.10: Starting iperf3-srv 16:54:39.038923 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13428): Launched 16:54:39.299083 run iperf3-srv_10.42.42.10(pid=13429): Launched 16:54:39.482714 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004... 16:54:39.607760 run iperf3-srv_10.42.42.10: Starting iperf3-srv 16:54:39.919314 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13430): Launched 16:54:40.176291 run iperf3-srv_10.42.42.10(pid=13431): Launched 16:54:40.358801 tst cs_paging_gprs_active.py:38: start network... 16:54:40.483410 run osmo-hlr_10.42.42.2: Starting osmo-hlr 16:54:40.967213 run create_hlr_db(pid=13432): Launched 16:54:41.139967 bus /sierra_2: Setting Powered False 16:54:42.162491 run create_hlr_db(pid=13432): Terminated: ok {rc=0} 16:54:42.563980 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13434): Launched 16:54:42.863930 run osmo-hlr_10.42.42.2(pid=13435): Launched 16:54:42.971901 run osmo-stp_10.42.42.5: Starting osmo-stp 16:54:43.492460 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13436): Launched 16:54:43.791487 run osmo-stp_10.42.42.5(pid=13437): Launched 16:54:43.898953 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 16:54:45.433400 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13438): Launched 16:54:45.540941 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 16:54:45.808973 run patchelf(pid=13439): Launched 16:54:46.990103 run patchelf(pid=13439): Terminated: ok {rc=0} 16:54:47.126377 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 16:54:47.404484 run osmo-gsm-tester_setcap_net_admin.sh(pid=13440): Launched 16:54:48.585211 run osmo-gsm-tester_setcap_net_admin.sh(pid=13440): Terminated: ok {rc=0} 16:54:48.888232 run osmo-ggsn_10.42.42.6(pid=13443): Launched 16:54:49.139305 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 16:54:54.283759 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13469): Launched 16:54:54.585731 run osmo-sgsn_10.42.42.7(pid=13470): Launched 16:54:54.697075 run osmo-msc_10.42.42.8: Starting osmo-msc 16:54:55.254143 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13471): Launched 16:54:55.555494 run osmo-msc_10.42.42.8(pid=13472): Launched 16:54:58.716642 run osmo-mgw_10.42.42.3: Starting osmo-mgw 16:54:59.256595 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13473): Launched 16:54:59.558990 run osmo-mgw_10.42.42.3(pid=13474): Launched 16:54:59.669147 run osmo-mgw_10.42.42.4: Starting osmo-mgw 16:55:00.182133 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13475): Launched 16:55:00.483598 run osmo-mgw_10.42.42.4(pid=13476): Launched 16:55:00.601469 run osmo-bsc_10.42.42.9: Starting osmo-bsc 16:55:01.282883 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13477): Launched 16:55:01.606975 run osmo-bsc_10.42.42.9(pid=13481): Launched 16:55:07.882743 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 16:55:08.458432 run rm-remote-dir(pid=13482): Launched 16:55:11.661384 run rm-remote-dir(pid=13482): ERR: Terminated: ERROR {rc=255} [trial-2072↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13482)] 16:55:11.803721 run rm-remote-dir(pid=13482): stdout: | (launched: 2023-03-07_16:55:08.291859) 16:55:11.946225 run rm-remote-dir(pid=13482): stderr: | (launched: 2023-03-07_16:55:08.343160) | ssh: connect to host 10.42.42.114 port 22: No route to host 16:55:12.005252 tst cs_paging_gprs_active.py:38: ERR: Error: rm-remote-dir(pid=13482): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13482)] [trial-2072↪gprs:sysmo↪cs_paging_gprs_active.py:38] 16:55:12.013639 tst cs_paging_gprs_active.py:38: Test FAILED (34.5 sec) 16:55:12.051903 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13428): Terminating (SIGTERM) 16:55:12.094149 run iperf3-srv_10.42.42.10(pid=13429): Terminating (SIGTERM) 16:55:12.136320 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13430): Terminating (SIGTERM) 16:55:12.178562 run iperf3-srv_10.42.42.10(pid=13431): Terminating (SIGTERM) 16:55:12.220945 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13434): Terminating (SIGTERM) 16:55:12.263164 run osmo-hlr_10.42.42.2(pid=13435): Terminating (SIGTERM) 16:55:12.305508 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13436): Terminating (SIGTERM) 16:55:12.350174 run osmo-stp_10.42.42.5(pid=13437): Terminating (SIGTERM) 16:55:12.418787 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13438): Terminating (SIGTERM) 16:55:12.464808 run osmo-ggsn_10.42.42.6(pid=13443): Terminating (SIGTERM) 16:55:12.569720 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13469): Terminating (SIGTERM) 16:55:12.614692 run osmo-sgsn_10.42.42.7(pid=13470): Terminating (SIGTERM) 16:55:12.665146 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13471): Terminating (SIGTERM) 16:55:12.709994 run osmo-msc_10.42.42.8(pid=13472): Terminating (SIGTERM) 16:55:12.754674 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13473): Terminating (SIGTERM) 16:55:12.798886 run osmo-mgw_10.42.42.3(pid=13474): Terminating (SIGTERM) 16:55:12.841832 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13475): Terminating (SIGTERM) 16:55:12.884342 run osmo-mgw_10.42.42.4(pid=13476): Terminating (SIGTERM) 16:55:12.926903 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13477): Terminating (SIGTERM) 16:55:12.969421 run osmo-bsc_10.42.42.9(pid=13481): Terminating (SIGTERM) 16:55:12.978918 --- ParallelTerminationStrategy: PID 13428 died... 16:55:13.035821 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13428): Terminated: ok {rc=0} 16:55:13.045897 --- ParallelTerminationStrategy: PID 13429 died... 16:55:13.102355 run iperf3-srv_10.42.42.10(pid=13429): Terminated {rc=256} 16:55:13.112493 --- ParallelTerminationStrategy: PID 13430 died... 16:55:13.169339 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13430): Terminated: ok {rc=0} 16:55:13.179488 --- ParallelTerminationStrategy: PID 13431 died... 16:55:13.235632 run iperf3-srv_10.42.42.10(pid=13431): Terminated {rc=256} 16:55:13.245912 --- ParallelTerminationStrategy: PID 13434 died... 16:55:13.302449 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13434): Terminated: ok {rc=0} 16:55:13.312636 --- ParallelTerminationStrategy: PID 13435 died... 16:55:13.368999 run osmo-hlr_10.42.42.2(pid=13435): Terminated: ok {rc=0} 16:55:13.379100 --- ParallelTerminationStrategy: PID 13436 died... 16:55:13.436348 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13436): Terminated: ok {rc=0} 16:55:13.446631 --- ParallelTerminationStrategy: PID 13437 died... 16:55:13.504537 run osmo-stp_10.42.42.5(pid=13437): Terminated: ok {rc=0} 16:55:13.514805 --- ParallelTerminationStrategy: PID 13438 died... 16:55:13.572224 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13438): Terminated: ok {rc=0} 16:55:13.582423 --- ParallelTerminationStrategy: PID 13443 died... 16:55:13.640235 run osmo-ggsn_10.42.42.6(pid=13443): Terminated: ok {rc=0} 16:55:13.655079 --- ParallelTerminationStrategy: PID 13469 died... 16:55:13.712558 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13469): Terminated: ok {rc=0} 16:55:13.731643 --- ParallelTerminationStrategy: PID 13471 died... 16:55:13.851229 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13471): Terminated: ok {rc=0} 16:55:13.870243 --- ParallelTerminationStrategy: PID 13472 died... 16:55:13.974105 run osmo-msc_10.42.42.8(pid=13472): Terminated: ok {rc=0} 16:55:13.984559 --- ParallelTerminationStrategy: PID 13473 died... 16:55:14.042280 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13473): Terminated: ok {rc=0} 16:55:14.052804 --- ParallelTerminationStrategy: PID 13474 died... 16:55:14.110498 run osmo-mgw_10.42.42.3(pid=13474): Terminated {rc=15} 16:55:14.120857 --- ParallelTerminationStrategy: PID 13475 died... 16:55:14.178510 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13475): Terminated: ok {rc=0} 16:55:14.188984 --- ParallelTerminationStrategy: PID 13476 died... 16:55:14.246443 run osmo-mgw_10.42.42.4(pid=13476): Terminated {rc=15} 16:55:14.256901 --- ParallelTerminationStrategy: PID 13477 died... 16:55:14.314377 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13477): Terminated: ok {rc=0} 16:55:14.325119 --- ParallelTerminationStrategy: PID 13481 died... 16:55:14.382461 run osmo-bsc_10.42.42.9(pid=13481): Terminated: ok {rc=0} 16:55:14.561224 --- ParallelTerminationStrategy: PID 13470 died... 16:55:14.619083 run osmo-sgsn_10.42.42.7(pid=13470): Terminated: ok {rc=0} 16:55:14.706222 bus /gobi_0: Setting Powered False 16:55:15.814258 bus /gobi_0: Setting Powered False 16:55:16.922451 bus /gobi_6: Setting Powered False 10gprs:sysmo28.0rm-remote-dir(pid=13538): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13538)]2063iperf3.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module> setup_run_iperf3_test_parallel(1) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel bts.start() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start rem_host.recreate_remote_dir(self.remote_inst) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir self.rm_remote_dir(remote_dir) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str)) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync proc.launch_sync() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync raise self.RunError('launch_sync()') osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=13538): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13538)] ---------------------------------------------- trial-2072 gprs:sysmo iperf3.py ---------------------------------------------- 16:55:18.092130 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:18.103799 tst gprs:sysmo: Using 1 x bts (candidates: 1) 16:55:18.233783 tst iperf3.py: using LAC 41724 16:55:18.386259 tst iperf3.py: using RAC 159 16:55:18.506296 tst iperf3.py: using CellId 41724 16:55:18.631608 tst iperf3.py: using BVCI 41725 16:55:18.656527 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:18.667479 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:18.678413 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:18.689539 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:18.700586 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:18.711466 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:18.722732 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:18.733948 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:18.745114 tst gprs:sysmo: Using 1 x modem (candidates: 4) 16:55:19.022854 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003... 16:55:19.148529 run iperf3-srv_10.42.42.10: Starting iperf3-srv 16:55:19.462423 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13495): Launched 16:55:19.721363 run iperf3-srv_10.42.42.10(pid=13496): Launched 16:55:19.905020 tst iperf3.py:8: start network... 16:55:20.030201 run osmo-hlr_10.42.42.2: Starting osmo-hlr 16:55:20.512291 run create_hlr_db(pid=13497): Launched 16:55:20.690213 bus /gobi_6: Setting Powered False 16:55:21.752046 run create_hlr_db(pid=13497): Terminated: ok {rc=0} 16:55:22.202457 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13499): Launched 16:55:22.533200 run osmo-hlr_10.42.42.2(pid=13500): Launched 16:55:22.650772 run osmo-stp_10.42.42.5: Starting osmo-stp 16:55:23.211321 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13501): Launched 16:55:23.589515 run osmo-stp_10.42.42.5(pid=13502): Launched 16:55:23.706513 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 16:55:24.233526 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13503): Launched 16:55:24.340932 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 16:55:24.604761 run patchelf(pid=13504): Launched 16:55:25.786169 run patchelf(pid=13504): Terminated: ok {rc=0} 16:55:25.897189 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 16:55:26.168536 run osmo-gsm-tester_setcap_net_admin.sh(pid=13505): Launched 16:55:27.350021 run osmo-gsm-tester_setcap_net_admin.sh(pid=13505): Terminated: ok {rc=0} 16:55:27.663067 run osmo-ggsn_10.42.42.6(pid=13508): Launched 16:55:27.879243 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 16:55:28.452077 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13528): Launched 16:55:28.758367 run osmo-sgsn_10.42.42.7(pid=13529): Launched 16:55:28.868875 run osmo-msc_10.42.42.8: Starting osmo-msc 16:55:29.423986 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13530): Launched 16:55:29.726137 run osmo-msc_10.42.42.8(pid=13531): Launched 16:55:32.887011 run osmo-mgw_10.42.42.3: Starting osmo-mgw 16:55:33.411599 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13532): Launched 16:55:33.712954 run osmo-mgw_10.42.42.3(pid=13533): Launched 16:55:33.820389 run osmo-mgw_10.42.42.4: Starting osmo-mgw 16:55:34.333924 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13534): Launched 16:55:34.634027 run osmo-mgw_10.42.42.4(pid=13535): Launched 16:55:34.741507 run osmo-bsc_10.42.42.9: Starting osmo-bsc 16:55:35.373812 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13536): Launched 16:55:35.675545 run osmo-bsc_10.42.42.9(pid=13537): Launched 16:55:41.933205 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 16:55:42.471955 run rm-remote-dir(pid=13538): Launched 16:55:45.662550 run rm-remote-dir(pid=13538): ERR: Terminated: ERROR {rc=255} [trial-2072↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13538)] 16:55:45.795859 run rm-remote-dir(pid=13538): stdout: | (launched: 2023-03-07_16:55:42.317281) 16:55:45.929947 run rm-remote-dir(pid=13538): stderr: | (launched: 2023-03-07_16:55:42.365389) | ssh: connect to host 10.42.42.114 port 22: No route to host 16:55:45.985329 tst iperf3.py:8: ERR: Error: rm-remote-dir(pid=13538): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13538)] [trial-2072↪gprs:sysmo↪iperf3.py:8] 16:55:45.993106 tst iperf3.py:8: Test FAILED (27.9 sec) 16:55:46.027919 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13495): Terminating (SIGTERM) 16:55:46.067685 run iperf3-srv_10.42.42.10(pid=13496): Terminating (SIGTERM) 16:55:46.107624 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13499): Terminating (SIGTERM) 16:55:46.147419 run osmo-hlr_10.42.42.2(pid=13500): Terminating (SIGTERM) 16:55:46.187810 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13501): Terminating (SIGTERM) 16:55:46.229351 run osmo-stp_10.42.42.5(pid=13502): Terminating (SIGTERM) 16:55:46.271447 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13503): Terminating (SIGTERM) 16:55:46.311890 run osmo-ggsn_10.42.42.6(pid=13508): Terminating (SIGTERM) 16:55:46.373542 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13528): Terminating (SIGTERM) 16:55:46.427214 run osmo-sgsn_10.42.42.7(pid=13529): Terminating (SIGTERM) 16:55:46.468219 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13530): Terminating (SIGTERM) 16:55:46.516297 run osmo-msc_10.42.42.8(pid=13531): Terminating (SIGTERM) 16:55:46.561652 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13532): Terminating (SIGTERM) 16:55:46.609392 run osmo-mgw_10.42.42.3(pid=13533): Terminating (SIGTERM) 16:55:46.649145 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13534): Terminating (SIGTERM) 16:55:46.688556 run osmo-mgw_10.42.42.4(pid=13535): Terminating (SIGTERM) 16:55:46.728696 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13536): Terminating (SIGTERM) 16:55:46.768071 run osmo-bsc_10.42.42.9(pid=13537): Terminating (SIGTERM) 16:55:46.777213 --- ParallelTerminationStrategy: PID 13495 died... 16:55:46.832085 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13495): Terminated: ok {rc=0} 16:55:46.842217 --- ParallelTerminationStrategy: PID 13496 died... 16:55:46.898021 run iperf3-srv_10.42.42.10(pid=13496): Terminated {rc=256} 16:55:46.925463 --- ParallelTerminationStrategy: PID 13499 died... 16:55:46.994099 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13499): Terminated: ok {rc=0} 16:55:47.004617 --- ParallelTerminationStrategy: PID 13500 died... 16:55:47.059472 run osmo-hlr_10.42.42.2(pid=13500): Terminated: ok {rc=0} 16:55:47.070289 --- ParallelTerminationStrategy: PID 13501 died... 16:55:47.123964 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13501): Terminated: ok {rc=0} 16:55:47.133593 --- ParallelTerminationStrategy: PID 13502 died... 16:55:47.186503 run osmo-stp_10.42.42.5(pid=13502): Terminated: ok {rc=0} 16:55:47.195970 --- ParallelTerminationStrategy: PID 13503 died... 16:55:47.248809 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13503): Terminated: ok {rc=0} 16:55:47.258435 --- ParallelTerminationStrategy: PID 13508 died... 16:55:47.311259 run osmo-ggsn_10.42.42.6(pid=13508): Terminated: ok {rc=0} 16:55:47.320985 --- ParallelTerminationStrategy: PID 13528 died... 16:55:47.373820 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13528): Terminated: ok {rc=0} 16:55:47.383380 --- ParallelTerminationStrategy: PID 13530 died... 16:55:47.436214 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13530): Terminated: ok {rc=0} 16:55:47.446455 --- ParallelTerminationStrategy: PID 13531 died... 16:55:47.500788 run osmo-msc_10.42.42.8(pid=13531): Terminated: ok {rc=0} 16:55:47.510834 --- ParallelTerminationStrategy: PID 13532 died... 16:55:47.566087 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13532): Terminated: ok {rc=0} 16:55:47.576438 --- ParallelTerminationStrategy: PID 13533 died... 16:55:47.631735 run osmo-mgw_10.42.42.3(pid=13533): Terminated {rc=15} 16:55:47.642054 --- ParallelTerminationStrategy: PID 13534 died... 16:55:47.697670 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13534): Terminated: ok {rc=0} 16:55:47.707895 --- ParallelTerminationStrategy: PID 13535 died... 16:55:47.762966 run osmo-mgw_10.42.42.4(pid=13535): Terminated {rc=15} 16:55:47.772693 --- ParallelTerminationStrategy: PID 13529 died... 16:55:47.825826 run osmo-sgsn_10.42.42.7(pid=13529): Terminated: ok {rc=0} 16:55:47.835445 --- ParallelTerminationStrategy: PID 13536 died... 16:55:47.888363 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13536): Terminated: ok {rc=0} 16:55:47.898084 --- ParallelTerminationStrategy: PID 13537 died... 16:55:47.951166 run osmo-bsc_10.42.42.9(pid=13537): Terminated: ok {rc=0} 16:55:48.035081 bus /gobi_6: Setting Powered False 10gprs:sysmo34.0rm-remote-dir(pid=13869): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13869)]2063iperf3m4.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module> setup_run_iperf3_test_parallel(4) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel bts.start() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start rem_host.recreate_remote_dir(self.remote_inst) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir self.rm_remote_dir(remote_dir) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str)) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync proc.launch_sync() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync raise self.RunError('launch_sync()') osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=13869): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13869)] ---------------------------------------------- trial-2072 gprs:sysmo iperf3m4.py ---------------------------------------------- 16:55:49.204549 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:49.216284 tst gprs:sysmo: Using 1 x bts (candidates: 1) 16:55:49.345585 tst iperf3m4.py: using LAC 41725 16:55:49.473148 tst iperf3m4.py: using RAC 160 16:55:49.600777 tst iperf3m4.py: using CellId 41725 16:55:49.729104 tst iperf3m4.py: using BVCI 41726 16:55:49.753824 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:49.764783 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:49.775658 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:49.786717 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:49.797632 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:49.808647 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:49.819691 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:49.830794 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:55:49.842014 tst gprs:sysmo: Using 1 x modem (candidates: 4) 16:55:49.962207 tst gprs:sysmo: Using 1 x modem (candidates: 4) 16:55:50.079545 tst gprs:sysmo: Using 1 x modem (candidates: 4) 16:55:50.199706 tst gprs:sysmo: Using 1 x modem (candidates: 4) 16:55:50.496231 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003... 16:55:50.631638 run iperf3-srv_10.42.42.10: Starting iperf3-srv 16:55:50.985507 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13551): Launched 16:55:51.256002 run iperf3-srv_10.42.42.10(pid=13552): Launched 16:55:51.455203 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004... 16:55:51.600455 run iperf3-srv_10.42.42.10: Starting iperf3-srv 16:55:52.030469 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13589): Launched 16:55:52.330442 run iperf3-srv_10.42.42.10(pid=13602): Launched 16:55:52.532054 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005... 16:55:52.680395 run iperf3-srv_10.42.42.10: Starting iperf3-srv 16:55:53.049883 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13628): Launched 16:55:53.384541 run iperf3-srv_10.42.42.10(pid=13647): Launched 16:55:53.618320 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006... 16:55:53.775809 run iperf3-srv_10.42.42.10: Starting iperf3-srv 16:55:54.175287 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13753): Launched 16:55:54.498138 run iperf3-srv_10.42.42.10(pid=13799): Launched 16:55:54.714890 tst iperf3m4.py:8: start network... 16:55:54.845254 run osmo-hlr_10.42.42.2: Starting osmo-hlr 16:55:55.390481 run create_hlr_db(pid=13829): Launched 16:55:55.584698 bus /gobi_6: Setting Powered False 16:55:56.604444 run create_hlr_db(pid=13829): Terminated: ok {rc=0} 16:55:57.035946 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13832): Launched 16:55:57.446464 run osmo-hlr_10.42.42.2(pid=13834): Launched 16:55:57.614972 run osmo-stp_10.42.42.5: Starting osmo-stp 16:55:58.193972 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13835): Launched 16:55:58.606910 run osmo-stp_10.42.42.5(pid=13836): Launched 16:55:58.768869 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 16:55:59.314930 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13837): Launched 16:55:59.424002 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 16:55:59.712293 run patchelf(pid=13838): Launched 16:56:00.901114 run patchelf(pid=13838): Terminated: ok {rc=0} 16:56:01.018212 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 16:56:01.326233 run osmo-gsm-tester_setcap_net_admin.sh(pid=13839): Launched 16:56:02.516111 run osmo-gsm-tester_setcap_net_admin.sh(pid=13839): Terminated: ok {rc=0} 16:56:02.928622 run osmo-ggsn_10.42.42.6(pid=13842): Launched 16:56:03.167036 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 16:56:03.890515 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13859): Launched 16:56:04.315757 run osmo-sgsn_10.42.42.7(pid=13860): Launched 16:56:04.562647 run osmo-msc_10.42.42.8: Starting osmo-msc 16:56:05.246184 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13861): Launched 16:56:05.675189 run osmo-msc_10.42.42.8(pid=13862): Launched 16:56:08.872418 run osmo-mgw_10.42.42.3: Starting osmo-mgw 16:56:09.429851 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13863): Launched 16:56:09.840034 run osmo-mgw_10.42.42.3(pid=13864): Launched 16:56:10.014097 run osmo-mgw_10.42.42.4: Starting osmo-mgw 16:56:10.558603 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13865): Launched 16:56:10.976446 run osmo-mgw_10.42.42.4(pid=13866): Launched 16:56:11.141323 run osmo-bsc_10.42.42.9: Starting osmo-bsc 16:56:11.827975 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13867): Launched 16:56:12.244935 run osmo-bsc_10.42.42.9(pid=13868): Launched 16:56:18.526382 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 16:56:19.065704 run rm-remote-dir(pid=13869): Launched 16:56:22.295517 run rm-remote-dir(pid=13869): ERR: Terminated: ERROR {rc=255} [trial-2072↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13869)] 16:56:22.429527 run rm-remote-dir(pid=13869): stdout: | (launched: 2023-03-07_16:56:18.910430) 16:56:22.563300 run rm-remote-dir(pid=13869): stderr: | (launched: 2023-03-07_16:56:18.958435) | ssh: connect to host 10.42.42.114 port 22: No route to host 16:56:22.618816 tst iperf3m4.py:8: ERR: Error: rm-remote-dir(pid=13869): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13869)] [trial-2072↪gprs:sysmo↪iperf3m4.py:8] 16:56:22.626578 tst iperf3m4.py:8: Test FAILED (33.5 sec) 16:56:22.661841 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13551): Terminating (SIGTERM) 16:56:22.701960 run iperf3-srv_10.42.42.10(pid=13552): Terminating (SIGTERM) 16:56:22.741991 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13589): Terminating (SIGTERM) 16:56:22.781976 run iperf3-srv_10.42.42.10(pid=13602): Terminating (SIGTERM) 16:56:22.821946 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13628): Terminating (SIGTERM) 16:56:22.861776 run iperf3-srv_10.42.42.10(pid=13647): Terminating (SIGTERM) 16:56:22.901521 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13753): Terminating (SIGTERM) 16:56:22.941261 run iperf3-srv_10.42.42.10(pid=13799): Terminating (SIGTERM) 16:56:22.981234 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13832): Terminating (SIGTERM) 16:56:23.021163 run osmo-hlr_10.42.42.2(pid=13834): Terminating (SIGTERM) 16:56:23.062052 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13835): Terminating (SIGTERM) 16:56:23.105520 run osmo-stp_10.42.42.5(pid=13836): Terminating (SIGTERM) 16:56:23.148688 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13837): Terminating (SIGTERM) 16:56:23.192837 run osmo-ggsn_10.42.42.6(pid=13842): Terminating (SIGTERM) 16:56:23.252083 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13859): Terminating (SIGTERM) 16:56:23.308481 run osmo-sgsn_10.42.42.7(pid=13860): Terminating (SIGTERM) 16:56:23.350413 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13861): Terminating (SIGTERM) 16:56:23.391782 run osmo-msc_10.42.42.8(pid=13862): Terminating (SIGTERM) 16:56:23.433035 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13863): Terminating (SIGTERM) 16:56:23.473191 run osmo-mgw_10.42.42.3(pid=13864): Terminating (SIGTERM) 16:56:23.513805 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13865): Terminating (SIGTERM) 16:56:23.553866 run osmo-mgw_10.42.42.4(pid=13866): Terminating (SIGTERM) 16:56:23.594507 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13867): Terminating (SIGTERM) 16:56:23.634453 run osmo-bsc_10.42.42.9(pid=13868): Terminating (SIGTERM) 16:56:23.643534 --- ParallelTerminationStrategy: PID 13551 died... 16:56:23.697170 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13551): Terminated: ok {rc=0} 16:56:23.706616 --- ParallelTerminationStrategy: PID 13552 died... 16:56:23.759835 run iperf3-srv_10.42.42.10(pid=13552): Terminated {rc=256} 16:56:23.769344 --- ParallelTerminationStrategy: PID 13589 died... 16:56:23.824015 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13589): Terminated: ok {rc=0} 16:56:23.833965 --- ParallelTerminationStrategy: PID 13602 died... 16:56:23.888668 run iperf3-srv_10.42.42.10(pid=13602): Terminated {rc=256} 16:56:23.898859 --- ParallelTerminationStrategy: PID 13628 died... 16:56:23.953062 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13628): Terminated: ok {rc=0} 16:56:23.962558 --- ParallelTerminationStrategy: PID 13647 died... 16:56:24.015786 run iperf3-srv_10.42.42.10(pid=13647): Terminated {rc=256} 16:56:24.025258 --- ParallelTerminationStrategy: PID 13753 died... 16:56:24.078697 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=13753): Terminated: ok {rc=0} 16:56:24.088225 --- ParallelTerminationStrategy: PID 13799 died... 16:56:24.141522 run iperf3-srv_10.42.42.10(pid=13799): Terminated {rc=256} 16:56:24.150947 --- ParallelTerminationStrategy: PID 13832 died... 16:56:24.204387 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13832): Terminated: ok {rc=0} 16:56:24.213925 --- ParallelTerminationStrategy: PID 13834 died... 16:56:24.267045 run osmo-hlr_10.42.42.2(pid=13834): Terminated: ok {rc=0} 16:56:24.276552 --- ParallelTerminationStrategy: PID 13835 died... 16:56:24.330242 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13835): Terminated: ok {rc=0} 16:56:24.340219 --- ParallelTerminationStrategy: PID 13836 died... 16:56:24.394725 run osmo-stp_10.42.42.5(pid=13836): Terminated: ok {rc=0} 16:56:24.421726 --- ParallelTerminationStrategy: PID 13837 died... 16:56:24.541848 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13837): Terminated: ok {rc=0} 16:56:24.560720 --- ParallelTerminationStrategy: PID 13842 died... 16:56:24.675288 run osmo-ggsn_10.42.42.6(pid=13842): Terminated: ok {rc=0} 16:56:24.686206 --- ParallelTerminationStrategy: PID 13859 died... 16:56:24.743750 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13859): Terminated: ok {rc=0} 16:56:24.758374 --- ParallelTerminationStrategy: PID 13861 died... 16:56:24.811386 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13861): Terminated: ok {rc=0} 16:56:24.821187 --- ParallelTerminationStrategy: PID 13860 died... 16:56:24.874021 run osmo-sgsn_10.42.42.7(pid=13860): Terminated: ok {rc=0} 16:56:24.883756 --- ParallelTerminationStrategy: PID 13862 died... 16:56:24.937001 run osmo-msc_10.42.42.8(pid=13862): Terminated: ok {rc=0} 16:56:24.946746 --- ParallelTerminationStrategy: PID 13863 died... 16:56:24.999742 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13863): Terminated: ok {rc=0} 16:56:25.009530 --- ParallelTerminationStrategy: PID 13864 died... 16:56:25.062948 run osmo-mgw_10.42.42.3(pid=13864): Terminated {rc=15} 16:56:25.072837 --- ParallelTerminationStrategy: PID 13865 died... 16:56:25.125942 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13865): Terminated: ok {rc=0} 16:56:25.136092 --- ParallelTerminationStrategy: PID 13866 died... 16:56:25.189620 run osmo-mgw_10.42.42.4(pid=13866): Terminated {rc=15} 16:56:25.199425 --- ParallelTerminationStrategy: PID 13867 died... 16:56:25.252772 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13867): Terminated: ok {rc=0} 16:56:25.262634 --- ParallelTerminationStrategy: PID 13868 died... 16:56:25.315782 run osmo-bsc_10.42.42.9(pid=13868): Terminated: ok {rc=0} 16:56:25.404668 bus /gobi_4: Setting Powered False 16:56:26.515800 bus /gobi_4: Setting Powered False 16:56:27.624860 bus /gobi_1: Setting Powered False 16:56:28.732623 bus /gobi_1: Setting Powered False 16:56:29.841440 bus /gobi_0: Setting Powered False 16:56:30.948659 bus /gobi_0: Setting Powered False 16:56:32.056687 bus /gobi_6: Setting Powered False 4gprs:sysmo28.0rm-remote-dir(pid=13923): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪ping.py:29↪ping.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13923)]2069ping.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 29, in <module> bts.start() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start rem_host.recreate_remote_dir(self.remote_inst) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir self.rm_remote_dir(remote_dir) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str)) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync proc.launch_sync() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync raise self.RunError('launch_sync()') osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=13923): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪ping.py:29↪ping.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13923)] ---------------------------------------------- trial-2072 gprs:sysmo ping.py ---------------------------------------------- 16:56:33.180373 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:56:33.192018 tst gprs:sysmo: Using 1 x bts (candidates: 1) 16:56:33.319820 tst ping.py: using LAC 41726 16:56:33.437563 tst ping.py: using RAC 161 16:56:33.553388 tst ping.py: using CellId 41726 16:56:33.668955 tst ping.py: using BVCI 41727 16:56:33.691799 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:56:33.702328 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:56:33.712912 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:56:33.723156 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:56:33.733379 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:56:33.743577 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:56:33.753998 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:56:33.764303 tst gprs:sysmo: Using 1 x modem (candidates: 4) 16:56:34.054364 tst ping.py:19: start network... 16:56:34.187488 run osmo-hlr_10.42.42.2: Starting osmo-hlr 16:56:34.688663 run create_hlr_db(pid=13883): Launched 16:56:34.874221 bus /gobi_6: Setting Powered False 16:56:35.931729 run create_hlr_db(pid=13883): Terminated: ok {rc=0} 16:56:36.380893 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13885): Launched 16:56:36.737880 run osmo-hlr_10.42.42.2(pid=13886): Launched 16:56:36.929209 run osmo-stp_10.42.42.5: Starting osmo-stp 16:56:37.559737 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13887): Launched 16:56:37.879600 run osmo-stp_10.42.42.5(pid=13888): Launched 16:56:37.993705 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 16:56:38.549987 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13889): Launched 16:56:38.663459 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 16:56:38.941894 run patchelf(pid=13890): Launched 16:56:40.132121 run patchelf(pid=13890): Terminated: ok {rc=0} 16:56:40.249682 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 16:56:40.533444 run osmo-gsm-tester_setcap_net_admin.sh(pid=13891): Launched 16:56:41.722571 run osmo-gsm-tester_setcap_net_admin.sh(pid=13891): Terminated: ok {rc=0} 16:56:42.048279 run osmo-ggsn_10.42.42.6(pid=13894): Launched 16:56:42.269109 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 16:56:42.820532 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13913): Launched 16:56:43.117748 run osmo-sgsn_10.42.42.7(pid=13914): Launched 16:56:43.226064 run osmo-msc_10.42.42.8: Starting osmo-msc 16:56:43.773962 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13915): Launched 16:56:44.072119 run osmo-msc_10.42.42.8(pid=13916): Launched 16:56:47.230013 run osmo-mgw_10.42.42.3: Starting osmo-mgw 16:56:47.748139 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13917): Launched 16:56:48.045036 run osmo-mgw_10.42.42.3(pid=13918): Launched 16:56:48.151436 run osmo-mgw_10.42.42.4: Starting osmo-mgw 16:56:48.657245 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13919): Launched 16:56:48.952920 run osmo-mgw_10.42.42.4(pid=13920): Launched 16:56:49.057820 run osmo-bsc_10.42.42.9: Starting osmo-bsc 16:56:49.693974 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13921): Launched 16:56:49.991635 run osmo-bsc_10.42.42.9(pid=13922): Launched 16:56:56.246408 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 16:56:56.779280 run rm-remote-dir(pid=13923): Launched 16:56:59.968569 run rm-remote-dir(pid=13923): ERR: Terminated: ERROR {rc=255} [trial-2072↪gprs:sysmo↪ping.py:29↪ping.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13923)] 16:57:00.100460 run rm-remote-dir(pid=13923): stdout: | (launched: 2023-03-07_16:56:56.626049) 16:57:00.233095 run rm-remote-dir(pid=13923): stderr: | (launched: 2023-03-07_16:56:56.673850) | ssh: connect to host 10.42.42.114 port 22: No route to host 16:57:00.287291 tst ping.py:29: ERR: Error: rm-remote-dir(pid=13923): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪ping.py:29↪ping.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13923)] [trial-2072↪gprs:sysmo↪ping.py:29] 16:57:00.294442 tst ping.py:29: Test FAILED (27.1 sec) 16:57:00.329352 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13885): Terminating (SIGTERM) 16:57:00.369232 run osmo-hlr_10.42.42.2(pid=13886): Terminating (SIGTERM) 16:57:00.410294 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13887): Terminating (SIGTERM) 16:57:00.453380 run osmo-stp_10.42.42.5(pid=13888): Terminating (SIGTERM) 16:57:00.496045 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13889): Terminating (SIGTERM) 16:57:00.539220 run osmo-ggsn_10.42.42.6(pid=13894): Terminating (SIGTERM) 16:57:00.589355 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13913): Terminating (SIGTERM) 16:57:00.642226 run osmo-sgsn_10.42.42.7(pid=13914): Terminating (SIGTERM) 16:57:00.686913 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13915): Terminating (SIGTERM) 16:57:00.730600 run osmo-msc_10.42.42.8(pid=13916): Terminating (SIGTERM) 16:57:00.778745 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13917): Terminating (SIGTERM) 16:57:00.824976 run osmo-mgw_10.42.42.3(pid=13918): Terminating (SIGTERM) 16:57:00.873605 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13919): Terminating (SIGTERM) 16:57:00.913561 run osmo-mgw_10.42.42.4(pid=13920): Terminating (SIGTERM) 16:57:00.953948 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13921): Terminating (SIGTERM) 16:57:00.993819 run osmo-bsc_10.42.42.9(pid=13922): Terminating (SIGTERM) 16:57:01.002853 --- ParallelTerminationStrategy: PID 13885 died... 16:57:01.056505 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13885): Terminated: ok {rc=0} 16:57:01.067052 --- ParallelTerminationStrategy: PID 13886 died... 16:57:01.122879 run osmo-hlr_10.42.42.2(pid=13886): Terminated: ok {rc=0} 16:57:01.133498 --- ParallelTerminationStrategy: PID 13887 died... 16:57:01.191598 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13887): Terminated: ok {rc=0} 16:57:01.202637 --- ParallelTerminationStrategy: PID 13888 died... 16:57:01.277864 run osmo-stp_10.42.42.5(pid=13888): Terminated: ok {rc=0} 16:57:01.288763 --- ParallelTerminationStrategy: PID 13889 died... 16:57:01.346366 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13889): Terminated: ok {rc=0} 16:57:01.362003 --- ParallelTerminationStrategy: PID 13894 died... 16:57:01.415495 run osmo-ggsn_10.42.42.6(pid=13894): Terminated: ok {rc=0} 16:57:01.425729 --- ParallelTerminationStrategy: PID 13913 died... 16:57:01.478836 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13913): Terminated: ok {rc=0} 16:57:01.488599 --- ParallelTerminationStrategy: PID 13915 died... 16:57:01.541407 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13915): Terminated: ok {rc=0} 16:57:01.551143 --- ParallelTerminationStrategy: PID 13916 died... 16:57:01.604371 run osmo-msc_10.42.42.8(pid=13916): Terminated: ok {rc=0} 16:57:01.614363 --- ParallelTerminationStrategy: PID 13917 died... 16:57:01.667835 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13917): Terminated: ok {rc=0} 16:57:01.678097 --- ParallelTerminationStrategy: PID 13918 died... 16:57:01.732479 run osmo-mgw_10.42.42.3(pid=13918): Terminated {rc=15} 16:57:01.742757 --- ParallelTerminationStrategy: PID 13919 died... 16:57:01.798498 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13919): Terminated: ok {rc=0} 16:57:01.809125 --- ParallelTerminationStrategy: PID 13920 died... 16:57:01.864694 run osmo-mgw_10.42.42.4(pid=13920): Terminated {rc=15} 16:57:01.875076 --- ParallelTerminationStrategy: PID 13921 died... 16:57:01.930372 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13921): Terminated: ok {rc=0} 16:57:01.940664 --- ParallelTerminationStrategy: PID 13922 died... 16:57:01.994632 run osmo-bsc_10.42.42.9(pid=13922): Terminated: ok {rc=0} 16:57:02.004438 --- ParallelTerminationStrategy: PID 13914 died... 16:57:02.057363 run osmo-sgsn_10.42.42.7(pid=13914): Terminated: ok {rc=0} 16:57:02.141002 bus /gobi_6: Setting Powered False 4gprs:sysmo28.0rm-remote-dir(pid=13979): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪ping_idle_ping.py:34↪ping_idle_ping.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13979)]2069ping_idle_ping.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 34, in <module> bts.start() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start rem_host.recreate_remote_dir(self.remote_inst) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir self.rm_remote_dir(remote_dir) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str)) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync proc.launch_sync() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync raise self.RunError('launch_sync()') osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=13979): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪ping_idle_ping.py:34↪ping_idle_ping.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13979)] ---------------------------------------------- trial-2072 gprs:sysmo ping_idle_ping.py ---------------------------------------------- 16:57:03.248313 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:57:03.259848 tst gprs:sysmo: Using 1 x bts (candidates: 1) 16:57:03.394372 tst ping_idle_ping.py: using LAC 41727 16:57:03.518332 tst ping_idle_ping.py: using RAC 162 16:57:03.641612 tst ping_idle_ping.py: using CellId 41727 16:57:03.764580 tst ping_idle_ping.py: using BVCI 41728 16:57:03.789113 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:57:03.800011 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:57:03.810977 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:57:03.822216 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:57:03.833062 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:57:03.843969 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:57:03.855136 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 16:57:03.866263 tst gprs:sysmo: Using 1 x modem (candidates: 4) 16:57:04.156022 tst ping_idle_ping.py:24: start network... 16:57:04.289316 run osmo-hlr_10.42.42.2: Starting osmo-hlr 16:57:04.784467 run create_hlr_db(pid=13935): Launched 16:57:04.958633 bus /gobi_6: Setting Powered False 16:57:06.012552 run create_hlr_db(pid=13935): Terminated: ok {rc=0} 16:57:06.469747 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13937): Launched 16:57:06.995458 run osmo-hlr_10.42.42.2(pid=13938): Launched 16:57:07.104728 run osmo-stp_10.42.42.5: Starting osmo-stp 16:57:07.651473 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13939): Launched 16:57:07.970273 run osmo-stp_10.42.42.5(pid=13940): Launched 16:57:08.084206 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 16:57:08.645648 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13941): Launched 16:57:08.759990 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 16:57:09.040926 run patchelf(pid=13942): Launched 16:57:10.232423 run patchelf(pid=13942): Terminated: ok {rc=0} 16:57:10.350411 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 16:57:10.637159 run osmo-gsm-tester_setcap_net_admin.sh(pid=13943): Launched 16:57:11.828091 run osmo-gsm-tester_setcap_net_admin.sh(pid=13943): Terminated: ok {rc=0} 16:57:12.161550 run osmo-ggsn_10.42.42.6(pid=13947): Launched 16:57:12.345184 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 16:57:12.949308 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13969): Launched 16:57:13.261097 run osmo-sgsn_10.42.42.7(pid=13970): Launched 16:57:13.375854 run osmo-msc_10.42.42.8: Starting osmo-msc 16:57:13.951782 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13971): Launched 16:57:14.267789 run osmo-msc_10.42.42.8(pid=13972): Launched 16:57:17.436914 run osmo-mgw_10.42.42.3: Starting osmo-mgw 16:57:17.985018 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13973): Launched 16:57:18.300992 run osmo-mgw_10.42.42.3(pid=13974): Launched 16:57:18.415498 run osmo-mgw_10.42.42.4: Starting osmo-mgw 16:57:18.956341 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13975): Launched 16:57:19.272086 run osmo-mgw_10.42.42.4(pid=13976): Launched 16:57:19.385985 run osmo-bsc_10.42.42.9: Starting osmo-bsc 16:57:20.050951 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13977): Launched 16:57:20.369252 run osmo-bsc_10.42.42.9(pid=13978): Launched 16:57:26.639903 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 16:57:27.205958 run rm-remote-dir(pid=13979): Launched 16:57:30.406328 run rm-remote-dir(pid=13979): ERR: Terminated: ERROR {rc=255} [trial-2072↪gprs:sysmo↪ping_idle_ping.py:34↪ping_idle_ping.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13979)] 16:57:30.548024 run rm-remote-dir(pid=13979): stdout: | (launched: 2023-03-07_16:57:27.042444) 16:57:30.689531 run rm-remote-dir(pid=13979): stderr: | (launched: 2023-03-07_16:57:27.092878) | ssh: connect to host 10.42.42.114 port 22: No route to host 16:57:30.747663 tst ping_idle_ping.py:34: ERR: Error: rm-remote-dir(pid=13979): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2072↪gprs:sysmo↪ping_idle_ping.py:34↪ping_idle_ping.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=13979)] [trial-2072↪gprs:sysmo↪ping_idle_ping.py:34] 16:57:30.755673 tst ping_idle_ping.py:34: Test FAILED (27.5 sec) 16:57:30.793184 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13937): Terminating (SIGTERM) 16:57:30.834966 run osmo-hlr_10.42.42.2(pid=13938): Terminating (SIGTERM) 16:57:30.878042 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13939): Terminating (SIGTERM) 16:57:30.922178 run osmo-stp_10.42.42.5(pid=13940): Terminating (SIGTERM) 16:57:30.966593 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13941): Terminating (SIGTERM) 16:57:31.014248 run osmo-ggsn_10.42.42.6(pid=13947): Terminating (SIGTERM) 16:57:31.098397 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13969): Terminating (SIGTERM) 16:57:31.153819 run osmo-sgsn_10.42.42.7(pid=13970): Terminating (SIGTERM) 16:57:31.211856 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13971): Terminating (SIGTERM) 16:57:31.256489 run osmo-msc_10.42.42.8(pid=13972): Terminating (SIGTERM) 16:57:31.301179 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13973): Terminating (SIGTERM) 16:57:31.345774 run osmo-mgw_10.42.42.3(pid=13974): Terminating (SIGTERM) 16:57:31.390492 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13975): Terminating (SIGTERM) 16:57:31.435685 run osmo-mgw_10.42.42.4(pid=13976): Terminating (SIGTERM) 16:57:31.487276 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13977): Terminating (SIGTERM) 16:57:31.532201 run osmo-bsc_10.42.42.9(pid=13978): Terminating (SIGTERM) 16:57:31.542474 --- ParallelTerminationStrategy: PID 13937 died... 16:57:31.602339 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13937): Terminated: ok {rc=0} 16:57:31.613702 --- ParallelTerminationStrategy: PID 13938 died... 16:57:31.672777 run osmo-hlr_10.42.42.2(pid=13938): Terminated: ok {rc=0} 16:57:31.683921 --- ParallelTerminationStrategy: PID 13939 died... 16:57:31.744781 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13939): Terminated: ok {rc=0} 16:57:31.755749 --- ParallelTerminationStrategy: PID 13940 died... 16:57:31.815089 run osmo-stp_10.42.42.5(pid=13940): Terminated: ok {rc=0} 16:57:31.826201 --- ParallelTerminationStrategy: PID 13941 died... 16:57:31.883328 run pcap-recorder_any(filters='host 10.42.42.6')(pid=13941): Terminated: ok {rc=0} 16:57:31.893941 --- ParallelTerminationStrategy: PID 13947 died... 16:57:31.950618 run osmo-ggsn_10.42.42.6(pid=13947): Terminated: ok {rc=0} 16:57:31.960892 --- ParallelTerminationStrategy: PID 13969 died... 16:57:32.017585 run pcap-recorder_any(filters='host 10.42.42.7')(pid=13969): Terminated: ok {rc=0} 16:57:32.027904 --- ParallelTerminationStrategy: PID 13971 died... 16:57:32.084727 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=13971): Terminated: ok {rc=0} 16:57:32.095009 --- ParallelTerminationStrategy: PID 13972 died... 16:57:32.151781 run osmo-msc_10.42.42.8(pid=13972): Terminated: ok {rc=0} 16:57:32.162128 --- ParallelTerminationStrategy: PID 13973 died... 16:57:32.219814 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13973): Terminated: ok {rc=0} 16:57:32.230703 --- ParallelTerminationStrategy: PID 13974 died... 16:57:32.288537 run osmo-mgw_10.42.42.3(pid=13974): Terminated {rc=15} 16:57:32.299264 --- ParallelTerminationStrategy: PID 13975 died... 16:57:32.357124 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13975): Terminated: ok {rc=0} 16:57:32.367951 --- ParallelTerminationStrategy: PID 13976 died... 16:57:32.425815 run osmo-mgw_10.42.42.4(pid=13976): Terminated {rc=15} 16:57:32.436645 --- ParallelTerminationStrategy: PID 13977 died... 16:57:32.494446 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=13977): Terminated: ok {rc=0} 16:57:32.505341 --- ParallelTerminationStrategy: PID 13978 died... 16:57:32.563580 run osmo-bsc_10.42.42.9(pid=13978): Terminated: ok {rc=0} 16:57:33.048892 --- ParallelTerminationStrategy: PID 13970 died... 16:57:33.103947 run osmo-sgsn_10.42.42.7(pid=13970): Terminated: ok {rc=0} 16:57:33.192332 bus /gobi_6: Setting Powered False 178.02gprs_sysmo2023-03-07T16:54:37