Failed
gprs:sysmo.cs_paging_gprs_active.py (from gprs_sysmo)
Failing for the past 8 builds
(Since Failed )
Stacktrace
rm-remote-dir(pid=9791): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9791)]
Standard Output
---------------------------------------------- trial-2070 gprs:sysmo cs_paging_gprs_active.py ---------------------------------------------- 15:42:28.104920 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:42:28.115669 tst gprs:sysmo: Using 1 x bts (candidates: 1) 15:42:28.242706 tst cs_paging_gprs_active.py: using LAC 41709 15:42:28.374016 tst cs_paging_gprs_active.py: using RAC 144 15:42:28.503187 tst cs_paging_gprs_active.py: using CellId 41709 15:42:28.626929 tst cs_paging_gprs_active.py: using BVCI 41710 15:42:28.651618 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:42:28.662556 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:42:28.673387 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:42:28.685162 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:42:28.695757 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:42:28.706691 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:42:28.717822 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:42:28.728746 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:42:28.739923 tst gprs:sysmo: Using 1 x modem (candidates: 4) 15:42:28.857642 tst gprs:sysmo: Using 1 x modem (candidates: 4) 15:42:29.133415 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003... 15:42:29.259168 run iperf3-srv_10.42.42.10: Starting iperf3-srv 15:42:29.568601 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9742): Launched 15:42:29.825569 run iperf3-srv_10.42.42.10(pid=9743): Launched 15:42:30.021532 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004... 15:42:30.156438 run iperf3-srv_10.42.42.10: Starting iperf3-srv 15:42:30.488575 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9744): Launched 15:42:30.764976 run iperf3-srv_10.42.42.10(pid=9745): Launched 15:42:30.962471 tst cs_paging_gprs_active.py:38: start network... 15:42:31.097057 run osmo-hlr_10.42.42.2: Starting osmo-hlr 15:42:31.603844 run create_hlr_db(pid=9746): Launched 15:42:31.790247 bus /sierra_2: Setting Powered False 15:42:32.811542 run create_hlr_db(pid=9746): Terminated: ok {rc=0} 15:42:33.213247 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9748): Launched 15:42:33.514054 run osmo-hlr_10.42.42.2(pid=9749): Launched 15:42:33.623368 run osmo-stp_10.42.42.5: Starting osmo-stp 15:42:34.142138 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9750): Launched 15:42:34.442215 run osmo-stp_10.42.42.5(pid=9751): Launched 15:42:34.549743 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 15:42:36.059100 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9752): Launched 15:42:36.171908 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 15:42:36.470633 run patchelf(pid=9753): Launched 15:42:37.650886 run patchelf(pid=9753): Terminated: ok {rc=0} 15:42:37.760889 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 15:42:38.031924 run osmo-gsm-tester_setcap_net_admin.sh(pid=9754): Launched 15:42:39.217490 run osmo-gsm-tester_setcap_net_admin.sh(pid=9754): Terminated: ok {rc=0} 15:42:39.521844 run osmo-ggsn_10.42.42.6(pid=9757): Launched 15:42:39.786371 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 15:42:44.485835 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9781): Launched 15:42:44.786863 run osmo-sgsn_10.42.42.7(pid=9782): Launched 15:42:44.897466 run osmo-msc_10.42.42.8: Starting osmo-msc 15:42:45.449033 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9783): Launched 15:42:45.749360 run osmo-msc_10.42.42.8(pid=9784): Launched 15:42:48.909681 run osmo-mgw_10.42.42.3: Starting osmo-mgw 15:42:49.450309 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9785): Launched 15:42:49.754257 run osmo-mgw_10.42.42.3(pid=9786): Launched 15:42:49.881295 run osmo-mgw_10.42.42.4: Starting osmo-mgw 15:42:50.394484 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9787): Launched 15:42:50.696130 run osmo-mgw_10.42.42.4(pid=9788): Launched 15:42:50.823268 run osmo-bsc_10.42.42.9: Starting osmo-bsc 15:42:51.549004 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9789): Launched 15:42:51.976872 run osmo-bsc_10.42.42.9(pid=9790): Launched 15:42:58.286774 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 15:42:58.861279 run rm-remote-dir(pid=9791): Launched 15:43:02.064880 run rm-remote-dir(pid=9791): ERR: Terminated: ERROR {rc=255} [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9791)] 15:43:02.207258 run rm-remote-dir(pid=9791): stdout: | (launched: 2022-12-23_15:42:58.695771) 15:43:02.352796 run rm-remote-dir(pid=9791): stderr: | (launched: 2022-12-23_15:42:58.747034) | ssh: connect to host 10.42.42.114 port 22: No route to host 15:43:02.412320 tst cs_paging_gprs_active.py:38: ERR: Error: rm-remote-dir(pid=9791): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9791)] [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38] 15:43:02.420934 tst cs_paging_gprs_active.py:38: Test FAILED (34.4 sec) 15:43:02.458426 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9742): Terminating (SIGTERM) 15:43:02.500361 run iperf3-srv_10.42.42.10(pid=9743): Terminating (SIGTERM) 15:43:02.542441 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9744): Terminating (SIGTERM) 15:43:02.584565 run iperf3-srv_10.42.42.10(pid=9745): Terminating (SIGTERM) 15:43:02.626573 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9748): Terminating (SIGTERM) 15:43:02.668427 run osmo-hlr_10.42.42.2(pid=9749): Terminating (SIGTERM) 15:43:02.710679 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9750): Terminating (SIGTERM) 15:43:02.753955 run osmo-stp_10.42.42.5(pid=9751): Terminating (SIGTERM) 15:43:02.830210 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9752): Terminating (SIGTERM) 15:43:02.890447 run osmo-ggsn_10.42.42.6(pid=9757): Terminating (SIGTERM) 15:43:02.938612 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9781): Terminating (SIGTERM) 15:43:02.983410 run osmo-sgsn_10.42.42.7(pid=9782): Terminating (SIGTERM) 15:43:03.026020 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9783): Terminating (SIGTERM) 15:43:03.067163 run osmo-msc_10.42.42.8(pid=9784): Terminating (SIGTERM) 15:43:03.109152 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9785): Terminating (SIGTERM) 15:43:03.150919 run osmo-mgw_10.42.42.3(pid=9786): Terminating (SIGTERM) 15:43:03.191071 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9787): Terminating (SIGTERM) 15:43:03.230262 run osmo-mgw_10.42.42.4(pid=9788): Terminating (SIGTERM) 15:43:03.270126 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9789): Terminating (SIGTERM) 15:43:03.310082 run osmo-bsc_10.42.42.9(pid=9790): Terminating (SIGTERM) 15:43:03.319144 --- ParallelTerminationStrategy: PID 9742 died... 15:43:03.372103 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9742): Terminated: ok {rc=0} 15:43:03.381575 --- ParallelTerminationStrategy: PID 9743 died... 15:43:03.433826 run iperf3-srv_10.42.42.10(pid=9743): Terminated {rc=256} 15:43:03.443171 --- ParallelTerminationStrategy: PID 9744 died... 15:43:03.496055 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9744): Terminated: ok {rc=0} 15:43:03.505484 --- ParallelTerminationStrategy: PID 9745 died... 15:43:03.557790 run iperf3-srv_10.42.42.10(pid=9745): Terminated {rc=256} 15:43:03.567161 --- ParallelTerminationStrategy: PID 9748 died... 15:43:03.619969 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9748): Terminated: ok {rc=0} 15:43:03.629429 --- ParallelTerminationStrategy: PID 9749 died... 15:43:03.681749 run osmo-hlr_10.42.42.2(pid=9749): Terminated: ok {rc=0} 15:43:03.691086 --- ParallelTerminationStrategy: PID 9750 died... 15:43:03.743541 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9750): Terminated: ok {rc=0} 15:43:03.752987 --- ParallelTerminationStrategy: PID 9751 died... 15:43:03.805236 run osmo-stp_10.42.42.5(pid=9751): Terminated: ok {rc=0} 15:43:03.814622 --- ParallelTerminationStrategy: PID 9752 died... 15:43:03.869382 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9752): Terminated: ok {rc=0} 15:43:03.879963 --- ParallelTerminationStrategy: PID 9757 died... 15:43:03.934633 run osmo-ggsn_10.42.42.6(pid=9757): Terminated: ok {rc=0} 15:43:03.944983 --- ParallelTerminationStrategy: PID 9781 died... 15:43:03.998050 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9781): Terminated: ok {rc=0} 15:43:04.007700 --- ParallelTerminationStrategy: PID 9783 died... 15:43:04.062267 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9783): Terminated: ok {rc=0} 15:43:04.073009 --- ParallelTerminationStrategy: PID 9784 died... 15:43:04.155559 run osmo-msc_10.42.42.8(pid=9784): Terminated: ok {rc=0} 15:43:04.182590 --- ParallelTerminationStrategy: PID 9785 died... 15:43:04.304368 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9785): Terminated: ok {rc=0} 15:43:04.316309 --- ParallelTerminationStrategy: PID 9786 died... 15:43:04.372319 run osmo-mgw_10.42.42.3(pid=9786): Terminated {rc=15} 15:43:04.382675 --- ParallelTerminationStrategy: PID 9787 died... 15:43:04.437984 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9787): Terminated: ok {rc=0} 15:43:04.448070 --- ParallelTerminationStrategy: PID 9788 died... 15:43:04.500934 run osmo-mgw_10.42.42.4(pid=9788): Terminated {rc=15} 15:43:04.510483 --- ParallelTerminationStrategy: PID 9782 died... 15:43:04.562897 run osmo-sgsn_10.42.42.7(pid=9782): Terminated: ok {rc=0} 15:43:04.572471 --- ParallelTerminationStrategy: PID 9789 died... 15:43:04.625018 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9789): Terminated: ok {rc=0} 15:43:04.634545 --- ParallelTerminationStrategy: PID 9790 died... 15:43:04.686923 run osmo-bsc_10.42.42.9(pid=9790): Terminated: ok {rc=0} 15:43:04.769775 bus /gobi_0: Setting Powered False 15:43:05.889065 bus /gobi_0: Setting Powered False 15:43:07.029953 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module> setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel bts.start() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start rem_host.recreate_remote_dir(self.remote_inst) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir self.rm_remote_dir(remote_dir) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str)) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync proc.launch_sync() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync raise self.RunError('launch_sync()') osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=9791): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2070↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9791)]