Failed
gprs:sysmo.cs_paging_gprs_active.py (from gprs_sysmo)
Failing for the past 9 builds
(Since Failed )
Stacktrace
rm-remote-dir(pid=31507): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31507)]
Standard Output
---------------------------------------------- trial-2071 gprs:sysmo cs_paging_gprs_active.py ---------------------------------------------- 21:58:45.595324 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 21:58:45.606504 tst gprs:sysmo: Using 1 x bts (candidates: 1) 21:58:45.742867 tst cs_paging_gprs_active.py: using LAC 41716 21:58:45.954066 tst cs_paging_gprs_active.py: using RAC 151 21:58:46.074657 tst cs_paging_gprs_active.py: using CellId 41716 21:58:46.194317 tst cs_paging_gprs_active.py: using BVCI 41717 21:58:46.217870 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 21:58:46.228652 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 21:58:46.239259 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 21:58:46.250192 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 21:58:46.260949 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 21:58:46.271664 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 21:58:46.282673 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 21:58:46.293554 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 21:58:46.304711 tst gprs:sysmo: Using 1 x modem (candidates: 4) 21:58:46.427159 tst gprs:sysmo: Using 1 x modem (candidates: 4) 21:58:46.707444 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003... 21:58:46.835506 run iperf3-srv_10.42.42.10: Starting iperf3-srv 21:58:47.153247 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31461): Launched 21:58:47.419627 run iperf3-srv_10.42.42.10(pid=31462): Launched 21:58:47.606262 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004... 21:58:47.733651 run iperf3-srv_10.42.42.10: Starting iperf3-srv 21:58:48.050248 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31463): Launched 21:58:48.313866 run iperf3-srv_10.42.42.10(pid=31464): Launched 21:58:48.500494 tst cs_paging_gprs_active.py:38: start network... 21:58:48.627864 run osmo-hlr_10.42.42.2: Starting osmo-hlr 21:58:49.108280 run create_hlr_db(pid=31465): Launched 21:58:49.285322 bus /sierra_2: Setting Powered False 21:58:50.305549 run create_hlr_db(pid=31465): Terminated: ok {rc=0} 21:58:50.712977 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31467): Launched 21:58:51.160948 run osmo-hlr_10.42.42.2(pid=31468): Launched 21:58:51.359295 run osmo-stp_10.42.42.5: Starting osmo-stp 21:58:51.938524 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31469): Launched 21:58:52.276016 run osmo-stp_10.42.42.5(pid=31470): Launched 21:58:52.389307 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 21:58:53.870859 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31471): Launched 21:58:53.979286 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 21:58:54.247505 run patchelf(pid=31472): Launched 21:58:55.430382 run patchelf(pid=31472): Terminated: ok {rc=0} 21:58:55.542608 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 21:58:55.831077 run osmo-gsm-tester_setcap_net_admin.sh(pid=31473): Launched 21:58:57.027355 run osmo-gsm-tester_setcap_net_admin.sh(pid=31473): Terminated: ok {rc=0} 21:58:57.358411 run osmo-ggsn_10.42.42.6(pid=31476): Launched 21:58:57.593148 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 21:59:02.485596 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31497): Launched 21:59:02.792607 run osmo-sgsn_10.42.42.7(pid=31498): Launched 21:59:02.907373 run osmo-msc_10.42.42.8: Starting osmo-msc 21:59:03.469605 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31499): Launched 21:59:03.777268 run osmo-msc_10.42.42.8(pid=31500): Launched 21:59:06.967481 run osmo-mgw_10.42.42.3: Starting osmo-mgw 21:59:07.518210 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31501): Launched 21:59:07.826571 run osmo-mgw_10.42.42.3(pid=31502): Launched 21:59:07.938563 run osmo-mgw_10.42.42.4: Starting osmo-mgw 21:59:08.463106 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31503): Launched 21:59:08.769931 run osmo-mgw_10.42.42.4(pid=31504): Launched 21:59:08.895961 run osmo-bsc_10.42.42.9: Starting osmo-bsc 21:59:09.587591 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31505): Launched 21:59:09.916084 run osmo-bsc_10.42.42.9(pid=31506): Launched 21:59:16.196893 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 21:59:16.779525 run rm-remote-dir(pid=31507): Launched 21:59:19.984803 run rm-remote-dir(pid=31507): ERR: Terminated: ERROR {rc=255} [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31507)] 21:59:20.128992 run rm-remote-dir(pid=31507): stdout: | (launched: 2023-01-03_21:59:16.612399) 21:59:20.276520 run rm-remote-dir(pid=31507): stderr: | (launched: 2023-01-03_21:59:16.664320) | ssh: connect to host 10.42.42.114 port 22: No route to host 21:59:20.336593 tst cs_paging_gprs_active.py:38: ERR: Error: rm-remote-dir(pid=31507): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31507)] [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38] 21:59:20.345255 tst cs_paging_gprs_active.py:38: Test FAILED (34.8 sec) 21:59:20.382439 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31461): Terminating (SIGTERM) 21:59:20.425066 run iperf3-srv_10.42.42.10(pid=31462): Terminating (SIGTERM) 21:59:20.467576 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31463): Terminating (SIGTERM) 21:59:20.509940 run iperf3-srv_10.42.42.10(pid=31464): Terminating (SIGTERM) 21:59:20.552443 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31467): Terminating (SIGTERM) 21:59:20.594585 run osmo-hlr_10.42.42.2(pid=31468): Terminating (SIGTERM) 21:59:20.638070 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31469): Terminating (SIGTERM) 21:59:20.683904 run osmo-stp_10.42.42.5(pid=31470): Terminating (SIGTERM) 21:59:20.797299 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31471): Terminating (SIGTERM) 21:59:20.882783 run osmo-ggsn_10.42.42.6(pid=31476): Terminating (SIGTERM) 21:59:20.979009 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31497): Terminating (SIGTERM) 21:59:21.080542 run osmo-sgsn_10.42.42.7(pid=31498): Terminating (SIGTERM) 21:59:21.146060 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31499): Terminating (SIGTERM) 21:59:21.239544 run osmo-msc_10.42.42.8(pid=31500): Terminating (SIGTERM) 21:59:21.312819 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31501): Terminating (SIGTERM) 21:59:21.358874 run osmo-mgw_10.42.42.3(pid=31502): Terminating (SIGTERM) 21:59:21.404331 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31503): Terminating (SIGTERM) 21:59:21.450717 run osmo-mgw_10.42.42.4(pid=31504): Terminating (SIGTERM) 21:59:21.495026 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31505): Terminating (SIGTERM) 21:59:21.539213 run osmo-bsc_10.42.42.9(pid=31506): Terminating (SIGTERM) 21:59:21.549346 --- ParallelTerminationStrategy: PID 31461 died... 21:59:21.607769 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31461): Terminated: ok {rc=0} 21:59:21.618505 --- ParallelTerminationStrategy: PID 31462 died... 21:59:21.676624 run iperf3-srv_10.42.42.10(pid=31462): Terminated {rc=256} 21:59:21.687321 --- ParallelTerminationStrategy: PID 31463 died... 21:59:21.745678 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31463): Terminated: ok {rc=0} 21:59:21.756471 --- ParallelTerminationStrategy: PID 31464 died... 21:59:21.814421 run iperf3-srv_10.42.42.10(pid=31464): Terminated {rc=256} 21:59:21.825152 --- ParallelTerminationStrategy: PID 31467 died... 21:59:21.884446 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31467): Terminated: ok {rc=0} 21:59:21.895343 --- ParallelTerminationStrategy: PID 31468 died... 21:59:21.953880 run osmo-hlr_10.42.42.2(pid=31468): Terminated: ok {rc=0} 21:59:21.964701 --- ParallelTerminationStrategy: PID 31469 died... 21:59:22.118935 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31469): Terminated: ok {rc=0} 21:59:22.160943 --- ParallelTerminationStrategy: PID 31470 died... 21:59:22.360199 run osmo-stp_10.42.42.5(pid=31470): Terminated: ok {rc=0} 21:59:22.370652 --- ParallelTerminationStrategy: PID 31471 died... 21:59:22.449727 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31471): Terminated: ok {rc=0} 21:59:22.468688 --- ParallelTerminationStrategy: PID 31476 died... 21:59:22.539973 run osmo-ggsn_10.42.42.6(pid=31476): Terminated: ok {rc=0} 21:59:22.550722 --- ParallelTerminationStrategy: PID 31497 died... 21:59:22.610347 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31497): Terminated: ok {rc=0} 21:59:22.621167 --- ParallelTerminationStrategy: PID 31499 died... 21:59:22.680358 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31499): Terminated: ok {rc=0} 21:59:22.691484 --- ParallelTerminationStrategy: PID 31500 died... 21:59:22.748027 run osmo-msc_10.42.42.8(pid=31500): Terminated: ok {rc=0} 21:59:22.758214 --- ParallelTerminationStrategy: PID 31498 died... 21:59:22.814520 run osmo-sgsn_10.42.42.7(pid=31498): Terminated: ok {rc=0} 21:59:22.825136 --- ParallelTerminationStrategy: PID 31501 died... 21:59:22.881818 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31501): Terminated: ok {rc=0} 21:59:22.892032 --- ParallelTerminationStrategy: PID 31502 died... 21:59:22.948299 run osmo-mgw_10.42.42.3(pid=31502): Terminated {rc=15} 21:59:22.958473 --- ParallelTerminationStrategy: PID 31503 died... 21:59:23.014932 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31503): Terminated: ok {rc=0} 21:59:23.025577 --- ParallelTerminationStrategy: PID 31504 died... 21:59:23.082097 run osmo-mgw_10.42.42.4(pid=31504): Terminated {rc=15} 21:59:23.092399 --- ParallelTerminationStrategy: PID 31505 died... 21:59:23.148753 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31505): Terminated: ok {rc=0} 21:59:23.159032 --- ParallelTerminationStrategy: PID 31506 died... 21:59:23.215415 run osmo-bsc_10.42.42.9(pid=31506): Terminated: ok {rc=0} 21:59:23.303956 bus /gobi_0: Setting Powered False 21:59:24.412482 bus /gobi_0: Setting Powered False 21:59:25.521964 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module> setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel bts.start() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start rem_host.recreate_remote_dir(self.remote_inst) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir self.rm_remote_dir(remote_dir) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str)) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync proc.launch_sync() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync raise self.RunError('launch_sync()') osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=31507): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31507)]