Failed
handover_2G:trx-sysmocell5000+bts1-sysmo.handover.py (from handover_2G_trx-sysmocell5000+bts1-sysmo)
Stacktrace
handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]
Standard Output
---------------------------------------------- trial-162 handover_2G:trx-sysmocell5000+bts1-sysmo handover.py ---------------------------------------------- 10:55:51.991498 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 10:55:52.005338 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2) 10:55:52.171392 tst handover.py: using LAC 16293 10:55:52.300839 tst handover.py: using RAC 228 10:55:52.431113 tst handover.py: using CellId 16292 10:55:52.563111 tst handover.py: using BVCI 2971 10:55:52.588792 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2) 10:55:52.730062 tst handover.py: using LAC 16294 10:55:52.862844 tst handover.py: using RAC 229 10:55:52.992466 tst handover.py: using CellId 16293 10:55:53.121647 tst handover.py: using BVCI 2972 10:55:53.147094 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 10:55:53.160955 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 10:55:53.174601 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 10:55:53.188255 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 10:55:53.201934 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 10:55:53.215438 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2) 10:55:53.347226 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2) 10:55:53.572404 run osmo-hlr_10.42.42.2: Starting osmo-hlr 10:55:54.090419 run create_hlr_db(pid=6513): Launched 10:55:54.240491 bus /sierra_2: Setting Powered False 10:55:55.323171 run create_hlr_db(pid=6513): Terminated: ok {rc=0} 10:55:55.750209 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6515): Launched 10:55:56.093749 run osmo-hlr_10.42.42.2(pid=6516): Launched 10:55:56.221843 run osmo-stp_10.42.42.5: Starting osmo-stp 10:55:56.797125 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6517): Launched 10:55:57.141803 run osmo-stp_10.42.42.5(pid=6518): Launched 10:55:57.267462 run osmo-msc_10.42.42.6: Starting osmo-msc 10:55:57.874810 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6519): Launched 10:55:58.223169 run osmo-msc_10.42.42.6(pid=6520): Launched 10:55:58.357728 run osmo-mgw_10.42.42.3: Starting osmo-mgw 10:55:58.949770 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6521): Launched 10:55:59.292868 run osmo-mgw_10.42.42.3(pid=6522): Launched 10:55:59.418896 run osmo-mgw_10.42.42.4: Starting osmo-mgw 10:55:59.982078 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6523): Launched 10:56:00.325020 run osmo-mgw_10.42.42.4(pid=6524): Launched 10:56:00.450509 run osmo-bsc_10.42.42.7: Starting osmo-bsc 10:56:01.221036 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6525): Launched 10:56:01.560195 run osmo-bsc_10.42.42.7(pid=6526): Launched 10:56:05.272585 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7 10:56:07.612663 run ssh_sc5_ccli(pid=6527): Launched 10:56:26.855970 run ssh_sc5_ccli(pid=6527): ERR: Terminated: ERROR {rc=255} [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=6527)] 10:56:27.010774 run ssh_sc5_ccli(pid=6527): stdout: | (launched: 2020-12-07_10:56:07.430962) 10:56:27.166710 run ssh_sc5_ccli(pid=6527): stderr: | (launched: 2020-12-07_10:56:07.485766) | ssh: connect to host 10.42.42.112 port 22: No route to host 10:56:27.299144 run sc5-trx: Configuring SC5 TRX failed, retrying 9 more times 10:56:32.664147 run ssh_sc5_ccli(pid=6528): Launched 10:56:35.876788 run ssh_sc5_ccli(pid=6528): Terminated: ok {rc=0} 10:56:36.001349 run osmo-bts-trx: Waiting for sc5-trx to start up... 10:56:37.304440 run osmo-bts-trx(pid=6529): Launched 11:01:38.063798 tst handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-sysmo/handover.py/01_bts0_started: FAIL (332.9s) 11:01:38.072189 tst handover.py:43: ERR: Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43] [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43] 11:01:38.080299 tst handover.py:43: Test FAILED (346.1 sec) 11:01:38.125059 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6515): Terminating (SIGTERM) 11:01:38.175991 run osmo-hlr_10.42.42.2(pid=6516): Terminating (SIGTERM) 11:01:38.229189 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6517): Terminating (SIGTERM) 11:01:38.283995 run osmo-stp_10.42.42.5(pid=6518): Terminating (SIGTERM) 11:01:38.337641 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6519): Terminating (SIGTERM) 11:01:38.393582 run osmo-msc_10.42.42.6(pid=6520): Terminating (SIGTERM) 11:01:38.444937 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6521): Terminating (SIGTERM) 11:01:38.495688 run osmo-mgw_10.42.42.3(pid=6522): Terminating (SIGTERM) 11:01:38.546498 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6523): Terminating (SIGTERM) 11:01:38.598254 run osmo-mgw_10.42.42.4(pid=6524): Terminating (SIGTERM) 11:01:38.649593 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6525): Terminating (SIGTERM) 11:01:38.700194 run osmo-bsc_10.42.42.7(pid=6526): Terminating (SIGTERM) 11:01:38.751063 run osmo-bts-trx(pid=6529): Terminating (SIGTERM) 11:01:38.763770 --- ParallelTerminationStrategy: PID 6515 died... 11:01:38.831279 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6515): Terminated: ok {rc=0} 11:01:38.843429 --- ParallelTerminationStrategy: PID 6516 died... 11:01:38.910186 run osmo-hlr_10.42.42.2(pid=6516): Terminated: ok {rc=0} 11:01:38.922162 --- ParallelTerminationStrategy: PID 6517 died... 11:01:38.989032 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6517): Terminated: ok {rc=0} 11:01:39.001161 --- ParallelTerminationStrategy: PID 6518 died... 11:01:39.068054 run osmo-stp_10.42.42.5(pid=6518): Terminated: ok {rc=0} 11:01:39.080105 --- ParallelTerminationStrategy: PID 6519 died... 11:01:39.146991 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6519): Terminated: ok {rc=0} 11:01:39.159052 --- ParallelTerminationStrategy: PID 6521 died... 11:01:39.226138 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6521): Terminated: ok {rc=0} 11:01:39.238256 --- ParallelTerminationStrategy: PID 6522 died... 11:01:39.305415 run osmo-mgw_10.42.42.3(pid=6522): Terminated {rc=15} 11:01:39.317857 --- ParallelTerminationStrategy: PID 6523 died... 11:01:39.386047 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6523): Terminated: ok {rc=0} 11:01:39.398209 --- ParallelTerminationStrategy: PID 6524 died... 11:01:39.465231 run osmo-mgw_10.42.42.4(pid=6524): Terminated {rc=15} 11:01:39.477440 --- ParallelTerminationStrategy: PID 6525 died... 11:01:39.544580 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6525): Terminated: ok {rc=0} 11:01:41.573641 --- ParallelTerminationStrategy: PID 6520 died... 11:01:41.643117 run osmo-msc_10.42.42.6(pid=6520): Terminated: ok {rc=0} 11:01:42.322148 --- ParallelTerminationStrategy: PID 6526 died... 11:01:42.391289 run osmo-bsc_10.42.42.7(pid=6526): Terminated: ok {rc=0} 11:01:42.403468 --- ParallelTerminationStrategy: PID 6529 died... 11:01:42.470286 run osmo-bts-trx(pid=6529): Terminated: ok {rc=0} 11:01:42.573768 bus /sierra_3: Setting Powered False 11:01:43.699233 bus /sierra_3: Setting Powered False 11:01:44.829739 bus /sierra_2: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 673, in exec_module File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module> wait(bsc.bts_is_connected, bts0) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]