Failed
handover_2G:trx-sysmocell5000+bts1-trx-umtrx.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-umtrx)
Stacktrace
handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]
Standard Output
---------------------------------------------- trial-162 handover_2G:trx-sysmocell5000+bts1-trx-umtrx handover.py ---------------------------------------------- 11:07:42.845857 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 11:07:42.860451 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2) 11:07:43.021940 tst handover.py: using LAC 16297 11:07:43.153402 tst handover.py: using RAC 232 11:07:43.286201 tst handover.py: using CellId 16296 11:07:43.423288 tst handover.py: using BVCI 2975 11:07:43.449297 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2) 11:07:43.600047 tst handover.py: using LAC 16298 11:07:43.730301 tst handover.py: using RAC 233 11:07:43.860578 tst handover.py: using CellId 16297 11:07:43.990239 tst handover.py: using BVCI 2976 11:07:44.016051 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 11:07:44.030093 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 11:07:44.043775 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 11:07:44.057633 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 11:07:44.071253 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 11:07:44.084945 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2) 11:07:44.219304 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2) 11:07:44.448371 run osmo-hlr_10.42.42.2: Starting osmo-hlr 11:07:44.968537 run create_hlr_db(pid=6601): Launched 11:07:45.119953 bus /sierra_2: Setting Powered False 11:07:46.204688 run create_hlr_db(pid=6601): Terminated: ok {rc=0} 11:07:46.641788 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6603): Launched 11:07:46.998658 run osmo-hlr_10.42.42.2(pid=6604): Launched 11:07:47.127716 run osmo-stp_10.42.42.5: Starting osmo-stp 11:07:47.711693 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6605): Launched 11:07:48.064866 run osmo-stp_10.42.42.5(pid=6606): Launched 11:07:48.185707 run osmo-msc_10.42.42.6: Starting osmo-msc 11:07:48.778927 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6607): Launched 11:07:49.127052 run osmo-msc_10.42.42.6(pid=6608): Launched 11:07:49.256822 run osmo-mgw_10.42.42.3: Starting osmo-mgw 11:07:49.830436 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6609): Launched 11:07:50.189784 run osmo-mgw_10.42.42.3(pid=6610): Launched 11:07:50.317766 run osmo-mgw_10.42.42.4: Starting osmo-mgw 11:07:50.892501 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6611): Launched 11:07:51.249304 run osmo-mgw_10.42.42.4(pid=6612): Launched 11:07:51.376595 run osmo-bsc_10.42.42.7: Starting osmo-bsc 11:07:52.157628 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6613): Launched 11:07:52.506213 run osmo-bsc_10.42.42.7(pid=6614): Launched 11:07:56.231902 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7 11:07:58.580850 run ssh_sc5_ccli(pid=6615): Launched 11:08:17.816764 run ssh_sc5_ccli(pid=6615): ERR: Terminated: ERROR {rc=255} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=6615)] 11:08:17.965961 run ssh_sc5_ccli(pid=6615): stdout: | (launched: 2020-12-07_11:07:58.401624) 11:08:18.116609 run ssh_sc5_ccli(pid=6615): stderr: | (launched: 2020-12-07_11:07:58.457213) | ssh: connect to host 10.42.42.112 port 22: No route to host 11:08:18.244913 run sc5-trx: Configuring SC5 TRX failed, retrying 9 more times 11:08:23.612757 run ssh_sc5_ccli(pid=6616): Launched 11:08:26.834881 run ssh_sc5_ccli(pid=6616): Terminated: ok {rc=0} 11:08:26.966431 run osmo-bts-trx: Waiting for sc5-trx to start up... 11:08:28.277964 run osmo-bts-trx(pid=6617): Launched 11:13:29.064769 tst handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-umtrx/handover.py/01_bts0_started: FAIL (332.9s) 11:13:29.073254 tst handover.py:43: ERR: Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43] [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43] 11:13:29.081576 tst handover.py:43: Test FAILED (346.2 sec) 11:13:29.126079 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6603): Terminating (SIGTERM) 11:13:29.177370 run osmo-hlr_10.42.42.2(pid=6604): Terminating (SIGTERM) 11:13:29.230519 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6605): Terminating (SIGTERM) 11:13:29.285911 run osmo-stp_10.42.42.5(pid=6606): Terminating (SIGTERM) 11:13:29.339165 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6607): Terminating (SIGTERM) 11:13:29.395384 run osmo-msc_10.42.42.6(pid=6608): Terminating (SIGTERM) 11:13:29.446843 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6609): Terminating (SIGTERM) 11:13:29.498133 run osmo-mgw_10.42.42.3(pid=6610): Terminating (SIGTERM) 11:13:29.549290 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6611): Terminating (SIGTERM) 11:13:29.600092 run osmo-mgw_10.42.42.4(pid=6612): Terminating (SIGTERM) 11:13:29.651193 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6613): Terminating (SIGTERM) 11:13:29.702388 run osmo-bsc_10.42.42.7(pid=6614): Terminating (SIGTERM) 11:13:29.753415 run osmo-bts-trx(pid=6617): Terminating (SIGTERM) 11:13:29.765046 --- ParallelTerminationStrategy: PID 6603 died... 11:13:29.832193 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6603): Terminated: ok {rc=0} 11:13:29.844296 --- ParallelTerminationStrategy: PID 6604 died... 11:13:29.911215 run osmo-hlr_10.42.42.2(pid=6604): Terminated: ok {rc=0} 11:13:29.923259 --- ParallelTerminationStrategy: PID 6605 died... 11:13:29.990108 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6605): Terminated: ok {rc=0} 11:13:30.002100 --- ParallelTerminationStrategy: PID 6606 died... 11:13:30.068937 run osmo-stp_10.42.42.5(pid=6606): Terminated: ok {rc=0} 11:13:30.081426 --- ParallelTerminationStrategy: PID 6607 died... 11:13:30.148265 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6607): Terminated: ok {rc=0} 11:13:30.160414 --- ParallelTerminationStrategy: PID 6609 died... 11:13:30.227293 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6609): Terminated: ok {rc=0} 11:13:30.240445 --- ParallelTerminationStrategy: PID 6610 died... 11:13:30.307191 run osmo-mgw_10.42.42.3(pid=6610): Terminated {rc=15} 11:13:30.319338 --- ParallelTerminationStrategy: PID 6611 died... 11:13:30.386809 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6611): Terminated: ok {rc=0} 11:13:30.398962 --- ParallelTerminationStrategy: PID 6612 died... 11:13:30.466018 run osmo-mgw_10.42.42.4(pid=6612): Terminated {rc=15} 11:13:30.478100 --- ParallelTerminationStrategy: PID 6613 died... 11:13:30.545250 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6613): Terminated: ok {rc=0} 11:13:32.574128 --- ParallelTerminationStrategy: PID 6608 died... 11:13:32.643346 run osmo-msc_10.42.42.6(pid=6608): Terminated: ok {rc=0} 11:13:33.322395 --- ParallelTerminationStrategy: PID 6614 died... 11:13:33.392035 run osmo-bsc_10.42.42.7(pid=6614): Terminated: ok {rc=0} 11:13:33.404174 --- ParallelTerminationStrategy: PID 6617 died... 11:13:33.471201 run osmo-bts-trx(pid=6617): Terminated: ok {rc=0} 11:13:33.574393 bus /sierra_3: Setting Powered False 11:13:34.698625 bus /sierra_3: Setting Powered False 11:13:35.830056 bus /sierra_2: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 673, in exec_module File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module> wait(bsc.bts_is_connected, bts0) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]