Failed
handover_2G:trx-sysmocell5000+bts1-trx-b200.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-b200)
Stacktrace
handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]
Standard Output
---------------------------------------------- trial-159 handover_2G:trx-sysmocell5000+bts1-trx-b200 handover.py ---------------------------------------------- 01:12:47.353215 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 01:12:47.367498 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2) 01:12:47.536443 tst handover.py: using LAC 16055 01:12:47.674265 tst handover.py: using RAC 245 01:12:47.809087 tst handover.py: using CellId 16054 01:12:47.938741 tst handover.py: using BVCI 2733 01:12:47.964715 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2) 01:12:48.123301 tst handover.py: using LAC 16056 01:12:48.262246 tst handover.py: using RAC 246 01:12:48.400140 tst handover.py: using CellId 16055 01:12:48.534003 tst handover.py: using BVCI 2734 01:12:48.560123 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 01:12:48.574074 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 01:12:48.587756 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 01:12:48.601410 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 01:12:48.615307 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 01:12:48.629032 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2) 01:12:48.756799 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2) 01:12:48.991211 run osmo-hlr_10.42.42.2: Starting osmo-hlr 01:12:49.534540 run create_hlr_db(pid=15579): Launched 01:12:49.685466 bus /sierra_2: Setting Powered False 01:12:50.770728 run create_hlr_db(pid=15579): Terminated: ok {rc=0} 01:12:51.202180 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15581): Launched 01:12:51.555533 run osmo-hlr_10.42.42.2(pid=15582): Launched 01:12:51.685943 run osmo-stp_10.42.42.5: Starting osmo-stp 01:12:52.274966 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15583): Launched 01:12:52.619286 run osmo-stp_10.42.42.5(pid=15584): Launched 01:12:52.742074 run osmo-msc_10.42.42.6: Starting osmo-msc 01:12:53.342118 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15585): Launched 01:12:53.694117 run osmo-msc_10.42.42.6(pid=15586): Launched 01:12:53.831594 run osmo-mgw_10.42.42.3: Starting osmo-mgw 01:12:54.442939 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15587): Launched 01:12:54.786695 run osmo-mgw_10.42.42.3(pid=15588): Launched 01:12:54.908602 run osmo-mgw_10.42.42.4: Starting osmo-mgw 01:12:55.469354 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15589): Launched 01:12:55.832485 run osmo-mgw_10.42.42.4(pid=15590): Launched 01:12:55.961971 run osmo-bsc_10.42.42.7: Starting osmo-bsc 01:12:56.761310 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15591): Launched 01:12:57.113323 run osmo-bsc_10.42.42.7(pid=15592): Launched 01:13:00.851278 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7 01:13:03.211365 run ssh_sc5_ccli(pid=15593): Launched 01:13:22.459459 run ssh_sc5_ccli(pid=15593): ERR: Terminated: ERROR {rc=255} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=15593)] 01:13:22.617406 run ssh_sc5_ccli(pid=15593): stdout: | (launched: 2020-12-07_01:13:03.026939) 01:13:22.776151 run ssh_sc5_ccli(pid=15593): stderr: | (launched: 2020-12-07_01:13:03.083247) | ssh: connect to host 10.42.42.112 port 22: No route to host 01:13:22.912125 run sc5-trx: Configuring SC5 TRX failed, retrying 9 more times 01:13:28.293801 run ssh_sc5_ccli(pid=15594): Launched 01:13:31.519463 run ssh_sc5_ccli(pid=15594): Terminated: ok {rc=0} 01:13:31.653984 run osmo-bts-trx: Waiting for sc5-trx to start up... 01:13:32.969053 run osmo-bts-trx(pid=15595): Launched 01:18:33.755234 tst handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-b200/handover.py/01_bts0_started: FAIL (333.0s) 01:18:33.763584 tst handover.py:43: ERR: Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43] [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43] 01:18:33.771846 tst handover.py:43: Test FAILED (346.4 sec) 01:18:33.817749 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15581): Terminating (SIGTERM) 01:18:33.869176 run osmo-hlr_10.42.42.2(pid=15582): Terminating (SIGTERM) 01:18:33.922622 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15583): Terminating (SIGTERM) 01:18:33.978454 run osmo-stp_10.42.42.5(pid=15584): Terminating (SIGTERM) 01:18:34.033330 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15585): Terminating (SIGTERM) 01:18:34.088241 run osmo-msc_10.42.42.6(pid=15586): Terminating (SIGTERM) 01:18:34.140069 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15587): Terminating (SIGTERM) 01:18:34.191677 run osmo-mgw_10.42.42.3(pid=15588): Terminating (SIGTERM) 01:18:34.244161 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15589): Terminating (SIGTERM) 01:18:34.295547 run osmo-mgw_10.42.42.4(pid=15590): Terminating (SIGTERM) 01:18:34.347024 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15591): Terminating (SIGTERM) 01:18:34.398258 run osmo-bsc_10.42.42.7(pid=15592): Terminating (SIGTERM) 01:18:34.449543 run osmo-bts-trx(pid=15595): Terminating (SIGTERM) 01:18:34.462318 --- ParallelTerminationStrategy: PID 15581 died... 01:18:34.530344 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15581): Terminated: ok {rc=0} 01:18:34.542599 --- ParallelTerminationStrategy: PID 15582 died... 01:18:34.610162 run osmo-hlr_10.42.42.2(pid=15582): Terminated: ok {rc=0} 01:18:34.622316 --- ParallelTerminationStrategy: PID 15583 died... 01:18:34.689992 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15583): Terminated: ok {rc=0} 01:18:34.702155 --- ParallelTerminationStrategy: PID 15584 died... 01:18:34.769659 run osmo-stp_10.42.42.5(pid=15584): Terminated: ok {rc=0} 01:18:34.781883 --- ParallelTerminationStrategy: PID 15585 died... 01:18:34.849605 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15585): Terminated: ok {rc=0} 01:18:34.861827 --- ParallelTerminationStrategy: PID 15587 died... 01:18:34.929704 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15587): Terminated: ok {rc=0} 01:18:34.941971 --- ParallelTerminationStrategy: PID 15588 died... 01:18:35.010631 run osmo-mgw_10.42.42.3(pid=15588): Terminated {rc=15} 01:18:35.023731 --- ParallelTerminationStrategy: PID 15589 died... 01:18:35.092912 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15589): Terminated: ok {rc=0} 01:18:35.105199 --- ParallelTerminationStrategy: PID 15590 died... 01:18:35.172827 run osmo-mgw_10.42.42.4(pid=15590): Terminated {rc=15} 01:18:35.185402 --- ParallelTerminationStrategy: PID 15591 died... 01:18:35.253373 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15591): Terminated: ok {rc=0} 01:18:37.282541 --- ParallelTerminationStrategy: PID 15586 died... 01:18:37.352711 run osmo-msc_10.42.42.6(pid=15586): Terminated: ok {rc=0} 01:18:38.031728 --- ParallelTerminationStrategy: PID 15592 died... 01:18:38.101371 run osmo-bsc_10.42.42.7(pid=15592): Terminated: ok {rc=0} 01:18:38.113636 --- ParallelTerminationStrategy: PID 15595 died... 01:18:38.181597 run osmo-bts-trx(pid=15595): Terminated: ok {rc=0} 01:18:38.285679 bus /sierra_3: Setting Powered False 01:18:39.410320 bus /sierra_3: Setting Powered False 01:18:40.539408 bus /sierra_2: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 673, in exec_module File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module> wait(bsc.bts_is_connected, bts0) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]