Failed
handover_2G:trx-sysmocell5000+bts1-trx-umtrx.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-umtrx)
Stacktrace
handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]
Standard Output
---------------------------------------------- trial-159 handover_2G:trx-sysmocell5000+bts1-trx-umtrx handover.py ---------------------------------------------- 01:18:42.994259 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 01:18:43.008789 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2) 01:18:43.170492 tst handover.py: using LAC 16057 01:18:43.305106 tst handover.py: using RAC 247 01:18:43.439999 tst handover.py: using CellId 16056 01:18:43.577571 tst handover.py: using BVCI 2735 01:18:43.605311 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2) 01:18:43.769065 tst handover.py: using LAC 16058 01:18:43.912195 tst handover.py: using RAC 248 01:18:44.050585 tst handover.py: using CellId 16057 01:18:44.188511 tst handover.py: using BVCI 2736 01:18:44.216033 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 01:18:44.230532 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 01:18:44.246024 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 01:18:44.261092 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 01:18:44.275662 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6) 01:18:44.290332 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2) 01:18:44.422537 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2) 01:18:44.657569 run osmo-hlr_10.42.42.2: Starting osmo-hlr 01:18:45.186278 run create_hlr_db(pid=15669): Launched 01:18:45.329476 bus /sierra_2: Setting Powered False 01:18:46.410154 run create_hlr_db(pid=15669): Terminated: ok {rc=0} 01:18:46.856656 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15671): Launched 01:18:47.198601 run osmo-hlr_10.42.42.2(pid=15672): Launched 01:18:47.321421 run osmo-stp_10.42.42.5: Starting osmo-stp 01:18:47.892310 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15673): Launched 01:18:48.255231 run osmo-stp_10.42.42.5(pid=15674): Launched 01:18:48.384454 run osmo-msc_10.42.42.6: Starting osmo-msc 01:18:49.013734 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15675): Launched 01:18:49.374324 run osmo-msc_10.42.42.6(pid=15676): Launched 01:18:49.499190 run osmo-mgw_10.42.42.3: Starting osmo-mgw 01:18:50.080497 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15677): Launched 01:18:50.445805 run osmo-mgw_10.42.42.3(pid=15678): Launched 01:18:50.575037 run osmo-mgw_10.42.42.4: Starting osmo-mgw 01:18:51.154202 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15679): Launched 01:18:51.500977 run osmo-mgw_10.42.42.4(pid=15680): Launched 01:18:51.622936 run osmo-bsc_10.42.42.7: Starting osmo-bsc 01:18:52.382600 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15681): Launched 01:18:52.734423 run osmo-bsc_10.42.42.7(pid=15682): Launched 01:18:56.477098 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7 01:18:58.822053 run ssh_sc5_ccli(pid=15686): Launched 01:19:18.070487 run ssh_sc5_ccli(pid=15686): ERR: Terminated: ERROR {rc=255} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=15686)] 01:19:18.230531 run ssh_sc5_ccli(pid=15686): stdout: | (launched: 2020-12-07_01:18:58.644495) 01:19:18.390204 run ssh_sc5_ccli(pid=15686): stderr: | (launched: 2020-12-07_01:18:58.697991) | ssh: connect to host 10.42.42.112 port 22: No route to host 01:19:18.526019 run sc5-trx: Configuring SC5 TRX failed, retrying 9 more times 01:19:23.908682 run ssh_sc5_ccli(pid=15687): Launched 01:19:27.133716 run ssh_sc5_ccli(pid=15687): Terminated: ok {rc=0} 01:19:27.267819 run osmo-bts-trx: Waiting for sc5-trx to start up... 01:19:28.582175 run osmo-bts-trx(pid=15688): Launched 01:24:29.335722 tst handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-umtrx/handover.py/01_bts0_started: FAIL (333.0s) 01:24:29.343045 tst handover.py:43: ERR: Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43] [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43] 01:24:29.349882 tst handover.py:43: Test FAILED (346.3 sec) 01:24:29.391551 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15671): Terminating (SIGTERM) 01:24:29.439983 run osmo-hlr_10.42.42.2(pid=15672): Terminating (SIGTERM) 01:24:29.489553 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15673): Terminating (SIGTERM) 01:24:29.540290 run osmo-stp_10.42.42.5(pid=15674): Terminating (SIGTERM) 01:24:29.590513 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15675): Terminating (SIGTERM) 01:24:29.641503 run osmo-msc_10.42.42.6(pid=15676): Terminating (SIGTERM) 01:24:29.690055 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15677): Terminating (SIGTERM) 01:24:29.738377 run osmo-mgw_10.42.42.3(pid=15678): Terminating (SIGTERM) 01:24:29.786825 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15679): Terminating (SIGTERM) 01:24:29.835115 run osmo-mgw_10.42.42.4(pid=15680): Terminating (SIGTERM) 01:24:29.884089 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15681): Terminating (SIGTERM) 01:24:29.932675 run osmo-bsc_10.42.42.7(pid=15682): Terminating (SIGTERM) 01:24:29.981162 run osmo-bts-trx(pid=15688): Terminating (SIGTERM) 01:24:29.992205 --- ParallelTerminationStrategy: PID 15671 died... 01:24:30.056383 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15671): Terminated: ok {rc=0} 01:24:30.067896 --- ParallelTerminationStrategy: PID 15672 died... 01:24:30.131896 run osmo-hlr_10.42.42.2(pid=15672): Terminated: ok {rc=0} 01:24:30.143460 --- ParallelTerminationStrategy: PID 15673 died... 01:24:30.207659 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15673): Terminated: ok {rc=0} 01:24:30.219269 --- ParallelTerminationStrategy: PID 15674 died... 01:24:30.283435 run osmo-stp_10.42.42.5(pid=15674): Terminated: ok {rc=0} 01:24:30.295007 --- ParallelTerminationStrategy: PID 15675 died... 01:24:30.359166 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15675): Terminated: ok {rc=0} 01:24:30.370761 --- ParallelTerminationStrategy: PID 15677 died... 01:24:30.434948 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15677): Terminated: ok {rc=0} 01:24:30.446570 --- ParallelTerminationStrategy: PID 15678 died... 01:24:30.510644 run osmo-mgw_10.42.42.3(pid=15678): Terminated {rc=15} 01:24:30.522252 --- ParallelTerminationStrategy: PID 15679 died... 01:24:30.586395 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15679): Terminated: ok {rc=0} 01:24:30.598096 --- ParallelTerminationStrategy: PID 15680 died... 01:24:30.662579 run osmo-mgw_10.42.42.4(pid=15680): Terminated {rc=15} 01:24:30.674250 --- ParallelTerminationStrategy: PID 15681 died... 01:24:30.738351 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15681): Terminated: ok {rc=0} 01:24:33.433282 --- ParallelTerminationStrategy: PID 15676 died... 01:24:33.499515 run osmo-msc_10.42.42.6(pid=15676): Terminated: ok {rc=0} 01:24:33.511189 --- ParallelTerminationStrategy: PID 15682 died... 01:24:33.575622 run osmo-bsc_10.42.42.7(pid=15682): Terminated: ok {rc=0} 01:24:33.587317 --- ParallelTerminationStrategy: PID 15688 died... 01:24:33.651484 run osmo-bts-trx(pid=15688): Terminated: ok {rc=0} 01:24:33.758303 bus /sierra_3: Setting Powered False 01:24:34.880849 bus /sierra_3: Setting Powered False 01:24:36.009797 bus /sierra_2: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 673, in exec_module File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module> wait(bsc.bts_is_connected, bts0) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]