Failed
handover_2G:trx-sysmocell5000+bts1-trx-b200.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-b200)
Stacktrace
handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]
Standard Output
---------------------------------------------- trial-164 handover_2G:trx-sysmocell5000+bts1-trx-b200 handover.py ---------------------------------------------- 16:09:27.528396 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 16:09:27.542868 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2) 16:09:27.707664 tst handover.py: using LAC 16440 16:09:27.845573 tst handover.py: using RAC 120 16:09:27.980965 tst handover.py: using CellId 16439 16:09:28.118895 tst handover.py: using BVCI 3118 16:09:28.145756 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2) 16:09:28.301972 tst handover.py: using LAC 16441 16:09:28.438230 tst handover.py: using RAC 121 16:09:28.575678 tst handover.py: using CellId 16440 16:09:28.718931 tst handover.py: using BVCI 3119 16:09:28.747417 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 16:09:28.761832 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 16:09:28.776185 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 16:09:28.790768 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 16:09:28.805335 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6) 16:09:28.820241 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2) 16:09:28.952858 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2) 16:09:29.185016 run osmo-hlr_10.42.42.2: Starting osmo-hlr 16:09:29.723249 run create_hlr_db(pid=13330): Launched 16:09:29.879639 bus /sierra_2: Setting Powered False 16:09:30.969010 run create_hlr_db(pid=13330): Terminated: ok {rc=0} 16:09:31.420122 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13332): Launched 16:09:31.797288 run osmo-hlr_10.42.42.2(pid=13333): Launched 16:09:31.931676 run osmo-stp_10.42.42.5: Starting osmo-stp 16:09:32.549996 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13334): Launched 16:09:32.923680 run osmo-stp_10.42.42.5(pid=13335): Launched 16:09:33.056633 run osmo-msc_10.42.42.6: Starting osmo-msc 16:09:33.709213 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13336): Launched 16:09:34.078897 run osmo-msc_10.42.42.6(pid=13337): Launched 16:09:34.206838 run osmo-mgw_10.42.42.3: Starting osmo-mgw 16:09:34.793764 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13338): Launched 16:09:35.143906 run osmo-mgw_10.42.42.3(pid=13339): Launched 16:09:35.267775 run osmo-mgw_10.42.42.4: Starting osmo-mgw 16:09:35.836532 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13340): Launched 16:09:36.185964 run osmo-mgw_10.42.42.4(pid=13341): Launched 16:09:36.311180 run osmo-bsc_10.42.42.7: Starting osmo-bsc 16:09:37.086148 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13342): Launched 16:09:37.437356 run osmo-bsc_10.42.42.7(pid=13343): Launched 16:09:41.153116 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7 16:09:43.488565 run ssh_sc5_ccli(pid=13344): Launched 16:10:02.729768 run ssh_sc5_ccli(pid=13344): ERR: Terminated: ERROR {rc=255} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=13344)] 16:10:02.882920 run ssh_sc5_ccli(pid=13344): stdout: | (launched: 2020-12-07_16:09:43.310034) 16:10:03.036959 run ssh_sc5_ccli(pid=13344): stderr: | (launched: 2020-12-07_16:09:43.364857) | ssh: connect to host 10.42.42.112 port 22: No route to host 16:10:03.167854 run sc5-trx: Configuring SC5 TRX failed, retrying 9 more times 16:10:08.557930 run ssh_sc5_ccli(pid=13348): Launched 16:10:11.790019 run ssh_sc5_ccli(pid=13348): Terminated: ok {rc=0} 16:10:11.928892 run osmo-bts-trx: Waiting for sc5-trx to start up... 16:10:13.254945 run osmo-bts-trx(pid=13349): Launched 16:15:14.032149 tst handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-b200/handover.py/01_bts0_started: FAIL (333.0s) 16:15:14.039977 tst handover.py:43: ERR: Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43] [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43] 16:15:14.048402 tst handover.py:43: Test FAILED (346.5 sec) 16:15:14.093475 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13332): Terminating (SIGTERM) 16:15:14.144945 run osmo-hlr_10.42.42.2(pid=13333): Terminating (SIGTERM) 16:15:14.198113 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13334): Terminating (SIGTERM) 16:15:14.255057 run osmo-stp_10.42.42.5(pid=13335): Terminating (SIGTERM) 16:15:14.308720 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13336): Terminating (SIGTERM) 16:15:14.365460 run osmo-msc_10.42.42.6(pid=13337): Terminating (SIGTERM) 16:15:14.418211 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13338): Terminating (SIGTERM) 16:15:14.470012 run osmo-mgw_10.42.42.3(pid=13339): Terminating (SIGTERM) 16:15:14.527427 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13340): Terminating (SIGTERM) 16:15:14.576558 run osmo-mgw_10.42.42.4(pid=13341): Terminating (SIGTERM) 16:15:14.625802 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13342): Terminating (SIGTERM) 16:15:14.675105 run osmo-bsc_10.42.42.7(pid=13343): Terminating (SIGTERM) 16:15:14.724286 run osmo-bts-trx(pid=13349): Terminating (SIGTERM) 16:15:14.735549 --- ParallelTerminationStrategy: PID 13332 died... 16:15:14.800483 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13332): Terminated: ok {rc=0} 16:15:14.812314 --- ParallelTerminationStrategy: PID 13333 died... 16:15:14.877173 run osmo-hlr_10.42.42.2(pid=13333): Terminated: ok {rc=0} 16:15:14.888815 --- ParallelTerminationStrategy: PID 13334 died... 16:15:14.952707 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13334): Terminated: ok {rc=0} 16:15:14.964285 --- ParallelTerminationStrategy: PID 13335 died... 16:15:15.028268 run osmo-stp_10.42.42.5(pid=13335): Terminated: ok {rc=0} 16:15:15.039901 --- ParallelTerminationStrategy: PID 13336 died... 16:15:15.103950 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13336): Terminated: ok {rc=0} 16:15:15.115580 --- ParallelTerminationStrategy: PID 13338 died... 16:15:15.180051 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13338): Terminated: ok {rc=0} 16:15:15.191676 --- ParallelTerminationStrategy: PID 13339 died... 16:15:15.255469 run osmo-mgw_10.42.42.3(pid=13339): Terminated {rc=15} 16:15:15.267054 --- ParallelTerminationStrategy: PID 13340 died... 16:15:15.331265 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13340): Terminated: ok {rc=0} 16:15:15.342932 --- ParallelTerminationStrategy: PID 13341 died... 16:15:15.406960 run osmo-mgw_10.42.42.4(pid=13341): Terminated {rc=15} 16:15:15.418558 --- ParallelTerminationStrategy: PID 13342 died... 16:15:15.482657 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13342): Terminated: ok {rc=0} 16:15:17.511051 --- ParallelTerminationStrategy: PID 13337 died... 16:15:17.577557 run osmo-msc_10.42.42.6(pid=13337): Terminated: ok {rc=0} 16:15:18.256148 --- ParallelTerminationStrategy: PID 13343 died... 16:15:18.324048 run osmo-bsc_10.42.42.7(pid=13343): Terminated: ok {rc=0} 16:15:18.335758 --- ParallelTerminationStrategy: PID 13349 died... 16:15:18.400110 run osmo-bts-trx(pid=13349): Terminated: ok {rc=0} 16:15:18.505646 bus /sierra_3: Setting Powered False 16:15:19.627139 bus /sierra_3: Setting Powered False 16:15:20.757145 bus /sierra_2: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 673, in exec_module File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module> wait(bsc.bts_is_connected, bts0) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]