Failed
handover_2G:trx-sysmocell5000+bts1-sysmo.handover.py (from handover_2G_trx-sysmocell5000+bts1-sysmo)
Stacktrace
handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]
Standard Output
---------------------------------------------- trial-160 handover_2G:trx-sysmocell5000+bts1-sysmo handover.py ---------------------------------------------- 06:04:50.768029 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 06:04:50.782017 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2) 06:04:50.937899 tst handover.py: using LAC 16149 06:04:51.071054 tst handover.py: using RAC 84 06:04:51.203792 tst handover.py: using CellId 16148 06:04:51.338626 tst handover.py: using BVCI 2827 06:04:51.364540 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2) 06:04:51.505319 tst handover.py: using LAC 16150 06:04:51.638217 tst handover.py: using RAC 85 06:04:51.769433 tst handover.py: using CellId 16149 06:04:51.899846 tst handover.py: using BVCI 2828 06:04:51.925768 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 06:04:51.939573 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 06:04:51.953279 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 06:04:51.967066 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 06:04:51.980822 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 06:04:51.994322 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2) 06:04:52.134670 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2) 06:04:52.377588 run osmo-hlr_10.42.42.2: Starting osmo-hlr 06:04:52.887342 run create_hlr_db(pid=29797): Launched 06:04:53.029420 bus /sierra_2: Setting Powered False 06:04:54.110620 run create_hlr_db(pid=29797): Terminated: ok {rc=0} 06:04:54.561540 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29799): Launched 06:04:54.924309 run osmo-hlr_10.42.42.2(pid=29800): Launched 06:04:55.054016 run osmo-stp_10.42.42.5: Starting osmo-stp 06:04:55.647256 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29801): Launched 06:04:56.007372 run osmo-stp_10.42.42.5(pid=29802): Launched 06:04:56.136189 run osmo-msc_10.42.42.6: Starting osmo-msc 06:04:56.765124 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29803): Launched 06:04:57.119642 run osmo-msc_10.42.42.6(pid=29804): Launched 06:04:57.244970 run osmo-mgw_10.42.42.3: Starting osmo-mgw 06:04:57.820123 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29805): Launched 06:04:58.174149 run osmo-mgw_10.42.42.3(pid=29806): Launched 06:04:58.295859 run osmo-mgw_10.42.42.4: Starting osmo-mgw 06:04:58.854305 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29807): Launched 06:04:59.217655 run osmo-mgw_10.42.42.4(pid=29808): Launched 06:04:59.346591 run osmo-bsc_10.42.42.7: Starting osmo-bsc 06:05:00.132796 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29809): Launched 06:05:00.485646 run osmo-bsc_10.42.42.7(pid=29810): Launched 06:05:04.214264 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7 06:05:06.572689 run ssh_sc5_ccli(pid=29815): Launched 06:05:25.820437 run ssh_sc5_ccli(pid=29815): ERR: Terminated: ERROR {rc=255} [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=29815)] 06:05:25.977005 run ssh_sc5_ccli(pid=29815): stdout: | (launched: 2020-12-07_06:05:06.388582) 06:05:26.134358 run ssh_sc5_ccli(pid=29815): stderr: | (launched: 2020-12-07_06:05:06.444712) | ssh: connect to host 10.42.42.112 port 22: No route to host 06:05:26.270052 run sc5-trx: Configuring SC5 TRX failed, retrying 9 more times 06:05:31.641504 run ssh_sc5_ccli(pid=29816): Launched 06:05:34.856588 run ssh_sc5_ccli(pid=29816): Terminated: ok {rc=0} 06:05:34.983012 run osmo-bts-trx: Waiting for sc5-trx to start up... 06:05:36.295767 run osmo-bts-trx(pid=29817): Launched 06:10:37.049183 tst handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-sysmo/handover.py/01_bts0_started: FAIL (332.9s) 06:10:37.057018 tst handover.py:43: ERR: Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43] [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43] 06:10:37.064585 tst handover.py:43: Test FAILED (346.3 sec) 06:10:37.109919 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29799): Terminating (SIGTERM) 06:10:37.161703 run osmo-hlr_10.42.42.2(pid=29800): Terminating (SIGTERM) 06:10:37.215282 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29801): Terminating (SIGTERM) 06:10:37.270971 run osmo-stp_10.42.42.5(pid=29802): Terminating (SIGTERM) 06:10:37.325067 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29803): Terminating (SIGTERM) 06:10:37.380832 run osmo-msc_10.42.42.6(pid=29804): Terminating (SIGTERM) 06:10:37.433561 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29805): Terminating (SIGTERM) 06:10:37.485845 run osmo-mgw_10.42.42.3(pid=29806): Terminating (SIGTERM) 06:10:37.538466 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29807): Terminating (SIGTERM) 06:10:37.591169 run osmo-mgw_10.42.42.4(pid=29808): Terminating (SIGTERM) 06:10:37.643684 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29809): Terminating (SIGTERM) 06:10:37.695875 run osmo-bsc_10.42.42.7(pid=29810): Terminating (SIGTERM) 06:10:37.748239 run osmo-bts-trx(pid=29817): Terminating (SIGTERM) 06:10:37.760207 --- ParallelTerminationStrategy: PID 29799 died... 06:10:37.829962 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29799): Terminated: ok {rc=0} 06:10:37.842467 --- ParallelTerminationStrategy: PID 29800 died... 06:10:37.911928 run osmo-hlr_10.42.42.2(pid=29800): Terminated: ok {rc=0} 06:10:37.924501 --- ParallelTerminationStrategy: PID 29801 died... 06:10:37.994035 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29801): Terminated: ok {rc=0} 06:10:38.006583 --- ParallelTerminationStrategy: PID 29802 died... 06:10:38.076173 run osmo-stp_10.42.42.5(pid=29802): Terminated: ok {rc=0} 06:10:38.088740 --- ParallelTerminationStrategy: PID 29803 died... 06:10:38.157442 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29803): Terminated: ok {rc=0} 06:10:38.169886 --- ParallelTerminationStrategy: PID 29805 died... 06:10:38.239779 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29805): Terminated: ok {rc=0} 06:10:38.252284 --- ParallelTerminationStrategy: PID 29806 died... 06:10:38.320865 run osmo-mgw_10.42.42.3(pid=29806): Terminated {rc=15} 06:10:38.333343 --- ParallelTerminationStrategy: PID 29807 died... 06:10:38.401535 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29807): Terminated: ok {rc=0} 06:10:38.413963 --- ParallelTerminationStrategy: PID 29808 died... 06:10:38.482195 run osmo-mgw_10.42.42.4(pid=29808): Terminated {rc=15} 06:10:38.494679 --- ParallelTerminationStrategy: PID 29809 died... 06:10:38.563500 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29809): Terminated: ok {rc=0} 06:10:40.593058 --- ParallelTerminationStrategy: PID 29804 died... 06:10:40.663719 run osmo-msc_10.42.42.6(pid=29804): Terminated: ok {rc=0} 06:10:41.342897 --- ParallelTerminationStrategy: PID 29810 died... 06:10:41.413971 run osmo-bsc_10.42.42.7(pid=29810): Terminated: ok {rc=0} 06:10:41.426301 --- ParallelTerminationStrategy: PID 29817 died... 06:10:41.494590 run osmo-bts-trx(pid=29817): Terminated: ok {rc=0} 06:10:41.598868 bus /sierra_3: Setting Powered False 06:10:42.725222 bus /sierra_3: Setting Powered False 06:10:43.857453 bus /sierra_2: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 673, in exec_module File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module> wait(bsc.bts_is_connected, bts0) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]