Failed

handover_2G:trx-sysmocell5000+bts1-sysmo.handover.py (from handover_2G_trx-sysmocell5000+bts1-sysmo)

Failing for the past 21 builds (Since #140 )

Took 5 min 47 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]

Standard Output

----------------------------------------------
trial-160 handover_2G:trx-sysmocell5000+bts1-sysmo handover.py
----------------------------------------------
06:04:50.768029 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
06:04:50.782017 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2)
06:04:50.937899 tst                      handover.py: using LAC 16149
06:04:51.071054 tst                      handover.py: using RAC 84
06:04:51.203792 tst                      handover.py: using CellId 16148
06:04:51.338626 tst                      handover.py: using BVCI 2827
06:04:51.364540 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2)
06:04:51.505319 tst                      handover.py: using LAC 16150
06:04:51.638217 tst                      handover.py: using RAC 85
06:04:51.769433 tst                      handover.py: using CellId 16149
06:04:51.899846 tst                      handover.py: using BVCI 2828
06:04:51.925768 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
06:04:51.939573 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
06:04:51.953279 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
06:04:51.967066 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
06:04:51.980822 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
06:04:51.994322 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2)
06:04:52.134670 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2)
06:04:52.377588 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
06:04:52.887342 run         create_hlr_db(pid=29797): Launched
06:04:53.029420 bus                        /sierra_2: Setting Powered False
06:04:54.110620 run         create_hlr_db(pid=29797): Terminated: ok {rc=0}
06:04:54.561540 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29799): Launched
06:04:54.924309 run   osmo-hlr_10.42.42.2(pid=29800): Launched
06:04:55.054016 run              osmo-stp_10.42.42.5: Starting osmo-stp
06:04:55.647256 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29801): Launched
06:04:56.007372 run   osmo-stp_10.42.42.5(pid=29802): Launched
06:04:56.136189 run              osmo-msc_10.42.42.6: Starting osmo-msc
06:04:56.765124 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29803): Launched
06:04:57.119642 run   osmo-msc_10.42.42.6(pid=29804): Launched
06:04:57.244970 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
06:04:57.820123 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29805): Launched
06:04:58.174149 run   osmo-mgw_10.42.42.3(pid=29806): Launched
06:04:58.295859 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
06:04:58.854305 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29807): Launched
06:04:59.217655 run   osmo-mgw_10.42.42.4(pid=29808): Launched
06:04:59.346591 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
06:05:00.132796 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29809): Launched
06:05:00.485646 run   osmo-bsc_10.42.42.7(pid=29810): Launched
06:05:04.214264 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
06:05:06.572689 run          ssh_sc5_ccli(pid=29815): Launched
06:05:25.820437 run          ssh_sc5_ccli(pid=29815): ERR: Terminated: ERROR {rc=255}  [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=29815)]
06:05:25.977005 run          ssh_sc5_ccli(pid=29815): stdout: 
| (launched: 2020-12-07_06:05:06.388582) 
06:05:26.134358 run          ssh_sc5_ccli(pid=29815): stderr: 
| (launched: 2020-12-07_06:05:06.444712)
| ssh: connect to host 10.42.42.112 port 22: No route to host 
06:05:26.270052 run                          sc5-trx: Configuring SC5 TRX failed, retrying 9 more times
06:05:31.641504 run          ssh_sc5_ccli(pid=29816): Launched
06:05:34.856588 run          ssh_sc5_ccli(pid=29816): Terminated: ok {rc=0}
06:05:34.983012 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
06:05:36.295767 run          osmo-bts-trx(pid=29817): Launched
06:10:37.049183 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-sysmo/handover.py/01_bts0_started: FAIL (332.9s)
06:10:37.057018 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]  [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]
06:10:37.064585 tst                   handover.py:43: Test FAILED (346.3 sec)
06:10:37.109919 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29799): Terminating (SIGTERM)
06:10:37.161703 run   osmo-hlr_10.42.42.2(pid=29800): Terminating (SIGTERM)
06:10:37.215282 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29801): Terminating (SIGTERM)
06:10:37.270971 run   osmo-stp_10.42.42.5(pid=29802): Terminating (SIGTERM)
06:10:37.325067 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29803): Terminating (SIGTERM)
06:10:37.380832 run   osmo-msc_10.42.42.6(pid=29804): Terminating (SIGTERM)
06:10:37.433561 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29805): Terminating (SIGTERM)
06:10:37.485845 run   osmo-mgw_10.42.42.3(pid=29806): Terminating (SIGTERM)
06:10:37.538466 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29807): Terminating (SIGTERM)
06:10:37.591169 run   osmo-mgw_10.42.42.4(pid=29808): Terminating (SIGTERM)
06:10:37.643684 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29809): Terminating (SIGTERM)
06:10:37.695875 run   osmo-bsc_10.42.42.7(pid=29810): Terminating (SIGTERM)
06:10:37.748239 run          osmo-bts-trx(pid=29817): Terminating (SIGTERM)
06:10:37.760207 ---      ParallelTerminationStrategy: PID 29799 died...
06:10:37.829962 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29799): Terminated: ok {rc=0}
06:10:37.842467 ---      ParallelTerminationStrategy: PID 29800 died...
06:10:37.911928 run   osmo-hlr_10.42.42.2(pid=29800): Terminated: ok {rc=0}
06:10:37.924501 ---      ParallelTerminationStrategy: PID 29801 died...
06:10:37.994035 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29801): Terminated: ok {rc=0}
06:10:38.006583 ---      ParallelTerminationStrategy: PID 29802 died...
06:10:38.076173 run   osmo-stp_10.42.42.5(pid=29802): Terminated: ok {rc=0}
06:10:38.088740 ---      ParallelTerminationStrategy: PID 29803 died...
06:10:38.157442 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29803): Terminated: ok {rc=0}
06:10:38.169886 ---      ParallelTerminationStrategy: PID 29805 died...
06:10:38.239779 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29805): Terminated: ok {rc=0}
06:10:38.252284 ---      ParallelTerminationStrategy: PID 29806 died...
06:10:38.320865 run   osmo-mgw_10.42.42.3(pid=29806): Terminated {rc=15}
06:10:38.333343 ---      ParallelTerminationStrategy: PID 29807 died...
06:10:38.401535 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29807): Terminated: ok {rc=0}
06:10:38.413963 ---      ParallelTerminationStrategy: PID 29808 died...
06:10:38.482195 run   osmo-mgw_10.42.42.4(pid=29808): Terminated {rc=15}
06:10:38.494679 ---      ParallelTerminationStrategy: PID 29809 died...
06:10:38.563500 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29809): Terminated: ok {rc=0}
06:10:40.593058 ---      ParallelTerminationStrategy: PID 29804 died...
06:10:40.663719 run   osmo-msc_10.42.42.6(pid=29804): Terminated: ok {rc=0}
06:10:41.342897 ---      ParallelTerminationStrategy: PID 29810 died...
06:10:41.413971 run   osmo-bsc_10.42.42.7(pid=29810): Terminated: ok {rc=0}
06:10:41.426301 ---      ParallelTerminationStrategy: PID 29817 died...
06:10:41.494590 run          osmo-bts-trx(pid=29817): Terminated: ok {rc=0}
06:10:41.598868 bus                        /sierra_3: Setting Powered False
06:10:42.725222 bus                        /sierra_3: Setting Powered False
06:10:43.857453 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-160↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]