Failed

handover_2G:trx-sysmocell5000+bts1-trx-b200.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-b200)

Failing for the past 20 builds (Since #140 )

Took 5 min 47 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]

Standard Output

----------------------------------------------
trial-159 handover_2G:trx-sysmocell5000+bts1-trx-b200 handover.py
----------------------------------------------
01:12:47.353215 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
01:12:47.367498 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
01:12:47.536443 tst                      handover.py: using LAC 16055
01:12:47.674265 tst                      handover.py: using RAC 245
01:12:47.809087 tst                      handover.py: using CellId 16054
01:12:47.938741 tst                      handover.py: using BVCI 2733
01:12:47.964715 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
01:12:48.123301 tst                      handover.py: using LAC 16056
01:12:48.262246 tst                      handover.py: using RAC 246
01:12:48.400140 tst                      handover.py: using CellId 16055
01:12:48.534003 tst                      handover.py: using BVCI 2734
01:12:48.560123 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
01:12:48.574074 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
01:12:48.587756 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
01:12:48.601410 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
01:12:48.615307 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
01:12:48.629032 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
01:12:48.756799 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
01:12:48.991211 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
01:12:49.534540 run         create_hlr_db(pid=15579): Launched
01:12:49.685466 bus                        /sierra_2: Setting Powered False
01:12:50.770728 run         create_hlr_db(pid=15579): Terminated: ok {rc=0}
01:12:51.202180 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15581): Launched
01:12:51.555533 run   osmo-hlr_10.42.42.2(pid=15582): Launched
01:12:51.685943 run              osmo-stp_10.42.42.5: Starting osmo-stp
01:12:52.274966 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15583): Launched
01:12:52.619286 run   osmo-stp_10.42.42.5(pid=15584): Launched
01:12:52.742074 run              osmo-msc_10.42.42.6: Starting osmo-msc
01:12:53.342118 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15585): Launched
01:12:53.694117 run   osmo-msc_10.42.42.6(pid=15586): Launched
01:12:53.831594 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
01:12:54.442939 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15587): Launched
01:12:54.786695 run   osmo-mgw_10.42.42.3(pid=15588): Launched
01:12:54.908602 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
01:12:55.469354 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15589): Launched
01:12:55.832485 run   osmo-mgw_10.42.42.4(pid=15590): Launched
01:12:55.961971 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
01:12:56.761310 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15591): Launched
01:12:57.113323 run   osmo-bsc_10.42.42.7(pid=15592): Launched
01:13:00.851278 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
01:13:03.211365 run          ssh_sc5_ccli(pid=15593): Launched
01:13:22.459459 run          ssh_sc5_ccli(pid=15593): ERR: Terminated: ERROR {rc=255}  [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=15593)]
01:13:22.617406 run          ssh_sc5_ccli(pid=15593): stdout: 
| (launched: 2020-12-07_01:13:03.026939) 
01:13:22.776151 run          ssh_sc5_ccli(pid=15593): stderr: 
| (launched: 2020-12-07_01:13:03.083247)
| ssh: connect to host 10.42.42.112 port 22: No route to host 
01:13:22.912125 run                          sc5-trx: Configuring SC5 TRX failed, retrying 9 more times
01:13:28.293801 run          ssh_sc5_ccli(pid=15594): Launched
01:13:31.519463 run          ssh_sc5_ccli(pid=15594): Terminated: ok {rc=0}
01:13:31.653984 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
01:13:32.969053 run          osmo-bts-trx(pid=15595): Launched
01:18:33.755234 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-b200/handover.py/01_bts0_started: FAIL (333.0s)
01:18:33.763584 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]  [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]
01:18:33.771846 tst                   handover.py:43: Test FAILED (346.4 sec)
01:18:33.817749 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15581): Terminating (SIGTERM)
01:18:33.869176 run   osmo-hlr_10.42.42.2(pid=15582): Terminating (SIGTERM)
01:18:33.922622 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15583): Terminating (SIGTERM)
01:18:33.978454 run   osmo-stp_10.42.42.5(pid=15584): Terminating (SIGTERM)
01:18:34.033330 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15585): Terminating (SIGTERM)
01:18:34.088241 run   osmo-msc_10.42.42.6(pid=15586): Terminating (SIGTERM)
01:18:34.140069 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15587): Terminating (SIGTERM)
01:18:34.191677 run   osmo-mgw_10.42.42.3(pid=15588): Terminating (SIGTERM)
01:18:34.244161 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15589): Terminating (SIGTERM)
01:18:34.295547 run   osmo-mgw_10.42.42.4(pid=15590): Terminating (SIGTERM)
01:18:34.347024 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15591): Terminating (SIGTERM)
01:18:34.398258 run   osmo-bsc_10.42.42.7(pid=15592): Terminating (SIGTERM)
01:18:34.449543 run          osmo-bts-trx(pid=15595): Terminating (SIGTERM)
01:18:34.462318 ---      ParallelTerminationStrategy: PID 15581 died...
01:18:34.530344 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15581): Terminated: ok {rc=0}
01:18:34.542599 ---      ParallelTerminationStrategy: PID 15582 died...
01:18:34.610162 run   osmo-hlr_10.42.42.2(pid=15582): Terminated: ok {rc=0}
01:18:34.622316 ---      ParallelTerminationStrategy: PID 15583 died...
01:18:34.689992 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15583): Terminated: ok {rc=0}
01:18:34.702155 ---      ParallelTerminationStrategy: PID 15584 died...
01:18:34.769659 run   osmo-stp_10.42.42.5(pid=15584): Terminated: ok {rc=0}
01:18:34.781883 ---      ParallelTerminationStrategy: PID 15585 died...
01:18:34.849605 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15585): Terminated: ok {rc=0}
01:18:34.861827 ---      ParallelTerminationStrategy: PID 15587 died...
01:18:34.929704 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15587): Terminated: ok {rc=0}
01:18:34.941971 ---      ParallelTerminationStrategy: PID 15588 died...
01:18:35.010631 run   osmo-mgw_10.42.42.3(pid=15588): Terminated {rc=15}
01:18:35.023731 ---      ParallelTerminationStrategy: PID 15589 died...
01:18:35.092912 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15589): Terminated: ok {rc=0}
01:18:35.105199 ---      ParallelTerminationStrategy: PID 15590 died...
01:18:35.172827 run   osmo-mgw_10.42.42.4(pid=15590): Terminated {rc=15}
01:18:35.185402 ---      ParallelTerminationStrategy: PID 15591 died...
01:18:35.253373 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15591): Terminated: ok {rc=0}
01:18:37.282541 ---      ParallelTerminationStrategy: PID 15586 died...
01:18:37.352711 run   osmo-msc_10.42.42.6(pid=15586): Terminated: ok {rc=0}
01:18:38.031728 ---      ParallelTerminationStrategy: PID 15592 died...
01:18:38.101371 run   osmo-bsc_10.42.42.7(pid=15592): Terminated: ok {rc=0}
01:18:38.113636 ---      ParallelTerminationStrategy: PID 15595 died...
01:18:38.181597 run          osmo-bts-trx(pid=15595): Terminated: ok {rc=0}
01:18:38.285679 bus                        /sierra_3: Setting Powered False
01:18:39.410320 bus                        /sierra_3: Setting Powered False
01:18:40.539408 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]