Failed

handover_2G:trx-sysmocell5000+bts1-trx-umtrx.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-umtrx)

Failing for the past 21 builds (Since #140 )

Took 5 min 50 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-160↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]

Standard Output

----------------------------------------------
trial-160 handover_2G:trx-sysmocell5000+bts1-trx-umtrx handover.py
----------------------------------------------
06:16:41.873824 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:16:41.887596 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
06:16:42.043979 tst                      handover.py: using LAC 16153
06:16:42.175252 tst                      handover.py: using RAC 88
06:16:42.307242 tst                      handover.py: using CellId 16152
06:16:42.439013 tst                      handover.py: using BVCI 2831
06:16:42.465098 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
06:16:42.622119 tst                      handover.py: using LAC 16154
06:16:42.751912 tst                      handover.py: using RAC 89
06:16:42.883075 tst                      handover.py: using CellId 16153
06:16:43.013356 tst                      handover.py: using BVCI 2832
06:16:43.039624 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:16:43.053527 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:16:43.067222 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:16:43.081048 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:16:43.095906 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:16:43.109669 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
06:16:43.244015 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
06:16:43.475487 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
06:16:43.987777 run         create_hlr_db(pid=30363): Launched
06:16:44.129612 bus                        /sierra_2: Setting Powered False
06:16:45.212467 run         create_hlr_db(pid=30363): Terminated: ok {rc=0}
06:16:45.657730 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30365): Launched
06:16:46.011311 run   osmo-hlr_10.42.42.2(pid=30366): Launched
06:16:46.140786 run              osmo-stp_10.42.42.5: Starting osmo-stp
06:16:46.726830 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30367): Launched
06:16:47.071194 run   osmo-stp_10.42.42.5(pid=30368): Launched
06:16:47.193986 run              osmo-msc_10.42.42.6: Starting osmo-msc
06:16:47.792857 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=30369): Launched
06:16:48.137013 run   osmo-msc_10.42.42.6(pid=30370): Launched
06:16:48.263189 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
06:16:48.834165 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=30371): Launched
06:16:49.180087 run   osmo-mgw_10.42.42.3(pid=30372): Launched
06:16:49.303056 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
06:16:49.855654 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=30373): Launched
06:16:50.199213 run   osmo-mgw_10.42.42.4(pid=30374): Launched
06:16:50.321561 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
06:16:51.084272 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=30375): Launched
06:16:51.447425 run   osmo-bsc_10.42.42.7(pid=30376): Launched
06:16:55.170444 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
06:16:57.493460 run          ssh_sc5_ccli(pid=30377): Launched
06:17:29.751065 run          ssh_sc5_ccli(pid=30377): Terminated: ok {rc=0}
06:17:29.877413 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
06:17:31.181354 run          osmo-bts-trx(pid=30381): Launched
06:22:31.845474 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-umtrx/handover.py/01_bts0_started: FAIL (336.8s)
06:22:31.853962 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-160↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]  [trial-160↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]
06:22:31.862182 tst                   handover.py:43: Test FAILED (350.0 sec)
06:22:31.907354 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30365): Terminating (SIGTERM)
06:22:31.959242 run   osmo-hlr_10.42.42.2(pid=30366): Terminating (SIGTERM)
06:22:32.012519 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30367): Terminating (SIGTERM)
06:22:32.067334 run   osmo-stp_10.42.42.5(pid=30368): Terminating (SIGTERM)
06:22:32.120847 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=30369): Terminating (SIGTERM)
06:22:32.175936 run   osmo-msc_10.42.42.6(pid=30370): Terminating (SIGTERM)
06:22:32.228385 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=30371): Terminating (SIGTERM)
06:22:32.281487 run   osmo-mgw_10.42.42.3(pid=30372): Terminating (SIGTERM)
06:22:32.332995 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=30373): Terminating (SIGTERM)
06:22:32.384385 run   osmo-mgw_10.42.42.4(pid=30374): Terminating (SIGTERM)
06:22:32.435731 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=30375): Terminating (SIGTERM)
06:22:32.487104 run   osmo-bsc_10.42.42.7(pid=30376): Terminating (SIGTERM)
06:22:32.538577 run          osmo-bts-trx(pid=30381): Terminating (SIGTERM)
06:22:32.551351 ---      ParallelTerminationStrategy: PID 30365 died...
06:22:32.620014 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30365): Terminated: ok {rc=0}
06:22:32.632460 ---      ParallelTerminationStrategy: PID 30366 died...
06:22:32.699995 run   osmo-hlr_10.42.42.2(pid=30366): Terminated: ok {rc=0}
06:22:32.712162 ---      ParallelTerminationStrategy: PID 30367 died...
06:22:32.779969 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30367): Terminated: ok {rc=0}
06:22:32.792215 ---      ParallelTerminationStrategy: PID 30368 died...
06:22:32.859914 run   osmo-stp_10.42.42.5(pid=30368): Terminated: ok {rc=0}
06:22:32.872052 ---      ParallelTerminationStrategy: PID 30369 died...
06:22:32.939996 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=30369): Terminated: ok {rc=0}
06:22:32.952232 ---      ParallelTerminationStrategy: PID 30371 died...
06:22:33.020078 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=30371): Terminated: ok {rc=0}
06:22:33.032274 ---      ParallelTerminationStrategy: PID 30372 died...
06:22:33.100485 run   osmo-mgw_10.42.42.3(pid=30372): Terminated {rc=15}
06:22:33.112805 ---      ParallelTerminationStrategy: PID 30373 died...
06:22:33.180928 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=30373): Terminated: ok {rc=0}
06:22:33.194138 ---      ParallelTerminationStrategy: PID 30374 died...
06:22:33.263662 run   osmo-mgw_10.42.42.4(pid=30374): Terminated {rc=15}
06:22:33.275898 ---      ParallelTerminationStrategy: PID 30375 died...
06:22:33.343918 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=30375): Terminated: ok {rc=0}
06:22:35.372917 ---      ParallelTerminationStrategy: PID 30370 died...
06:22:35.443222 run   osmo-msc_10.42.42.6(pid=30370): Terminated: ok {rc=0}
06:22:36.125425 ---      ParallelTerminationStrategy: PID 30376 died...
06:22:36.192297 run   osmo-bsc_10.42.42.7(pid=30376): Terminated: ok {rc=0}
06:22:36.204176 ---      ParallelTerminationStrategy: PID 30381 died...
06:22:36.268970 run          osmo-bts-trx(pid=30381): Terminated: ok {rc=0}
06:22:36.374643 bus                        /sierra_3: Setting Powered False
06:22:37.496016 bus                        /sierra_3: Setting Powered False
06:22:38.625671 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-160↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]