Failed

handover_2G:trx-sysmocell5000+bts1-trx-b200.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-b200)

Failing for the past 24 builds (Since #140 )

Took 5 min 51 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-163↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]

Standard Output

----------------------------------------------
trial-163 handover_2G:trx-sysmocell5000+bts1-trx-b200 handover.py
----------------------------------------------
13:30:18.387671 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:30:18.402229 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
13:30:18.566044 tst                      handover.py: using LAC 16367
13:30:18.704107 tst                      handover.py: using RAC 47
13:30:18.850713 tst                      handover.py: using CellId 16366
13:30:18.992313 tst                      handover.py: using BVCI 3045
13:30:19.020520 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
13:30:19.182195 tst                      handover.py: using LAC 16368
13:30:19.323162 tst                      handover.py: using RAC 48
13:30:19.463253 tst                      handover.py: using CellId 16367
13:30:19.603186 tst                      handover.py: using BVCI 3046
13:30:19.631182 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:30:19.645691 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:30:19.660056 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:30:19.674669 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:30:19.689169 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:30:19.703474 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
13:30:19.835265 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
13:30:20.069808 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
13:30:20.621650 run          create_hlr_db(pid=9739): Launched
13:30:20.774684 bus                        /sierra_2: Setting Powered False
13:30:21.862750 run          create_hlr_db(pid=9739): Terminated: ok {rc=0}
13:30:22.302915 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9741): Launched
13:30:22.649056 run    osmo-hlr_10.42.42.2(pid=9742): Launched
13:30:22.773239 run              osmo-stp_10.42.42.5: Starting osmo-stp
13:30:23.345709 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9743): Launched
13:30:23.691241 run    osmo-stp_10.42.42.5(pid=9744): Launched
13:30:23.815101 run              osmo-msc_10.42.42.6: Starting osmo-msc
13:30:24.421184 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=9745): Launched
13:30:24.769462 run    osmo-msc_10.42.42.6(pid=9746): Launched
13:30:24.896218 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
13:30:25.476112 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9747): Launched
13:30:25.833995 run    osmo-mgw_10.42.42.3(pid=9748): Launched
13:30:25.966426 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
13:30:26.560023 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9749): Launched
13:30:26.913245 run    osmo-mgw_10.42.42.4(pid=9750): Launched
13:30:27.037894 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
13:30:27.809845 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=9751): Launched
13:30:28.172247 run    osmo-bsc_10.42.42.7(pid=9752): Launched
13:30:31.920184 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
13:30:34.287706 run           ssh_sc5_ccli(pid=9753): Launched
13:31:06.550260 run           ssh_sc5_ccli(pid=9753): Terminated: ok {rc=0}
13:31:06.679275 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
13:31:07.989411 run           osmo-bts-trx(pid=9754): Launched
13:36:08.681844 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-b200/handover.py/01_bts0_started: FAIL (336.9s)
13:36:08.690996 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-163↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]  [trial-163↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]
13:36:08.699253 tst                   handover.py:43: Test FAILED (350.3 sec)
13:36:08.746717 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9741): Terminating (SIGTERM)
13:36:08.799442 run    osmo-hlr_10.42.42.2(pid=9742): Terminating (SIGTERM)
13:36:08.852824 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9743): Terminating (SIGTERM)
13:36:08.907943 run    osmo-stp_10.42.42.5(pid=9744): Terminating (SIGTERM)
13:36:08.961388 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=9745): Terminating (SIGTERM)
13:36:09.017612 run    osmo-msc_10.42.42.6(pid=9746): Terminating (SIGTERM)
13:36:09.071424 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9747): Terminating (SIGTERM)
13:36:09.124177 run    osmo-mgw_10.42.42.3(pid=9748): Terminating (SIGTERM)
13:36:09.176980 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9749): Terminating (SIGTERM)
13:36:09.229618 run    osmo-mgw_10.42.42.4(pid=9750): Terminating (SIGTERM)
13:36:09.282864 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=9751): Terminating (SIGTERM)
13:36:09.335497 run    osmo-bsc_10.42.42.7(pid=9752): Terminating (SIGTERM)
13:36:09.388798 run           osmo-bts-trx(pid=9754): Terminating (SIGTERM)
13:36:09.401711 ---      ParallelTerminationStrategy: PID 9741 died...
13:36:09.471533 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9741): Terminated: ok {rc=0}
13:36:09.483856 ---      ParallelTerminationStrategy: PID 9742 died...
13:36:09.553266 run    osmo-hlr_10.42.42.2(pid=9742): Terminated: ok {rc=0}
13:36:09.565566 ---      ParallelTerminationStrategy: PID 9743 died...
13:36:09.635410 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9743): Terminated: ok {rc=0}
13:36:09.647762 ---      ParallelTerminationStrategy: PID 9744 died...
13:36:09.717307 run    osmo-stp_10.42.42.5(pid=9744): Terminated: ok {rc=0}
13:36:09.729769 ---      ParallelTerminationStrategy: PID 9745 died...
13:36:09.799396 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=9745): Terminated: ok {rc=0}
13:36:09.811719 ---      ParallelTerminationStrategy: PID 9747 died...
13:36:09.881650 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9747): Terminated: ok {rc=0}
13:36:09.894043 ---      ParallelTerminationStrategy: PID 9748 died...
13:36:09.963719 run    osmo-mgw_10.42.42.3(pid=9748): Terminated {rc=15}
13:36:09.976155 ---      ParallelTerminationStrategy: PID 9749 died...
13:36:10.046941 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9749): Terminated: ok {rc=0}
13:36:10.060918 ---      ParallelTerminationStrategy: PID 9750 died...
13:36:10.130701 run    osmo-mgw_10.42.42.4(pid=9750): Terminated {rc=15}
13:36:10.143082 ---      ParallelTerminationStrategy: PID 9751 died...
13:36:10.212975 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=9751): Terminated: ok {rc=0}
13:36:12.242289 ---      ParallelTerminationStrategy: PID 9746 died...
13:36:12.315861 run    osmo-msc_10.42.42.6(pid=9746): Terminated: ok {rc=0}
13:36:12.995197 ---      ParallelTerminationStrategy: PID 9752 died...
13:36:13.067012 run    osmo-bsc_10.42.42.7(pid=9752): Terminated: ok {rc=0}
13:36:13.079472 ---      ParallelTerminationStrategy: PID 9754 died...
13:36:13.149169 run           osmo-bts-trx(pid=9754): Terminated: ok {rc=0}
13:36:13.253217 bus                        /sierra_3: Setting Powered False
13:36:14.382062 bus                        /sierra_3: Setting Powered False
13:36:15.517326 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-163↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]