Failed

handover_2G:trx-sysmocell5000+bts1-trx-umtrx.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-umtrx)

Failing for the past 22 builds (Since #140 )

Took 5 min 51 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]

Standard Output

----------------------------------------------
trial-161 handover_2G:trx-sysmocell5000+bts1-trx-umtrx handover.py
----------------------------------------------
08:43:15.077072 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
08:43:15.091915 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
08:43:15.262140 tst                      handover.py: using LAC 16225
08:43:15.403065 tst                      handover.py: using RAC 160
08:43:15.542146 tst                      handover.py: using CellId 16224
08:43:15.681124 tst                      handover.py: using BVCI 2903
08:43:15.708281 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
08:43:15.870936 tst                      handover.py: using LAC 16226
08:43:16.009596 tst                      handover.py: using RAC 161
08:43:16.150063 tst                      handover.py: using CellId 16225
08:43:16.294203 tst                      handover.py: using BVCI 2904
08:43:16.321390 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
08:43:16.336033 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
08:43:16.351024 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
08:43:16.366238 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
08:43:16.381294 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
08:43:16.396611 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
08:43:16.528309 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
08:43:16.753904 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
08:43:17.260211 run          create_hlr_db(pid=1900): Launched
08:43:17.403138 bus                        /sierra_2: Setting Powered False
08:43:18.483956 run          create_hlr_db(pid=1900): Terminated: ok {rc=0}
08:43:18.917193 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1902): Launched
08:43:19.273016 run    osmo-hlr_10.42.42.2(pid=1903): Launched
08:43:19.393962 run              osmo-stp_10.42.42.5: Starting osmo-stp
08:43:19.955876 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1904): Launched
08:43:20.306514 run    osmo-stp_10.42.42.5(pid=1905): Launched
08:43:20.434236 run              osmo-msc_10.42.42.6: Starting osmo-msc
08:43:21.058437 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1906): Launched
08:43:21.413779 run    osmo-msc_10.42.42.6(pid=1907): Launched
08:43:21.537693 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
08:43:22.111247 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1908): Launched
08:43:22.465461 run    osmo-mgw_10.42.42.3(pid=1909): Launched
08:43:22.586076 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
08:43:23.134804 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1910): Launched
08:43:23.484152 run    osmo-mgw_10.42.42.4(pid=1911): Launched
08:43:23.612655 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
08:43:24.409588 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1912): Launched
08:43:24.769592 run    osmo-bsc_10.42.42.7(pid=1913): Launched
08:43:28.498111 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
08:43:30.843219 run           ssh_sc5_ccli(pid=1914): Launched
08:44:03.105727 run           ssh_sc5_ccli(pid=1914): Terminated: ok {rc=0}
08:44:03.237832 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
08:44:04.549977 run           osmo-bts-trx(pid=1916): Launched
08:49:05.212099 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-umtrx/handover.py/01_bts0_started: FAIL (336.8s)
08:49:05.220118 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]  [trial-161↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]
08:49:05.228039 tst                   handover.py:43: Test FAILED (350.1 sec)
08:49:05.273109 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1902): Terminating (SIGTERM)
08:49:05.324905 run    osmo-hlr_10.42.42.2(pid=1903): Terminating (SIGTERM)
08:49:05.377679 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1904): Terminating (SIGTERM)
08:49:05.433141 run    osmo-stp_10.42.42.5(pid=1905): Terminating (SIGTERM)
08:49:05.487598 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1906): Terminating (SIGTERM)
08:49:05.543340 run    osmo-msc_10.42.42.6(pid=1907): Terminating (SIGTERM)
08:49:05.596774 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1908): Terminating (SIGTERM)
08:49:05.648986 run    osmo-mgw_10.42.42.3(pid=1909): Terminating (SIGTERM)
08:49:05.701283 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1910): Terminating (SIGTERM)
08:49:05.753404 run    osmo-mgw_10.42.42.4(pid=1911): Terminating (SIGTERM)
08:49:05.805324 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1912): Terminating (SIGTERM)
08:49:05.857793 run    osmo-bsc_10.42.42.7(pid=1913): Terminating (SIGTERM)
08:49:05.909923 run           osmo-bts-trx(pid=1916): Terminating (SIGTERM)
08:49:05.921763 ---      ParallelTerminationStrategy: PID 1902 died...
08:49:05.991369 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1902): Terminated: ok {rc=0}
08:49:06.005940 ---      ParallelTerminationStrategy: PID 1903 died...
08:49:06.073709 run    osmo-hlr_10.42.42.2(pid=1903): Terminated: ok {rc=0}
08:49:06.085811 ---      ParallelTerminationStrategy: PID 1904 died...
08:49:06.153218 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1904): Terminated: ok {rc=0}
08:49:06.165448 ---      ParallelTerminationStrategy: PID 1905 died...
08:49:06.233830 run    osmo-stp_10.42.42.5(pid=1905): Terminated: ok {rc=0}
08:49:06.246093 ---      ParallelTerminationStrategy: PID 1906 died...
08:49:06.314135 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1906): Terminated: ok {rc=0}
08:49:06.326344 ---      ParallelTerminationStrategy: PID 1908 died...
08:49:06.394051 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1908): Terminated: ok {rc=0}
08:49:06.406222 ---      ParallelTerminationStrategy: PID 1909 died...
08:49:06.474351 run    osmo-mgw_10.42.42.3(pid=1909): Terminated {rc=15}
08:49:06.486571 ---      ParallelTerminationStrategy: PID 1910 died...
08:49:06.554236 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1910): Terminated: ok {rc=0}
08:49:06.566466 ---      ParallelTerminationStrategy: PID 1911 died...
08:49:06.633980 run    osmo-mgw_10.42.42.4(pid=1911): Terminated {rc=15}
08:49:06.646202 ---      ParallelTerminationStrategy: PID 1912 died...
08:49:06.713823 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1912): Terminated: ok {rc=0}
08:49:08.742929 ---      ParallelTerminationStrategy: PID 1907 died...
08:49:08.812607 run    osmo-msc_10.42.42.6(pid=1907): Terminated: ok {rc=0}
08:49:09.491610 ---      ParallelTerminationStrategy: PID 1913 died...
08:49:09.561296 run    osmo-bsc_10.42.42.7(pid=1913): Terminated: ok {rc=0}
08:49:09.573589 ---      ParallelTerminationStrategy: PID 1916 died...
08:49:09.641411 run           osmo-bts-trx(pid=1916): Terminated: ok {rc=0}
08:49:09.745913 bus                        /sierra_3: Setting Powered False
08:49:10.870154 bus                        /sierra_3: Setting Powered False
08:49:12.002123 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]