Failed

handover_2G:trx-sysmocell5000+bts1-trx-umtrx.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-umtrx)

Failing for the past 25 builds (Since #140 )

Took 5 min 51 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]

Standard Output

----------------------------------------------
trial-164 handover_2G:trx-sysmocell5000+bts1-trx-umtrx handover.py
----------------------------------------------
16:15:23.216946 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
16:15:23.231605 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
16:15:23.402763 tst                      handover.py: using LAC 16442
16:15:23.541080 tst                      handover.py: using RAC 122
16:15:23.677687 tst                      handover.py: using CellId 16441
16:15:23.815698 tst                      handover.py: using BVCI 3120
16:15:23.842520 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
16:15:23.997143 tst                      handover.py: using LAC 16443
16:15:24.134950 tst                      handover.py: using RAC 123
16:15:24.271360 tst                      handover.py: using CellId 16442
16:15:24.406393 tst                      handover.py: using BVCI 3121
16:15:24.433099 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
16:15:24.446720 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
16:15:24.460197 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
16:15:24.473857 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
16:15:24.487448 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
16:15:24.500960 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
16:15:24.636438 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
16:15:24.870609 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
16:15:25.391435 run         create_hlr_db(pid=13432): Launched
16:15:25.535428 bus                        /sierra_2: Setting Powered False
16:15:26.619870 run         create_hlr_db(pid=13432): Terminated: ok {rc=0}
16:15:27.085026 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13434): Launched
16:15:27.452247 run   osmo-hlr_10.42.42.2(pid=13435): Launched
16:15:27.576238 run              osmo-stp_10.42.42.5: Starting osmo-stp
16:15:28.155401 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13436): Launched
16:15:28.517651 run   osmo-stp_10.42.42.5(pid=13437): Launched
16:15:28.650995 run              osmo-msc_10.42.42.6: Starting osmo-msc
16:15:29.293547 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13438): Launched
16:15:29.657744 run   osmo-msc_10.42.42.6(pid=13439): Launched
16:15:29.800276 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
16:15:30.426594 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13440): Launched
16:15:30.788167 run   osmo-mgw_10.42.42.3(pid=13441): Launched
16:15:30.921753 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
16:15:31.520578 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13442): Launched
16:15:31.884008 run   osmo-mgw_10.42.42.4(pid=13443): Launched
16:15:32.017913 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
16:15:32.835958 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13444): Launched
16:15:33.188953 run   osmo-bsc_10.42.42.7(pid=13445): Launched
16:15:36.908902 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
16:15:39.264242 run          ssh_sc5_ccli(pid=13446): Launched
16:16:11.524094 run          ssh_sc5_ccli(pid=13446): Terminated: ok {rc=0}
16:16:11.654668 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
16:16:12.967241 run          osmo-bts-trx(pid=13449): Launched
16:21:13.599164 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-umtrx/handover.py/01_bts0_started: FAIL (336.8s)
16:21:13.606778 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]  [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]
16:21:13.614213 tst                   handover.py:43: Test FAILED (350.4 sec)
16:21:13.658213 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13434): Terminating (SIGTERM)
16:21:13.707079 run   osmo-hlr_10.42.42.2(pid=13435): Terminating (SIGTERM)
16:21:13.756851 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13436): Terminating (SIGTERM)
16:21:13.807787 run   osmo-stp_10.42.42.5(pid=13437): Terminating (SIGTERM)
16:21:13.857957 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13438): Terminating (SIGTERM)
16:21:13.910140 run   osmo-msc_10.42.42.6(pid=13439): Terminating (SIGTERM)
16:21:13.959416 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13440): Terminating (SIGTERM)
16:21:14.008620 run   osmo-mgw_10.42.42.3(pid=13441): Terminating (SIGTERM)
16:21:14.057479 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13442): Terminating (SIGTERM)
16:21:14.106051 run   osmo-mgw_10.42.42.4(pid=13443): Terminating (SIGTERM)
16:21:14.154750 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13444): Terminating (SIGTERM)
16:21:14.203299 run   osmo-bsc_10.42.42.7(pid=13445): Terminating (SIGTERM)
16:21:14.251987 run          osmo-bts-trx(pid=13449): Terminating (SIGTERM)
16:21:14.264706 ---      ParallelTerminationStrategy: PID 13434 died...
16:21:14.329425 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13434): Terminated: ok {rc=0}
16:21:14.341467 ---      ParallelTerminationStrategy: PID 13435 died...
16:21:14.405761 run   osmo-hlr_10.42.42.2(pid=13435): Terminated: ok {rc=0}
16:21:14.417305 ---      ParallelTerminationStrategy: PID 13436 died...
16:21:14.481412 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13436): Terminated: ok {rc=0}
16:21:14.493025 ---      ParallelTerminationStrategy: PID 13437 died...
16:21:14.557222 run   osmo-stp_10.42.42.5(pid=13437): Terminated: ok {rc=0}
16:21:14.568819 ---      ParallelTerminationStrategy: PID 13438 died...
16:21:14.632957 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13438): Terminated: ok {rc=0}
16:21:14.644625 ---      ParallelTerminationStrategy: PID 13440 died...
16:21:14.708726 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13440): Terminated: ok {rc=0}
16:21:14.720273 ---      ParallelTerminationStrategy: PID 13441 died...
16:21:14.784559 run   osmo-mgw_10.42.42.3(pid=13441): Terminated {rc=15}
16:21:14.796092 ---      ParallelTerminationStrategy: PID 13442 died...
16:21:14.860431 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13442): Terminated: ok {rc=0}
16:21:14.872020 ---      ParallelTerminationStrategy: PID 13443 died...
16:21:14.936663 run   osmo-mgw_10.42.42.4(pid=13443): Terminated {rc=15}
16:21:14.948282 ---      ParallelTerminationStrategy: PID 13444 died...
16:21:15.013024 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13444): Terminated: ok {rc=0}
16:21:17.711031 ---      ParallelTerminationStrategy: PID 13439 died...
16:21:17.781698 run   osmo-msc_10.42.42.6(pid=13439): Terminated: ok {rc=0}
16:21:17.794248 ---      ParallelTerminationStrategy: PID 13445 died...
16:21:17.862720 run   osmo-bsc_10.42.42.7(pid=13445): Terminated: ok {rc=0}
16:21:17.875215 ---      ParallelTerminationStrategy: PID 13449 died...
16:21:17.943833 run          osmo-bts-trx(pid=13449): Terminated: ok {rc=0}
16:21:18.048301 bus                        /sierra_3: Setting Powered False
16:21:19.175407 bus                        /sierra_3: Setting Powered False
16:21:20.308907 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]