Failed

handover_2G:trx-sysmocell5000+bts1-trx-b200.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-b200)

Failing for the past 25 builds (Since #140 )

Took 5 min 47 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]

Standard Output

----------------------------------------------
trial-164 handover_2G:trx-sysmocell5000+bts1-trx-b200 handover.py
----------------------------------------------
16:09:27.528396 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
16:09:27.542868 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
16:09:27.707664 tst                      handover.py: using LAC 16440
16:09:27.845573 tst                      handover.py: using RAC 120
16:09:27.980965 tst                      handover.py: using CellId 16439
16:09:28.118895 tst                      handover.py: using BVCI 3118
16:09:28.145756 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
16:09:28.301972 tst                      handover.py: using LAC 16441
16:09:28.438230 tst                      handover.py: using RAC 121
16:09:28.575678 tst                      handover.py: using CellId 16440
16:09:28.718931 tst                      handover.py: using BVCI 3119
16:09:28.747417 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
16:09:28.761832 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
16:09:28.776185 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
16:09:28.790768 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
16:09:28.805335 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
16:09:28.820241 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
16:09:28.952858 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
16:09:29.185016 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
16:09:29.723249 run         create_hlr_db(pid=13330): Launched
16:09:29.879639 bus                        /sierra_2: Setting Powered False
16:09:30.969010 run         create_hlr_db(pid=13330): Terminated: ok {rc=0}
16:09:31.420122 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13332): Launched
16:09:31.797288 run   osmo-hlr_10.42.42.2(pid=13333): Launched
16:09:31.931676 run              osmo-stp_10.42.42.5: Starting osmo-stp
16:09:32.549996 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13334): Launched
16:09:32.923680 run   osmo-stp_10.42.42.5(pid=13335): Launched
16:09:33.056633 run              osmo-msc_10.42.42.6: Starting osmo-msc
16:09:33.709213 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13336): Launched
16:09:34.078897 run   osmo-msc_10.42.42.6(pid=13337): Launched
16:09:34.206838 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
16:09:34.793764 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13338): Launched
16:09:35.143906 run   osmo-mgw_10.42.42.3(pid=13339): Launched
16:09:35.267775 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
16:09:35.836532 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13340): Launched
16:09:36.185964 run   osmo-mgw_10.42.42.4(pid=13341): Launched
16:09:36.311180 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
16:09:37.086148 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13342): Launched
16:09:37.437356 run   osmo-bsc_10.42.42.7(pid=13343): Launched
16:09:41.153116 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
16:09:43.488565 run          ssh_sc5_ccli(pid=13344): Launched
16:10:02.729768 run          ssh_sc5_ccli(pid=13344): ERR: Terminated: ERROR {rc=255}  [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=13344)]
16:10:02.882920 run          ssh_sc5_ccli(pid=13344): stdout: 
| (launched: 2020-12-07_16:09:43.310034) 
16:10:03.036959 run          ssh_sc5_ccli(pid=13344): stderr: 
| (launched: 2020-12-07_16:09:43.364857)
| ssh: connect to host 10.42.42.112 port 22: No route to host 
16:10:03.167854 run                          sc5-trx: Configuring SC5 TRX failed, retrying 9 more times
16:10:08.557930 run          ssh_sc5_ccli(pid=13348): Launched
16:10:11.790019 run          ssh_sc5_ccli(pid=13348): Terminated: ok {rc=0}
16:10:11.928892 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
16:10:13.254945 run          osmo-bts-trx(pid=13349): Launched
16:15:14.032149 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-b200/handover.py/01_bts0_started: FAIL (333.0s)
16:15:14.039977 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]  [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]
16:15:14.048402 tst                   handover.py:43: Test FAILED (346.5 sec)
16:15:14.093475 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13332): Terminating (SIGTERM)
16:15:14.144945 run   osmo-hlr_10.42.42.2(pid=13333): Terminating (SIGTERM)
16:15:14.198113 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13334): Terminating (SIGTERM)
16:15:14.255057 run   osmo-stp_10.42.42.5(pid=13335): Terminating (SIGTERM)
16:15:14.308720 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13336): Terminating (SIGTERM)
16:15:14.365460 run   osmo-msc_10.42.42.6(pid=13337): Terminating (SIGTERM)
16:15:14.418211 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13338): Terminating (SIGTERM)
16:15:14.470012 run   osmo-mgw_10.42.42.3(pid=13339): Terminating (SIGTERM)
16:15:14.527427 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13340): Terminating (SIGTERM)
16:15:14.576558 run   osmo-mgw_10.42.42.4(pid=13341): Terminating (SIGTERM)
16:15:14.625802 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13342): Terminating (SIGTERM)
16:15:14.675105 run   osmo-bsc_10.42.42.7(pid=13343): Terminating (SIGTERM)
16:15:14.724286 run          osmo-bts-trx(pid=13349): Terminating (SIGTERM)
16:15:14.735549 ---      ParallelTerminationStrategy: PID 13332 died...
16:15:14.800483 run pcap-recorder_any(filters='host 10.42.42.2')(pid=13332): Terminated: ok {rc=0}
16:15:14.812314 ---      ParallelTerminationStrategy: PID 13333 died...
16:15:14.877173 run   osmo-hlr_10.42.42.2(pid=13333): Terminated: ok {rc=0}
16:15:14.888815 ---      ParallelTerminationStrategy: PID 13334 died...
16:15:14.952707 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=13334): Terminated: ok {rc=0}
16:15:14.964285 ---      ParallelTerminationStrategy: PID 13335 died...
16:15:15.028268 run   osmo-stp_10.42.42.5(pid=13335): Terminated: ok {rc=0}
16:15:15.039901 ---      ParallelTerminationStrategy: PID 13336 died...
16:15:15.103950 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=13336): Terminated: ok {rc=0}
16:15:15.115580 ---      ParallelTerminationStrategy: PID 13338 died...
16:15:15.180051 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=13338): Terminated: ok {rc=0}
16:15:15.191676 ---      ParallelTerminationStrategy: PID 13339 died...
16:15:15.255469 run   osmo-mgw_10.42.42.3(pid=13339): Terminated {rc=15}
16:15:15.267054 ---      ParallelTerminationStrategy: PID 13340 died...
16:15:15.331265 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=13340): Terminated: ok {rc=0}
16:15:15.342932 ---      ParallelTerminationStrategy: PID 13341 died...
16:15:15.406960 run   osmo-mgw_10.42.42.4(pid=13341): Terminated {rc=15}
16:15:15.418558 ---      ParallelTerminationStrategy: PID 13342 died...
16:15:15.482657 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=13342): Terminated: ok {rc=0}
16:15:17.511051 ---      ParallelTerminationStrategy: PID 13337 died...
16:15:17.577557 run   osmo-msc_10.42.42.6(pid=13337): Terminated: ok {rc=0}
16:15:18.256148 ---      ParallelTerminationStrategy: PID 13343 died...
16:15:18.324048 run   osmo-bsc_10.42.42.7(pid=13343): Terminated: ok {rc=0}
16:15:18.335758 ---      ParallelTerminationStrategy: PID 13349 died...
16:15:18.400110 run          osmo-bts-trx(pid=13349): Terminated: ok {rc=0}
16:15:18.505646 bus                        /sierra_3: Setting Powered False
16:15:19.627139 bus                        /sierra_3: Setting Powered False
16:15:20.757145 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-164↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]