Failed

handover_2G:trx-sysmocell5000+bts1-trx-umtrx.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-umtrx)

Failing for the past 23 builds (Since #140 )

Took 5 min 47 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]

Standard Output

----------------------------------------------
trial-162 handover_2G:trx-sysmocell5000+bts1-trx-umtrx handover.py
----------------------------------------------
11:07:42.845857 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
11:07:42.860451 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
11:07:43.021940 tst                      handover.py: using LAC 16297
11:07:43.153402 tst                      handover.py: using RAC 232
11:07:43.286201 tst                      handover.py: using CellId 16296
11:07:43.423288 tst                      handover.py: using BVCI 2975
11:07:43.449297 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
11:07:43.600047 tst                      handover.py: using LAC 16298
11:07:43.730301 tst                      handover.py: using RAC 233
11:07:43.860578 tst                      handover.py: using CellId 16297
11:07:43.990239 tst                      handover.py: using BVCI 2976
11:07:44.016051 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
11:07:44.030093 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
11:07:44.043775 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
11:07:44.057633 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
11:07:44.071253 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
11:07:44.084945 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
11:07:44.219304 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
11:07:44.448371 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
11:07:44.968537 run          create_hlr_db(pid=6601): Launched
11:07:45.119953 bus                        /sierra_2: Setting Powered False
11:07:46.204688 run          create_hlr_db(pid=6601): Terminated: ok {rc=0}
11:07:46.641788 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6603): Launched
11:07:46.998658 run    osmo-hlr_10.42.42.2(pid=6604): Launched
11:07:47.127716 run              osmo-stp_10.42.42.5: Starting osmo-stp
11:07:47.711693 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6605): Launched
11:07:48.064866 run    osmo-stp_10.42.42.5(pid=6606): Launched
11:07:48.185707 run              osmo-msc_10.42.42.6: Starting osmo-msc
11:07:48.778927 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6607): Launched
11:07:49.127052 run    osmo-msc_10.42.42.6(pid=6608): Launched
11:07:49.256822 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
11:07:49.830436 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6609): Launched
11:07:50.189784 run    osmo-mgw_10.42.42.3(pid=6610): Launched
11:07:50.317766 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
11:07:50.892501 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6611): Launched
11:07:51.249304 run    osmo-mgw_10.42.42.4(pid=6612): Launched
11:07:51.376595 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
11:07:52.157628 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6613): Launched
11:07:52.506213 run    osmo-bsc_10.42.42.7(pid=6614): Launched
11:07:56.231902 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
11:07:58.580850 run           ssh_sc5_ccli(pid=6615): Launched
11:08:17.816764 run           ssh_sc5_ccli(pid=6615): ERR: Terminated: ERROR {rc=255}  [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=6615)]
11:08:17.965961 run           ssh_sc5_ccli(pid=6615): stdout: 
| (launched: 2020-12-07_11:07:58.401624) 
11:08:18.116609 run           ssh_sc5_ccli(pid=6615): stderr: 
| (launched: 2020-12-07_11:07:58.457213)
| ssh: connect to host 10.42.42.112 port 22: No route to host 
11:08:18.244913 run                          sc5-trx: Configuring SC5 TRX failed, retrying 9 more times
11:08:23.612757 run           ssh_sc5_ccli(pid=6616): Launched
11:08:26.834881 run           ssh_sc5_ccli(pid=6616): Terminated: ok {rc=0}
11:08:26.966431 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
11:08:28.277964 run           osmo-bts-trx(pid=6617): Launched
11:13:29.064769 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-umtrx/handover.py/01_bts0_started: FAIL (332.9s)
11:13:29.073254 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]  [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]
11:13:29.081576 tst                   handover.py:43: Test FAILED (346.2 sec)
11:13:29.126079 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6603): Terminating (SIGTERM)
11:13:29.177370 run    osmo-hlr_10.42.42.2(pid=6604): Terminating (SIGTERM)
11:13:29.230519 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6605): Terminating (SIGTERM)
11:13:29.285911 run    osmo-stp_10.42.42.5(pid=6606): Terminating (SIGTERM)
11:13:29.339165 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6607): Terminating (SIGTERM)
11:13:29.395384 run    osmo-msc_10.42.42.6(pid=6608): Terminating (SIGTERM)
11:13:29.446843 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6609): Terminating (SIGTERM)
11:13:29.498133 run    osmo-mgw_10.42.42.3(pid=6610): Terminating (SIGTERM)
11:13:29.549290 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6611): Terminating (SIGTERM)
11:13:29.600092 run    osmo-mgw_10.42.42.4(pid=6612): Terminating (SIGTERM)
11:13:29.651193 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6613): Terminating (SIGTERM)
11:13:29.702388 run    osmo-bsc_10.42.42.7(pid=6614): Terminating (SIGTERM)
11:13:29.753415 run           osmo-bts-trx(pid=6617): Terminating (SIGTERM)
11:13:29.765046 ---      ParallelTerminationStrategy: PID 6603 died...
11:13:29.832193 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6603): Terminated: ok {rc=0}
11:13:29.844296 ---      ParallelTerminationStrategy: PID 6604 died...
11:13:29.911215 run    osmo-hlr_10.42.42.2(pid=6604): Terminated: ok {rc=0}
11:13:29.923259 ---      ParallelTerminationStrategy: PID 6605 died...
11:13:29.990108 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6605): Terminated: ok {rc=0}
11:13:30.002100 ---      ParallelTerminationStrategy: PID 6606 died...
11:13:30.068937 run    osmo-stp_10.42.42.5(pid=6606): Terminated: ok {rc=0}
11:13:30.081426 ---      ParallelTerminationStrategy: PID 6607 died...
11:13:30.148265 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6607): Terminated: ok {rc=0}
11:13:30.160414 ---      ParallelTerminationStrategy: PID 6609 died...
11:13:30.227293 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6609): Terminated: ok {rc=0}
11:13:30.240445 ---      ParallelTerminationStrategy: PID 6610 died...
11:13:30.307191 run    osmo-mgw_10.42.42.3(pid=6610): Terminated {rc=15}
11:13:30.319338 ---      ParallelTerminationStrategy: PID 6611 died...
11:13:30.386809 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6611): Terminated: ok {rc=0}
11:13:30.398962 ---      ParallelTerminationStrategy: PID 6612 died...
11:13:30.466018 run    osmo-mgw_10.42.42.4(pid=6612): Terminated {rc=15}
11:13:30.478100 ---      ParallelTerminationStrategy: PID 6613 died...
11:13:30.545250 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6613): Terminated: ok {rc=0}
11:13:32.574128 ---      ParallelTerminationStrategy: PID 6608 died...
11:13:32.643346 run    osmo-msc_10.42.42.6(pid=6608): Terminated: ok {rc=0}
11:13:33.322395 ---      ParallelTerminationStrategy: PID 6614 died...
11:13:33.392035 run    osmo-bsc_10.42.42.7(pid=6614): Terminated: ok {rc=0}
11:13:33.404174 ---      ParallelTerminationStrategy: PID 6617 died...
11:13:33.471201 run           osmo-bts-trx(pid=6617): Terminated: ok {rc=0}
11:13:33.574393 bus                        /sierra_3: Setting Powered False
11:13:34.698625 bus                        /sierra_3: Setting Powered False
11:13:35.830056 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]