Failed

handover_2G:trx-sysmocell5000+bts1-trx-umtrx.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-umtrx)

Failing for the past 20 builds (Since #140 )

Took 5 min 47 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]

Standard Output

----------------------------------------------
trial-159 handover_2G:trx-sysmocell5000+bts1-trx-umtrx handover.py
----------------------------------------------
01:18:42.994259 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
01:18:43.008789 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
01:18:43.170492 tst                      handover.py: using LAC 16057
01:18:43.305106 tst                      handover.py: using RAC 247
01:18:43.439999 tst                      handover.py: using CellId 16056
01:18:43.577571 tst                      handover.py: using BVCI 2735
01:18:43.605311 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
01:18:43.769065 tst                      handover.py: using LAC 16058
01:18:43.912195 tst                      handover.py: using RAC 248
01:18:44.050585 tst                      handover.py: using CellId 16057
01:18:44.188511 tst                      handover.py: using BVCI 2736
01:18:44.216033 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
01:18:44.230532 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
01:18:44.246024 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
01:18:44.261092 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
01:18:44.275662 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
01:18:44.290332 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
01:18:44.422537 tst handover_2G:trx-sysmocell5000+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
01:18:44.657569 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
01:18:45.186278 run         create_hlr_db(pid=15669): Launched
01:18:45.329476 bus                        /sierra_2: Setting Powered False
01:18:46.410154 run         create_hlr_db(pid=15669): Terminated: ok {rc=0}
01:18:46.856656 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15671): Launched
01:18:47.198601 run   osmo-hlr_10.42.42.2(pid=15672): Launched
01:18:47.321421 run              osmo-stp_10.42.42.5: Starting osmo-stp
01:18:47.892310 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15673): Launched
01:18:48.255231 run   osmo-stp_10.42.42.5(pid=15674): Launched
01:18:48.384454 run              osmo-msc_10.42.42.6: Starting osmo-msc
01:18:49.013734 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15675): Launched
01:18:49.374324 run   osmo-msc_10.42.42.6(pid=15676): Launched
01:18:49.499190 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
01:18:50.080497 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15677): Launched
01:18:50.445805 run   osmo-mgw_10.42.42.3(pid=15678): Launched
01:18:50.575037 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
01:18:51.154202 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15679): Launched
01:18:51.500977 run   osmo-mgw_10.42.42.4(pid=15680): Launched
01:18:51.622936 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
01:18:52.382600 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15681): Launched
01:18:52.734423 run   osmo-bsc_10.42.42.7(pid=15682): Launched
01:18:56.477098 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
01:18:58.822053 run          ssh_sc5_ccli(pid=15686): Launched
01:19:18.070487 run          ssh_sc5_ccli(pid=15686): ERR: Terminated: ERROR {rc=255}  [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=15686)]
01:19:18.230531 run          ssh_sc5_ccli(pid=15686): stdout: 
| (launched: 2020-12-07_01:18:58.644495) 
01:19:18.390204 run          ssh_sc5_ccli(pid=15686): stderr: 
| (launched: 2020-12-07_01:18:58.697991)
| ssh: connect to host 10.42.42.112 port 22: No route to host 
01:19:18.526019 run                          sc5-trx: Configuring SC5 TRX failed, retrying 9 more times
01:19:23.908682 run          ssh_sc5_ccli(pid=15687): Launched
01:19:27.133716 run          ssh_sc5_ccli(pid=15687): Terminated: ok {rc=0}
01:19:27.267819 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
01:19:28.582175 run          osmo-bts-trx(pid=15688): Launched
01:24:29.335722 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-umtrx/handover.py/01_bts0_started: FAIL (333.0s)
01:24:29.343045 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]  [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]
01:24:29.349882 tst                   handover.py:43: Test FAILED (346.3 sec)
01:24:29.391551 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15671): Terminating (SIGTERM)
01:24:29.439983 run   osmo-hlr_10.42.42.2(pid=15672): Terminating (SIGTERM)
01:24:29.489553 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15673): Terminating (SIGTERM)
01:24:29.540290 run   osmo-stp_10.42.42.5(pid=15674): Terminating (SIGTERM)
01:24:29.590513 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15675): Terminating (SIGTERM)
01:24:29.641503 run   osmo-msc_10.42.42.6(pid=15676): Terminating (SIGTERM)
01:24:29.690055 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15677): Terminating (SIGTERM)
01:24:29.738377 run   osmo-mgw_10.42.42.3(pid=15678): Terminating (SIGTERM)
01:24:29.786825 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15679): Terminating (SIGTERM)
01:24:29.835115 run   osmo-mgw_10.42.42.4(pid=15680): Terminating (SIGTERM)
01:24:29.884089 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15681): Terminating (SIGTERM)
01:24:29.932675 run   osmo-bsc_10.42.42.7(pid=15682): Terminating (SIGTERM)
01:24:29.981162 run          osmo-bts-trx(pid=15688): Terminating (SIGTERM)
01:24:29.992205 ---      ParallelTerminationStrategy: PID 15671 died...
01:24:30.056383 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15671): Terminated: ok {rc=0}
01:24:30.067896 ---      ParallelTerminationStrategy: PID 15672 died...
01:24:30.131896 run   osmo-hlr_10.42.42.2(pid=15672): Terminated: ok {rc=0}
01:24:30.143460 ---      ParallelTerminationStrategy: PID 15673 died...
01:24:30.207659 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15673): Terminated: ok {rc=0}
01:24:30.219269 ---      ParallelTerminationStrategy: PID 15674 died...
01:24:30.283435 run   osmo-stp_10.42.42.5(pid=15674): Terminated: ok {rc=0}
01:24:30.295007 ---      ParallelTerminationStrategy: PID 15675 died...
01:24:30.359166 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=15675): Terminated: ok {rc=0}
01:24:30.370761 ---      ParallelTerminationStrategy: PID 15677 died...
01:24:30.434948 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=15677): Terminated: ok {rc=0}
01:24:30.446570 ---      ParallelTerminationStrategy: PID 15678 died...
01:24:30.510644 run   osmo-mgw_10.42.42.3(pid=15678): Terminated {rc=15}
01:24:30.522252 ---      ParallelTerminationStrategy: PID 15679 died...
01:24:30.586395 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=15679): Terminated: ok {rc=0}
01:24:30.598096 ---      ParallelTerminationStrategy: PID 15680 died...
01:24:30.662579 run   osmo-mgw_10.42.42.4(pid=15680): Terminated {rc=15}
01:24:30.674250 ---      ParallelTerminationStrategy: PID 15681 died...
01:24:30.738351 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=15681): Terminated: ok {rc=0}
01:24:33.433282 ---      ParallelTerminationStrategy: PID 15676 died...
01:24:33.499515 run   osmo-msc_10.42.42.6(pid=15676): Terminated: ok {rc=0}
01:24:33.511189 ---      ParallelTerminationStrategy: PID 15682 died...
01:24:33.575622 run   osmo-bsc_10.42.42.7(pid=15682): Terminated: ok {rc=0}
01:24:33.587317 ---      ParallelTerminationStrategy: PID 15688 died...
01:24:33.651484 run          osmo-bts-trx(pid=15688): Terminated: ok {rc=0}
01:24:33.758303 bus                        /sierra_3: Setting Powered False
01:24:34.880849 bus                        /sierra_3: Setting Powered False
01:24:36.009797 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-159↪handover_2G:trx-sysmocell5000+bts1-trx-umtrx↪handover.py:43]