Failed

handover_2G:trx-sysmocell5000+bts1-sysmo.handover.py (from handover_2G_trx-sysmocell5000+bts1-sysmo)

Failing for the past 22 builds (Since #140 )

Took 5 min 47 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]

Standard Output

----------------------------------------------
trial-161 handover_2G:trx-sysmocell5000+bts1-sysmo handover.py
----------------------------------------------
08:31:20.771643 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:31:20.785834 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2)
08:31:20.945114 tst                      handover.py: using LAC 16221
08:31:21.076643 tst                      handover.py: using RAC 156
08:31:21.212784 tst                      handover.py: using CellId 16220
08:31:21.342980 tst                      handover.py: using BVCI 2899
08:31:21.368874 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2)
08:31:21.509170 tst                      handover.py: using LAC 16222
08:31:21.639388 tst                      handover.py: using RAC 157
08:31:21.769206 tst                      handover.py: using CellId 16221
08:31:21.898976 tst                      handover.py: using BVCI 2900
08:31:21.924715 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:31:21.938850 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:31:21.952864 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:31:21.966876 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:31:21.981117 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:31:21.994939 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2)
08:31:22.126064 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2)
08:31:22.353759 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
08:31:22.856454 run          create_hlr_db(pid=1820): Launched
08:31:22.996613 bus                        /sierra_2: Setting Powered False
08:31:24.079141 run          create_hlr_db(pid=1820): Terminated: ok {rc=0}
08:31:24.520478 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1822): Launched
08:31:24.867890 run    osmo-hlr_10.42.42.2(pid=1823): Launched
08:31:24.996819 run              osmo-stp_10.42.42.5: Starting osmo-stp
08:31:25.577892 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1824): Launched
08:31:25.924493 run    osmo-stp_10.42.42.5(pid=1825): Launched
08:31:26.053342 run              osmo-msc_10.42.42.6: Starting osmo-msc
08:31:26.704237 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1826): Launched
08:31:27.065099 run    osmo-msc_10.42.42.6(pid=1827): Launched
08:31:27.199588 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
08:31:27.799965 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1828): Launched
08:31:28.138313 run    osmo-mgw_10.42.42.3(pid=1829): Launched
08:31:28.258176 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
08:31:28.805964 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1830): Launched
08:31:29.143644 run    osmo-mgw_10.42.42.4(pid=1831): Launched
08:31:29.263703 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
08:31:30.022154 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1832): Launched
08:31:30.375450 run    osmo-bsc_10.42.42.7(pid=1833): Launched
08:31:34.063739 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
08:31:36.377623 run           ssh_sc5_ccli(pid=1834): Launched
08:31:55.613128 run           ssh_sc5_ccli(pid=1834): ERR: Terminated: ERROR {rc=255}  [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=1834)]
08:31:55.761930 run           ssh_sc5_ccli(pid=1834): stdout: 
| (launched: 2020-12-07_08:31:36.201671) 
08:31:55.911326 run           ssh_sc5_ccli(pid=1834): stderr: 
| (launched: 2020-12-07_08:31:36.254756)
| ssh: connect to host 10.42.42.112 port 22: No route to host 
08:31:56.038361 run                          sc5-trx: Configuring SC5 TRX failed, retrying 9 more times
08:32:01.402211 run           ssh_sc5_ccli(pid=1841): Launched
08:32:04.625438 run           ssh_sc5_ccli(pid=1841): Terminated: ok {rc=0}
08:32:04.757715 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
08:32:06.075644 run           osmo-bts-trx(pid=1842): Launched
08:37:06.830591 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-sysmo/handover.py/01_bts0_started: FAIL (332.9s)
08:37:06.838208 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]  [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]
08:37:06.845636 tst                   handover.py:43: Test FAILED (346.1 sec)
08:37:06.889473 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1822): Terminating (SIGTERM)
08:37:06.938750 run    osmo-hlr_10.42.42.2(pid=1823): Terminating (SIGTERM)
08:37:06.988885 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1824): Terminating (SIGTERM)
08:37:07.041178 run    osmo-stp_10.42.42.5(pid=1825): Terminating (SIGTERM)
08:37:07.092825 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1826): Terminating (SIGTERM)
08:37:07.145473 run    osmo-msc_10.42.42.6(pid=1827): Terminating (SIGTERM)
08:37:07.196871 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1828): Terminating (SIGTERM)
08:37:07.246888 run    osmo-mgw_10.42.42.3(pid=1829): Terminating (SIGTERM)
08:37:07.297275 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1830): Terminating (SIGTERM)
08:37:07.346234 run    osmo-mgw_10.42.42.4(pid=1831): Terminating (SIGTERM)
08:37:07.395407 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1832): Terminating (SIGTERM)
08:37:07.444374 run    osmo-bsc_10.42.42.7(pid=1833): Terminating (SIGTERM)
08:37:07.493302 run           osmo-bts-trx(pid=1842): Terminating (SIGTERM)
08:37:07.505661 ---      ParallelTerminationStrategy: PID 1822 died...
08:37:07.570611 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1822): Terminated: ok {rc=0}
08:37:07.582455 ---      ParallelTerminationStrategy: PID 1823 died...
08:37:07.646821 run    osmo-hlr_10.42.42.2(pid=1823): Terminated: ok {rc=0}
08:37:07.658414 ---      ParallelTerminationStrategy: PID 1824 died...
08:37:07.722880 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1824): Terminated: ok {rc=0}
08:37:07.734477 ---      ParallelTerminationStrategy: PID 1825 died...
08:37:07.798623 run    osmo-stp_10.42.42.5(pid=1825): Terminated: ok {rc=0}
08:37:07.810240 ---      ParallelTerminationStrategy: PID 1826 died...
08:37:07.874495 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1826): Terminated: ok {rc=0}
08:37:07.886150 ---      ParallelTerminationStrategy: PID 1828 died...
08:37:07.950416 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1828): Terminated: ok {rc=0}
08:37:07.962042 ---      ParallelTerminationStrategy: PID 1829 died...
08:37:08.026388 run    osmo-mgw_10.42.42.3(pid=1829): Terminated {rc=15}
08:37:08.038056 ---      ParallelTerminationStrategy: PID 1830 died...
08:37:08.102935 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1830): Terminated: ok {rc=0}
08:37:08.114640 ---      ParallelTerminationStrategy: PID 1831 died...
08:37:08.180377 run    osmo-mgw_10.42.42.4(pid=1831): Terminated {rc=15}
08:37:08.192044 ---      ParallelTerminationStrategy: PID 1832 died...
08:37:08.256770 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1832): Terminated: ok {rc=0}
08:37:10.285259 ---      ParallelTerminationStrategy: PID 1827 died...
08:37:10.352658 run    osmo-msc_10.42.42.6(pid=1827): Terminated: ok {rc=0}
08:37:11.031346 ---      ParallelTerminationStrategy: PID 1833 died...
08:37:11.097747 run    osmo-bsc_10.42.42.7(pid=1833): Terminated: ok {rc=0}
08:37:11.109461 ---      ParallelTerminationStrategy: PID 1842 died...
08:37:11.174165 run           osmo-bts-trx(pid=1842): Terminated: ok {rc=0}
08:37:11.280060 bus                        /sierra_3: Setting Powered False
08:37:12.401859 bus                        /sierra_3: Setting Powered False
08:37:13.532699 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]