Failed
handover_2G:trx-sysmocell5000+bts1-sysmo.handover.py (from handover_2G_trx-sysmocell5000+bts1-sysmo)
Stacktrace
handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]
Standard Output
---------------------------------------------- trial-161 handover_2G:trx-sysmocell5000+bts1-sysmo handover.py ---------------------------------------------- 08:31:20.771643 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 08:31:20.785834 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2) 08:31:20.945114 tst handover.py: using LAC 16221 08:31:21.076643 tst handover.py: using RAC 156 08:31:21.212784 tst handover.py: using CellId 16220 08:31:21.342980 tst handover.py: using BVCI 2899 08:31:21.368874 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2) 08:31:21.509170 tst handover.py: using LAC 16222 08:31:21.639388 tst handover.py: using RAC 157 08:31:21.769206 tst handover.py: using CellId 16221 08:31:21.898976 tst handover.py: using BVCI 2900 08:31:21.924715 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 08:31:21.938850 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 08:31:21.952864 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 08:31:21.966876 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 08:31:21.981117 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6) 08:31:21.994939 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2) 08:31:22.126064 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2) 08:31:22.353759 run osmo-hlr_10.42.42.2: Starting osmo-hlr 08:31:22.856454 run create_hlr_db(pid=1820): Launched 08:31:22.996613 bus /sierra_2: Setting Powered False 08:31:24.079141 run create_hlr_db(pid=1820): Terminated: ok {rc=0} 08:31:24.520478 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1822): Launched 08:31:24.867890 run osmo-hlr_10.42.42.2(pid=1823): Launched 08:31:24.996819 run osmo-stp_10.42.42.5: Starting osmo-stp 08:31:25.577892 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1824): Launched 08:31:25.924493 run osmo-stp_10.42.42.5(pid=1825): Launched 08:31:26.053342 run osmo-msc_10.42.42.6: Starting osmo-msc 08:31:26.704237 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1826): Launched 08:31:27.065099 run osmo-msc_10.42.42.6(pid=1827): Launched 08:31:27.199588 run osmo-mgw_10.42.42.3: Starting osmo-mgw 08:31:27.799965 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1828): Launched 08:31:28.138313 run osmo-mgw_10.42.42.3(pid=1829): Launched 08:31:28.258176 run osmo-mgw_10.42.42.4: Starting osmo-mgw 08:31:28.805964 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1830): Launched 08:31:29.143644 run osmo-mgw_10.42.42.4(pid=1831): Launched 08:31:29.263703 run osmo-bsc_10.42.42.7: Starting osmo-bsc 08:31:30.022154 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1832): Launched 08:31:30.375450 run osmo-bsc_10.42.42.7(pid=1833): Launched 08:31:34.063739 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7 08:31:36.377623 run ssh_sc5_ccli(pid=1834): Launched 08:31:55.613128 run ssh_sc5_ccli(pid=1834): ERR: Terminated: ERROR {rc=255} [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=1834)] 08:31:55.761930 run ssh_sc5_ccli(pid=1834): stdout: | (launched: 2020-12-07_08:31:36.201671) 08:31:55.911326 run ssh_sc5_ccli(pid=1834): stderr: | (launched: 2020-12-07_08:31:36.254756) | ssh: connect to host 10.42.42.112 port 22: No route to host 08:31:56.038361 run sc5-trx: Configuring SC5 TRX failed, retrying 9 more times 08:32:01.402211 run ssh_sc5_ccli(pid=1841): Launched 08:32:04.625438 run ssh_sc5_ccli(pid=1841): Terminated: ok {rc=0} 08:32:04.757715 run osmo-bts-trx: Waiting for sc5-trx to start up... 08:32:06.075644 run osmo-bts-trx(pid=1842): Launched 08:37:06.830591 tst handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-sysmo/handover.py/01_bts0_started: FAIL (332.9s) 08:37:06.838208 tst handover.py:43: ERR: Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43] [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43] 08:37:06.845636 tst handover.py:43: Test FAILED (346.1 sec) 08:37:06.889473 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1822): Terminating (SIGTERM) 08:37:06.938750 run osmo-hlr_10.42.42.2(pid=1823): Terminating (SIGTERM) 08:37:06.988885 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1824): Terminating (SIGTERM) 08:37:07.041178 run osmo-stp_10.42.42.5(pid=1825): Terminating (SIGTERM) 08:37:07.092825 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1826): Terminating (SIGTERM) 08:37:07.145473 run osmo-msc_10.42.42.6(pid=1827): Terminating (SIGTERM) 08:37:07.196871 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1828): Terminating (SIGTERM) 08:37:07.246888 run osmo-mgw_10.42.42.3(pid=1829): Terminating (SIGTERM) 08:37:07.297275 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1830): Terminating (SIGTERM) 08:37:07.346234 run osmo-mgw_10.42.42.4(pid=1831): Terminating (SIGTERM) 08:37:07.395407 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1832): Terminating (SIGTERM) 08:37:07.444374 run osmo-bsc_10.42.42.7(pid=1833): Terminating (SIGTERM) 08:37:07.493302 run osmo-bts-trx(pid=1842): Terminating (SIGTERM) 08:37:07.505661 --- ParallelTerminationStrategy: PID 1822 died... 08:37:07.570611 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1822): Terminated: ok {rc=0} 08:37:07.582455 --- ParallelTerminationStrategy: PID 1823 died... 08:37:07.646821 run osmo-hlr_10.42.42.2(pid=1823): Terminated: ok {rc=0} 08:37:07.658414 --- ParallelTerminationStrategy: PID 1824 died... 08:37:07.722880 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1824): Terminated: ok {rc=0} 08:37:07.734477 --- ParallelTerminationStrategy: PID 1825 died... 08:37:07.798623 run osmo-stp_10.42.42.5(pid=1825): Terminated: ok {rc=0} 08:37:07.810240 --- ParallelTerminationStrategy: PID 1826 died... 08:37:07.874495 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1826): Terminated: ok {rc=0} 08:37:07.886150 --- ParallelTerminationStrategy: PID 1828 died... 08:37:07.950416 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1828): Terminated: ok {rc=0} 08:37:07.962042 --- ParallelTerminationStrategy: PID 1829 died... 08:37:08.026388 run osmo-mgw_10.42.42.3(pid=1829): Terminated {rc=15} 08:37:08.038056 --- ParallelTerminationStrategy: PID 1830 died... 08:37:08.102935 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1830): Terminated: ok {rc=0} 08:37:08.114640 --- ParallelTerminationStrategy: PID 1831 died... 08:37:08.180377 run osmo-mgw_10.42.42.4(pid=1831): Terminated {rc=15} 08:37:08.192044 --- ParallelTerminationStrategy: PID 1832 died... 08:37:08.256770 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1832): Terminated: ok {rc=0} 08:37:10.285259 --- ParallelTerminationStrategy: PID 1827 died... 08:37:10.352658 run osmo-msc_10.42.42.6(pid=1827): Terminated: ok {rc=0} 08:37:11.031346 --- ParallelTerminationStrategy: PID 1833 died... 08:37:11.097747 run osmo-bsc_10.42.42.7(pid=1833): Terminated: ok {rc=0} 08:37:11.109461 --- ParallelTerminationStrategy: PID 1842 died... 08:37:11.174165 run osmo-bts-trx(pid=1842): Terminated: ok {rc=0} 08:37:11.280060 bus /sierra_3: Setting Powered False 08:37:12.401859 bus /sierra_3: Setting Powered False 08:37:13.532699 bus /sierra_2: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 673, in exec_module File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module> wait(bsc.bts_is_connected, bts0) File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]