Failed

handover_2G:trx-sysmocell5000+bts1-trx-b200.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-b200)

Failing for the past 22 builds (Since #140 )

Took 5 min 50 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]

Standard Output

----------------------------------------------
trial-161 handover_2G:trx-sysmocell5000+bts1-trx-b200 handover.py
----------------------------------------------
08:37:15.965896 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:37:15.979983 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
08:37:16.137890 tst                      handover.py: using LAC 16223
08:37:16.269062 tst                      handover.py: using RAC 158
08:37:16.397440 tst                      handover.py: using CellId 16222
08:37:16.525692 tst                      handover.py: using BVCI 2901
08:37:16.551325 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
08:37:16.702342 tst                      handover.py: using LAC 16224
08:37:16.831470 tst                      handover.py: using RAC 159
08:37:16.959799 tst                      handover.py: using CellId 16223
08:37:17.089018 tst                      handover.py: using BVCI 2902
08:37:17.114743 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:37:17.128773 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:37:17.142720 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:37:17.156797 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:37:17.170711 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:37:17.184616 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
08:37:17.312776 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
08:37:17.550498 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
08:37:18.075156 run          create_hlr_db(pid=1864): Launched
08:37:18.214940 bus                        /sierra_2: Setting Powered False
08:37:19.297834 run          create_hlr_db(pid=1864): Terminated: ok {rc=0}
08:37:19.737103 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1866): Launched
08:37:20.084730 run    osmo-hlr_10.42.42.2(pid=1867): Launched
08:37:20.213954 run              osmo-stp_10.42.42.5: Starting osmo-stp
08:37:20.795015 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1868): Launched
08:37:21.143712 run    osmo-stp_10.42.42.5(pid=1869): Launched
08:37:21.271005 run              osmo-msc_10.42.42.6: Starting osmo-msc
08:37:21.891674 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1870): Launched
08:37:22.231892 run    osmo-msc_10.42.42.6(pid=1871): Launched
08:37:22.353891 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
08:37:22.918932 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1872): Launched
08:37:23.259188 run    osmo-mgw_10.42.42.3(pid=1873): Launched
08:37:23.378573 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
08:37:23.929958 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1874): Launched
08:37:24.269222 run    osmo-mgw_10.42.42.4(pid=1875): Launched
08:37:24.389597 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
08:37:25.152093 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1876): Launched
08:37:25.507098 run    osmo-bsc_10.42.42.7(pid=1877): Launched
08:37:29.199702 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
08:37:31.521548 run           ssh_sc5_ccli(pid=1878): Launched
08:38:03.786646 run           ssh_sc5_ccli(pid=1878): Terminated: ok {rc=0}
08:38:03.919703 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
08:38:05.239083 run           osmo-bts-trx(pid=1879): Launched
08:43:05.856101 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-b200/handover.py/01_bts0_started: FAIL (336.7s)
08:43:05.863439 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]  [trial-161↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]
08:43:05.870583 tst                   handover.py:43: Test FAILED (349.9 sec)
08:43:05.914358 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1866): Terminating (SIGTERM)
08:43:05.963441 run    osmo-hlr_10.42.42.2(pid=1867): Terminating (SIGTERM)
08:43:06.013532 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1868): Terminating (SIGTERM)
08:43:06.063509 run    osmo-stp_10.42.42.5(pid=1869): Terminating (SIGTERM)
08:43:06.113332 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1870): Terminating (SIGTERM)
08:43:06.164764 run    osmo-msc_10.42.42.6(pid=1871): Terminating (SIGTERM)
08:43:06.214820 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1872): Terminating (SIGTERM)
08:43:06.263517 run    osmo-mgw_10.42.42.3(pid=1873): Terminating (SIGTERM)
08:43:06.314056 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1874): Terminating (SIGTERM)
08:43:06.363097 run    osmo-mgw_10.42.42.4(pid=1875): Terminating (SIGTERM)
08:43:06.412241 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1876): Terminating (SIGTERM)
08:43:06.461147 run    osmo-bsc_10.42.42.7(pid=1877): Terminating (SIGTERM)
08:43:06.510121 run           osmo-bts-trx(pid=1879): Terminating (SIGTERM)
08:43:06.522417 ---      ParallelTerminationStrategy: PID 1866 died...
08:43:06.594114 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1866): Terminated: ok {rc=0}
08:43:06.606487 ---      ParallelTerminationStrategy: PID 1867 died...
08:43:06.674321 run    osmo-hlr_10.42.42.2(pid=1867): Terminated: ok {rc=0}
08:43:06.686442 ---      ParallelTerminationStrategy: PID 1868 died...
08:43:06.754351 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1868): Terminated: ok {rc=0}
08:43:06.766525 ---      ParallelTerminationStrategy: PID 1869 died...
08:43:06.834476 run    osmo-stp_10.42.42.5(pid=1869): Terminated: ok {rc=0}
08:43:06.846687 ---      ParallelTerminationStrategy: PID 1870 died...
08:43:06.915108 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1870): Terminated: ok {rc=0}
08:43:06.927285 ---      ParallelTerminationStrategy: PID 1872 died...
08:43:06.995631 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1872): Terminated: ok {rc=0}
08:43:07.007868 ---      ParallelTerminationStrategy: PID 1873 died...
08:43:07.076296 run    osmo-mgw_10.42.42.3(pid=1873): Terminated {rc=15}
08:43:07.088610 ---      ParallelTerminationStrategy: PID 1874 died...
08:43:07.156968 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1874): Terminated: ok {rc=0}
08:43:07.169398 ---      ParallelTerminationStrategy: PID 1875 died...
08:43:07.237925 run    osmo-mgw_10.42.42.4(pid=1875): Terminated {rc=15}
08:43:07.250434 ---      ParallelTerminationStrategy: PID 1876 died...
08:43:07.319234 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1876): Terminated: ok {rc=0}
08:43:09.348305 ---      ParallelTerminationStrategy: PID 1871 died...
08:43:09.419653 run    osmo-msc_10.42.42.6(pid=1871): Terminated: ok {rc=0}
08:43:10.098847 ---      ParallelTerminationStrategy: PID 1877 died...
08:43:10.168891 run    osmo-bsc_10.42.42.7(pid=1877): Terminated: ok {rc=0}
08:43:10.181238 ---      ParallelTerminationStrategy: PID 1879 died...
08:43:10.250134 run           osmo-bts-trx(pid=1879): Terminated: ok {rc=0}
08:43:10.354892 bus                        /sierra_3: Setting Powered False
08:43:11.477012 bus                        /sierra_3: Setting Powered False
08:43:12.607519 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]