![[Jenkins]](/jenkins/static/342c126a/images/svgs/logo.svg){kind=logo}
Failed
handover_2G:trx-sysmocell5000+bts1-trx-b200.handover.py (from handover_2G_trx-sysmocell5000+bts1-trx-b200)
Stacktrace
handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]Standard Output
----------------------------------------------
trial-162 handover_2G:trx-sysmocell5000+bts1-trx-b200 handover.py
----------------------------------------------
11:01:47.269620 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
11:01:47.283455 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
11:01:47.441003 tst handover.py: using LAC 16295
11:01:47.581196 tst handover.py: using RAC 230
11:01:47.722346 tst handover.py: using CellId 16294
11:01:47.860153 tst handover.py: using BVCI 2973
11:01:47.888093 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x bts (candidates: 2)
11:01:48.048715 tst handover.py: using LAC 16296
11:01:48.189745 tst handover.py: using RAC 231
11:01:48.328939 tst handover.py: using CellId 16295
11:01:48.486116 tst handover.py: using BVCI 2974
11:01:48.512213 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
11:01:48.526253 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
11:01:48.540524 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
11:01:48.554351 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
11:01:48.568017 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
11:01:48.581629 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
11:01:48.709931 tst handover_2G:trx-sysmocell5000+bts1-trx-b200: Using 1 x modem (candidates: 2)
11:01:48.943845 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:01:49.466791 run create_hlr_db(pid=6557): Launched
11:01:49.607848 bus /sierra_2: Setting Powered False
11:01:50.698496 run create_hlr_db(pid=6557): Terminated: ok {rc=0}
11:01:51.129999 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6559): Launched
11:01:51.471623 run osmo-hlr_10.42.42.2(pid=6560): Launched
11:01:51.592817 run osmo-stp_10.42.42.5: Starting osmo-stp
11:01:52.151222 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6561): Launched
11:01:52.491806 run osmo-stp_10.42.42.5(pid=6562): Launched
11:01:52.612714 run osmo-msc_10.42.42.6: Starting osmo-msc
11:01:53.205208 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6563): Launched
11:01:53.555874 run osmo-msc_10.42.42.6(pid=6564): Launched
11:01:53.693417 run osmo-mgw_10.42.42.3: Starting osmo-mgw
11:01:54.296519 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6565): Launched
11:01:54.646840 run osmo-mgw_10.42.42.3(pid=6566): Launched
11:01:54.773852 run osmo-mgw_10.42.42.4: Starting osmo-mgw
11:01:55.345648 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6567): Launched
11:01:55.694300 run osmo-mgw_10.42.42.4(pid=6568): Launched
11:01:55.822037 run osmo-bsc_10.42.42.7: Starting osmo-bsc
11:01:56.609414 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6570): Launched
11:01:56.960041 run osmo-bsc_10.42.42.7(pid=6571): Launched
11:02:00.688038 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
11:02:03.030441 run ssh_sc5_ccli(pid=6572): Launched
11:02:06.242359 run ssh_sc5_ccli(pid=6572): ERR: Terminated: ERROR {rc=255} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=6572)]
11:02:06.391938 run ssh_sc5_ccli(pid=6572): stdout:
| (launched: 2020-12-07_11:02:02.851806)
11:02:06.541691 run ssh_sc5_ccli(pid=6572): stderr:
| (launched: 2020-12-07_11:02:02.907661)
| ssh: connect to host 10.42.42.112 port 22: No route to host
11:02:06.669297 run sc5-trx: Configuring SC5 TRX failed, retrying 9 more times
11:02:12.036951 run ssh_sc5_ccli(pid=6573): Launched
11:02:15.259698 run ssh_sc5_ccli(pid=6573): ERR: Terminated: ERROR {rc=255} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=6573)]
11:02:15.416430 run ssh_sc5_ccli(pid=6573): stdout:
| (launched: 2020-12-07_11:02:11.854651)
11:02:15.572704 run ssh_sc5_ccli(pid=6573): stderr:
| (launched: 2020-12-07_11:02:11.908133)
| ssh: connect to host 10.42.42.112 port 22: No route to host
11:02:15.706044 run sc5-trx: Configuring SC5 TRX failed, retrying 8 more times
11:02:21.078364 run ssh_sc5_ccli(pid=6574): Launched
11:02:24.302110 run ssh_sc5_ccli(pid=6574): ERR: Terminated: ERROR {rc=255} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=6574)]
11:02:24.459030 run ssh_sc5_ccli(pid=6574): stdout:
| (launched: 2020-12-07_11:02:20.896598)
11:02:24.615482 run ssh_sc5_ccli(pid=6574): stderr:
| (launched: 2020-12-07_11:02:20.951960)
| ssh: connect to host 10.42.42.112 port 22: No route to host
11:02:24.748811 run sc5-trx: Configuring SC5 TRX failed, retrying 7 more times
11:02:30.124400 run ssh_sc5_ccli(pid=6575): Launched
11:02:31.343707 run ssh_sc5_ccli(pid=6575): Terminated: ok {rc=0}
11:02:31.475664 run osmo-bts-trx: Waiting for sc5-trx to start up...
11:02:32.788099 run osmo-bts-trx(pid=6576): Launched
11:07:33.594724 tst handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-trx-b200/handover.py/01_bts0_started: FAIL (333.0s)
11:07:33.603146 tst handover.py:43: ERR: Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43] [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]
11:07:33.611259 tst handover.py:43: Test FAILED (346.3 sec)
11:07:33.655912 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6559): Terminating (SIGTERM)
11:07:33.707880 run osmo-hlr_10.42.42.2(pid=6560): Terminating (SIGTERM)
11:07:33.760781 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6561): Terminating (SIGTERM)
11:07:33.816028 run osmo-stp_10.42.42.5(pid=6562): Terminating (SIGTERM)
11:07:33.870846 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6563): Terminating (SIGTERM)
11:07:33.929246 run osmo-msc_10.42.42.6(pid=6564): Terminating (SIGTERM)
11:07:33.981729 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6565): Terminating (SIGTERM)
11:07:34.032703 run osmo-mgw_10.42.42.3(pid=6566): Terminating (SIGTERM)
11:07:34.085364 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6567): Terminating (SIGTERM)
11:07:34.137462 run osmo-mgw_10.42.42.4(pid=6568): Terminating (SIGTERM)
11:07:34.189544 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6570): Terminating (SIGTERM)
11:07:34.242847 run osmo-bsc_10.42.42.7(pid=6571): Terminating (SIGTERM)
11:07:34.294585 run osmo-bts-trx(pid=6576): Terminating (SIGTERM)
11:07:34.307441 --- ParallelTerminationStrategy: PID 6559 died...
11:07:34.376043 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6559): Terminated: ok {rc=0}
11:07:34.388269 --- ParallelTerminationStrategy: PID 6560 died...
11:07:34.457179 run osmo-hlr_10.42.42.2(pid=6560): Terminated: ok {rc=0}
11:07:34.469452 --- ParallelTerminationStrategy: PID 6561 died...
11:07:34.538076 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6561): Terminated: ok {rc=0}
11:07:34.550519 --- ParallelTerminationStrategy: PID 6562 died...
11:07:34.618857 run osmo-stp_10.42.42.5(pid=6562): Terminated: ok {rc=0}
11:07:34.631033 --- ParallelTerminationStrategy: PID 6563 died...
11:07:34.698176 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6563): Terminated: ok {rc=0}
11:07:34.710460 --- ParallelTerminationStrategy: PID 6565 died...
11:07:34.777961 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6565): Terminated: ok {rc=0}
11:07:34.791111 --- ParallelTerminationStrategy: PID 6566 died...
11:07:34.860132 run osmo-mgw_10.42.42.3(pid=6566): Terminated {rc=15}
11:07:34.872635 --- ParallelTerminationStrategy: PID 6567 died...
11:07:34.940073 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6567): Terminated: ok {rc=0}
11:07:34.952507 --- ParallelTerminationStrategy: PID 6568 died...
11:07:35.019797 run osmo-mgw_10.42.42.4(pid=6568): Terminated {rc=15}
11:07:35.032127 --- ParallelTerminationStrategy: PID 6570 died...
11:07:35.099699 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6570): Terminated: ok {rc=0}
11:07:37.129277 --- ParallelTerminationStrategy: PID 6564 died...
11:07:37.198753 run osmo-msc_10.42.42.6(pid=6564): Terminated: ok {rc=0}
11:07:37.878006 --- ParallelTerminationStrategy: PID 6571 died...
11:07:37.947330 run osmo-bsc_10.42.42.7(pid=6571): Terminated: ok {rc=0}
11:07:37.959526 --- ParallelTerminationStrategy: PID 6576 died...
11:07:38.026895 run osmo-bts-trx(pid=6576): Terminated: ok {rc=0}
11:07:38.130543 bus /sierra_3: Setting Powered False
11:07:39.250176 bus /sierra_3: Setting Powered False
11:07:40.382210 bus /sierra_2: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 673, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
wait(bsc.bts_is_connected, bts0)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-trx-b200↪handover.py:43]