![[Jenkins]](/jenkins/static/342c126a/images/svgs/logo.svg){kind=logo}
Failed
handover_2G:oc2g+bts1-trx-sysmocell5000.handover.py (from handover_2G_oc2g+bts1-trx-sysmocell5000)
Stacktrace
handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:oc2g+bts1-trx-sysmocell5000↪handover.py:84]Standard Output
----------------------------------------------
trial-161 handover_2G:oc2g+bts1-trx-sysmocell5000 handover.py
----------------------------------------------
07:46:35.754669 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
07:46:35.768689 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x bts (candidates: 2)
07:46:35.909105 tst handover.py: using LAC 16183
07:46:36.041256 tst handover.py: using RAC 118
07:46:36.171574 tst handover.py: using CellId 16182
07:46:36.301022 tst handover.py: using BVCI 2861
07:46:36.326816 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x bts (candidates: 2)
07:46:36.484633 tst handover.py: using LAC 16184
07:46:36.614587 tst handover.py: using RAC 119
07:46:36.745165 tst handover.py: using CellId 16183
07:46:36.874974 tst handover.py: using BVCI 2862
07:46:36.900845 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
07:46:36.914895 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
07:46:36.928889 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
07:46:36.942792 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
07:46:36.956684 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
07:46:36.970482 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x modem (candidates: 2)
07:46:37.103046 tst handover_2G:oc2g+bts1-trx-sysmocell5000: Using 1 x modem (candidates: 2)
07:46:37.333019 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:46:37.838285 run create_hlr_db(pid=32671): Launched
07:46:37.977935 bus /sierra_2: Setting Powered False
07:46:39.061266 run create_hlr_db(pid=32671): Terminated: ok {rc=0}
07:46:39.496241 run pcap-recorder_any(filters='host 10.42.42.2')(pid=32673): Launched
07:46:39.854330 run osmo-hlr_10.42.42.2(pid=32674): Launched
07:46:39.983281 run osmo-stp_10.42.42.5: Starting osmo-stp
07:46:40.573587 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=32675): Launched
07:46:40.933336 run osmo-stp_10.42.42.5(pid=32676): Launched
07:46:41.060982 run osmo-msc_10.42.42.6: Starting osmo-msc
07:46:41.687475 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=32677): Launched
07:46:42.040542 run osmo-msc_10.42.42.6(pid=32678): Launched
07:46:42.163759 run osmo-mgw_10.42.42.3: Starting osmo-mgw
07:46:42.742028 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=32679): Launched
07:46:43.099402 run osmo-mgw_10.42.42.3(pid=32680): Launched
07:46:43.226793 run osmo-mgw_10.42.42.4: Starting osmo-mgw
07:46:43.799304 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=32681): Launched
07:46:44.147808 run osmo-mgw_10.42.42.4(pid=32682): Launched
07:46:44.275812 run osmo-bsc_10.42.42.7: Starting osmo-bsc
07:46:45.072446 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=32683): Launched
07:46:45.421425 run osmo-bsc_10.42.42.7(pid=32684): Launched
07:46:49.152570 run osmo-bts-oc2g: Starting OsmoBtsOC2G to connect to osmo-bsc_10.42.42.7
07:46:49.758114 run rm-remote-dir(pid=32685): Launched
07:46:50.970028 run rm-remote-dir(pid=32685): Terminated: ok {rc=0}
07:46:51.345441 run mk-remote-dir(pid=32686): Launched
07:46:52.568718 run mk-remote-dir(pid=32686): Terminated: ok {rc=0}
07:46:52.888562 run scp-inst-to-remote(pid=32687): Launched
07:46:59.106867 run scp-inst-to-remote(pid=32687): Terminated: ok {rc=0}
07:46:59.467651 run mk-remote-dir(pid=32689): Launched
07:47:00.675869 run mk-remote-dir(pid=32689): Terminated: ok {rc=0}
07:47:00.992862 run scp-cfg-to-remote(pid=32690): Launched
07:47:02.213258 run scp-cfg-to-remote(pid=32690): Terminated: ok {rc=0}
07:47:02.579315 run osmo-bts-oc2g(pid=32692): Launched
07:47:09.036457 tst handover.py:43: ----- Report fragment: handover_2G:oc2g+bts1-trx-sysmocell5000/handover.py/01_bts0_started: pass (19.8s)
07:47:09.194560 tst handover.py: using MSISDN 13074
07:47:09.393324 bus /sierra_2: Setting Powered True
07:47:12.924276 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13074', subscriber_id=1}
07:47:13.081591 tst handover.py: using MSISDN 13075
07:47:13.269043 bus /sierra_3: Setting Powered True
07:47:19.730601 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13075', subscriber_id=2}
07:47:20.042962 bus /sierra_2: Setting Online False
07:47:21.251710 bus /sierra_2: Setting Powered False
07:47:27.003680 bus /sierra_2: Setting Powered True
07:47:28.373674 bus /sierra_2: Setting Online True
07:47:30.982699 tst /sierra_2: Connect to ('901', '70')
07:47:31.392664 bus /sierra_3: Setting Online False
07:47:32.604636 bus /sierra_3: Setting Powered False
07:47:38.861846 bus /sierra_3: Setting Powered True
07:47:43.305243 bus /sierra_3: Setting Online True
07:47:45.629045 tst /sierra_2: Already registered with network ('901', '70')
07:47:46.077404 tst /sierra_3: Connect to ('901', '70')
07:47:46.259127 tst /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
07:47:46.392665 tst /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
07:47:46.605986 tst handover.py:54: waiting for modems to attach...
07:47:49.213030 tst handover.py:57: ----- Report fragment: handover_2G:oc2g+bts1-trx-sysmocell5000/handover.py/02.1_ms0_attach: pass (2.4s)
07:48:00.873868 tst /sierra_3: Already registered with network ('901', '70')
07:48:05.688988 tst handover.py:60: ----- Report fragment: handover_2G:oc2g+bts1-trx-sysmocell5000/handover.py/02.2_ms1_attach: pass (16.2s)
07:48:07.211146 tst handover.py:63: ----- Report fragment: handover_2G:oc2g+bts1-trx-sysmocell5000/handover.py/02.3_subscribed_in_msc: pass (1.3s)
07:48:12.798357 tst handover.py:68: dial success
07:48:15.534699 tst handover.py:75: answer success, call established and ongoing
07:48:15.846046 bus Vty(host='10.42.42.7', port=4242): Found matching lchan activity (2 of 2 requirements): ['0-0-2-0 TCH/F ESTABLISHED', '0-0-3-0 TCH/F ESTABLISHED']
07:48:16.071347 tst handover.py:79: ----- Report fragment: handover_2G:oc2g+bts1-trx-sysmocell5000/handover.py/03_call_established: pass (3.0s)
07:48:16.223872 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
07:48:18.578258 run ssh_sc5_ccli(pid=32694): Launched
07:48:37.825444 run ssh_sc5_ccli(pid=32694): ERR: Terminated: ERROR {rc=255} [trial-161↪handover_2G:oc2g+bts1-trx-sysmocell5000↪handover.py:83↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=32694)]
07:48:37.985461 run ssh_sc5_ccli(pid=32694): stdout:
| (launched: 2020-12-07_07:48:18.401672)
07:48:38.144124 run ssh_sc5_ccli(pid=32694): stderr:
| (launched: 2020-12-07_07:48:18.457969)
| ssh: connect to host 10.42.42.112 port 22: No route to host
07:48:38.281801 run sc5-trx: Configuring SC5 TRX failed, retrying 9 more times
07:48:43.653659 run ssh_sc5_ccli(pid=32695): Launched
07:48:46.865185 run ssh_sc5_ccli(pid=32695): ERR: Terminated: ERROR {rc=255} [trial-161↪handover_2G:oc2g+bts1-trx-sysmocell5000↪handover.py:83↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=32695)]
07:48:47.013805 run ssh_sc5_ccli(pid=32695): stdout:
| (launched: 2020-12-07_07:48:43.475836)
07:48:47.161795 run ssh_sc5_ccli(pid=32695): stderr:
| (launched: 2020-12-07_07:48:43.532306)
| ssh: connect to host 10.42.42.112 port 22: No route to host
07:48:47.290103 run sc5-trx: Configuring SC5 TRX failed, retrying 8 more times
07:48:52.646548 run ssh_sc5_ccli(pid=32696): Launched
07:48:53.870858 run ssh_sc5_ccli(pid=32696): Terminated: ok {rc=0}
07:48:54.005734 run osmo-bts-trx: Waiting for sc5-trx to start up...
07:48:55.338802 run osmo-bts-trx(pid=32700): Launched
07:53:55.994188 tst handover.py:84: ----- Report fragment: handover_2G:oc2g+bts1-trx-sysmocell5000/handover.py/04.1_bts1_started: FAIL (339.9s)
07:53:56.002068 tst handover.py:84: ERR: Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:oc2g+bts1-trx-sysmocell5000↪handover.py:84] [trial-161↪handover_2G:oc2g+bts1-trx-sysmocell5000↪handover.py:84]
07:53:56.010066 tst handover.py:84: Test FAILED (440.2 sec)
07:53:56.057182 run pcap-recorder_any(filters='host 10.42.42.2')(pid=32673): Terminating (SIGTERM)
07:53:56.110890 run osmo-hlr_10.42.42.2(pid=32674): Terminating (SIGTERM)
07:53:56.166183 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=32675): Terminating (SIGTERM)
07:53:56.223591 run osmo-stp_10.42.42.5(pid=32676): Terminating (SIGTERM)
07:53:56.279843 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=32677): Terminating (SIGTERM)
07:53:56.337479 run osmo-msc_10.42.42.6(pid=32678): Terminating (SIGTERM)
07:53:56.391859 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=32679): Terminating (SIGTERM)
07:53:56.444471 run osmo-mgw_10.42.42.3(pid=32680): Terminating (SIGTERM)
07:53:56.499853 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=32681): Terminating (SIGTERM)
07:53:56.558126 run osmo-mgw_10.42.42.4(pid=32682): Terminating (SIGTERM)
07:53:56.616129 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=32683): Terminating (SIGTERM)
07:53:56.669297 run osmo-bsc_10.42.42.7(pid=32684): Terminating (SIGTERM)
07:53:56.719533 run osmo-bts-oc2g(pid=32692): Terminating (SIGTERM)
07:53:56.769670 run osmo-bts-trx(pid=32700): Terminating (SIGTERM)
07:53:56.781138 --- ParallelTerminationStrategy: PID 32673 died...
07:53:56.847884 run pcap-recorder_any(filters='host 10.42.42.2')(pid=32673): Terminated: ok {rc=0}
07:53:56.860085 --- ParallelTerminationStrategy: PID 32674 died...
07:53:56.926717 run osmo-hlr_10.42.42.2(pid=32674): Terminated: ok {rc=0}
07:53:56.938806 --- ParallelTerminationStrategy: PID 32675 died...
07:53:57.005303 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=32675): Terminated: ok {rc=0}
07:53:57.017435 --- ParallelTerminationStrategy: PID 32676 died...
07:53:57.083556 run osmo-stp_10.42.42.5(pid=32676): Terminated: ok {rc=0}
07:53:57.095534 --- ParallelTerminationStrategy: PID 32677 died...
07:53:57.161381 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=32677): Terminated: ok {rc=0}
07:53:57.173433 --- ParallelTerminationStrategy: PID 32679 died...
07:53:57.239112 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=32679): Terminated: ok {rc=0}
07:53:57.251271 --- ParallelTerminationStrategy: PID 32680 died...
07:53:57.317423 run osmo-mgw_10.42.42.3(pid=32680): Terminated {rc=15}
07:53:57.329396 --- ParallelTerminationStrategy: PID 32681 died...
07:53:57.394856 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=32681): Terminated: ok {rc=0}
07:53:57.406768 --- ParallelTerminationStrategy: PID 32682 died...
07:53:57.471268 run osmo-mgw_10.42.42.4(pid=32682): Terminated {rc=15}
07:53:57.483024 --- ParallelTerminationStrategy: PID 32683 died...
07:53:57.547999 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=32683): Terminated: ok {rc=0}
07:53:57.559820 --- ParallelTerminationStrategy: PID 32692 died...
07:53:57.624426 run osmo-bts-oc2g(pid=32692): Terminated {rc=36608}
07:53:59.652927 --- ParallelTerminationStrategy: PID 32678 died...
07:53:59.721351 run osmo-msc_10.42.42.6(pid=32678): Terminated: ok {rc=0}
07:54:00.400035 --- ParallelTerminationStrategy: PID 32684 died...
07:54:00.466294 run osmo-bsc_10.42.42.7(pid=32684): Terminated: ok {rc=0}
07:54:00.478008 --- ParallelTerminationStrategy: PID 32700 died...
07:54:00.542596 run osmo-bts-trx(pid=32700): Terminated: ok {rc=0}
07:54:00.701221 bus /sierra_3: Setting Online False
07:54:01.855976 bus /sierra_3: Setting Powered False
07:54:17.642415 bus /sierra_3: Setting Powered False
07:54:18.762449 bus /sierra_3: Setting Powered False
07:54:19.940880 bus /sierra_2: Setting Online False
07:54:21.105303 bus /sierra_2: Setting Powered False
07:54:32.637121 bus /sierra_2: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 673, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 84, in <module>
wait(bsc.bts_is_connected, bts1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-161↪handover_2G:oc2g+bts1-trx-sysmocell5000↪handover.py:84]