Skip to content

Failed

handover_2G:trx-lms-limenet-micro+bts1-trx-b200.handover.py (from handover_2G_trx-lms-limenet-micro+bts1-trx-b200)

Failing for the past 24 builds (Since #140 )
Took 1 min 56 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms-limenet-micro+bts1-trx-b200↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-163 handover_2G:trx-lms-limenet-micro+bts1-trx-b200 handover.py
----------------------------------------------
13:19:40.759684 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:19:40.773725 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x bts (candidates: 2)
13:19:40.934410 tst                      handover.py: using LAC 16361
13:19:41.072653 tst                      handover.py: using RAC 41
13:19:41.210481 tst                      handover.py: using CellId 16360
13:19:41.345818 tst                      handover.py: using BVCI 3039
13:19:41.372586 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x bts (candidates: 2)
13:19:41.528925 tst                      handover.py: using LAC 16362
13:19:41.664295 tst                      handover.py: using RAC 42
13:19:41.801759 tst                      handover.py: using CellId 16361
13:19:41.936993 tst                      handover.py: using BVCI 3040
13:19:41.963550 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:19:41.977626 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:19:41.991832 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:19:42.006190 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:19:42.020525 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:19:42.034199 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x modem (candidates: 2)
13:19:42.169576 tst handover_2G:trx-lms-limenet-micro+bts1-trx-b200: Using 1 x modem (candidates: 2)
13:19:42.407613 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
13:19:42.946212 run          create_hlr_db(pid=8860): Launched
13:19:43.090370 bus                        /sierra_2: Setting Powered False
13:19:44.175064 run          create_hlr_db(pid=8860): Terminated: ok {rc=0}
13:19:44.634785 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8862): Launched
13:19:44.993352 run    osmo-hlr_10.42.42.2(pid=8863): Launched
13:19:45.116585 run              osmo-stp_10.42.42.5: Starting osmo-stp
13:19:45.691223 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8864): Launched
13:19:46.036178 run    osmo-stp_10.42.42.5(pid=8865): Launched
13:19:46.158701 run              osmo-msc_10.42.42.6: Starting osmo-msc
13:19:46.766967 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8866): Launched
13:19:47.121575 run    osmo-msc_10.42.42.6(pid=8867): Launched
13:19:47.260498 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
13:19:47.882882 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8868): Launched
13:19:48.247933 run    osmo-mgw_10.42.42.3(pid=8869): Launched
13:19:48.378427 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
13:19:48.963110 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8870): Launched
13:19:49.319006 run    osmo-mgw_10.42.42.4(pid=8871): Launched
13:19:49.448894 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
13:19:50.245287 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8872): Launched
13:19:50.601571 run    osmo-bsc_10.42.42.7(pid=8873): Launched
13:19:54.347292 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
13:19:54.839365 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'channels': [{}], 'type': 'lms', 'bts_ip': '10.42.42.53', 'clock_reference': 'external', 'remote_user': 'pi', 'dev_args': 'LimeNET-Micro,serial=005839AF25B352', 'trx_ip': '10.42.42.123', 'launch_trx': 'True', 'egprs': 'disable'}}
13:19:55.473896 run                uname-m(pid=8874): Launched
13:19:56.693530 run                uname-m(pid=8874): Terminated: ok {rc=0}
13:19:57.087239 run          rm-remote-dir(pid=8875): Launched
13:19:58.319639 run          rm-remote-dir(pid=8875): Terminated: ok {rc=0}
13:19:58.717660 run          mk-remote-dir(pid=8876): Launched
13:19:59.939524 run          mk-remote-dir(pid=8876): Terminated: ok {rc=0}
13:20:00.320482 run          mk-remote-dir(pid=8877): Launched
13:20:01.539070 run          mk-remote-dir(pid=8877): Terminated: ok {rc=0}
13:20:01.912277 run      scp-cfg-to-remote(pid=8882): Launched
13:20:03.131487 run      scp-cfg-to-remote(pid=8882): Terminated: ok {rc=0}
13:20:03.514599 run          mk-remote-dir(pid=8891): Launched
13:20:04.736493 run          mk-remote-dir(pid=8891): Terminated: ok {rc=0}
13:20:05.076020 run  scp-wrapper-to-remote(pid=8892): Launched
13:20:06.310171 run  scp-wrapper-to-remote(pid=8892): Terminated: ok {rc=0}
13:20:06.695449 run           osmo-trx-lms(pid=8894): Launched
13:20:06.827110 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
13:20:09.139599 run           osmo-bts-trx(pid=8958): Launched
13:20:14.630123 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-trx-b200/handover.py/01_bts0_started: pass (20.2s)
13:20:14.793979 tst                      handover.py: using MSISDN 13204
13:20:14.989871 bus                        /sierra_2: Setting Powered True
13:20:18.795060 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13204', subscriber_id=1}
13:20:19.026343 tst                      handover.py: using MSISDN 13205
13:20:19.318052 bus                        /sierra_3: Setting Powered True
13:20:33.698482 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13205', subscriber_id=2}
13:20:34.060009 bus                        /sierra_2: Setting Online False
13:20:35.306860 bus                        /sierra_2: Setting Powered False
13:20:41.129417 bus                        /sierra_2: Setting Powered True
13:20:42.612461 bus                        /sierra_2: Setting Online True
13:20:45.392712 tst                        /sierra_2: Connect to ('901', '70')
13:20:45.805997 bus                        /sierra_3: Setting Online False
13:20:47.053459 bus                        /sierra_3: Setting Powered False
13:20:55.137194 bus                        /sierra_3: Setting Powered True
13:21:09.965318 bus                        /sierra_3: Setting Online True
13:21:16.066848 tst                        /sierra_3: Connect to ('901', '70')
13:21:16.248745 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
13:21:16.379374 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
13:21:16.587398 tst                   handover.py:54: waiting for modems to attach...
13:21:35.841756 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-trx-b200/handover.py/02.1_ms0_attach: FAIL (19.2s)
13:21:35.850793 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms-limenet-micro+bts1-trx-b200↪handover.py:57↪handover.py↪/sierra_2]  [trial-163↪handover_2G:trx-lms-limenet-micro+bts1-trx-b200↪handover.py:57]
13:21:35.859718 tst                   handover.py:57: Test FAILED (115.1 sec)
13:21:35.903320 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8862): Terminating (SIGTERM)
13:21:35.953310 run    osmo-hlr_10.42.42.2(pid=8863): Terminating (SIGTERM)
13:21:36.003858 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8864): Terminating (SIGTERM)
13:21:36.057488 run    osmo-stp_10.42.42.5(pid=8865): Terminating (SIGTERM)
13:21:36.114906 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8866): Terminating (SIGTERM)
13:21:36.175286 run    osmo-msc_10.42.42.6(pid=8867): Terminating (SIGTERM)
13:21:36.225334 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8868): Terminating (SIGTERM)
13:21:36.276127 run    osmo-mgw_10.42.42.3(pid=8869): Terminating (SIGTERM)
13:21:36.326525 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8870): Terminating (SIGTERM)
13:21:36.376916 run    osmo-mgw_10.42.42.4(pid=8871): Terminating (SIGTERM)
13:21:36.427199 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8872): Terminating (SIGTERM)
13:21:36.476863 run    osmo-bsc_10.42.42.7(pid=8873): Terminating (SIGTERM)
13:21:36.526314 run           osmo-trx-lms(pid=8894): Terminating (SIGTERM)
13:21:36.575578 run           osmo-bts-trx(pid=8958): Terminating (SIGTERM)
13:21:36.586756 ---      ParallelTerminationStrategy: PID 8862 died...
13:21:36.652033 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8862): Terminated: ok {rc=0}
13:21:36.664165 ---      ParallelTerminationStrategy: PID 8863 died...
13:21:36.729428 run    osmo-hlr_10.42.42.2(pid=8863): Terminated: ok {rc=0}
13:21:36.741130 ---      ParallelTerminationStrategy: PID 8864 died...
13:21:36.806215 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8864): Terminated: ok {rc=0}
13:21:36.818007 ---      ParallelTerminationStrategy: PID 8865 died...
13:21:36.883102 run    osmo-stp_10.42.42.5(pid=8865): Terminated: ok {rc=0}
13:21:36.894914 ---      ParallelTerminationStrategy: PID 8866 died...
13:21:36.959979 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8866): Terminated: ok {rc=0}
13:21:36.971728 ---      ParallelTerminationStrategy: PID 8868 died...
13:21:37.036900 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8868): Terminated: ok {rc=0}
13:21:37.049369 ---      ParallelTerminationStrategy: PID 8869 died...
13:21:37.114557 run    osmo-mgw_10.42.42.3(pid=8869): Terminated {rc=15}
13:21:37.126308 ---      ParallelTerminationStrategy: PID 8870 died...
13:21:37.192072 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8870): Terminated: ok {rc=0}
13:21:37.203847 ---      ParallelTerminationStrategy: PID 8871 died...
13:21:37.269055 run    osmo-mgw_10.42.42.4(pid=8871): Terminated {rc=15}
13:21:37.280896 ---      ParallelTerminationStrategy: PID 8872 died...
13:21:37.346237 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8872): Terminated: ok {rc=0}
13:21:37.358479 ---      ParallelTerminationStrategy: PID 8894 died...
13:21:37.424164 run           osmo-trx-lms(pid=8894): Terminated {rc=36608}
13:21:39.452526 ---      ParallelTerminationStrategy: PID 8867 died...
13:21:39.521038 run    osmo-msc_10.42.42.6(pid=8867): Terminated: ok {rc=0}
13:21:40.199973 ---      ParallelTerminationStrategy: PID 8873 died...
13:21:40.267771 run    osmo-bsc_10.42.42.7(pid=8873): Terminated: ok {rc=0}
13:21:43.459104 run           osmo-bts-trx(pid=8958): Terminating (SIGINT)
13:21:43.638470 ---      ParallelTerminationStrategy: PID 8958 died...
13:21:43.705780 run           osmo-bts-trx(pid=8958): Terminated: ok {rc=0}
13:21:43.880796 bus                        /sierra_3: Setting Online False
13:21:45.385559 bus                        /sierra_3: Setting Powered False
13:21:51.934328 bus                        /sierra_3: Setting Powered False
13:21:53.055815 bus                        /sierra_3: Setting Powered False
13:21:54.238507 bus                        /sierra_2: Setting Online False
13:21:55.401404 bus                        /sierra_2: Setting Powered False
13:22:01.643780 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms-limenet-micro+bts1-trx-b200↪handover.py:57↪handover.py↪/sierra_2]