Skip to content

Failed

handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx.handover.py (from handover_2G_trx-lms-limenet-micro+bts1-trx-umtrx)

Failing for the past 21 builds (Since #140 )
Took 1 min 56 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-160 handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx handover.py
----------------------------------------------
06:02:28.023288 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:02:28.037434 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
06:02:28.193220 tst                      handover.py: using LAC 16147
06:02:28.323335 tst                      handover.py: using RAC 82
06:02:28.452859 tst                      handover.py: using CellId 16146
06:02:28.581835 tst                      handover.py: using BVCI 2825
06:02:28.607613 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
06:02:28.759905 tst                      handover.py: using LAC 16148
06:02:28.891940 tst                      handover.py: using RAC 83
06:02:29.028549 tst                      handover.py: using CellId 16147
06:02:29.164620 tst                      handover.py: using BVCI 2826
06:02:29.191880 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:02:29.206936 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:02:29.222175 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:02:29.237288 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:02:29.252176 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
06:02:29.266924 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
06:02:29.398572 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
06:02:29.642086 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
06:02:30.150303 run         create_hlr_db(pid=29757): Launched
06:02:30.292411 bus                        /sierra_2: Setting Powered False
06:02:31.375072 run         create_hlr_db(pid=29757): Terminated: ok {rc=0}
06:02:31.821395 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29759): Launched
06:02:32.176673 run   osmo-hlr_10.42.42.2(pid=29760): Launched
06:02:32.299065 run              osmo-stp_10.42.42.5: Starting osmo-stp
06:02:32.871043 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29761): Launched
06:02:33.234112 run   osmo-stp_10.42.42.5(pid=29762): Launched
06:02:33.362134 run              osmo-msc_10.42.42.6: Starting osmo-msc
06:02:34.168234 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29763): Launched
06:02:34.520721 run   osmo-msc_10.42.42.6(pid=29764): Launched
06:02:34.644589 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
06:02:35.209595 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29765): Launched
06:02:35.557479 run   osmo-mgw_10.42.42.3(pid=29766): Launched
06:02:35.684557 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
06:02:36.256548 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29767): Launched
06:02:36.605532 run   osmo-mgw_10.42.42.4(pid=29768): Launched
06:02:36.731945 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
06:02:37.512271 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29769): Launched
06:02:37.852891 run   osmo-bsc_10.42.42.7(pid=29770): Launched
06:02:41.546290 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
06:02:42.009383 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'dev_args': 'LimeNET-Micro,serial=005839AF25B352', 'bts_ip': '10.42.42.53', 'trx_ip': '10.42.42.123', 'channels': [{}], 'remote_user': 'pi', 'type': 'lms', 'launch_trx': 'True', 'egprs': 'disable', 'clock_reference': 'external'}}
06:02:42.618384 run               uname-m(pid=29771): Launched
06:02:43.843681 run               uname-m(pid=29771): Terminated: ok {rc=0}
06:02:44.234944 run         rm-remote-dir(pid=29772): Launched
06:02:45.454809 run         rm-remote-dir(pid=29772): Terminated: ok {rc=0}
06:02:45.842793 run         mk-remote-dir(pid=29773): Launched
06:02:47.071007 run         mk-remote-dir(pid=29773): Terminated: ok {rc=0}
06:02:47.457814 run         mk-remote-dir(pid=29774): Launched
06:02:48.671754 run         mk-remote-dir(pid=29774): Terminated: ok {rc=0}
06:02:48.997624 run     scp-cfg-to-remote(pid=29775): Launched
06:02:50.222398 run     scp-cfg-to-remote(pid=29775): Terminated: ok {rc=0}
06:02:50.620548 run         mk-remote-dir(pid=29777): Launched
06:02:51.848256 run         mk-remote-dir(pid=29777): Terminated: ok {rc=0}
06:02:52.191078 run scp-wrapper-to-remote(pid=29778): Launched
06:02:53.419274 run scp-wrapper-to-remote(pid=29778): Terminated: ok {rc=0}
06:02:53.788874 run          osmo-trx-lms(pid=29780): Launched
06:02:53.911189 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
06:02:56.223575 run          osmo-bts-trx(pid=29781): Launched
06:03:01.765266 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx/handover.py/01_bts0_started: pass (20.1s)
06:03:01.954996 tst                      handover.py: using MSISDN 13044
06:03:02.182131 bus                        /sierra_2: Setting Powered True
06:03:05.772176 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13044', subscriber_id=1}
06:03:05.950072 tst                      handover.py: using MSISDN 13045
06:03:06.169299 bus                        /sierra_3: Setting Powered True
06:03:21.948732 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13045', subscriber_id=2}
06:03:22.301905 bus                        /sierra_2: Setting Online False
06:03:23.543609 bus                        /sierra_2: Setting Powered False
06:03:29.916653 bus                        /sierra_2: Setting Powered True
06:03:31.334413 bus                        /sierra_2: Setting Online True
06:03:34.097435 tst                        /sierra_2: Connect to ('901', '70')
06:03:34.503276 bus                        /sierra_3: Setting Online False
06:03:35.746914 bus                        /sierra_3: Setting Powered False
06:03:44.134059 bus                        /sierra_3: Setting Powered True
06:03:57.375930 bus                        /sierra_3: Setting Online True
06:04:03.495585 tst                        /sierra_3: Connect to ('901', '70')
06:04:03.704548 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
06:04:03.854889 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
06:04:04.095133 tst                   handover.py:54: waiting for modems to attach...
06:04:23.303031 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx/handover.py/02.1_ms0_attach: FAIL (19.1s)
06:04:23.312404 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx↪handover.py:57↪handover.py↪/sierra_2]  [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx↪handover.py:57]
06:04:23.321227 tst                   handover.py:57: Test FAILED (115.3 sec)
06:04:23.365702 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29759): Terminating (SIGTERM)
06:04:23.415434 run   osmo-hlr_10.42.42.2(pid=29760): Terminating (SIGTERM)
06:04:23.465050 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29761): Terminating (SIGTERM)
06:04:23.517827 run   osmo-stp_10.42.42.5(pid=29762): Terminating (SIGTERM)
06:04:23.569680 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29763): Terminating (SIGTERM)
06:04:23.622488 run   osmo-msc_10.42.42.6(pid=29764): Terminating (SIGTERM)
06:04:23.672600 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29765): Terminating (SIGTERM)
06:04:23.722446 run   osmo-mgw_10.42.42.3(pid=29766): Terminating (SIGTERM)
06:04:23.779180 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29767): Terminating (SIGTERM)
06:04:23.828069 run   osmo-mgw_10.42.42.4(pid=29768): Terminating (SIGTERM)
06:04:23.877951 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29769): Terminating (SIGTERM)
06:04:23.927926 run   osmo-bsc_10.42.42.7(pid=29770): Terminating (SIGTERM)
06:04:23.977488 run          osmo-trx-lms(pid=29780): Terminating (SIGTERM)
06:04:24.026654 run          osmo-bts-trx(pid=29781): Terminating (SIGTERM)
06:04:24.037811 ---      ParallelTerminationStrategy: PID 29759 died...
06:04:24.103272 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29759): Terminated: ok {rc=0}
06:04:24.114998 ---      ParallelTerminationStrategy: PID 29760 died...
06:04:24.179939 run   osmo-hlr_10.42.42.2(pid=29760): Terminated: ok {rc=0}
06:04:24.191523 ---      ParallelTerminationStrategy: PID 29761 died...
06:04:24.256666 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29761): Terminated: ok {rc=0}
06:04:24.268307 ---      ParallelTerminationStrategy: PID 29762 died...
06:04:24.333203 run   osmo-stp_10.42.42.5(pid=29762): Terminated: ok {rc=0}
06:04:24.344770 ---      ParallelTerminationStrategy: PID 29763 died...
06:04:24.408954 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29763): Terminated: ok {rc=0}
06:04:24.420574 ---      ParallelTerminationStrategy: PID 29765 died...
06:04:24.484880 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29765): Terminated: ok {rc=0}
06:04:24.496522 ---      ParallelTerminationStrategy: PID 29766 died...
06:04:24.560528 run   osmo-mgw_10.42.42.3(pid=29766): Terminated {rc=15}
06:04:24.572233 ---      ParallelTerminationStrategy: PID 29767 died...
06:04:24.636479 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29767): Terminated: ok {rc=0}
06:04:24.648105 ---      ParallelTerminationStrategy: PID 29768 died...
06:04:24.722519 run   osmo-mgw_10.42.42.4(pid=29768): Terminated {rc=15}
06:04:24.736592 ---      ParallelTerminationStrategy: PID 29769 died...
06:04:24.813312 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29769): Terminated: ok {rc=0}
06:04:24.827443 ---      ParallelTerminationStrategy: PID 29780 died...
06:04:24.904997 run          osmo-trx-lms(pid=29780): Terminated {rc=36608}
06:04:26.935494 ---      ParallelTerminationStrategy: PID 29764 died...
06:04:27.008527 run   osmo-msc_10.42.42.6(pid=29764): Terminated: ok {rc=0}
06:04:27.687719 ---      ParallelTerminationStrategy: PID 29770 died...
06:04:27.758392 run   osmo-bsc_10.42.42.7(pid=29770): Terminated: ok {rc=0}
06:04:30.952249 run          osmo-bts-trx(pid=29781): Terminating (SIGINT)
06:04:31.133861 ---      ParallelTerminationStrategy: PID 29781 died...
06:04:31.204589 run          osmo-bts-trx(pid=29781): Terminated: ok {rc=0}
06:04:31.387106 bus                        /sierra_3: Setting Online False
06:04:32.791952 bus                        /sierra_3: Setting Powered False
06:04:38.983677 bus                        /sierra_3: Setting Powered False
06:04:40.105281 bus                        /sierra_3: Setting Powered False
06:04:41.286548 bus                        /sierra_2: Setting Online False
06:04:42.435999 bus                        /sierra_2: Setting Powered False
06:04:48.635167 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx↪handover.py:57↪handover.py↪/sierra_2]