Skip to content

Failed

handover_2G:trx-lms+bts1-trx-umtrx.handover.py (from handover_2G_trx-lms+bts1-trx-umtrx)

Failing for the past 21 builds (Since #140 )
Took 1 min 44 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms+bts1-trx-umtrx↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-160 handover_2G:trx-lms+bts1-trx-umtrx handover.py
----------------------------------------------
05:53:32.822363 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
05:53:32.837477 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
05:53:32.998365 tst                      handover.py: using LAC 16139
05:53:33.136575 tst                      handover.py: using RAC 74
05:53:33.271386 tst                      handover.py: using CellId 16138
05:53:33.403107 tst                      handover.py: using BVCI 2817
05:53:33.429642 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
05:53:33.580712 tst                      handover.py: using LAC 16140
05:53:33.712073 tst                      handover.py: using RAC 75
05:53:33.844420 tst                      handover.py: using CellId 16139
05:53:33.975328 tst                      handover.py: using BVCI 2818
05:53:34.001646 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
05:53:34.015594 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
05:53:34.029456 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
05:53:34.043275 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
05:53:34.056916 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
05:53:34.070474 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
05:53:34.203652 tst handover_2G:trx-lms+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
05:53:34.435721 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
05:53:34.960700 run         create_hlr_db(pid=29600): Launched
05:53:35.110933 bus                        /sierra_2: Setting Powered False
05:53:36.197487 run         create_hlr_db(pid=29600): Terminated: ok {rc=0}
05:53:36.627473 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29602): Launched
05:53:36.968212 run   osmo-hlr_10.42.42.2(pid=29603): Launched
05:53:37.090894 run              osmo-stp_10.42.42.5: Starting osmo-stp
05:53:37.659295 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29604): Launched
05:53:38.014465 run   osmo-stp_10.42.42.5(pid=29605): Launched
05:53:38.136237 run              osmo-msc_10.42.42.6: Starting osmo-msc
05:53:38.732607 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29606): Launched
05:53:39.084953 run   osmo-msc_10.42.42.6(pid=29607): Launched
05:53:39.223153 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
05:53:39.839152 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29608): Launched
05:53:40.194614 run   osmo-mgw_10.42.42.3(pid=29609): Launched
05:53:40.316408 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
05:53:40.876446 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29610): Launched
05:53:41.231424 run   osmo-mgw_10.42.42.4(pid=29611): Launched
05:53:41.355078 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
05:53:42.118953 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29612): Launched
05:53:42.475481 run   osmo-bsc_10.42.42.7(pid=29613): Launched
05:53:46.173028 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
05:53:46.637396 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'dev_args': 'LimeSDR-USB,serial=00090706024F0A22', 'bts_ip': '10.42.42.53', 'trx_ip': '10.42.42.117', 'channels': [{'rx_path': 'LNAW'}], 'remote_user': 'jenkins', 'type': 'lms', 'launch_trx': 'True', 'egprs': 'disable', 'clock_reference': 'external'}}
05:53:47.243285 run               uname-m(pid=29615): Launched
05:53:48.459989 run               uname-m(pid=29615): Terminated: ok {rc=0}
05:53:48.851849 run         rm-remote-dir(pid=29616): Launched
05:53:50.084684 run         rm-remote-dir(pid=29616): Terminated: ok {rc=0}
05:53:50.478958 run         mk-remote-dir(pid=29617): Launched
05:53:51.696622 run         mk-remote-dir(pid=29617): Terminated: ok {rc=0}
05:53:52.071641 run         mk-remote-dir(pid=29618): Launched
05:53:53.288147 run         mk-remote-dir(pid=29618): Terminated: ok {rc=0}
05:53:53.618230 run    scp-inst-to-remote(pid=29619): Launched
05:53:55.838082 run    scp-inst-to-remote(pid=29619): Terminated: ok {rc=0}
05:53:56.212483 run         mk-remote-dir(pid=29624): Launched
05:53:57.428840 run         mk-remote-dir(pid=29624): Terminated: ok {rc=0}
05:53:57.748551 run     scp-cfg-to-remote(pid=29625): Launched
05:53:58.963598 run     scp-cfg-to-remote(pid=29625): Terminated: ok {rc=0}
05:53:59.354661 run         mk-remote-dir(pid=29627): Launched
05:54:00.584527 run         mk-remote-dir(pid=29627): Terminated: ok {rc=0}
05:54:00.924063 run scp-wrapper-to-remote(pid=29628): Launched
05:54:02.142425 run scp-wrapper-to-remote(pid=29628): Terminated: ok {rc=0}
05:54:02.511341 run          osmo-trx-lms(pid=29630): Launched
05:54:02.642045 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
05:54:03.955320 run          osmo-bts-trx(pid=29631): Launched
05:54:06.425675 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms+bts1-trx-umtrx/handover.py/01_bts0_started: pass (20.2s)
05:54:06.590424 tst                      handover.py: using MSISDN 13036
05:54:06.778054 bus                        /sierra_2: Setting Powered True
05:54:10.296254 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13036', subscriber_id=1}
05:54:10.452911 tst                      handover.py: using MSISDN 13037
05:54:10.639838 bus                        /sierra_3: Setting Powered True
05:54:24.283551 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13037', subscriber_id=2}
05:54:24.578060 bus                        /sierra_2: Setting Online False
05:54:25.781446 bus                        /sierra_2: Setting Powered False
05:54:31.845163 bus                        /sierra_2: Setting Powered True
05:54:33.208948 bus                        /sierra_2: Setting Online True
05:54:35.764434 tst                        /sierra_2: Connect to ('901', '70')
05:54:36.105535 bus                        /sierra_3: Setting Online False
05:54:37.312675 bus                        /sierra_3: Setting Powered False
05:54:45.208202 bus                        /sierra_3: Setting Powered True
05:54:58.945153 bus                        /sierra_3: Setting Online True
05:55:04.852957 tst                        /sierra_3: Connect to ('901', '70')
05:55:05.079317 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
05:55:05.217118 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
05:55:05.422388 tst                   handover.py:54: waiting for modems to attach...
05:55:16.292227 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms+bts1-trx-umtrx/handover.py/02.1_ms0_attach: FAIL (10.8s)
05:55:16.301472 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms+bts1-trx-umtrx↪handover.py:57↪handover.py↪/sierra_2]  [trial-160↪handover_2G:trx-lms+bts1-trx-umtrx↪handover.py:57]
05:55:16.310720 tst                   handover.py:57: Test FAILED (103.5 sec)
05:55:16.353287 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29602): Terminating (SIGTERM)
05:55:16.403855 run   osmo-hlr_10.42.42.2(pid=29603): Terminating (SIGTERM)
05:55:16.453404 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29604): Terminating (SIGTERM)
05:55:16.505845 run   osmo-stp_10.42.42.5(pid=29605): Terminating (SIGTERM)
05:55:16.557744 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29606): Terminating (SIGTERM)
05:55:16.610140 run   osmo-msc_10.42.42.6(pid=29607): Terminating (SIGTERM)
05:55:16.659543 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29608): Terminating (SIGTERM)
05:55:16.708114 run   osmo-mgw_10.42.42.3(pid=29609): Terminating (SIGTERM)
05:55:16.757345 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29610): Terminating (SIGTERM)
05:55:16.805871 run   osmo-mgw_10.42.42.4(pid=29611): Terminating (SIGTERM)
05:55:16.854980 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29612): Terminating (SIGTERM)
05:55:16.903540 run   osmo-bsc_10.42.42.7(pid=29613): Terminating (SIGTERM)
05:55:16.952381 run          osmo-trx-lms(pid=29630): Terminating (SIGTERM)
05:55:17.001089 run          osmo-bts-trx(pid=29631): Terminating (SIGTERM)
05:55:17.012097 ---      ParallelTerminationStrategy: PID 29602 died...
05:55:17.076960 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29602): Terminated: ok {rc=0}
05:55:17.088595 ---      ParallelTerminationStrategy: PID 29603 died...
05:55:17.152521 run   osmo-hlr_10.42.42.2(pid=29603): Terminated: ok {rc=0}
05:55:17.163986 ---      ParallelTerminationStrategy: PID 29604 died...
05:55:17.228229 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29604): Terminated: ok {rc=0}
05:55:17.239779 ---      ParallelTerminationStrategy: PID 29605 died...
05:55:17.305060 run   osmo-stp_10.42.42.5(pid=29605): Terminated: ok {rc=0}
05:55:17.316764 ---      ParallelTerminationStrategy: PID 29606 died...
05:55:17.380814 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29606): Terminated: ok {rc=0}
05:55:17.392288 ---      ParallelTerminationStrategy: PID 29608 died...
05:55:17.456360 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29608): Terminated: ok {rc=0}
05:55:17.468537 ---      ParallelTerminationStrategy: PID 29609 died...
05:55:17.532455 run   osmo-mgw_10.42.42.3(pid=29609): Terminated {rc=15}
05:55:17.544007 ---      ParallelTerminationStrategy: PID 29610 died...
05:55:17.608905 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29610): Terminated: ok {rc=0}
05:55:17.620890 ---      ParallelTerminationStrategy: PID 29611 died...
05:55:17.684952 run   osmo-mgw_10.42.42.4(pid=29611): Terminated {rc=15}
05:55:17.696701 ---      ParallelTerminationStrategy: PID 29612 died...
05:55:17.760844 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29612): Terminated: ok {rc=0}
05:55:17.772517 ---      ParallelTerminationStrategy: PID 29630 died...
05:55:17.837113 run          osmo-trx-lms(pid=29630): Terminated {rc=36608}
05:55:19.865622 ---      ParallelTerminationStrategy: PID 29607 died...
05:55:19.932278 run   osmo-msc_10.42.42.6(pid=29607): Terminated: ok {rc=0}
05:55:20.611252 ---      ParallelTerminationStrategy: PID 29613 died...
05:55:20.677292 run   osmo-bsc_10.42.42.7(pid=29613): Terminated: ok {rc=0}
05:55:23.867562 run          osmo-bts-trx(pid=29631): Terminating (SIGINT)
05:55:24.046806 ---      ParallelTerminationStrategy: PID 29631 died...
05:55:24.113568 run          osmo-bts-trx(pid=29631): Terminated: ok {rc=0}
05:55:24.286074 bus                        /sierra_3: Setting Online False
05:55:25.763249 bus                        /sierra_3: Setting Powered False
05:55:31.900892 bus                        /sierra_3: Setting Powered False
05:55:33.025659 bus                        /sierra_3: Setting Powered False
05:55:34.202674 bus                        /sierra_2: Setting Online False
05:55:35.337177 bus                        /sierra_2: Setting Powered False
05:55:40.844028 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms+bts1-trx-umtrx↪handover.py:57↪handover.py↪/sierra_2]