Skip to content

Failed

handover_2G:trx-lms+bts1-trx-sysmocell5000.handover.py (from handover_2G_trx-lms+bts1-trx-sysmocell5000)

Failing for the past 22 builds (Since #140 )
Took 1 min 45 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-161↪handover_2G:trx-lms+bts1-trx-sysmocell5000↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-161 handover_2G:trx-lms+bts1-trx-sysmocell5000 handover.py
----------------------------------------------
08:13:16.817316 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
08:13:16.831287 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x bts (candidates: 2)
08:13:16.988754 tst                      handover.py: using LAC 16205
08:13:17.118975 tst                      handover.py: using RAC 140
08:13:17.247970 tst                      handover.py: using CellId 16204
08:13:17.379702 tst                      handover.py: using BVCI 2883
08:13:17.405760 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x bts (candidates: 2)
08:13:17.561386 tst                      handover.py: using LAC 16206
08:13:17.697507 tst                      handover.py: using RAC 141
08:13:17.842832 tst                      handover.py: using CellId 16205
08:13:17.972707 tst                      handover.py: using BVCI 2884
08:13:17.998739 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
08:13:18.013255 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
08:13:18.027230 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
08:13:18.041183 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
08:13:18.055507 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
08:13:18.069547 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x modem (candidates: 2)
08:13:18.204499 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x modem (candidates: 2)
08:13:18.432273 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
08:13:18.937659 run           create_hlr_db(pid=639): Launched
08:13:19.077816 bus                        /sierra_2: Setting Powered False
08:13:20.160993 run           create_hlr_db(pid=639): Terminated: ok {rc=0}
08:13:20.610708 run pcap-recorder_any(filters='host 10.42.42.2')(pid=641): Launched
08:13:20.971807 run     osmo-hlr_10.42.42.2(pid=642): Launched
08:13:21.100504 run              osmo-stp_10.42.42.5: Starting osmo-stp
08:13:21.693426 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=643): Launched
08:13:22.046557 run     osmo-stp_10.42.42.5(pid=644): Launched
08:13:22.166740 run              osmo-msc_10.42.42.6: Starting osmo-msc
08:13:22.771106 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=645): Launched
08:13:23.125863 run     osmo-msc_10.42.42.6(pid=646): Launched
08:13:23.270704 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
08:13:23.844799 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=647): Launched
08:13:24.197920 run     osmo-mgw_10.42.42.3(pid=648): Launched
08:13:24.317592 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
08:13:24.868985 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=649): Launched
08:13:25.217597 run     osmo-mgw_10.42.42.4(pid=650): Launched
08:13:25.345240 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
08:13:26.118028 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=651): Launched
08:13:26.467104 run     osmo-bsc_10.42.42.7(pid=652): Launched
08:13:30.197083 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
08:13:30.686029 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'remote_user': 'jenkins', 'launch_trx': 'True', 'dev_args': 'LimeSDR-USB,serial=00090706024F0A22', 'type': 'lms', 'channels': [{'rx_path': 'LNAW'}], 'trx_ip': '10.42.42.117', 'egprs': 'disable', 'clock_reference': 'external', 'bts_ip': '10.42.42.53'}}
08:13:31.320956 run                 uname-m(pid=654): Launched
08:13:32.546102 run                 uname-m(pid=654): Terminated: ok {rc=0}
08:13:32.944742 run           rm-remote-dir(pid=655): Launched
08:13:34.172295 run           rm-remote-dir(pid=655): Terminated: ok {rc=0}
08:13:34.565398 run           mk-remote-dir(pid=656): Launched
08:13:35.783220 run           mk-remote-dir(pid=656): Terminated: ok {rc=0}
08:13:36.165220 run           mk-remote-dir(pid=657): Launched
08:13:37.390149 run           mk-remote-dir(pid=657): Terminated: ok {rc=0}
08:13:37.719588 run      scp-inst-to-remote(pid=658): Launched
08:13:39.944433 run      scp-inst-to-remote(pid=658): Terminated: ok {rc=0}
08:13:40.331577 run           mk-remote-dir(pid=660): Launched
08:13:41.544497 run           mk-remote-dir(pid=660): Terminated: ok {rc=0}
08:13:41.873049 run       scp-cfg-to-remote(pid=661): Launched
08:13:43.099408 run       scp-cfg-to-remote(pid=661): Terminated: ok {rc=0}
08:13:43.491963 run           mk-remote-dir(pid=663): Launched
08:13:44.708633 run           mk-remote-dir(pid=663): Terminated: ok {rc=0}
08:13:45.035001 run   scp-wrapper-to-remote(pid=664): Launched
08:13:46.253108 run   scp-wrapper-to-remote(pid=664): Terminated: ok {rc=0}
08:13:46.621657 run            osmo-trx-lms(pid=666): Launched
08:13:46.751331 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
08:13:48.071690 run            osmo-bts-trx(pid=667): Launched
08:13:50.599187 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms+bts1-trx-sysmocell5000/handover.py/01_bts0_started: pass (20.3s)
08:13:50.782938 tst                      handover.py: using MSISDN 13084
08:13:50.997874 bus                        /sierra_2: Setting Powered True
08:13:54.584209 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13084', subscriber_id=1}
08:13:54.760809 tst                      handover.py: using MSISDN 13085
08:13:54.976037 bus                        /sierra_3: Setting Powered True
08:14:10.456440 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13085', subscriber_id=2}
08:14:10.801465 bus                        /sierra_2: Setting Online False
08:14:12.039378 bus                        /sierra_2: Setting Powered False
08:14:17.904596 bus                        /sierra_2: Setting Powered True
08:14:19.317171 bus                        /sierra_2: Setting Online True
08:14:22.038228 tst                        /sierra_2: Connect to ('901', '70')
08:14:22.440464 bus                        /sierra_3: Setting Online False
08:14:23.677424 bus                        /sierra_3: Setting Powered False
08:14:31.984463 bus                        /sierra_3: Setting Powered True
08:14:44.895089 bus                        /sierra_3: Setting Online True
08:14:50.905414 tst                        /sierra_3: Connect to ('901', '70')
08:14:51.110018 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
08:14:51.266705 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
08:14:51.502337 tst                   handover.py:54: waiting for modems to attach...
08:15:01.072557 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms+bts1-trx-sysmocell5000/handover.py/02.1_ms0_attach: FAIL (9.5s)
08:15:01.081676 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-161↪handover_2G:trx-lms+bts1-trx-sysmocell5000↪handover.py:57↪handover.py↪/sierra_2]  [trial-161↪handover_2G:trx-lms+bts1-trx-sysmocell5000↪handover.py:57]
08:15:01.090666 tst                   handover.py:57: Test FAILED (104.3 sec)
08:15:01.133020 run pcap-recorder_any(filters='host 10.42.42.2')(pid=641): Terminating (SIGTERM)
08:15:01.182206 run     osmo-hlr_10.42.42.2(pid=642): Terminating (SIGTERM)
08:15:01.231641 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=643): Terminating (SIGTERM)
08:15:01.284887 run     osmo-stp_10.42.42.5(pid=644): Terminating (SIGTERM)
08:15:01.335091 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=645): Terminating (SIGTERM)
08:15:01.387760 run     osmo-msc_10.42.42.6(pid=646): Terminating (SIGTERM)
08:15:01.437947 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=647): Terminating (SIGTERM)
08:15:01.487740 run     osmo-mgw_10.42.42.3(pid=648): Terminating (SIGTERM)
08:15:01.538212 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=649): Terminating (SIGTERM)
08:15:01.588597 run     osmo-mgw_10.42.42.4(pid=650): Terminating (SIGTERM)
08:15:01.647179 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=651): Terminating (SIGTERM)
08:15:01.698588 run     osmo-bsc_10.42.42.7(pid=652): Terminating (SIGTERM)
08:15:01.748968 run            osmo-trx-lms(pid=666): Terminating (SIGTERM)
08:15:01.800468 run            osmo-bts-trx(pid=667): Terminating (SIGTERM)
08:15:01.811518 ---      ParallelTerminationStrategy: PID 641 died...
08:15:01.881335 run pcap-recorder_any(filters='host 10.42.42.2')(pid=641): Terminated: ok {rc=0}
08:15:01.893701 ---      ParallelTerminationStrategy: PID 642 died...
08:15:01.959197 run     osmo-hlr_10.42.42.2(pid=642): Terminated: ok {rc=0}
08:15:01.971277 ---      ParallelTerminationStrategy: PID 643 died...
08:15:02.038204 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=643): Terminated: ok {rc=0}
08:15:02.050828 ---      ParallelTerminationStrategy: PID 644 died...
08:15:02.118044 run     osmo-stp_10.42.42.5(pid=644): Terminated: ok {rc=0}
08:15:02.131577 ---      ParallelTerminationStrategy: PID 645 died...
08:15:02.196194 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=645): Terminated: ok {rc=0}
08:15:02.208042 ---      ParallelTerminationStrategy: PID 647 died...
08:15:02.272750 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=647): Terminated: ok {rc=0}
08:15:02.284480 ---      ParallelTerminationStrategy: PID 648 died...
08:15:02.348739 run     osmo-mgw_10.42.42.3(pid=648): Terminated {rc=15}
08:15:02.360416 ---      ParallelTerminationStrategy: PID 649 died...
08:15:02.424682 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=649): Terminated: ok {rc=0}
08:15:02.436412 ---      ParallelTerminationStrategy: PID 650 died...
08:15:02.500743 run     osmo-mgw_10.42.42.4(pid=650): Terminated {rc=15}
08:15:02.512271 ---      ParallelTerminationStrategy: PID 651 died...
08:15:02.577256 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=651): Terminated: ok {rc=0}
08:15:02.589170 ---      ParallelTerminationStrategy: PID 666 died...
08:15:02.653660 run            osmo-trx-lms(pid=666): Terminated {rc=36608}
08:15:04.682289 ---      ParallelTerminationStrategy: PID 646 died...
08:15:04.749514 run     osmo-msc_10.42.42.6(pid=646): Terminated: ok {rc=0}
08:15:05.428166 ---      ParallelTerminationStrategy: PID 652 died...
08:15:05.495087 run     osmo-bsc_10.42.42.7(pid=652): Terminated: ok {rc=0}
08:15:08.686284 run            osmo-bts-trx(pid=667): Terminating (SIGINT)
08:15:08.865620 ---      ParallelTerminationStrategy: PID 667 died...
08:15:08.931976 run            osmo-bts-trx(pid=667): Terminated: ok {rc=0}
08:15:09.105437 bus                        /sierra_3: Setting Online False
08:15:10.567742 bus                        /sierra_3: Setting Powered False
08:15:16.941662 bus                        /sierra_3: Setting Powered False
08:15:18.063808 bus                        /sierra_3: Setting Powered False
08:15:19.247082 bus                        /sierra_2: Setting Online False
08:15:20.375091 bus                        /sierra_2: Setting Powered False
08:15:26.638973 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-161↪handover_2G:trx-lms+bts1-trx-sysmocell5000↪handover.py:57↪handover.py↪/sierra_2]