Skip to content

Failed

handover_2G:trx-lms+bts1-trx-b200.handover.py (from handover_2G_trx-lms+bts1-trx-b200)

Failing for the past 24 builds (Since #140 )
Took 1 min 49 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms+bts1-trx-b200↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-163 handover_2G:trx-lms+bts1-trx-b200 handover.py
----------------------------------------------
13:10:34.799116 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:10:34.813894 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x bts (candidates: 2)
13:10:34.973581 tst                      handover.py: using LAC 16353
13:10:35.107713 tst                      handover.py: using RAC 33
13:10:35.244404 tst                      handover.py: using CellId 16352
13:10:35.378678 tst                      handover.py: using BVCI 3031
13:10:35.405430 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x bts (candidates: 2)
13:10:35.558228 tst                      handover.py: using LAC 16354
13:10:35.690753 tst                      handover.py: using RAC 34
13:10:35.824045 tst                      handover.py: using CellId 16353
13:10:35.956753 tst                      handover.py: using BVCI 3032
13:10:35.983094 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:10:35.996732 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:10:36.010176 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:10:36.023625 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:10:36.036986 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
13:10:36.050243 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x modem (candidates: 2)
13:10:36.184731 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x modem (candidates: 2)
13:10:36.418934 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
13:10:36.969215 run          create_hlr_db(pid=8640): Launched
13:10:37.122932 bus                        /sierra_2: Setting Powered False
13:10:38.209834 run          create_hlr_db(pid=8640): Terminated: ok {rc=0}
13:10:38.649914 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8642): Launched
13:10:39.006357 run    osmo-hlr_10.42.42.2(pid=8643): Launched
13:10:39.138633 run              osmo-stp_10.42.42.5: Starting osmo-stp
13:10:39.735511 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8644): Launched
13:10:40.090538 run    osmo-stp_10.42.42.5(pid=8645): Launched
13:10:40.220181 run              osmo-msc_10.42.42.6: Starting osmo-msc
13:10:40.852214 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8646): Launched
13:10:41.206468 run    osmo-msc_10.42.42.6(pid=8647): Launched
13:10:41.345774 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
13:10:41.964989 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8648): Launched
13:10:42.322597 run    osmo-mgw_10.42.42.3(pid=8649): Launched
13:10:42.454043 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
13:10:43.046357 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8650): Launched
13:10:43.413207 run    osmo-mgw_10.42.42.4(pid=8651): Launched
13:10:43.543111 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
13:10:44.342555 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8652): Launched
13:10:44.698646 run    osmo-bsc_10.42.42.7(pid=8653): Launched
13:10:48.443468 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
13:10:48.936940 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'channels': [{'rx_path': 'LNAW'}], 'type': 'lms', 'bts_ip': '10.42.42.53', 'clock_reference': 'external', 'remote_user': 'jenkins', 'dev_args': 'LimeSDR-USB,serial=00090706024F0A22', 'trx_ip': '10.42.42.117', 'launch_trx': 'True', 'egprs': 'disable'}}
13:10:49.581125 run                uname-m(pid=8654): Launched
13:10:50.802660 run                uname-m(pid=8654): Terminated: ok {rc=0}
13:10:51.199857 run          rm-remote-dir(pid=8655): Launched
13:10:52.435895 run          rm-remote-dir(pid=8655): Terminated: ok {rc=0}
13:10:52.836577 run          mk-remote-dir(pid=8656): Launched
13:10:54.060207 run          mk-remote-dir(pid=8656): Terminated: ok {rc=0}
13:10:54.451357 run          mk-remote-dir(pid=8657): Launched
13:10:55.682585 run          mk-remote-dir(pid=8657): Terminated: ok {rc=0}
13:10:56.028033 run     scp-inst-to-remote(pid=8658): Launched
13:10:58.252088 run     scp-inst-to-remote(pid=8658): Terminated: ok {rc=0}
13:10:58.643916 run          mk-remote-dir(pid=8660): Launched
13:10:59.873767 run          mk-remote-dir(pid=8660): Terminated: ok {rc=0}
13:11:00.219388 run      scp-cfg-to-remote(pid=8661): Launched
13:11:01.452776 run      scp-cfg-to-remote(pid=8661): Terminated: ok {rc=0}
13:11:01.859844 run          mk-remote-dir(pid=8663): Launched
13:11:03.093098 run          mk-remote-dir(pid=8663): Terminated: ok {rc=0}
13:11:03.442256 run  scp-wrapper-to-remote(pid=8664): Launched
13:11:04.677305 run  scp-wrapper-to-remote(pid=8664): Terminated: ok {rc=0}
13:11:05.063115 run           osmo-trx-lms(pid=8666): Launched
13:11:05.195575 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
13:11:06.521186 run           osmo-bts-trx(pid=8667): Launched
13:11:09.060169 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms+bts1-trx-b200/handover.py/01_bts0_started: pass (20.5s)
13:11:09.244692 tst                      handover.py: using MSISDN 13196
13:11:09.467188 bus                        /sierra_2: Setting Powered True
13:11:13.025262 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13196', subscriber_id=1}
13:11:13.205601 tst                      handover.py: using MSISDN 13197
13:11:13.426974 bus                        /sierra_3: Setting Powered True
13:11:29.141101 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13197', subscriber_id=2}
13:11:29.446900 bus                        /sierra_2: Setting Online False
13:11:30.658538 bus                        /sierra_2: Setting Powered False
13:11:36.854862 bus                        /sierra_2: Setting Powered True
13:11:38.265932 bus                        /sierra_2: Setting Online True
13:11:40.877540 tst                        /sierra_2: Connect to ('901', '70')
13:11:41.228268 bus                        /sierra_3: Setting Online False
13:11:42.440930 bus                        /sierra_3: Setting Powered False
13:11:51.356252 bus                        /sierra_3: Setting Powered True
13:12:05.199848 bus                        /sierra_3: Setting Online True
13:12:11.465045 tst                        /sierra_3: Connect to ('901', '70')
13:12:11.675376 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
13:12:11.827553 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
13:12:12.070668 tst                   handover.py:54: waiting for modems to attach...
13:12:23.622247 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms+bts1-trx-b200/handover.py/02.1_ms0_attach: FAIL (11.5s)
13:12:23.632464 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms+bts1-trx-b200↪handover.py:57↪handover.py↪/sierra_2]  [trial-163↪handover_2G:trx-lms+bts1-trx-b200↪handover.py:57]
13:12:23.641163 tst                   handover.py:57: Test FAILED (108.8 sec)
13:12:23.683822 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8642): Terminating (SIGTERM)
13:12:23.733815 run    osmo-hlr_10.42.42.2(pid=8643): Terminating (SIGTERM)
13:12:23.783897 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8644): Terminating (SIGTERM)
13:12:23.836784 run    osmo-stp_10.42.42.5(pid=8645): Terminating (SIGTERM)
13:12:23.893049 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8646): Terminating (SIGTERM)
13:12:23.945897 run    osmo-msc_10.42.42.6(pid=8647): Terminating (SIGTERM)
13:12:23.995925 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8648): Terminating (SIGTERM)
13:12:24.045974 run    osmo-mgw_10.42.42.3(pid=8649): Terminating (SIGTERM)
13:12:24.096558 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8650): Terminating (SIGTERM)
13:12:24.146008 run    osmo-mgw_10.42.42.4(pid=8651): Terminating (SIGTERM)
13:12:24.203335 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8652): Terminating (SIGTERM)
13:12:24.253816 run    osmo-bsc_10.42.42.7(pid=8653): Terminating (SIGTERM)
13:12:24.303534 run           osmo-trx-lms(pid=8666): Terminating (SIGTERM)
13:12:24.352936 run           osmo-bts-trx(pid=8667): Terminating (SIGTERM)
13:12:24.364113 ---      ParallelTerminationStrategy: PID 8642 died...
13:12:24.429929 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8642): Terminated: ok {rc=0}
13:12:24.441666 ---      ParallelTerminationStrategy: PID 8643 died...
13:12:24.506586 run    osmo-hlr_10.42.42.2(pid=8643): Terminated: ok {rc=0}
13:12:24.518262 ---      ParallelTerminationStrategy: PID 8644 died...
13:12:24.583180 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8644): Terminated: ok {rc=0}
13:12:24.594886 ---      ParallelTerminationStrategy: PID 8645 died...
13:12:24.659753 run    osmo-stp_10.42.42.5(pid=8645): Terminated: ok {rc=0}
13:12:24.671465 ---      ParallelTerminationStrategy: PID 8646 died...
13:12:24.736405 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8646): Terminated: ok {rc=0}
13:12:24.748073 ---      ParallelTerminationStrategy: PID 8648 died...
13:12:24.813152 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8648): Terminated: ok {rc=0}
13:12:24.825557 ---      ParallelTerminationStrategy: PID 8649 died...
13:12:24.890744 run    osmo-mgw_10.42.42.3(pid=8649): Terminated {rc=15}
13:12:24.902845 ---      ParallelTerminationStrategy: PID 8650 died...
13:12:24.968124 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8650): Terminated: ok {rc=0}
13:12:24.979969 ---      ParallelTerminationStrategy: PID 8651 died...
13:12:25.045674 run    osmo-mgw_10.42.42.4(pid=8651): Terminated {rc=15}
13:12:25.057436 ---      ParallelTerminationStrategy: PID 8652 died...
13:12:25.122546 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8652): Terminated: ok {rc=0}
13:12:25.134412 ---      ParallelTerminationStrategy: PID 8666 died...
13:12:25.199563 run           osmo-trx-lms(pid=8666): Terminated {rc=36608}
13:12:27.228048 ---      ParallelTerminationStrategy: PID 8647 died...
13:12:27.296067 run    osmo-msc_10.42.42.6(pid=8647): Terminated: ok {rc=0}
13:12:27.974839 ---      ParallelTerminationStrategy: PID 8653 died...
13:12:28.042140 run    osmo-bsc_10.42.42.7(pid=8653): Terminated: ok {rc=0}
13:12:31.234088 run           osmo-bts-trx(pid=8667): Terminating (SIGINT)
13:12:31.413486 ---      ParallelTerminationStrategy: PID 8667 died...
13:12:31.480757 run           osmo-bts-trx(pid=8667): Terminated: ok {rc=0}
13:12:31.663444 bus                        /sierra_3: Setting Online False
13:12:33.099097 bus                        /sierra_3: Setting Powered False
13:12:38.943842 bus                        /sierra_3: Setting Powered False
13:12:40.065794 bus                        /sierra_3: Setting Powered False
13:12:41.247167 bus                        /sierra_2: Setting Online False
13:12:42.405423 bus                        /sierra_2: Setting Powered False
13:12:48.643366 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms+bts1-trx-b200↪handover.py:57↪handover.py↪/sierra_2]