Skip to content

Failed

handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000.handover.py (from handover_2G_trx-lms-limenet-micro+bts1-trx-sysmocell5000)

Failing for the past 21 builds (Since #140 )
Took 1 min 56 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-160 handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000 handover.py
----------------------------------------------
05:55:43.015842 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
05:55:43.029880 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x bts (candidates: 2)
05:55:43.182279 tst                      handover.py: using LAC 16141
05:55:43.315394 tst                      handover.py: using RAC 76
05:55:43.448015 tst                      handover.py: using CellId 16140
05:55:43.580114 tst                      handover.py: using BVCI 2819
05:55:43.606230 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x bts (candidates: 2)
05:55:43.765531 tst                      handover.py: using LAC 16142
05:55:43.897192 tst                      handover.py: using RAC 77
05:55:44.028995 tst                      handover.py: using CellId 16141
05:55:44.161586 tst                      handover.py: using BVCI 2820
05:55:44.187676 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
05:55:44.202093 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
05:55:44.216162 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
05:55:44.230282 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
05:55:44.244197 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
05:55:44.258208 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x modem (candidates: 2)
05:55:44.392820 tst handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000: Using 1 x modem (candidates: 2)
05:55:44.622906 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
05:55:45.128629 run         create_hlr_db(pid=29645): Launched
05:55:45.270239 bus                        /sierra_2: Setting Powered False
05:55:46.352707 run         create_hlr_db(pid=29645): Terminated: ok {rc=0}
05:55:46.793716 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29647): Launched
05:55:47.133450 run   osmo-hlr_10.42.42.2(pid=29648): Launched
05:55:47.255760 run              osmo-stp_10.42.42.5: Starting osmo-stp
05:55:47.825383 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29649): Launched
05:55:48.179490 run   osmo-stp_10.42.42.5(pid=29650): Launched
05:55:48.301000 run              osmo-msc_10.42.42.6: Starting osmo-msc
05:55:48.896802 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29651): Launched
05:55:49.237699 run   osmo-msc_10.42.42.6(pid=29652): Launched
05:55:49.362350 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
05:55:49.937643 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29653): Launched
05:55:50.292036 run   osmo-mgw_10.42.42.3(pid=29654): Launched
05:55:50.413671 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
05:55:50.974959 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29655): Launched
05:55:51.331690 run   osmo-mgw_10.42.42.4(pid=29656): Launched
05:55:51.454124 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
05:55:52.220494 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29657): Launched
05:55:52.581371 run   osmo-bsc_10.42.42.7(pid=29658): Launched
05:55:56.313311 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
05:55:56.798818 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'dev_args': 'LimeNET-Micro,serial=005839AF25B352', 'bts_ip': '10.42.42.53', 'trx_ip': '10.42.42.123', 'channels': [{}], 'remote_user': 'pi', 'type': 'lms', 'launch_trx': 'True', 'egprs': 'disable', 'clock_reference': 'external'}}
05:55:57.432817 run               uname-m(pid=29659): Launched
05:55:58.658986 run               uname-m(pid=29659): Terminated: ok {rc=0}
05:55:59.052575 run         rm-remote-dir(pid=29660): Launched
05:56:00.281801 run         rm-remote-dir(pid=29660): Terminated: ok {rc=0}
05:56:00.684550 run         mk-remote-dir(pid=29661): Launched
05:56:01.913466 run         mk-remote-dir(pid=29661): Terminated: ok {rc=0}
05:56:02.301882 run         mk-remote-dir(pid=29662): Launched
05:56:03.518370 run         mk-remote-dir(pid=29662): Terminated: ok {rc=0}
05:56:03.838571 run     scp-cfg-to-remote(pid=29663): Launched
05:56:05.054925 run     scp-cfg-to-remote(pid=29663): Terminated: ok {rc=0}
05:56:05.436572 run         mk-remote-dir(pid=29665): Launched
05:56:06.655016 run         mk-remote-dir(pid=29665): Terminated: ok {rc=0}
05:56:06.992419 run scp-wrapper-to-remote(pid=29666): Launched
05:56:08.222600 run scp-wrapper-to-remote(pid=29666): Terminated: ok {rc=0}
05:56:08.594931 run          osmo-trx-lms(pid=29668): Launched
05:56:08.717189 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
05:56:11.030629 run          osmo-bts-trx(pid=29669): Launched
05:56:16.580379 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000/handover.py/01_bts0_started: pass (20.1s)
05:56:16.764685 tst                      handover.py: using MSISDN 13038
05:56:16.986208 bus                        /sierra_2: Setting Powered True
05:56:20.555171 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13038', subscriber_id=1}
05:56:20.734258 tst                      handover.py: using MSISDN 13039
05:56:20.958661 bus                        /sierra_3: Setting Powered True
05:56:37.015559 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13039', subscriber_id=2}
05:56:37.363468 bus                        /sierra_2: Setting Online False
05:56:38.603577 bus                        /sierra_2: Setting Powered False
05:56:44.909708 bus                        /sierra_2: Setting Powered True
05:56:46.332199 bus                        /sierra_2: Setting Online True
05:56:49.092008 tst                        /sierra_2: Connect to ('901', '70')
05:56:49.494805 bus                        /sierra_3: Setting Online False
05:56:50.733959 bus                        /sierra_3: Setting Powered False
05:56:59.030770 bus                        /sierra_3: Setting Powered True
05:57:12.666710 bus                        /sierra_3: Setting Online True
05:57:18.989198 tst                        /sierra_3: Connect to ('901', '70')
05:57:19.199427 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
05:57:19.351687 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
05:57:19.594800 tst                   handover.py:54: waiting for modems to attach...
05:57:38.846577 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000/handover.py/02.1_ms0_attach: FAIL (19.2s)
05:57:38.857692 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000↪handover.py:57↪handover.py↪/sierra_2]  [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000↪handover.py:57]
05:57:38.869397 tst                   handover.py:57: Test FAILED (115.8 sec)
05:57:38.918761 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29647): Terminating (SIGTERM)
05:57:38.976030 run   osmo-hlr_10.42.42.2(pid=29648): Terminating (SIGTERM)
05:57:39.036721 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29649): Terminating (SIGTERM)
05:57:39.096991 run   osmo-stp_10.42.42.5(pid=29650): Terminating (SIGTERM)
05:57:39.163841 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29651): Terminating (SIGTERM)
05:57:39.224754 run   osmo-msc_10.42.42.6(pid=29652): Terminating (SIGTERM)
05:57:39.282065 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29653): Terminating (SIGTERM)
05:57:39.339709 run   osmo-mgw_10.42.42.3(pid=29654): Terminating (SIGTERM)
05:57:39.395978 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29655): Terminating (SIGTERM)
05:57:39.452157 run   osmo-mgw_10.42.42.4(pid=29656): Terminating (SIGTERM)
05:57:39.508790 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29657): Terminating (SIGTERM)
05:57:39.564799 run   osmo-bsc_10.42.42.7(pid=29658): Terminating (SIGTERM)
05:57:39.620832 run          osmo-trx-lms(pid=29668): Terminating (SIGTERM)
05:57:39.676097 run          osmo-bts-trx(pid=29669): Terminating (SIGTERM)
05:57:39.687930 ---      ParallelTerminationStrategy: PID 29647 died...
05:57:39.757460 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29647): Terminated: ok {rc=0}
05:57:39.770011 ---      ParallelTerminationStrategy: PID 29648 died...
05:57:39.839107 run   osmo-hlr_10.42.42.2(pid=29648): Terminated: ok {rc=0}
05:57:39.851650 ---      ParallelTerminationStrategy: PID 29649 died...
05:57:39.920786 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29649): Terminated: ok {rc=0}
05:57:39.933383 ---      ParallelTerminationStrategy: PID 29650 died...
05:57:40.002236 run   osmo-stp_10.42.42.5(pid=29650): Terminated: ok {rc=0}
05:57:40.014770 ---      ParallelTerminationStrategy: PID 29651 died...
05:57:40.086121 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29651): Terminated: ok {rc=0}
05:57:40.098711 ---      ParallelTerminationStrategy: PID 29653 died...
05:57:40.168045 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29653): Terminated: ok {rc=0}
05:57:40.180801 ---      ParallelTerminationStrategy: PID 29654 died...
05:57:40.250676 run   osmo-mgw_10.42.42.3(pid=29654): Terminated {rc=15}
05:57:40.263132 ---      ParallelTerminationStrategy: PID 29655 died...
05:57:40.332220 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29655): Terminated: ok {rc=0}
05:57:40.344855 ---      ParallelTerminationStrategy: PID 29656 died...
05:57:40.413536 run   osmo-mgw_10.42.42.4(pid=29656): Terminated {rc=15}
05:57:40.426093 ---      ParallelTerminationStrategy: PID 29657 died...
05:57:40.495027 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29657): Terminated: ok {rc=0}
05:57:40.507378 ---      ParallelTerminationStrategy: PID 29668 died...
05:57:40.577040 run          osmo-trx-lms(pid=29668): Terminated {rc=36608}
05:57:42.606117 ---      ParallelTerminationStrategy: PID 29652 died...
05:57:42.675113 run   osmo-msc_10.42.42.6(pid=29652): Terminated: ok {rc=0}
05:57:43.357980 ---      ParallelTerminationStrategy: PID 29658 died...
05:57:43.428061 run   osmo-bsc_10.42.42.7(pid=29658): Terminated: ok {rc=0}
05:57:46.620941 run          osmo-bts-trx(pid=29669): Terminating (SIGINT)
05:57:46.801006 ---      ParallelTerminationStrategy: PID 29669 died...
05:57:46.871331 run          osmo-bts-trx(pid=29669): Terminated: ok {rc=0}
05:57:47.051574 bus                        /sierra_3: Setting Online False
05:57:48.535787 bus                        /sierra_3: Setting Powered False
05:57:54.910687 bus                        /sierra_3: Setting Powered False
05:57:56.031598 bus                        /sierra_3: Setting Powered False
05:57:57.211238 bus                        /sierra_2: Setting Online False
05:57:58.346713 bus                        /sierra_2: Setting Powered False
05:58:03.863783 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-trx-sysmocell5000↪handover.py:57↪handover.py↪/sierra_2]