Skip to content

Failed

handover_2G:trx-lms-limenet-micro+bts1-sysmo.handover.py (from handover_2G_trx-lms-limenet-micro+bts1-sysmo)

Failing for the past 21 builds (Since #140 )
Took 1 min 47 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-sysmo↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-160 handover_2G:trx-lms-limenet-micro+bts1-sysmo handover.py
----------------------------------------------
05:58:05.999631 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x ip_address (candidates: 6)
05:58:06.015317 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x bts (candidates: 2)
05:58:06.167064 tst                      handover.py: using LAC 16143
05:58:06.296553 tst                      handover.py: using RAC 78
05:58:06.426425 tst                      handover.py: using CellId 16142
05:58:06.556139 tst                      handover.py: using BVCI 2821
05:58:06.582086 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x bts (candidates: 2)
05:58:06.719294 tst                      handover.py: using LAC 16144
05:58:06.849199 tst                      handover.py: using RAC 79
05:58:06.979583 tst                      handover.py: using CellId 16143
05:58:07.115758 tst                      handover.py: using BVCI 2822
05:58:07.141848 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x ip_address (candidates: 6)
05:58:07.155841 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x ip_address (candidates: 6)
05:58:07.169706 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x ip_address (candidates: 6)
05:58:07.183647 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x ip_address (candidates: 6)
05:58:07.197459 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x ip_address (candidates: 6)
05:58:07.211162 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x modem (candidates: 2)
05:58:07.342536 tst handover_2G:trx-lms-limenet-micro+bts1-sysmo: Using 1 x modem (candidates: 2)
05:58:07.571163 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
05:58:08.074756 run         create_hlr_db(pid=29680): Launched
05:58:08.214947 bus                        /sierra_2: Setting Powered False
05:58:09.298218 run         create_hlr_db(pid=29680): Terminated: ok {rc=0}
05:58:09.745191 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29682): Launched
05:58:10.097920 run   osmo-hlr_10.42.42.2(pid=29683): Launched
05:58:10.220377 run              osmo-stp_10.42.42.5: Starting osmo-stp
05:58:10.780901 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29684): Launched
05:58:11.120201 run   osmo-stp_10.42.42.5(pid=29685): Launched
05:58:11.241110 run              osmo-msc_10.42.42.6: Starting osmo-msc
05:58:11.842855 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29686): Launched
05:58:12.196981 run   osmo-msc_10.42.42.6(pid=29687): Launched
05:58:12.321209 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
05:58:12.892239 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29688): Launched
05:58:13.245620 run   osmo-mgw_10.42.42.3(pid=29689): Launched
05:58:13.367579 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
05:58:13.916716 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29690): Launched
05:58:14.267054 run   osmo-mgw_10.42.42.4(pid=29691): Launched
05:58:14.394880 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
05:58:15.180172 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29692): Launched
05:58:15.520072 run   osmo-bsc_10.42.42.7(pid=29693): Launched
05:58:19.215080 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
05:58:19.681792 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'dev_args': 'LimeNET-Micro,serial=005839AF25B352', 'bts_ip': '10.42.42.53', 'trx_ip': '10.42.42.123', 'channels': [{}], 'remote_user': 'pi', 'type': 'lms', 'launch_trx': 'True', 'egprs': 'disable', 'clock_reference': 'external'}}
05:58:20.283763 run               uname-m(pid=29694): Launched
05:58:21.499196 run               uname-m(pid=29694): Terminated: ok {rc=0}
05:58:21.885386 run         rm-remote-dir(pid=29695): Launched
05:58:23.114979 run         rm-remote-dir(pid=29695): Terminated: ok {rc=0}
05:58:23.507091 run         mk-remote-dir(pid=29696): Launched
05:58:24.725579 run         mk-remote-dir(pid=29696): Terminated: ok {rc=0}
05:58:25.107262 run         mk-remote-dir(pid=29697): Launched
05:58:26.333751 run         mk-remote-dir(pid=29697): Terminated: ok {rc=0}
05:58:26.661972 run     scp-cfg-to-remote(pid=29698): Launched
05:58:27.878484 run     scp-cfg-to-remote(pid=29698): Terminated: ok {rc=0}
05:58:28.266964 run         mk-remote-dir(pid=29700): Launched
05:58:29.495172 run         mk-remote-dir(pid=29700): Terminated: ok {rc=0}
05:58:29.832086 run scp-wrapper-to-remote(pid=29701): Launched
05:58:31.050696 run scp-wrapper-to-remote(pid=29701): Terminated: ok {rc=0}
05:58:31.421697 run          osmo-trx-lms(pid=29703): Launched
05:58:31.553572 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
05:58:33.867018 run          osmo-bts-trx(pid=29704): Launched
05:58:39.420895 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-sysmo/handover.py/01_bts0_started: pass (20.1s)
05:58:39.645501 tst                      handover.py: using MSISDN 13040
05:58:39.852994 bus                        /sierra_2: Setting Powered True
05:58:43.409829 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13040', subscriber_id=1}
05:58:43.605759 tst                      handover.py: using MSISDN 13041
05:58:43.836738 bus                        /sierra_3: Setting Powered True
05:58:58.513049 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13041', subscriber_id=2}
05:58:58.858789 bus                        /sierra_2: Setting Online False
05:59:00.095364 bus                        /sierra_2: Setting Powered False
05:59:05.907640 bus                        /sierra_2: Setting Powered True
05:59:07.314699 bus                        /sierra_2: Setting Online True
05:59:10.085148 tst                        /sierra_2: Connect to ('901', '70')
05:59:10.485710 bus                        /sierra_3: Setting Online False
05:59:11.785553 bus                        /sierra_3: Setting Powered False
05:59:19.711933 bus                        /sierra_3: Setting Powered True
05:59:33.505055 bus                        /sierra_3: Setting Online True
05:59:39.785899 tst                        /sierra_3: Connect to ('901', '70')
05:59:39.996631 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
05:59:40.148753 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
05:59:40.398479 tst                   handover.py:54: waiting for modems to attach...
05:59:52.319291 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-sysmo/handover.py/02.1_ms0_attach: FAIL (11.9s)
05:59:52.330514 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-sysmo↪handover.py:57↪handover.py↪/sierra_2]  [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-sysmo↪handover.py:57]
05:59:52.341961 tst                   handover.py:57: Test FAILED (106.3 sec)
05:59:52.392314 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29682): Terminating (SIGTERM)
05:59:52.449272 run   osmo-hlr_10.42.42.2(pid=29683): Terminating (SIGTERM)
05:59:52.508747 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29684): Terminating (SIGTERM)
05:59:52.570097 run   osmo-stp_10.42.42.5(pid=29685): Terminating (SIGTERM)
05:59:52.628928 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29686): Terminating (SIGTERM)
05:59:52.687365 run   osmo-msc_10.42.42.6(pid=29687): Terminating (SIGTERM)
05:59:52.744587 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29688): Terminating (SIGTERM)
05:59:52.801479 run   osmo-mgw_10.42.42.3(pid=29689): Terminating (SIGTERM)
05:59:52.858314 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29690): Terminating (SIGTERM)
05:59:52.914808 run   osmo-mgw_10.42.42.4(pid=29691): Terminating (SIGTERM)
05:59:52.971160 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29692): Terminating (SIGTERM)
05:59:53.027516 run   osmo-bsc_10.42.42.7(pid=29693): Terminating (SIGTERM)
05:59:53.084426 run          osmo-trx-lms(pid=29703): Terminating (SIGTERM)
05:59:53.139143 run          osmo-bts-trx(pid=29704): Terminating (SIGTERM)
05:59:53.150955 ---      ParallelTerminationStrategy: PID 29682 died...
05:59:53.220662 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29682): Terminated: ok {rc=0}
05:59:53.233642 ---      ParallelTerminationStrategy: PID 29683 died...
05:59:53.303055 run   osmo-hlr_10.42.42.2(pid=29683): Terminated: ok {rc=0}
05:59:53.315653 ---      ParallelTerminationStrategy: PID 29684 died...
05:59:53.384843 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29684): Terminated: ok {rc=0}
05:59:53.397330 ---      ParallelTerminationStrategy: PID 29685 died...
05:59:53.466630 run   osmo-stp_10.42.42.5(pid=29685): Terminated: ok {rc=0}
05:59:53.479247 ---      ParallelTerminationStrategy: PID 29686 died...
05:59:53.548144 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29686): Terminated: ok {rc=0}
05:59:53.560844 ---      ParallelTerminationStrategy: PID 29688 died...
05:59:53.630190 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29688): Terminated: ok {rc=0}
05:59:53.643042 ---      ParallelTerminationStrategy: PID 29689 died...
05:59:53.712018 run   osmo-mgw_10.42.42.3(pid=29689): Terminated {rc=15}
05:59:53.724669 ---      ParallelTerminationStrategy: PID 29690 died...
05:59:53.793992 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29690): Terminated: ok {rc=0}
05:59:53.806546 ---      ParallelTerminationStrategy: PID 29691 died...
05:59:53.875357 run   osmo-mgw_10.42.42.4(pid=29691): Terminated {rc=15}
05:59:53.888030 ---      ParallelTerminationStrategy: PID 29692 died...
05:59:53.956904 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29692): Terminated: ok {rc=0}
05:59:53.969457 ---      ParallelTerminationStrategy: PID 29703 died...
05:59:54.038749 run          osmo-trx-lms(pid=29703): Terminated {rc=36608}
05:59:56.068842 ---      ParallelTerminationStrategy: PID 29687 died...
05:59:56.137902 run   osmo-msc_10.42.42.6(pid=29687): Terminated: ok {rc=0}
05:59:56.817154 ---      ParallelTerminationStrategy: PID 29693 died...
05:59:56.884518 run   osmo-bsc_10.42.42.7(pid=29693): Terminated: ok {rc=0}
06:00:00.077132 run          osmo-bts-trx(pid=29704): Terminating (SIGINT)
06:00:00.258675 ---      ParallelTerminationStrategy: PID 29704 died...
06:00:00.329250 run          osmo-bts-trx(pid=29704): Terminated: ok {rc=0}
06:00:00.506087 bus                        /sierra_3: Setting Online False
06:00:01.978691 bus                        /sierra_3: Setting Powered False
06:00:07.962910 bus                        /sierra_3: Setting Powered False
06:00:09.087829 bus                        /sierra_3: Setting Powered False
06:00:10.275151 bus                        /sierra_2: Setting Online False
06:00:11.425200 bus                        /sierra_2: Setting Powered False
06:00:17.642398 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-160↪handover_2G:trx-lms-limenet-micro+bts1-sysmo↪handover.py:57↪handover.py↪/sierra_2]