Skip to content

Failed

handover_2G:trx-lms+bts1-trx-sysmocell5000.handover.py (from handover_2G_trx-lms+bts1-trx-sysmocell5000)

Failing for the past 24 builds (Since #140 )
Took 1 min 49 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms+bts1-trx-sysmocell5000↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-163 handover_2G:trx-lms+bts1-trx-sysmocell5000 handover.py
----------------------------------------------
13:05:55.833753 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
13:05:55.848049 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x bts (candidates: 2)
13:05:56.010170 tst                      handover.py: using LAC 16349
13:05:56.153191 tst                      handover.py: using RAC 29
13:05:56.296037 tst                      handover.py: using CellId 16348
13:05:56.439534 tst                      handover.py: using BVCI 3027
13:05:56.466343 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x bts (candidates: 2)
13:05:56.623781 tst                      handover.py: using LAC 16350
13:05:56.757052 tst                      handover.py: using RAC 30
13:05:56.891051 tst                      handover.py: using CellId 16349
13:05:57.024700 tst                      handover.py: using BVCI 3028
13:05:57.051364 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
13:05:57.065017 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
13:05:57.078506 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
13:05:57.092654 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
13:05:57.106568 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x ip_address (candidates: 6)
13:05:57.120565 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x modem (candidates: 2)
13:05:57.255296 tst handover_2G:trx-lms+bts1-trx-sysmocell5000: Using 1 x modem (candidates: 2)
13:05:57.484286 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
13:05:58.000179 run          create_hlr_db(pid=8544): Launched
13:05:58.142993 bus                        /sierra_2: Setting Powered False
13:05:59.228947 run          create_hlr_db(pid=8544): Terminated: ok {rc=0}
13:05:59.685798 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8546): Launched
13:06:00.051658 run    osmo-hlr_10.42.42.2(pid=8547): Launched
13:06:00.184773 run              osmo-stp_10.42.42.5: Starting osmo-stp
13:06:00.783694 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8548): Launched
13:06:01.130593 run    osmo-stp_10.42.42.5(pid=8549): Launched
13:06:01.254134 run              osmo-msc_10.42.42.6: Starting osmo-msc
13:06:01.861165 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8550): Launched
13:06:02.208111 run    osmo-msc_10.42.42.6(pid=8551): Launched
13:06:02.334827 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
13:06:02.923199 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8552): Launched
13:06:03.284029 run    osmo-mgw_10.42.42.3(pid=8553): Launched
13:06:03.407409 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
13:06:03.979095 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8554): Launched
13:06:04.342099 run    osmo-mgw_10.42.42.4(pid=8555): Launched
13:06:04.465632 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
13:06:05.246453 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8556): Launched
13:06:05.609677 run    osmo-bsc_10.42.42.7(pid=8557): Launched
13:06:09.345609 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
13:06:09.843377 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'channels': [{'rx_path': 'LNAW'}], 'type': 'lms', 'bts_ip': '10.42.42.53', 'clock_reference': 'external', 'remote_user': 'jenkins', 'dev_args': 'LimeSDR-USB,serial=00090706024F0A22', 'trx_ip': '10.42.42.117', 'launch_trx': 'True', 'egprs': 'disable'}}
13:06:10.497230 run                uname-m(pid=8558): Launched
13:06:11.728727 run                uname-m(pid=8558): Terminated: ok {rc=0}
13:06:12.139273 run          rm-remote-dir(pid=8559): Launched
13:06:13.374880 run          rm-remote-dir(pid=8559): Terminated: ok {rc=0}
13:06:13.776264 run          mk-remote-dir(pid=8560): Launched
13:06:14.999623 run          mk-remote-dir(pid=8560): Terminated: ok {rc=0}
13:06:15.381893 run          mk-remote-dir(pid=8561): Launched
13:06:16.602505 run          mk-remote-dir(pid=8561): Terminated: ok {rc=0}
13:06:16.930924 run     scp-inst-to-remote(pid=8562): Launched
13:06:19.154094 run     scp-inst-to-remote(pid=8562): Terminated: ok {rc=0}
13:06:19.545916 run          mk-remote-dir(pid=8564): Launched
13:06:20.776979 run          mk-remote-dir(pid=8564): Terminated: ok {rc=0}
13:06:21.123237 run      scp-cfg-to-remote(pid=8565): Launched
13:06:22.345282 run      scp-cfg-to-remote(pid=8565): Terminated: ok {rc=0}
13:06:22.742356 run          mk-remote-dir(pid=8567): Launched
13:06:23.977342 run          mk-remote-dir(pid=8567): Terminated: ok {rc=0}
13:06:24.330748 run  scp-wrapper-to-remote(pid=8568): Launched
13:06:25.557725 run  scp-wrapper-to-remote(pid=8568): Terminated: ok {rc=0}
13:06:25.934128 run           osmo-trx-lms(pid=8570): Launched
13:06:26.067583 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
13:06:27.391892 run           osmo-bts-trx(pid=8571): Launched
13:06:29.989767 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms+bts1-trx-sysmocell5000/handover.py/01_bts0_started: pass (20.5s)
13:06:30.177684 tst                      handover.py: using MSISDN 13192
13:06:30.401703 bus                        /sierra_2: Setting Powered True
13:06:33.950011 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13192', subscriber_id=1}
13:06:34.129272 tst                      handover.py: using MSISDN 13193
13:06:34.352297 bus                        /sierra_3: Setting Powered True
13:06:50.089993 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13193', subscriber_id=2}
13:06:50.440827 bus                        /sierra_2: Setting Online False
13:06:51.679513 bus                        /sierra_2: Setting Powered False
13:06:57.910541 bus                        /sierra_2: Setting Powered True
13:06:59.336615 bus                        /sierra_2: Setting Online True
13:07:02.063917 tst                        /sierra_2: Connect to ('901', '70')
13:07:02.471782 bus                        /sierra_3: Setting Online False
13:07:03.712856 bus                        /sierra_3: Setting Powered False
13:07:12.027984 bus                        /sierra_3: Setting Powered True
13:07:25.802667 bus                        /sierra_3: Setting Online True
13:07:31.931172 tst                        /sierra_3: Connect to ('901', '70')
13:07:32.138396 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
13:07:32.289239 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
13:07:32.528969 tst                   handover.py:54: waiting for modems to attach...
13:07:44.476237 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms+bts1-trx-sysmocell5000/handover.py/02.1_ms0_attach: FAIL (11.9s)
13:07:44.486975 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms+bts1-trx-sysmocell5000↪handover.py:57↪handover.py↪/sierra_2]  [trial-163↪handover_2G:trx-lms+bts1-trx-sysmocell5000↪handover.py:57]
13:07:44.498065 tst                   handover.py:57: Test FAILED (108.7 sec)
13:07:44.549328 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8546): Terminating (SIGTERM)
13:07:44.605517 run    osmo-hlr_10.42.42.2(pid=8547): Terminating (SIGTERM)
13:07:44.663436 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8548): Terminating (SIGTERM)
13:07:44.724157 run    osmo-stp_10.42.42.5(pid=8549): Terminating (SIGTERM)
13:07:44.785497 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8550): Terminating (SIGTERM)
13:07:44.851705 run    osmo-msc_10.42.42.6(pid=8551): Terminating (SIGTERM)
13:07:44.908769 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8552): Terminating (SIGTERM)
13:07:44.964795 run    osmo-mgw_10.42.42.3(pid=8553): Terminating (SIGTERM)
13:07:45.020823 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8554): Terminating (SIGTERM)
13:07:45.076643 run    osmo-mgw_10.42.42.4(pid=8555): Terminating (SIGTERM)
13:07:45.133149 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8556): Terminating (SIGTERM)
13:07:45.188578 run    osmo-bsc_10.42.42.7(pid=8557): Terminating (SIGTERM)
13:07:45.244073 run           osmo-trx-lms(pid=8570): Terminating (SIGTERM)
13:07:45.298349 run           osmo-bts-trx(pid=8571): Terminating (SIGTERM)
13:07:45.310322 ---      ParallelTerminationStrategy: PID 8546 died...
13:07:45.380225 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8546): Terminated: ok {rc=0}
13:07:45.392942 ---      ParallelTerminationStrategy: PID 8547 died...
13:07:45.462604 run    osmo-hlr_10.42.42.2(pid=8547): Terminated: ok {rc=0}
13:07:45.475102 ---      ParallelTerminationStrategy: PID 8548 died...
13:07:45.544901 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8548): Terminated: ok {rc=0}
13:07:45.557457 ---      ParallelTerminationStrategy: PID 8549 died...
13:07:45.627130 run    osmo-stp_10.42.42.5(pid=8549): Terminated: ok {rc=0}
13:07:45.639821 ---      ParallelTerminationStrategy: PID 8550 died...
13:07:45.710015 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8550): Terminated: ok {rc=0}
13:07:45.722806 ---      ParallelTerminationStrategy: PID 8552 died...
13:07:45.793192 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8552): Terminated: ok {rc=0}
13:07:45.805807 ---      ParallelTerminationStrategy: PID 8553 died...
13:07:45.875721 run    osmo-mgw_10.42.42.3(pid=8553): Terminated {rc=15}
13:07:45.888430 ---      ParallelTerminationStrategy: PID 8554 died...
13:07:45.958336 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8554): Terminated: ok {rc=0}
13:07:45.970965 ---      ParallelTerminationStrategy: PID 8555 died...
13:07:46.040868 run    osmo-mgw_10.42.42.4(pid=8555): Terminated {rc=15}
13:07:46.053512 ---      ParallelTerminationStrategy: PID 8556 died...
13:07:46.123069 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8556): Terminated: ok {rc=0}
13:07:46.135607 ---      ParallelTerminationStrategy: PID 8570 died...
13:07:46.205519 run           osmo-trx-lms(pid=8570): Terminated {rc=36608}
13:07:48.234338 ---      ParallelTerminationStrategy: PID 8551 died...
13:07:48.310238 run    osmo-msc_10.42.42.6(pid=8551): Terminated: ok {rc=0}
13:07:48.323258 ---      ParallelTerminationStrategy: PID 8557 died...
13:07:48.391832 run    osmo-bsc_10.42.42.7(pid=8557): Terminated: ok {rc=0}
13:07:52.253495 run           osmo-bts-trx(pid=8571): Terminating (SIGINT)
13:07:52.433495 ---      ParallelTerminationStrategy: PID 8571 died...
13:07:52.504102 run           osmo-bts-trx(pid=8571): Terminated: ok {rc=0}
13:07:52.684786 bus                        /sierra_3: Setting Online False
13:07:54.188555 bus                        /sierra_3: Setting Powered False
13:07:59.984828 bus                        /sierra_3: Setting Powered False
13:08:01.111010 bus                        /sierra_3: Setting Powered False
13:08:02.297705 bus                        /sierra_2: Setting Online False
13:08:03.433941 bus                        /sierra_2: Setting Powered False
13:08:09.633424 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms+bts1-trx-sysmocell5000↪handover.py:57↪handover.py↪/sierra_2]