Skip to content

Failed

handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx.handover.py (from handover_2G_trx-lms-limenet-micro+bts1-trx-umtrx)

Failing for the past 23 builds (Since #140 )
Took 1 min 48 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-162↪handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-162 handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx handover.py
----------------------------------------------
10:53:37.782504 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
10:53:37.796630 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
10:53:37.945761 tst                      handover.py: using LAC 16291
10:53:38.075633 tst                      handover.py: using RAC 226
10:53:38.205323 tst                      handover.py: using CellId 16290
10:53:38.337717 tst                      handover.py: using BVCI 2969
10:53:38.363335 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x bts (candidates: 2)
10:53:38.516723 tst                      handover.py: using LAC 16292
10:53:38.648905 tst                      handover.py: using RAC 227
10:53:38.785412 tst                      handover.py: using CellId 16291
10:53:38.924065 tst                      handover.py: using BVCI 2970
10:53:38.951192 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
10:53:38.966064 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
10:53:38.980932 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
10:53:38.995782 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
10:53:39.010777 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x ip_address (candidates: 6)
10:53:39.025406 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
10:53:39.153985 tst handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx: Using 1 x modem (candidates: 2)
10:53:39.385282 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
10:53:39.916983 run          create_hlr_db(pid=6472): Launched
10:53:40.065800 bus                        /sierra_2: Setting Powered False
10:53:41.151155 run          create_hlr_db(pid=6472): Terminated: ok {rc=0}
10:53:41.593249 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6474): Launched
10:53:41.945830 run    osmo-hlr_10.42.42.2(pid=6475): Launched
10:53:42.066529 run              osmo-stp_10.42.42.5: Starting osmo-stp
10:53:42.622380 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6476): Launched
10:53:42.958701 run    osmo-stp_10.42.42.5(pid=6477): Launched
10:53:43.078625 run              osmo-msc_10.42.42.6: Starting osmo-msc
10:53:43.667475 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6478): Launched
10:53:44.006521 run    osmo-msc_10.42.42.6(pid=6479): Launched
10:53:44.128596 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
10:53:44.693021 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6480): Launched
10:53:45.046415 run    osmo-mgw_10.42.42.3(pid=6481): Launched
10:53:45.171989 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
10:53:45.739689 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6482): Launched
10:53:46.088045 run    osmo-mgw_10.42.42.4(pid=6483): Launched
10:53:46.209011 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
10:53:46.961624 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6484): Launched
10:53:47.313651 run    osmo-bsc_10.42.42.7(pid=6485): Launched
10:53:51.001110 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
10:53:51.459381 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'launch_trx': 'True', 'dev_args': 'LimeNET-Micro,serial=005839AF25B352', 'clock_reference': 'external', 'trx_ip': '10.42.42.123', 'egprs': 'disable', 'channels': [{}], 'type': 'lms', 'bts_ip': '10.42.42.53', 'remote_user': 'pi'}}
10:53:52.052171 run                uname-m(pid=6486): Launched
10:53:53.264848 run                uname-m(pid=6486): Terminated: ok {rc=0}
10:53:53.639264 run          rm-remote-dir(pid=6487): Launched
10:53:54.853330 run          rm-remote-dir(pid=6487): Terminated: ok {rc=0}
10:53:55.237395 run          mk-remote-dir(pid=6491): Launched
10:53:56.463430 run          mk-remote-dir(pid=6491): Terminated: ok {rc=0}
10:53:56.852269 run          mk-remote-dir(pid=6492): Launched
10:53:58.072300 run          mk-remote-dir(pid=6492): Terminated: ok {rc=0}
10:53:58.403438 run      scp-cfg-to-remote(pid=6493): Launched
10:53:59.625097 run      scp-cfg-to-remote(pid=6493): Terminated: ok {rc=0}
10:54:00.021124 run          mk-remote-dir(pid=6495): Launched
10:54:01.246694 run          mk-remote-dir(pid=6495): Terminated: ok {rc=0}
10:54:01.583374 run  scp-wrapper-to-remote(pid=6496): Launched
10:54:02.807798 run  scp-wrapper-to-remote(pid=6496): Terminated: ok {rc=0}
10:54:03.178567 run           osmo-trx-lms(pid=6498): Launched
10:54:03.307543 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
10:54:05.626763 run           osmo-bts-trx(pid=6499): Launched
10:54:11.198937 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx/handover.py/01_bts0_started: pass (20.0s)
10:54:11.403215 tst                      handover.py: using MSISDN 13152
10:54:11.627150 bus                        /sierra_2: Setting Powered True
10:54:15.196071 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13152', subscriber_id=1}
10:54:15.380250 tst                      handover.py: using MSISDN 13153
10:54:15.597543 bus                        /sierra_3: Setting Powered True
10:54:31.387555 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13153', subscriber_id=2}
10:54:31.737352 bus                        /sierra_2: Setting Online False
10:54:32.977940 bus                        /sierra_2: Setting Powered False
10:54:38.914314 bus                        /sierra_2: Setting Powered True
10:54:40.347681 bus                        /sierra_2: Setting Online True
10:54:43.090973 tst                        /sierra_2: Connect to ('901', '70')
10:54:43.495366 bus                        /sierra_3: Setting Online False
10:54:44.737255 bus                        /sierra_3: Setting Powered False
10:54:53.037343 bus                        /sierra_3: Setting Powered True
10:55:05.763903 bus                        /sierra_3: Setting Online True
10:55:11.146726 tst                        /sierra_3: Connect to ('901', '70')
10:55:11.333825 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
10:55:11.459562 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
10:55:11.663780 tst                   handover.py:54: waiting for modems to attach...
10:55:25.201414 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx/handover.py/02.1_ms0_attach: FAIL (13.5s)
10:55:25.210571 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-162↪handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx↪handover.py:57↪handover.py↪/sierra_2]  [trial-162↪handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx↪handover.py:57]
10:55:25.219566 tst                   handover.py:57: Test FAILED (107.4 sec)
10:55:25.261680 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6474): Terminating (SIGTERM)
10:55:25.311521 run    osmo-hlr_10.42.42.2(pid=6475): Terminating (SIGTERM)
10:55:25.360866 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6476): Terminating (SIGTERM)
10:55:25.412897 run    osmo-stp_10.42.42.5(pid=6477): Terminating (SIGTERM)
10:55:25.470929 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6478): Terminating (SIGTERM)
10:55:25.522911 run    osmo-msc_10.42.42.6(pid=6479): Terminating (SIGTERM)
10:55:25.572057 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6480): Terminating (SIGTERM)
10:55:25.620519 run    osmo-mgw_10.42.42.3(pid=6481): Terminating (SIGTERM)
10:55:25.675339 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6482): Terminating (SIGTERM)
10:55:25.723738 run    osmo-mgw_10.42.42.4(pid=6483): Terminating (SIGTERM)
10:55:25.772913 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6484): Terminating (SIGTERM)
10:55:25.821296 run    osmo-bsc_10.42.42.7(pid=6485): Terminating (SIGTERM)
10:55:25.869673 run           osmo-trx-lms(pid=6498): Terminating (SIGTERM)
10:55:25.918045 run           osmo-bts-trx(pid=6499): Terminating (SIGTERM)
10:55:25.929102 ---      ParallelTerminationStrategy: PID 6474 died...
10:55:25.993019 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6474): Terminated: ok {rc=0}
10:55:26.004544 ---      ParallelTerminationStrategy: PID 6475 died...
10:55:26.067835 run    osmo-hlr_10.42.42.2(pid=6475): Terminated: ok {rc=0}
10:55:26.079290 ---      ParallelTerminationStrategy: PID 6476 died...
10:55:26.142789 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6476): Terminated: ok {rc=0}
10:55:26.154320 ---      ParallelTerminationStrategy: PID 6477 died...
10:55:26.217765 run    osmo-stp_10.42.42.5(pid=6477): Terminated: ok {rc=0}
10:55:26.229302 ---      ParallelTerminationStrategy: PID 6478 died...
10:55:26.292977 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6478): Terminated: ok {rc=0}
10:55:26.304538 ---      ParallelTerminationStrategy: PID 6480 died...
10:55:26.368719 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6480): Terminated: ok {rc=0}
10:55:26.380259 ---      ParallelTerminationStrategy: PID 6481 died...
10:55:26.443817 run    osmo-mgw_10.42.42.3(pid=6481): Terminated {rc=15}
10:55:26.455392 ---      ParallelTerminationStrategy: PID 6482 died...
10:55:26.519031 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6482): Terminated: ok {rc=0}
10:55:26.530608 ---      ParallelTerminationStrategy: PID 6483 died...
10:55:26.594652 run    osmo-mgw_10.42.42.4(pid=6483): Terminated {rc=15}
10:55:26.606219 ---      ParallelTerminationStrategy: PID 6484 died...
10:55:26.670799 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6484): Terminated: ok {rc=0}
10:55:26.682528 ---      ParallelTerminationStrategy: PID 6498 died...
10:55:26.746279 run           osmo-trx-lms(pid=6498): Terminated {rc=36608}
10:55:28.774857 ---      ParallelTerminationStrategy: PID 6479 died...
10:55:28.840280 run    osmo-msc_10.42.42.6(pid=6479): Terminated: ok {rc=0}
10:55:29.518936 ---      ParallelTerminationStrategy: PID 6485 died...
10:55:29.584171 run    osmo-bsc_10.42.42.7(pid=6485): Terminated: ok {rc=0}
10:55:32.774239 run           osmo-bts-trx(pid=6499): Terminating (SIGINT)
10:55:32.953476 ---      ParallelTerminationStrategy: PID 6499 died...
10:55:33.019053 run           osmo-bts-trx(pid=6499): Terminated: ok {rc=0}
10:55:33.189606 bus                        /sierra_3: Setting Online False
10:55:34.653750 bus                        /sierra_3: Setting Powered False
10:55:40.913857 bus                        /sierra_3: Setting Powered False
10:55:42.033567 bus                        /sierra_3: Setting Powered False
10:55:43.213584 bus                        /sierra_2: Setting Online False
10:55:44.339452 bus                        /sierra_2: Setting Powered False
10:55:49.865148 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-162↪handover_2G:trx-lms-limenet-micro+bts1-trx-umtrx↪handover.py:57↪handover.py↪/sierra_2]