Skip to content

Failed

handover_2G:trx-lms+bts1-trx-lms-limenet-micro.handover.py (from handover_2G_trx-lms+bts1-trx-lms-limenet-micro)

Failing for the past 24 builds (Since #140 )
Took 2 min 0 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms+bts1-trx-lms-limenet-micro↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-163 handover_2G:trx-lms+bts1-trx-lms-limenet-micro handover.py
----------------------------------------------
13:03:28.326439 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x ip_address (candidates: 6)
13:03:28.341026 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x bts (candidates: 2)
13:03:28.503265 tst                      handover.py: using LAC 16347
13:03:28.645444 tst                      handover.py: using RAC 27
13:03:28.784200 tst                      handover.py: using CellId 16346
13:03:28.927304 tst                      handover.py: using BVCI 3025
13:03:28.955459 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x bts (candidates: 2)
13:03:29.118538 tst                      handover.py: using LAC 16348
13:03:29.258256 tst                      handover.py: using RAC 28
13:03:29.393653 tst                      handover.py: using CellId 16347
13:03:29.530857 tst                      handover.py: using BVCI 3026
13:03:29.557567 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x ip_address (candidates: 6)
13:03:29.571473 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x ip_address (candidates: 6)
13:03:29.585330 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x ip_address (candidates: 6)
13:03:29.599071 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x ip_address (candidates: 6)
13:03:29.612962 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x ip_address (candidates: 6)
13:03:29.626542 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x modem (candidates: 2)
13:03:29.762316 tst handover_2G:trx-lms+bts1-trx-lms-limenet-micro: Using 1 x modem (candidates: 2)
13:03:29.997803 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
13:03:30.532149 run          create_hlr_db(pid=8500): Launched
13:03:30.684732 bus                        /sierra_2: Setting Powered False
13:03:31.773879 run          create_hlr_db(pid=8500): Terminated: ok {rc=0}
13:03:32.212808 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8502): Launched
13:03:32.556634 run    osmo-hlr_10.42.42.2(pid=8503): Launched
13:03:32.679569 run              osmo-stp_10.42.42.5: Starting osmo-stp
13:03:33.250809 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8504): Launched
13:03:33.606043 run    osmo-stp_10.42.42.5(pid=8505): Launched
13:03:33.735705 run              osmo-msc_10.42.42.6: Starting osmo-msc
13:03:34.370292 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8506): Launched
13:03:34.734302 run    osmo-msc_10.42.42.6(pid=8507): Launched
13:03:34.873787 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
13:03:35.491578 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8508): Launched
13:03:35.852233 run    osmo-mgw_10.42.42.3(pid=8509): Launched
13:03:35.975837 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
13:03:36.547110 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8510): Launched
13:03:36.906363 run    osmo-mgw_10.42.42.4(pid=8511): Launched
13:03:37.029488 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
13:03:37.808494 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8512): Launched
13:03:38.169025 run    osmo-bsc_10.42.42.7(pid=8513): Launched
13:03:41.882577 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
13:03:42.359669 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'channels': [{'rx_path': 'LNAW'}], 'type': 'lms', 'bts_ip': '10.42.42.53', 'clock_reference': 'external', 'remote_user': 'jenkins', 'dev_args': 'LimeSDR-USB,serial=00090706024F0A22', 'trx_ip': '10.42.42.117', 'launch_trx': 'True', 'egprs': 'disable'}}
13:03:42.984736 run                uname-m(pid=8514): Launched
13:03:44.216296 run                uname-m(pid=8514): Terminated: ok {rc=0}
13:03:44.617558 run          rm-remote-dir(pid=8515): Launched
13:03:45.839988 run          rm-remote-dir(pid=8515): Terminated: ok {rc=0}
13:03:46.235728 run          mk-remote-dir(pid=8516): Launched
13:03:47.471046 run          mk-remote-dir(pid=8516): Terminated: ok {rc=0}
13:03:47.866095 run          mk-remote-dir(pid=8517): Launched
13:03:49.085262 run          mk-remote-dir(pid=8517): Terminated: ok {rc=0}
13:03:49.411576 run     scp-inst-to-remote(pid=8518): Launched
13:03:51.632269 run     scp-inst-to-remote(pid=8518): Terminated: ok {rc=0}
13:03:52.021301 run          mk-remote-dir(pid=8520): Launched
13:03:53.252207 run          mk-remote-dir(pid=8520): Terminated: ok {rc=0}
13:03:53.589105 run      scp-cfg-to-remote(pid=8521): Launched
13:03:54.807644 run      scp-cfg-to-remote(pid=8521): Terminated: ok {rc=0}
13:03:55.206264 run          mk-remote-dir(pid=8526): Launched
13:03:56.441465 run          mk-remote-dir(pid=8526): Terminated: ok {rc=0}
13:03:56.783924 run  scp-wrapper-to-remote(pid=8527): Launched
13:03:58.007913 run  scp-wrapper-to-remote(pid=8527): Terminated: ok {rc=0}
13:03:58.374635 run           osmo-trx-lms(pid=8529): Launched
13:03:58.499378 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
13:03:59.815220 run           osmo-bts-trx(pid=8530): Launched
13:04:02.419112 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms+bts1-trx-lms-limenet-micro/handover.py/01_bts0_started: pass (20.4s)
13:04:02.621352 tst                      handover.py: using MSISDN 13190
13:04:02.863306 bus                        /sierra_2: Setting Powered True
13:04:06.448601 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13190', subscriber_id=1}
13:04:06.656298 tst                      handover.py: using MSISDN 13191
13:04:06.933314 bus                        /sierra_3: Setting Powered True
13:04:22.257808 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13191', subscriber_id=2}
13:04:22.562472 bus                        /sierra_2: Setting Online False
13:04:23.773761 bus                        /sierra_2: Setting Powered False
13:04:29.853476 bus                        /sierra_2: Setting Powered True
13:04:31.247691 bus                        /sierra_2: Setting Online True
13:04:33.822269 tst                        /sierra_2: Connect to ('901', '70')
13:04:34.172113 bus                        /sierra_3: Setting Online False
13:04:35.382621 bus                        /sierra_3: Setting Powered False
13:04:43.572436 bus                        /sierra_3: Setting Powered True
13:04:56.655158 bus                        /sierra_3: Setting Online True
13:05:02.708024 tst                        /sierra_3: Connect to ('901', '70')
13:05:02.892166 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
13:05:03.023763 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
13:05:03.234580 tst                   handover.py:54: waiting for modems to attach...
13:05:28.194958 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms+bts1-trx-lms-limenet-micro/handover.py/02.1_ms0_attach: FAIL (24.9s)
13:05:28.206386 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms+bts1-trx-lms-limenet-micro↪handover.py:57↪handover.py↪/sierra_2]  [trial-163↪handover_2G:trx-lms+bts1-trx-lms-limenet-micro↪handover.py:57]
13:05:28.217629 tst                   handover.py:57: Test FAILED (119.9 sec)
13:05:28.268017 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8502): Terminating (SIGTERM)
13:05:28.325195 run    osmo-hlr_10.42.42.2(pid=8503): Terminating (SIGTERM)
13:05:28.384478 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8504): Terminating (SIGTERM)
13:05:28.444707 run    osmo-stp_10.42.42.5(pid=8505): Terminating (SIGTERM)
13:05:28.504122 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8506): Terminating (SIGTERM)
13:05:28.564089 run    osmo-msc_10.42.42.6(pid=8507): Terminating (SIGTERM)
13:05:28.621467 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8508): Terminating (SIGTERM)
13:05:28.678774 run    osmo-mgw_10.42.42.3(pid=8509): Terminating (SIGTERM)
13:05:28.735866 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8510): Terminating (SIGTERM)
13:05:28.791987 run    osmo-mgw_10.42.42.4(pid=8511): Terminating (SIGTERM)
13:05:28.848949 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8512): Terminating (SIGTERM)
13:05:28.905114 run    osmo-bsc_10.42.42.7(pid=8513): Terminating (SIGTERM)
13:05:28.961661 run           osmo-trx-lms(pid=8529): Terminating (SIGTERM)
13:05:29.016182 run           osmo-bts-trx(pid=8530): Terminating (SIGTERM)
13:05:29.028236 ---      ParallelTerminationStrategy: PID 8502 died...
13:05:29.098949 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8502): Terminated: ok {rc=0}
13:05:29.111615 ---      ParallelTerminationStrategy: PID 8503 died...
13:05:29.181416 run    osmo-hlr_10.42.42.2(pid=8503): Terminated: ok {rc=0}
13:05:29.193993 ---      ParallelTerminationStrategy: PID 8504 died...
13:05:29.264551 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8504): Terminated: ok {rc=0}
13:05:29.277158 ---      ParallelTerminationStrategy: PID 8505 died...
13:05:29.346976 run    osmo-stp_10.42.42.5(pid=8505): Terminated: ok {rc=0}
13:05:29.359498 ---      ParallelTerminationStrategy: PID 8506 died...
13:05:29.430034 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=8506): Terminated: ok {rc=0}
13:05:29.442706 ---      ParallelTerminationStrategy: PID 8508 died...
13:05:29.512856 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8508): Terminated: ok {rc=0}
13:05:29.525520 ---      ParallelTerminationStrategy: PID 8509 died...
13:05:29.595916 run    osmo-mgw_10.42.42.3(pid=8509): Terminated {rc=15}
13:05:29.608724 ---      ParallelTerminationStrategy: PID 8510 died...
13:05:29.679102 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8510): Terminated: ok {rc=0}
13:05:29.691791 ---      ParallelTerminationStrategy: PID 8511 died...
13:05:29.761937 run    osmo-mgw_10.42.42.4(pid=8511): Terminated {rc=15}
13:05:29.774591 ---      ParallelTerminationStrategy: PID 8512 died...
13:05:29.845284 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=8512): Terminated: ok {rc=0}
13:05:29.857916 ---      ParallelTerminationStrategy: PID 8529 died...
13:05:29.928306 run           osmo-trx-lms(pid=8529): Terminated {rc=36608}
13:05:31.957907 ---      ParallelTerminationStrategy: PID 8507 died...
13:05:32.030060 run    osmo-msc_10.42.42.6(pid=8507): Terminated: ok {rc=0}
13:05:32.043299 ---      ParallelTerminationStrategy: PID 8513 died...
13:05:32.109571 run    osmo-bsc_10.42.42.7(pid=8513): Terminated: ok {rc=0}
13:05:35.967864 run           osmo-bts-trx(pid=8530): Terminating (SIGINT)
13:05:36.150568 ---      ParallelTerminationStrategy: PID 8530 died...
13:05:36.222220 run           osmo-bts-trx(pid=8530): Terminated: ok {rc=0}
13:05:36.408659 bus                        /sierra_3: Setting Online False
13:05:37.906839 bus                        /sierra_3: Setting Powered False
13:05:43.951670 bus                        /sierra_3: Setting Powered False
13:05:45.074000 bus                        /sierra_3: Setting Powered False
13:05:46.256449 bus                        /sierra_2: Setting Online False
13:05:47.402302 bus                        /sierra_2: Setting Powered False
13:05:53.643713 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-163↪handover_2G:trx-lms+bts1-trx-lms-limenet-micro↪handover.py:57↪handover.py↪/sierra_2]