Skip to content

Failed

handover_2G:trx-lms+bts1-sysmo.handover.py (from handover_2G_trx-lms+bts1-sysmo)

Failing for the past 22 builds (Since #140 )
Took 1 min 53 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-161↪handover_2G:trx-lms+bts1-sysmo↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-161 handover_2G:trx-lms+bts1-sysmo handover.py
----------------------------------------------
08:15:28.984133 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:15:28.998805 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x bts (candidates: 2)
08:15:29.161721 tst                      handover.py: using LAC 16207
08:15:29.292952 tst                      handover.py: using RAC 142
08:15:29.421458 tst                      handover.py: using CellId 16206
08:15:29.550251 tst                      handover.py: using BVCI 2885
08:15:29.576081 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x bts (candidates: 2)
08:15:29.713093 tst                      handover.py: using LAC 16208
08:15:29.848544 tst                      handover.py: using RAC 143
08:15:29.984776 tst                      handover.py: using CellId 16207
08:15:30.118464 tst                      handover.py: using BVCI 2886
08:15:30.144445 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:15:30.158527 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:15:30.172841 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:15:30.186707 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:15:30.200696 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x ip_address (candidates: 6)
08:15:30.214474 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x modem (candidates: 2)
08:15:30.346200 tst   handover_2G:trx-lms+bts1-sysmo: Using 1 x modem (candidates: 2)
08:15:30.575211 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
08:15:31.077335 run          create_hlr_db(pid=1125): Launched
08:15:31.216264 bus                        /sierra_2: Setting Powered False
08:15:32.298884 run          create_hlr_db(pid=1125): Terminated: ok {rc=0}
08:15:32.743597 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1127): Launched
08:15:33.098809 run    osmo-hlr_10.42.42.2(pid=1128): Launched
08:15:33.227555 run              osmo-stp_10.42.42.5: Starting osmo-stp
08:15:33.814215 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1129): Launched
08:15:34.168725 run    osmo-stp_10.42.42.5(pid=1130): Launched
08:15:34.295444 run              osmo-msc_10.42.42.6: Starting osmo-msc
08:15:34.911972 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1132): Launched
08:15:35.267007 run    osmo-msc_10.42.42.6(pid=1133): Launched
08:15:35.400380 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
08:15:36.007006 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1134): Launched
08:15:36.363522 run    osmo-mgw_10.42.42.3(pid=1135): Launched
08:15:36.489774 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
08:15:37.065834 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1136): Launched
08:15:37.417490 run    osmo-mgw_10.42.42.4(pid=1137): Launched
08:15:37.537460 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
08:15:38.297064 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1138): Launched
08:15:38.648175 run    osmo-bsc_10.42.42.7(pid=1139): Launched
08:15:42.339750 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
08:15:42.804269 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'remote_user': 'jenkins', 'launch_trx': 'True', 'dev_args': 'LimeSDR-USB,serial=00090706024F0A22', 'type': 'lms', 'channels': [{'rx_path': 'LNAW'}], 'trx_ip': '10.42.42.117', 'egprs': 'disable', 'clock_reference': 'external', 'bts_ip': '10.42.42.53'}}
08:15:43.413149 run                uname-m(pid=1140): Launched
08:15:44.639012 run                uname-m(pid=1140): Terminated: ok {rc=0}
08:15:45.035684 run          rm-remote-dir(pid=1141): Launched
08:15:46.262689 run          rm-remote-dir(pid=1141): Terminated: ok {rc=0}
08:15:46.656771 run          mk-remote-dir(pid=1142): Launched
08:15:47.872942 run          mk-remote-dir(pid=1142): Terminated: ok {rc=0}
08:15:48.244858 run          mk-remote-dir(pid=1143): Launched
08:15:49.458737 run          mk-remote-dir(pid=1143): Terminated: ok {rc=0}
08:15:49.773965 run     scp-inst-to-remote(pid=1144): Launched
08:15:51.988933 run     scp-inst-to-remote(pid=1144): Terminated: ok {rc=0}
08:15:52.358844 run          mk-remote-dir(pid=1146): Launched
08:15:53.571816 run          mk-remote-dir(pid=1146): Terminated: ok {rc=0}
08:15:53.887900 run      scp-cfg-to-remote(pid=1147): Launched
08:15:55.101086 run      scp-cfg-to-remote(pid=1147): Terminated: ok {rc=0}
08:15:55.487444 run          mk-remote-dir(pid=1149): Launched
08:15:56.715551 run          mk-remote-dir(pid=1149): Terminated: ok {rc=0}
08:15:57.052106 run  scp-wrapper-to-remote(pid=1150): Launched
08:15:58.267384 run  scp-wrapper-to-remote(pid=1150): Terminated: ok {rc=0}
08:15:58.634764 run           osmo-trx-lms(pid=1152): Launched
08:15:58.766089 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
08:16:00.084222 run           osmo-bts-trx(pid=1153): Launched
08:16:02.608642 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms+bts1-sysmo/handover.py/01_bts0_started: pass (20.1s)
08:16:02.789118 tst                      handover.py: using MSISDN 13086
08:16:03.004990 bus                        /sierra_2: Setting Powered True
08:16:06.571633 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13086', subscriber_id=1}
08:16:06.746360 tst                      handover.py: using MSISDN 13087
08:16:06.962257 bus                        /sierra_3: Setting Powered True
08:16:21.709042 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13087', subscriber_id=2}
08:16:22.004226 bus                        /sierra_2: Setting Online False
08:16:23.241658 bus                        /sierra_2: Setting Powered False
08:16:29.038687 bus                        /sierra_2: Setting Powered True
08:16:30.461173 bus                        /sierra_2: Setting Online True
08:16:33.195240 tst                        /sierra_2: Connect to ('901', '70')
08:16:33.592013 bus                        /sierra_3: Setting Online False
08:16:34.829359 bus                        /sierra_3: Setting Powered False
08:16:42.989295 bus                        /sierra_3: Setting Powered True
08:16:56.564794 bus                        /sierra_3: Setting Online True
08:17:02.647597 tst                        /sierra_3: Connect to ('901', '70')
08:17:02.853525 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
08:17:03.000799 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
08:17:03.238129 tst                   handover.py:54: waiting for modems to attach...
08:17:21.417879 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms+bts1-sysmo/handover.py/02.1_ms0_attach: FAIL (18.1s)
08:17:21.428781 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-161↪handover_2G:trx-lms+bts1-sysmo↪handover.py:57↪handover.py↪/sierra_2]  [trial-161↪handover_2G:trx-lms+bts1-sysmo↪handover.py:57]
08:17:21.439701 tst                   handover.py:57: Test FAILED (112.4 sec)
08:17:21.489144 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1127): Terminating (SIGTERM)
08:17:21.544737 run    osmo-hlr_10.42.42.2(pid=1128): Terminating (SIGTERM)
08:17:21.602589 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1129): Terminating (SIGTERM)
08:17:21.662813 run    osmo-stp_10.42.42.5(pid=1130): Terminating (SIGTERM)
08:17:21.725545 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1132): Terminating (SIGTERM)
08:17:21.785293 run    osmo-msc_10.42.42.6(pid=1133): Terminating (SIGTERM)
08:17:21.841619 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1134): Terminating (SIGTERM)
08:17:21.897053 run    osmo-mgw_10.42.42.3(pid=1135): Terminating (SIGTERM)
08:17:21.952815 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1136): Terminating (SIGTERM)
08:17:22.008247 run    osmo-mgw_10.42.42.4(pid=1137): Terminating (SIGTERM)
08:17:22.064852 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1138): Terminating (SIGTERM)
08:17:22.123477 run    osmo-bsc_10.42.42.7(pid=1139): Terminating (SIGTERM)
08:17:22.180130 run           osmo-trx-lms(pid=1152): Terminating (SIGTERM)
08:17:22.235628 run           osmo-bts-trx(pid=1153): Terminating (SIGTERM)
08:17:22.247554 ---      ParallelTerminationStrategy: PID 1127 died...
08:17:22.317259 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1127): Terminated: ok {rc=0}
08:17:22.329667 ---      ParallelTerminationStrategy: PID 1128 died...
08:17:22.399388 run    osmo-hlr_10.42.42.2(pid=1128): Terminated: ok {rc=0}
08:17:22.411914 ---      ParallelTerminationStrategy: PID 1129 died...
08:17:22.481060 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1129): Terminated: ok {rc=0}
08:17:22.493589 ---      ParallelTerminationStrategy: PID 1130 died...
08:17:22.562486 run    osmo-stp_10.42.42.5(pid=1130): Terminated: ok {rc=0}
08:17:22.574971 ---      ParallelTerminationStrategy: PID 1132 died...
08:17:22.643886 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1132): Terminated: ok {rc=0}
08:17:22.656456 ---      ParallelTerminationStrategy: PID 1134 died...
08:17:22.725482 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1134): Terminated: ok {rc=0}
08:17:22.738179 ---      ParallelTerminationStrategy: PID 1135 died...
08:17:22.807099 run    osmo-mgw_10.42.42.3(pid=1135): Terminated {rc=15}
08:17:22.819602 ---      ParallelTerminationStrategy: PID 1136 died...
08:17:22.888956 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1136): Terminated: ok {rc=0}
08:17:22.901515 ---      ParallelTerminationStrategy: PID 1137 died...
08:17:22.970486 run    osmo-mgw_10.42.42.4(pid=1137): Terminated {rc=15}
08:17:22.983068 ---      ParallelTerminationStrategy: PID 1138 died...
08:17:23.052279 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1138): Terminated: ok {rc=0}
08:17:23.064967 ---      ParallelTerminationStrategy: PID 1152 died...
08:17:23.134695 run           osmo-trx-lms(pid=1152): Terminated {rc=36608}
08:17:25.164738 ---      ParallelTerminationStrategy: PID 1133 died...
08:17:25.234607 run    osmo-msc_10.42.42.6(pid=1133): Terminated: ok {rc=0}
08:17:25.913966 ---      ParallelTerminationStrategy: PID 1139 died...
08:17:25.980489 run    osmo-bsc_10.42.42.7(pid=1139): Terminated: ok {rc=0}
08:17:29.170569 run           osmo-bts-trx(pid=1153): Terminating (SIGINT)
08:17:29.355901 ---      ParallelTerminationStrategy: PID 1153 died...
08:17:29.422384 run           osmo-bts-trx(pid=1153): Terminated: ok {rc=0}
08:17:29.595033 bus                        /sierra_3: Setting Online False
08:17:31.082660 bus                        /sierra_3: Setting Powered False
08:17:36.933061 bus                        /sierra_3: Setting Powered False
08:17:38.055905 bus                        /sierra_3: Setting Powered False
08:17:39.236449 bus                        /sierra_2: Setting Online False
08:17:40.390350 bus                        /sierra_2: Setting Powered False
08:17:46.633088 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-161↪handover_2G:trx-lms+bts1-sysmo↪handover.py:57↪handover.py↪/sierra_2]