Skip to content

Failed

handover_2G:trx-lms+bts1-trx-b200.handover.py (from handover_2G_trx-lms+bts1-trx-b200)

Failing for the past 22 builds (Since #140 )
Took 1 min 47 sec.

Stacktrace

/sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-161↪handover_2G:trx-lms+bts1-trx-b200↪handover.py:57↪handover.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-161 handover_2G:trx-lms+bts1-trx-b200 handover.py
----------------------------------------------
08:17:48.788172 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:17:48.802393 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x bts (candidates: 2)
08:17:48.964041 tst                      handover.py: using LAC 16209
08:17:49.098619 tst                      handover.py: using RAC 144
08:17:49.228714 tst                      handover.py: using CellId 16208
08:17:49.358799 tst                      handover.py: using BVCI 2887
08:17:49.384563 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x bts (candidates: 2)
08:17:49.543532 tst                      handover.py: using LAC 16210
08:17:49.673360 tst                      handover.py: using RAC 145
08:17:49.803008 tst                      handover.py: using CellId 16209
08:17:49.933213 tst                      handover.py: using BVCI 2888
08:17:49.958998 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:17:49.972925 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:17:49.986648 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:17:50.000417 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:17:50.014210 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x ip_address (candidates: 6)
08:17:50.027866 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x modem (candidates: 2)
08:17:50.161064 tst handover_2G:trx-lms+bts1-trx-b200: Using 1 x modem (candidates: 2)
08:17:50.388836 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
08:17:50.890568 run          create_hlr_db(pid=1166): Launched
08:17:51.030592 bus                        /sierra_2: Setting Powered False
08:17:52.112192 run          create_hlr_db(pid=1166): Terminated: ok {rc=0}
08:17:52.555744 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1168): Launched
08:17:52.905685 run    osmo-hlr_10.42.42.2(pid=1169): Launched
08:17:53.025899 run              osmo-stp_10.42.42.5: Starting osmo-stp
08:17:53.594676 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1170): Launched
08:17:53.947801 run    osmo-stp_10.42.42.5(pid=1171): Launched
08:17:54.068149 run              osmo-msc_10.42.42.6: Starting osmo-msc
08:17:54.659972 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1172): Launched
08:17:55.012405 run    osmo-msc_10.42.42.6(pid=1173): Launched
08:17:55.149556 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
08:17:55.758952 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1174): Launched
08:17:56.110412 run    osmo-mgw_10.42.42.3(pid=1175): Launched
08:17:56.230402 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
08:17:56.777846 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1176): Launched
08:17:57.125111 run    osmo-mgw_10.42.42.4(pid=1177): Launched
08:17:57.253069 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
08:17:58.037649 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1178): Launched
08:17:58.386097 run    osmo-bsc_10.42.42.7(pid=1179): Launched
08:18:02.094540 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
08:18:02.557667 run                     osmo-trx-lms: OSMOTRX CONF: {'osmo_trx': {'remote_user': 'jenkins', 'launch_trx': 'True', 'dev_args': 'LimeSDR-USB,serial=00090706024F0A22', 'type': 'lms', 'channels': [{'rx_path': 'LNAW'}], 'trx_ip': '10.42.42.117', 'egprs': 'disable', 'clock_reference': 'external', 'bts_ip': '10.42.42.53'}}
08:18:03.168718 run                uname-m(pid=1180): Launched
08:18:04.395981 run                uname-m(pid=1180): Terminated: ok {rc=0}
08:18:04.789660 run          rm-remote-dir(pid=1181): Launched
08:18:06.019019 run          rm-remote-dir(pid=1181): Terminated: ok {rc=0}
08:18:06.412619 run          mk-remote-dir(pid=1182): Launched
08:18:07.629606 run          mk-remote-dir(pid=1182): Terminated: ok {rc=0}
08:18:08.013405 run          mk-remote-dir(pid=1183): Launched
08:18:09.239937 run          mk-remote-dir(pid=1183): Terminated: ok {rc=0}
08:18:09.569269 run     scp-inst-to-remote(pid=1184): Launched
08:18:11.794814 run     scp-inst-to-remote(pid=1184): Terminated: ok {rc=0}
08:18:12.180877 run          mk-remote-dir(pid=1186): Launched
08:18:13.396211 run          mk-remote-dir(pid=1186): Terminated: ok {rc=0}
08:18:13.715184 run      scp-cfg-to-remote(pid=1187): Launched
08:18:14.929857 run      scp-cfg-to-remote(pid=1187): Terminated: ok {rc=0}
08:18:15.319178 run          mk-remote-dir(pid=1189): Launched
08:18:16.547236 run          mk-remote-dir(pid=1189): Terminated: ok {rc=0}
08:18:16.884134 run  scp-wrapper-to-remote(pid=1190): Launched
08:18:18.101182 run  scp-wrapper-to-remote(pid=1190): Terminated: ok {rc=0}
08:18:18.460421 run           osmo-trx-lms(pid=1192): Launched
08:18:18.582126 run                     osmo-bts-trx: Waiting for osmo-trx-lms to start up...
08:18:19.892778 run           osmo-bts-trx(pid=1193): Launched
08:18:22.436739 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-lms+bts1-trx-b200/handover.py/01_bts0_started: pass (20.2s)
08:18:22.617139 tst                      handover.py: using MSISDN 13088
08:18:22.834515 bus                        /sierra_2: Setting Powered True
08:18:26.385433 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='13088', subscriber_id=1}
08:18:26.573463 tst                      handover.py: using MSISDN 13089
08:18:26.842787 bus                        /sierra_3: Setting Powered True
08:18:42.832664 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='13089', subscriber_id=2}
08:18:43.133447 bus                        /sierra_2: Setting Online False
08:18:44.337964 bus                        /sierra_2: Setting Powered False
08:18:50.118796 bus                        /sierra_2: Setting Powered True
08:18:51.786335 bus                        /sierra_2: Setting Online True
08:18:54.251916 tst                        /sierra_2: Connect to ('901', '70')
08:18:54.590644 bus                        /sierra_3: Setting Online False
08:18:55.795240 bus                        /sierra_3: Setting Powered False
08:19:03.699704 bus                        /sierra_3: Setting Powered True
08:19:17.391216 bus                        /sierra_3: Setting Online True
08:19:23.413708 tst                        /sierra_3: Connect to ('901', '70')
08:19:23.619432 tst                        /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053'
08:19:23.767101 tst                        /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028'
08:19:24.004929 tst                   handover.py:54: waiting for modems to attach...
08:19:35.539786 tst                   handover.py:57: ----- Report fragment: handover_2G:trx-lms+bts1-trx-b200/handover.py/02.1_ms0_attach: FAIL (11.5s)
08:19:35.550015 tst                   handover.py:57: ERR: Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-161↪handover_2G:trx-lms+bts1-trx-b200↪handover.py:57↪handover.py↪/sierra_2]  [trial-161↪handover_2G:trx-lms+bts1-trx-b200↪handover.py:57]
08:19:35.559168 tst                   handover.py:57: Test FAILED (106.8 sec)
08:19:35.601739 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1168): Terminating (SIGTERM)
08:19:35.650782 run    osmo-hlr_10.42.42.2(pid=1169): Terminating (SIGTERM)
08:19:35.700874 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1170): Terminating (SIGTERM)
08:19:35.753803 run    osmo-stp_10.42.42.5(pid=1171): Terminating (SIGTERM)
08:19:35.816418 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1172): Terminating (SIGTERM)
08:19:35.874888 run    osmo-msc_10.42.42.6(pid=1173): Terminating (SIGTERM)
08:19:35.924621 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1174): Terminating (SIGTERM)
08:19:35.975510 run    osmo-mgw_10.42.42.3(pid=1175): Terminating (SIGTERM)
08:19:36.026562 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1176): Terminating (SIGTERM)
08:19:36.076909 run    osmo-mgw_10.42.42.4(pid=1177): Terminating (SIGTERM)
08:19:36.127195 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1178): Terminating (SIGTERM)
08:19:36.177167 run    osmo-bsc_10.42.42.7(pid=1179): Terminating (SIGTERM)
08:19:36.228568 run           osmo-trx-lms(pid=1192): Terminating (SIGTERM)
08:19:36.279019 run           osmo-bts-trx(pid=1193): Terminating (SIGTERM)
08:19:36.290348 ---      ParallelTerminationStrategy: PID 1168 died...
08:19:36.356574 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1168): Terminated: ok {rc=0}
08:19:36.368604 ---      ParallelTerminationStrategy: PID 1169 died...
08:19:36.434688 run    osmo-hlr_10.42.42.2(pid=1169): Terminated: ok {rc=0}
08:19:36.446503 ---      ParallelTerminationStrategy: PID 1170 died...
08:19:36.512368 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1170): Terminated: ok {rc=0}
08:19:36.524395 ---      ParallelTerminationStrategy: PID 1171 died...
08:19:36.589922 run    osmo-stp_10.42.42.5(pid=1171): Terminated: ok {rc=0}
08:19:36.601927 ---      ParallelTerminationStrategy: PID 1172 died...
08:19:36.668049 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1172): Terminated: ok {rc=0}
08:19:36.680071 ---      ParallelTerminationStrategy: PID 1174 died...
08:19:36.745920 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1174): Terminated: ok {rc=0}
08:19:36.757947 ---      ParallelTerminationStrategy: PID 1175 died...
08:19:36.823761 run    osmo-mgw_10.42.42.3(pid=1175): Terminated {rc=15}
08:19:36.835816 ---      ParallelTerminationStrategy: PID 1176 died...
08:19:36.901821 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1176): Terminated: ok {rc=0}
08:19:36.914114 ---      ParallelTerminationStrategy: PID 1177 died...
08:19:36.980299 run    osmo-mgw_10.42.42.4(pid=1177): Terminated {rc=15}
08:19:36.992712 ---      ParallelTerminationStrategy: PID 1178 died...
08:19:37.058808 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1178): Terminated: ok {rc=0}
08:19:37.071230 ---      ParallelTerminationStrategy: PID 1192 died...
08:19:37.137411 run           osmo-trx-lms(pid=1192): Terminated {rc=36608}
08:19:39.166888 ---      ParallelTerminationStrategy: PID 1173 died...
08:19:39.235429 run    osmo-msc_10.42.42.6(pid=1173): Terminated: ok {rc=0}
08:19:39.914116 ---      ParallelTerminationStrategy: PID 1179 died...
08:19:39.980498 run    osmo-bsc_10.42.42.7(pid=1179): Terminated: ok {rc=0}
08:19:43.171988 run           osmo-bts-trx(pid=1193): Terminating (SIGINT)
08:19:43.351650 ---      ParallelTerminationStrategy: PID 1193 died...
08:19:43.417690 run           osmo-bts-trx(pid=1193): Terminated: ok {rc=0}
08:19:43.589981 bus                        /sierra_3: Setting Online False
08:19:45.088014 bus                        /sierra_3: Setting Powered False
08:19:50.913250 bus                        /sierra_3: Setting Powered False
08:19:52.035159 bus                        /sierra_3: Setting Powered False
08:19:53.214702 bus                        /sierra_2: Setting Online False
08:19:54.354685 bus                        /sierra_2: Setting Powered False
08:19:59.877214 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 57, in <module>
    wait(ms0.is_registered, msc.mcc_mnc())
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 581, in scan_cb_register
    self.schedule_scan_register(mcc_mnc)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 524, in schedule_scan_register
    raise log.Error('Failed to find Network Operator', mcc_mnc=mcc_mnc, attempts=self.register_attempts)
osmo_gsm_tester.core.log.Error: /sierra_2: Failed to find Network Operator {attempts=4, mcc_mnc=('901', '70')} [trial-161↪handover_2G:trx-lms+bts1-trx-b200↪handover.py:57↪handover.py↪/sierra_2]