Failed

sms:sysmo.mo_mt_sms.py (from sms_sysmo)

Failing for the past 95 builds (Since #751 )

Took 1 min 24 sec.

Stacktrace

/sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f2a4cd1f378>, timeout=10, timestep=1} [trial-845↪sms:sysmo↪mo_mt_sms.py:27↪mo_mt_sms.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-845 sms:sysmo mo_mt_sms.py
----------------------------------------------
10:30:52.755239 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
10:30:52.766147 tst                        sms:sysmo: Using 1 x bts (candidates: 1)
10:30:52.893339 tst                     mo_mt_sms.py: using LAC 10589
10:30:53.013063 tst                     mo_mt_sms.py: using RAC 134
10:30:53.136000 tst                     mo_mt_sms.py: using CellId 10589
10:30:53.258417 tst                     mo_mt_sms.py: using BVCI 10585
10:30:53.281671 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
10:30:53.292120 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
10:30:53.302639 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
10:30:53.314070 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
10:30:53.324745 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
10:30:53.335571 tst                        sms:sysmo: Using 1 x modem (candidates: 2)
10:30:53.444462 tst                        sms:sysmo: Using 1 x modem (candidates: 2)
10:30:53.640248 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
10:30:55.314293 run          create_hlr_db(pid=5424): Launched
10:30:55.503444 bus                        /sierra_1: Setting Powered False
10:30:56.524533 run          create_hlr_db(pid=5424): Terminated: ok {rc=0}
10:30:56.938578 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5426): Launched
10:30:57.248509 run    osmo-hlr_10.42.42.2(pid=5428): Launched
10:30:57.362154 run              osmo-stp_10.42.42.5: Starting osmo-stp
10:30:59.843206 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5429): Launched
10:31:00.169092 run    osmo-stp_10.42.42.5(pid=5430): Launched
10:31:00.285949 run              osmo-msc_10.42.42.6: Starting osmo-msc
10:31:05.726049 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=5431): Launched
10:31:06.054140 run    osmo-msc_10.42.42.6(pid=5432): Launched
10:31:09.230318 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
10:31:11.001339 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=5433): Launched
10:31:11.335260 run    osmo-mgw_10.42.42.3(pid=5434): Launched
10:31:11.452545 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
10:31:12.012185 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=5435): Launched
10:31:12.341456 run    osmo-mgw_10.42.42.4(pid=5436): Launched
10:31:12.459203 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
10:31:17.005994 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=5437): Launched
10:31:17.338449 run    osmo-bsc_10.42.42.7(pid=5438): Launched
10:31:23.612601 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7
10:31:24.535079 run          rm-remote-dir(pid=5439): Launched
10:31:26.728512 run          rm-remote-dir(pid=5439): Terminated: ok {rc=0}
10:31:27.059495 run          mk-remote-dir(pid=5440): Launched
10:31:29.250982 run          mk-remote-dir(pid=5440): Terminated: ok {rc=0}
10:31:29.533003 run     scp-inst-to-remote(pid=5441): Launched
10:31:43.750478 run     scp-inst-to-remote(pid=5441): Terminated: ok {rc=0}
10:31:44.098065 run          mk-remote-dir(pid=5443): Launched
10:31:47.299582 run          mk-remote-dir(pid=5443): Terminated: ok {rc=0}
10:31:47.597817 run      scp-cfg-to-remote(pid=5444): Launched
10:31:49.799300 run      scp-cfg-to-remote(pid=5444): Terminated: ok {rc=0}
10:31:50.138307 run    reload-dsp-firmware(pid=5446): Launched
10:31:54.328629 run    reload-dsp-firmware(pid=5446): Terminated: ok {rc=0}
10:31:54.642693 run         osmo-bts-sysmo(pid=5447): Launched
10:31:59.881813 tst                     mo_mt_sms.py: using MSISDN 6886
10:32:00.052909 bus                        /sierra_1: Setting Powered True
10:32:03.527217 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000009031', msisdn='6886', subscriber_id=1}
10:32:03.667645 tst                     mo_mt_sms.py: using MSISDN 6887
10:32:03.846768 bus                        /sierra_2: Setting Powered True
10:32:16.202861 tst                  mo_mt_sms.py:27: ERR: Error: /sierra_2: Wait timeout {condition=&lt;function Modem.imsi.&lt;locals&gt;.&lt;lambda&gt; at 0x7f2a4cd1f378&gt;, timeout=10, timestep=1} [trial-845↪sms:sysmo↪mo_mt_sms.py:27↪mo_mt_sms.py↪/sierra_2]  [trial-845↪sms:sysmo↪mo_mt_sms.py:27]
10:32:16.210248 tst                  mo_mt_sms.py:27: Test FAILED (83.4 sec)
10:32:16.247256 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5426): Terminating (SIGTERM)
10:32:16.290468 run    osmo-hlr_10.42.42.2(pid=5428): Terminating (SIGTERM)
10:32:16.334901 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5429): Terminating (SIGTERM)
10:32:16.379803 run    osmo-stp_10.42.42.5(pid=5430): Terminating (SIGTERM)
10:32:16.458067 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=5431): Terminating (SIGTERM)
10:32:16.505986 run    osmo-msc_10.42.42.6(pid=5432): Terminating (SIGTERM)
10:32:16.549016 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=5433): Terminating (SIGTERM)
10:32:16.594377 run    osmo-mgw_10.42.42.3(pid=5434): Terminating (SIGTERM)
10:32:16.637942 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=5435): Terminating (SIGTERM)
10:32:16.683434 run    osmo-mgw_10.42.42.4(pid=5436): Terminating (SIGTERM)
10:32:16.727387 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=5437): Terminating (SIGTERM)
10:32:16.770164 run    osmo-bsc_10.42.42.7(pid=5438): Terminating (SIGTERM)
10:32:16.813228 run         osmo-bts-sysmo(pid=5447): Terminating (SIGTERM)
10:32:16.823032 ---      ParallelTerminationStrategy: PID 5426 died...
10:32:16.881015 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5426): Terminated: ok {rc=0}
10:32:16.891185 ---      ParallelTerminationStrategy: PID 5428 died...
10:32:16.947902 run    osmo-hlr_10.42.42.2(pid=5428): Terminated: ok {rc=0}
10:32:16.958019 ---      ParallelTerminationStrategy: PID 5429 died...
10:32:17.014856 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5429): Terminated: ok {rc=0}
10:32:17.024997 ---      ParallelTerminationStrategy: PID 5430 died...
10:32:17.081566 run    osmo-stp_10.42.42.5(pid=5430): Terminated: ok {rc=0}
10:32:17.091639 ---      ParallelTerminationStrategy: PID 5431 died...
10:32:17.148235 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=5431): Terminated: ok {rc=0}
10:32:17.158360 ---      ParallelTerminationStrategy: PID 5432 died...
10:32:17.215055 run    osmo-msc_10.42.42.6(pid=5432): Terminated: ok {rc=0}
10:32:17.225240 ---      ParallelTerminationStrategy: PID 5433 died...
10:32:17.283524 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=5433): Terminated: ok {rc=0}
10:32:17.294186 ---      ParallelTerminationStrategy: PID 5434 died...
10:32:17.352540 run    osmo-mgw_10.42.42.3(pid=5434): Terminated {rc=15}
10:32:17.363351 ---      ParallelTerminationStrategy: PID 5435 died...
10:32:17.421102 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=5435): Terminated: ok {rc=0}
10:32:17.431791 ---      ParallelTerminationStrategy: PID 5436 died...
10:32:17.489912 run    osmo-mgw_10.42.42.4(pid=5436): Terminated {rc=15}
10:32:17.500113 ---      ParallelTerminationStrategy: PID 5437 died...
10:32:17.557447 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=5437): Terminated: ok {rc=0}
10:32:17.567625 ---      ParallelTerminationStrategy: PID 5438 died...
10:32:17.624172 run    osmo-bsc_10.42.42.7(pid=5438): Terminated: ok {rc=0}
10:32:17.634322 ---      ParallelTerminationStrategy: PID 5447 died...
10:32:17.691663 run         osmo-bts-sysmo(pid=5447): Terminated: ok {rc=0}
10:32:17.798107 bus                        /sierra_2: Setting Online False
10:32:18.881821 bus                        /sierra_2: Setting Powered False
10:32:24.960026 bus                        /sierra_2: Setting Powered False
10:32:26.066869 bus                        /sierra_2: Setting Powered False
10:32:27.196373 bus                        /sierra_1: Setting Online False
10:32:28.284975 bus                        /sierra_1: Setting Powered False
10:32:33.973226 bus                        /sierra_1: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/sms/mo_mt_sms.py", line 27, in <module>
    hlr.subscriber_add(ms_mt)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/hlr_osmo.py", line 121, in subscriber_add
    self.log('Add subscriber', msisdn=msisdn, imsi=modem.imsi(), subscriber_id=subscriber_id,
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 467, in imsi
    MainLoop.wait(lambda: simmgr.GetProperties().get('SubscriberIdentity', None) is not None, timeout=10)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f2a4cd1f378>, timeout=10, timestep=1} [trial-845↪sms:sysmo↪mo_mt_sms.py:27↪mo_mt_sms.py↪/sierra_2]