Failed

sms:sysmo.mo_mt_sms.py (from sms_sysmo)

Failing for the past 83 builds (Since #751 )

Took 1 min 22 sec.

Stacktrace

/sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f5de46cb378>, timeout=10, timestep=1} [trial-833↪sms:sysmo↪mo_mt_sms.py:27↪mo_mt_sms.py↪/sierra_2]

Standard Output

----------------------------------------------
trial-833 sms:sysmo mo_mt_sms.py
----------------------------------------------
20:31:01.141552 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
20:31:01.152137 tst                        sms:sysmo: Using 1 x bts (candidates: 1)
20:31:01.274348 tst                     mo_mt_sms.py: using LAC 9629
20:31:01.393827 tst                     mo_mt_sms.py: using RAC 194
20:31:01.512566 tst                     mo_mt_sms.py: using CellId 9629
20:31:01.637228 tst                     mo_mt_sms.py: using BVCI 9625
20:31:01.660198 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
20:31:01.670792 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
20:31:01.681116 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
20:31:01.691929 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
20:31:01.702683 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
20:31:01.713607 tst                        sms:sysmo: Using 1 x modem (candidates: 2)
20:31:01.816599 tst                        sms:sysmo: Using 1 x modem (candidates: 2)
20:31:02.011297 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
20:31:03.672978 run          create_hlr_db(pid=9537): Launched
20:31:03.849712 bus                        /sierra_1: Setting Powered False
20:31:04.870518 run          create_hlr_db(pid=9537): Terminated: ok {rc=0}
20:31:05.303984 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9539): Launched
20:31:05.629928 run    osmo-hlr_10.42.42.2(pid=9540): Launched
20:31:05.752940 run              osmo-stp_10.42.42.5: Starting osmo-stp
20:31:08.268431 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9541): Launched
20:31:08.592128 run    osmo-stp_10.42.42.5(pid=9542): Launched
20:31:08.709068 run              osmo-msc_10.42.42.6: Starting osmo-msc
20:31:14.144575 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=9543): Launched
20:31:14.467245 run    osmo-msc_10.42.42.6(pid=9544): Launched
20:31:17.641091 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
20:31:19.397879 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9545): Launched
20:31:19.726673 run    osmo-mgw_10.42.42.3(pid=9546): Launched
20:31:19.842900 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
20:31:20.395692 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9547): Launched
20:31:20.719271 run    osmo-mgw_10.42.42.4(pid=9548): Launched
20:31:20.836144 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
20:31:25.355924 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=9549): Launched
20:31:25.682960 run    osmo-bsc_10.42.42.7(pid=9550): Launched
20:31:31.963789 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7
20:31:32.931671 run          rm-remote-dir(pid=9551): Launched
20:31:35.132304 run          rm-remote-dir(pid=9551): Terminated: ok {rc=0}
20:31:35.481699 run          mk-remote-dir(pid=9552): Launched
20:31:37.681764 run          mk-remote-dir(pid=9552): Terminated: ok {rc=0}
20:31:37.974650 run     scp-inst-to-remote(pid=9553): Launched
20:31:51.190664 run     scp-inst-to-remote(pid=9553): Terminated: ok {rc=0}
20:31:51.535692 run          mk-remote-dir(pid=9555): Launched
20:31:54.734137 run          mk-remote-dir(pid=9555): Terminated: ok {rc=0}
20:31:55.027892 run      scp-cfg-to-remote(pid=9556): Launched
20:31:57.226430 run      scp-cfg-to-remote(pid=9556): Terminated: ok {rc=0}
20:31:57.564448 run    reload-dsp-firmware(pid=9558): Launched
20:32:00.761019 run    reload-dsp-firmware(pid=9558): Terminated: ok {rc=0}
20:32:01.092062 run         osmo-bts-sysmo(pid=9559): Launched
20:32:06.330113 tst                     mo_mt_sms.py: using MSISDN 6250
20:32:06.499088 bus                        /sierra_1: Setting Powered True
20:32:09.982522 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000009031', msisdn='6250', subscriber_id=1}
20:32:10.116974 tst                     mo_mt_sms.py: using MSISDN 6251
20:32:10.283787 bus                        /sierra_2: Setting Powered True
20:32:22.638102 tst                  mo_mt_sms.py:27: ERR: Error: /sierra_2: Wait timeout {condition=&lt;function Modem.imsi.&lt;locals&gt;.&lt;lambda&gt; at 0x7f5de46cb378&gt;, timeout=10, timestep=1} [trial-833↪sms:sysmo↪mo_mt_sms.py:27↪mo_mt_sms.py↪/sierra_2]  [trial-833↪sms:sysmo↪mo_mt_sms.py:27]
20:32:22.645472 tst                  mo_mt_sms.py:27: Test FAILED (81.5 sec)
20:32:22.681217 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9539): Terminating (SIGTERM)
20:32:22.722722 run    osmo-hlr_10.42.42.2(pid=9540): Terminating (SIGTERM)
20:32:22.765730 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9541): Terminating (SIGTERM)
20:32:22.809090 run    osmo-stp_10.42.42.5(pid=9542): Terminating (SIGTERM)
20:32:22.874008 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=9543): Terminating (SIGTERM)
20:32:22.916146 run    osmo-msc_10.42.42.6(pid=9544): Terminating (SIGTERM)
20:32:22.959206 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9545): Terminating (SIGTERM)
20:32:23.002296 run    osmo-mgw_10.42.42.3(pid=9546): Terminating (SIGTERM)
20:32:23.056804 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9547): Terminating (SIGTERM)
20:32:23.099523 run    osmo-mgw_10.42.42.4(pid=9548): Terminating (SIGTERM)
20:32:23.155390 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=9549): Terminating (SIGTERM)
20:32:23.198284 run    osmo-bsc_10.42.42.7(pid=9550): Terminating (SIGTERM)
20:32:23.239807 run         osmo-bts-sysmo(pid=9559): Terminating (SIGTERM)
20:32:23.249515 ---      ParallelTerminationStrategy: PID 9539 died...
20:32:23.305088 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9539): Terminated: ok {rc=0}
20:32:23.314974 ---      ParallelTerminationStrategy: PID 9540 died...
20:32:23.370122 run    osmo-hlr_10.42.42.2(pid=9540): Terminated: ok {rc=0}
20:32:23.379945 ---      ParallelTerminationStrategy: PID 9541 died...
20:32:23.435082 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9541): Terminated: ok {rc=0}
20:32:23.444888 ---      ParallelTerminationStrategy: PID 9542 died...
20:32:23.500417 run    osmo-stp_10.42.42.5(pid=9542): Terminated: ok {rc=0}
20:32:23.510291 ---      ParallelTerminationStrategy: PID 9543 died...
20:32:23.565059 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=9543): Terminated: ok {rc=0}
20:32:23.574929 ---      ParallelTerminationStrategy: PID 9544 died...
20:32:23.630005 run    osmo-msc_10.42.42.6(pid=9544): Terminated: ok {rc=0}
20:32:23.639843 ---      ParallelTerminationStrategy: PID 9545 died...
20:32:23.695357 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9545): Terminated: ok {rc=0}
20:32:23.705393 ---      ParallelTerminationStrategy: PID 9546 died...
20:32:23.760791 run    osmo-mgw_10.42.42.3(pid=9546): Terminated {rc=15}
20:32:23.770752 ---      ParallelTerminationStrategy: PID 9547 died...
20:32:23.826760 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9547): Terminated: ok {rc=0}
20:32:23.837285 ---      ParallelTerminationStrategy: PID 9548 died...
20:32:23.893721 run    osmo-mgw_10.42.42.4(pid=9548): Terminated {rc=15}
20:32:23.904117 ---      ParallelTerminationStrategy: PID 9549 died...
20:32:23.961272 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=9549): Terminated: ok {rc=0}
20:32:23.971868 ---      ParallelTerminationStrategy: PID 9559 died...
20:32:24.028584 run         osmo-bts-sysmo(pid=9559): Terminated: ok {rc=0}
20:32:24.039029 ---      ParallelTerminationStrategy: PID 9550 died...
20:32:24.094288 run    osmo-bsc_10.42.42.7(pid=9550): Terminated: ok {rc=0}
20:32:24.199539 bus                        /sierra_2: Setting Online False
20:32:25.285743 bus                        /sierra_2: Setting Powered False
20:32:30.971081 bus                        /sierra_2: Setting Powered False
20:32:32.077618 bus                        /sierra_2: Setting Powered False
20:32:33.206214 bus                        /sierra_1: Setting Online False
20:32:34.293335 bus                        /sierra_1: Setting Powered False
20:32:39.963205 bus                        /sierra_1: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/sms/mo_mt_sms.py", line 27, in <module>
    hlr.subscriber_add(ms_mt)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/hlr_osmo.py", line 121, in subscriber_add
    self.log('Add subscriber', msisdn=msisdn, imsi=modem.imsi(), subscriber_id=subscriber_id,
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 467, in imsi
    MainLoop.wait(lambda: simmgr.GetProperties().get('SubscriberIdentity', None) is not None, timeout=10)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f5de46cb378>, timeout=10, timestep=1} [trial-833↪sms:sysmo↪mo_mt_sms.py:27↪mo_mt_sms.py↪/sierra_2]