jenkins LTS upgrade
Failed
voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any.mo_mt_call_osmux.py (from voice_sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any)
Stacktrace
/sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7fd2c44acea0>, timeout=10, timestep=1} [trial-822↪voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪/sierra_2]
Standard Output
---------------------------------------------- trial-822 voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any mo_mt_call_osmux.py ---------------------------------------------- 07:47:16.111906 tst voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any: Using 1 x ip_address (candidates: 6) 07:47:16.123653 tst voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any: Using 1 x bts (candidates: 1) 07:47:16.271428 tst mo_mt_call_osmux.py: using LAC 8763 07:47:16.397721 tst mo_mt_call_osmux.py: using RAC 93 07:47:16.518213 tst mo_mt_call_osmux.py: using CellId 8763 07:47:16.638180 tst mo_mt_call_osmux.py: using BVCI 8759 07:47:16.661770 tst voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any: Using 1 x ip_address (candidates: 6) 07:47:16.672885 tst voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any: Using 1 x ip_address (candidates: 6) 07:47:16.683956 tst voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any: Using 1 x ip_address (candidates: 6) 07:47:16.695271 tst voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any: Using 1 x ip_address (candidates: 6) 07:47:16.706555 tst voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any: Using 1 x ip_address (candidates: 6) 07:47:16.717788 tst voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any: Using 1 x modem (candidates: 2) 07:47:16.818382 tst voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any: Using 1 x modem (candidates: 2) 07:47:17.006002 run osmo-hlr_10.42.42.2: Starting osmo-hlr 07:47:17.494319 run create_hlr_db(pid=11863): Launched 07:47:17.676458 bus /sierra_1: Setting Powered False 07:47:18.696088 run create_hlr_db(pid=11863): Terminated: ok {rc=0} 07:47:19.141622 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11865): Launched 07:47:19.478996 run osmo-hlr_10.42.42.2(pid=11867): Launched 07:47:19.600127 run osmo-stp_10.42.42.5: Starting osmo-stp 07:47:20.178170 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11868): Launched 07:47:20.514622 run osmo-stp_10.42.42.5(pid=11869): Launched 07:47:20.633023 run osmo-msc_10.42.42.6: Starting osmo-msc 07:47:21.242992 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=11870): Launched 07:47:21.581041 run osmo-msc_10.42.42.6(pid=11871): Launched 07:47:24.758836 run osmo-mgw_10.42.42.3: Starting osmo-mgw 07:47:25.344794 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=11872): Launched 07:47:25.682896 run osmo-mgw_10.42.42.3(pid=11873): Launched 07:47:25.801875 run osmo-mgw_10.42.42.4: Starting osmo-mgw 07:47:26.371609 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=11874): Launched 07:47:26.707879 run osmo-mgw_10.42.42.4(pid=11875): Launched 07:47:26.827922 run osmo-bsc_10.42.42.7: Starting osmo-bsc 07:47:27.526791 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=11876): Launched 07:47:27.863958 run osmo-bsc_10.42.42.7(pid=11877): Launched 07:47:34.146984 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7 07:47:34.745796 run rm-remote-dir(pid=11879): Launched 07:47:36.951984 run rm-remote-dir(pid=11879): Terminated: ok {rc=0} 07:47:37.311812 run mk-remote-dir(pid=11880): Launched 07:47:39.517134 run mk-remote-dir(pid=11880): Terminated: ok {rc=0} 07:47:39.821047 run scp-inst-to-remote(pid=11881): Launched 07:47:53.043157 run scp-inst-to-remote(pid=11881): Terminated: ok {rc=0} 07:47:53.405101 run mk-remote-dir(pid=11883): Launched 07:47:55.608756 run mk-remote-dir(pid=11883): Terminated: ok {rc=0} 07:47:55.911933 run scp-cfg-to-remote(pid=11884): Launched 07:47:58.115228 run scp-cfg-to-remote(pid=11884): Terminated: ok {rc=0} 07:47:58.465361 run reload-dsp-firmware(pid=11886): Launched 07:48:01.666759 run reload-dsp-firmware(pid=11886): Terminated: ok {rc=0} 07:48:02.010111 run osmo-bts-sysmo(pid=11887): Launched 07:48:07.254203 tst mo_mt_call_osmux.py: using MSISDN 5692 07:48:07.437084 bus /sierra_1: Setting Powered True 07:48:10.929041 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000009031', msisdn='5692', subscriber_id=1} 07:48:11.072291 tst mo_mt_call_osmux.py: using MSISDN 5693 07:48:11.254052 bus /sierra_2: Setting Powered True 07:48:23.615235 tst mo_mt_call_osmux.py:8: ERR: Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7fd2c44acea0>, timeout=10, timestep=1} [trial-822↪voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪/sierra_2] [trial-822↪voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any↪mo_mt_call_osmux.py:8] 07:48:23.623009 tst mo_mt_call_osmux.py:8: Test FAILED (67.5 sec) 07:48:23.658596 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11865): Terminating (SIGTERM) 07:48:23.699800 run osmo-hlr_10.42.42.2(pid=11867): Terminating (SIGTERM) 07:48:23.743043 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11868): Terminating (SIGTERM) 07:48:23.785625 run osmo-stp_10.42.42.5(pid=11869): Terminating (SIGTERM) 07:48:23.849095 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=11870): Terminating (SIGTERM) 07:48:23.892455 run osmo-msc_10.42.42.6(pid=11871): Terminating (SIGTERM) 07:48:23.936682 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=11872): Terminating (SIGTERM) 07:48:23.979626 run osmo-mgw_10.42.42.3(pid=11873): Terminating (SIGTERM) 07:48:24.034499 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=11874): Terminating (SIGTERM) 07:48:24.077019 run osmo-mgw_10.42.42.4(pid=11875): Terminating (SIGTERM) 07:48:24.126121 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=11876): Terminating (SIGTERM) 07:48:24.168598 run osmo-bsc_10.42.42.7(pid=11877): Terminating (SIGTERM) 07:48:24.210454 run osmo-bts-sysmo(pid=11887): Terminating (SIGTERM) 07:48:24.222038 --- ParallelTerminationStrategy: PID 11865 died... 07:48:24.279167 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11865): Terminated: ok {rc=0} 07:48:24.289803 --- ParallelTerminationStrategy: PID 11867 died... 07:48:24.345998 run osmo-hlr_10.42.42.2(pid=11867): Terminated: ok {rc=0} 07:48:24.356534 --- ParallelTerminationStrategy: PID 11868 died... 07:48:24.412642 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11868): Terminated: ok {rc=0} 07:48:24.423237 --- ParallelTerminationStrategy: PID 11869 died... 07:48:24.479556 run osmo-stp_10.42.42.5(pid=11869): Terminated: ok {rc=0} 07:48:24.490124 --- ParallelTerminationStrategy: PID 11870 died... 07:48:24.545650 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=11870): Terminated: ok {rc=0} 07:48:24.555828 --- ParallelTerminationStrategy: PID 11871 died... 07:48:24.611242 run osmo-msc_10.42.42.6(pid=11871): Terminated: ok {rc=0} 07:48:24.621489 --- ParallelTerminationStrategy: PID 11872 died... 07:48:24.676994 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=11872): Terminated: ok {rc=0} 07:48:24.687295 --- ParallelTerminationStrategy: PID 11873 died... 07:48:24.742772 run osmo-mgw_10.42.42.3(pid=11873): Terminated {rc=15} 07:48:24.753027 --- ParallelTerminationStrategy: PID 11874 died... 07:48:24.808399 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=11874): Terminated: ok {rc=0} 07:48:24.818652 --- ParallelTerminationStrategy: PID 11875 died... 07:48:24.873775 run osmo-mgw_10.42.42.4(pid=11875): Terminated {rc=15} 07:48:24.883987 --- ParallelTerminationStrategy: PID 11876 died... 07:48:24.939587 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=11876): Terminated: ok {rc=0} 07:48:24.949969 --- ParallelTerminationStrategy: PID 11877 died... 07:48:25.005446 run osmo-bsc_10.42.42.7(pid=11877): Terminated: ok {rc=0} 07:48:25.015724 --- ParallelTerminationStrategy: PID 11887 died... 07:48:25.071446 run osmo-bts-sysmo(pid=11887): Terminated: ok {rc=0} 07:48:25.177537 bus /sierra_2: Setting Online False 07:48:26.263420 bus /sierra_2: Setting Powered False 07:48:31.972006 bus /sierra_2: Setting Powered False 07:48:33.074964 bus /sierra_2: Setting Powered False 07:48:34.202379 bus /sierra_1: Setting Online False 07:48:35.289047 bus /sierra_1: Setting Powered False 07:48:40.962822 bus /sierra_1: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/mo_mt_call_osmux.py", line 8, in <module> test_mo_mt_call(True, True) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/lib/testlib.py", line 39, in test_mo_mt_call hlr.subscriber_add(ms_mt) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/hlr_osmo.py", line 121, in subscriber_add self.log('Add subscriber', msisdn=msisdn, imsi=modem.imsi(), subscriber_id=subscriber_id, File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 467, in imsi MainLoop.wait(lambda: simmgr.GetProperties().get('SubscriberIdentity', None) is not None, timeout=10) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7fd2c44acea0>, timeout=10, timestep=1} [trial-822↪voice:sysmo+mod-bts0-dynts-ipa+cfg-codec-fr-any↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪/sierra_2]