jenkins LTS upgrade
Failed
voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3.mo_mt_call.py (from voice_sysmo+mod-bts0-ts-tchf+cfg-codec-fr3)
Stacktrace
/sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f8282e02268>, timeout=10, timestep=1} [trial-814↪voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2]
Standard Output
---------------------------------------------- trial-814 voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3 mo_mt_call.py ---------------------------------------------- 21:54:25.990094 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 21:54:26.001730 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x bts (candidates: 1) 21:54:26.154098 tst mo_mt_call.py: using LAC 8116 21:54:26.279187 tst mo_mt_call.py: using RAC 211 21:54:26.406958 tst mo_mt_call.py: using CellId 8116 21:54:26.527763 tst mo_mt_call.py: using BVCI 8112 21:54:26.551370 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 21:54:26.562673 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 21:54:26.573852 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 21:54:26.585725 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 21:54:26.597890 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 21:54:26.610311 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x modem (candidates: 2) 21:54:26.716342 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x modem (candidates: 2) 21:54:26.913593 run osmo-hlr_10.42.42.2: Starting osmo-hlr 21:54:27.427887 run create_hlr_db(pid=19838): Launched 21:54:27.605314 bus /sierra_1: Setting Powered False 21:54:28.627447 run create_hlr_db(pid=19838): Terminated: ok {rc=0} 21:54:29.040536 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19840): Launched 21:54:29.349971 run osmo-hlr_10.42.42.2(pid=19841): Launched 21:54:29.460419 run osmo-stp_10.42.42.5: Starting osmo-stp 21:54:29.999048 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19842): Launched 21:54:30.308689 run osmo-stp_10.42.42.5(pid=19843): Launched 21:54:30.418240 run osmo-msc_10.42.42.6: Starting osmo-msc 21:54:30.980652 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=19844): Launched 21:54:31.292073 run osmo-msc_10.42.42.6(pid=19845): Launched 21:54:34.456557 run osmo-mgw_10.42.42.3: Starting osmo-mgw 21:54:34.997742 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=19846): Launched 21:54:35.307237 run osmo-mgw_10.42.42.3(pid=19847): Launched 21:54:35.416706 run osmo-mgw_10.42.42.4: Starting osmo-mgw 21:54:35.946049 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=19848): Launched 21:54:36.255454 run osmo-mgw_10.42.42.4(pid=19849): Launched 21:54:36.364604 run osmo-bsc_10.42.42.7: Starting osmo-bsc 21:54:37.008907 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=19850): Launched 21:54:37.319255 run osmo-bsc_10.42.42.7(pid=19851): Launched 21:54:43.595052 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7 21:54:44.185973 run rm-remote-dir(pid=19852): Launched 21:54:46.389032 run rm-remote-dir(pid=19852): Terminated: ok {rc=0} 21:54:46.742338 run mk-remote-dir(pid=19853): Launched 21:54:48.944284 run mk-remote-dir(pid=19853): Terminated: ok {rc=0} 21:54:49.242017 run scp-inst-to-remote(pid=19854): Launched 21:55:01.459556 run scp-inst-to-remote(pid=19854): Terminated: ok {rc=0} 21:55:01.830706 run mk-remote-dir(pid=19860): Launched 21:55:04.032323 run mk-remote-dir(pid=19860): Terminated: ok {rc=0} 21:55:04.335198 run scp-cfg-to-remote(pid=20228): Launched 21:55:06.541905 run scp-cfg-to-remote(pid=20228): Terminated: ok {rc=0} 21:55:06.913920 run reload-dsp-firmware(pid=20395): Launched 21:55:10.102611 run reload-dsp-firmware(pid=20395): Terminated: ok {rc=0} 21:55:10.416608 run osmo-bts-sysmo(pid=20397): Launched 21:55:14.659836 tst mo_mt_call.py: using MSISDN 5254 21:55:14.839626 bus /sierra_1: Setting Powered True 21:55:18.338158 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000009031', msisdn='5254', subscriber_id=1} 21:55:18.472687 tst mo_mt_call.py: using MSISDN 5255 21:55:18.641123 bus /sierra_2: Setting Powered True 21:55:30.984551 tst mo_mt_call.py:8: ERR: Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f8282e02268>, timeout=10, timestep=1} [trial-814↪voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2] [trial-814↪voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3↪mo_mt_call.py:8] 21:55:30.991735 tst mo_mt_call.py:8: Test FAILED (65.0 sec) 21:55:31.025906 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19840): Terminating (SIGTERM) 21:55:31.064909 run osmo-hlr_10.42.42.2(pid=19841): Terminating (SIGTERM) 21:55:31.105090 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19842): Terminating (SIGTERM) 21:55:31.146990 run osmo-stp_10.42.42.5(pid=19843): Terminating (SIGTERM) 21:55:31.190295 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=19844): Terminating (SIGTERM) 21:55:31.231223 run osmo-msc_10.42.42.6(pid=19845): Terminating (SIGTERM) 21:55:31.273499 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=19846): Terminating (SIGTERM) 21:55:31.314599 run osmo-mgw_10.42.42.3(pid=19847): Terminating (SIGTERM) 21:55:31.354527 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=19848): Terminating (SIGTERM) 21:55:31.394601 run osmo-mgw_10.42.42.4(pid=19849): Terminating (SIGTERM) 21:55:31.448292 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=19850): Terminating (SIGTERM) 21:55:31.487477 run osmo-bsc_10.42.42.7(pid=19851): Terminating (SIGTERM) 21:55:31.526665 run osmo-bts-sysmo(pid=20397): Terminating (SIGTERM) 21:55:31.536135 --- ParallelTerminationStrategy: PID 19840 died... 21:55:31.590469 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19840): Terminated: ok {rc=0} 21:55:31.600442 --- ParallelTerminationStrategy: PID 19841 died... 21:55:31.654572 run osmo-hlr_10.42.42.2(pid=19841): Terminated: ok {rc=0} 21:55:31.664616 --- ParallelTerminationStrategy: PID 19842 died... 21:55:31.717542 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19842): Terminated: ok {rc=0} 21:55:31.727201 --- ParallelTerminationStrategy: PID 19843 died... 21:55:31.779651 run osmo-stp_10.42.42.5(pid=19843): Terminated: ok {rc=0} 21:55:31.789277 --- ParallelTerminationStrategy: PID 19844 died... 21:55:31.841482 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=19844): Terminated: ok {rc=0} 21:55:31.851181 --- ParallelTerminationStrategy: PID 19845 died... 21:55:31.903367 run osmo-msc_10.42.42.6(pid=19845): Terminated: ok {rc=0} 21:55:31.913412 --- ParallelTerminationStrategy: PID 19846 died... 21:55:31.966032 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=19846): Terminated: ok {rc=0} 21:55:31.975771 --- ParallelTerminationStrategy: PID 19847 died... 21:55:32.028091 run osmo-mgw_10.42.42.3(pid=19847): Terminated {rc=15} 21:55:32.037894 --- ParallelTerminationStrategy: PID 19848 died... 21:55:32.090669 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=19848): Terminated: ok {rc=0} 21:55:32.100304 --- ParallelTerminationStrategy: PID 19849 died... 21:55:32.152545 run osmo-mgw_10.42.42.4(pid=19849): Terminated {rc=15} 21:55:32.162289 --- ParallelTerminationStrategy: PID 19850 died... 21:55:32.214731 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=19850): Terminated: ok {rc=0} 21:55:32.224299 --- ParallelTerminationStrategy: PID 19851 died... 21:55:32.276811 run osmo-bsc_10.42.42.7(pid=19851): Terminated: ok {rc=0} 21:55:32.286595 --- ParallelTerminationStrategy: PID 20397 died... 21:55:32.339258 run osmo-bts-sysmo(pid=20397): Terminated: ok {rc=0} 21:55:32.440816 bus /sierra_2: Setting Online False 21:55:33.526226 bus /sierra_2: Setting Powered False 21:55:38.963432 bus /sierra_2: Setting Powered False 21:55:40.065501 bus /sierra_2: Setting Powered False 21:55:41.196851 bus /sierra_1: Setting Online False 21:55:42.281495 bus /sierra_1: Setting Powered False 21:55:47.965654 bus /sierra_1: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/mo_mt_call.py", line 8, in <module> test_mo_mt_call(False, False) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/lib/testlib.py", line 39, in test_mo_mt_call hlr.subscriber_add(ms_mt) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/hlr_osmo.py", line 121, in subscriber_add self.log('Add subscriber', msisdn=msisdn, imsi=modem.imsi(), subscriber_id=subscriber_id, File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 467, in imsi MainLoop.wait(lambda: simmgr.GetProperties().get('SubscriberIdentity', None) is not None, timeout=10) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f8282e02268>, timeout=10, timestep=1} [trial-814↪voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2]