jenkins LTS upgrade
Failed
voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3.mo_mt_call_osmux.py (from voice_sysmo+mod-bts0-ts-tchf+cfg-codec-fr3)
Stacktrace
/sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f6e843538c8>, timeout=10, timestep=1} [trial-813↪voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪/sierra_2]
Standard Output
---------------------------------------------- trial-813 voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3 mo_mt_call_osmux.py ---------------------------------------------- 19:24:41.120671 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 19:24:41.132465 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x bts (candidates: 1) 19:24:41.281662 tst mo_mt_call_osmux.py: using LAC 8037 19:24:41.405689 tst mo_mt_call_osmux.py: using RAC 132 19:24:41.531544 tst mo_mt_call_osmux.py: using CellId 8037 19:24:41.656389 tst mo_mt_call_osmux.py: using BVCI 8033 19:24:41.680430 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 19:24:41.691535 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 19:24:41.702484 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 19:24:41.713669 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 19:24:41.724786 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x ip_address (candidates: 6) 19:24:41.735840 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x modem (candidates: 2) 19:24:41.836585 tst voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3: Using 1 x modem (candidates: 2) 19:24:42.028066 run osmo-hlr_10.42.42.2: Starting osmo-hlr 19:24:42.532993 run create_hlr_db(pid=16683): Launched 19:24:42.727733 bus /sierra_1: Setting Powered False 19:24:43.747771 run create_hlr_db(pid=16683): Terminated: ok {rc=0} 19:24:44.197271 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16685): Launched 19:24:44.532568 run osmo-hlr_10.42.42.2(pid=16686): Launched 19:24:44.654348 run osmo-stp_10.42.42.5: Starting osmo-stp 19:24:45.232945 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16687): Launched 19:24:45.567808 run osmo-stp_10.42.42.5(pid=16688): Launched 19:24:45.686898 run osmo-msc_10.42.42.6: Starting osmo-msc 19:24:46.298331 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=16689): Launched 19:24:46.634848 run osmo-msc_10.42.42.6(pid=16690): Launched 19:24:49.805843 run osmo-mgw_10.42.42.3: Starting osmo-mgw 19:24:50.353050 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=16691): Launched 19:24:50.674485 run osmo-mgw_10.42.42.3(pid=16692): Launched 19:24:50.786634 run osmo-mgw_10.42.42.4: Starting osmo-mgw 19:24:51.323714 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=16693): Launched 19:24:51.642116 run osmo-mgw_10.42.42.4(pid=16694): Launched 19:24:51.754918 run osmo-bsc_10.42.42.7: Starting osmo-bsc 19:24:52.413811 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=16695): Launched 19:24:52.730728 run osmo-bsc_10.42.42.7(pid=16696): Launched 19:24:59.000323 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7 19:24:59.567129 run rm-remote-dir(pid=16697): Launched 19:25:01.762738 run rm-remote-dir(pid=16697): Terminated: ok {rc=0} 19:25:02.101664 run mk-remote-dir(pid=16701): Launched 19:25:04.294536 run mk-remote-dir(pid=16701): Terminated: ok {rc=0} 19:25:04.580656 run scp-inst-to-remote(pid=16702): Launched 19:25:16.799930 run scp-inst-to-remote(pid=16702): Terminated: ok {rc=0} 19:25:17.152334 run mk-remote-dir(pid=16704): Launched 19:25:19.355463 run mk-remote-dir(pid=16704): Terminated: ok {rc=0} 19:25:19.658836 run scp-cfg-to-remote(pid=16705): Launched 19:25:21.863716 run scp-cfg-to-remote(pid=16705): Terminated: ok {rc=0} 19:25:22.213766 run reload-dsp-firmware(pid=16707): Launched 19:25:25.406531 run reload-dsp-firmware(pid=16707): Terminated: ok {rc=0} 19:25:25.725523 run osmo-bts-sysmo(pid=16708): Launched 19:25:29.965082 tst mo_mt_call_osmux.py: using MSISDN 5203 19:25:30.149408 bus /sierra_1: Setting Powered True 19:25:33.661231 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000009031', msisdn='5203', subscriber_id=1} 19:25:33.806492 tst mo_mt_call_osmux.py: using MSISDN 5204 19:25:33.981463 bus /sierra_2: Setting Powered True 19:25:46.325716 tst mo_mt_call_osmux.py:8: ERR: Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f6e843538c8>, timeout=10, timestep=1} [trial-813↪voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪/sierra_2] [trial-813↪voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3↪mo_mt_call_osmux.py:8] 19:25:46.333079 tst mo_mt_call_osmux.py:8: Test FAILED (65.2 sec) 19:25:46.367945 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16685): Terminating (SIGTERM) 19:25:46.407885 run osmo-hlr_10.42.42.2(pid=16686): Terminating (SIGTERM) 19:25:46.449690 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16687): Terminating (SIGTERM) 19:25:46.492404 run osmo-stp_10.42.42.5(pid=16688): Terminating (SIGTERM) 19:25:46.543700 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=16689): Terminating (SIGTERM) 19:25:46.590386 run osmo-msc_10.42.42.6(pid=16690): Terminating (SIGTERM) 19:25:46.633004 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=16691): Terminating (SIGTERM) 19:25:46.675234 run osmo-mgw_10.42.42.3(pid=16692): Terminating (SIGTERM) 19:25:46.717651 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=16693): Terminating (SIGTERM) 19:25:46.763480 run osmo-mgw_10.42.42.4(pid=16694): Terminating (SIGTERM) 19:25:46.814357 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=16695): Terminating (SIGTERM) 19:25:46.857812 run osmo-bsc_10.42.42.7(pid=16696): Terminating (SIGTERM) 19:25:46.903861 run osmo-bts-sysmo(pid=16708): Terminating (SIGTERM) 19:25:46.913646 --- ParallelTerminationStrategy: PID 16685 died... 19:25:46.970780 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16685): Terminated: ok {rc=0} 19:25:46.981742 --- ParallelTerminationStrategy: PID 16686 died... 19:25:47.038938 run osmo-hlr_10.42.42.2(pid=16686): Terminated: ok {rc=0} 19:25:47.049940 --- ParallelTerminationStrategy: PID 16687 died... 19:25:47.106822 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16687): Terminated: ok {rc=0} 19:25:47.117641 --- ParallelTerminationStrategy: PID 16688 died... 19:25:47.174509 run osmo-stp_10.42.42.5(pid=16688): Terminated: ok {rc=0} 19:25:47.185240 --- ParallelTerminationStrategy: PID 16689 died... 19:25:47.241027 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=16689): Terminated: ok {rc=0} 19:25:47.251333 --- ParallelTerminationStrategy: PID 16690 died... 19:25:47.306838 run osmo-msc_10.42.42.6(pid=16690): Terminated: ok {rc=0} 19:25:47.317071 --- ParallelTerminationStrategy: PID 16691 died... 19:25:47.372573 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=16691): Terminated: ok {rc=0} 19:25:47.382659 --- ParallelTerminationStrategy: PID 16692 died... 19:25:47.438099 run osmo-mgw_10.42.42.3(pid=16692): Terminated {rc=15} 19:25:47.448177 --- ParallelTerminationStrategy: PID 16693 died... 19:25:47.503664 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=16693): Terminated: ok {rc=0} 19:25:47.513839 --- ParallelTerminationStrategy: PID 16694 died... 19:25:47.569032 run osmo-mgw_10.42.42.4(pid=16694): Terminated {rc=15} 19:25:47.579136 --- ParallelTerminationStrategy: PID 16695 died... 19:25:47.634890 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=16695): Terminated: ok {rc=0} 19:25:47.644986 --- ParallelTerminationStrategy: PID 16696 died... 19:25:47.700362 run osmo-bsc_10.42.42.7(pid=16696): Terminated: ok {rc=0} 19:25:47.710455 --- ParallelTerminationStrategy: PID 16708 died... 19:25:47.765822 run osmo-bts-sysmo(pid=16708): Terminated: ok {rc=0} 19:25:47.872226 bus /sierra_2: Setting Online False 19:25:48.958767 bus /sierra_2: Setting Powered False 19:25:54.969670 bus /sierra_2: Setting Powered False 19:25:56.075896 bus /sierra_2: Setting Powered False 19:25:57.208234 bus /sierra_1: Setting Online False 19:25:58.294934 bus /sierra_1: Setting Powered False 19:26:03.962690 bus /sierra_1: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/mo_mt_call_osmux.py", line 8, in <module> test_mo_mt_call(True, True) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/lib/testlib.py", line 39, in test_mo_mt_call hlr.subscriber_add(ms_mt) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/hlr_osmo.py", line 121, in subscriber_add self.log('Add subscriber', msisdn=msisdn, imsi=modem.imsi(), subscriber_id=subscriber_id, File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 467, in imsi MainLoop.wait(lambda: simmgr.GetProperties().get('SubscriberIdentity', None) is not None, timeout=10) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f6e843538c8>, timeout=10, timestep=1} [trial-813↪voice:sysmo+mod-bts0-ts-tchf+cfg-codec-fr3↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪/sierra_2]