jenkins LTS upgrade
Failed
voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3.mo_mt_call.py (from voice_sysmo+mod-bts0-ts-tchh+cfg-codec-hr3)
Stacktrace
/sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f37780d6a60>, timeout=10, timestep=1} [trial-841↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2]
Standard Output
---------------------------------------------- trial-841 voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3 mo_mt_call.py ---------------------------------------------- 06:44:23.992097 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 06:44:24.003206 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x bts (candidates: 1) 06:44:24.149722 tst mo_mt_call.py: using LAC 10280 06:44:24.274410 tst mo_mt_call.py: using RAC 80 06:44:24.403682 tst mo_mt_call.py: using CellId 10280 06:44:24.533542 tst mo_mt_call.py: using BVCI 10276 06:44:24.559327 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 06:44:24.571110 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 06:44:24.582667 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 06:44:24.594674 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 06:44:24.607657 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 06:44:24.619347 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x modem (candidates: 2) 06:44:24.723878 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x modem (candidates: 2) 06:44:24.915548 run osmo-hlr_10.42.42.2: Starting osmo-hlr 06:44:25.413104 run create_hlr_db(pid=29323): Launched 06:44:25.594515 bus /sierra_1: Setting Powered False 06:44:26.616730 run create_hlr_db(pid=29323): Terminated: ok {rc=0} 06:44:27.062183 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29325): Launched 06:44:27.400126 run osmo-hlr_10.42.42.2(pid=29326): Launched 06:44:27.521957 run osmo-stp_10.42.42.5: Starting osmo-stp 06:44:28.103012 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29327): Launched 06:44:28.439565 run osmo-stp_10.42.42.5(pid=29328): Launched 06:44:28.558914 run osmo-msc_10.42.42.6: Starting osmo-msc 06:44:29.170377 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29329): Launched 06:44:29.508327 run osmo-msc_10.42.42.6(pid=29330): Launched 06:44:32.687052 run osmo-mgw_10.42.42.3: Starting osmo-mgw 06:44:33.269754 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29331): Launched 06:44:33.605006 run osmo-mgw_10.42.42.3(pid=29332): Launched 06:44:33.725276 run osmo-mgw_10.42.42.4: Starting osmo-mgw 06:44:34.298207 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29333): Launched 06:44:34.635692 run osmo-mgw_10.42.42.4(pid=29334): Launched 06:44:34.756801 run osmo-bsc_10.42.42.7: Starting osmo-bsc 06:44:35.455581 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29335): Launched 06:44:35.793734 run osmo-bsc_10.42.42.7(pid=29336): Launched 06:44:42.078917 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7 06:44:42.679130 run rm-remote-dir(pid=29337): Launched 06:44:44.887597 run rm-remote-dir(pid=29337): Terminated: ok {rc=0} 06:44:45.249669 run mk-remote-dir(pid=29338): Launched 06:44:47.457644 run mk-remote-dir(pid=29338): Terminated: ok {rc=0} 06:44:47.761121 run scp-inst-to-remote(pid=29339): Launched 06:45:01.987104 run scp-inst-to-remote(pid=29339): Terminated: ok {rc=0} 06:45:02.344059 run mk-remote-dir(pid=29344): Launched 06:45:04.548250 run mk-remote-dir(pid=29344): Terminated: ok {rc=0} 06:45:04.850479 run scp-cfg-to-remote(pid=29345): Launched 06:45:08.058063 run scp-cfg-to-remote(pid=29345): Terminated: ok {rc=0} 06:45:08.407511 run reload-dsp-firmware(pid=29347): Launched 06:45:11.610044 run reload-dsp-firmware(pid=29347): Terminated: ok {rc=0} 06:45:11.952405 run osmo-bts-sysmo(pid=29348): Launched 06:45:17.209504 tst mo_mt_call.py: using MSISDN 6693 06:45:17.389707 bus /sierra_1: Setting Powered True 06:45:20.870526 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000009031', msisdn='6693', subscriber_id=1} 06:45:21.024880 tst mo_mt_call.py: using MSISDN 6694 06:45:21.198054 bus /sierra_2: Setting Powered True 06:45:33.564545 tst mo_mt_call.py:8: ERR: Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f37780d6a60>, timeout=10, timestep=1} [trial-841↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2] [trial-841↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3↪mo_mt_call.py:8] 06:45:33.571926 tst mo_mt_call.py:8: Test FAILED (69.6 sec) 06:45:33.606820 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29325): Terminating (SIGTERM) 06:45:33.646712 run osmo-hlr_10.42.42.2(pid=29326): Terminating (SIGTERM) 06:45:33.687943 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29327): Terminating (SIGTERM) 06:45:33.731763 run osmo-stp_10.42.42.5(pid=29328): Terminating (SIGTERM) 06:45:33.775932 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29329): Terminating (SIGTERM) 06:45:33.820448 run osmo-msc_10.42.42.6(pid=29330): Terminating (SIGTERM) 06:45:33.861835 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29331): Terminating (SIGTERM) 06:45:33.906548 run osmo-mgw_10.42.42.3(pid=29332): Terminating (SIGTERM) 06:45:33.948969 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29333): Terminating (SIGTERM) 06:45:33.991336 run osmo-mgw_10.42.42.4(pid=29334): Terminating (SIGTERM) 06:45:34.034318 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29335): Terminating (SIGTERM) 06:45:34.074157 run osmo-bsc_10.42.42.7(pid=29336): Terminating (SIGTERM) 06:45:34.114193 run osmo-bts-sysmo(pid=29348): Terminating (SIGTERM) 06:45:34.123504 --- ParallelTerminationStrategy: PID 29325 died... 06:45:34.176904 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29325): Terminated: ok {rc=0} 06:45:34.187033 --- ParallelTerminationStrategy: PID 29326 died... 06:45:34.240484 run osmo-hlr_10.42.42.2(pid=29326): Terminated: ok {rc=0} 06:45:34.250331 --- ParallelTerminationStrategy: PID 29327 died... 06:45:34.303787 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29327): Terminated: ok {rc=0} 06:45:34.313748 --- ParallelTerminationStrategy: PID 29328 died... 06:45:34.366767 run osmo-stp_10.42.42.5(pid=29328): Terminated: ok {rc=0} 06:45:34.376758 --- ParallelTerminationStrategy: PID 29329 died... 06:45:34.430145 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=29329): Terminated: ok {rc=0} 06:45:34.440221 --- ParallelTerminationStrategy: PID 29330 died... 06:45:34.493710 run osmo-msc_10.42.42.6(pid=29330): Terminated: ok {rc=0} 06:45:34.503646 --- ParallelTerminationStrategy: PID 29331 died... 06:45:34.557277 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=29331): Terminated: ok {rc=0} 06:45:34.567358 --- ParallelTerminationStrategy: PID 29332 died... 06:45:34.620891 run osmo-mgw_10.42.42.3(pid=29332): Terminated {rc=15} 06:45:34.630787 --- ParallelTerminationStrategy: PID 29333 died... 06:45:34.684598 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=29333): Terminated: ok {rc=0} 06:45:34.694519 --- ParallelTerminationStrategy: PID 29334 died... 06:45:34.747856 run osmo-mgw_10.42.42.4(pid=29334): Terminated {rc=15} 06:45:34.757754 --- ParallelTerminationStrategy: PID 29335 died... 06:45:34.811572 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=29335): Terminated: ok {rc=0} 06:45:34.821696 --- ParallelTerminationStrategy: PID 29348 died... 06:45:34.875263 run osmo-bts-sysmo(pid=29348): Terminated: ok {rc=0} 06:45:35.360780 --- ParallelTerminationStrategy: PID 29336 died... 06:45:35.415979 run osmo-bsc_10.42.42.7(pid=29336): Terminated: ok {rc=0} 06:45:35.525236 bus /sierra_2: Setting Online False 06:45:36.610813 bus /sierra_2: Setting Powered False 06:45:41.958274 bus /sierra_2: Setting Powered False 06:45:43.065200 bus /sierra_2: Setting Powered False 06:45:44.194607 bus /sierra_1: Setting Online False 06:45:45.282707 bus /sierra_1: Setting Powered False 06:45:50.960085 bus /sierra_1: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/mo_mt_call.py", line 8, in <module> test_mo_mt_call(False, False) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/lib/testlib.py", line 39, in test_mo_mt_call hlr.subscriber_add(ms_mt) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/hlr_osmo.py", line 121, in subscriber_add self.log('Add subscriber', msisdn=msisdn, imsi=modem.imsi(), subscriber_id=subscriber_id, File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 467, in imsi MainLoop.wait(lambda: simmgr.GetProperties().get('SubscriberIdentity', None) is not None, timeout=10) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f37780d6a60>, timeout=10, timestep=1} [trial-841↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2]