jenkins LTS upgrade
Failed
voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3.mo_mt_call.py (from voice_sysmo+mod-bts0-ts-tchh+cfg-codec-hr3)
Stacktrace
/sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f6e8441ca60>, timeout=10, timestep=1} [trial-813↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2]
Standard Output
---------------------------------------------- trial-813 voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3 mo_mt_call.py ---------------------------------------------- 19:28:54.984438 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 19:28:54.995493 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x bts (candidates: 1) 19:28:55.146405 tst mo_mt_call.py: using LAC 8040 19:28:55.268268 tst mo_mt_call.py: using RAC 135 19:28:55.390056 tst mo_mt_call.py: using CellId 8040 19:28:55.515657 tst mo_mt_call.py: using BVCI 8036 19:28:55.539737 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 19:28:55.550778 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 19:28:55.561617 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 19:28:55.573028 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 19:28:55.584129 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x ip_address (candidates: 6) 19:28:55.595245 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x modem (candidates: 2) 19:28:55.697683 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3: Using 1 x modem (candidates: 2) 19:28:55.887548 run osmo-hlr_10.42.42.2: Starting osmo-hlr 19:28:56.383611 run create_hlr_db(pid=16782): Launched 19:28:56.565784 bus /sierra_1: Setting Powered False 19:28:57.587783 run create_hlr_db(pid=16782): Terminated: ok {rc=0} 19:28:58.010542 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16784): Launched 19:28:58.334145 run osmo-hlr_10.42.42.2(pid=16785): Launched 19:28:58.447255 run osmo-stp_10.42.42.5: Starting osmo-stp 19:28:58.995215 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16786): Launched 19:28:59.318699 run osmo-stp_10.42.42.5(pid=16787): Launched 19:28:59.431341 run osmo-msc_10.42.42.6: Starting osmo-msc 19:29:00.010400 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=16788): Launched 19:29:00.330956 run osmo-msc_10.42.42.6(pid=16789): Launched 19:29:03.503439 run osmo-mgw_10.42.42.3: Starting osmo-mgw 19:29:04.055720 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=16790): Launched 19:29:04.374826 run osmo-mgw_10.42.42.3(pid=16791): Launched 19:29:04.487560 run osmo-mgw_10.42.42.4: Starting osmo-mgw 19:29:05.029550 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=16792): Launched 19:29:05.348849 run osmo-mgw_10.42.42.4(pid=16793): Launched 19:29:05.462163 run osmo-bsc_10.42.42.7: Starting osmo-bsc 19:29:06.122323 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=16794): Launched 19:29:06.442998 run osmo-bsc_10.42.42.7(pid=16795): Launched 19:29:12.714482 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7 19:29:13.280694 run rm-remote-dir(pid=16796): Launched 19:29:15.476962 run rm-remote-dir(pid=16796): Terminated: ok {rc=0} 19:29:15.816918 run mk-remote-dir(pid=16797): Launched 19:29:18.012657 run mk-remote-dir(pid=16797): Terminated: ok {rc=0} 19:29:18.299577 run scp-inst-to-remote(pid=16798): Launched 19:29:30.519717 run scp-inst-to-remote(pid=16798): Terminated: ok {rc=0} 19:29:30.874939 run mk-remote-dir(pid=16800): Launched 19:29:33.078434 run mk-remote-dir(pid=16800): Terminated: ok {rc=0} 19:29:33.380314 run scp-cfg-to-remote(pid=16801): Launched 19:29:35.584531 run scp-cfg-to-remote(pid=16801): Terminated: ok {rc=0} 19:29:35.933582 run reload-dsp-firmware(pid=16803): Launched 19:29:39.135221 run reload-dsp-firmware(pid=16803): Terminated: ok {rc=0} 19:29:39.475574 run osmo-bts-sysmo(pid=16804): Launched 19:29:43.737932 tst mo_mt_call.py: using MSISDN 5209 19:29:43.922245 bus /sierra_1: Setting Powered True 19:29:47.418151 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000009031', msisdn='5209', subscriber_id=1} 19:29:47.562302 tst mo_mt_call.py: using MSISDN 5210 19:29:47.738602 bus /sierra_2: Setting Powered True 19:30:00.093650 tst mo_mt_call.py:8: ERR: Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f6e8441ca60>, timeout=10, timestep=1} [trial-813↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2] [trial-813↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3↪mo_mt_call.py:8] 19:30:00.100881 tst mo_mt_call.py:8: Test FAILED (65.2 sec) 19:30:00.135765 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16784): Terminating (SIGTERM) 19:30:00.175199 run osmo-hlr_10.42.42.2(pid=16785): Terminating (SIGTERM) 19:30:00.215982 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16786): Terminating (SIGTERM) 19:30:00.260808 run osmo-stp_10.42.42.5(pid=16787): Terminating (SIGTERM) 19:30:00.310900 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=16788): Terminating (SIGTERM) 19:30:00.354999 run osmo-msc_10.42.42.6(pid=16789): Terminating (SIGTERM) 19:30:00.398962 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=16790): Terminating (SIGTERM) 19:30:00.441708 run osmo-mgw_10.42.42.3(pid=16791): Terminating (SIGTERM) 19:30:00.499145 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=16792): Terminating (SIGTERM) 19:30:00.542618 run osmo-mgw_10.42.42.4(pid=16793): Terminating (SIGTERM) 19:30:00.582846 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=16794): Terminating (SIGTERM) 19:30:00.623174 run osmo-bsc_10.42.42.7(pid=16795): Terminating (SIGTERM) 19:30:00.663665 run osmo-bts-sysmo(pid=16804): Terminating (SIGTERM) 19:30:00.674817 --- ParallelTerminationStrategy: PID 16784 died... 19:30:00.729770 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16784): Terminated: ok {rc=0} 19:30:00.739777 --- ParallelTerminationStrategy: PID 16785 died... 19:30:00.794415 run osmo-hlr_10.42.42.2(pid=16785): Terminated: ok {rc=0} 19:30:00.804594 --- ParallelTerminationStrategy: PID 16786 died... 19:30:00.858958 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16786): Terminated: ok {rc=0} 19:30:00.868843 --- ParallelTerminationStrategy: PID 16787 died... 19:30:00.921978 run osmo-stp_10.42.42.5(pid=16787): Terminated: ok {rc=0} 19:30:00.931843 --- ParallelTerminationStrategy: PID 16788 died... 19:30:00.985439 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=16788): Terminated: ok {rc=0} 19:30:00.995193 --- ParallelTerminationStrategy: PID 16789 died... 19:30:01.048345 run osmo-msc_10.42.42.6(pid=16789): Terminated: ok {rc=0} 19:30:01.058070 --- ParallelTerminationStrategy: PID 16790 died... 19:30:01.111256 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=16790): Terminated: ok {rc=0} 19:30:01.120930 --- ParallelTerminationStrategy: PID 16791 died... 19:30:01.173890 run osmo-mgw_10.42.42.3(pid=16791): Terminated {rc=15} 19:30:01.183760 --- ParallelTerminationStrategy: PID 16792 died... 19:30:01.236776 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=16792): Terminated: ok {rc=0} 19:30:01.246819 --- ParallelTerminationStrategy: PID 16793 died... 19:30:01.300023 run osmo-mgw_10.42.42.4(pid=16793): Terminated {rc=15} 19:30:01.309781 --- ParallelTerminationStrategy: PID 16794 died... 19:30:01.363026 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=16794): Terminated: ok {rc=0} 19:30:01.372750 --- ParallelTerminationStrategy: PID 16795 died... 19:30:01.426541 run osmo-bsc_10.42.42.7(pid=16795): Terminated: ok {rc=0} 19:30:01.436286 --- ParallelTerminationStrategy: PID 16804 died... 19:30:01.489629 run osmo-bts-sysmo(pid=16804): Terminated: ok {rc=0} 19:30:01.597879 bus /sierra_2: Setting Online False 19:30:02.683600 bus /sierra_2: Setting Powered False 19:30:08.038469 bus /sierra_2: Setting Powered False 19:30:09.145707 bus /sierra_2: Setting Powered False 19:30:10.273357 bus /sierra_1: Setting Online False 19:30:11.360711 bus /sierra_1: Setting Powered False 19:30:16.956036 bus /sierra_1: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/mo_mt_call.py", line 8, in <module> test_mo_mt_call(False, False) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/lib/testlib.py", line 39, in test_mo_mt_call hlr.subscriber_add(ms_mt) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/hlr_osmo.py", line 121, in subscriber_add self.log('Add subscriber', msisdn=msisdn, imsi=modem.imsi(), subscriber_id=subscriber_id, File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 467, in imsi MainLoop.wait(lambda: simmgr.GetProperties().get('SubscriberIdentity', None) is not None, timeout=10) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7f6e8441ca60>, timeout=10, timestep=1} [trial-813↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr3↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2]