jenkins LTS upgrade
Failed
voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1.mo_mt_call_osmux.py (from voice_sysmo+mod-bts0-ts-tchh+cfg-codec-hr1)
Stacktrace
/sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7fd2c443c400>, timeout=10, timestep=1} [trial-822↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪/sierra_2]
Standard Output
---------------------------------------------- trial-822 voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1 mo_mt_call_osmux.py ---------------------------------------------- 07:41:31.122255 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 07:41:31.134578 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x bts (candidates: 1) 07:41:31.290248 tst mo_mt_call_osmux.py: using LAC 8759 07:41:31.419274 tst mo_mt_call_osmux.py: using RAC 89 07:41:31.548321 tst mo_mt_call_osmux.py: using CellId 8759 07:41:31.679041 tst mo_mt_call_osmux.py: using BVCI 8755 07:41:31.704254 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 07:41:31.715902 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 07:41:31.727412 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 07:41:31.738904 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 07:41:31.750857 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 07:41:31.762438 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x modem (candidates: 2) 07:41:31.866465 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x modem (candidates: 2) 07:41:32.065714 run osmo-hlr_10.42.42.2: Starting osmo-hlr 07:41:32.596120 run create_hlr_db(pid=11729): Launched 07:41:32.788566 bus /sierra_1: Setting Powered False 07:41:33.808165 run create_hlr_db(pid=11729): Terminated: ok {rc=0} 07:41:34.229495 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11731): Launched 07:41:34.548421 run osmo-hlr_10.42.42.2(pid=11732): Launched 07:41:34.661126 run osmo-stp_10.42.42.5: Starting osmo-stp 07:41:35.209257 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11733): Launched 07:41:35.525222 run osmo-stp_10.42.42.5(pid=11734): Launched 07:41:35.639996 run osmo-msc_10.42.42.6: Starting osmo-msc 07:41:36.213530 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=11735): Launched 07:41:36.530662 run osmo-msc_10.42.42.6(pid=11736): Launched 07:41:39.698354 run osmo-mgw_10.42.42.3: Starting osmo-mgw 07:41:40.242587 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=11737): Launched 07:41:40.558343 run osmo-mgw_10.42.42.3(pid=11738): Launched 07:41:40.669522 run osmo-mgw_10.42.42.4: Starting osmo-mgw 07:41:41.206537 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=11739): Launched 07:41:41.522340 run osmo-mgw_10.42.42.4(pid=11740): Launched 07:41:41.633915 run osmo-bsc_10.42.42.7: Starting osmo-bsc 07:41:42.291257 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=11741): Launched 07:41:42.606365 run osmo-bsc_10.42.42.7(pid=11742): Launched 07:41:48.872218 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7 07:41:49.439672 run rm-remote-dir(pid=11744): Launched 07:41:51.645221 run rm-remote-dir(pid=11744): Terminated: ok {rc=0} 07:41:52.002962 run mk-remote-dir(pid=11745): Launched 07:41:54.205513 run mk-remote-dir(pid=11745): Terminated: ok {rc=0} 07:41:54.506362 run scp-inst-to-remote(pid=11746): Launched 07:42:07.725772 run scp-inst-to-remote(pid=11746): Terminated: ok {rc=0} 07:42:08.078075 run mk-remote-dir(pid=11755): Launched 07:42:10.279681 run mk-remote-dir(pid=11755): Terminated: ok {rc=0} 07:42:10.580390 run scp-cfg-to-remote(pid=11756): Launched 07:42:12.783711 run scp-cfg-to-remote(pid=11756): Terminated: ok {rc=0} 07:42:13.131774 run reload-dsp-firmware(pid=11758): Launched 07:42:16.331314 run reload-dsp-firmware(pid=11758): Terminated: ok {rc=0} 07:42:16.668711 run osmo-bts-sysmo(pid=11759): Launched 07:42:21.910044 tst mo_mt_call_osmux.py: using MSISDN 5684 07:42:22.092048 bus /sierra_1: Setting Powered True 07:42:25.575403 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000009031', msisdn='5684', subscriber_id=1} 07:42:25.711787 tst mo_mt_call_osmux.py: using MSISDN 5685 07:42:25.882111 bus /sierra_2: Setting Powered True 07:42:38.235754 tst mo_mt_call_osmux.py:8: ERR: Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7fd2c443c400>, timeout=10, timestep=1} [trial-822↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪/sierra_2] [trial-822↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call_osmux.py:8] 07:42:38.243610 tst mo_mt_call_osmux.py:8: Test FAILED (67.2 sec) 07:42:38.279320 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11731): Terminating (SIGTERM) 07:42:38.320432 run osmo-hlr_10.42.42.2(pid=11732): Terminating (SIGTERM) 07:42:38.363137 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11733): Terminating (SIGTERM) 07:42:38.407387 run osmo-stp_10.42.42.5(pid=11734): Terminating (SIGTERM) 07:42:38.455069 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=11735): Terminating (SIGTERM) 07:42:38.504047 run osmo-msc_10.42.42.6(pid=11736): Terminating (SIGTERM) 07:42:38.547497 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=11737): Terminating (SIGTERM) 07:42:38.595951 run osmo-mgw_10.42.42.3(pid=11738): Terminating (SIGTERM) 07:42:38.642753 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=11739): Terminating (SIGTERM) 07:42:38.684621 run osmo-mgw_10.42.42.4(pid=11740): Terminating (SIGTERM) 07:42:38.745067 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=11741): Terminating (SIGTERM) 07:42:38.786467 run osmo-bsc_10.42.42.7(pid=11742): Terminating (SIGTERM) 07:42:38.828028 run osmo-bts-sysmo(pid=11759): Terminating (SIGTERM) 07:42:38.838274 --- ParallelTerminationStrategy: PID 11731 died... 07:42:38.895328 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11731): Terminated: ok {rc=0} 07:42:38.906174 --- ParallelTerminationStrategy: PID 11732 died... 07:42:38.962568 run osmo-hlr_10.42.42.2(pid=11732): Terminated: ok {rc=0} 07:42:38.973427 --- ParallelTerminationStrategy: PID 11733 died... 07:42:39.029898 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11733): Terminated: ok {rc=0} 07:42:39.040378 --- ParallelTerminationStrategy: PID 11734 died... 07:42:39.096609 run osmo-stp_10.42.42.5(pid=11734): Terminated: ok {rc=0} 07:42:39.106701 --- ParallelTerminationStrategy: PID 11735 died... 07:42:39.161990 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=11735): Terminated: ok {rc=0} 07:42:39.172321 --- ParallelTerminationStrategy: PID 11736 died... 07:42:39.227477 run osmo-msc_10.42.42.6(pid=11736): Terminated: ok {rc=0} 07:42:39.237608 --- ParallelTerminationStrategy: PID 11737 died... 07:42:39.292715 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=11737): Terminated: ok {rc=0} 07:42:39.302832 --- ParallelTerminationStrategy: PID 11738 died... 07:42:39.357958 run osmo-mgw_10.42.42.3(pid=11738): Terminated {rc=15} 07:42:39.368067 --- ParallelTerminationStrategy: PID 11739 died... 07:42:39.423444 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=11739): Terminated: ok {rc=0} 07:42:39.433733 --- ParallelTerminationStrategy: PID 11740 died... 07:42:39.488966 run osmo-mgw_10.42.42.4(pid=11740): Terminated {rc=15} 07:42:39.499126 --- ParallelTerminationStrategy: PID 11741 died... 07:42:39.554615 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=11741): Terminated: ok {rc=0} 07:42:39.564856 --- ParallelTerminationStrategy: PID 11742 died... 07:42:39.620555 run osmo-bsc_10.42.42.7(pid=11742): Terminated: ok {rc=0} 07:42:39.630720 --- ParallelTerminationStrategy: PID 11759 died... 07:42:39.686238 run osmo-bts-sysmo(pid=11759): Terminated: ok {rc=0} 07:42:39.791498 bus /sierra_2: Setting Online False 07:42:40.894852 bus /sierra_2: Setting Powered False 07:42:46.958742 bus /sierra_2: Setting Powered False 07:42:48.065900 bus /sierra_2: Setting Powered False 07:42:49.191011 bus /sierra_1: Setting Online False 07:42:50.277731 bus /sierra_1: Setting Powered False 07:42:55.961559 bus /sierra_1: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/mo_mt_call_osmux.py", line 8, in <module> test_mo_mt_call(True, True) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/lib/testlib.py", line 39, in test_mo_mt_call hlr.subscriber_add(ms_mt) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/hlr_osmo.py", line 121, in subscriber_add self.log('Add subscriber', msisdn=msisdn, imsi=modem.imsi(), subscriber_id=subscriber_id, File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 467, in imsi MainLoop.wait(lambda: simmgr.GetProperties().get('SubscriberIdentity', None) is not None, timeout=10) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7fd2c443c400>, timeout=10, timestep=1} [trial-822↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪/sierra_2]