jenkins LTS upgrade
Failed
voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1.mo_mt_call.py (from voice_sysmo+mod-bts0-ts-tchh+cfg-codec-hr1)
Stacktrace
/sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7ff1005fae18>, timeout=10, timestep=1} [trial-808↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2]
Standard Output
---------------------------------------------- trial-808 voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1 mo_mt_call.py ---------------------------------------------- 13:28:46.980654 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 13:28:46.992017 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x bts (candidates: 1) 13:28:47.144829 tst mo_mt_call.py: using LAC 7638 13:28:47.274488 tst mo_mt_call.py: using RAC 243 13:28:47.404216 tst mo_mt_call.py: using CellId 7638 13:28:47.531297 tst mo_mt_call.py: using BVCI 7634 13:28:47.556115 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 13:28:47.567540 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 13:28:47.578742 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 13:28:47.589977 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 13:28:47.601102 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 13:28:47.612342 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x modem (candidates: 2) 13:28:47.713585 tst voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x modem (candidates: 2) 13:28:47.900735 run osmo-hlr_10.42.42.2: Starting osmo-hlr 13:28:48.384901 run create_hlr_db(pid=1556): Launched 13:28:48.562501 bus /sierra_1: Setting Powered False 13:28:49.585888 run create_hlr_db(pid=1556): Terminated: ok {rc=0} 13:28:50.023280 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1558): Launched 13:28:50.355347 run osmo-hlr_10.42.42.2(pid=1559): Launched 13:28:50.474671 run osmo-stp_10.42.42.5: Starting osmo-stp 13:28:51.044267 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1560): Launched 13:28:51.379491 run osmo-stp_10.42.42.5(pid=1561): Launched 13:28:51.498444 run osmo-msc_10.42.42.6: Starting osmo-msc 13:28:52.104124 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1562): Launched 13:28:52.439152 run osmo-msc_10.42.42.6(pid=1563): Launched 13:28:55.617487 run osmo-mgw_10.42.42.3: Starting osmo-mgw 13:28:56.195102 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1564): Launched 13:28:56.529921 run osmo-mgw_10.42.42.3(pid=1565): Launched 13:28:56.648594 run osmo-mgw_10.42.42.4: Starting osmo-mgw 13:28:57.211811 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1566): Launched 13:28:57.545938 run osmo-mgw_10.42.42.4(pid=1567): Launched 13:28:57.665254 run osmo-bsc_10.42.42.7: Starting osmo-bsc 13:28:58.356815 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1568): Launched 13:28:58.691843 run osmo-bsc_10.42.42.7(pid=1569): Launched 13:29:04.964627 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7 13:29:05.538455 run rm-remote-dir(pid=1570): Launched 13:29:07.733038 run rm-remote-dir(pid=1570): Terminated: ok {rc=0} 13:29:08.069079 run mk-remote-dir(pid=1571): Launched 13:29:09.260018 run mk-remote-dir(pid=1571): Terminated: ok {rc=0} 13:29:09.543071 run scp-inst-to-remote(pid=1572): Launched 13:29:22.763835 run scp-inst-to-remote(pid=1572): Terminated: ok {rc=0} 13:29:23.114611 run mk-remote-dir(pid=1574): Launched 13:29:25.315764 run mk-remote-dir(pid=1574): Terminated: ok {rc=0} 13:29:25.616293 run scp-cfg-to-remote(pid=1575): Launched 13:29:27.818148 run scp-cfg-to-remote(pid=1575): Terminated: ok {rc=0} 13:29:28.163903 run reload-dsp-firmware(pid=1577): Launched 13:29:30.361791 run reload-dsp-firmware(pid=1577): Terminated: ok {rc=0} 13:29:30.699055 run osmo-bts-sysmo(pid=1578): Launched 13:29:35.950908 tst mo_mt_call.py: using MSISDN 4940 13:29:36.134299 bus /sierra_1: Setting Powered True 13:29:39.618461 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000009031', msisdn='4940', subscriber_id=1} 13:29:39.761797 tst mo_mt_call.py: using MSISDN 4941 13:29:39.935642 bus /sierra_2: Setting Powered True 13:29:52.298169 tst mo_mt_call.py:8: ERR: Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7ff1005fae18>, timeout=10, timestep=1} [trial-808↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2] [trial-808↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call.py:8] 13:29:52.306041 tst mo_mt_call.py:8: Test FAILED (65.4 sec) 13:29:52.341813 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1558): Terminating (SIGTERM) 13:29:52.382951 run osmo-hlr_10.42.42.2(pid=1559): Terminating (SIGTERM) 13:29:52.427169 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1560): Terminating (SIGTERM) 13:29:52.472347 run osmo-stp_10.42.42.5(pid=1561): Terminating (SIGTERM) 13:29:52.524968 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1562): Terminating (SIGTERM) 13:29:52.572924 run osmo-msc_10.42.42.6(pid=1563): Terminating (SIGTERM) 13:29:52.656624 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1564): Terminating (SIGTERM) 13:29:52.699540 run osmo-mgw_10.42.42.3(pid=1565): Terminating (SIGTERM) 13:29:52.743018 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1566): Terminating (SIGTERM) 13:29:52.786453 run osmo-mgw_10.42.42.4(pid=1567): Terminating (SIGTERM) 13:29:52.840654 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1568): Terminating (SIGTERM) 13:29:52.883014 run osmo-bsc_10.42.42.7(pid=1569): Terminating (SIGTERM) 13:29:52.925090 run osmo-bts-sysmo(pid=1578): Terminating (SIGTERM) 13:29:52.937001 --- ParallelTerminationStrategy: PID 1558 died... 13:29:52.995118 run pcap-recorder_any(filters='host 10.42.42.2')(pid=1558): Terminated: ok {rc=0} 13:29:53.005788 --- ParallelTerminationStrategy: PID 1559 died... 13:29:53.063300 run osmo-hlr_10.42.42.2(pid=1559): Terminated: ok {rc=0} 13:29:53.074033 --- ParallelTerminationStrategy: PID 1560 died... 13:29:53.131625 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=1560): Terminated: ok {rc=0} 13:29:53.142349 --- ParallelTerminationStrategy: PID 1561 died... 13:29:53.199516 run osmo-stp_10.42.42.5(pid=1561): Terminated: ok {rc=0} 13:29:53.209973 --- ParallelTerminationStrategy: PID 1562 died... 13:29:53.266415 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=1562): Terminated: ok {rc=0} 13:29:53.276824 --- ParallelTerminationStrategy: PID 1563 died... 13:29:53.333761 run osmo-msc_10.42.42.6(pid=1563): Terminated: ok {rc=0} 13:29:53.344348 --- ParallelTerminationStrategy: PID 1564 died... 13:29:53.401353 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=1564): Terminated: ok {rc=0} 13:29:53.411706 --- ParallelTerminationStrategy: PID 1565 died... 13:29:53.468683 run osmo-mgw_10.42.42.3(pid=1565): Terminated {rc=15} 13:29:53.478400 --- ParallelTerminationStrategy: PID 1566 died... 13:29:53.532100 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=1566): Terminated: ok {rc=0} 13:29:53.542083 --- ParallelTerminationStrategy: PID 1567 died... 13:29:53.594803 run osmo-mgw_10.42.42.4(pid=1567): Terminated {rc=15} 13:29:53.604590 --- ParallelTerminationStrategy: PID 1568 died... 13:29:53.657701 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=1568): Terminated: ok {rc=0} 13:29:53.667485 --- ParallelTerminationStrategy: PID 1569 died... 13:29:53.720100 run osmo-bsc_10.42.42.7(pid=1569): Terminated: ok {rc=0} 13:29:53.730036 --- ParallelTerminationStrategy: PID 1578 died... 13:29:53.782860 run osmo-bts-sysmo(pid=1578): Terminated: ok {rc=0} 13:29:53.885379 bus /sierra_2: Setting Online False 13:29:54.968002 bus /sierra_2: Setting Powered False 13:30:00.962003 bus /sierra_2: Setting Powered False 13:30:02.062754 bus /sierra_2: Setting Powered False 13:30:03.195387 bus /sierra_1: Setting Online False 13:30:04.279025 bus /sierra_1: Setting Powered False 13:30:09.970063 bus /sierra_1: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/mo_mt_call.py", line 8, in <module> test_mo_mt_call(False, False) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/lib/testlib.py", line 39, in test_mo_mt_call hlr.subscriber_add(ms_mt) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/hlr_osmo.py", line 121, in subscriber_add self.log('Add subscriber', msisdn=msisdn, imsi=modem.imsi(), subscriber_id=subscriber_id, File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 467, in imsi MainLoop.wait(lambda: simmgr.GetProperties().get('SubscriberIdentity', None) is not None, timeout=10) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep) osmo_gsm_tester.core.log.Error: /sierra_2: Wait timeout {condition=<function Modem.imsi.<locals>.<lambda> at 0x7ff1005fae18>, timeout=10, timestep=1} [trial-808↪voice:sysmo+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call.py:8↪mo_mt_call.py↪/sierra_2]