jenkins LTS upgrade
Failed
voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1.mo_mt_call_osmux.py (from voice_trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1)
Stacktrace
uname-m(pid=27476): launch_sync(): local ssh process exited with status 255 [trial-805↪voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=27476)]
Standard Output
---------------------------------------------- trial-805 voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1 mo_mt_call_osmux.py ---------------------------------------------- 10:49:32.902803 tst voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 10:49:32.915225 tst voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x bts (candidates: 1) 10:49:33.082058 tst mo_mt_call_osmux.py: using LAC 7421 10:49:33.214928 tst mo_mt_call_osmux.py: using RAC 26 10:49:33.346324 tst mo_mt_call_osmux.py: using CellId 7421 10:49:33.479129 tst mo_mt_call_osmux.py: using BVCI 7417 10:49:33.504589 tst voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 10:49:33.516245 tst voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 10:49:33.527698 tst voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 10:49:33.539277 tst voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 10:49:33.550842 tst voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x ip_address (candidates: 6) 10:49:33.562380 tst voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x modem (candidates: 2) 10:49:33.669472 tst voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1: Using 1 x modem (candidates: 2) 10:49:33.870056 run osmo-hlr_10.42.42.2: Starting osmo-hlr 10:49:34.403484 run create_hlr_db(pid=27462): Launched 10:49:34.597682 bus /sierra_1: Setting Powered False 10:49:35.618391 run create_hlr_db(pid=27462): Terminated: ok {rc=0} 10:49:36.061523 run pcap-recorder_any(filters='host 10.42.42.2')(pid=27464): Launched 10:49:36.395391 run osmo-hlr_10.42.42.2(pid=27465): Launched 10:49:36.516641 run osmo-stp_10.42.42.5: Starting osmo-stp 10:49:37.092966 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=27466): Launched 10:49:37.427882 run osmo-stp_10.42.42.5(pid=27467): Launched 10:49:37.546982 run osmo-msc_10.42.42.6: Starting osmo-msc 10:49:38.155244 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=27468): Launched 10:49:38.491059 run osmo-msc_10.42.42.6(pid=27469): Launched 10:49:41.669945 run osmo-mgw_10.42.42.3: Starting osmo-mgw 10:49:42.248438 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=27470): Launched 10:49:42.584788 run osmo-mgw_10.42.42.3(pid=27471): Launched 10:49:42.704834 run osmo-mgw_10.42.42.4: Starting osmo-mgw 10:49:43.273103 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=27472): Launched 10:49:43.608339 run osmo-mgw_10.42.42.4(pid=27473): Launched 10:49:43.728232 run osmo-bsc_10.42.42.7: Starting osmo-bsc 10:49:44.425825 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=27474): Launched 10:49:44.760985 run osmo-bsc_10.42.42.7(pid=27475): Launched 10:49:51.046702 run osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7 10:49:51.531530 run osmo-trx-uhd: OSMOTRX CONF: {'osmo_trx': {'clock_reference': 'external', 'launch_trx': 'True', 'type': 'uhd', 'bts_ip': '10.42.42.50', 'trx_ip': '127.0.0.1', 'egprs': 'disable', 'channels': [{}], 'dev_args': 'type=b200,serial=30A9FFB', 'multi_arfcn': 'True', 'remote_user': 'jenkins'}} 10:49:52.149323 run uname-m(pid=27476): Launched 10:49:53.359100 run uname-m(pid=27476): ERR: Terminated: ERROR {rc=255} [trial-805↪voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=27476)] 10:49:53.508489 run uname-m(pid=27476): stdout: | (launched: 2022-12-04_10:49:51.970823) 10:49:53.657900 run uname-m(pid=27476): stderr: | (launched: 2022-12-04_10:49:52.024637) | Host key verification failed. 10:49:53.719826 tst mo_mt_call_osmux.py:8: ERR: Error: uname-m(pid=27476): launch_sync(): local ssh process exited with status 255 [trial-805↪voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=27476)] [trial-805↪voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call_osmux.py:8] 10:49:53.728878 tst mo_mt_call_osmux.py:8: Test FAILED (20.9 sec) 10:49:53.766619 run pcap-recorder_any(filters='host 10.42.42.2')(pid=27464): Terminating (SIGTERM) 10:49:53.807049 run osmo-hlr_10.42.42.2(pid=27465): Terminating (SIGTERM) 10:49:53.848750 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=27466): Terminating (SIGTERM) 10:49:53.894771 run osmo-stp_10.42.42.5(pid=27467): Terminating (SIGTERM) 10:49:53.939802 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=27468): Terminating (SIGTERM) 10:49:53.986381 run osmo-msc_10.42.42.6(pid=27469): Terminating (SIGTERM) 10:49:54.029239 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=27470): Terminating (SIGTERM) 10:49:54.073662 run osmo-mgw_10.42.42.3(pid=27471): Terminating (SIGTERM) 10:49:54.116713 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=27472): Terminating (SIGTERM) 10:49:54.159833 run osmo-mgw_10.42.42.4(pid=27473): Terminating (SIGTERM) 10:49:54.203128 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=27474): Terminating (SIGTERM) 10:49:54.246487 run osmo-bsc_10.42.42.7(pid=27475): Terminating (SIGTERM) 10:49:54.256307 --- ParallelTerminationStrategy: PID 27464 died... 10:49:54.313235 run pcap-recorder_any(filters='host 10.42.42.2')(pid=27464): Terminated: ok {rc=0} 10:49:54.323395 --- ParallelTerminationStrategy: PID 27465 died... 10:49:54.379409 run osmo-hlr_10.42.42.2(pid=27465): Terminated: ok {rc=0} 10:49:54.389541 --- ParallelTerminationStrategy: PID 27466 died... 10:49:54.448878 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=27466): Terminated: ok {rc=0} 10:49:54.478458 --- ParallelTerminationStrategy: PID 27467 died... 10:49:54.553675 run osmo-stp_10.42.42.5(pid=27467): Terminated: ok {rc=0} 10:49:54.563807 --- ParallelTerminationStrategy: PID 27468 died... 10:49:54.620263 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=27468): Terminated: ok {rc=0} 10:49:54.630535 --- ParallelTerminationStrategy: PID 27469 died... 10:49:54.686380 run osmo-msc_10.42.42.6(pid=27469): Terminated: ok {rc=0} 10:49:54.696538 --- ParallelTerminationStrategy: PID 27470 died... 10:49:54.752390 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=27470): Terminated: ok {rc=0} 10:49:54.762528 --- ParallelTerminationStrategy: PID 27471 died... 10:49:54.818392 run osmo-mgw_10.42.42.3(pid=27471): Terminated {rc=15} 10:49:54.828528 --- ParallelTerminationStrategy: PID 27472 died... 10:49:54.885808 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=27472): Terminated: ok {rc=0} 10:49:54.896924 --- ParallelTerminationStrategy: PID 27473 died... 10:49:54.954566 run osmo-mgw_10.42.42.4(pid=27473): Terminated {rc=15} 10:49:54.965725 --- ParallelTerminationStrategy: PID 27474 died... 10:49:55.023272 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=27474): Terminated: ok {rc=0} 10:49:55.202018 --- ParallelTerminationStrategy: PID 27475 died... 10:49:55.259267 run osmo-bsc_10.42.42.7(pid=27475): Terminated: ok {rc=0} 10:49:55.346195 bus /sierra_2: Setting Powered False 10:49:56.452637 bus /sierra_2: Setting Powered False 10:49:57.561424 bus /sierra_1: Setting Powered False
Standard Error
Traceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/mo_mt_call_osmux.py", line 8, in <module> test_mo_mt_call(True, True) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/voice/lib/testlib.py", line 35, in test_mo_mt_call bts.start() File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 174, in start self.trx.start(keepalive) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 327, in start self.start_remotely(keepalive) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 293, in start_remotely have_inst = rem_host.inst_compatible_for_remote() File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 150, in inst_compatible_for_remote proc = self.run_remote_sync('uname-m', ('uname', '-m')) File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync proc.launch_sync() File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync raise self.RunError('launch_sync()') osmo_gsm_tester.core.log.Error: uname-m(pid=27476): launch_sync(): local ssh process exited with status 255 [trial-805↪voice:trx-b200+mod-bts0-ts-tchh+cfg-codec-hr1↪mo_mt_call_osmux.py:8↪mo_mt_call_osmux.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=27476)]