Failed

sms:trx-b200.mo_mt_sms.py (from sms_trx-b200)

Failing for the past 90 builds (Since #749 )

Took 21 sec.

Stacktrace

uname-m(pid=22177): launch_sync(): local ssh process exited with status 255 [trial-838↪sms:trx-b200↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=22177)]

Standard Output

----------------------------------------------
trial-838 sms:trx-b200 mo_mt_sms.py
----------------------------------------------
02:38:06.695138 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
02:38:06.705642 tst                     sms:trx-b200: Using 1 x bts (candidates: 1)
02:38:06.842264 tst                     mo_mt_sms.py: using LAC 10051
02:38:06.964380 tst                     mo_mt_sms.py: using RAC 106
02:38:07.091713 tst                     mo_mt_sms.py: using CellId 10051
02:38:07.214464 tst                     mo_mt_sms.py: using BVCI 10047
02:38:07.238252 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
02:38:07.248837 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
02:38:07.259312 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
02:38:07.269968 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
02:38:07.280507 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
02:38:07.291082 tst                     sms:trx-b200: Using 1 x modem (candidates: 2)
02:38:07.396855 tst                     sms:trx-b200: Using 1 x modem (candidates: 2)
02:38:07.593831 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
02:38:08.121468 run         create_hlr_db(pid=22163): Launched
02:38:08.312821 bus                        /sierra_1: Setting Powered False
02:38:09.333224 run         create_hlr_db(pid=22163): Terminated: ok {rc=0}
02:38:09.750209 run pcap-recorder_any(filters='host 10.42.42.2')(pid=22165): Launched
02:38:10.060920 run   osmo-hlr_10.42.42.2(pid=22166): Launched
02:38:10.172851 run              osmo-stp_10.42.42.5: Starting osmo-stp
02:38:10.740348 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=22167): Launched
02:38:11.071590 run   osmo-stp_10.42.42.5(pid=22168): Launched
02:38:11.189574 run              osmo-msc_10.42.42.6: Starting osmo-msc
02:38:11.792000 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=22169): Launched
02:38:12.125042 run   osmo-msc_10.42.42.6(pid=22170): Launched
02:38:15.301995 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
02:38:15.875520 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=22171): Launched
02:38:16.206116 run   osmo-mgw_10.42.42.3(pid=22172): Launched
02:38:16.324248 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
02:38:16.887206 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=22173): Launched
02:38:17.219161 run   osmo-mgw_10.42.42.4(pid=22174): Launched
02:38:17.337945 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
02:38:18.031577 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=22175): Launched
02:38:18.364080 run   osmo-bsc_10.42.42.7(pid=22176): Launched
02:38:24.648317 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
02:38:25.128507 run                     osmo-trx-uhd: OSMOTRX CONF: {'osmo_trx': {'clock_reference': 'external', 'launch_trx': 'True', 'type': 'uhd', 'bts_ip': '10.42.42.50', 'trx_ip': '127.0.0.1', 'egprs': 'disable', 'channels': [{}], 'dev_args': 'type=b200,serial=30A9FFB', 'multi_arfcn': 'True', 'remote_user': 'jenkins'}}
02:38:25.718065 run               uname-m(pid=22177): Launched
02:38:26.911443 run               uname-m(pid=22177): ERR: Terminated: ERROR {rc=255}  [trial-838↪sms:trx-b200↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=22177)]
02:38:27.050882 run               uname-m(pid=22177): stdout: 
| (launched: 2022-12-06_02:38:25.551086) 
02:38:27.190454 run               uname-m(pid=22177): stderr: 
| (launched: 2022-12-06_02:38:25.601254)
| Host key verification failed. 
02:38:27.247612 tst                  mo_mt_sms.py:23: ERR: Error: uname-m(pid=22177): launch_sync(): local ssh process exited with status 255 [trial-838↪sms:trx-b200↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=22177)]  [trial-838↪sms:trx-b200↪mo_mt_sms.py:23]
02:38:27.255290 tst                  mo_mt_sms.py:23: Test FAILED (20.6 sec)
02:38:27.290438 run pcap-recorder_any(filters='host 10.42.42.2')(pid=22165): Terminating (SIGTERM)
02:38:27.329865 run   osmo-hlr_10.42.42.2(pid=22166): Terminating (SIGTERM)
02:38:27.371080 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=22167): Terminating (SIGTERM)
02:38:27.414506 run   osmo-stp_10.42.42.5(pid=22168): Terminating (SIGTERM)
02:38:27.455950 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=22169): Terminating (SIGTERM)
02:38:27.499803 run   osmo-msc_10.42.42.6(pid=22170): Terminating (SIGTERM)
02:38:27.540927 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=22171): Terminating (SIGTERM)
02:38:27.585216 run   osmo-mgw_10.42.42.3(pid=22172): Terminating (SIGTERM)
02:38:27.642084 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=22173): Terminating (SIGTERM)
02:38:27.683417 run   osmo-mgw_10.42.42.4(pid=22174): Terminating (SIGTERM)
02:38:27.723724 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=22175): Terminating (SIGTERM)
02:38:27.763828 run   osmo-bsc_10.42.42.7(pid=22176): Terminating (SIGTERM)
02:38:27.773350 ---      ParallelTerminationStrategy: PID 22165 died...
02:38:27.826434 run pcap-recorder_any(filters='host 10.42.42.2')(pid=22165): Terminated: ok {rc=0}
02:38:27.835800 ---      ParallelTerminationStrategy: PID 22166 died...
02:38:27.888514 run   osmo-hlr_10.42.42.2(pid=22166): Terminated: ok {rc=0}
02:38:27.897949 ---      ParallelTerminationStrategy: PID 22167 died...
02:38:27.950538 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=22167): Terminated: ok {rc=0}
02:38:27.960115 ---      ParallelTerminationStrategy: PID 22168 died...
02:38:28.013338 run   osmo-stp_10.42.42.5(pid=22168): Terminated: ok {rc=0}
02:38:28.022849 ---      ParallelTerminationStrategy: PID 22169 died...
02:38:28.075881 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=22169): Terminated: ok {rc=0}
02:38:28.085442 ---      ParallelTerminationStrategy: PID 22170 died...
02:38:28.138009 run   osmo-msc_10.42.42.6(pid=22170): Terminated: ok {rc=0}
02:38:28.147655 ---      ParallelTerminationStrategy: PID 22171 died...
02:38:28.201037 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=22171): Terminated: ok {rc=0}
02:38:28.210616 ---      ParallelTerminationStrategy: PID 22172 died...
02:38:28.263856 run   osmo-mgw_10.42.42.3(pid=22172): Terminated {rc=15}
02:38:28.273445 ---      ParallelTerminationStrategy: PID 22173 died...
02:38:28.326079 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=22173): Terminated: ok {rc=0}
02:38:28.335763 ---      ParallelTerminationStrategy: PID 22174 died...
02:38:28.389016 run   osmo-mgw_10.42.42.4(pid=22174): Terminated {rc=15}
02:38:28.398556 ---      ParallelTerminationStrategy: PID 22175 died...
02:38:28.451666 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=22175): Terminated: ok {rc=0}
02:38:28.630142 ---      ParallelTerminationStrategy: PID 22176 died...
02:38:28.686422 run   osmo-bsc_10.42.42.7(pid=22176): Terminated: ok {rc=0}
02:38:28.774908 bus                        /sierra_2: Setting Powered False
02:38:29.877479 bus                        /sierra_2: Setting Powered False
02:38:30.983375 bus                        /sierra_1: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/sms/mo_mt_sms.py", line 23, in <module>
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 174, in start
    self.trx.start(keepalive)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 327, in start
    self.start_remotely(keepalive)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 293, in start_remotely
    have_inst = rem_host.inst_compatible_for_remote()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 150, in inst_compatible_for_remote
    proc = self.run_remote_sync('uname-m', ('uname', '-m'))
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: uname-m(pid=22177): launch_sync(): local ssh process exited with status 255 [trial-838↪sms:trx-b200↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=22177)]