Failed

sms:trx-b200.mo_mt_sms.py (from sms_trx-b200)

Failing for the past 67 builds (Since #749 )

Took 21 sec.

Stacktrace

uname-m(pid=22700): launch_sync(): local ssh process exited with status 255 [trial-815↪sms:trx-b200↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=22700)]

Standard Output

----------------------------------------------
trial-815 sms:trx-b200 mo_mt_sms.py
----------------------------------------------
23:14:13.677720 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
23:14:13.688207 tst                     sms:trx-b200: Using 1 x bts (candidates: 1)
23:14:13.825488 tst                     mo_mt_sms.py: using LAC 8211
23:14:13.952261 tst                     mo_mt_sms.py: using RAC 51
23:14:14.072537 tst                     mo_mt_sms.py: using CellId 8211
23:14:14.194417 tst                     mo_mt_sms.py: using BVCI 8207
23:14:14.218074 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
23:14:14.228619 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
23:14:14.239225 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
23:14:14.249945 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
23:14:14.260604 tst                     sms:trx-b200: Using 1 x ip_address (candidates: 6)
23:14:14.271249 tst                     sms:trx-b200: Using 1 x modem (candidates: 2)
23:14:14.371118 tst                     sms:trx-b200: Using 1 x modem (candidates: 2)
23:14:14.558147 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
23:14:15.049533 run         create_hlr_db(pid=22686): Launched
23:14:15.229667 bus                        /sierra_1: Setting Powered False
23:14:16.249699 run         create_hlr_db(pid=22686): Terminated: ok {rc=0}
23:14:16.688693 run pcap-recorder_any(filters='host 10.42.42.2')(pid=22688): Launched
23:14:17.021376 run   osmo-hlr_10.42.42.2(pid=22689): Launched
23:14:17.141844 run              osmo-stp_10.42.42.5: Starting osmo-stp
23:14:17.717324 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=22690): Launched
23:14:18.051191 run   osmo-stp_10.42.42.5(pid=22691): Launched
23:14:18.169210 run              osmo-msc_10.42.42.6: Starting osmo-msc
23:14:18.776148 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=22692): Launched
23:14:19.108808 run   osmo-msc_10.42.42.6(pid=22693): Launched
23:14:22.285239 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
23:14:22.860650 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=22694): Launched
23:14:23.194339 run   osmo-mgw_10.42.42.3(pid=22695): Launched
23:14:23.312077 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
23:14:23.877066 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=22696): Launched
23:14:24.207785 run   osmo-mgw_10.42.42.4(pid=22697): Launched
23:14:24.326188 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
23:14:25.017463 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=22698): Launched
23:14:25.352756 run   osmo-bsc_10.42.42.7(pid=22699): Launched
23:14:31.636044 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
23:14:32.117943 run                     osmo-trx-uhd: OSMOTRX CONF: {'osmo_trx': {'clock_reference': 'external', 'launch_trx': 'True', 'type': 'uhd', 'bts_ip': '10.42.42.50', 'trx_ip': '127.0.0.1', 'egprs': 'disable', 'channels': [{}], 'dev_args': 'type=b200,serial=30A9FFB', 'multi_arfcn': 'True', 'remote_user': 'jenkins'}}
23:14:32.731187 run               uname-m(pid=22700): Launched
23:14:33.936309 run               uname-m(pid=22700): ERR: Terminated: ERROR {rc=255}  [trial-815↪sms:trx-b200↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=22700)]
23:14:34.082750 run               uname-m(pid=22700): stdout: 
| (launched: 2022-12-04_23:14:32.555259) 
23:14:34.229902 run               uname-m(pid=22700): stderr: 
| (launched: 2022-12-04_23:14:32.608418)
| Host key verification failed. 
23:14:34.290744 tst                  mo_mt_sms.py:23: ERR: Error: uname-m(pid=22700): launch_sync(): local ssh process exited with status 255 [trial-815↪sms:trx-b200↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=22700)]  [trial-815↪sms:trx-b200↪mo_mt_sms.py:23]
23:14:34.298702 tst                  mo_mt_sms.py:23: Test FAILED (20.6 sec)
23:14:34.334757 run pcap-recorder_any(filters='host 10.42.42.2')(pid=22688): Terminating (SIGTERM)
23:14:34.375970 run   osmo-hlr_10.42.42.2(pid=22689): Terminating (SIGTERM)
23:14:34.418318 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=22690): Terminating (SIGTERM)
23:14:34.462972 run   osmo-stp_10.42.42.5(pid=22691): Terminating (SIGTERM)
23:14:34.521762 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=22692): Terminating (SIGTERM)
23:14:34.574574 run   osmo-msc_10.42.42.6(pid=22693): Terminating (SIGTERM)
23:14:34.619056 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=22694): Terminating (SIGTERM)
23:14:34.661525 run   osmo-mgw_10.42.42.3(pid=22695): Terminating (SIGTERM)
23:14:34.704138 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=22696): Terminating (SIGTERM)
23:14:34.746381 run   osmo-mgw_10.42.42.4(pid=22697): Terminating (SIGTERM)
23:14:34.788909 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=22698): Terminating (SIGTERM)
23:14:34.831346 run   osmo-bsc_10.42.42.7(pid=22699): Terminating (SIGTERM)
23:14:34.841110 ---      ParallelTerminationStrategy: PID 22688 died...
23:14:34.896545 run pcap-recorder_any(filters='host 10.42.42.2')(pid=22688): Terminated: ok {rc=0}
23:14:34.906398 ---      ParallelTerminationStrategy: PID 22689 died...
23:14:34.961052 run   osmo-hlr_10.42.42.2(pid=22689): Terminated: ok {rc=0}
23:14:34.970882 ---      ParallelTerminationStrategy: PID 22690 died...
23:14:35.025674 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=22690): Terminated: ok {rc=0}
23:14:35.035505 ---      ParallelTerminationStrategy: PID 22691 died...
23:14:35.090283 run   osmo-stp_10.42.42.5(pid=22691): Terminated: ok {rc=0}
23:14:35.100103 ---      ParallelTerminationStrategy: PID 22692 died...
23:14:35.154957 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=22692): Terminated: ok {rc=0}
23:14:35.164810 ---      ParallelTerminationStrategy: PID 22693 died...
23:14:35.220004 run   osmo-msc_10.42.42.6(pid=22693): Terminated: ok {rc=0}
23:14:35.230092 ---      ParallelTerminationStrategy: PID 22694 died...
23:14:35.285063 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=22694): Terminated: ok {rc=0}
23:14:35.295025 ---      ParallelTerminationStrategy: PID 22695 died...
23:14:35.350559 run   osmo-mgw_10.42.42.3(pid=22695): Terminated {rc=15}
23:14:35.360468 ---      ParallelTerminationStrategy: PID 22696 died...
23:14:35.415369 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=22696): Terminated: ok {rc=0}
23:14:35.425440 ---      ParallelTerminationStrategy: PID 22697 died...
23:14:35.481346 run   osmo-mgw_10.42.42.4(pid=22697): Terminated {rc=15}
23:14:35.491672 ---      ParallelTerminationStrategy: PID 22698 died...
23:14:35.547683 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=22698): Terminated: ok {rc=0}
23:14:35.726118 ---      ParallelTerminationStrategy: PID 22699 died...
23:14:35.782280 run   osmo-bsc_10.42.42.7(pid=22699): Terminated: ok {rc=0}
23:14:35.870293 bus                        /sierra_2: Setting Powered False
23:14:36.976252 bus                        /sierra_2: Setting Powered False
23:14:38.082390 bus                        /sierra_1: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/sms/mo_mt_sms.py", line 23, in <module>
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 174, in start
    self.trx.start(keepalive)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 327, in start
    self.start_remotely(keepalive)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 293, in start_remotely
    have_inst = rem_host.inst_compatible_for_remote()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 150, in inst_compatible_for_remote
    proc = self.run_remote_sync('uname-m', ('uname', '-m'))
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: uname-m(pid=22700): launch_sync(): local ssh process exited with status 255 [trial-815↪sms:trx-b200↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=22700)]