assert_extension.py ussd:trx-b200

Failing for the past 61 builds since #749

Took 20 sec (from ussd_trx-b200)

Stack Trace

uname-m(pid=4809): launch_sync(): local ssh process exited with status 255 [trial-809↪ussd:trx-b200↪assert_extension.py:24↪assert_extension.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=4809)]

Standard Output


----------------------------------------------
trial-809 ussd:trx-b200 assert_extension.py
----------------------------------------------
14:40:45.038369 tst                    ussd:trx-b200: Using 1 x ip_address (candidates: 6)
14:40:45.049054 tst                    ussd:trx-b200: Using 1 x bts (candidates: 1)
14:40:45.183782 tst              assert_extension.py: using LAC 7733
14:40:45.309552 tst              assert_extension.py: using RAC 83
14:40:45.433850 tst              assert_extension.py: using CellId 7733
14:40:45.559450 tst              assert_extension.py: using BVCI 7729
14:40:45.584293 tst                    ussd:trx-b200: Using 1 x ip_address (candidates: 6)
14:40:45.595665 tst                    ussd:trx-b200: Using 1 x ip_address (candidates: 6)
14:40:45.606896 tst                    ussd:trx-b200: Using 1 x ip_address (candidates: 6)
14:40:45.618154 tst                    ussd:trx-b200: Using 1 x ip_address (candidates: 6)
14:40:45.629373 tst                    ussd:trx-b200: Using 1 x ip_address (candidates: 6)
14:40:45.640397 tst                    ussd:trx-b200: Using 1 x modem (candidates: 1)
14:40:45.826238 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
14:40:46.321847 run          create_hlr_db(pid=4794): Launched
14:40:46.510414 bus                        /sierra_1: Setting Powered False
14:40:47.530891 run          create_hlr_db(pid=4794): Terminated: ok {rc=0}
14:40:47.939839 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4796): Launched
14:40:48.245021 run    osmo-hlr_10.42.42.2(pid=4797): Launched
14:40:48.355419 run              osmo-stp_10.42.42.5: Starting osmo-stp
14:40:48.888320 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4799): Launched
14:40:49.194776 run    osmo-stp_10.42.42.5(pid=4800): Launched
14:40:49.302434 run              osmo-msc_10.42.42.6: Starting osmo-msc
14:40:49.855093 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=4801): Launched
14:40:50.161126 run    osmo-msc_10.42.42.6(pid=4802): Launched
14:40:53.323202 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
14:40:53.849291 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=4803): Launched
14:40:54.154726 run    osmo-mgw_10.42.42.3(pid=4804): Launched
14:40:54.262351 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
14:40:54.783190 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=4805): Launched
14:40:55.090943 run    osmo-mgw_10.42.42.4(pid=4806): Launched
14:40:55.199278 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
14:40:55.846593 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=4807): Launched
14:40:56.154918 run    osmo-bsc_10.42.42.7(pid=4808): Launched
14:41:02.417135 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
14:41:02.857092 run                     osmo-trx-uhd: OSMOTRX CONF: {'osmo_trx': {'clock_reference': 'external', 'launch_trx': 'True', 'type': 'uhd', 'bts_ip': '10.42.42.50', 'trx_ip': '127.0.0.1', 'egprs': 'disable', 'channels': [{}], 'dev_args': 'type=b200,serial=30A9FFB', 'multi_arfcn': 'True', 'remote_user': 'jenkins'}}
14:41:03.425552 run                uname-m(pid=4809): Launched
14:41:04.615709 run                uname-m(pid=4809): ERR: Terminated: ERROR {rc=255}  [trial-809↪ussd:trx-b200↪assert_extension.py:24↪assert_extension.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=4809)]
14:41:04.752027 run                uname-m(pid=4809): stdout: 
| (launched: 2022-12-04_14:41:03.257423) 
14:41:04.888380 run                uname-m(pid=4809): stderr: 
| (launched: 2022-12-04_14:41:03.306522)
| Host key verification failed. 
14:41:04.944707 tst           assert_extension.py:24: ERR: Error: uname-m(pid=4809): launch_sync(): local ssh process exited with status 255 [trial-809↪ussd:trx-b200↪assert_extension.py:24↪assert_extension.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=4809)]  [trial-809↪ussd:trx-b200↪assert_extension.py:24]
14:41:04.952412 tst           assert_extension.py:24: Test FAILED (19.9 sec)
14:41:04.988408 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4796): Terminating (SIGTERM)
14:41:05.028881 run    osmo-hlr_10.42.42.2(pid=4797): Terminating (SIGTERM)
14:41:05.070583 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4799): Terminating (SIGTERM)
14:41:05.114090 run    osmo-stp_10.42.42.5(pid=4800): Terminating (SIGTERM)
14:41:05.156315 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=4801): Terminating (SIGTERM)
14:41:05.198788 run    osmo-msc_10.42.42.6(pid=4802): Terminating (SIGTERM)
14:41:05.240930 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=4803): Terminating (SIGTERM)
14:41:05.283264 run    osmo-mgw_10.42.42.3(pid=4804): Terminating (SIGTERM)
14:41:05.340481 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=4805): Terminating (SIGTERM)
14:41:05.381491 run    osmo-mgw_10.42.42.4(pid=4806): Terminating (SIGTERM)
14:41:05.422351 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=4807): Terminating (SIGTERM)
14:41:05.462422 run    osmo-bsc_10.42.42.7(pid=4808): Terminating (SIGTERM)
14:41:05.471892 ---      ParallelTerminationStrategy: PID 4796 died...
14:41:05.525450 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4796): Terminated: ok {rc=0}
14:41:05.535296 ---      ParallelTerminationStrategy: PID 4797 died...
14:41:05.588409 run    osmo-hlr_10.42.42.2(pid=4797): Terminated: ok {rc=0}
14:41:05.598330 ---      ParallelTerminationStrategy: PID 4799 died...
14:41:05.651816 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4799): Terminated: ok {rc=0}
14:41:05.661454 ---      ParallelTerminationStrategy: PID 4800 died...
14:41:05.714589 run    osmo-stp_10.42.42.5(pid=4800): Terminated: ok {rc=0}
14:41:05.724323 ---      ParallelTerminationStrategy: PID 4801 died...
14:41:05.777711 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=4801): Terminated: ok {rc=0}
14:41:05.787389 ---      ParallelTerminationStrategy: PID 4802 died...
14:41:05.840279 run    osmo-msc_10.42.42.6(pid=4802): Terminated: ok {rc=0}
14:41:05.850227 ---      ParallelTerminationStrategy: PID 4803 died...
14:41:05.903457 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=4803): Terminated: ok {rc=0}
14:41:05.913351 ---      ParallelTerminationStrategy: PID 4804 died...
14:41:05.966753 run    osmo-mgw_10.42.42.3(pid=4804): Terminated {rc=15}
14:41:05.976671 ---      ParallelTerminationStrategy: PID 4805 died...
14:41:06.029795 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=4805): Terminated: ok {rc=0}
14:41:06.040096 ---      ParallelTerminationStrategy: PID 4806 died...
14:41:06.093358 run    osmo-mgw_10.42.42.4(pid=4806): Terminated {rc=15}
14:41:06.103265 ---      ParallelTerminationStrategy: PID 4807 died...
14:41:06.156411 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=4807): Terminated: ok {rc=0}
14:41:06.641726 ---      ParallelTerminationStrategy: PID 4808 died...
14:41:06.698653 run    osmo-bsc_10.42.42.7(pid=4808): Terminated: ok {rc=0}
14:41:06.785836 bus                        /sierra_1: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/sysmocom/suites/ussd/assert_extension.py", line 24, in <module>
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 174, in start
    self.trx.start(keepalive)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 327, in start
    self.start_remotely(keepalive)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_osmotrx.py", line 293, in start_remotely
    have_inst = rem_host.inst_compatible_for_remote()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 150, in inst_compatible_for_remote
    proc = self.run_remote_sync('uname-m', ('uname', '-m'))
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-rnd/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: uname-m(pid=4809): launch_sync(): local ssh process exited with status 255 [trial-809↪ussd:trx-b200↪assert_extension.py:24↪assert_extension.py↪osmo-bts-trx↪osmo-trx-uhd↪host-jenkins@127.0.0.1↪uname-m(pid=4809)]