mo_mt_sms.py sms:sysmo

Failing for the past 3 builds since #2069

Took 35 sec (from sms_sysmo)

Stack Trace

rm-remote-dir(pid=31453): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪sms:sysmo↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31453)]

Standard Output


----------------------------------------------
trial-2071 sms:sysmo mo_mt_sms.py
----------------------------------------------
21:58:04.668773 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
21:58:04.679844 tst                        sms:sysmo: Using 1 x bts (candidates: 1)
21:58:04.810847 tst                     mo_mt_sms.py: using LAC 41715
21:58:04.937307 tst                     mo_mt_sms.py: using RAC 150
21:58:05.068070 tst                     mo_mt_sms.py: using CellId 41715
21:58:05.192379 tst                     mo_mt_sms.py: using BVCI 41716
21:58:05.216693 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
21:58:05.227542 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
21:58:05.238522 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
21:58:05.249503 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
21:58:05.260501 tst                        sms:sysmo: Using 1 x ip_address (candidates: 6)
21:58:05.271799 tst                        sms:sysmo: Using 1 x modem (candidates: 2)
21:58:05.387565 tst                        sms:sysmo: Using 1 x modem (candidates: 2)
21:58:05.596645 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:07.293198 run         create_hlr_db(pid=31439): Launched
21:58:07.466051 bus                        /sierra_2: Setting Powered False
21:58:08.487360 run         create_hlr_db(pid=31439): Terminated: ok {rc=0}
21:58:08.885550 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31441): Launched
21:58:09.185224 run   osmo-hlr_10.42.42.2(pid=31442): Launched
21:58:09.293423 run              osmo-stp_10.42.42.5: Starting osmo-stp
21:58:11.751811 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31443): Launched
21:58:12.050935 run   osmo-stp_10.42.42.5(pid=31444): Launched
21:58:12.157739 run              osmo-msc_10.42.42.6: Starting osmo-msc
21:58:17.334948 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=31445): Launched
21:58:17.699564 run   osmo-msc_10.42.42.6(pid=31446): Launched
21:58:20.877933 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
21:58:22.594343 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31447): Launched
21:58:22.893289 run   osmo-mgw_10.42.42.3(pid=31448): Launched
21:58:23.002789 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
21:58:23.508685 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31449): Launched
21:58:23.806703 run   osmo-mgw_10.42.42.4(pid=31450): Launched
21:58:23.916414 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
21:58:28.282259 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=31451): Launched
21:58:28.582816 run   osmo-bsc_10.42.42.7(pid=31452): Launched
21:58:34.857727 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7
21:58:35.855547 run         rm-remote-dir(pid=31453): Launched
21:58:39.044028 run         rm-remote-dir(pid=31453): ERR: Terminated: ERROR {rc=255}  [trial-2071↪sms:sysmo↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31453)]
21:58:39.177011 run         rm-remote-dir(pid=31453): stdout: 
| (launched: 2023-01-03_21:58:35.663170) 
21:58:39.309319 run         rm-remote-dir(pid=31453): stderr: 
| (launched: 2023-01-03_21:58:35.714012)
| ssh: connect to host 10.42.42.114 port 22: No route to host 
21:58:39.363736 tst                  mo_mt_sms.py:23: ERR: Error: rm-remote-dir(pid=31453): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪sms:sysmo↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31453)]  [trial-2071↪sms:sysmo↪mo_mt_sms.py:23]
21:58:39.371062 tst                  mo_mt_sms.py:23: Test FAILED (34.7 sec)
21:58:39.407011 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31441): Terminating (SIGTERM)
21:58:39.448016 run   osmo-hlr_10.42.42.2(pid=31442): Terminating (SIGTERM)
21:58:39.490768 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31443): Terminating (SIGTERM)
21:58:39.535369 run   osmo-stp_10.42.42.5(pid=31444): Terminating (SIGTERM)
21:58:39.578286 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=31445): Terminating (SIGTERM)
21:58:39.620496 run   osmo-msc_10.42.42.6(pid=31446): Terminating (SIGTERM)
21:58:39.662629 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31447): Terminating (SIGTERM)
21:58:39.705328 run   osmo-mgw_10.42.42.3(pid=31448): Terminating (SIGTERM)
21:58:39.747648 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31449): Terminating (SIGTERM)
21:58:39.790075 run   osmo-mgw_10.42.42.4(pid=31450): Terminating (SIGTERM)
21:58:39.838645 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=31451): Terminating (SIGTERM)
21:58:39.882149 run   osmo-bsc_10.42.42.7(pid=31452): Terminating (SIGTERM)
21:58:39.892028 ---      ParallelTerminationStrategy: PID 31441 died...
21:58:39.950601 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31441): Terminated: ok {rc=0}
21:58:39.960995 ---      ParallelTerminationStrategy: PID 31442 died...
21:58:40.019110 run   osmo-hlr_10.42.42.2(pid=31442): Terminated: ok {rc=0}
21:58:40.029915 ---      ParallelTerminationStrategy: PID 31443 died...
21:58:40.088375 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31443): Terminated: ok {rc=0}
21:58:40.098671 ---      ParallelTerminationStrategy: PID 31444 died...
21:58:40.156598 run   osmo-stp_10.42.42.5(pid=31444): Terminated: ok {rc=0}
21:58:40.166811 ---      ParallelTerminationStrategy: PID 31445 died...
21:58:40.224954 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=31445): Terminated: ok {rc=0}
21:58:40.235181 ---      ParallelTerminationStrategy: PID 31446 died...
21:58:40.293387 run   osmo-msc_10.42.42.6(pid=31446): Terminated: ok {rc=0}
21:58:40.303673 ---      ParallelTerminationStrategy: PID 31447 died...
21:58:40.361855 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31447): Terminated: ok {rc=0}
21:58:40.372322 ---      ParallelTerminationStrategy: PID 31448 died...
21:58:40.430238 run   osmo-mgw_10.42.42.3(pid=31448): Terminated {rc=15}
21:58:40.440547 ---      ParallelTerminationStrategy: PID 31449 died...
21:58:40.498508 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31449): Terminated: ok {rc=0}
21:58:40.509199 ---      ParallelTerminationStrategy: PID 31450 died...
21:58:40.566898 run   osmo-mgw_10.42.42.4(pid=31450): Terminated {rc=15}
21:58:40.577693 ---      ParallelTerminationStrategy: PID 31451 died...
21:58:40.635786 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=31451): Terminated: ok {rc=0}
21:58:41.121346 ---      ParallelTerminationStrategy: PID 31452 died...
21:58:41.179536 run   osmo-bsc_10.42.42.7(pid=31452): Terminated: ok {rc=0}
21:58:41.269511 bus                        /sierra_3: Setting Powered False
21:58:42.375049 bus                        /sierra_3: Setting Powered False
21:58:43.488203 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/sms/mo_mt_sms.py", line 23, in <module>
    bts.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
    rem_host.recreate_remote_dir(self.remote_inst)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
    self.rm_remote_dir(remote_dir)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
    self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=31453): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2071↪sms:sysmo↪mo_mt_sms.py:23↪mo_mt_sms.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=31453)]