Stack Trace
rm-remote-dir(pid=9508): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2069↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9508)]Standard Output
----------------------------------------------
trial-2069 gprs:sysmo iperf3.py
----------------------------------------------
18:29:01.328078 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.339899 tst gprs:sysmo: Using 1 x bts (candidates: 1)
18:29:01.478275 tst iperf3.py: using LAC 41703
18:29:01.599605 tst iperf3.py: using RAC 138
18:29:01.721723 tst iperf3.py: using CellId 41703
18:29:01.841755 tst iperf3.py: using BVCI 41704
18:29:01.864845 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.875432 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.886001 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.896476 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.906794 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.917102 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.927610 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.938047 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:29:01.949312 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:29:02.241929 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
18:29:02.377366 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:29:02.712109 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9466): Launched
18:29:02.989703 run iperf3-srv_10.42.42.10(pid=9467): Launched
18:29:03.187870 tst iperf3.py:8: start network...
18:29:03.323005 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:29:03.835008 run create_hlr_db(pid=9468): Launched
18:29:04.022935 bus /gobi_6: Setting Powered False
18:29:05.043205 run create_hlr_db(pid=9468): Terminated: ok {rc=0}
18:29:05.448615 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9470): Launched
18:29:05.751549 run osmo-hlr_10.42.42.2(pid=9471): Launched
18:29:05.915369 run osmo-stp_10.42.42.5: Starting osmo-stp
18:29:06.469444 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9472): Launched
18:29:06.894837 run osmo-stp_10.42.42.5(pid=9473): Launched
18:29:07.046214 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:29:07.592450 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9474): Launched
18:29:07.699788 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:29:07.962859 run patchelf(pid=9475): Launched
18:29:09.164291 run patchelf(pid=9475): Terminated: ok {rc=0}
18:29:09.285946 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
18:29:09.554877 run osmo-gsm-tester_setcap_net_admin.sh(pid=9476): Launched
18:29:10.735170 run osmo-gsm-tester_setcap_net_admin.sh(pid=9476): Terminated: ok {rc=0}
18:29:11.040409 run osmo-ggsn_10.42.42.6(pid=9479): Launched
18:29:11.304791 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
18:29:11.852518 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9498): Launched
18:29:12.154883 run osmo-sgsn_10.42.42.7(pid=9499): Launched
18:29:12.265485 run osmo-msc_10.42.42.8: Starting osmo-msc
18:29:12.818010 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9500): Launched
18:29:13.120499 run osmo-msc_10.42.42.8(pid=9501): Launched
18:29:16.282026 run osmo-mgw_10.42.42.3: Starting osmo-mgw
18:29:16.806950 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9502): Launched
18:29:17.109013 run osmo-mgw_10.42.42.3(pid=9503): Launched
18:29:17.216499 run osmo-mgw_10.42.42.4: Starting osmo-mgw
18:29:17.729953 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9504): Launched
18:29:18.031519 run osmo-mgw_10.42.42.4(pid=9505): Launched
18:29:18.139235 run osmo-bsc_10.42.42.9: Starting osmo-bsc
18:29:18.773801 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9506): Launched
18:29:19.075670 run osmo-bsc_10.42.42.9(pid=9507): Launched
18:29:25.333155 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
18:29:25.872269 run rm-remote-dir(pid=9508): Launched
18:29:29.063418 run rm-remote-dir(pid=9508): ERR: Terminated: ERROR {rc=255} [trial-2069↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9508)]
18:29:29.196636 run rm-remote-dir(pid=9508): stdout:
| (launched: 2022-12-20_18:29:25.717831)
18:29:29.330055 run rm-remote-dir(pid=9508): stderr:
| (launched: 2022-12-20_18:29:25.765791)
| ssh: connect to host 10.42.42.114 port 22: No route to host
18:29:29.385115 tst iperf3.py:8: ERR: Error: rm-remote-dir(pid=9508): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2069↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9508)] [trial-2069↪gprs:sysmo↪iperf3.py:8]
18:29:29.392699 tst iperf3.py:8: Test FAILED (28.1 sec)
18:29:29.427755 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9466): Terminating (SIGTERM)
18:29:29.468419 run iperf3-srv_10.42.42.10(pid=9467): Terminating (SIGTERM)
18:29:29.508618 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9470): Terminating (SIGTERM)
18:29:29.548553 run osmo-hlr_10.42.42.2(pid=9471): Terminating (SIGTERM)
18:29:29.589723 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9472): Terminating (SIGTERM)
18:29:29.631775 run osmo-stp_10.42.42.5(pid=9473): Terminating (SIGTERM)
18:29:29.674567 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9474): Terminating (SIGTERM)
18:29:29.717921 run osmo-ggsn_10.42.42.6(pid=9479): Terminating (SIGTERM)
18:29:29.765567 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9498): Terminating (SIGTERM)
18:29:29.818607 run osmo-sgsn_10.42.42.7(pid=9499): Terminating (SIGTERM)
18:29:29.866823 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9500): Terminating (SIGTERM)
18:29:29.915230 run osmo-msc_10.42.42.8(pid=9501): Terminating (SIGTERM)
18:29:29.961149 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9502): Terminating (SIGTERM)
18:29:30.009367 run osmo-mgw_10.42.42.3(pid=9503): Terminating (SIGTERM)
18:29:30.061927 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9504): Terminating (SIGTERM)
18:29:30.105077 run osmo-mgw_10.42.42.4(pid=9505): Terminating (SIGTERM)
18:29:30.150381 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9506): Terminating (SIGTERM)
18:29:30.195001 run osmo-bsc_10.42.42.9(pid=9507): Terminating (SIGTERM)
18:29:30.205964 --- ParallelTerminationStrategy: PID 9466 died...
18:29:30.280370 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=9466): Terminated: ok {rc=0}
18:29:30.291423 --- ParallelTerminationStrategy: PID 9467 died...
18:29:30.349424 run iperf3-srv_10.42.42.10(pid=9467): Terminated {rc=256}
18:29:30.361037 --- ParallelTerminationStrategy: PID 9470 died...
18:29:30.420347 run pcap-recorder_any(filters='host 10.42.42.2')(pid=9470): Terminated: ok {rc=0}
18:29:30.432961 --- ParallelTerminationStrategy: PID 9471 died...
18:29:30.513500 run osmo-hlr_10.42.42.2(pid=9471): Terminated: ok {rc=0}
18:29:30.525216 --- ParallelTerminationStrategy: PID 9472 died...
18:29:30.586674 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=9472): Terminated: ok {rc=0}
18:29:30.597857 --- ParallelTerminationStrategy: PID 9473 died...
18:29:30.651578 run osmo-stp_10.42.42.5(pid=9473): Terminated: ok {rc=0}
18:29:30.661552 --- ParallelTerminationStrategy: PID 9474 died...
18:29:30.715223 run pcap-recorder_any(filters='host 10.42.42.6')(pid=9474): Terminated: ok {rc=0}
18:29:30.725288 --- ParallelTerminationStrategy: PID 9479 died...
18:29:30.779014 run osmo-ggsn_10.42.42.6(pid=9479): Terminated: ok {rc=0}
18:29:30.788990 --- ParallelTerminationStrategy: PID 9498 died...
18:29:30.842842 run pcap-recorder_any(filters='host 10.42.42.7')(pid=9498): Terminated: ok {rc=0}
18:29:30.853253 --- ParallelTerminationStrategy: PID 9500 died...
18:29:30.908872 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=9500): Terminated: ok {rc=0}
18:29:30.919552 --- ParallelTerminationStrategy: PID 9501 died...
18:29:30.976112 run osmo-msc_10.42.42.8(pid=9501): Terminated: ok {rc=0}
18:29:30.986706 --- ParallelTerminationStrategy: PID 9502 died...
18:29:31.042956 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=9502): Terminated: ok {rc=0}
18:29:31.053730 --- ParallelTerminationStrategy: PID 9503 died...
18:29:31.110106 run osmo-mgw_10.42.42.3(pid=9503): Terminated {rc=15}
18:29:31.120611 --- ParallelTerminationStrategy: PID 9504 died...
18:29:31.175033 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=9504): Terminated: ok {rc=0}
18:29:31.185105 --- ParallelTerminationStrategy: PID 9499 died...
18:29:31.239152 run osmo-sgsn_10.42.42.7(pid=9499): Terminated: ok {rc=0}
18:29:31.249090 --- ParallelTerminationStrategy: PID 9505 died...
18:29:31.302845 run osmo-mgw_10.42.42.4(pid=9505): Terminated {rc=15}
18:29:31.313200 --- ParallelTerminationStrategy: PID 9506 died...
18:29:31.366956 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=9506): Terminated: ok {rc=0}
18:29:31.377088 --- ParallelTerminationStrategy: PID 9507 died...
18:29:31.430560 run osmo-bsc_10.42.42.9(pid=9507): Terminated: ok {rc=0}
18:29:31.520823 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 74, in setup_run_iperf3_test_parallel
bts.start()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/bts_sysmo.py", line 113, in start
rem_host.recreate_remote_dir(self.remote_inst)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 146, in recreate_remote_dir
self.rm_remote_dir(remote_dir)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 139, in rm_remote_dir
self.run_remote_sync('rm-remote-dir', ('test', '!', '-d', remote_dir_str, '||', 'rm', '-rf', remote_dir_str))
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/remote.py", line 134, in run_remote_sync
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: rm-remote-dir(pid=9508): launch_sync(): local ssh process exited with status 255 (ssh: connect to host 10.42.42.114 port 22: No route to host) [trial-2069↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-bts-sysmo↪host-root@10.42.42.114↪rm-remote-dir(pid=9508)]