56gprs:trx-umtrx9.0osmo-stp_10.42.42.5(pid=28965): Process ended prematurely: osmo-stp_10.42.42.5(pid=28965) [trial-2373↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28965)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=28965): Process ended prematurely: osmo-stp_10.42.42.5(pid=28965) [trial-2373↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28965)]
----------------------------------------------
trial-2373 gprs:trx-umtrx cs_paging_gprs_active.py
----------------------------------------------
10:27:35.705759 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:35.716278 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
10:27:35.854010 tst cs_paging_gprs_active.py: using LAC 27756
10:27:35.979454 tst cs_paging_gprs_active.py: using RAC 216
10:27:36.096906 tst cs_paging_gprs_active.py: using CellId 27756
10:27:36.216354 tst cs_paging_gprs_active.py: using BVCI 27757
10:27:36.239991 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:36.250673 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:36.261049 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:36.271435 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:36.281700 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:36.291914 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:36.302386 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:36.312552 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:36.323100 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
10:27:36.440636 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
10:27:36.721499 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
10:27:36.849278 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:27:37.165687 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28955): Launched
10:27:37.428096 run iperf3-srv_10.42.42.10(pid=28956): Launched
10:27:37.615198 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
10:27:37.743086 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:27:38.058992 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28957): Launched
10:27:38.321775 run iperf3-srv_10.42.42.10(pid=28958): Launched
10:27:38.507598 tst cs_paging_gprs_active.py:38: start network...
10:27:38.634498 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:27:39.124366 run create_hlr_db(pid=28959): Launched
10:27:39.300010 bus /sierra_2: Setting Powered False
10:27:40.322194 run create_hlr_db(pid=28959): Terminated: ok {rc=0}
10:27:40.757727 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28961): Launched
10:27:41.085108 run osmo-hlr_10.42.42.2(pid=28962): Launched
10:27:41.206286 run osmo-stp_10.42.42.5: Starting osmo-stp
10:27:41.774926 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28964): Launched
10:27:42.101344 run osmo-stp_10.42.42.5(pid=28965): Launched
10:27:42.220694 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:27:42.790360 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28967): Launched
10:27:42.907047 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:27:43.191590 run patchelf(pid=28968): Launched
10:27:43.383396 run osmo-stp_10.42.42.5(pid=28965): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28965)]
10:27:43.523554 run osmo-stp_10.42.42.5(pid=28965): stdout:
| (launched: 2022-12-10_10:27:41.941291)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:27:43.664038 run osmo-stp_10.42.42.5(pid=28965): stderr:
| �[0;m�[38;5;43m20221210102742133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102742133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102742133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102742134 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102742134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102742134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102742134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:27:43.800034 run osmo-stp_10.42.42.5(pid=28965): stdout:
| (launched: 2022-12-10_10:27:41.941291)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:27:43.934930 run osmo-stp_10.42.42.5(pid=28965): stderr:
| �[0;m�[38;5;43m20221210102742133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102742133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102742133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102742134 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102742134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102742134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102742134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:27:44.091502 run patchelf(pid=28968): Terminating (SIGINT)
10:27:44.276133 run patchelf(pid=28968): Terminated: ok {rc=0}
10:27:44.310057 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=28965): Process ended prematurely: osmo-stp_10.42.42.5(pid=28965) [trial-2373↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28965)] [trial-2373↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38]
10:27:44.319832 tst cs_paging_gprs_active.py:38: Test FAILED (8.6 sec)
10:27:44.384446 run osmo-hlr_10.42.42.2(pid=28962): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:trx-umtrx↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=28962)]
10:27:44.436556 run osmo-hlr_10.42.42.2(pid=28962): stdout:
| (launched: 2022-12-10_10:27:40.923882)
10:27:44.489259 run osmo-hlr_10.42.42.2(pid=28962): stderr:
| �[0;m�[1;31m20221210102741123 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210102741123 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210102741123 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210102741123 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210102741123 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210102741123 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210102741136 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:trx-umtrx/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210102741145 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210102741145 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210102741145 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:27:44.534019 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28955): Terminating (SIGTERM)
10:27:44.576695 run iperf3-srv_10.42.42.10(pid=28956): Terminating (SIGTERM)
10:27:44.619307 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28957): Terminating (SIGTERM)
10:27:44.661838 run iperf3-srv_10.42.42.10(pid=28958): Terminating (SIGTERM)
10:27:44.704305 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28961): Terminating (SIGTERM)
10:27:44.746721 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28964): Terminating (SIGTERM)
10:27:44.789120 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28967): Terminating (SIGTERM)
10:27:44.799128 --- ParallelTerminationStrategy: PID 28955 died...
10:27:44.855494 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28955): Terminated: ok {rc=0}
10:27:44.865922 --- ParallelTerminationStrategy: PID 28956 died...
10:27:44.921921 run iperf3-srv_10.42.42.10(pid=28956): Terminated {rc=256}
10:27:44.932307 --- ParallelTerminationStrategy: PID 28957 died...
10:27:44.988793 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28957): Terminated: ok {rc=0}
10:27:44.998997 --- ParallelTerminationStrategy: PID 28958 died...
10:27:45.055632 run iperf3-srv_10.42.42.10(pid=28958): Terminated {rc=256}
10:27:45.065865 --- ParallelTerminationStrategy: PID 28961 died...
10:27:45.122400 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28961): Terminated: ok {rc=0}
10:27:45.132579 --- ParallelTerminationStrategy: PID 28964 died...
10:27:45.189021 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28964): Terminated: ok {rc=0}
10:27:45.199465 --- ParallelTerminationStrategy: PID 28967 died...
10:27:45.256405 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28967): Terminated: ok {rc=0}
10:27:45.345356 bus /gobi_0: Setting Powered False
10:27:46.453442 bus /gobi_0: Setting Powered False
10:27:47.564797 bus /gobi_6: Setting Powered False
56gprs:trx-umtrx8.0osmo-stp_10.42.42.5(pid=28977): Process ended prematurely: osmo-stp_10.42.42.5(pid=28977) [trial-2373↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28977)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=28977): Process ended prematurely: osmo-stp_10.42.42.5(pid=28977) [trial-2373↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28977)]
----------------------------------------------
trial-2373 gprs:trx-umtrx iperf3.py
----------------------------------------------
10:27:48.675454 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:48.686759 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
10:27:48.825485 tst iperf3.py: using LAC 27757
10:27:48.945280 tst iperf3.py: using RAC 217
10:27:49.066508 tst iperf3.py: using CellId 27757
10:27:49.188749 tst iperf3.py: using BVCI 27758
10:27:49.211979 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:49.222779 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:49.233383 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:49.243822 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:49.254234 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:49.264602 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:49.275187 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:49.285661 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:49.296204 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
10:27:49.591612 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
10:27:49.728394 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:27:50.066379 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28969): Launched
10:27:50.347785 run iperf3-srv_10.42.42.10(pid=28970): Launched
10:27:50.548616 tst iperf3.py:8: start network...
10:27:50.685631 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:27:51.199369 run create_hlr_db(pid=28971): Launched
10:27:51.389986 bus /gobi_6: Setting Powered False
10:27:52.410203 run create_hlr_db(pid=28971): Terminated: ok {rc=0}
10:27:52.846362 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28973): Launched
10:27:53.173105 run osmo-hlr_10.42.42.2(pid=28974): Launched
10:27:53.294015 run osmo-stp_10.42.42.5: Starting osmo-stp
10:27:53.863563 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28976): Launched
10:27:54.188464 run osmo-stp_10.42.42.5(pid=28977): Launched
10:27:54.306536 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:27:54.874397 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28979): Launched
10:27:54.991441 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:27:55.275878 run patchelf(pid=28980): Launched
10:27:55.467019 run osmo-stp_10.42.42.5(pid=28977): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28977)]
10:27:55.607156 run osmo-stp_10.42.42.5(pid=28977): stdout:
| (launched: 2022-12-10_10:27:54.028914)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:27:55.747705 run osmo-stp_10.42.42.5(pid=28977): stderr:
| �[0;m�[38;5;43m20221210102754220 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102754220 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102754220 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102754221 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102754223 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102754223 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102754223 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:27:55.883034 run osmo-stp_10.42.42.5(pid=28977): stdout:
| (launched: 2022-12-10_10:27:54.028914)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:27:56.016577 run osmo-stp_10.42.42.5(pid=28977): stderr:
| �[0;m�[38;5;43m20221210102754220 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102754220 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102754220 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102754221 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102754223 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102754223 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102754223 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:27:56.172557 run patchelf(pid=28980): Terminating (SIGINT)
10:27:56.356202 run patchelf(pid=28980): Terminated: ok {rc=0}
10:27:56.389753 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=28977): Process ended prematurely: osmo-stp_10.42.42.5(pid=28977) [trial-2373↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28977)] [trial-2373↪gprs:trx-umtrx↪iperf3.py:8]
10:27:56.399549 tst iperf3.py:8: Test FAILED (7.8 sec)
10:27:56.463610 run osmo-hlr_10.42.42.2(pid=28974): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:trx-umtrx↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=28974)]
10:27:56.515406 run osmo-hlr_10.42.42.2(pid=28974): stdout:
| (launched: 2022-12-10_10:27:53.012379)
10:27:56.567805 run osmo-hlr_10.42.42.2(pid=28974): stderr:
| �[0;m�[1;31m20221210102753211 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210102753212 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210102753212 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210102753212 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210102753212 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210102753212 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210102753225 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:trx-umtrx/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210102753233 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210102753233 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210102753233 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:27:56.612301 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28969): Terminating (SIGTERM)
10:27:56.654884 run iperf3-srv_10.42.42.10(pid=28970): Terminating (SIGTERM)
10:27:56.697491 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28973): Terminating (SIGTERM)
10:27:56.739947 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28976): Terminating (SIGTERM)
10:27:56.782638 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28979): Terminating (SIGTERM)
10:27:56.792520 --- ParallelTerminationStrategy: PID 28969 died...
10:27:56.849705 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28969): Terminated: ok {rc=0}
10:27:56.860027 --- ParallelTerminationStrategy: PID 28970 died...
10:27:56.916457 run iperf3-srv_10.42.42.10(pid=28970): Terminated {rc=256}
10:27:56.926906 --- ParallelTerminationStrategy: PID 28973 died...
10:27:56.983384 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28973): Terminated: ok {rc=0}
10:27:56.994017 --- ParallelTerminationStrategy: PID 28976 died...
10:27:57.050472 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28976): Terminated: ok {rc=0}
10:27:57.061063 --- ParallelTerminationStrategy: PID 28979 died...
10:27:57.117541 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28979): Terminated: ok {rc=0}
10:27:57.205355 bus /gobi_6: Setting Powered False
199gprs:trx-umtrx12.0osmo-stp_10.42.42.5(pid=28995): Process ended prematurely: osmo-stp_10.42.42.5(pid=28995) [trial-2373↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28995)]2175iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=28995): Process ended prematurely: osmo-stp_10.42.42.5(pid=28995) [trial-2373↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28995)]
----------------------------------------------
trial-2373 gprs:trx-umtrx iperf3m4.py
----------------------------------------------
10:27:58.318652 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:58.329970 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
10:27:58.469332 tst iperf3m4.py: using LAC 27758
10:27:58.591350 tst iperf3m4.py: using RAC 218
10:27:58.714421 tst iperf3m4.py: using CellId 27758
10:27:58.841841 tst iperf3m4.py: using BVCI 27759
10:27:58.866795 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:58.878318 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:58.889302 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:58.900579 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:58.911971 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:58.923294 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:58.934762 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:58.946042 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:27:58.957371 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
10:27:59.076261 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
10:27:59.204355 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
10:27:59.325489 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
10:27:59.606608 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
10:27:59.734480 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:28:00.052778 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28981): Launched
10:28:00.315862 run iperf3-srv_10.42.42.10(pid=28982): Launched
10:28:00.502462 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
10:28:00.629385 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:28:00.974279 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28983): Launched
10:28:01.286062 run iperf3-srv_10.42.42.10(pid=28984): Launched
10:28:01.494211 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
10:28:01.636373 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:28:01.997542 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28985): Launched
10:28:02.303371 run iperf3-srv_10.42.42.10(pid=28986): Launched
10:28:02.504178 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
10:28:02.640720 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:28:02.979059 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28987): Launched
10:28:03.259783 run iperf3-srv_10.42.42.10(pid=28988): Launched
10:28:03.459815 tst iperf3m4.py:8: start network...
10:28:03.596392 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:28:04.110769 run create_hlr_db(pid=28989): Launched
10:28:04.301522 bus /gobi_6: Setting Powered False
10:28:05.322432 run create_hlr_db(pid=28989): Terminated: ok {rc=0}
10:28:05.757115 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28991): Launched
10:28:06.083706 run osmo-hlr_10.42.42.2(pid=28992): Launched
10:28:06.206920 run osmo-stp_10.42.42.5: Starting osmo-stp
10:28:06.775149 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28994): Launched
10:28:07.102985 run osmo-stp_10.42.42.5(pid=28995): Launched
10:28:07.222482 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:28:07.792768 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28997): Launched
10:28:07.909947 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:28:08.194168 run patchelf(pid=28998): Launched
10:28:08.386089 run osmo-stp_10.42.42.5(pid=28995): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28995)]
10:28:08.526573 run osmo-stp_10.42.42.5(pid=28995): stdout:
| (launched: 2022-12-10_10:28:06.941912)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:08.667307 run osmo-stp_10.42.42.5(pid=28995): stderr:
| �[0;m�[38;5;43m20221210102807133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102807133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102807133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102807134 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102807134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102807134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102807134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:08.802493 run osmo-stp_10.42.42.5(pid=28995): stdout:
| (launched: 2022-12-10_10:28:06.941912)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:08.936355 run osmo-stp_10.42.42.5(pid=28995): stderr:
| �[0;m�[38;5;43m20221210102807133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102807133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102807133 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102807134 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102807134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102807134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102807134 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:09.092180 run patchelf(pid=28998): Terminating (SIGINT)
10:28:09.276314 run patchelf(pid=28998): Terminated: ok {rc=0}
10:28:09.310027 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=28995): Process ended prematurely: osmo-stp_10.42.42.5(pid=28995) [trial-2373↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28995)] [trial-2373↪gprs:trx-umtrx↪iperf3m4.py:8]
10:28:09.319930 tst iperf3m4.py:8: Test FAILED (11.0 sec)
10:28:09.384484 run osmo-hlr_10.42.42.2(pid=28992): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:trx-umtrx↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=28992)]
10:28:09.436538 run osmo-hlr_10.42.42.2(pid=28992): stdout:
| (launched: 2022-12-10_10:28:05.923087)
10:28:09.489250 run osmo-hlr_10.42.42.2(pid=28992): stderr:
| �[0;m�[1;31m20221210102806120 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210102806120 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210102806120 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210102806120 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210102806120 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210102806120 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210102806135 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:trx-umtrx/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210102806146 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210102806146 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210102806146 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:28:09.534402 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28981): Terminating (SIGTERM)
10:28:09.576678 run iperf3-srv_10.42.42.10(pid=28982): Terminating (SIGTERM)
10:28:09.619605 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28983): Terminating (SIGTERM)
10:28:09.661997 run iperf3-srv_10.42.42.10(pid=28984): Terminating (SIGTERM)
10:28:09.704561 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28985): Terminating (SIGTERM)
10:28:09.747201 run iperf3-srv_10.42.42.10(pid=28986): Terminating (SIGTERM)
10:28:09.789611 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28987): Terminating (SIGTERM)
10:28:09.831936 run iperf3-srv_10.42.42.10(pid=28988): Terminating (SIGTERM)
10:28:09.874894 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28991): Terminating (SIGTERM)
10:28:09.917603 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28994): Terminating (SIGTERM)
10:28:09.960013 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28997): Terminating (SIGTERM)
10:28:09.970104 --- ParallelTerminationStrategy: PID 28981 died...
10:28:10.026668 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28981): Terminated: ok {rc=0}
10:28:10.036691 --- ParallelTerminationStrategy: PID 28982 died...
10:28:10.092867 run iperf3-srv_10.42.42.10(pid=28982): Terminated {rc=256}
10:28:10.102997 --- ParallelTerminationStrategy: PID 28983 died...
10:28:10.159293 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28983): Terminated: ok {rc=0}
10:28:10.169867 --- ParallelTerminationStrategy: PID 28984 died...
10:28:10.225614 run iperf3-srv_10.42.42.10(pid=28984): Terminated {rc=256}
10:28:10.235922 --- ParallelTerminationStrategy: PID 28985 died...
10:28:10.291541 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28985): Terminated: ok {rc=0}
10:28:10.301804 --- ParallelTerminationStrategy: PID 28986 died...
10:28:10.357466 run iperf3-srv_10.42.42.10(pid=28986): Terminated {rc=256}
10:28:10.367771 --- ParallelTerminationStrategy: PID 28987 died...
10:28:10.423376 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28987): Terminated: ok {rc=0}
10:28:10.433777 --- ParallelTerminationStrategy: PID 28988 died...
10:28:10.489535 run iperf3-srv_10.42.42.10(pid=28988): Terminated {rc=256}
10:28:10.499681 --- ParallelTerminationStrategy: PID 28991 died...
10:28:10.555635 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28991): Terminated: ok {rc=0}
10:28:10.565875 --- ParallelTerminationStrategy: PID 28994 died...
10:28:10.621943 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28994): Terminated: ok {rc=0}
10:28:10.632041 --- ParallelTerminationStrategy: PID 28997 died...
10:28:10.687988 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28997): Terminated: ok {rc=0}
10:28:10.775284 bus /gobi_4: Setting Powered False
10:28:11.879240 bus /gobi_4: Setting Powered False
10:28:12.990999 bus /gobi_1: Setting Powered False
10:28:14.093634 bus /gobi_1: Setting Powered False
10:28:15.205096 bus /gobi_0: Setting Powered False
10:28:16.315797 bus /gobi_0: Setting Powered False
10:28:17.427208 bus /gobi_6: Setting Powered False
174gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=29005): Process ended prematurely: osmo-stp_10.42.42.5(pid=29005) [trial-2373↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29005)]2200ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=29005): Process ended prematurely: osmo-stp_10.42.42.5(pid=29005) [trial-2373↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29005)]
----------------------------------------------
trial-2373 gprs:trx-umtrx ping.py
----------------------------------------------
10:28:18.494274 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:18.505444 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
10:28:18.640863 tst ping.py: using LAC 27759
10:28:18.759517 tst ping.py: using RAC 219
10:28:18.877051 tst ping.py: using CellId 27759
10:28:18.996124 tst ping.py: using BVCI 27760
10:28:19.019051 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:19.029595 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:19.039944 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:19.050363 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:19.060582 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:19.071009 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:19.081712 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:19.092033 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
10:28:19.366718 tst ping.py:19: start network...
10:28:19.490995 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:28:19.970395 run create_hlr_db(pid=28999): Launched
10:28:20.145175 bus /gobi_6: Setting Powered False
10:28:21.165075 run create_hlr_db(pid=28999): Terminated: ok {rc=0}
10:28:21.591021 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29001): Launched
10:28:21.911306 run osmo-hlr_10.42.42.2(pid=29002): Launched
10:28:22.027520 run osmo-stp_10.42.42.5: Starting osmo-stp
10:28:22.586843 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29004): Launched
10:28:22.906942 run osmo-stp_10.42.42.5(pid=29005): Launched
10:28:23.023604 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:28:23.583319 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29007): Launched
10:28:23.698012 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:28:23.978029 run patchelf(pid=29008): Launched
10:28:24.166758 run osmo-stp_10.42.42.5(pid=29005): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29005)]
10:28:24.304917 run osmo-stp_10.42.42.5(pid=29005): stdout:
| (launched: 2022-12-10_10:28:22.749734)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:24.443189 run osmo-stp_10.42.42.5(pid=29005): stderr:
| �[0;m�[38;5;43m20221210102822942 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102822943 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102822943 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102822943 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102822944 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102822944 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102822944 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:24.576575 run osmo-stp_10.42.42.5(pid=29005): stdout:
| (launched: 2022-12-10_10:28:22.749734)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:24.708768 run osmo-stp_10.42.42.5(pid=29005): stderr:
| �[0;m�[38;5;43m20221210102822942 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102822943 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102822943 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102822943 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102822944 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102822944 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102822944 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:24.862911 run patchelf(pid=29008): Terminating (SIGINT)
10:28:25.044323 run patchelf(pid=29008): Terminated: ok {rc=0}
10:28:25.077455 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=29005): Process ended prematurely: osmo-stp_10.42.42.5(pid=29005) [trial-2373↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29005)] [trial-2373↪gprs:trx-umtrx↪ping.py:22]
10:28:25.086787 tst ping.py:22: Test FAILED (6.6 sec)
10:28:25.150611 run osmo-hlr_10.42.42.2(pid=29002): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:trx-umtrx↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=29002)]
10:28:25.202527 run osmo-hlr_10.42.42.2(pid=29002): stdout:
| (launched: 2022-12-10_10:28:21.753804)
10:28:25.255291 run osmo-hlr_10.42.42.2(pid=29002): stderr:
| �[0;m�[1;31m20221210102821952 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210102821952 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210102821952 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210102821952 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210102821952 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210102821952 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210102821964 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:trx-umtrx/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210102821972 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210102821973 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210102821973 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:28:25.299562 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29001): Terminating (SIGTERM)
10:28:25.341883 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29004): Terminating (SIGTERM)
10:28:25.384217 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29007): Terminating (SIGTERM)
10:28:25.393970 --- ParallelTerminationStrategy: PID 29001 died...
10:28:25.450146 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29001): Terminated: ok {rc=0}
10:28:25.460609 --- ParallelTerminationStrategy: PID 29004 died...
10:28:25.516702 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29004): Terminated: ok {rc=0}
10:28:25.526779 --- ParallelTerminationStrategy: PID 29007 died...
10:28:25.583561 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29007): Terminated: ok {rc=0}
10:28:25.670159 bus /gobi_6: Setting Powered False
235gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=29015): Process ended prematurely: osmo-stp_10.42.42.5(pid=29015) [trial-2373↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29015)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=29015): Process ended prematurely: osmo-stp_10.42.42.5(pid=29015) [trial-2373↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29015)]
----------------------------------------------
trial-2373 gprs:trx-umtrx ping_idle_ping.py
----------------------------------------------
10:28:26.722640 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:26.734359 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
10:28:26.879395 tst ping_idle_ping.py: using LAC 27760
10:28:27.007045 tst ping_idle_ping.py: using RAC 220
10:28:27.127389 tst ping_idle_ping.py: using CellId 27760
10:28:27.249949 tst ping_idle_ping.py: using BVCI 27761
10:28:27.273066 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:27.283942 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:27.294656 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:27.305485 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:27.316101 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:27.326724 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:27.337519 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
10:28:27.348185 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
10:28:27.639738 tst ping_idle_ping.py:24: start network...
10:28:27.774074 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:28:28.280703 run create_hlr_db(pid=29009): Launched
10:28:28.467756 bus /gobi_6: Setting Powered False
10:28:29.488142 run create_hlr_db(pid=29009): Terminated: ok {rc=0}
10:28:29.893389 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29011): Launched
10:28:30.195068 run osmo-hlr_10.42.42.2(pid=29012): Launched
10:28:30.306131 run osmo-stp_10.42.42.5: Starting osmo-stp
10:28:30.829990 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29014): Launched
10:28:31.303813 run osmo-stp_10.42.42.5(pid=29015): Launched
10:28:31.526911 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:28:32.155491 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29017): Launched
10:28:32.274455 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:28:32.556320 run patchelf(pid=29018): Launched
10:28:32.745010 run osmo-stp_10.42.42.5(pid=29015): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29015)]
10:28:32.883052 run osmo-stp_10.42.42.5(pid=29015): stdout:
| (launched: 2022-12-10_10:28:31.016850)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:33.021680 run osmo-stp_10.42.42.5(pid=29015): stderr:
| �[0;m�[38;5;43m20221210102831247 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102831247 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102831247 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102831248 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102831248 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102831248 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102831248 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:33.155215 run osmo-stp_10.42.42.5(pid=29015): stdout:
| (launched: 2022-12-10_10:28:31.016850)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:33.287799 run osmo-stp_10.42.42.5(pid=29015): stderr:
| �[0;m�[38;5;43m20221210102831247 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210102831247 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210102831247 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210102831248 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210102831248 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210102831248 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210102831248 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:33.441725 run patchelf(pid=29018): Terminating (SIGINT)
10:28:33.622010 run patchelf(pid=29018): Terminated: ok {rc=0}
10:28:33.654492 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=29015): Process ended prematurely: osmo-stp_10.42.42.5(pid=29015) [trial-2373↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29015)] [trial-2373↪gprs:trx-umtrx↪ping_idle_ping.py:27]
10:28:33.663778 tst ping_idle_ping.py:27: Test FAILED (6.9 sec)
10:28:33.728119 run osmo-hlr_10.42.42.2(pid=29012): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:trx-umtrx↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=29012)]
10:28:33.780328 run osmo-hlr_10.42.42.2(pid=29012): stdout:
| (launched: 2022-12-10_10:28:30.046068)
10:28:33.833542 run osmo-hlr_10.42.42.2(pid=29012): stderr:
| �[0;m�[1;31m20221210102830224 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210102830224 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210102830224 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210102830224 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210102830224 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210102830224 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210102830242 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:trx-umtrx/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210102830251 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210102830251 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210102830251 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:28:33.878566 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29011): Terminating (SIGTERM)
10:28:33.921742 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29014): Terminating (SIGTERM)
10:28:33.964563 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29017): Terminating (SIGTERM)
10:28:33.974588 --- ParallelTerminationStrategy: PID 29011 died...
10:28:34.031835 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29011): Terminated: ok {rc=0}
10:28:34.042479 --- ParallelTerminationStrategy: PID 29014 died...
10:28:34.099177 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29014): Terminated: ok {rc=0}
10:28:34.109878 --- ParallelTerminationStrategy: PID 29017 died...
10:28:34.166893 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29017): Terminated: ok {rc=0}
10:28:34.254593 bus /gobi_6: Setting Powered False
5gprs:trx-umtrx00