50gprs:sysmo10.0osmo-stp_10.42.42.5(pid=2288): Process ended prematurely: osmo-stp_10.42.42.5(pid=2288) [trial-2363↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2288)]2314cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2288): Process ended prematurely: osmo-stp_10.42.42.5(pid=2288) [trial-2363↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2288)]
----------------------------------------------
trial-2363 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
18:43:06.931348 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:06.942373 tst gprs:sysmo: Using 1 x bts (candidates: 1)
18:43:07.073100 tst cs_paging_gprs_active.py: using LAC 22967
18:43:07.198805 tst cs_paging_gprs_active.py: using RAC 17
18:43:07.326402 tst cs_paging_gprs_active.py: using CellId 22967
18:43:07.453834 tst cs_paging_gprs_active.py: using BVCI 22968
18:43:07.478424 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:07.489290 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:07.499975 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:07.510918 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:07.521445 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:07.532120 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:07.542843 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:07.553990 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:07.565026 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:43:07.678726 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:43:07.958375 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
18:43:08.084327 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:43:08.394735 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2278): Launched
18:43:08.652940 run iperf3-srv_10.42.42.10(pid=2279): Launched
18:43:08.837078 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
18:43:08.961955 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:43:09.271395 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2280): Launched
18:43:09.527392 run iperf3-srv_10.42.42.10(pid=2281): Launched
18:43:09.709946 tst cs_paging_gprs_active.py:38: start network...
18:43:09.834441 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:43:10.313668 run create_hlr_db(pid=2282): Launched
18:43:10.486016 bus /sierra_2: Setting Powered False
18:43:11.507713 run create_hlr_db(pid=2282): Terminated: ok {rc=0}
18:43:11.934801 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2284): Launched
18:43:12.257141 run osmo-hlr_10.42.42.2(pid=2285): Launched
18:43:12.381195 run osmo-stp_10.42.42.5: Starting osmo-stp
18:43:12.939004 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2287): Launched
18:43:13.261581 run osmo-stp_10.42.42.5(pid=2288): Launched
18:43:13.383327 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:43:14.935322 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2290): Launched
18:43:15.050054 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:43:15.330621 run patchelf(pid=2291): Launched
18:43:15.518313 run osmo-stp_10.42.42.5(pid=2288): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2288)]
18:43:15.655311 run osmo-stp_10.42.42.5(pid=2288): stdout:
| (launched: 2022-12-09_18:43:13.102585)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:43:15.793179 run osmo-stp_10.42.42.5(pid=2288): stderr:
| �[0;m�[38;5;43m20221209184313294 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184313294 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184313294 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184313295 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184313297 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184313297 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184313297 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:43:15.953565 run osmo-stp_10.42.42.5(pid=2288): stdout:
| (launched: 2022-12-09_18:43:13.102585)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:43:16.088323 run osmo-stp_10.42.42.5(pid=2288): stderr:
| �[0;m�[38;5;43m20221209184313294 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184313294 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184313294 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184313295 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184313297 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184313297 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184313297 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:43:16.244512 run patchelf(pid=2291): Terminating (SIGINT)
18:43:16.429446 run patchelf(pid=2291): Terminated: ok {rc=0}
18:43:16.463472 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=2288): Process ended prematurely: osmo-stp_10.42.42.5(pid=2288) [trial-2363↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2288)] [trial-2363↪gprs:sysmo↪cs_paging_gprs_active.py:38]
18:43:16.473741 tst cs_paging_gprs_active.py:38: Test FAILED (9.6 sec)
18:43:16.537255 run osmo-hlr_10.42.42.2(pid=2285): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:sysmo↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2285)]
18:43:16.588670 run osmo-hlr_10.42.42.2(pid=2285): stdout:
| (launched: 2022-12-09_18:43:12.098876)
18:43:16.640577 run osmo-hlr_10.42.42.2(pid=2285): stderr:
| �[0;m�[1;31m20221209184312297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209184312297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209184312297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209184312297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209184312297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209184312297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209184312308 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:sysmo/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209184312316 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209184312317 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209184312317 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:43:16.685104 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2278): Terminating (SIGTERM)
18:43:16.727549 run iperf3-srv_10.42.42.10(pid=2279): Terminating (SIGTERM)
18:43:16.772338 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2280): Terminating (SIGTERM)
18:43:16.815887 run iperf3-srv_10.42.42.10(pid=2281): Terminating (SIGTERM)
18:43:16.861111 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2284): Terminating (SIGTERM)
18:43:16.905054 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2287): Terminating (SIGTERM)
18:43:16.947566 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2290): Terminating (SIGTERM)
18:43:16.957127 --- ParallelTerminationStrategy: PID 2278 died...
18:43:17.013331 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2278): Terminated: ok {rc=0}
18:43:17.023920 --- ParallelTerminationStrategy: PID 2279 died...
18:43:17.079890 run iperf3-srv_10.42.42.10(pid=2279): Terminated {rc=256}
18:43:17.090121 --- ParallelTerminationStrategy: PID 2280 died...
18:43:17.146016 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2280): Terminated: ok {rc=0}
18:43:17.156488 --- ParallelTerminationStrategy: PID 2281 died...
18:43:17.212357 run iperf3-srv_10.42.42.10(pid=2281): Terminated {rc=256}
18:43:17.222313 --- ParallelTerminationStrategy: PID 2284 died...
18:43:17.277009 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2284): Terminated: ok {rc=0}
18:43:17.286770 --- ParallelTerminationStrategy: PID 2287 died...
18:43:17.342056 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2287): Terminated: ok {rc=0}
18:43:17.352068 --- ParallelTerminationStrategy: PID 2290 died...
18:43:17.407503 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2290): Terminated: ok {rc=0}
18:43:17.493695 bus /gobi_0: Setting Powered False
18:43:18.598412 bus /gobi_0: Setting Powered False
18:43:19.705174 bus /gobi_6: Setting Powered False
50gprs:sysmo8.0osmo-stp_10.42.42.5(pid=2300): Process ended prematurely: osmo-stp_10.42.42.5(pid=2300) [trial-2363↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2300)]2314iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2300): Process ended prematurely: osmo-stp_10.42.42.5(pid=2300) [trial-2363↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2300)]
----------------------------------------------
trial-2363 gprs:sysmo iperf3.py
----------------------------------------------
18:43:20.819401 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:20.830629 tst gprs:sysmo: Using 1 x bts (candidates: 1)
18:43:20.960135 tst iperf3.py: using LAC 22968
18:43:21.087589 tst iperf3.py: using RAC 18
18:43:21.212490 tst iperf3.py: using CellId 22968
18:43:21.337442 tst iperf3.py: using BVCI 22969
18:43:21.361949 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:21.372652 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:21.383394 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:21.394259 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:21.405012 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:21.415775 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:21.426686 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:21.437555 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:21.448479 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:43:21.740777 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
18:43:21.875938 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:43:22.207358 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2292): Launched
18:43:22.482003 run iperf3-srv_10.42.42.10(pid=2293): Launched
18:43:22.678507 tst iperf3.py:8: start network...
18:43:22.813131 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:43:23.315110 run create_hlr_db(pid=2294): Launched
18:43:23.500434 bus /gobi_6: Setting Powered False
18:43:24.520866 run create_hlr_db(pid=2294): Terminated: ok {rc=0}
18:43:24.921032 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2296): Launched
18:43:25.219303 run osmo-hlr_10.42.42.2(pid=2297): Launched
18:43:25.335841 run osmo-stp_10.42.42.5: Starting osmo-stp
18:43:25.861104 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2299): Launched
18:43:26.158953 run osmo-stp_10.42.42.5(pid=2300): Launched
18:43:26.267809 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:43:26.789048 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2302): Launched
18:43:26.896094 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:43:27.156254 run patchelf(pid=2303): Launched
18:43:27.330351 run osmo-stp_10.42.42.5(pid=2300): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2300)]
18:43:27.457844 run osmo-stp_10.42.42.5(pid=2300): stdout:
| (launched: 2022-12-09_18:43:26.012034)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:43:27.585646 run osmo-stp_10.42.42.5(pid=2300): stderr:
| �[0;m�[38;5;43m20221209184326183 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184326183 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184326183 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184326184 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184326184 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184326184 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184326185 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:43:27.708851 run osmo-stp_10.42.42.5(pid=2300): stdout:
| (launched: 2022-12-09_18:43:26.012034)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:43:27.831148 run osmo-stp_10.42.42.5(pid=2300): stderr:
| �[0;m�[38;5;43m20221209184326183 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184326183 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184326183 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184326184 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184326184 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184326184 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184326185 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:43:27.973624 run patchelf(pid=2303): Terminating (SIGINT)
18:43:28.141231 run patchelf(pid=2303): Terminated: ok {rc=0}
18:43:28.171923 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=2300): Process ended prematurely: osmo-stp_10.42.42.5(pid=2300) [trial-2363↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2300)] [trial-2363↪gprs:sysmo↪iperf3.py:8]
18:43:28.180802 tst iperf3.py:8: Test FAILED (7.4 sec)
18:43:28.239408 run osmo-hlr_10.42.42.2(pid=2297): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:sysmo↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2297)]
18:43:28.287265 run osmo-hlr_10.42.42.2(pid=2297): stdout:
| (launched: 2022-12-09_18:43:25.072560)
18:43:28.335945 run osmo-hlr_10.42.42.2(pid=2297): stderr:
| �[0;m�[1;31m20221209184325252 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209184325252 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209184325252 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209184325252 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209184325252 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209184325252 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209184325265 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:sysmo/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209184325273 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209184325273 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209184325274 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:43:28.377180 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2292): Terminating (SIGTERM)
18:43:28.415985 run iperf3-srv_10.42.42.10(pid=2293): Terminating (SIGTERM)
18:43:28.455073 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2296): Terminating (SIGTERM)
18:43:28.493971 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2299): Terminating (SIGTERM)
18:43:28.532698 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2302): Terminating (SIGTERM)
18:43:28.541777 --- ParallelTerminationStrategy: PID 2292 died...
18:43:28.594286 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2292): Terminated: ok {rc=0}
18:43:28.603832 --- ParallelTerminationStrategy: PID 2293 died...
18:43:28.655743 run iperf3-srv_10.42.42.10(pid=2293): Terminated {rc=256}
18:43:28.665192 --- ParallelTerminationStrategy: PID 2296 died...
18:43:28.716832 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2296): Terminated: ok {rc=0}
18:43:28.726242 --- ParallelTerminationStrategy: PID 2299 died...
18:43:28.778340 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2299): Terminated: ok {rc=0}
18:43:28.787757 --- ParallelTerminationStrategy: PID 2302 died...
18:43:28.839633 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2302): Terminated: ok {rc=0}
18:43:28.926627 bus /gobi_6: Setting Powered False
51gprs:sysmo11.0osmo-stp_10.42.42.5(pid=2318): Process ended prematurely: osmo-stp_10.42.42.5(pid=2318) [trial-2363↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2318)]2313iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2318): Process ended prematurely: osmo-stp_10.42.42.5(pid=2318) [trial-2363↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2318)]
----------------------------------------------
trial-2363 gprs:sysmo iperf3m4.py
----------------------------------------------
18:43:30.035795 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:30.047529 tst gprs:sysmo: Using 1 x bts (candidates: 1)
18:43:30.169092 tst iperf3m4.py: using LAC 22969
18:43:30.295587 tst iperf3m4.py: using RAC 19
18:43:30.421386 tst iperf3m4.py: using CellId 22969
18:43:30.553052 tst iperf3m4.py: using BVCI 22970
18:43:30.577788 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:30.588848 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:30.599778 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:30.611495 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:30.623032 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:30.634743 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:30.646523 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:30.658513 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:30.670085 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:43:30.791799 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:43:30.924935 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:43:31.049689 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:43:31.362273 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
18:43:31.491306 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:43:31.842688 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2304): Launched
18:43:32.130521 run iperf3-srv_10.42.42.10(pid=2305): Launched
18:43:32.334049 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
18:43:32.469661 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:43:32.802262 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2306): Launched
18:43:33.077274 run iperf3-srv_10.42.42.10(pid=2307): Launched
18:43:33.274500 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
18:43:33.408856 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:43:33.740156 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2308): Launched
18:43:34.015246 run iperf3-srv_10.42.42.10(pid=2309): Launched
18:43:34.212400 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
18:43:34.346608 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:43:34.677913 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2310): Launched
18:43:34.963418 run iperf3-srv_10.42.42.10(pid=2311): Launched
18:43:35.145328 tst iperf3m4.py:8: start network...
18:43:35.269967 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:43:35.748483 run create_hlr_db(pid=2312): Launched
18:43:35.921289 bus /gobi_6: Setting Powered False
18:43:36.942889 run create_hlr_db(pid=2312): Terminated: ok {rc=0}
18:43:37.367951 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2314): Launched
18:43:37.688890 run osmo-hlr_10.42.42.2(pid=2315): Launched
18:43:37.808590 run osmo-stp_10.42.42.5: Starting osmo-stp
18:43:38.366688 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2317): Launched
18:43:38.687091 run osmo-stp_10.42.42.5(pid=2318): Launched
18:43:38.803936 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:43:39.361386 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2320): Launched
18:43:39.476217 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:43:39.754771 run patchelf(pid=2321): Launched
18:43:39.942261 run osmo-stp_10.42.42.5(pid=2318): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2318)]
18:43:40.079331 run osmo-stp_10.42.42.5(pid=2318): stdout:
| (launched: 2022-12-09_18:43:38.529732)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:43:40.217128 run osmo-stp_10.42.42.5(pid=2318): stderr:
| �[0;m�[38;5;43m20221209184338720 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184338720 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184338720 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184338721 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184338721 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184338721 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184338721 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:43:40.349613 run osmo-stp_10.42.42.5(pid=2318): stdout:
| (launched: 2022-12-09_18:43:38.529732)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:43:40.480725 run osmo-stp_10.42.42.5(pid=2318): stderr:
| �[0;m�[38;5;43m20221209184338720 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184338720 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184338720 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184338721 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184338721 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184338721 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184338721 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:43:40.633434 run patchelf(pid=2321): Terminating (SIGINT)
18:43:40.813062 run patchelf(pid=2321): Terminated: ok {rc=0}
18:43:40.845943 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=2318): Process ended prematurely: osmo-stp_10.42.42.5(pid=2318) [trial-2363↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2318)] [trial-2363↪gprs:sysmo↪iperf3m4.py:8]
18:43:40.855603 tst iperf3m4.py:8: Test FAILED (10.9 sec)
18:43:40.918100 run osmo-hlr_10.42.42.2(pid=2315): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:sysmo↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2315)]
18:43:40.968598 run osmo-hlr_10.42.42.2(pid=2315): stdout:
| (launched: 2022-12-09_18:43:37.531814)
18:43:41.019736 run osmo-hlr_10.42.42.2(pid=2315): stderr:
| �[0;m�[1;31m20221209184337730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209184337730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209184337730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209184337730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209184337730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209184337730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209184337743 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:sysmo/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209184337753 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209184337754 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209184337754 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:43:41.063046 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2304): Terminating (SIGTERM)
18:43:41.104085 run iperf3-srv_10.42.42.10(pid=2305): Terminating (SIGTERM)
18:43:41.144937 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2306): Terminating (SIGTERM)
18:43:41.185874 run iperf3-srv_10.42.42.10(pid=2307): Terminating (SIGTERM)
18:43:41.226782 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2308): Terminating (SIGTERM)
18:43:41.267409 run iperf3-srv_10.42.42.10(pid=2309): Terminating (SIGTERM)
18:43:41.308731 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2310): Terminating (SIGTERM)
18:43:41.349625 run iperf3-srv_10.42.42.10(pid=2311): Terminating (SIGTERM)
18:43:41.390409 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2314): Terminating (SIGTERM)
18:43:41.431136 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2317): Terminating (SIGTERM)
18:43:41.471964 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2320): Terminating (SIGTERM)
18:43:41.481753 --- ParallelTerminationStrategy: PID 2304 died...
18:43:41.536903 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2304): Terminated: ok {rc=0}
18:43:41.547120 --- ParallelTerminationStrategy: PID 2305 died...
18:43:41.601357 run iperf3-srv_10.42.42.10(pid=2305): Terminated {rc=256}
18:43:41.611313 --- ParallelTerminationStrategy: PID 2306 died...
18:43:41.666262 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2306): Terminated: ok {rc=0}
18:43:41.676115 --- ParallelTerminationStrategy: PID 2307 died...
18:43:41.731088 run iperf3-srv_10.42.42.10(pid=2307): Terminated {rc=256}
18:43:41.741058 --- ParallelTerminationStrategy: PID 2308 died...
18:43:41.796419 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2308): Terminated: ok {rc=0}
18:43:41.806348 --- ParallelTerminationStrategy: PID 2309 died...
18:43:41.861666 run iperf3-srv_10.42.42.10(pid=2309): Terminated {rc=256}
18:43:41.871439 --- ParallelTerminationStrategy: PID 2310 died...
18:43:41.926381 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2310): Terminated: ok {rc=0}
18:43:41.936463 --- ParallelTerminationStrategy: PID 2311 died...
18:43:41.991099 run iperf3-srv_10.42.42.10(pid=2311): Terminated {rc=256}
18:43:42.001366 --- ParallelTerminationStrategy: PID 2314 died...
18:43:42.056050 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2314): Terminated: ok {rc=0}
18:43:42.066210 --- ParallelTerminationStrategy: PID 2317 died...
18:43:42.121084 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2317): Terminated: ok {rc=0}
18:43:42.131203 --- ParallelTerminationStrategy: PID 2320 died...
18:43:42.186888 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2320): Terminated: ok {rc=0}
18:43:42.273308 bus /gobi_4: Setting Powered False
18:43:43.378758 bus /gobi_4: Setting Powered False
18:43:44.485288 bus /gobi_1: Setting Powered False
18:43:45.589550 bus /gobi_1: Setting Powered False
18:43:46.712930 bus /gobi_0: Setting Powered False
18:43:47.816014 bus /gobi_0: Setting Powered False
18:43:48.926515 bus /gobi_6: Setting Powered False
33gprs:sysmo7.0osmo-stp_10.42.42.5(pid=2328): Process ended prematurely: osmo-stp_10.42.42.5(pid=2328) [trial-2363↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2328)]2331ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2328): Process ended prematurely: osmo-stp_10.42.42.5(pid=2328) [trial-2363↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2328)]
----------------------------------------------
trial-2363 gprs:sysmo ping.py
----------------------------------------------
18:43:49.992604 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:50.003410 tst gprs:sysmo: Using 1 x bts (candidates: 1)
18:43:50.125588 tst ping.py: using LAC 22970
18:43:50.242143 tst ping.py: using RAC 20
18:43:50.361536 tst ping.py: using CellId 22970
18:43:50.478548 tst ping.py: using BVCI 22971
18:43:50.501027 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:50.511300 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:50.521573 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:50.531662 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:50.541753 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:50.551732 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:50.561970 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:50.572041 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:43:50.860039 tst ping.py:19: start network...
18:43:50.992472 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:43:51.489990 run create_hlr_db(pid=2322): Launched
18:43:51.673757 bus /gobi_6: Setting Powered False
18:43:52.694542 run create_hlr_db(pid=2322): Terminated: ok {rc=0}
18:43:53.114309 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2324): Launched
18:43:53.430221 run osmo-hlr_10.42.42.2(pid=2325): Launched
18:43:53.544138 run osmo-stp_10.42.42.5: Starting osmo-stp
18:43:54.093304 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2327): Launched
18:43:54.409302 run osmo-stp_10.42.42.5(pid=2328): Launched
18:43:54.523801 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:43:55.073911 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2330): Launched
18:43:55.186526 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:43:55.454974 run patchelf(pid=2331): Launched
18:43:55.627264 run osmo-stp_10.42.42.5(pid=2328): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2328)]
18:43:55.752889 run osmo-stp_10.42.42.5(pid=2328): stdout:
| (launched: 2022-12-09_18:43:54.253268)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:43:55.878829 run osmo-stp_10.42.42.5(pid=2328): stderr:
| �[0;m�[38;5;43m20221209184354445 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184354446 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184354446 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184354446 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184354447 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184354447 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184354447 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:43:56.000263 run osmo-stp_10.42.42.5(pid=2328): stdout:
| (launched: 2022-12-09_18:43:54.253268)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:43:56.120789 run osmo-stp_10.42.42.5(pid=2328): stderr:
| �[0;m�[38;5;43m20221209184354445 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184354446 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184354446 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184354446 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184354447 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184354447 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184354447 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:43:56.260608 run patchelf(pid=2331): Terminating (SIGINT)
18:43:56.425504 run patchelf(pid=2331): Terminated: ok {rc=0}
18:43:56.455257 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=2328): Process ended prematurely: osmo-stp_10.42.42.5(pid=2328) [trial-2363↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2328)] [trial-2363↪gprs:sysmo↪ping.py:22]
18:43:56.463793 tst ping.py:22: Test FAILED (6.5 sec)
18:43:56.522763 run osmo-hlr_10.42.42.2(pid=2325): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:sysmo↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2325)]
18:43:56.570765 run osmo-hlr_10.42.42.2(pid=2325): stdout:
| (launched: 2022-12-09_18:43:53.274689)
18:43:56.619340 run osmo-hlr_10.42.42.2(pid=2325): stderr:
| �[0;m�[1;31m20221209184353474 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209184353474 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209184353475 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209184353475 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209184353475 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209184353475 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209184353682 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:sysmo/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209184353693 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209184353693 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209184353693 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:43:56.660886 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2324): Terminating (SIGTERM)
18:43:56.700327 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2327): Terminating (SIGTERM)
18:43:56.739834 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2330): Terminating (SIGTERM)
18:43:56.749173 --- ParallelTerminationStrategy: PID 2324 died...
18:43:56.802141 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2324): Terminated: ok {rc=0}
18:43:56.811686 --- ParallelTerminationStrategy: PID 2327 died...
18:43:56.863982 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2327): Terminated: ok {rc=0}
18:43:56.873541 --- ParallelTerminationStrategy: PID 2330 died...
18:43:56.925480 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2330): Terminated: ok {rc=0}
18:43:57.007696 bus /gobi_6: Setting Powered False
33gprs:sysmo7.0osmo-stp_10.42.42.5(pid=2338): Process ended prematurely: osmo-stp_10.42.42.5(pid=2338) [trial-2363↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2338)]2331ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2338): Process ended prematurely: osmo-stp_10.42.42.5(pid=2338) [trial-2363↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2338)]
----------------------------------------------
trial-2363 gprs:sysmo ping_idle_ping.py
----------------------------------------------
18:43:58.056518 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:58.067338 tst gprs:sysmo: Using 1 x bts (candidates: 1)
18:43:58.197249 tst ping_idle_ping.py: using LAC 22971
18:43:58.321177 tst ping_idle_ping.py: using RAC 21
18:43:58.445978 tst ping_idle_ping.py: using CellId 22971
18:43:58.570459 tst ping_idle_ping.py: using BVCI 22972
18:43:58.594898 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:58.605903 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:58.616681 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:58.627680 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:58.638763 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:58.649735 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:58.660745 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:43:58.671971 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:43:58.950859 tst ping_idle_ping.py:24: start network...
18:43:59.074191 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:43:59.544931 run create_hlr_db(pid=2332): Launched
18:43:59.715186 bus /gobi_6: Setting Powered False
18:44:00.735274 run create_hlr_db(pid=2332): Terminated: ok {rc=0}
18:44:01.166020 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2334): Launched
18:44:01.568677 run osmo-hlr_10.42.42.2(pid=2335): Launched
18:44:01.693782 run osmo-stp_10.42.42.5: Starting osmo-stp
18:44:02.368270 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2337): Launched
18:44:02.661974 run osmo-stp_10.42.42.5(pid=2338): Launched
18:44:02.768666 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:44:03.282694 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2340): Launched
18:44:03.387132 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:44:03.643706 run patchelf(pid=2341): Launched
18:44:03.815524 run osmo-stp_10.42.42.5(pid=2338): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2338)]
18:44:03.941474 run osmo-stp_10.42.42.5(pid=2338): stdout:
| (launched: 2022-12-09_18:44:02.517549)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:44:04.067485 run osmo-stp_10.42.42.5(pid=2338): stderr:
| �[0;m�[38;5;43m20221209184402688 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184402689 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184402689 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184402689 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184402690 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184402690 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184402690 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:44:04.189215 run osmo-stp_10.42.42.5(pid=2338): stdout:
| (launched: 2022-12-09_18:44:02.517549)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:44:04.309921 run osmo-stp_10.42.42.5(pid=2338): stderr:
| �[0;m�[38;5;43m20221209184402688 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209184402689 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209184402689 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209184402689 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209184402690 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209184402690 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209184402690 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:44:04.451091 run patchelf(pid=2341): Terminating (SIGINT)
18:44:04.615270 run patchelf(pid=2341): Terminated: ok {rc=0}
18:44:04.645256 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=2338): Process ended prematurely: osmo-stp_10.42.42.5(pid=2338) [trial-2363↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2338)] [trial-2363↪gprs:sysmo↪ping_idle_ping.py:27]
18:44:04.653749 tst ping_idle_ping.py:27: Test FAILED (6.6 sec)
18:44:04.711568 run osmo-hlr_10.42.42.2(pid=2335): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:sysmo↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2335)]
18:44:04.758916 run osmo-hlr_10.42.42.2(pid=2335): stdout:
| (launched: 2022-12-09_18:44:01.320476)
18:44:04.807098 run osmo-hlr_10.42.42.2(pid=2335): stderr:
| �[0;m�[1;31m20221209184401678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209184401678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209184401678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209184401678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209184401678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209184401678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209184401710 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:sysmo/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209184401718 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209184401718 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209184401718 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:44:04.848773 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2334): Terminating (SIGTERM)
18:44:04.887550 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2337): Terminating (SIGTERM)
18:44:04.926444 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2340): Terminating (SIGTERM)
18:44:04.935443 --- ParallelTerminationStrategy: PID 2334 died...
18:44:04.987445 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2334): Terminated: ok {rc=0}
18:44:04.996794 --- ParallelTerminationStrategy: PID 2337 died...
18:44:05.048561 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2337): Terminated: ok {rc=0}
18:44:05.058078 --- ParallelTerminationStrategy: PID 2340 died...
18:44:05.109799 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2340): Terminated: ok {rc=0}
18:44:05.196941 bus /gobi_6: Setting Powered False
5gprs:sysmo00