58gprs:sysmo11.0osmo-stp_10.42.42.5(pid=8358): Process ended prematurely: osmo-stp_10.42.42.5(pid=8358) [trial-2371↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8358)]2314cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8358): Process ended prematurely: osmo-stp_10.42.42.5(pid=8358) [trial-2371↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8358)]
----------------------------------------------
trial-2371 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
07:00:10.458230 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:10.468699 tst gprs:sysmo: Using 1 x bts (candidates: 1)
07:00:10.593673 tst cs_paging_gprs_active.py: using LAC 26655
07:00:10.714801 tst cs_paging_gprs_active.py: using RAC 135
07:00:10.837722 tst cs_paging_gprs_active.py: using CellId 26655
07:00:10.958246 tst cs_paging_gprs_active.py: using BVCI 26656
07:00:10.981907 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:10.992847 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:11.003363 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:11.014093 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:11.024627 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:11.035292 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:11.046067 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:11.056633 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:11.067509 tst gprs:sysmo: Using 1 x modem (candidates: 4)
07:00:11.184659 tst gprs:sysmo: Using 1 x modem (candidates: 4)
07:00:11.488485 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
07:00:11.629618 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:00:11.976226 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8348): Launched
07:00:12.266982 run iperf3-srv_10.42.42.10(pid=8349): Launched
07:00:12.471991 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
07:00:12.611173 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:00:12.958460 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8350): Launched
07:00:13.247432 run iperf3-srv_10.42.42.10(pid=8351): Launched
07:00:13.451709 tst cs_paging_gprs_active.py:38: start network...
07:00:13.590273 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:00:14.126002 run create_hlr_db(pid=8352): Launched
07:00:14.319597 bus /sierra_2: Setting Powered False
07:00:15.340339 run create_hlr_db(pid=8352): Terminated: ok {rc=0}
07:00:15.768504 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8354): Launched
07:00:16.253742 run osmo-hlr_10.42.42.2(pid=8355): Launched
07:00:16.477712 run osmo-stp_10.42.42.5: Starting osmo-stp
07:00:17.066495 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8357): Launched
07:00:17.422907 run osmo-stp_10.42.42.5(pid=8358): Launched
07:00:17.550028 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:00:19.137231 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8360): Launched
07:00:19.256268 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:00:19.550153 run patchelf(pid=8361): Launched
07:00:19.743987 run osmo-stp_10.42.42.5(pid=8358): ERR: Terminated: ERROR {rc=1} [trial-2371↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8358)]
07:00:19.885850 run osmo-stp_10.42.42.5(pid=8358): stdout:
| (launched: 2022-12-10_07:00:17.238175)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:00:20.028510 run osmo-stp_10.42.42.5(pid=8358): stderr:
| �[0;m�[38;5;43m20221210070017457 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070017457 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070017457 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070017458 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070017458 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070017458 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070017458 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:00:20.165940 run osmo-stp_10.42.42.5(pid=8358): stdout:
| (launched: 2022-12-10_07:00:17.238175)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:00:20.301757 run osmo-stp_10.42.42.5(pid=8358): stderr:
| �[0;m�[38;5;43m20221210070017457 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070017457 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070017457 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070017458 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070017458 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070017458 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070017458 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:00:20.460884 run patchelf(pid=8361): Terminating (SIGINT)
07:00:20.648326 run patchelf(pid=8361): Terminated: ok {rc=0}
07:00:20.682572 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=8358): Process ended prematurely: osmo-stp_10.42.42.5(pid=8358) [trial-2371↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8358)] [trial-2371↪gprs:sysmo↪cs_paging_gprs_active.py:38]
07:00:20.692223 tst cs_paging_gprs_active.py:38: Test FAILED (10.3 sec)
07:00:20.755366 run osmo-hlr_10.42.42.2(pid=8355): ERR: Terminated: ERROR {rc=237} [trial-2371↪gprs:sysmo↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8355)]
07:00:20.807004 run osmo-hlr_10.42.42.2(pid=8355): stdout:
| (launched: 2022-12-10_07:00:15.950163)
07:00:20.859244 run osmo-hlr_10.42.42.2(pid=8355): stderr:
| �[0;m�[1;31m20221210070016206 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210070016206 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210070016206 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210070016206 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210070016206 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210070016206 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210070016222 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2371/run.2022-12-10_06-54-20/gprs:sysmo/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210070016236 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210070016238 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210070016238 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:00:20.904041 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8348): Terminating (SIGTERM)
07:00:20.946774 run iperf3-srv_10.42.42.10(pid=8349): Terminating (SIGTERM)
07:00:20.989728 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8350): Terminating (SIGTERM)
07:00:21.032520 run iperf3-srv_10.42.42.10(pid=8351): Terminating (SIGTERM)
07:00:21.075223 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8354): Terminating (SIGTERM)
07:00:21.117792 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8357): Terminating (SIGTERM)
07:00:21.160235 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8360): Terminating (SIGTERM)
07:00:21.170061 --- ParallelTerminationStrategy: PID 8348 died...
07:00:21.227545 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8348): Terminated: ok {rc=0}
07:00:21.237771 --- ParallelTerminationStrategy: PID 8349 died...
07:00:21.294015 run iperf3-srv_10.42.42.10(pid=8349): Terminated {rc=256}
07:00:21.304060 --- ParallelTerminationStrategy: PID 8350 died...
07:00:21.360529 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8350): Terminated: ok {rc=0}
07:00:21.370663 --- ParallelTerminationStrategy: PID 8351 died...
07:00:21.427101 run iperf3-srv_10.42.42.10(pid=8351): Terminated {rc=256}
07:00:21.437284 --- ParallelTerminationStrategy: PID 8354 died...
07:00:21.493768 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8354): Terminated: ok {rc=0}
07:00:21.503909 --- ParallelTerminationStrategy: PID 8357 died...
07:00:21.560646 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8357): Terminated: ok {rc=0}
07:00:21.570808 --- ParallelTerminationStrategy: PID 8360 died...
07:00:21.627454 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8360): Terminated: ok {rc=0}
07:00:21.716734 bus /gobi_0: Setting Powered False
07:00:22.822619 bus /gobi_0: Setting Powered False
07:00:23.934981 bus /gobi_6: Setting Powered False
58gprs:sysmo8.0osmo-stp_10.42.42.5(pid=8370): Process ended prematurely: osmo-stp_10.42.42.5(pid=8370) [trial-2371↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8370)]2314iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8370): Process ended prematurely: osmo-stp_10.42.42.5(pid=8370) [trial-2371↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8370)]
----------------------------------------------
trial-2371 gprs:sysmo iperf3.py
----------------------------------------------
07:00:25.053660 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:25.065340 tst gprs:sysmo: Using 1 x bts (candidates: 1)
07:00:25.197544 tst iperf3.py: using LAC 26656
07:00:25.329560 tst iperf3.py: using RAC 136
07:00:25.466624 tst iperf3.py: using CellId 26656
07:00:25.594560 tst iperf3.py: using BVCI 26657
07:00:25.619990 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:25.631098 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:25.642060 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:25.652934 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:25.663772 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:25.674588 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:25.685550 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:25.696425 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:25.707956 tst gprs:sysmo: Using 1 x modem (candidates: 4)
07:00:26.008000 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
07:00:26.147408 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:00:26.493998 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8362): Launched
07:00:26.782141 run iperf3-srv_10.42.42.10(pid=8363): Launched
07:00:26.986315 tst iperf3.py:8: start network...
07:00:27.125658 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:00:27.658907 run create_hlr_db(pid=8364): Launched
07:00:27.851188 bus /gobi_6: Setting Powered False
07:00:28.871491 run create_hlr_db(pid=8364): Terminated: ok {rc=0}
07:00:29.289858 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8366): Launched
07:00:29.603870 run osmo-hlr_10.42.42.2(pid=8367): Launched
07:00:29.725031 run osmo-stp_10.42.42.5: Starting osmo-stp
07:00:30.274115 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8369): Launched
07:00:30.588902 run osmo-stp_10.42.42.5(pid=8370): Launched
07:00:30.701931 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:00:31.308592 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8372): Launched
07:00:31.431958 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:00:31.741340 run patchelf(pid=8373): Launched
07:00:31.941239 run osmo-stp_10.42.42.5(pid=8370): ERR: Terminated: ERROR {rc=1} [trial-2371↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8370)]
07:00:32.088412 run osmo-stp_10.42.42.5(pid=8370): stdout:
| (launched: 2022-12-10_07:00:30.431879)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:00:32.232428 run osmo-stp_10.42.42.5(pid=8370): stderr:
| �[0;m�[38;5;43m20221210070030605 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070030605 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070030606 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070030606 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070030606 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070030607 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070030607 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:00:32.370238 run osmo-stp_10.42.42.5(pid=8370): stdout:
| (launched: 2022-12-10_07:00:30.431879)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:00:32.506173 run osmo-stp_10.42.42.5(pid=8370): stderr:
| �[0;m�[38;5;43m20221210070030605 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070030605 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070030606 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070030606 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070030606 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070030607 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070030607 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:00:32.665520 run patchelf(pid=8373): Terminating (SIGINT)
07:00:32.853543 run patchelf(pid=8373): Terminated: ok {rc=0}
07:00:32.887869 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=8370): Process ended prematurely: osmo-stp_10.42.42.5(pid=8370) [trial-2371↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8370)] [trial-2371↪gprs:sysmo↪iperf3.py:8]
07:00:32.897669 tst iperf3.py:8: Test FAILED (7.9 sec)
07:00:32.961894 run osmo-hlr_10.42.42.2(pid=8367): ERR: Terminated: ERROR {rc=237} [trial-2371↪gprs:sysmo↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8367)]
07:00:33.014362 run osmo-hlr_10.42.42.2(pid=8367): stdout:
| (launched: 2022-12-10_07:00:29.447678)
07:00:33.067957 run osmo-hlr_10.42.42.2(pid=8367): stderr:
| �[0;m�[1;31m20221210070029628 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210070029628 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210070029628 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210070029628 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210070029629 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210070029629 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210070029640 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2371/run.2022-12-10_06-54-20/gprs:sysmo/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210070029651 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210070029651 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210070029651 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:00:33.113513 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8362): Terminating (SIGTERM)
07:00:33.156198 run iperf3-srv_10.42.42.10(pid=8363): Terminating (SIGTERM)
07:00:33.199003 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8366): Terminating (SIGTERM)
07:00:33.241666 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8369): Terminating (SIGTERM)
07:00:33.285047 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8372): Terminating (SIGTERM)
07:00:33.294892 --- ParallelTerminationStrategy: PID 8362 died...
07:00:33.351931 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8362): Terminated: ok {rc=0}
07:00:33.362166 --- ParallelTerminationStrategy: PID 8363 died...
07:00:33.418895 run iperf3-srv_10.42.42.10(pid=8363): Terminated {rc=256}
07:00:33.429240 --- ParallelTerminationStrategy: PID 8366 died...
07:00:33.485952 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8366): Terminated: ok {rc=0}
07:00:33.496138 --- ParallelTerminationStrategy: PID 8369 died...
07:00:33.553122 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8369): Terminated: ok {rc=0}
07:00:33.563347 --- ParallelTerminationStrategy: PID 8372 died...
07:00:33.620169 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8372): Terminated: ok {rc=0}
07:00:33.709614 bus /gobi_6: Setting Powered False
59gprs:sysmo11.0osmo-stp_10.42.42.5(pid=8388): Process ended prematurely: osmo-stp_10.42.42.5(pid=8388) [trial-2371↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8388)]2313iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8388): Process ended prematurely: osmo-stp_10.42.42.5(pid=8388) [trial-2371↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8388)]
----------------------------------------------
trial-2371 gprs:sysmo iperf3m4.py
----------------------------------------------
07:00:34.828315 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:34.840078 tst gprs:sysmo: Using 1 x bts (candidates: 1)
07:00:34.971597 tst iperf3m4.py: using LAC 26657
07:00:35.103882 tst iperf3m4.py: using RAC 137
07:00:35.235779 tst iperf3m4.py: using CellId 26657
07:00:35.365696 tst iperf3m4.py: using BVCI 26658
07:00:35.391049 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:35.402157 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:35.413123 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:35.424022 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:35.434917 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:35.445753 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:35.456723 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:35.467614 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:35.478776 tst gprs:sysmo: Using 1 x modem (candidates: 4)
07:00:35.596346 tst gprs:sysmo: Using 1 x modem (candidates: 4)
07:00:35.716863 tst gprs:sysmo: Using 1 x modem (candidates: 4)
07:00:35.833463 tst gprs:sysmo: Using 1 x modem (candidates: 4)
07:00:36.117843 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
07:00:36.247330 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:00:36.572714 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8374): Launched
07:00:36.843432 run iperf3-srv_10.42.42.10(pid=8375): Launched
07:00:37.033513 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
07:00:37.164314 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:00:37.487735 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8376): Launched
07:00:37.758685 run iperf3-srv_10.42.42.10(pid=8377): Launched
07:00:37.948135 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
07:00:38.077647 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:00:38.401928 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8378): Launched
07:00:38.671338 run iperf3-srv_10.42.42.10(pid=8379): Launched
07:00:38.860492 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
07:00:38.989863 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:00:39.315623 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8380): Launched
07:00:39.601457 run iperf3-srv_10.42.42.10(pid=8381): Launched
07:00:39.790312 tst iperf3m4.py:8: start network...
07:00:39.919350 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:00:40.411624 run create_hlr_db(pid=8382): Launched
07:00:40.590757 bus /gobi_6: Setting Powered False
07:00:41.612014 run create_hlr_db(pid=8382): Terminated: ok {rc=0}
07:00:42.056171 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8384): Launched
07:00:42.392722 run osmo-hlr_10.42.42.2(pid=8385): Launched
07:00:42.516764 run osmo-stp_10.42.42.5: Starting osmo-stp
07:00:43.097974 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8387): Launched
07:00:43.431616 run osmo-stp_10.42.42.5(pid=8388): Launched
07:00:43.552142 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:00:44.133526 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8390): Launched
07:00:44.252589 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:00:44.545131 run patchelf(pid=8391): Launched
07:00:44.739817 run osmo-stp_10.42.42.5(pid=8388): ERR: Terminated: ERROR {rc=1} [trial-2371↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8388)]
07:00:44.882382 run osmo-stp_10.42.42.5(pid=8388): stdout:
| (launched: 2022-12-10_07:00:43.267650)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:00:45.025327 run osmo-stp_10.42.42.5(pid=8388): stderr:
| �[0;m�[38;5;43m20221210070043464 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070043464 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070043464 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070043465 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070043465 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070043465 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070043465 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:00:45.163772 run osmo-stp_10.42.42.5(pid=8388): stdout:
| (launched: 2022-12-10_07:00:43.267650)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:00:45.300161 run osmo-stp_10.42.42.5(pid=8388): stderr:
| �[0;m�[38;5;43m20221210070043464 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070043464 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070043464 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070043465 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070043465 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070043465 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070043465 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:00:45.459487 run patchelf(pid=8391): Terminating (SIGINT)
07:00:45.646849 run patchelf(pid=8391): Terminated: ok {rc=0}
07:00:45.680868 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=8388): Process ended prematurely: osmo-stp_10.42.42.5(pid=8388) [trial-2371↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8388)] [trial-2371↪gprs:sysmo↪iperf3m4.py:8]
07:00:45.690687 tst iperf3m4.py:8: Test FAILED (10.9 sec)
07:00:45.754801 run osmo-hlr_10.42.42.2(pid=8385): ERR: Terminated: ERROR {rc=237} [trial-2371↪gprs:sysmo↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8385)]
07:00:45.807133 run osmo-hlr_10.42.42.2(pid=8385): stdout:
| (launched: 2022-12-10_07:00:42.226415)
07:00:45.870711 run osmo-hlr_10.42.42.2(pid=8385): stderr:
| �[0;m�[1;31m20221210070042428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210070042428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210070042428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210070042428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210070042428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210070042429 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210070042441 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2371/run.2022-12-10_06-54-20/gprs:sysmo/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210070042450 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210070042450 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210070042450 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:00:45.917552 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8374): Terminating (SIGTERM)
07:00:45.981136 run iperf3-srv_10.42.42.10(pid=8375): Terminating (SIGTERM)
07:00:46.025336 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8376): Terminating (SIGTERM)
07:00:46.083014 run iperf3-srv_10.42.42.10(pid=8377): Terminating (SIGTERM)
07:00:46.140111 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8378): Terminating (SIGTERM)
07:00:46.197521 run iperf3-srv_10.42.42.10(pid=8379): Terminating (SIGTERM)
07:00:46.246871 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8380): Terminating (SIGTERM)
07:00:46.288764 run iperf3-srv_10.42.42.10(pid=8381): Terminating (SIGTERM)
07:00:46.334470 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8384): Terminating (SIGTERM)
07:00:46.377730 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8387): Terminating (SIGTERM)
07:00:46.424653 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8390): Terminating (SIGTERM)
07:00:46.436799 --- ParallelTerminationStrategy: PID 8374 died...
07:00:46.496899 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8374): Terminated: ok {rc=0}
07:00:46.508091 --- ParallelTerminationStrategy: PID 8375 died...
07:00:46.566412 run iperf3-srv_10.42.42.10(pid=8375): Terminated {rc=256}
07:00:46.577249 --- ParallelTerminationStrategy: PID 8376 died...
07:00:46.636481 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8376): Terminated: ok {rc=0}
07:00:46.647263 --- ParallelTerminationStrategy: PID 8377 died...
07:00:46.706914 run iperf3-srv_10.42.42.10(pid=8377): Terminated {rc=256}
07:00:46.717696 --- ParallelTerminationStrategy: PID 8378 died...
07:00:46.776052 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8378): Terminated: ok {rc=0}
07:00:46.786932 --- ParallelTerminationStrategy: PID 8379 died...
07:00:46.845955 run iperf3-srv_10.42.42.10(pid=8379): Terminated {rc=256}
07:00:46.856742 --- ParallelTerminationStrategy: PID 8380 died...
07:00:46.914657 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8380): Terminated: ok {rc=0}
07:00:46.925435 --- ParallelTerminationStrategy: PID 8381 died...
07:00:46.982124 run iperf3-srv_10.42.42.10(pid=8381): Terminated {rc=256}
07:00:46.994015 --- ParallelTerminationStrategy: PID 8384 died...
07:00:47.053543 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8384): Terminated: ok {rc=0}
07:00:47.064334 --- ParallelTerminationStrategy: PID 8387 died...
07:00:47.122523 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8387): Terminated: ok {rc=0}
07:00:47.133420 --- ParallelTerminationStrategy: PID 8390 died...
07:00:47.189869 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8390): Terminated: ok {rc=0}
07:00:47.280681 bus /gobi_4: Setting Powered False
07:00:48.389609 bus /gobi_4: Setting Powered False
07:00:49.503647 bus /gobi_1: Setting Powered False
07:00:50.608643 bus /gobi_1: Setting Powered False
07:00:51.721837 bus /gobi_0: Setting Powered False
07:00:52.827077 bus /gobi_0: Setting Powered False
07:00:53.937604 bus /gobi_6: Setting Powered False
41gprs:sysmo7.0osmo-stp_10.42.42.5(pid=8398): Process ended prematurely: osmo-stp_10.42.42.5(pid=8398) [trial-2371↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8398)]2331ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8398): Process ended prematurely: osmo-stp_10.42.42.5(pid=8398) [trial-2371↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8398)]
----------------------------------------------
trial-2371 gprs:sysmo ping.py
----------------------------------------------
07:00:55.007379 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:55.019135 tst gprs:sysmo: Using 1 x bts (candidates: 1)
07:00:55.156847 tst ping.py: using LAC 26658
07:00:55.286090 tst ping.py: using RAC 138
07:00:55.416510 tst ping.py: using CellId 26658
07:00:55.544770 tst ping.py: using BVCI 26659
07:00:55.570065 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:55.581252 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:55.592259 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:55.603224 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:55.614142 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:55.624936 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:55.636176 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:00:55.647200 tst gprs:sysmo: Using 1 x modem (candidates: 4)
07:00:55.944563 tst ping.py:19: start network...
07:00:56.081789 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:00:56.606857 run create_hlr_db(pid=8392): Launched
07:00:56.796307 bus /gobi_6: Setting Powered False
07:00:57.816833 run create_hlr_db(pid=8392): Terminated: ok {rc=0}
07:00:58.238265 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8394): Launched
07:00:58.546325 run osmo-hlr_10.42.42.2(pid=8395): Launched
07:00:58.662939 run osmo-stp_10.42.42.5: Starting osmo-stp
07:00:59.200052 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8397): Launched
07:00:59.507540 run osmo-stp_10.42.42.5(pid=8398): Launched
07:00:59.619313 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:01:00.151284 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8400): Launched
07:01:00.259623 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:01:00.527729 run patchelf(pid=8401): Launched
07:01:00.706082 run osmo-stp_10.42.42.5(pid=8398): ERR: Terminated: ERROR {rc=1} [trial-2371↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8398)]
07:01:00.836796 run osmo-stp_10.42.42.5(pid=8398): stdout:
| (launched: 2022-12-10_07:00:59.354662)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:01:00.976734 run osmo-stp_10.42.42.5(pid=8398): stderr:
| �[0;m�[38;5;43m20221210070059524 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070059524 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070059525 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070059525 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070059527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070059527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070059528 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:01:01.106377 run osmo-stp_10.42.42.5(pid=8398): stdout:
| (launched: 2022-12-10_07:00:59.354662)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:01:01.235752 run osmo-stp_10.42.42.5(pid=8398): stderr:
| �[0;m�[38;5;43m20221210070059524 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070059524 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070059525 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070059525 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070059527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070059527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070059528 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:01:01.386648 run patchelf(pid=8401): Terminating (SIGINT)
07:01:01.562748 run patchelf(pid=8401): Terminated: ok {rc=0}
07:01:01.595696 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=8398): Process ended prematurely: osmo-stp_10.42.42.5(pid=8398) [trial-2371↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8398)] [trial-2371↪gprs:sysmo↪ping.py:22]
07:01:01.605210 tst ping.py:22: Test FAILED (6.6 sec)
07:01:01.670728 run osmo-hlr_10.42.42.2(pid=8395): ERR: Terminated: ERROR {rc=237} [trial-2371↪gprs:sysmo↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8395)]
07:01:01.722954 run osmo-hlr_10.42.42.2(pid=8395): stdout:
| (launched: 2022-12-10_07:00:58.392690)
07:01:01.776140 run osmo-hlr_10.42.42.2(pid=8395): stderr:
| �[0;m�[1;31m20221210070058571 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210070058571 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210070058571 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210070058571 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210070058571 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210070058571 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210070058583 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2371/run.2022-12-10_06-54-20/gprs:sysmo/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210070058592 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210070058592 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210070058592 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:01:01.821454 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8394): Terminating (SIGTERM)
07:01:01.863647 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8397): Terminating (SIGTERM)
07:01:01.906580 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8400): Terminating (SIGTERM)
07:01:01.918190 --- ParallelTerminationStrategy: PID 8394 died...
07:01:01.977318 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8394): Terminated: ok {rc=0}
07:01:01.988069 --- ParallelTerminationStrategy: PID 8397 died...
07:01:02.045828 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8397): Terminated: ok {rc=0}
07:01:02.056520 --- ParallelTerminationStrategy: PID 8400 died...
07:01:02.114636 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8400): Terminated: ok {rc=0}
07:01:02.203817 bus /gobi_6: Setting Powered False
41gprs:sysmo7.0osmo-stp_10.42.42.5(pid=8408): Process ended prematurely: osmo-stp_10.42.42.5(pid=8408) [trial-2371↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8408)]2331ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8408): Process ended prematurely: osmo-stp_10.42.42.5(pid=8408) [trial-2371↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8408)]
----------------------------------------------
trial-2371 gprs:sysmo ping_idle_ping.py
----------------------------------------------
07:01:03.257210 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:01:03.269103 tst gprs:sysmo: Using 1 x bts (candidates: 1)
07:01:03.403194 tst ping_idle_ping.py: using LAC 26659
07:01:03.530781 tst ping_idle_ping.py: using RAC 139
07:01:03.659061 tst ping_idle_ping.py: using CellId 26659
07:01:03.789483 tst ping_idle_ping.py: using BVCI 26660
07:01:03.814619 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:01:03.825719 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:01:03.836636 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:01:03.847769 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:01:03.858900 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:01:03.869920 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:01:03.881219 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
07:01:03.892260 tst gprs:sysmo: Using 1 x modem (candidates: 4)
07:01:04.185177 tst ping_idle_ping.py:24: start network...
07:01:04.313325 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:01:04.796385 run create_hlr_db(pid=8402): Launched
07:01:04.973287 bus /gobi_6: Setting Powered False
07:01:05.992866 run create_hlr_db(pid=8402): Terminated: ok {rc=0}
07:01:06.401447 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8404): Launched
07:01:06.710938 run osmo-hlr_10.42.42.2(pid=8405): Launched
07:01:06.831343 run osmo-stp_10.42.42.5: Starting osmo-stp
07:01:07.371851 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8407): Launched
07:01:07.691997 run osmo-stp_10.42.42.5(pid=8408): Launched
07:01:07.812892 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:01:08.387450 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8410): Launched
07:01:08.504873 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:01:08.794158 run patchelf(pid=8411): Launched
07:01:08.985915 run osmo-stp_10.42.42.5(pid=8408): ERR: Terminated: ERROR {rc=1} [trial-2371↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8408)]
07:01:09.127020 run osmo-stp_10.42.42.5(pid=8408): stdout:
| (launched: 2022-12-10_07:01:07.526602)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:01:09.267369 run osmo-stp_10.42.42.5(pid=8408): stderr:
| �[0;m�[38;5;43m20221210070107715 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070107715 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070107716 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070107716 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070107717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070107717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070107717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:01:09.403715 run osmo-stp_10.42.42.5(pid=8408): stdout:
| (launched: 2022-12-10_07:01:07.526602)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:01:09.537897 run osmo-stp_10.42.42.5(pid=8408): stderr:
| �[0;m�[38;5;43m20221210070107715 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210070107715 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210070107716 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210070107716 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210070107717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210070107717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210070107717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:01:09.695619 run patchelf(pid=8411): Terminating (SIGINT)
07:01:09.879132 run patchelf(pid=8411): Terminated: ok {rc=0}
07:01:09.912341 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=8408): Process ended prematurely: osmo-stp_10.42.42.5(pid=8408) [trial-2371↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8408)] [trial-2371↪gprs:sysmo↪ping_idle_ping.py:27]
07:01:09.921518 tst ping_idle_ping.py:27: Test FAILED (6.7 sec)
07:01:09.984635 run osmo-hlr_10.42.42.2(pid=8405): ERR: Terminated: ERROR {rc=237} [trial-2371↪gprs:sysmo↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8405)]
07:01:10.036429 run osmo-hlr_10.42.42.2(pid=8405): stdout:
| (launched: 2022-12-10_07:01:06.556430)
07:01:10.089374 run osmo-hlr_10.42.42.2(pid=8405): stderr:
| �[0;m�[1;31m20221210070106735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210070106735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210070106735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210070106735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210070106735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210070106735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210070106747 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2371/run.2022-12-10_06-54-20/gprs:sysmo/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210070106755 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210070106755 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210070106755 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:01:10.134279 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8404): Terminating (SIGTERM)
07:01:10.176430 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8407): Terminating (SIGTERM)
07:01:10.218824 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8410): Terminating (SIGTERM)
07:01:10.228547 --- ParallelTerminationStrategy: PID 8404 died...
07:01:10.285240 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8404): Terminated: ok {rc=0}
07:01:10.295509 --- ParallelTerminationStrategy: PID 8407 died...
07:01:10.352181 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8407): Terminated: ok {rc=0}
07:01:10.362489 --- ParallelTerminationStrategy: PID 8410 died...
07:01:10.419024 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8410): Terminated: ok {rc=0}
07:01:10.507958 bus /gobi_6: Setting Powered False
5gprs:sysmo00