221gprs:oc2g10.0osmo-stp_10.42.42.5(pid=2835): Process ended prematurely: osmo-stp_10.42.42.5(pid=2835) [trial-2363↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2835)]2143cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2835): Process ended prematurely: osmo-stp_10.42.42.5(pid=2835) [trial-2363↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2835)]
----------------------------------------------
trial-2363 gprs:oc2g cs_paging_gprs_active.py
----------------------------------------------
18:50:55.218593 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:50:55.229755 tst gprs:oc2g: Using 1 x bts (candidates: 1)
18:50:55.354905 tst cs_paging_gprs_active.py: using LAC 23009
18:50:55.477909 tst cs_paging_gprs_active.py: using RAC 59
18:50:55.603293 tst cs_paging_gprs_active.py: using CellId 23009
18:50:55.730016 tst cs_paging_gprs_active.py: using BVCI 23010
18:50:55.754735 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:50:55.765493 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:50:55.776079 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:50:55.787098 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:50:55.797832 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:50:55.808529 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:50:55.819463 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:50:55.830326 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:50:55.841321 tst gprs:oc2g: Using 1 x modem (candidates: 4)
18:50:55.958414 tst gprs:oc2g: Using 1 x modem (candidates: 4)
18:50:56.252615 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
18:50:56.389007 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:50:56.724306 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2825): Launched
18:50:57.002792 run iperf3-srv_10.42.42.10(pid=2826): Launched
18:50:57.202641 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
18:50:57.338442 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:50:57.673764 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2827): Launched
18:50:57.951418 run iperf3-srv_10.42.42.10(pid=2828): Launched
18:50:58.151057 tst cs_paging_gprs_active.py:38: start network...
18:50:58.286804 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:50:58.795276 run create_hlr_db(pid=2829): Launched
18:50:58.983037 bus /sierra_2: Setting Powered False
18:51:00.003670 run create_hlr_db(pid=2829): Terminated: ok {rc=0}
18:51:00.410797 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2831): Launched
18:51:00.713496 run osmo-hlr_10.42.42.2(pid=2832): Launched
18:51:00.846961 run osmo-stp_10.42.42.5: Starting osmo-stp
18:51:01.440872 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2834): Launched
18:51:01.833780 run osmo-stp_10.42.42.5(pid=2835): Launched
18:51:02.127783 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:51:02.727841 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2837): Launched
18:51:02.844317 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:51:03.127413 run patchelf(pid=2838): Launched
18:51:03.316778 run osmo-stp_10.42.42.5(pid=2835): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2835)]
18:51:03.455425 run osmo-stp_10.42.42.5(pid=2835): stdout:
| (launched: 2022-12-09_18:51:01.596438)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:03.594809 run osmo-stp_10.42.42.5(pid=2835): stderr:
| �[0;m�[38;5;43m20221209185101852 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185101852 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185101852 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185101853 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185101855 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185101855 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185101856 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:03.729080 run osmo-stp_10.42.42.5(pid=2835): stdout:
| (launched: 2022-12-09_18:51:01.596438)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:03.861648 run osmo-stp_10.42.42.5(pid=2835): stderr:
| �[0;m�[38;5;43m20221209185101852 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185101852 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185101852 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185101853 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185101855 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185101855 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185101856 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:04.016311 run patchelf(pid=2838): Terminating (SIGINT)
18:51:04.198327 run patchelf(pid=2838): Terminated: ok {rc=0}
18:51:04.231343 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=2835): Process ended prematurely: osmo-stp_10.42.42.5(pid=2835) [trial-2363↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2835)] [trial-2363↪gprs:oc2g↪cs_paging_gprs_active.py:38]
18:51:04.241080 tst cs_paging_gprs_active.py:38: Test FAILED (9.1 sec)
18:51:04.304206 run osmo-hlr_10.42.42.2(pid=2832): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:oc2g↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2832)]
18:51:04.355754 run osmo-hlr_10.42.42.2(pid=2832): stdout:
| (launched: 2022-12-09_18:51:00.564564)
18:51:04.407589 run osmo-hlr_10.42.42.2(pid=2832): stderr:
| �[0;m�[1;31m20221209185100741 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209185100741 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209185100741 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209185100741 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209185100741 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209185100741 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209185100754 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:oc2g/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209185100764 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209185100764 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209185100764 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:51:04.451347 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2825): Terminating (SIGTERM)
18:51:04.492665 run iperf3-srv_10.42.42.10(pid=2826): Terminating (SIGTERM)
18:51:04.534153 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2827): Terminating (SIGTERM)
18:51:04.575289 run iperf3-srv_10.42.42.10(pid=2828): Terminating (SIGTERM)
18:51:04.616442 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2831): Terminating (SIGTERM)
18:51:04.657384 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2834): Terminating (SIGTERM)
18:51:04.698342 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2837): Terminating (SIGTERM)
18:51:04.708174 --- ParallelTerminationStrategy: PID 2825 died...
18:51:04.763718 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2825): Terminated: ok {rc=0}
18:51:04.773806 --- ParallelTerminationStrategy: PID 2826 died...
18:51:04.828529 run iperf3-srv_10.42.42.10(pid=2826): Terminated {rc=256}
18:51:04.838312 --- ParallelTerminationStrategy: PID 2827 died...
18:51:04.893217 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2827): Terminated: ok {rc=0}
18:51:04.903379 --- ParallelTerminationStrategy: PID 2828 died...
18:51:04.958013 run iperf3-srv_10.42.42.10(pid=2828): Terminated {rc=256}
18:51:04.968012 --- ParallelTerminationStrategy: PID 2831 died...
18:51:05.022630 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2831): Terminated: ok {rc=0}
18:51:05.032706 --- ParallelTerminationStrategy: PID 2834 died...
18:51:05.087344 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2834): Terminated: ok {rc=0}
18:51:05.097704 --- ParallelTerminationStrategy: PID 2837 died...
18:51:05.152650 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2837): Terminated: ok {rc=0}
18:51:05.237488 bus /gobi_0: Setting Powered False
18:51:06.342408 bus /gobi_0: Setting Powered False
18:51:07.447662 bus /gobi_6: Setting Powered False
221gprs:oc2g8.0osmo-stp_10.42.42.5(pid=2848): Process ended prematurely: osmo-stp_10.42.42.5(pid=2848) [trial-2363↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2848)]2143iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2848): Process ended prematurely: osmo-stp_10.42.42.5(pid=2848) [trial-2363↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2848)]
----------------------------------------------
trial-2363 gprs:oc2g iperf3.py
----------------------------------------------
18:51:08.561094 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:08.572333 tst gprs:oc2g: Using 1 x bts (candidates: 1)
18:51:08.703531 tst iperf3.py: using LAC 23010
18:51:08.829872 tst iperf3.py: using RAC 60
18:51:08.957576 tst iperf3.py: using CellId 23010
18:51:09.086009 tst iperf3.py: using BVCI 23011
18:51:09.110639 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:09.121414 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:09.132079 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:09.143160 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:09.154126 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:09.164694 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:09.175565 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:09.186429 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:09.197445 tst gprs:oc2g: Using 1 x modem (candidates: 4)
18:51:09.477873 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
18:51:09.603818 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:51:09.917863 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2840): Launched
18:51:10.176663 run iperf3-srv_10.42.42.10(pid=2841): Launched
18:51:10.360352 tst iperf3.py:8: start network...
18:51:10.485808 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:51:10.969510 run create_hlr_db(pid=2842): Launched
18:51:11.143283 bus /gobi_6: Setting Powered False
18:51:12.163810 run create_hlr_db(pid=2842): Terminated: ok {rc=0}
18:51:12.593475 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2844): Launched
18:51:12.916518 run osmo-hlr_10.42.42.2(pid=2845): Launched
18:51:13.035331 run osmo-stp_10.42.42.5: Starting osmo-stp
18:51:13.596231 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2847): Launched
18:51:13.920058 run osmo-stp_10.42.42.5(pid=2848): Launched
18:51:14.038615 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:51:14.603996 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2850): Launched
18:51:14.720178 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:51:15.001724 run patchelf(pid=2851): Launched
18:51:15.191100 run osmo-stp_10.42.42.5(pid=2848): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2848)]
18:51:15.329411 run osmo-stp_10.42.42.5(pid=2848): stdout:
| (launched: 2022-12-09_18:51:13.761317)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:15.467489 run osmo-stp_10.42.42.5(pid=2848): stderr:
| �[0;m�[38;5;43m20221209185113953 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185113954 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185113954 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185113954 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185113955 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185113955 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185113955 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:15.602034 run osmo-stp_10.42.42.5(pid=2848): stdout:
| (launched: 2022-12-09_18:51:13.761317)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:15.734585 run osmo-stp_10.42.42.5(pid=2848): stderr:
| �[0;m�[38;5;43m20221209185113953 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185113954 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185113954 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185113954 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185113955 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185113955 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185113955 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:15.923043 run patchelf(pid=2851): Terminating (SIGINT)
18:51:16.111808 run patchelf(pid=2851): Terminated: ok {rc=0}
18:51:16.144180 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=2848): Process ended prematurely: osmo-stp_10.42.42.5(pid=2848) [trial-2363↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2848)] [trial-2363↪gprs:oc2g↪iperf3.py:8]
18:51:16.154004 tst iperf3.py:8: Test FAILED (7.6 sec)
18:51:16.215993 run osmo-hlr_10.42.42.2(pid=2845): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:oc2g↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2845)]
18:51:16.266274 run osmo-hlr_10.42.42.2(pid=2845): stdout:
| (launched: 2022-12-09_18:51:12.757915)
18:51:16.316502 run osmo-hlr_10.42.42.2(pid=2845): stderr:
| �[0;m�[1;31m20221209185112955 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209185112955 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209185112955 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209185112955 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209185112955 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209185112955 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209185112968 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:oc2g/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209185112976 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209185112976 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209185112976 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:51:16.360219 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2840): Terminating (SIGTERM)
18:51:16.402201 run iperf3-srv_10.42.42.10(pid=2841): Terminating (SIGTERM)
18:51:16.445747 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2844): Terminating (SIGTERM)
18:51:16.488524 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2847): Terminating (SIGTERM)
18:51:16.530105 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2850): Terminating (SIGTERM)
18:51:16.541356 --- ParallelTerminationStrategy: PID 2840 died...
18:51:16.598847 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2840): Terminated: ok {rc=0}
18:51:16.609577 --- ParallelTerminationStrategy: PID 2841 died...
18:51:16.663659 run iperf3-srv_10.42.42.10(pid=2841): Terminated {rc=256}
18:51:16.674250 --- ParallelTerminationStrategy: PID 2844 died...
18:51:16.727498 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2844): Terminated: ok {rc=0}
18:51:16.738750 --- ParallelTerminationStrategy: PID 2847 died...
18:51:16.793392 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2847): Terminated: ok {rc=0}
18:51:16.803399 --- ParallelTerminationStrategy: PID 2850 died...
18:51:16.856698 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2850): Terminated: ok {rc=0}
18:51:16.945465 bus /gobi_6: Setting Powered False
225gprs:oc2g11.0osmo-stp_10.42.42.5(pid=2866): Process ended prematurely: osmo-stp_10.42.42.5(pid=2866) [trial-2363↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2866)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2866): Process ended prematurely: osmo-stp_10.42.42.5(pid=2866) [trial-2363↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2866)]
----------------------------------------------
trial-2363 gprs:oc2g iperf3m4.py
----------------------------------------------
18:51:18.053449 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:18.064182 tst gprs:oc2g: Using 1 x bts (candidates: 1)
18:51:18.190095 tst iperf3m4.py: using LAC 23011
18:51:18.307214 tst iperf3m4.py: using RAC 61
18:51:18.426881 tst iperf3m4.py: using CellId 23011
18:51:18.543759 tst iperf3m4.py: using BVCI 23012
18:51:18.566903 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:18.577258 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:18.587413 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:18.597549 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:18.607516 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:18.617652 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:18.627781 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:18.637857 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:18.648046 tst gprs:oc2g: Using 1 x modem (candidates: 4)
18:51:18.762111 tst gprs:oc2g: Using 1 x modem (candidates: 4)
18:51:18.877976 tst gprs:oc2g: Using 1 x modem (candidates: 4)
18:51:18.996114 tst gprs:oc2g: Using 1 x modem (candidates: 4)
18:51:19.288760 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
18:51:19.425084 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:51:19.762317 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2852): Launched
18:51:20.042137 run iperf3-srv_10.42.42.10(pid=2853): Launched
18:51:20.242108 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
18:51:20.378439 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:51:20.713469 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2854): Launched
18:51:20.991053 run iperf3-srv_10.42.42.10(pid=2855): Launched
18:51:21.190559 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
18:51:21.326402 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:51:21.661488 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2856): Launched
18:51:21.940695 run iperf3-srv_10.42.42.10(pid=2857): Launched
18:51:22.140545 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
18:51:22.276074 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:51:22.611583 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2858): Launched
18:51:22.888476 run iperf3-srv_10.42.42.10(pid=2859): Launched
18:51:23.087987 tst iperf3m4.py:8: start network...
18:51:23.223443 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:51:23.733754 run create_hlr_db(pid=2860): Launched
18:51:23.921745 bus /gobi_6: Setting Powered False
18:51:24.942315 run create_hlr_db(pid=2860): Terminated: ok {rc=0}
18:51:25.373160 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2862): Launched
18:51:25.697874 run osmo-hlr_10.42.42.2(pid=2863): Launched
18:51:25.814568 run osmo-stp_10.42.42.5: Starting osmo-stp
18:51:26.377072 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2865): Launched
18:51:26.701076 run osmo-stp_10.42.42.5(pid=2866): Launched
18:51:26.818493 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:51:27.384391 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2868): Launched
18:51:27.500446 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:51:27.782575 run patchelf(pid=2869): Launched
18:51:27.971986 run osmo-stp_10.42.42.5(pid=2866): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2866)]
18:51:28.110905 run osmo-stp_10.42.42.5(pid=2866): stdout:
| (launched: 2022-12-09_18:51:26.542030)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:28.249569 run osmo-stp_10.42.42.5(pid=2866): stderr:
| �[0;m�[38;5;43m20221209185126731 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185126731 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185126731 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185126732 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185126732 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185126732 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185126732 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:28.383179 run osmo-stp_10.42.42.5(pid=2866): stdout:
| (launched: 2022-12-09_18:51:26.542030)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:28.516159 run osmo-stp_10.42.42.5(pid=2866): stderr:
| �[0;m�[38;5;43m20221209185126731 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185126731 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185126731 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185126732 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185126732 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185126732 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185126732 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:28.670822 run patchelf(pid=2869): Terminating (SIGINT)
18:51:28.851996 run patchelf(pid=2869): Terminated: ok {rc=0}
18:51:28.885300 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=2866): Process ended prematurely: osmo-stp_10.42.42.5(pid=2866) [trial-2363↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2866)] [trial-2363↪gprs:oc2g↪iperf3m4.py:8]
18:51:28.895222 tst iperf3m4.py:8: Test FAILED (10.9 sec)
18:51:28.958221 run osmo-hlr_10.42.42.2(pid=2863): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:oc2g↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2863)]
18:51:29.009133 run osmo-hlr_10.42.42.2(pid=2863): stdout:
| (launched: 2022-12-09_18:51:25.538625)
18:51:29.060688 run osmo-hlr_10.42.42.2(pid=2863): stderr:
| �[0;m�[1;31m20221209185125734 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209185125734 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209185125734 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209185125734 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209185125734 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209185125734 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209185125747 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:oc2g/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209185125758 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209185125758 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209185125758 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:51:29.104526 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2852): Terminating (SIGTERM)
18:51:29.145930 run iperf3-srv_10.42.42.10(pid=2853): Terminating (SIGTERM)
18:51:29.187101 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2854): Terminating (SIGTERM)
18:51:29.228008 run iperf3-srv_10.42.42.10(pid=2855): Terminating (SIGTERM)
18:51:29.269467 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2856): Terminating (SIGTERM)
18:51:29.310656 run iperf3-srv_10.42.42.10(pid=2857): Terminating (SIGTERM)
18:51:29.352278 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2858): Terminating (SIGTERM)
18:51:29.393315 run iperf3-srv_10.42.42.10(pid=2859): Terminating (SIGTERM)
18:51:29.434564 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2862): Terminating (SIGTERM)
18:51:29.475721 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2865): Terminating (SIGTERM)
18:51:29.516872 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2868): Terminating (SIGTERM)
18:51:29.526799 --- ParallelTerminationStrategy: PID 2852 died...
18:51:29.581962 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2852): Terminated: ok {rc=0}
18:51:29.592152 --- ParallelTerminationStrategy: PID 2853 died...
18:51:29.647043 run iperf3-srv_10.42.42.10(pid=2853): Terminated {rc=256}
18:51:29.657206 --- ParallelTerminationStrategy: PID 2854 died...
18:51:29.712026 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2854): Terminated: ok {rc=0}
18:51:29.722034 --- ParallelTerminationStrategy: PID 2855 died...
18:51:29.776470 run iperf3-srv_10.42.42.10(pid=2855): Terminated {rc=256}
18:51:29.786685 --- ParallelTerminationStrategy: PID 2856 died...
18:51:29.841565 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2856): Terminated: ok {rc=0}
18:51:29.851744 --- ParallelTerminationStrategy: PID 2857 died...
18:51:29.907062 run iperf3-srv_10.42.42.10(pid=2857): Terminated {rc=256}
18:51:29.917183 --- ParallelTerminationStrategy: PID 2858 died...
18:51:29.972076 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2858): Terminated: ok {rc=0}
18:51:29.982193 --- ParallelTerminationStrategy: PID 2859 died...
18:51:30.037523 run iperf3-srv_10.42.42.10(pid=2859): Terminated {rc=256}
18:51:30.047492 --- ParallelTerminationStrategy: PID 2862 died...
18:51:30.102530 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2862): Terminated: ok {rc=0}
18:51:30.112691 --- ParallelTerminationStrategy: PID 2865 died...
18:51:30.167952 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2865): Terminated: ok {rc=0}
18:51:30.178003 --- ParallelTerminationStrategy: PID 2868 died...
18:51:30.233375 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2868): Terminated: ok {rc=0}
18:51:30.318997 bus /gobi_4: Setting Powered False
18:51:31.424091 bus /gobi_4: Setting Powered False
18:51:32.533620 bus /gobi_1: Setting Powered False
18:51:33.638401 bus /gobi_1: Setting Powered False
18:51:34.743505 bus /gobi_0: Setting Powered False
18:51:35.847849 bus /gobi_0: Setting Powered False
18:51:36.955001 bus /gobi_6: Setting Powered False
225gprs:oc2g7.0osmo-stp_10.42.42.5(pid=2876): Process ended prematurely: osmo-stp_10.42.42.5(pid=2876) [trial-2363↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2876)]2139ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2876): Process ended prematurely: osmo-stp_10.42.42.5(pid=2876) [trial-2363↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2876)]
----------------------------------------------
trial-2363 gprs:oc2g ping.py
----------------------------------------------
18:51:38.022754 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:38.034294 tst gprs:oc2g: Using 1 x bts (candidates: 1)
18:51:38.163989 tst ping.py: using LAC 23012
18:51:38.287821 tst ping.py: using RAC 62
18:51:38.412528 tst ping.py: using CellId 23012
18:51:38.537275 tst ping.py: using BVCI 23013
18:51:38.561760 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:38.572438 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:38.583109 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:38.594045 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:38.604792 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:38.615577 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:38.626570 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:38.637395 tst gprs:oc2g: Using 1 x modem (candidates: 4)
18:51:38.910876 tst ping.py:19: start network...
18:51:39.035605 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:51:39.510281 run create_hlr_db(pid=2870): Launched
18:51:39.681445 bus /gobi_6: Setting Powered False
18:51:40.702768 run create_hlr_db(pid=2870): Terminated: ok {rc=0}
18:51:41.123156 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2872): Launched
18:51:41.421468 run osmo-hlr_10.42.42.2(pid=2873): Launched
18:51:41.528752 run osmo-stp_10.42.42.5: Starting osmo-stp
18:51:42.046346 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2875): Launched
18:51:42.343810 run osmo-stp_10.42.42.5(pid=2876): Launched
18:51:42.451281 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:51:42.968577 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2878): Launched
18:51:43.074073 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:51:43.333474 run patchelf(pid=2879): Launched
18:51:43.506489 run osmo-stp_10.42.42.5(pid=2876): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2876)]
18:51:43.632839 run osmo-stp_10.42.42.5(pid=2876): stdout:
| (launched: 2022-12-09_18:51:42.197451)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:43.759486 run osmo-stp_10.42.42.5(pid=2876): stderr:
| �[0;m�[38;5;43m20221209185142367 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185142367 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185142367 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185142368 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185142368 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185142368 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185142368 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:43.882544 run osmo-stp_10.42.42.5(pid=2876): stdout:
| (launched: 2022-12-09_18:51:42.197451)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:44.004661 run osmo-stp_10.42.42.5(pid=2876): stderr:
| �[0;m�[38;5;43m20221209185142367 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185142367 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185142367 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185142368 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185142368 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185142368 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185142368 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:44.146309 run patchelf(pid=2879): Terminating (SIGINT)
18:51:44.311813 run patchelf(pid=2879): Terminated: ok {rc=0}
18:51:44.342123 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=2876): Process ended prematurely: osmo-stp_10.42.42.5(pid=2876) [trial-2363↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2876)] [trial-2363↪gprs:oc2g↪ping.py:22]
18:51:44.350784 tst ping.py:22: Test FAILED (6.3 sec)
18:51:44.409950 run osmo-hlr_10.42.42.2(pid=2873): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:oc2g↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2873)]
18:51:44.458329 run osmo-hlr_10.42.42.2(pid=2873): stdout:
| (launched: 2022-12-09_18:51:41.274530)
18:51:44.507215 run osmo-hlr_10.42.42.2(pid=2873): stderr:
| �[0;m�[1;31m20221209185141451 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209185141451 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209185141451 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209185141451 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209185141451 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209185141451 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209185141464 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:oc2g/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209185141475 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209185141475 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209185141475 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:51:44.548737 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2872): Terminating (SIGTERM)
18:51:44.588318 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2875): Terminating (SIGTERM)
18:51:44.627625 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2878): Terminating (SIGTERM)
18:51:44.636847 --- ParallelTerminationStrategy: PID 2872 died...
18:51:44.689387 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2872): Terminated: ok {rc=0}
18:51:44.699009 --- ParallelTerminationStrategy: PID 2875 died...
18:51:44.751643 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2875): Terminated: ok {rc=0}
18:51:44.761318 --- ParallelTerminationStrategy: PID 2878 died...
18:51:44.813419 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2878): Terminated: ok {rc=0}
18:51:44.900013 bus /gobi_6: Setting Powered False
225gprs:oc2g7.0osmo-stp_10.42.42.5(pid=2886): Process ended prematurely: osmo-stp_10.42.42.5(pid=2886) [trial-2363↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2886)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2886): Process ended prematurely: osmo-stp_10.42.42.5(pid=2886) [trial-2363↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2886)]
----------------------------------------------
trial-2363 gprs:oc2g ping_idle_ping.py
----------------------------------------------
18:51:45.957244 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:45.969572 tst gprs:oc2g: Using 1 x bts (candidates: 1)
18:51:46.098639 tst ping_idle_ping.py: using LAC 23013
18:51:46.225746 tst ping_idle_ping.py: using RAC 63
18:51:46.354335 tst ping_idle_ping.py: using CellId 23013
18:51:46.497981 tst ping_idle_ping.py: using BVCI 23014
18:51:46.529478 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:46.542894 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:46.555602 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:46.568361 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:46.584013 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:46.597059 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:46.608942 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
18:51:46.623129 tst gprs:oc2g: Using 1 x modem (candidates: 4)
18:51:46.906817 tst ping_idle_ping.py:24: start network...
18:51:47.033272 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:51:47.518496 run create_hlr_db(pid=2880): Launched
18:51:47.705632 bus /gobi_6: Setting Powered False
18:51:48.724807 run create_hlr_db(pid=2880): Terminated: ok {rc=0}
18:51:49.124840 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2882): Launched
18:51:49.423129 run osmo-hlr_10.42.42.2(pid=2883): Launched
18:51:49.536891 run osmo-stp_10.42.42.5: Starting osmo-stp
18:51:50.053950 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2885): Launched
18:51:50.352542 run osmo-stp_10.42.42.5(pid=2886): Launched
18:51:50.460691 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:51:50.978419 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2888): Launched
18:51:51.083788 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:51:51.342566 run patchelf(pid=2889): Launched
18:51:51.516160 run osmo-stp_10.42.42.5(pid=2886): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2886)]
18:51:51.643161 run osmo-stp_10.42.42.5(pid=2886): stdout:
| (launched: 2022-12-09_18:51:50.204918)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:51.770269 run osmo-stp_10.42.42.5(pid=2886): stderr:
| �[0;m�[38;5;43m20221209185150375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185150375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185150375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185150376 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185150376 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185150376 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185150376 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:51.893443 run osmo-stp_10.42.42.5(pid=2886): stdout:
| (launched: 2022-12-09_18:51:50.204918)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:51:52.015495 run osmo-stp_10.42.42.5(pid=2886): stderr:
| �[0;m�[38;5;43m20221209185150375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209185150375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209185150375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209185150376 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209185150376 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209185150376 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209185150376 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:51:52.158117 run patchelf(pid=2889): Terminating (SIGINT)
18:51:52.324563 run patchelf(pid=2889): Terminated: ok {rc=0}
18:51:52.354970 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=2886): Process ended prematurely: osmo-stp_10.42.42.5(pid=2886) [trial-2363↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2886)] [trial-2363↪gprs:oc2g↪ping_idle_ping.py:27]
18:51:52.363396 tst ping_idle_ping.py:27: Test FAILED (6.4 sec)
18:51:52.422069 run osmo-hlr_10.42.42.2(pid=2883): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:oc2g↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2883)]
18:51:52.470210 run osmo-hlr_10.42.42.2(pid=2883): stdout:
| (launched: 2022-12-09_18:51:49.276339)
18:51:52.518562 run osmo-hlr_10.42.42.2(pid=2883): stderr:
| �[0;m�[1;31m20221209185149453 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209185149453 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209185149453 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209185149453 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209185149453 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209185149453 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209185149464 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:oc2g/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209185149473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209185149473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209185149474 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:51:52.559714 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2882): Terminating (SIGTERM)
18:51:52.598582 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2885): Terminating (SIGTERM)
18:51:52.637450 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2888): Terminating (SIGTERM)
18:51:52.646600 --- ParallelTerminationStrategy: PID 2882 died...
18:51:52.698847 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2882): Terminated: ok {rc=0}
18:51:52.708288 --- ParallelTerminationStrategy: PID 2885 died...
18:51:52.760337 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2885): Terminated: ok {rc=0}
18:51:52.769900 --- ParallelTerminationStrategy: PID 2888 died...
18:51:52.821976 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2888): Terminated: ok {rc=0}
18:51:52.908398 bus /gobi_6: Setting Powered False
5gprs:oc2g00