253gprs:oc2g9.0osmo-stp_10.42.42.5(pid=4975): Process ended prematurely: osmo-stp_10.42.42.5(pid=4975) [trial-2395↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4975)]2143cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=4975): Process ended prematurely: osmo-stp_10.42.42.5(pid=4975) [trial-2395↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4975)]
----------------------------------------------
trial-2395 gprs:oc2g cs_paging_gprs_active.py
----------------------------------------------
21:58:12.133911 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:12.144746 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:58:12.278183 tst cs_paging_gprs_active.py: using LAC 36492
21:58:12.407811 tst cs_paging_gprs_active.py: using RAC 27
21:58:12.536913 tst cs_paging_gprs_active.py: using CellId 36492
21:58:12.666697 tst cs_paging_gprs_active.py: using BVCI 36493
21:58:12.691547 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:12.702354 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:12.713070 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:12.723750 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:12.735174 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:12.746682 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:12.758260 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:12.769661 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:12.781279 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:12.898642 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:13.196081 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
21:58:13.333372 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:13.671650 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4965): Launched
21:58:13.952519 run iperf3-srv_10.42.42.10(pid=4966): Launched
21:58:14.154333 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
21:58:14.291167 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:14.629878 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4967): Launched
21:58:14.911174 run iperf3-srv_10.42.42.10(pid=4968): Launched
21:58:15.113414 tst cs_paging_gprs_active.py:38: start network...
21:58:15.250232 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:15.763860 run create_hlr_db(pid=4969): Launched
21:58:15.969284 bus /sierra_2: Setting Powered False
21:58:16.965408 run create_hlr_db(pid=4969): Terminated: ok {rc=0}
21:58:17.412281 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4971): Launched
21:58:17.719506 run osmo-hlr_10.42.42.2(pid=4972): Launched
21:58:17.832260 run osmo-stp_10.42.42.5: Starting osmo-stp
21:58:18.357425 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4974): Launched
21:58:18.660634 run osmo-stp_10.42.42.5(pid=4975): Launched
21:58:18.770519 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:58:19.298148 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4977): Launched
21:58:19.406055 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:58:19.670844 run patchelf(pid=4978): Launched
21:58:19.848800 run osmo-stp_10.42.42.5(pid=4975): ERR: Terminated: ERROR {rc=1} [trial-2395↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4975)]
21:58:19.978354 run osmo-stp_10.42.42.5(pid=4975): stdout:
| (launched: 2022-12-13_21:58:18.511311)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:20.108051 run osmo-stp_10.42.42.5(pid=4975): stderr:
| �[0;m�[38;5;43m20221213215818683 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215818683 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215818683 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215818683 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215818684 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215818684 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215818684 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:20.233256 run osmo-stp_10.42.42.5(pid=4975): stdout:
| (launched: 2022-12-13_21:58:18.511311)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:20.357158 run osmo-stp_10.42.42.5(pid=4975): stderr:
| �[0;m�[38;5;43m20221213215818683 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215818683 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215818683 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215818683 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215818684 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215818684 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215818684 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:20.501600 run patchelf(pid=4978): Terminating (SIGINT)
21:58:20.671409 run patchelf(pid=4978): Terminated: ok {rc=0}
21:58:20.702523 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=4975): Process ended prematurely: osmo-stp_10.42.42.5(pid=4975) [trial-2395↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4975)] [trial-2395↪gprs:oc2g↪cs_paging_gprs_active.py:38]
21:58:20.711600 tst cs_paging_gprs_active.py:38: Test FAILED (8.6 sec)
21:58:20.771260 run osmo-hlr_10.42.42.2(pid=4972): ERR: Terminated: ERROR {rc=237} [trial-2395↪gprs:oc2g↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=4972)]
21:58:20.820048 run osmo-hlr_10.42.42.2(pid=4972): stdout:
| (launched: 2022-12-13_21:58:17.567219)
21:58:20.869444 run osmo-hlr_10.42.42.2(pid=4972): stderr:
| �[0;m�[1;31m20221213215817744 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213215817745 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213215817745 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213215817745 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213215817745 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213215817745 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213215817759 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2395/run.2022-12-13_21-44-31/gprs:oc2g/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213215817769 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213215817770 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213215817770 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:58:20.911257 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4965): Terminating (SIGTERM)
21:58:20.950706 run iperf3-srv_10.42.42.10(pid=4966): Terminating (SIGTERM)
21:58:20.990312 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4967): Terminating (SIGTERM)
21:58:21.029710 run iperf3-srv_10.42.42.10(pid=4968): Terminating (SIGTERM)
21:58:21.070950 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4971): Terminating (SIGTERM)
21:58:21.110531 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4974): Terminating (SIGTERM)
21:58:21.150207 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4977): Terminating (SIGTERM)
21:58:21.159460 --- ParallelTerminationStrategy: PID 4965 died...
21:58:21.212684 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4965): Terminated: ok {rc=0}
21:58:21.222353 --- ParallelTerminationStrategy: PID 4966 died...
21:58:21.274714 run iperf3-srv_10.42.42.10(pid=4966): Terminated {rc=256}
21:58:21.284070 --- ParallelTerminationStrategy: PID 4967 died...
21:58:21.336692 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4967): Terminated: ok {rc=0}
21:58:21.346388 --- ParallelTerminationStrategy: PID 4968 died...
21:58:21.398820 run iperf3-srv_10.42.42.10(pid=4968): Terminated {rc=256}
21:58:21.408295 --- ParallelTerminationStrategy: PID 4971 died...
21:58:21.460741 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4971): Terminated: ok {rc=0}
21:58:21.470369 --- ParallelTerminationStrategy: PID 4974 died...
21:58:21.522994 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4974): Terminated: ok {rc=0}
21:58:21.532487 --- ParallelTerminationStrategy: PID 4977 died...
21:58:21.585094 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4977): Terminated: ok {rc=0}
21:58:21.672713 bus /gobi_0: Setting Powered False
21:58:22.776213 bus /gobi_0: Setting Powered False
21:58:23.887013 bus /gobi_6: Setting Powered False
253gprs:oc2g8.0osmo-stp_10.42.42.5(pid=4987): Process ended prematurely: osmo-stp_10.42.42.5(pid=4987) [trial-2395↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4987)]2143iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=4987): Process ended prematurely: osmo-stp_10.42.42.5(pid=4987) [trial-2395↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4987)]
----------------------------------------------
trial-2395 gprs:oc2g iperf3.py
----------------------------------------------
21:58:24.997052 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:25.008047 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:58:25.139828 tst iperf3.py: using LAC 36493
21:58:25.268315 tst iperf3.py: using RAC 28
21:58:25.395900 tst iperf3.py: using CellId 36493
21:58:25.522730 tst iperf3.py: using BVCI 36494
21:58:25.547457 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:25.558494 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:25.569397 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:25.580179 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:25.591281 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:25.602364 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:25.613539 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:25.624397 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:25.635551 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:25.934336 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
21:58:26.071197 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:26.407920 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4979): Launched
21:58:26.688717 run iperf3-srv_10.42.42.10(pid=4980): Launched
21:58:26.888754 tst iperf3.py:8: start network...
21:58:27.025220 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:27.540683 run create_hlr_db(pid=4981): Launched
21:58:27.730209 bus /gobi_6: Setting Powered False
21:58:28.751062 run create_hlr_db(pid=4981): Terminated: ok {rc=0}
21:58:29.170505 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4983): Launched
21:58:29.498289 run osmo-hlr_10.42.42.2(pid=4984): Launched
21:58:29.619359 run osmo-stp_10.42.42.5: Starting osmo-stp
21:58:30.185622 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4986): Launched
21:58:30.512518 run osmo-stp_10.42.42.5(pid=4987): Launched
21:58:30.631460 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:58:31.232381 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4989): Launched
21:58:31.343573 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:58:31.640509 run patchelf(pid=4990): Launched
21:58:31.825462 run osmo-stp_10.42.42.5(pid=4987): ERR: Terminated: ERROR {rc=1} [trial-2395↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4987)]
21:58:31.959031 run osmo-stp_10.42.42.5(pid=4987): stdout:
| (launched: 2022-12-13_21:58:30.351373)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:32.092663 run osmo-stp_10.42.42.5(pid=4987): stderr:
| �[0;m�[38;5;43m20221213215830545 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215830545 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215830545 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215830546 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215830546 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215830546 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215830546 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:32.221463 run osmo-stp_10.42.42.5(pid=4987): stdout:
| (launched: 2022-12-13_21:58:30.351373)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:32.348295 run osmo-stp_10.42.42.5(pid=4987): stderr:
| �[0;m�[38;5;43m20221213215830545 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215830545 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215830545 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215830546 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215830546 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215830546 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215830546 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:32.494581 run patchelf(pid=4990): Terminating (SIGINT)
21:58:32.665455 run patchelf(pid=4990): Terminated: ok {rc=0}
21:58:32.696689 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=4987): Process ended prematurely: osmo-stp_10.42.42.5(pid=4987) [trial-2395↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4987)] [trial-2395↪gprs:oc2g↪iperf3.py:8]
21:58:32.705952 tst iperf3.py:8: Test FAILED (7.7 sec)
21:58:32.765835 run osmo-hlr_10.42.42.2(pid=4984): ERR: Terminated: ERROR {rc=237} [trial-2395↪gprs:oc2g↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=4984)]
21:58:32.814727 run osmo-hlr_10.42.42.2(pid=4984): stdout:
| (launched: 2022-12-13_21:58:29.336642)
21:58:32.864123 run osmo-hlr_10.42.42.2(pid=4984): stderr:
| �[0;m�[1;31m20221213215829536 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213215829536 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213215829536 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213215829536 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213215829536 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213215829536 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213215829548 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2395/run.2022-12-13_21-44-31/gprs:oc2g/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213215829558 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213215829558 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213215829558 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:58:32.906311 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4979): Terminating (SIGTERM)
21:58:32.946182 run iperf3-srv_10.42.42.10(pid=4980): Terminating (SIGTERM)
21:58:32.985727 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4983): Terminating (SIGTERM)
21:58:33.025307 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4986): Terminating (SIGTERM)
21:58:33.064986 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4989): Terminating (SIGTERM)
21:58:33.074213 --- ParallelTerminationStrategy: PID 4979 died...
21:58:33.127183 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4979): Terminated: ok {rc=0}
21:58:33.136821 --- ParallelTerminationStrategy: PID 4980 died...
21:58:33.189570 run iperf3-srv_10.42.42.10(pid=4980): Terminated {rc=256}
21:58:33.199161 --- ParallelTerminationStrategy: PID 4983 died...
21:58:33.252054 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4983): Terminated: ok {rc=0}
21:58:33.261714 --- ParallelTerminationStrategy: PID 4986 died...
21:58:33.314693 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4986): Terminated: ok {rc=0}
21:58:33.324272 --- ParallelTerminationStrategy: PID 4989 died...
21:58:33.377154 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4989): Terminated: ok {rc=0}
21:58:33.464270 bus /gobi_6: Setting Powered False
257gprs:oc2g11.0osmo-stp_10.42.42.5(pid=5005): Process ended prematurely: osmo-stp_10.42.42.5(pid=5005) [trial-2395↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5005)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=5005): Process ended prematurely: osmo-stp_10.42.42.5(pid=5005) [trial-2395↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5005)]
----------------------------------------------
trial-2395 gprs:oc2g iperf3m4.py
----------------------------------------------
21:58:34.579983 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:34.591575 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:58:34.719710 tst iperf3m4.py: using LAC 36494
21:58:34.845018 tst iperf3m4.py: using RAC 29
21:58:34.970963 tst iperf3m4.py: using CellId 36494
21:58:35.097376 tst iperf3m4.py: using BVCI 36495
21:58:35.122473 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:35.133411 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:35.144117 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:35.154967 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:35.166246 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:35.177507 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:35.188738 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:35.199857 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:35.211200 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:35.332214 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:35.450108 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:35.569322 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:35.863083 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
21:58:35.999340 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:36.339189 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4991): Launched
21:58:36.620527 run iperf3-srv_10.42.42.10(pid=4992): Launched
21:58:36.821013 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
21:58:36.958019 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:37.297249 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4993): Launched
21:58:37.579117 run iperf3-srv_10.42.42.10(pid=4994): Launched
21:58:37.779079 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
21:58:37.915945 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:38.254342 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4995): Launched
21:58:38.535259 run iperf3-srv_10.42.42.10(pid=4996): Launched
21:58:38.735067 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
21:58:38.871257 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:39.208749 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4997): Launched
21:58:39.489126 run iperf3-srv_10.42.42.10(pid=4998): Launched
21:58:39.688870 tst iperf3m4.py:8: start network...
21:58:39.825181 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:40.339765 run create_hlr_db(pid=4999): Launched
21:58:40.528782 bus /gobi_6: Setting Powered False
21:58:41.550845 run create_hlr_db(pid=4999): Terminated: ok {rc=0}
21:58:41.985743 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5001): Launched
21:58:42.313825 run osmo-hlr_10.42.42.2(pid=5002): Launched
21:58:42.434621 run osmo-stp_10.42.42.5: Starting osmo-stp
21:58:43.000140 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5004): Launched
21:58:43.328446 run osmo-stp_10.42.42.5(pid=5005): Launched
21:58:43.447370 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:58:44.019148 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5007): Launched
21:58:44.136918 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:58:44.421948 run patchelf(pid=5008): Launched
21:58:44.612428 run osmo-stp_10.42.42.5(pid=5005): ERR: Terminated: ERROR {rc=1} [trial-2395↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5005)]
21:58:44.751393 run osmo-stp_10.42.42.5(pid=5005): stdout:
| (launched: 2022-12-13_21:58:43.166806)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:44.891562 run osmo-stp_10.42.42.5(pid=5005): stderr:
| �[0;m�[38;5;43m20221213215843362 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215843362 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215843362 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215843363 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215843363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215843363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215843363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:45.026446 run osmo-stp_10.42.42.5(pid=5005): stdout:
| (launched: 2022-12-13_21:58:43.166806)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:45.160846 run osmo-stp_10.42.42.5(pid=5005): stderr:
| �[0;m�[38;5;43m20221213215843362 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215843362 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215843362 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215843363 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215843363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215843363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215843363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:45.316250 run patchelf(pid=5008): Terminating (SIGINT)
21:58:45.498566 run patchelf(pid=5008): Terminated: ok {rc=0}
21:58:45.531890 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=5005): Process ended prematurely: osmo-stp_10.42.42.5(pid=5005) [trial-2395↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5005)] [trial-2395↪gprs:oc2g↪iperf3m4.py:8]
21:58:45.541872 tst iperf3m4.py:8: Test FAILED (11.0 sec)
21:58:45.605368 run osmo-hlr_10.42.42.2(pid=5002): ERR: Terminated: ERROR {rc=237} [trial-2395↪gprs:oc2g↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=5002)]
21:58:45.656720 run osmo-hlr_10.42.42.2(pid=5002): stdout:
| (launched: 2022-12-13_21:58:42.151961)
21:58:45.708590 run osmo-hlr_10.42.42.2(pid=5002): stderr:
| �[0;m�[1;31m20221213215842351 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213215842351 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213215842351 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213215842351 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213215842351 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213215842351 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213215842364 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2395/run.2022-12-13_21-44-31/gprs:oc2g/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213215842372 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213215842372 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213215842373 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:58:45.752895 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4991): Terminating (SIGTERM)
21:58:45.794638 run iperf3-srv_10.42.42.10(pid=4992): Terminating (SIGTERM)
21:58:45.837796 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4993): Terminating (SIGTERM)
21:58:45.896261 run iperf3-srv_10.42.42.10(pid=4994): Terminating (SIGTERM)
21:58:45.939392 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4995): Terminating (SIGTERM)
21:58:45.981649 run iperf3-srv_10.42.42.10(pid=4996): Terminating (SIGTERM)
21:58:46.024174 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4997): Terminating (SIGTERM)
21:58:46.066872 run iperf3-srv_10.42.42.10(pid=4998): Terminating (SIGTERM)
21:58:46.109690 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5001): Terminating (SIGTERM)
21:58:46.152104 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5004): Terminating (SIGTERM)
21:58:46.194690 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5007): Terminating (SIGTERM)
21:58:46.204453 --- ParallelTerminationStrategy: PID 4991 died...
21:58:46.261496 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4991): Terminated: ok {rc=0}
21:58:46.271982 --- ParallelTerminationStrategy: PID 4992 died...
21:58:46.328797 run iperf3-srv_10.42.42.10(pid=4992): Terminated {rc=256}
21:58:46.339342 --- ParallelTerminationStrategy: PID 4993 died...
21:58:46.396223 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4993): Terminated: ok {rc=0}
21:58:46.406825 --- ParallelTerminationStrategy: PID 4994 died...
21:58:46.463926 run iperf3-srv_10.42.42.10(pid=4994): Terminated {rc=256}
21:58:46.474476 --- ParallelTerminationStrategy: PID 4995 died...
21:58:46.532032 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4995): Terminated: ok {rc=0}
21:58:46.546297 --- ParallelTerminationStrategy: PID 4996 died...
21:58:46.616101 run iperf3-srv_10.42.42.10(pid=4996): Terminated {rc=256}
21:58:46.630975 --- ParallelTerminationStrategy: PID 4997 died...
21:58:46.690906 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4997): Terminated: ok {rc=0}
21:58:46.701523 --- ParallelTerminationStrategy: PID 4998 died...
21:58:46.758423 run iperf3-srv_10.42.42.10(pid=4998): Terminated {rc=256}
21:58:46.769011 --- ParallelTerminationStrategy: PID 5001 died...
21:58:46.826164 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5001): Terminated: ok {rc=0}
21:58:46.836713 --- ParallelTerminationStrategy: PID 5004 died...
21:58:46.893934 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5004): Terminated: ok {rc=0}
21:58:46.904594 --- ParallelTerminationStrategy: PID 5007 died...
21:58:46.961572 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5007): Terminated: ok {rc=0}
21:58:47.054602 bus /gobi_4: Setting Powered False
21:58:48.159636 bus /gobi_4: Setting Powered False
21:58:49.267241 bus /gobi_1: Setting Powered False
21:58:50.372543 bus /gobi_1: Setting Powered False
21:58:51.479710 bus /gobi_0: Setting Powered False
21:58:52.585147 bus /gobi_0: Setting Powered False
21:58:53.694179 bus /gobi_6: Setting Powered False
257gprs:oc2g7.0osmo-stp_10.42.42.5(pid=5016): Process ended prematurely: osmo-stp_10.42.42.5(pid=5016) [trial-2395↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5016)]2139ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=5016): Process ended prematurely: osmo-stp_10.42.42.5(pid=5016) [trial-2395↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5016)]
----------------------------------------------
trial-2395 gprs:oc2g ping.py
----------------------------------------------
21:58:54.762552 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:54.774198 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:58:54.896122 tst ping.py: using LAC 36495
21:58:55.015513 tst ping.py: using RAC 30
21:58:55.135178 tst ping.py: using CellId 36495
21:58:55.254380 tst ping.py: using BVCI 36496
21:58:55.277359 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:55.287761 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:55.298161 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:55.308414 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:55.318688 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:55.328876 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:55.339333 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:55.349641 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:55.643086 tst ping.py:19: start network...
21:58:55.777811 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:56.282704 run create_hlr_db(pid=5010): Launched
21:58:56.468489 bus /gobi_6: Setting Powered False
21:58:57.490694 run create_hlr_db(pid=5010): Terminated: ok {rc=0}
21:58:57.917520 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5012): Launched
21:58:58.240766 run osmo-hlr_10.42.42.2(pid=5013): Launched
21:58:58.357606 run osmo-stp_10.42.42.5: Starting osmo-stp
21:58:58.916562 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5015): Launched
21:58:59.239496 run osmo-stp_10.42.42.5(pid=5016): Launched
21:58:59.356851 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:58:59.917342 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5018): Launched
21:59:00.032764 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:59:00.313322 run patchelf(pid=5019): Launched
21:59:00.500912 run osmo-stp_10.42.42.5(pid=5016): ERR: Terminated: ERROR {rc=1} [trial-2395↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5016)]
21:59:00.638240 run osmo-stp_10.42.42.5(pid=5016): stdout:
| (launched: 2022-12-13_21:58:59.080091)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:59:00.775757 run osmo-stp_10.42.42.5(pid=5016): stderr:
| �[0;m�[38;5;43m20221213215859272 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215859272 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215859272 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215859273 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215859273 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215859273 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215859273 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:59:00.931017 run osmo-stp_10.42.42.5(pid=5016): stdout:
| (launched: 2022-12-13_21:58:59.080091)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:59:01.058456 run osmo-stp_10.42.42.5(pid=5016): stderr:
| �[0;m�[38;5;43m20221213215859272 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215859272 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215859272 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215859273 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215859273 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215859273 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215859273 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:59:01.206779 run patchelf(pid=5019): Terminating (SIGINT)
21:59:01.380732 run patchelf(pid=5019): Terminated: ok {rc=0}
21:59:01.412833 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=5016): Process ended prematurely: osmo-stp_10.42.42.5(pid=5016) [trial-2395↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5016)] [trial-2395↪gprs:oc2g↪ping.py:22]
21:59:01.422851 tst ping.py:22: Test FAILED (6.6 sec)
21:59:01.487443 run osmo-hlr_10.42.42.2(pid=5013): ERR: Terminated: ERROR {rc=237} [trial-2395↪gprs:oc2g↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=5013)]
21:59:01.538396 run osmo-hlr_10.42.42.2(pid=5013): stdout:
| (launched: 2022-12-13_21:58:58.082120)
21:59:01.591177 run osmo-hlr_10.42.42.2(pid=5013): stderr:
| �[0;m�[1;31m20221213215858281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213215858281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213215858281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213215858281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213215858281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213215858281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213215858294 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2395/run.2022-12-13_21-44-31/gprs:oc2g/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213215858302 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213215858302 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213215858302 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:59:01.634928 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5012): Terminating (SIGTERM)
21:59:01.678794 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5015): Terminating (SIGTERM)
21:59:01.722097 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5018): Terminating (SIGTERM)
21:59:01.732814 --- ParallelTerminationStrategy: PID 5012 died...
21:59:01.790992 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5012): Terminated: ok {rc=0}
21:59:01.801354 --- ParallelTerminationStrategy: PID 5015 died...
21:59:01.859064 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5015): Terminated: ok {rc=0}
21:59:01.869293 --- ParallelTerminationStrategy: PID 5018 died...
21:59:01.924173 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5018): Terminated: ok {rc=0}
21:59:02.010380 bus /gobi_6: Setting Powered False
257gprs:oc2g7.0osmo-stp_10.42.42.5(pid=5026): Process ended prematurely: osmo-stp_10.42.42.5(pid=5026) [trial-2395↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5026)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=5026): Process ended prematurely: osmo-stp_10.42.42.5(pid=5026) [trial-2395↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5026)]
----------------------------------------------
trial-2395 gprs:oc2g ping_idle_ping.py
----------------------------------------------
21:59:03.061410 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:59:03.073417 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:59:03.202543 tst ping_idle_ping.py: using LAC 36496
21:59:03.327161 tst ping_idle_ping.py: using RAC 31
21:59:03.452304 tst ping_idle_ping.py: using CellId 36496
21:59:03.578085 tst ping_idle_ping.py: using BVCI 36497
21:59:03.602718 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:59:03.613643 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:59:03.624444 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:59:03.635200 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:59:03.646275 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:59:03.657358 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:59:03.668476 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:59:03.679606 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:59:03.955534 tst ping_idle_ping.py:24: start network...
21:59:04.080584 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:59:04.563683 run create_hlr_db(pid=5020): Launched
21:59:04.736366 bus /gobi_6: Setting Powered False
21:59:05.757906 run create_hlr_db(pid=5020): Terminated: ok {rc=0}
21:59:06.186309 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5022): Launched
21:59:06.508268 run osmo-hlr_10.42.42.2(pid=5023): Launched
21:59:06.627738 run osmo-stp_10.42.42.5: Starting osmo-stp
21:59:07.185666 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5025): Launched
21:59:07.508704 run osmo-stp_10.42.42.5(pid=5026): Launched
21:59:07.625783 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:59:08.188792 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5028): Launched
21:59:08.303619 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:59:08.584195 run patchelf(pid=5029): Launched
21:59:08.772281 run osmo-stp_10.42.42.5(pid=5026): ERR: Terminated: ERROR {rc=1} [trial-2395↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5026)]
21:59:08.910143 run osmo-stp_10.42.42.5(pid=5026): stdout:
| (launched: 2022-12-13_21:59:07.349088)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:59:09.047750 run osmo-stp_10.42.42.5(pid=5026): stderr:
| �[0;m�[38;5;43m20221213215907543 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215907544 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215907544 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215907544 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215907545 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215907545 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215907545 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:59:09.181932 run osmo-stp_10.42.42.5(pid=5026): stdout:
| (launched: 2022-12-13_21:59:07.349088)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:59:09.314291 run osmo-stp_10.42.42.5(pid=5026): stderr:
| �[0;m�[38;5;43m20221213215907543 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213215907544 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213215907544 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213215907544 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213215907545 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213215907545 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213215907545 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:59:09.467993 run patchelf(pid=5029): Terminating (SIGINT)
21:59:09.648052 run patchelf(pid=5029): Terminated: ok {rc=0}
21:59:09.680610 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=5026): Process ended prematurely: osmo-stp_10.42.42.5(pid=5026) [trial-2395↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5026)] [trial-2395↪gprs:oc2g↪ping_idle_ping.py:27]
21:59:09.689934 tst ping_idle_ping.py:27: Test FAILED (6.6 sec)
21:59:09.753292 run osmo-hlr_10.42.42.2(pid=5023): ERR: Terminated: ERROR {rc=237} [trial-2395↪gprs:oc2g↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=5023)]
21:59:09.804854 run osmo-hlr_10.42.42.2(pid=5023): stdout:
| (launched: 2022-12-13_21:59:06.349797)
21:59:09.856912 run osmo-hlr_10.42.42.2(pid=5023): stderr:
| �[0;m�[1;31m20221213215906547 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213215906547 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213215906547 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213215906547 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213215906547 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213215906547 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213215906559 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2395/run.2022-12-13_21-44-31/gprs:oc2g/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213215906567 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213215906568 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213215906568 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:59:09.901651 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5022): Terminating (SIGTERM)
21:59:09.943413 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5025): Terminating (SIGTERM)
21:59:09.985510 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5028): Terminating (SIGTERM)
21:59:09.995376 --- ParallelTerminationStrategy: PID 5022 died...
21:59:10.052083 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5022): Terminated: ok {rc=0}
21:59:10.062349 --- ParallelTerminationStrategy: PID 5025 died...
21:59:10.119496 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5025): Terminated: ok {rc=0}
21:59:10.129997 --- ParallelTerminationStrategy: PID 5028 died...
21:59:10.186285 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5028): Terminated: ok {rc=0}
21:59:10.273918 bus /gobi_6: Setting Powered False
5gprs:oc2g00