57gprs:trx-b2009.0osmo-stp_10.42.42.5(pid=2855): Process ended prematurely: osmo-stp_10.42.42.5(pid=2855) [trial-2374↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2855)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2855): Process ended prematurely: osmo-stp_10.42.42.5(pid=2855) [trial-2374↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2855)]
----------------------------------------------
trial-2374 gprs:trx-b200 cs_paging_gprs_active.py
----------------------------------------------
11:37:30.794549 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:30.805380 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
11:37:30.957505 tst cs_paging_gprs_active.py: using LAC 28126
11:37:31.096232 tst cs_paging_gprs_active.py: using RAC 76
11:37:31.222903 tst cs_paging_gprs_active.py: using CellId 28126
11:37:31.347407 tst cs_paging_gprs_active.py: using BVCI 28127
11:37:31.371615 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:31.383250 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:31.395641 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:31.408623 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:31.420450 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:31.431739 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:31.445783 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:31.456882 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:31.470021 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
11:37:31.618830 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
11:37:31.947774 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
11:37:32.084896 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:37:32.431464 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2840): Launched
11:37:32.708065 run iperf3-srv_10.42.42.10(pid=2841): Launched
11:37:32.905777 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
11:37:33.040572 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:37:33.375025 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2842): Launched
11:37:33.650962 run iperf3-srv_10.42.42.10(pid=2843): Launched
11:37:33.848318 tst cs_paging_gprs_active.py:38: start network...
11:37:33.982650 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:37:34.499302 run create_hlr_db(pid=2844): Launched
11:37:34.685751 bus /sierra_2: Setting Powered False
11:37:35.707275 run create_hlr_db(pid=2844): Terminated: ok {rc=0}
11:37:36.136429 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2851): Launched
11:37:36.458827 run osmo-hlr_10.42.42.2(pid=2852): Launched
11:37:36.575530 run osmo-stp_10.42.42.5: Starting osmo-stp
11:37:37.135186 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2854): Launched
11:37:37.457138 run osmo-stp_10.42.42.5(pid=2855): Launched
11:37:37.573923 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:37:38.135525 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2857): Launched
11:37:38.250331 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:37:38.531320 run patchelf(pid=2858): Launched
11:37:38.719854 run osmo-stp_10.42.42.5(pid=2855): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2855)]
11:37:38.858246 run osmo-stp_10.42.42.5(pid=2855): stdout:
| (launched: 2022-12-10_11:37:37.299517)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:37:38.996775 run osmo-stp_10.42.42.5(pid=2855): stderr:
| �[0;m�[38;5;43m20221210113737492 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113737492 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113737492 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113737493 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113737493 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113737493 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113737493 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:37:39.130552 run osmo-stp_10.42.42.5(pid=2855): stdout:
| (launched: 2022-12-10_11:37:37.299517)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:37:39.262843 run osmo-stp_10.42.42.5(pid=2855): stderr:
| �[0;m�[38;5;43m20221210113737492 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113737492 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113737492 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113737493 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113737493 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113737493 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113737493 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:37:39.417079 run patchelf(pid=2858): Terminating (SIGINT)
11:37:39.597358 run patchelf(pid=2858): Terminated: ok {rc=0}
11:37:39.630360 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=2855): Process ended prematurely: osmo-stp_10.42.42.5(pid=2855) [trial-2374↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2855)] [trial-2374↪gprs:trx-b200↪cs_paging_gprs_active.py:38]
11:37:39.640163 tst cs_paging_gprs_active.py:38: Test FAILED (8.9 sec)
11:37:39.702510 run osmo-hlr_10.42.42.2(pid=2852): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:trx-b200↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2852)]
11:37:39.753478 run osmo-hlr_10.42.42.2(pid=2852): stdout:
| (launched: 2022-12-10_11:37:36.301238)
11:37:39.804984 run osmo-hlr_10.42.42.2(pid=2852): stderr:
| �[0;m�[1;31m20221210113736499 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210113736499 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210113736500 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210113736500 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210113736500 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210113736500 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210113736511 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:trx-b200/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210113736520 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210113736520 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210113736520 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:37:39.848988 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2840): Terminating (SIGTERM)
11:37:39.890705 run iperf3-srv_10.42.42.10(pid=2841): Terminating (SIGTERM)
11:37:39.932374 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2842): Terminating (SIGTERM)
11:37:39.974035 run iperf3-srv_10.42.42.10(pid=2843): Terminating (SIGTERM)
11:37:40.015789 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2851): Terminating (SIGTERM)
11:37:40.057612 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2854): Terminating (SIGTERM)
11:37:40.099403 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2857): Terminating (SIGTERM)
11:37:40.109111 --- ParallelTerminationStrategy: PID 2840 died...
11:37:40.164822 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2840): Terminated: ok {rc=0}
11:37:40.174930 --- ParallelTerminationStrategy: PID 2841 died...
11:37:40.230379 run iperf3-srv_10.42.42.10(pid=2841): Terminated {rc=256}
11:37:40.240312 --- ParallelTerminationStrategy: PID 2842 died...
11:37:40.296054 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2842): Terminated: ok {rc=0}
11:37:40.306046 --- ParallelTerminationStrategy: PID 2843 died...
11:37:40.361528 run iperf3-srv_10.42.42.10(pid=2843): Terminated {rc=256}
11:37:40.371489 --- ParallelTerminationStrategy: PID 2851 died...
11:37:40.426976 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2851): Terminated: ok {rc=0}
11:37:40.436907 --- ParallelTerminationStrategy: PID 2854 died...
11:37:40.492547 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2854): Terminated: ok {rc=0}
11:37:40.502591 --- ParallelTerminationStrategy: PID 2857 died...
11:37:40.557924 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2857): Terminated: ok {rc=0}
11:37:40.647216 bus /gobi_0: Setting Powered False
11:37:41.754092 bus /gobi_0: Setting Powered False
11:37:42.861408 bus /gobi_6: Setting Powered False
57gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=2867): Process ended prematurely: osmo-stp_10.42.42.5(pid=2867) [trial-2374↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2867)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2867): Process ended prematurely: osmo-stp_10.42.42.5(pid=2867) [trial-2374↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2867)]
----------------------------------------------
trial-2374 gprs:trx-b200 iperf3.py
----------------------------------------------
11:37:43.970202 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:43.981294 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
11:37:44.123120 tst iperf3.py: using LAC 28127
11:37:44.239104 tst iperf3.py: using RAC 77
11:37:44.358821 tst iperf3.py: using CellId 28127
11:37:44.482368 tst iperf3.py: using BVCI 28128
11:37:44.507064 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:44.518221 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:44.529270 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:44.539952 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:44.550764 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:44.561615 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:44.572478 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:44.583067 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:44.594060 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
11:37:44.884891 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
11:37:45.019018 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:37:45.352737 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2859): Launched
11:37:45.629134 run iperf3-srv_10.42.42.10(pid=2860): Launched
11:37:45.826504 tst iperf3.py:8: start network...
11:37:45.963152 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:37:46.506874 run create_hlr_db(pid=2861): Launched
11:37:46.745848 bus /gobi_6: Setting Powered False
11:37:47.709524 run create_hlr_db(pid=2861): Terminated: ok {rc=0}
11:37:48.203531 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2863): Launched
11:37:48.525502 run osmo-hlr_10.42.42.2(pid=2864): Launched
11:37:48.646045 run osmo-stp_10.42.42.5: Starting osmo-stp
11:37:49.206433 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2866): Launched
11:37:49.528649 run osmo-stp_10.42.42.5(pid=2867): Launched
11:37:49.646047 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:37:50.207328 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2869): Launched
11:37:50.322416 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:37:50.602098 run patchelf(pid=2870): Launched
11:37:50.790522 run osmo-stp_10.42.42.5(pid=2867): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2867)]
11:37:50.928165 run osmo-stp_10.42.42.5(pid=2867): stdout:
| (launched: 2022-12-10_11:37:49.370729)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:37:51.066127 run osmo-stp_10.42.42.5(pid=2867): stderr:
| �[0;m�[38;5;43m20221210113749563 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113749563 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113749564 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113749564 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113749564 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113749565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113749565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:37:51.200296 run osmo-stp_10.42.42.5(pid=2867): stdout:
| (launched: 2022-12-10_11:37:49.370729)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:37:51.332670 run osmo-stp_10.42.42.5(pid=2867): stderr:
| �[0;m�[38;5;43m20221210113749563 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113749563 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113749564 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113749564 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113749564 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113749565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113749565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:37:51.486802 run patchelf(pid=2870): Terminating (SIGINT)
11:37:51.667297 run patchelf(pid=2870): Terminated: ok {rc=0}
11:37:51.700334 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=2867): Process ended prematurely: osmo-stp_10.42.42.5(pid=2867) [trial-2374↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2867)] [trial-2374↪gprs:trx-b200↪iperf3.py:8]
11:37:51.710106 tst iperf3.py:8: Test FAILED (7.8 sec)
11:37:51.773046 run osmo-hlr_10.42.42.2(pid=2864): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:trx-b200↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2864)]
11:37:51.824245 run osmo-hlr_10.42.42.2(pid=2864): stdout:
| (launched: 2022-12-10_11:37:48.368337)
11:37:51.875668 run osmo-hlr_10.42.42.2(pid=2864): stderr:
| �[0;m�[1;31m20221210113748567 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210113748567 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210113748568 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210113748568 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210113748568 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210113748568 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210113748579 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:trx-b200/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210113748589 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210113748589 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210113748589 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:37:51.919612 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2859): Terminating (SIGTERM)
11:37:51.961078 run iperf3-srv_10.42.42.10(pid=2860): Terminating (SIGTERM)
11:37:52.002545 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2863): Terminating (SIGTERM)
11:37:52.043999 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2866): Terminating (SIGTERM)
11:37:52.085657 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2869): Terminating (SIGTERM)
11:37:52.095163 --- ParallelTerminationStrategy: PID 2859 died...
11:37:52.150734 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2859): Terminated: ok {rc=0}
11:37:52.160673 --- ParallelTerminationStrategy: PID 2860 died...
11:37:52.215811 run iperf3-srv_10.42.42.10(pid=2860): Terminated {rc=256}
11:37:52.225794 --- ParallelTerminationStrategy: PID 2863 died...
11:37:52.280832 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2863): Terminated: ok {rc=0}
11:37:52.290798 --- ParallelTerminationStrategy: PID 2866 died...
11:37:52.346503 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2866): Terminated: ok {rc=0}
11:37:52.356420 --- ParallelTerminationStrategy: PID 2869 died...
11:37:52.411442 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2869): Terminated: ok {rc=0}
11:37:52.499160 bus /gobi_6: Setting Powered False
189gprs:trx-b20011.0osmo-stp_10.42.42.5(pid=2885): Process ended prematurely: osmo-stp_10.42.42.5(pid=2885) [trial-2374↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2885)]2186iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2885): Process ended prematurely: osmo-stp_10.42.42.5(pid=2885) [trial-2374↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2885)]
----------------------------------------------
trial-2374 gprs:trx-b200 iperf3m4.py
----------------------------------------------
11:37:53.614623 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:53.626130 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
11:37:53.766788 tst iperf3m4.py: using LAC 28128
11:37:53.894087 tst iperf3m4.py: using RAC 78
11:37:54.019306 tst iperf3m4.py: using CellId 28128
11:37:54.144078 tst iperf3m4.py: using BVCI 28129
11:37:54.168607 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:54.179576 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:54.190429 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:54.201198 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:54.211870 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:54.222613 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:54.233478 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:54.244172 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:37:54.255043 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
11:37:54.373738 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
11:37:54.491451 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
11:37:54.607049 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
11:37:54.884199 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
11:37:55.009618 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:37:55.321376 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2871): Launched
11:37:55.579883 run iperf3-srv_10.42.42.10(pid=2872): Launched
11:37:55.762645 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
11:37:55.887208 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:37:56.198204 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2873): Launched
11:37:56.455655 run iperf3-srv_10.42.42.10(pid=2874): Launched
11:37:56.638286 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
11:37:56.763014 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:37:57.072459 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2875): Launched
11:37:57.329975 run iperf3-srv_10.42.42.10(pid=2876): Launched
11:37:57.511983 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
11:37:57.636254 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:37:57.946537 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2877): Launched
11:37:58.204196 run iperf3-srv_10.42.42.10(pid=2878): Launched
11:37:58.386871 tst iperf3m4.py:8: start network...
11:37:58.511568 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:37:58.982721 run create_hlr_db(pid=2879): Launched
11:37:59.156498 bus /gobi_6: Setting Powered False
11:38:00.177536 run create_hlr_db(pid=2879): Terminated: ok {rc=0}
11:38:00.576910 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2881): Launched
11:38:00.911394 run osmo-hlr_10.42.42.2(pid=2882): Launched
11:38:01.073044 run osmo-stp_10.42.42.5: Starting osmo-stp
11:38:01.716172 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2884): Launched
11:38:02.258462 run osmo-stp_10.42.42.5(pid=2885): Launched
11:38:02.498252 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:38:03.059608 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2887): Launched
11:38:03.174477 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:38:03.456109 run patchelf(pid=2888): Launched
11:38:03.644290 run osmo-stp_10.42.42.5(pid=2885): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2885)]
11:38:03.782346 run osmo-stp_10.42.42.5(pid=2885): stdout:
| (launched: 2022-12-10_11:38:01.953129)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:38:03.920436 run osmo-stp_10.42.42.5(pid=2885): stderr:
| �[0;m�[38;5;43m20221210113802353 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113802354 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113802354 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113802356 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113802359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113802359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113802359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:38:04.054016 run osmo-stp_10.42.42.5(pid=2885): stdout:
| (launched: 2022-12-10_11:38:01.953129)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:38:04.185970 run osmo-stp_10.42.42.5(pid=2885): stderr:
| �[0;m�[38;5;43m20221210113802353 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113802354 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113802354 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113802356 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113802359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113802359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113802359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:38:04.340087 run patchelf(pid=2888): Terminating (SIGINT)
11:38:04.520658 run patchelf(pid=2888): Terminated: ok {rc=0}
11:38:04.553783 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=2885): Process ended prematurely: osmo-stp_10.42.42.5(pid=2885) [trial-2374↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2885)] [trial-2374↪gprs:trx-b200↪iperf3m4.py:8]
11:38:04.563480 tst iperf3m4.py:8: Test FAILED (11.0 sec)
11:38:04.626476 run osmo-hlr_10.42.42.2(pid=2882): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:trx-b200↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2882)]
11:38:04.677877 run osmo-hlr_10.42.42.2(pid=2882): stdout:
| (launched: 2022-12-10_11:38:00.729335)
11:38:04.729607 run osmo-hlr_10.42.42.2(pid=2882): stderr:
| �[0;m�[1;31m20221210113800973 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210113800973 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210113800973 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210113800973 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210113800973 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210113800974 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210113800996 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:trx-b200/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210113801021 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210113801021 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210113801021 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:38:04.773684 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2871): Terminating (SIGTERM)
11:38:04.815321 run iperf3-srv_10.42.42.10(pid=2872): Terminating (SIGTERM)
11:38:04.857018 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2873): Terminating (SIGTERM)
11:38:04.898585 run iperf3-srv_10.42.42.10(pid=2874): Terminating (SIGTERM)
11:38:04.940366 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2875): Terminating (SIGTERM)
11:38:04.982567 run iperf3-srv_10.42.42.10(pid=2876): Terminating (SIGTERM)
11:38:05.024199 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2877): Terminating (SIGTERM)
11:38:05.065999 run iperf3-srv_10.42.42.10(pid=2878): Terminating (SIGTERM)
11:38:05.107645 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2881): Terminating (SIGTERM)
11:38:05.149343 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2884): Terminating (SIGTERM)
11:38:05.190993 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2887): Terminating (SIGTERM)
11:38:05.200633 --- ParallelTerminationStrategy: PID 2871 died...
11:38:05.256555 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2871): Terminated: ok {rc=0}
11:38:05.266626 --- ParallelTerminationStrategy: PID 2872 died...
11:38:05.322685 run iperf3-srv_10.42.42.10(pid=2872): Terminated {rc=256}
11:38:05.332689 --- ParallelTerminationStrategy: PID 2873 died...
11:38:05.388476 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2873): Terminated: ok {rc=0}
11:38:05.398486 --- ParallelTerminationStrategy: PID 2874 died...
11:38:05.454048 run iperf3-srv_10.42.42.10(pid=2874): Terminated {rc=256}
11:38:05.464011 --- ParallelTerminationStrategy: PID 2875 died...
11:38:05.519713 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2875): Terminated: ok {rc=0}
11:38:05.529842 --- ParallelTerminationStrategy: PID 2876 died...
11:38:05.585664 run iperf3-srv_10.42.42.10(pid=2876): Terminated {rc=256}
11:38:05.595696 --- ParallelTerminationStrategy: PID 2877 died...
11:38:05.651383 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2877): Terminated: ok {rc=0}
11:38:05.661894 --- ParallelTerminationStrategy: PID 2878 died...
11:38:05.718104 run iperf3-srv_10.42.42.10(pid=2878): Terminated {rc=256}
11:38:05.729801 --- ParallelTerminationStrategy: PID 2881 died...
11:38:05.815792 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2881): Terminated: ok {rc=0}
11:38:05.825979 --- ParallelTerminationStrategy: PID 2884 died...
11:38:05.881117 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2884): Terminated: ok {rc=0}
11:38:05.891704 --- ParallelTerminationStrategy: PID 2887 died...
11:38:05.946010 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2887): Terminated: ok {rc=0}
11:38:06.037282 bus /gobi_4: Setting Powered False
11:38:07.141409 bus /gobi_4: Setting Powered False
11:38:08.253862 bus /gobi_1: Setting Powered False
11:38:09.357275 bus /gobi_1: Setting Powered False
11:38:10.468537 bus /gobi_0: Setting Powered False
11:38:11.570673 bus /gobi_0: Setting Powered False
11:38:12.683396 bus /gobi_6: Setting Powered False
189gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=3065): Process ended prematurely: osmo-stp_10.42.42.5(pid=3065) [trial-2374↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=3065)]2186ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=3065): Process ended prematurely: osmo-stp_10.42.42.5(pid=3065) [trial-2374↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=3065)]
----------------------------------------------
trial-2374 gprs:trx-b200 ping.py
----------------------------------------------
11:38:13.749746 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:13.760633 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
11:38:13.902499 tst ping.py: using LAC 28129
11:38:14.025846 tst ping.py: using RAC 79
11:38:14.151970 tst ping.py: using CellId 28129
11:38:14.275875 tst ping.py: using BVCI 28130
11:38:14.300160 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:14.311520 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:14.322484 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:14.333365 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:14.344459 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:14.355225 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:14.366231 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:14.377130 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
11:38:14.647520 tst ping.py:19: start network...
11:38:14.770667 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:38:15.235980 run create_hlr_db(pid=2891): Launched
11:38:15.407389 bus /gobi_6: Setting Powered False
11:38:16.442521 run create_hlr_db(pid=2891): Terminated: ok {rc=0}
11:38:16.956514 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2921): Launched
11:38:17.435595 run osmo-hlr_10.42.42.2(pid=2930): Launched
11:38:17.719363 run osmo-stp_10.42.42.5: Starting osmo-stp
11:38:18.472533 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2993): Launched
11:38:18.970480 run osmo-stp_10.42.42.5(pid=3065): Launched
11:38:19.220585 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:38:19.892018 run pcap-recorder_any(filters='host 10.42.42.6')(pid=3168): Launched
11:38:20.007806 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:38:20.305515 run patchelf(pid=3170): Launched
11:38:20.497908 run osmo-stp_10.42.42.5(pid=3065): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=3065)]
11:38:20.638644 run osmo-stp_10.42.42.5(pid=3065): stdout:
| (launched: 2022-12-10_11:38:18.667353)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:38:20.777335 run osmo-stp_10.42.42.5(pid=3065): stderr:
| �[0;m�[38;5;43m20221210113818976 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113818976 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113818985 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113818985 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113818986 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113818986 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113818986 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:38:20.911335 run osmo-stp_10.42.42.5(pid=3065): stdout:
| (launched: 2022-12-10_11:38:18.667353)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:38:21.045719 run osmo-stp_10.42.42.5(pid=3065): stderr:
| �[0;m�[38;5;43m20221210113818976 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113818976 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113818985 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113818985 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113818986 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113818986 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113818986 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:38:21.199056 run patchelf(pid=3170): Terminating (SIGINT)
11:38:21.382189 run patchelf(pid=3170): Terminated: ok {rc=0}
11:38:21.416592 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=3065): Process ended prematurely: osmo-stp_10.42.42.5(pid=3065) [trial-2374↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=3065)] [trial-2374↪gprs:trx-b200↪ping.py:22]
11:38:21.427154 tst ping.py:22: Test FAILED (7.7 sec)
11:38:21.491370 run osmo-hlr_10.42.42.2(pid=2930): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:trx-b200↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2930)]
11:38:21.543624 run osmo-hlr_10.42.42.2(pid=2930): stdout:
| (launched: 2022-12-10_11:38:17.155861)
11:38:21.596642 run osmo-hlr_10.42.42.2(pid=2930): stderr:
| �[0;m�[1;31m20221210113817439 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210113817439 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210113817439 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210113817439 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210113817439 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210113817439 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210113817455 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:trx-b200/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210113817467 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210113817468 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210113817468 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:38:21.642309 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2921): Terminating (SIGTERM)
11:38:21.687062 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2993): Terminating (SIGTERM)
11:38:21.729492 run pcap-recorder_any(filters='host 10.42.42.6')(pid=3168): Terminating (SIGTERM)
11:38:21.740122 --- ParallelTerminationStrategy: PID 2921 died...
11:38:21.797615 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2921): Terminated: ok {rc=0}
11:38:21.809476 --- ParallelTerminationStrategy: PID 2993 died...
11:38:21.865726 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2993): Terminated: ok {rc=0}
11:38:21.876561 --- ParallelTerminationStrategy: PID 3168 died...
11:38:21.932481 run pcap-recorder_any(filters='host 10.42.42.6')(pid=3168): Terminated: ok {rc=0}
11:38:22.023164 bus /gobi_6: Setting Powered False
189gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=3179): Process ended prematurely: osmo-stp_10.42.42.5(pid=3179) [trial-2374↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=3179)]2186ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=3179): Process ended prematurely: osmo-stp_10.42.42.5(pid=3179) [trial-2374↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=3179)]
----------------------------------------------
trial-2374 gprs:trx-b200 ping_idle_ping.py
----------------------------------------------
11:38:23.077533 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:23.089898 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
11:38:23.235808 tst ping_idle_ping.py: using LAC 28130
11:38:23.366760 tst ping_idle_ping.py: using RAC 80
11:38:23.490210 tst ping_idle_ping.py: using CellId 28130
11:38:23.615206 tst ping_idle_ping.py: using BVCI 28131
11:38:23.640266 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:23.651915 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:23.663610 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:23.674947 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:23.686134 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:23.697420 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:23.708933 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
11:38:23.720203 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
11:38:24.012128 tst ping_idle_ping.py:24: start network...
11:38:24.146441 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:38:24.707243 run create_hlr_db(pid=3173): Launched
11:38:24.896593 bus /gobi_6: Setting Powered False
11:38:25.917789 run create_hlr_db(pid=3173): Terminated: ok {rc=0}
11:38:26.367619 run pcap-recorder_any(filters='host 10.42.42.2')(pid=3175): Launched
11:38:26.796510 run osmo-hlr_10.42.42.2(pid=3176): Launched
11:38:27.043313 run osmo-stp_10.42.42.5: Starting osmo-stp
11:38:27.625690 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=3178): Launched
11:38:28.055745 run osmo-stp_10.42.42.5(pid=3179): Launched
11:38:28.284717 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:38:28.870226 run pcap-recorder_any(filters='host 10.42.42.6')(pid=3181): Launched
11:38:28.988095 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:38:29.291548 run patchelf(pid=3182): Launched
11:38:29.484848 run osmo-stp_10.42.42.5(pid=3179): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=3179)]
11:38:29.625959 run osmo-stp_10.42.42.5(pid=3179): stdout:
| (launched: 2022-12-10_11:38:27.790775)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:38:29.765865 run osmo-stp_10.42.42.5(pid=3179): stderr:
| �[0;m�[38;5;43m20221210113828087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113828087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113828087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113828088 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113828088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113828088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113828088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:38:29.901875 run osmo-stp_10.42.42.5(pid=3179): stdout:
| (launched: 2022-12-10_11:38:27.790775)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:38:30.038110 run osmo-stp_10.42.42.5(pid=3179): stderr:
| �[0;m�[38;5;43m20221210113828087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210113828087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210113828087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210113828088 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210113828088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210113828088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210113828088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:38:30.194136 run patchelf(pid=3182): Terminating (SIGINT)
11:38:30.379319 run patchelf(pid=3182): Terminated: ok {rc=0}
11:38:30.413813 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=3179): Process ended prematurely: osmo-stp_10.42.42.5(pid=3179) [trial-2374↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=3179)] [trial-2374↪gprs:trx-b200↪ping_idle_ping.py:27]
11:38:30.424325 tst ping_idle_ping.py:27: Test FAILED (7.3 sec)
11:38:30.489746 run osmo-hlr_10.42.42.2(pid=3176): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:trx-b200↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=3176)]
11:38:30.543318 run osmo-hlr_10.42.42.2(pid=3176): stdout:
| (launched: 2022-12-10_11:38:26.533491)
11:38:30.597725 run osmo-hlr_10.42.42.2(pid=3176): stderr:
| �[0;m�[1;31m20221210113826843 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210113826844 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210113826844 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210113826844 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210113826844 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210113826844 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210113826862 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:trx-b200/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210113826878 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210113826879 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210113826879 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:38:30.645099 run pcap-recorder_any(filters='host 10.42.42.2')(pid=3175): Terminating (SIGTERM)
11:38:30.691061 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=3178): Terminating (SIGTERM)
11:38:30.736933 run pcap-recorder_any(filters='host 10.42.42.6')(pid=3181): Terminating (SIGTERM)
11:38:30.747821 --- ParallelTerminationStrategy: PID 3175 died...
11:38:30.806957 run pcap-recorder_any(filters='host 10.42.42.2')(pid=3175): Terminated: ok {rc=0}
11:38:30.819020 --- ParallelTerminationStrategy: PID 3178 died...
11:38:30.881013 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=3178): Terminated: ok {rc=0}
11:38:30.891686 --- ParallelTerminationStrategy: PID 3181 died...
11:38:30.954699 run pcap-recorder_any(filters='host 10.42.42.6')(pid=3181): Terminated: ok {rc=0}
11:38:31.049598 bus /gobi_6: Setting Powered False
5gprs:trx-b20000