76gprs:trx-umtrx9.0osmo-stp_10.42.42.5(pid=21597): Process ended prematurely: osmo-stp_10.42.42.5(pid=21597) [trial-2393↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21597)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21597): Process ended prematurely: osmo-stp_10.42.42.5(pid=21597) [trial-2393↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21597)]
----------------------------------------------
trial-2393 gprs:trx-umtrx cs_paging_gprs_active.py
----------------------------------------------
19:18:08.719790 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:08.730565 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
19:18:08.864458 tst cs_paging_gprs_active.py: using LAC 35707
19:18:08.989181 tst cs_paging_gprs_active.py: using RAC 7
19:18:09.107442 tst cs_paging_gprs_active.py: using CellId 35707
19:18:09.235603 tst cs_paging_gprs_active.py: using BVCI 35708
19:18:09.261126 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:09.272356 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:09.283672 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:09.294983 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:09.306400 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:09.317797 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:09.329296 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:09.340419 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:09.351947 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
19:18:09.469469 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
19:18:09.751704 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
19:18:09.879868 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:18:10.198539 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21587): Launched
19:18:10.462691 run iperf3-srv_10.42.42.10(pid=21588): Launched
19:18:10.650000 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
19:18:10.777615 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:18:11.095304 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21589): Launched
19:18:11.359494 run iperf3-srv_10.42.42.10(pid=21590): Launched
19:18:11.547646 tst cs_paging_gprs_active.py:38: start network...
19:18:11.675288 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:18:12.167768 run create_hlr_db(pid=21591): Launched
19:18:12.345092 bus /sierra_2: Setting Powered False
19:18:13.367089 run create_hlr_db(pid=21591): Terminated: ok {rc=0}
19:18:13.806066 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21593): Launched
19:18:14.137134 run osmo-hlr_10.42.42.2(pid=21594): Launched
19:18:14.260838 run osmo-stp_10.42.42.5: Starting osmo-stp
19:18:14.836145 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21596): Launched
19:18:15.166455 run osmo-stp_10.42.42.5(pid=21597): Launched
19:18:15.287408 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:18:15.861999 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21599): Launched
19:18:15.996865 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:18:16.301924 run patchelf(pid=21600): Launched
19:18:16.497938 run osmo-stp_10.42.42.5(pid=21597): ERR: Terminated: ERROR {rc=1} [trial-2393↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21597)]
19:18:16.642619 run osmo-stp_10.42.42.5(pid=21597): stdout:
| (launched: 2022-12-13_19:18:15.004255)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:18:16.787527 run osmo-stp_10.42.42.5(pid=21597): stderr:
| �[0;m�[38;5;43m20221213191815194 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191815194 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191815194 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191815194 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191815195 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191815195 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191815195 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:18:16.927287 run osmo-stp_10.42.42.5(pid=21597): stdout:
| (launched: 2022-12-13_19:18:15.004255)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:18:17.066865 run osmo-stp_10.42.42.5(pid=21597): stderr:
| �[0;m�[38;5;43m20221213191815194 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191815194 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191815194 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191815194 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191815195 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191815195 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191815195 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:18:17.228170 run patchelf(pid=21600): Terminating (SIGINT)
19:18:17.414431 run patchelf(pid=21600): Terminated: ok {rc=0}
19:18:17.448104 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=21597): Process ended prematurely: osmo-stp_10.42.42.5(pid=21597) [trial-2393↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21597)] [trial-2393↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38]
19:18:17.458038 tst cs_paging_gprs_active.py:38: Test FAILED (8.8 sec)
19:18:17.523440 run osmo-hlr_10.42.42.2(pid=21594): ERR: Terminated: ERROR {rc=237} [trial-2393↪gprs:trx-umtrx↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21594)]
19:18:17.576610 run osmo-hlr_10.42.42.2(pid=21594): stdout:
| (launched: 2022-12-13_19:18:13.974548)
19:18:17.630035 run osmo-hlr_10.42.42.2(pid=21594): stderr:
| �[0;m�[1;31m20221213191814169 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213191814169 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213191814170 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213191814170 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213191814170 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213191814170 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213191814182 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2393/run.2022-12-13_18-38-33/gprs:trx-umtrx/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213191814190 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213191814190 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213191814190 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:18:17.676026 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21587): Terminating (SIGTERM)
19:18:17.719258 run iperf3-srv_10.42.42.10(pid=21588): Terminating (SIGTERM)
19:18:17.762425 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21589): Terminating (SIGTERM)
19:18:17.805324 run iperf3-srv_10.42.42.10(pid=21590): Terminating (SIGTERM)
19:18:17.848462 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21593): Terminating (SIGTERM)
19:18:17.891862 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21596): Terminating (SIGTERM)
19:18:17.935029 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21599): Terminating (SIGTERM)
19:18:17.944863 --- ParallelTerminationStrategy: PID 21587 died...
19:18:18.002237 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21587): Terminated: ok {rc=0}
19:18:18.012426 --- ParallelTerminationStrategy: PID 21588 died...
19:18:18.069388 run iperf3-srv_10.42.42.10(pid=21588): Terminated {rc=256}
19:18:18.079548 --- ParallelTerminationStrategy: PID 21589 died...
19:18:18.136828 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21589): Terminated: ok {rc=0}
19:18:18.147119 --- ParallelTerminationStrategy: PID 21590 died...
19:18:18.204374 run iperf3-srv_10.42.42.10(pid=21590): Terminated {rc=256}
19:18:18.214631 --- ParallelTerminationStrategy: PID 21593 died...
19:18:18.272020 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21593): Terminated: ok {rc=0}
19:18:18.282352 --- ParallelTerminationStrategy: PID 21596 died...
19:18:18.339530 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21596): Terminated: ok {rc=0}
19:18:18.349870 --- ParallelTerminationStrategy: PID 21599 died...
19:18:18.406970 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21599): Terminated: ok {rc=0}
19:18:18.494849 bus /gobi_0: Setting Powered False
19:18:19.602321 bus /gobi_0: Setting Powered False
19:18:20.711565 bus /gobi_6: Setting Powered False
76gprs:trx-umtrx8.0osmo-stp_10.42.42.5(pid=21609): Process ended prematurely: osmo-stp_10.42.42.5(pid=21609) [trial-2393↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21609)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21609): Process ended prematurely: osmo-stp_10.42.42.5(pid=21609) [trial-2393↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21609)]
----------------------------------------------
trial-2393 gprs:trx-umtrx iperf3.py
----------------------------------------------
19:18:21.828439 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:21.840441 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
19:18:21.988502 tst iperf3.py: using LAC 35708
19:18:22.116799 tst iperf3.py: using RAC 8
19:18:22.245848 tst iperf3.py: using CellId 35708
19:18:22.379979 tst iperf3.py: using BVCI 35709
19:18:22.405199 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:22.416420 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:22.427455 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:22.438481 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:22.449728 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:22.460852 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:22.472323 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:22.483572 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:22.494989 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
19:18:22.777421 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
19:18:22.905012 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:18:23.221992 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21601): Launched
19:18:23.486042 run iperf3-srv_10.42.42.10(pid=21602): Launched
19:18:23.673330 tst iperf3.py:8: start network...
19:18:23.800820 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:18:24.291895 run create_hlr_db(pid=21603): Launched
19:18:24.468179 bus /gobi_6: Setting Powered False
19:18:25.489464 run create_hlr_db(pid=21603): Terminated: ok {rc=0}
19:18:25.927363 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21605): Launched
19:18:26.257525 run osmo-hlr_10.42.42.2(pid=21606): Launched
19:18:26.382564 run osmo-stp_10.42.42.5: Starting osmo-stp
19:18:26.953060 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21608): Launched
19:18:27.281524 run osmo-stp_10.42.42.5(pid=21609): Launched
19:18:27.401263 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:18:27.972561 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21611): Launched
19:18:28.090276 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:18:28.376889 run patchelf(pid=21612): Launched
19:18:28.569565 run osmo-stp_10.42.42.5(pid=21609): ERR: Terminated: ERROR {rc=1} [trial-2393↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21609)]
19:18:28.710794 run osmo-stp_10.42.42.5(pid=21609): stdout:
| (launched: 2022-12-13_19:18:27.120132)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:18:28.852231 run osmo-stp_10.42.42.5(pid=21609): stderr:
| �[0;m�[38;5;43m20221213191827309 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191827309 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191827309 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191827310 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191827310 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191827310 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191827310 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:18:28.988933 run osmo-stp_10.42.42.5(pid=21609): stdout:
| (launched: 2022-12-13_19:18:27.120132)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:18:29.124455 run osmo-stp_10.42.42.5(pid=21609): stderr:
| �[0;m�[38;5;43m20221213191827309 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191827309 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191827309 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191827310 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191827310 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191827310 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191827310 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:18:29.282089 run patchelf(pid=21612): Terminating (SIGINT)
19:18:29.466508 run patchelf(pid=21612): Terminated: ok {rc=0}
19:18:29.499955 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=21609): Process ended prematurely: osmo-stp_10.42.42.5(pid=21609) [trial-2393↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21609)] [trial-2393↪gprs:trx-umtrx↪iperf3.py:8]
19:18:29.509740 tst iperf3.py:8: Test FAILED (7.7 sec)
19:18:29.573507 run osmo-hlr_10.42.42.2(pid=21606): ERR: Terminated: ERROR {rc=237} [trial-2393↪gprs:trx-umtrx↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21606)]
19:18:29.625596 run osmo-hlr_10.42.42.2(pid=21606): stdout:
| (launched: 2022-12-13_19:18:26.095554)
19:18:29.678404 run osmo-hlr_10.42.42.2(pid=21606): stderr:
| �[0;m�[1;31m20221213191826291 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213191826291 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213191826291 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213191826291 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213191826291 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213191826291 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213191826303 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2393/run.2022-12-13_18-38-33/gprs:trx-umtrx/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213191826314 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213191826315 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213191826315 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:18:29.723309 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21601): Terminating (SIGTERM)
19:18:29.766050 run iperf3-srv_10.42.42.10(pid=21602): Terminating (SIGTERM)
19:18:29.808693 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21605): Terminating (SIGTERM)
19:18:29.851289 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21608): Terminating (SIGTERM)
19:18:29.894086 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21611): Terminating (SIGTERM)
19:18:29.903961 --- ParallelTerminationStrategy: PID 21601 died...
19:18:29.960772 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21601): Terminated: ok {rc=0}
19:18:29.971017 --- ParallelTerminationStrategy: PID 21602 died...
19:18:30.027479 run iperf3-srv_10.42.42.10(pid=21602): Terminated {rc=256}
19:18:30.038025 --- ParallelTerminationStrategy: PID 21605 died...
19:18:30.094460 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21605): Terminated: ok {rc=0}
19:18:30.104682 --- ParallelTerminationStrategy: PID 21608 died...
19:18:30.161188 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21608): Terminated: ok {rc=0}
19:18:30.171393 --- ParallelTerminationStrategy: PID 21611 died...
19:18:30.227873 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21611): Terminated: ok {rc=0}
19:18:30.314352 bus /gobi_6: Setting Powered False
219gprs:trx-umtrx11.0osmo-stp_10.42.42.5(pid=21627): Process ended prematurely: osmo-stp_10.42.42.5(pid=21627) [trial-2393↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21627)]2175iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21627): Process ended prematurely: osmo-stp_10.42.42.5(pid=21627) [trial-2393↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21627)]
----------------------------------------------
trial-2393 gprs:trx-umtrx iperf3m4.py
----------------------------------------------
19:18:31.449469 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:31.465849 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
19:18:31.610348 tst iperf3m4.py: using LAC 35709
19:18:31.766019 tst iperf3m4.py: using RAC 9
19:18:31.935965 tst iperf3m4.py: using CellId 35709
19:18:32.090950 tst iperf3m4.py: using BVCI 35710
19:18:32.124444 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:32.139628 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:32.152780 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:32.171954 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:32.187518 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:32.202856 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:32.218242 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:32.232074 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:32.246071 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
19:18:32.373425 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
19:18:32.499922 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
19:18:32.620274 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
19:18:32.905155 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
19:18:33.033634 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:18:33.351405 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21613): Launched
19:18:33.615607 run iperf3-srv_10.42.42.10(pid=21614): Launched
19:18:33.803025 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
19:18:33.930467 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:18:34.248008 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21615): Launched
19:18:34.511911 run iperf3-srv_10.42.42.10(pid=21616): Launched
19:18:34.698868 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
19:18:34.826774 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:18:35.144355 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21617): Launched
19:18:35.408281 run iperf3-srv_10.42.42.10(pid=21618): Launched
19:18:35.596195 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
19:18:35.724596 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:18:36.042461 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21619): Launched
19:18:36.306443 run iperf3-srv_10.42.42.10(pid=21620): Launched
19:18:36.493353 tst iperf3m4.py:8: start network...
19:18:36.620997 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:18:37.112461 run create_hlr_db(pid=21621): Launched
19:18:37.288722 bus /gobi_6: Setting Powered False
19:18:38.309983 run create_hlr_db(pid=21621): Terminated: ok {rc=0}
19:18:38.745938 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21623): Launched
19:18:39.074332 run osmo-hlr_10.42.42.2(pid=21624): Launched
19:18:39.195982 run osmo-stp_10.42.42.5: Starting osmo-stp
19:18:39.769119 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21626): Launched
19:18:40.098780 run osmo-stp_10.42.42.5(pid=21627): Launched
19:18:40.218567 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:18:40.792001 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21629): Launched
19:18:40.909810 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:18:41.196220 run patchelf(pid=21630): Launched
19:18:41.388757 run osmo-stp_10.42.42.5(pid=21627): ERR: Terminated: ERROR {rc=1} [trial-2393↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21627)]
19:18:41.529681 run osmo-stp_10.42.42.5(pid=21627): stdout:
| (launched: 2022-12-13_19:18:39.937392)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:18:41.670864 run osmo-stp_10.42.42.5(pid=21627): stderr:
| �[0;m�[38;5;43m20221213191840126 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191840126 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191840126 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191840127 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191840127 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191840127 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191840127 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:18:41.807594 run osmo-stp_10.42.42.5(pid=21627): stdout:
| (launched: 2022-12-13_19:18:39.937392)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:18:41.943226 run osmo-stp_10.42.42.5(pid=21627): stderr:
| �[0;m�[38;5;43m20221213191840126 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191840126 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191840126 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191840127 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191840127 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191840127 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191840127 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:18:42.100461 run patchelf(pid=21630): Terminating (SIGINT)
19:18:42.285358 run patchelf(pid=21630): Terminated: ok {rc=0}
19:18:42.319112 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=21627): Process ended prematurely: osmo-stp_10.42.42.5(pid=21627) [trial-2393↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21627)] [trial-2393↪gprs:trx-umtrx↪iperf3m4.py:8]
19:18:42.328867 tst iperf3m4.py:8: Test FAILED (10.9 sec)
19:18:42.393112 run osmo-hlr_10.42.42.2(pid=21624): ERR: Terminated: ERROR {rc=237} [trial-2393↪gprs:trx-umtrx↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21624)]
19:18:42.445031 run osmo-hlr_10.42.42.2(pid=21624): stdout:
| (launched: 2022-12-13_19:18:38.913707)
19:18:42.497585 run osmo-hlr_10.42.42.2(pid=21624): stderr:
| �[0;m�[1;31m20221213191839109 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213191839109 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213191839109 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213191839109 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213191839109 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213191839109 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213191839122 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2393/run.2022-12-13_18-38-33/gprs:trx-umtrx/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213191839131 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213191839131 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213191839131 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:18:42.542651 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21613): Terminating (SIGTERM)
19:18:42.585397 run iperf3-srv_10.42.42.10(pid=21614): Terminating (SIGTERM)
19:18:42.627871 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21615): Terminating (SIGTERM)
19:18:42.670551 run iperf3-srv_10.42.42.10(pid=21616): Terminating (SIGTERM)
19:18:42.713241 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21617): Terminating (SIGTERM)
19:18:42.755691 run iperf3-srv_10.42.42.10(pid=21618): Terminating (SIGTERM)
19:18:42.798110 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21619): Terminating (SIGTERM)
19:18:42.840392 run iperf3-srv_10.42.42.10(pid=21620): Terminating (SIGTERM)
19:18:42.882842 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21623): Terminating (SIGTERM)
19:18:42.925075 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21626): Terminating (SIGTERM)
19:18:42.967376 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21629): Terminating (SIGTERM)
19:18:42.977221 --- ParallelTerminationStrategy: PID 21613 died...
19:18:43.034536 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21613): Terminated: ok {rc=0}
19:18:43.044812 --- ParallelTerminationStrategy: PID 21614 died...
19:18:43.101615 run iperf3-srv_10.42.42.10(pid=21614): Terminated {rc=256}
19:18:43.111752 --- ParallelTerminationStrategy: PID 21615 died...
19:18:43.168475 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21615): Terminated: ok {rc=0}
19:18:43.178742 --- ParallelTerminationStrategy: PID 21616 died...
19:18:43.235140 run iperf3-srv_10.42.42.10(pid=21616): Terminated {rc=256}
19:18:43.246014 --- ParallelTerminationStrategy: PID 21617 died...
19:18:43.303066 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21617): Terminated: ok {rc=0}
19:18:43.313494 --- ParallelTerminationStrategy: PID 21618 died...
19:18:43.370130 run iperf3-srv_10.42.42.10(pid=21618): Terminated {rc=256}
19:18:43.380161 --- ParallelTerminationStrategy: PID 21619 died...
19:18:43.436421 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21619): Terminated: ok {rc=0}
19:18:43.446568 --- ParallelTerminationStrategy: PID 21620 died...
19:18:43.502877 run iperf3-srv_10.42.42.10(pid=21620): Terminated {rc=256}
19:18:43.513200 --- ParallelTerminationStrategy: PID 21623 died...
19:18:43.569674 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21623): Terminated: ok {rc=0}
19:18:43.579979 --- ParallelTerminationStrategy: PID 21626 died...
19:18:43.636342 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21626): Terminated: ok {rc=0}
19:18:43.646560 --- ParallelTerminationStrategy: PID 21629 died...
19:18:43.703159 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21629): Terminated: ok {rc=0}
19:18:43.790594 bus /gobi_4: Setting Powered False
19:18:44.897287 bus /gobi_4: Setting Powered False
19:18:46.032652 bus /gobi_1: Setting Powered False
19:18:47.134217 bus /gobi_1: Setting Powered False
19:18:48.240263 bus /gobi_0: Setting Powered False
19:18:49.347051 bus /gobi_0: Setting Powered False
19:18:50.455598 bus /gobi_6: Setting Powered False
194gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=21637): Process ended prematurely: osmo-stp_10.42.42.5(pid=21637) [trial-2393↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21637)]2200ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21637): Process ended prematurely: osmo-stp_10.42.42.5(pid=21637) [trial-2393↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21637)]
----------------------------------------------
trial-2393 gprs:trx-umtrx ping.py
----------------------------------------------
19:18:51.526084 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:51.538086 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
19:18:51.684119 tst ping.py: using LAC 35710
19:18:51.810049 tst ping.py: using RAC 10
19:18:51.938312 tst ping.py: using CellId 35710
19:18:52.069263 tst ping.py: using BVCI 35711
19:18:52.094086 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:52.105251 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:52.116214 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:52.127211 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:52.138931 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:52.150673 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:52.162652 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:52.174627 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
19:18:52.451971 tst ping.py:19: start network...
19:18:52.577831 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:18:53.060459 run create_hlr_db(pid=21631): Launched
19:18:53.233865 bus /gobi_6: Setting Powered False
19:18:54.255110 run create_hlr_db(pid=21631): Terminated: ok {rc=0}
19:18:54.684699 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21633): Launched
19:18:55.007895 run osmo-hlr_10.42.42.2(pid=21634): Launched
19:18:55.125570 run osmo-stp_10.42.42.5: Starting osmo-stp
19:18:55.687921 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21636): Launched
19:18:56.012034 run osmo-stp_10.42.42.5(pid=21637): Launched
19:18:56.129967 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:18:56.694407 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21639): Launched
19:18:56.810448 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:18:57.092212 run patchelf(pid=21640): Launched
19:18:57.282384 run osmo-stp_10.42.42.5(pid=21637): ERR: Terminated: ERROR {rc=1} [trial-2393↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21637)]
19:18:57.420599 run osmo-stp_10.42.42.5(pid=21637): stdout:
| (launched: 2022-12-13_19:18:55.852835)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:18:57.559329 run osmo-stp_10.42.42.5(pid=21637): stderr:
| �[0;m�[38;5;43m20221213191856040 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191856040 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191856040 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191856041 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191856041 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191856041 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191856041 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:18:57.693567 run osmo-stp_10.42.42.5(pid=21637): stdout:
| (launched: 2022-12-13_19:18:55.852835)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:18:57.826962 run osmo-stp_10.42.42.5(pid=21637): stderr:
| �[0;m�[38;5;43m20221213191856040 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191856040 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191856040 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191856041 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191856041 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191856041 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191856041 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:18:57.981615 run patchelf(pid=21640): Terminating (SIGINT)
19:18:58.162807 run patchelf(pid=21640): Terminated: ok {rc=0}
19:18:58.195594 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=21637): Process ended prematurely: osmo-stp_10.42.42.5(pid=21637) [trial-2393↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21637)] [trial-2393↪gprs:trx-umtrx↪ping.py:22]
19:18:58.204778 tst ping.py:22: Test FAILED (6.7 sec)
19:18:58.269173 run osmo-hlr_10.42.42.2(pid=21634): ERR: Terminated: ERROR {rc=237} [trial-2393↪gprs:trx-umtrx↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21634)]
19:18:58.321644 run osmo-hlr_10.42.42.2(pid=21634): stdout:
| (launched: 2022-12-13_19:18:54.849645)
19:18:58.374602 run osmo-hlr_10.42.42.2(pid=21634): stderr:
| �[0;m�[1;31m20221213191855044 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213191855044 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213191855044 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213191855044 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213191855044 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213191855044 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213191855057 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2393/run.2022-12-13_18-38-33/gprs:trx-umtrx/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213191855066 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213191855066 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213191855066 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:18:58.419868 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21633): Terminating (SIGTERM)
19:18:58.462968 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21636): Terminating (SIGTERM)
19:18:58.505839 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21639): Terminating (SIGTERM)
19:18:58.515625 --- ParallelTerminationStrategy: PID 21633 died...
19:18:58.572704 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21633): Terminated: ok {rc=0}
19:18:58.583123 --- ParallelTerminationStrategy: PID 21636 died...
19:18:58.639899 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21636): Terminated: ok {rc=0}
19:18:58.650163 --- ParallelTerminationStrategy: PID 21639 died...
19:18:58.706749 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21639): Terminated: ok {rc=0}
19:18:58.794295 bus /gobi_6: Setting Powered False
255gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=21647): Process ended prematurely: osmo-stp_10.42.42.5(pid=21647) [trial-2393↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21647)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21647): Process ended prematurely: osmo-stp_10.42.42.5(pid=21647) [trial-2393↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21647)]
----------------------------------------------
trial-2393 gprs:trx-umtrx ping_idle_ping.py
----------------------------------------------
19:18:59.847315 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:18:59.860289 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
19:18:59.998975 tst ping_idle_ping.py: using LAC 35711
19:19:00.119550 tst ping_idle_ping.py: using RAC 11
19:19:00.240279 tst ping_idle_ping.py: using CellId 35711
19:19:00.360912 tst ping_idle_ping.py: using BVCI 35712
19:19:00.384241 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:19:00.395722 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:19:00.406771 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:19:00.417468 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:19:00.427938 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:19:00.438660 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:19:00.449590 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
19:19:00.460357 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
19:19:00.755478 tst ping_idle_ping.py:24: start network...
19:19:00.907164 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:19:01.453867 run create_hlr_db(pid=21641): Launched
19:19:01.634635 bus /gobi_6: Setting Powered False
19:19:02.663040 run create_hlr_db(pid=21641): Terminated: ok {rc=0}
19:19:03.070489 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21643): Launched
19:19:03.373601 run osmo-hlr_10.42.42.2(pid=21644): Launched
19:19:03.483302 run osmo-stp_10.42.42.5: Starting osmo-stp
19:19:04.009449 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21646): Launched
19:19:04.313616 run osmo-stp_10.42.42.5(pid=21647): Launched
19:19:04.424354 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:19:04.952239 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21649): Launched
19:19:05.059797 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:19:05.323393 run patchelf(pid=21650): Launched
19:19:05.499719 run osmo-stp_10.42.42.5(pid=21647): ERR: Terminated: ERROR {rc=1} [trial-2393↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21647)]
19:19:05.628921 run osmo-stp_10.42.42.5(pid=21647): stdout:
| (launched: 2022-12-13_19:19:04.163208)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:19:05.758298 run osmo-stp_10.42.42.5(pid=21647): stderr:
| �[0;m�[38;5;43m20221213191904331 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191904331 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191904331 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191904331 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191904332 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191904332 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191904332 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:19:05.883276 run osmo-stp_10.42.42.5(pid=21647): stdout:
| (launched: 2022-12-13_19:19:04.163208)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:19:06.007176 run osmo-stp_10.42.42.5(pid=21647): stderr:
| �[0;m�[38;5;43m20221213191904331 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213191904331 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213191904331 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213191904331 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213191904332 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213191904332 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213191904332 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:19:06.150927 run patchelf(pid=21650): Terminating (SIGINT)
19:19:06.319436 run patchelf(pid=21650): Terminated: ok {rc=0}
19:19:06.350226 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=21647): Process ended prematurely: osmo-stp_10.42.42.5(pid=21647) [trial-2393↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21647)] [trial-2393↪gprs:trx-umtrx↪ping_idle_ping.py:27]
19:19:06.358770 tst ping_idle_ping.py:27: Test FAILED (6.5 sec)
19:19:06.418730 run osmo-hlr_10.42.42.2(pid=21644): ERR: Terminated: ERROR {rc=237} [trial-2393↪gprs:trx-umtrx↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21644)]
19:19:06.467737 run osmo-hlr_10.42.42.2(pid=21644): stdout:
| (launched: 2022-12-13_19:19:03.224389)
19:19:06.517643 run osmo-hlr_10.42.42.2(pid=21644): stderr:
| �[0;m�[1;31m20221213191903396 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213191903396 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213191903396 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213191903396 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213191903397 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213191903397 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213191903421 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2393/run.2022-12-13_18-38-33/gprs:trx-umtrx/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213191903430 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213191903431 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213191903431 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:19:06.560091 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21643): Terminating (SIGTERM)
19:19:06.599987 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21646): Terminating (SIGTERM)
19:19:06.640063 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21649): Terminating (SIGTERM)
19:19:06.649516 --- ParallelTerminationStrategy: PID 21643 died...
19:19:06.703157 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21643): Terminated: ok {rc=0}
19:19:06.712714 --- ParallelTerminationStrategy: PID 21646 died...
19:19:06.765865 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21646): Terminated: ok {rc=0}
19:19:06.775647 --- ParallelTerminationStrategy: PID 21649 died...
19:19:06.828558 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21649): Terminated: ok {rc=0}
19:19:06.912611 bus /gobi_6: Setting Powered False
5gprs:trx-umtrx00