63gprs:trx-umtrx9.0osmo-stp_10.42.42.5(pid=11715): Process ended prematurely: osmo-stp_10.42.42.5(pid=11715) [trial-2380↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11715)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=11715): Process ended prematurely: osmo-stp_10.42.42.5(pid=11715) [trial-2380↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11715)]
----------------------------------------------
trial-2380 gprs:trx-umtrx cs_paging_gprs_active.py
----------------------------------------------
18:21:06.285380 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:06.296357 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
18:21:06.440928 tst cs_paging_gprs_active.py: using LAC 30636
18:21:06.568783 tst cs_paging_gprs_active.py: using RAC 36
18:21:06.698348 tst cs_paging_gprs_active.py: using CellId 30636
18:21:06.827926 tst cs_paging_gprs_active.py: using BVCI 30637
18:21:06.852925 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:06.864210 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:06.875150 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:06.886085 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:06.896925 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:06.907774 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:06.918922 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:06.929715 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:06.940731 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
18:21:07.059834 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
18:21:07.341962 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
18:21:07.469605 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:21:07.785518 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11705): Launched
18:21:08.048676 run iperf3-srv_10.42.42.10(pid=11706): Launched
18:21:08.234251 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
18:21:08.361198 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:21:08.677639 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11707): Launched
18:21:08.941132 run iperf3-srv_10.42.42.10(pid=11708): Launched
18:21:09.126874 tst cs_paging_gprs_active.py:38: start network...
18:21:09.253906 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:21:09.735329 run create_hlr_db(pid=11709): Launched
18:21:09.910432 bus /sierra_2: Setting Powered False
18:21:10.932279 run create_hlr_db(pid=11709): Terminated: ok {rc=0}
18:21:11.338653 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11711): Launched
18:21:11.644376 run osmo-hlr_10.42.42.2(pid=11712): Launched
18:21:11.759346 run osmo-stp_10.42.42.5: Starting osmo-stp
18:21:12.287622 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11714): Launched
18:21:12.594798 run osmo-stp_10.42.42.5(pid=11715): Launched
18:21:12.705841 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:21:13.235448 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11717): Launched
18:21:13.343583 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:21:13.609244 run patchelf(pid=11718): Launched
18:21:13.786954 run osmo-stp_10.42.42.5(pid=11715): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11715)]
18:21:13.916596 run osmo-stp_10.42.42.5(pid=11715): stdout:
| (launched: 2022-12-12_18:21:12.442302)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:21:14.046906 run osmo-stp_10.42.42.5(pid=11715): stderr:
| �[0;m�[38;5;43m20221212182112609 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182112610 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182112610 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182112610 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182112611 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182112611 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182112611 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:21:14.172460 run osmo-stp_10.42.42.5(pid=11715): stdout:
| (launched: 2022-12-12_18:21:12.442302)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:21:14.296825 run osmo-stp_10.42.42.5(pid=11715): stderr:
| �[0;m�[38;5;43m20221212182112609 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182112610 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182112610 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182112610 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182112611 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182112611 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182112611 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:21:14.442329 run patchelf(pid=11718): Terminating (SIGINT)
18:21:14.612584 run patchelf(pid=11718): Terminated: ok {rc=0}
18:21:14.644167 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=11715): Process ended prematurely: osmo-stp_10.42.42.5(pid=11715) [trial-2380↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11715)] [trial-2380↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38]
18:21:14.653566 tst cs_paging_gprs_active.py:38: Test FAILED (8.4 sec)
18:21:14.714664 run osmo-hlr_10.42.42.2(pid=11712): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-umtrx↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=11712)]
18:21:14.763613 run osmo-hlr_10.42.42.2(pid=11712): stdout:
| (launched: 2022-12-12_18:21:11.493184)
18:21:14.813069 run osmo-hlr_10.42.42.2(pid=11712): stderr:
| �[0;m�[1;31m20221212182111666 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212182111666 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212182111666 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212182111666 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212182111666 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212182111666 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212182111681 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-umtrx/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212182111691 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212182111691 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212182111691 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:21:14.855018 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11705): Terminating (SIGTERM)
18:21:14.894838 run iperf3-srv_10.42.42.10(pid=11706): Terminating (SIGTERM)
18:21:14.934413 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11707): Terminating (SIGTERM)
18:21:14.974112 run iperf3-srv_10.42.42.10(pid=11708): Terminating (SIGTERM)
18:21:15.013830 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11711): Terminating (SIGTERM)
18:21:15.053457 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11714): Terminating (SIGTERM)
18:21:15.093609 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11717): Terminating (SIGTERM)
18:21:15.103162 --- ParallelTerminationStrategy: PID 11705 died...
18:21:15.156425 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11705): Terminated: ok {rc=0}
18:21:15.166265 --- ParallelTerminationStrategy: PID 11706 died...
18:21:15.219466 run iperf3-srv_10.42.42.10(pid=11706): Terminated {rc=256}
18:21:15.229354 --- ParallelTerminationStrategy: PID 11707 died...
18:21:15.282906 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11707): Terminated: ok {rc=0}
18:21:15.292716 --- ParallelTerminationStrategy: PID 11708 died...
18:21:15.345662 run iperf3-srv_10.42.42.10(pid=11708): Terminated {rc=256}
18:21:15.355502 --- ParallelTerminationStrategy: PID 11711 died...
18:21:15.408536 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11711): Terminated: ok {rc=0}
18:21:15.418433 --- ParallelTerminationStrategy: PID 11714 died...
18:21:15.471847 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11714): Terminated: ok {rc=0}
18:21:15.481951 --- ParallelTerminationStrategy: PID 11717 died...
18:21:15.535849 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11717): Terminated: ok {rc=0}
18:21:15.621055 bus /gobi_0: Setting Powered False
18:21:16.736284 bus /gobi_0: Setting Powered False
18:21:17.847898 bus /gobi_6: Setting Powered False
63gprs:trx-umtrx8.0osmo-stp_10.42.42.5(pid=11727): Process ended prematurely: osmo-stp_10.42.42.5(pid=11727) [trial-2380↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11727)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=11727): Process ended prematurely: osmo-stp_10.42.42.5(pid=11727) [trial-2380↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11727)]
----------------------------------------------
trial-2380 gprs:trx-umtrx iperf3.py
----------------------------------------------
18:21:18.958091 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:18.969518 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
18:21:19.112309 tst iperf3.py: using LAC 30637
18:21:19.232499 tst iperf3.py: using RAC 37
18:21:19.356726 tst iperf3.py: using CellId 30637
18:21:19.476406 tst iperf3.py: using BVCI 30638
18:21:19.499591 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:19.510309 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:19.520735 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:19.531475 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:19.542104 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:19.552759 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:19.563637 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:19.574231 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:19.585118 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
18:21:19.882723 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
18:21:20.019148 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:21:20.356176 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11719): Launched
18:21:20.639362 run iperf3-srv_10.42.42.10(pid=11720): Launched
18:21:20.840740 tst iperf3.py:8: start network...
18:21:20.977007 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:21:21.500269 run create_hlr_db(pid=11721): Launched
18:21:21.689134 bus /gobi_6: Setting Powered False
18:21:22.711558 run create_hlr_db(pid=11721): Terminated: ok {rc=0}
18:21:23.120784 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11723): Launched
18:21:23.427528 run osmo-hlr_10.42.42.2(pid=11724): Launched
18:21:23.543549 run osmo-stp_10.42.42.5: Starting osmo-stp
18:21:24.072418 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11726): Launched
18:21:24.377383 run osmo-stp_10.42.42.5(pid=11727): Launched
18:21:24.488624 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:21:25.019751 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11729): Launched
18:21:25.128444 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:21:25.394721 run patchelf(pid=11730): Launched
18:21:25.572702 run osmo-stp_10.42.42.5(pid=11727): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11727)]
18:21:25.702985 run osmo-stp_10.42.42.5(pid=11727): stdout:
| (launched: 2022-12-12_18:21:24.226934)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:21:25.833307 run osmo-stp_10.42.42.5(pid=11727): stderr:
| �[0;m�[38;5;43m20221212182124395 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182124395 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182124396 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182124396 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182124397 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182124397 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182124397 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:21:25.958716 run osmo-stp_10.42.42.5(pid=11727): stdout:
| (launched: 2022-12-12_18:21:24.226934)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:21:26.083105 run osmo-stp_10.42.42.5(pid=11727): stderr:
| �[0;m�[38;5;43m20221212182124395 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182124395 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182124396 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182124396 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182124397 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182124397 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182124397 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:21:26.228751 run patchelf(pid=11730): Terminating (SIGINT)
18:21:26.399071 run patchelf(pid=11730): Terminated: ok {rc=0}
18:21:26.430425 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=11727): Process ended prematurely: osmo-stp_10.42.42.5(pid=11727) [trial-2380↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11727)] [trial-2380↪gprs:trx-umtrx↪iperf3.py:8]
18:21:26.439749 tst iperf3.py:8: Test FAILED (7.5 sec)
18:21:26.500019 run osmo-hlr_10.42.42.2(pid=11724): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-umtrx↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=11724)]
18:21:26.548677 run osmo-hlr_10.42.42.2(pid=11724): stdout:
| (launched: 2022-12-12_18:21:23.275259)
18:21:26.597961 run osmo-hlr_10.42.42.2(pid=11724): stderr:
| �[0;m�[1;31m20221212182123449 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212182123449 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212182123449 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212182123449 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212182123449 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212182123449 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212182123462 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-umtrx/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212182123470 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212182123470 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212182123470 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:21:26.639943 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11719): Terminating (SIGTERM)
18:21:26.679769 run iperf3-srv_10.42.42.10(pid=11720): Terminating (SIGTERM)
18:21:26.720134 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11723): Terminating (SIGTERM)
18:21:26.759968 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11726): Terminating (SIGTERM)
18:21:26.799828 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11729): Terminating (SIGTERM)
18:21:26.809465 --- ParallelTerminationStrategy: PID 11719 died...
18:21:26.862693 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11719): Terminated: ok {rc=0}
18:21:26.872234 --- ParallelTerminationStrategy: PID 11720 died...
18:21:26.924982 run iperf3-srv_10.42.42.10(pid=11720): Terminated {rc=256}
18:21:26.934779 --- ParallelTerminationStrategy: PID 11723 died...
18:21:26.987786 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11723): Terminated: ok {rc=0}
18:21:26.997671 --- ParallelTerminationStrategy: PID 11726 died...
18:21:27.050703 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11726): Terminated: ok {rc=0}
18:21:27.060486 --- ParallelTerminationStrategy: PID 11729 died...
18:21:27.113378 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11729): Terminated: ok {rc=0}
18:21:27.202506 bus /gobi_6: Setting Powered False
206gprs:trx-umtrx12.0osmo-stp_10.42.42.5(pid=11745): Process ended prematurely: osmo-stp_10.42.42.5(pid=11745) [trial-2380↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11745)]2175iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=11745): Process ended prematurely: osmo-stp_10.42.42.5(pid=11745) [trial-2380↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11745)]
----------------------------------------------
trial-2380 gprs:trx-umtrx iperf3m4.py
----------------------------------------------
18:21:28.316436 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:28.328334 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
18:21:28.474049 tst iperf3m4.py: using LAC 30638
18:21:28.601303 tst iperf3m4.py: using RAC 38
18:21:28.730314 tst iperf3m4.py: using CellId 30638
18:21:28.859908 tst iperf3m4.py: using BVCI 30639
18:21:28.884811 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:28.896066 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:28.907109 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:28.918171 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:28.929218 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:28.940095 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:28.951305 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:28.962201 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:28.973322 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
18:21:29.095943 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
18:21:29.219040 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
18:21:29.338115 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
18:21:29.634264 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
18:21:29.770760 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:21:30.108031 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11731): Launched
18:21:30.389701 run iperf3-srv_10.42.42.10(pid=11732): Launched
18:21:30.588622 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
18:21:30.724434 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:21:31.097164 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11733): Launched
18:21:31.394184 run iperf3-srv_10.42.42.10(pid=11734): Launched
18:21:31.596789 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
18:21:31.735400 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:21:32.089465 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11735): Launched
18:21:32.373430 run iperf3-srv_10.42.42.10(pid=11736): Launched
18:21:32.573269 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
18:21:32.709578 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:21:33.049963 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11737): Launched
18:21:33.332532 run iperf3-srv_10.42.42.10(pid=11738): Launched
18:21:33.532272 tst iperf3m4.py:8: start network...
18:21:33.668235 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:21:34.187396 run create_hlr_db(pid=11739): Launched
18:21:34.363817 bus /gobi_6: Setting Powered False
18:21:35.386013 run create_hlr_db(pid=11739): Terminated: ok {rc=0}
18:21:35.821922 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11741): Launched
18:21:36.151870 run osmo-hlr_10.42.42.2(pid=11742): Launched
18:21:36.271442 run osmo-stp_10.42.42.5: Starting osmo-stp
18:21:36.841866 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11744): Launched
18:21:37.172053 run osmo-stp_10.42.42.5(pid=11745): Launched
18:21:37.290170 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:21:37.862560 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11747): Launched
18:21:37.980900 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:21:38.269299 run patchelf(pid=11748): Launched
18:21:38.460694 run osmo-stp_10.42.42.5(pid=11745): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11745)]
18:21:38.600234 run osmo-stp_10.42.42.5(pid=11745): stdout:
| (launched: 2022-12-12_18:21:37.009366)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:21:38.740172 run osmo-stp_10.42.42.5(pid=11745): stderr:
| �[0;m�[38;5;43m20221212182137199 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182137199 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182137199 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182137200 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182137200 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182137200 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182137200 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:21:38.875714 run osmo-stp_10.42.42.5(pid=11745): stdout:
| (launched: 2022-12-12_18:21:37.009366)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:21:39.009565 run osmo-stp_10.42.42.5(pid=11745): stderr:
| �[0;m�[38;5;43m20221212182137199 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182137199 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182137199 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182137200 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182137200 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182137200 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182137200 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:21:39.165939 run patchelf(pid=11748): Terminating (SIGINT)
18:21:39.348530 run patchelf(pid=11748): Terminated: ok {rc=0}
18:21:39.382325 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=11745): Process ended prematurely: osmo-stp_10.42.42.5(pid=11745) [trial-2380↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11745)] [trial-2380↪gprs:trx-umtrx↪iperf3m4.py:8]
18:21:39.392346 tst iperf3m4.py:8: Test FAILED (11.1 sec)
18:21:39.456002 run osmo-hlr_10.42.42.2(pid=11742): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-umtrx↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=11742)]
18:21:39.507907 run osmo-hlr_10.42.42.2(pid=11742): stdout:
| (launched: 2022-12-12_18:21:35.989122)
18:21:39.560370 run osmo-hlr_10.42.42.2(pid=11742): stderr:
| �[0;m�[1;31m20221212182136184 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212182136184 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212182136184 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212182136184 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212182136184 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212182136184 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212182136196 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-umtrx/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212182136205 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212182136206 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212182136206 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:21:39.605269 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11731): Terminating (SIGTERM)
18:21:39.647654 run iperf3-srv_10.42.42.10(pid=11732): Terminating (SIGTERM)
18:21:39.690126 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11733): Terminating (SIGTERM)
18:21:39.732158 run iperf3-srv_10.42.42.10(pid=11734): Terminating (SIGTERM)
18:21:39.774515 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11735): Terminating (SIGTERM)
18:21:39.816319 run iperf3-srv_10.42.42.10(pid=11736): Terminating (SIGTERM)
18:21:39.858048 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11737): Terminating (SIGTERM)
18:21:39.899628 run iperf3-srv_10.42.42.10(pid=11738): Terminating (SIGTERM)
18:21:39.941398 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11741): Terminating (SIGTERM)
18:21:39.983706 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11744): Terminating (SIGTERM)
18:21:40.025424 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11747): Terminating (SIGTERM)
18:21:40.035115 --- ParallelTerminationStrategy: PID 11731 died...
18:21:40.090938 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11731): Terminated: ok {rc=0}
18:21:40.100819 --- ParallelTerminationStrategy: PID 11732 died...
18:21:40.155889 run iperf3-srv_10.42.42.10(pid=11732): Terminated {rc=256}
18:21:40.165867 --- ParallelTerminationStrategy: PID 11733 died...
18:21:40.223471 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11733): Terminated: ok {rc=0}
18:21:40.233269 --- ParallelTerminationStrategy: PID 11734 died...
18:21:40.286968 run iperf3-srv_10.42.42.10(pid=11734): Terminated {rc=256}
18:21:40.296842 --- ParallelTerminationStrategy: PID 11735 died...
18:21:40.350543 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11735): Terminated: ok {rc=0}
18:21:40.360437 --- ParallelTerminationStrategy: PID 11736 died...
18:21:40.414419 run iperf3-srv_10.42.42.10(pid=11736): Terminated {rc=256}
18:21:40.424049 --- ParallelTerminationStrategy: PID 11737 died...
18:21:40.477535 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=11737): Terminated: ok {rc=0}
18:21:40.487393 --- ParallelTerminationStrategy: PID 11738 died...
18:21:40.540486 run iperf3-srv_10.42.42.10(pid=11738): Terminated {rc=256}
18:21:40.550415 --- ParallelTerminationStrategy: PID 11741 died...
18:21:40.603937 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11741): Terminated: ok {rc=0}
18:21:40.613716 --- ParallelTerminationStrategy: PID 11744 died...
18:21:40.667219 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11744): Terminated: ok {rc=0}
18:21:40.676878 --- ParallelTerminationStrategy: PID 11747 died...
18:21:40.730146 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11747): Terminated: ok {rc=0}
18:21:40.819402 bus /gobi_4: Setting Powered False
18:21:41.923131 bus /gobi_4: Setting Powered False
18:21:43.034609 bus /gobi_1: Setting Powered False
18:21:44.136636 bus /gobi_1: Setting Powered False
18:21:45.247521 bus /gobi_0: Setting Powered False
18:21:46.355697 bus /gobi_0: Setting Powered False
18:21:47.467441 bus /gobi_6: Setting Powered False
181gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=11756): Process ended prematurely: osmo-stp_10.42.42.5(pid=11756) [trial-2380↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11756)]2200ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=11756): Process ended prematurely: osmo-stp_10.42.42.5(pid=11756) [trial-2380↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11756)]
----------------------------------------------
trial-2380 gprs:trx-umtrx ping.py
----------------------------------------------
18:21:48.532903 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:48.544444 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
18:21:48.681079 tst ping.py: using LAC 30639
18:21:48.806181 tst ping.py: using RAC 39
18:21:48.933594 tst ping.py: using CellId 30639
18:21:49.060422 tst ping.py: using BVCI 30640
18:21:49.084895 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:49.096301 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:49.107444 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:49.119228 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:49.130666 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:49.142128 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:49.153747 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:49.165262 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
18:21:49.455112 tst ping.py:19: start network...
18:21:49.588762 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:21:50.102010 run create_hlr_db(pid=11750): Launched
18:21:50.286817 bus /gobi_6: Setting Powered False
18:21:51.307509 run create_hlr_db(pid=11750): Terminated: ok {rc=0}
18:21:51.709279 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11752): Launched
18:21:52.009997 run osmo-hlr_10.42.42.2(pid=11753): Launched
18:21:52.120315 run osmo-stp_10.42.42.5: Starting osmo-stp
18:21:52.643280 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11755): Launched
18:21:52.945623 run osmo-stp_10.42.42.5(pid=11756): Launched
18:21:53.053781 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:21:53.576238 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11758): Launched
18:21:53.682950 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:21:53.946697 run patchelf(pid=11759): Launched
18:21:54.122194 run osmo-stp_10.42.42.5(pid=11756): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11756)]
18:21:54.250238 run osmo-stp_10.42.42.5(pid=11756): stdout:
| (launched: 2022-12-12_18:21:52.794899)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:21:54.378859 run osmo-stp_10.42.42.5(pid=11756): stderr:
| �[0;m�[38;5;43m20221212182152964 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182152964 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182152964 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182152965 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182152965 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182152965 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182152965 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:21:54.502590 run osmo-stp_10.42.42.5(pid=11756): stdout:
| (launched: 2022-12-12_18:21:52.794899)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:21:54.625327 run osmo-stp_10.42.42.5(pid=11756): stderr:
| �[0;m�[38;5;43m20221212182152964 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182152964 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182152964 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182152965 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182152965 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182152965 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182152965 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:21:54.768167 run patchelf(pid=11759): Terminating (SIGINT)
18:21:54.937737 run patchelf(pid=11759): Terminated: ok {rc=0}
18:21:54.968537 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=11756): Process ended prematurely: osmo-stp_10.42.42.5(pid=11756) [trial-2380↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11756)] [trial-2380↪gprs:trx-umtrx↪ping.py:22]
18:21:54.977425 tst ping.py:22: Test FAILED (6.4 sec)
18:21:55.037388 run osmo-hlr_10.42.42.2(pid=11753): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-umtrx↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=11753)]
18:21:55.087103 run osmo-hlr_10.42.42.2(pid=11753): stdout:
| (launched: 2022-12-12_18:21:51.861214)
18:21:55.136368 run osmo-hlr_10.42.42.2(pid=11753): stderr:
| �[0;m�[1;31m20221212182152034 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212182152034 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212182152034 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212182152034 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212182152034 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212182152035 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212182152047 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-umtrx/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212182152058 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212182152058 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212182152058 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:21:55.178528 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11752): Terminating (SIGTERM)
18:21:55.218308 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11755): Terminating (SIGTERM)
18:21:55.257950 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11758): Terminating (SIGTERM)
18:21:55.267142 --- ParallelTerminationStrategy: PID 11752 died...
18:21:55.320180 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11752): Terminated: ok {rc=0}
18:21:55.329787 --- ParallelTerminationStrategy: PID 11755 died...
18:21:55.382598 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11755): Terminated: ok {rc=0}
18:21:55.392173 --- ParallelTerminationStrategy: PID 11758 died...
18:21:55.445613 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11758): Terminated: ok {rc=0}
18:21:55.529238 bus /gobi_6: Setting Powered False
242gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=11766): Process ended prematurely: osmo-stp_10.42.42.5(pid=11766) [trial-2380↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11766)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=11766): Process ended prematurely: osmo-stp_10.42.42.5(pid=11766) [trial-2380↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11766)]
----------------------------------------------
trial-2380 gprs:trx-umtrx ping_idle_ping.py
----------------------------------------------
18:21:56.583142 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:56.595176 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
18:21:56.742927 tst ping_idle_ping.py: using LAC 30640
18:21:56.871124 tst ping_idle_ping.py: using RAC 40
18:21:57.000003 tst ping_idle_ping.py: using CellId 30640
18:21:57.126166 tst ping_idle_ping.py: using BVCI 30641
18:21:57.151027 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:57.162304 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:57.173351 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:57.184822 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:57.196164 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:57.207620 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:57.219299 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
18:21:57.230927 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
18:21:57.522591 tst ping_idle_ping.py:24: start network...
18:21:57.656390 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:21:58.176936 run create_hlr_db(pid=11760): Launched
18:21:58.364437 bus /gobi_6: Setting Powered False
18:21:59.386082 run create_hlr_db(pid=11760): Terminated: ok {rc=0}
18:21:59.794058 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11762): Launched
18:22:00.097559 run osmo-hlr_10.42.42.2(pid=11763): Launched
18:22:00.216365 run osmo-stp_10.42.42.5: Starting osmo-stp
18:22:00.748246 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11765): Launched
18:22:01.079183 run osmo-stp_10.42.42.5(pid=11766): Launched
18:22:01.444150 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:22:02.008260 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11768): Launched
18:22:02.129997 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:22:02.422112 run patchelf(pid=11769): Launched
18:22:02.611049 run osmo-stp_10.42.42.5(pid=11766): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11766)]
18:22:02.748706 run osmo-stp_10.42.42.5(pid=11766): stdout:
| (launched: 2022-12-12_18:22:00.910995)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:22:02.886670 run osmo-stp_10.42.42.5(pid=11766): stderr:
| �[0;m�[38;5;43m20221212182201280 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182201280 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182201280 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182201282 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182201283 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182201283 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182201283 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:22:03.020296 run osmo-stp_10.42.42.5(pid=11766): stdout:
| (launched: 2022-12-12_18:22:00.910995)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:22:03.152505 run osmo-stp_10.42.42.5(pid=11766): stderr:
| �[0;m�[38;5;43m20221212182201280 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212182201280 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212182201280 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212182201282 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212182201283 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212182201283 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212182201283 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:22:03.305706 run patchelf(pid=11769): Terminating (SIGINT)
18:22:03.484543 run patchelf(pid=11769): Terminated: ok {rc=0}
18:22:03.517363 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=11766): Process ended prematurely: osmo-stp_10.42.42.5(pid=11766) [trial-2380↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=11766)] [trial-2380↪gprs:trx-umtrx↪ping_idle_ping.py:27]
18:22:03.526799 tst ping_idle_ping.py:27: Test FAILED (6.9 sec)
18:22:03.591613 run osmo-hlr_10.42.42.2(pid=11763): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-umtrx↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=11763)]
18:22:03.643161 run osmo-hlr_10.42.42.2(pid=11763): stdout:
| (launched: 2022-12-12_18:21:59.947692)
18:22:03.695116 run osmo-hlr_10.42.42.2(pid=11763): stderr:
| �[0;m�[1;31m20221212182200122 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212182200122 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212182200122 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212182200122 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212182200122 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212182200122 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212182200134 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-umtrx/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212182200144 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212182200145 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212182200145 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:22:03.739616 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11762): Terminating (SIGTERM)
18:22:03.782381 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11765): Terminating (SIGTERM)
18:22:03.824697 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11768): Terminating (SIGTERM)
18:22:03.834624 --- ParallelTerminationStrategy: PID 11762 died...
18:22:03.891050 run pcap-recorder_any(filters='host 10.42.42.2')(pid=11762): Terminated: ok {rc=0}
18:22:03.901202 --- ParallelTerminationStrategy: PID 11765 died...
18:22:03.957185 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=11765): Terminated: ok {rc=0}
18:22:03.967542 --- ParallelTerminationStrategy: PID 11768 died...
18:22:04.023817 run pcap-recorder_any(filters='host 10.42.42.6')(pid=11768): Terminated: ok {rc=0}
18:22:04.110971 bus /gobi_6: Setting Powered False
5gprs:trx-umtrx00