50gprs:trx-b2009.0osmo-stp_10.42.42.5(pid=5895): Process ended prematurely: osmo-stp_10.42.42.5(pid=5895) [trial-2367↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5895)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=5895): Process ended prematurely: osmo-stp_10.42.42.5(pid=5895) [trial-2367↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5895)]
----------------------------------------------
trial-2367 gprs:trx-b200 cs_paging_gprs_active.py
----------------------------------------------
01:02:22.190401 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:22.201310 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
01:02:22.346297 tst cs_paging_gprs_active.py: using LAC 24899
01:02:22.474132 tst cs_paging_gprs_active.py: using RAC 164
01:02:22.603693 tst cs_paging_gprs_active.py: using CellId 24899
01:02:22.732052 tst cs_paging_gprs_active.py: using BVCI 24900
01:02:22.757019 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:22.768054 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:22.778873 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:22.789789 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:22.800489 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:22.811242 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:22.822204 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:22.832834 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:22.843847 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
01:02:22.963201 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
01:02:23.258678 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
01:02:23.395123 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:02:23.732379 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5885): Launched
01:02:24.013541 run iperf3-srv_10.42.42.10(pid=5886): Launched
01:02:24.213643 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
01:02:24.349779 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:02:24.687338 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5887): Launched
01:02:24.966858 run iperf3-srv_10.42.42.10(pid=5888): Launched
01:02:25.166637 tst cs_paging_gprs_active.py:38: start network...
01:02:25.302801 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:02:25.823990 run create_hlr_db(pid=5889): Launched
01:02:26.011606 bus /sierra_2: Setting Powered False
01:02:27.022740 run create_hlr_db(pid=5889): Terminated: ok {rc=0}
01:02:27.462459 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5891): Launched
01:02:27.789712 run osmo-hlr_10.42.42.2(pid=5892): Launched
01:02:27.908833 run osmo-stp_10.42.42.5: Starting osmo-stp
01:02:28.474533 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5894): Launched
01:02:28.801940 run osmo-stp_10.42.42.5(pid=5895): Launched
01:02:28.920109 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:02:29.488914 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5897): Launched
01:02:29.605502 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:02:29.889134 run patchelf(pid=5898): Launched
01:02:30.081028 run osmo-stp_10.42.42.5(pid=5895): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5895)]
01:02:30.220156 run osmo-stp_10.42.42.5(pid=5895): stdout:
| (launched: 2022-12-10_01:02:28.641321)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:02:30.359551 run osmo-stp_10.42.42.5(pid=5895): stderr:
| �[0;m�[38;5;43m20221210010228834 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010228834 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010228834 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010228834 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010228835 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010228835 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010228835 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:02:30.494391 run osmo-stp_10.42.42.5(pid=5895): stdout:
| (launched: 2022-12-10_01:02:28.641321)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:02:30.628560 run osmo-stp_10.42.42.5(pid=5895): stderr:
| �[0;m�[38;5;43m20221210010228834 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010228834 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010228834 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010228834 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010228835 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010228835 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010228835 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:02:30.783598 run patchelf(pid=5898): Terminating (SIGINT)
01:02:30.994489 run patchelf(pid=5898): Terminated: ok {rc=0}
01:02:31.039629 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=5895): Process ended prematurely: osmo-stp_10.42.42.5(pid=5895) [trial-2367↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5895)] [trial-2367↪gprs:trx-b200↪cs_paging_gprs_active.py:38]
01:02:31.050491 tst cs_paging_gprs_active.py:38: Test FAILED (8.9 sec)
01:02:31.116188 run osmo-hlr_10.42.42.2(pid=5892): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-b200↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=5892)]
01:02:31.169227 run osmo-hlr_10.42.42.2(pid=5892): stdout:
| (launched: 2022-12-10_01:02:27.628442)
01:02:31.222294 run osmo-hlr_10.42.42.2(pid=5892): stderr:
| �[0;m�[1;31m20221210010227827 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210010227828 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210010227828 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210010227828 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210010227828 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210010227828 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210010227852 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-b200/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210010227862 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210010227862 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210010227862 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:02:31.268331 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5885): Terminating (SIGTERM)
01:02:31.311128 run iperf3-srv_10.42.42.10(pid=5886): Terminating (SIGTERM)
01:02:31.355978 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5887): Terminating (SIGTERM)
01:02:31.399354 run iperf3-srv_10.42.42.10(pid=5888): Terminating (SIGTERM)
01:02:31.442409 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5891): Terminating (SIGTERM)
01:02:31.487171 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5894): Terminating (SIGTERM)
01:02:31.530716 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5897): Terminating (SIGTERM)
01:02:31.540648 --- ParallelTerminationStrategy: PID 5885 died...
01:02:31.597936 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5885): Terminated: ok {rc=0}
01:02:31.608400 --- ParallelTerminationStrategy: PID 5886 died...
01:02:31.665040 run iperf3-srv_10.42.42.10(pid=5886): Terminated {rc=256}
01:02:31.675378 --- ParallelTerminationStrategy: PID 5887 died...
01:02:31.731828 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5887): Terminated: ok {rc=0}
01:02:31.742326 --- ParallelTerminationStrategy: PID 5888 died...
01:02:31.799004 run iperf3-srv_10.42.42.10(pid=5888): Terminated {rc=256}
01:02:31.809434 --- ParallelTerminationStrategy: PID 5891 died...
01:02:31.865978 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5891): Terminated: ok {rc=0}
01:02:31.876372 --- ParallelTerminationStrategy: PID 5894 died...
01:02:31.932878 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5894): Terminated: ok {rc=0}
01:02:31.943306 --- ParallelTerminationStrategy: PID 5897 died...
01:02:31.999789 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5897): Terminated: ok {rc=0}
01:02:32.089428 bus /gobi_0: Setting Powered False
01:02:33.195074 bus /gobi_0: Setting Powered False
01:02:34.304239 bus /gobi_6: Setting Powered False
50gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=5908): Process ended prematurely: osmo-stp_10.42.42.5(pid=5908) [trial-2367↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5908)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=5908): Process ended prematurely: osmo-stp_10.42.42.5(pid=5908) [trial-2367↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5908)]
----------------------------------------------
trial-2367 gprs:trx-b200 iperf3.py
----------------------------------------------
01:02:35.420661 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:35.432431 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
01:02:35.577890 tst iperf3.py: using LAC 24900
01:02:35.697826 tst iperf3.py: using RAC 165
01:02:35.817115 tst iperf3.py: using CellId 24900
01:02:35.939041 tst iperf3.py: using BVCI 24901
01:02:35.962494 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:35.973248 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:35.983911 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:35.995019 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:36.005996 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:36.016839 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:36.027950 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:36.038955 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:36.050128 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
01:02:36.345168 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
01:02:36.481286 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:02:36.818930 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5900): Launched
01:02:37.098889 run iperf3-srv_10.42.42.10(pid=5901): Launched
01:02:37.298658 tst iperf3.py:8: start network...
01:02:37.434675 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:02:37.954456 run create_hlr_db(pid=5902): Launched
01:02:38.148064 bus /gobi_6: Setting Powered False
01:02:39.162684 run create_hlr_db(pid=5902): Terminated: ok {rc=0}
01:02:39.595809 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5904): Launched
01:02:39.922345 run osmo-hlr_10.42.42.2(pid=5905): Launched
01:02:40.041196 run osmo-stp_10.42.42.5: Starting osmo-stp
01:02:40.608538 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5907): Launched
01:02:40.935565 run osmo-stp_10.42.42.5(pid=5908): Launched
01:02:41.053724 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:02:41.619581 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5910): Launched
01:02:41.735090 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:02:42.018461 run patchelf(pid=5911): Launched
01:02:42.209219 run osmo-stp_10.42.42.5(pid=5908): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5908)]
01:02:42.348093 run osmo-stp_10.42.42.5(pid=5908): stdout:
| (launched: 2022-12-10_01:02:40.775206)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:02:42.487205 run osmo-stp_10.42.42.5(pid=5908): stderr:
| �[0;m�[38;5;43m20221210010240968 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010240968 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010240968 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010240969 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010240969 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010240969 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010240969 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:02:42.622306 run osmo-stp_10.42.42.5(pid=5908): stdout:
| (launched: 2022-12-10_01:02:40.775206)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:02:42.756363 run osmo-stp_10.42.42.5(pid=5908): stderr:
| �[0;m�[38;5;43m20221210010240968 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010240968 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010240968 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010240969 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010240969 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010240969 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010240969 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:02:42.911194 run patchelf(pid=5911): Terminating (SIGINT)
01:02:43.092710 run patchelf(pid=5911): Terminated: ok {rc=0}
01:02:43.126604 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=5908): Process ended prematurely: osmo-stp_10.42.42.5(pid=5908) [trial-2367↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5908)] [trial-2367↪gprs:trx-b200↪iperf3.py:8]
01:02:43.136255 tst iperf3.py:8: Test FAILED (7.7 sec)
01:02:43.199170 run osmo-hlr_10.42.42.2(pid=5905): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-b200↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=5905)]
01:02:43.250294 run osmo-hlr_10.42.42.2(pid=5905): stdout:
| (launched: 2022-12-10_01:02:39.761553)
01:02:43.302143 run osmo-hlr_10.42.42.2(pid=5905): stderr:
| �[0;m�[1;31m20221210010239964 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210010239964 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210010239964 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210010239964 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210010239964 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210010239964 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210010239982 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-b200/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210010239991 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210010239991 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210010239991 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:02:43.346489 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5900): Terminating (SIGTERM)
01:02:43.388201 run iperf3-srv_10.42.42.10(pid=5901): Terminating (SIGTERM)
01:02:43.430137 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5904): Terminating (SIGTERM)
01:02:43.471864 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5907): Terminating (SIGTERM)
01:02:43.513720 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5910): Terminating (SIGTERM)
01:02:43.523365 --- ParallelTerminationStrategy: PID 5900 died...
01:02:43.579518 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5900): Terminated: ok {rc=0}
01:02:43.589560 --- ParallelTerminationStrategy: PID 5901 died...
01:02:43.644777 run iperf3-srv_10.42.42.10(pid=5901): Terminated {rc=256}
01:02:43.654722 --- ParallelTerminationStrategy: PID 5904 died...
01:02:43.710034 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5904): Terminated: ok {rc=0}
01:02:43.720025 --- ParallelTerminationStrategy: PID 5907 died...
01:02:43.775365 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5907): Terminated: ok {rc=0}
01:02:43.785422 --- ParallelTerminationStrategy: PID 5910 died...
01:02:43.840792 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5910): Terminated: ok {rc=0}
01:02:43.928150 bus /gobi_6: Setting Powered False
182gprs:trx-b20012.0osmo-stp_10.42.42.5(pid=5926): Process ended prematurely: osmo-stp_10.42.42.5(pid=5926) [trial-2367↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5926)]2186iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=5926): Process ended prematurely: osmo-stp_10.42.42.5(pid=5926) [trial-2367↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5926)]
----------------------------------------------
trial-2367 gprs:trx-b200 iperf3m4.py
----------------------------------------------
01:02:45.043234 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:45.055269 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
01:02:45.201181 tst iperf3m4.py: using LAC 24901
01:02:45.329802 tst iperf3m4.py: using RAC 166
01:02:45.457033 tst iperf3m4.py: using CellId 24901
01:02:45.585722 tst iperf3m4.py: using BVCI 24902
01:02:45.610508 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:45.621710 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:45.632657 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:45.643634 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:45.654615 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:45.665499 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:45.676459 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:45.687405 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:02:45.698466 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
01:02:45.818333 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
01:02:45.958433 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
01:02:46.078189 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
01:02:46.395404 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
01:02:46.528277 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:02:46.877230 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5912): Launched
01:02:47.177717 run iperf3-srv_10.42.42.10(pid=5913): Launched
01:02:47.378942 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
01:02:47.514952 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:02:47.852192 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5914): Launched
01:02:48.133644 run iperf3-srv_10.42.42.10(pid=5915): Launched
01:02:48.334144 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
01:02:48.470538 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:02:48.808462 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5916): Launched
01:02:49.087961 run iperf3-srv_10.42.42.10(pid=5917): Launched
01:02:49.286776 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
01:02:49.422021 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:02:49.757021 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5918): Launched
01:02:50.036421 run iperf3-srv_10.42.42.10(pid=5919): Launched
01:02:50.235717 tst iperf3m4.py:8: start network...
01:02:50.370994 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:02:50.891825 run create_hlr_db(pid=5920): Launched
01:02:51.079023 bus /gobi_6: Setting Powered False
01:02:52.098497 run create_hlr_db(pid=5920): Terminated: ok {rc=0}
01:02:52.534894 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5922): Launched
01:02:52.860836 run osmo-hlr_10.42.42.2(pid=5923): Launched
01:02:52.978767 run osmo-stp_10.42.42.5: Starting osmo-stp
01:02:53.545426 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5925): Launched
01:02:53.871591 run osmo-stp_10.42.42.5(pid=5926): Launched
01:02:53.989438 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:02:54.556781 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5928): Launched
01:02:54.672776 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:02:54.957405 run patchelf(pid=5929): Launched
01:02:55.148410 run osmo-stp_10.42.42.5(pid=5926): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5926)]
01:02:55.287865 run osmo-stp_10.42.42.5(pid=5926): stdout:
| (launched: 2022-12-10_01:02:53.710873)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:02:55.427148 run osmo-stp_10.42.42.5(pid=5926): stderr:
| �[0;m�[38;5;43m20221210010253903 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010253903 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010253903 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010253904 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010253904 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010253904 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010253904 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:02:55.562872 run osmo-stp_10.42.42.5(pid=5926): stdout:
| (launched: 2022-12-10_01:02:53.710873)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:02:55.696931 run osmo-stp_10.42.42.5(pid=5926): stderr:
| �[0;m�[38;5;43m20221210010253903 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010253903 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010253903 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010253904 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010253904 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010253904 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010253904 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:02:55.851843 run patchelf(pid=5929): Terminating (SIGINT)
01:02:56.033777 run patchelf(pid=5929): Terminated: ok {rc=0}
01:02:56.067183 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=5926): Process ended prematurely: osmo-stp_10.42.42.5(pid=5926) [trial-2367↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5926)] [trial-2367↪gprs:trx-b200↪iperf3m4.py:8]
01:02:56.076875 tst iperf3m4.py:8: Test FAILED (11.1 sec)
01:02:56.140509 run osmo-hlr_10.42.42.2(pid=5923): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-b200↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=5923)]
01:02:56.192273 run osmo-hlr_10.42.42.2(pid=5923): stdout:
| (launched: 2022-12-10_01:02:52.701137)
01:02:56.244408 run osmo-hlr_10.42.42.2(pid=5923): stderr:
| �[0;m�[1;31m20221210010252903 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210010252903 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210010252903 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210010252903 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210010252903 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210010252903 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210010252929 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-b200/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210010252939 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210010252939 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210010252939 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:02:56.288976 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5912): Terminating (SIGTERM)
01:02:56.330960 run iperf3-srv_10.42.42.10(pid=5913): Terminating (SIGTERM)
01:02:56.373135 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5914): Terminating (SIGTERM)
01:02:56.415265 run iperf3-srv_10.42.42.10(pid=5915): Terminating (SIGTERM)
01:02:56.457529 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5916): Terminating (SIGTERM)
01:02:56.499532 run iperf3-srv_10.42.42.10(pid=5917): Terminating (SIGTERM)
01:02:56.541775 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5918): Terminating (SIGTERM)
01:02:56.583928 run iperf3-srv_10.42.42.10(pid=5919): Terminating (SIGTERM)
01:02:56.626116 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5922): Terminating (SIGTERM)
01:02:56.668176 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5925): Terminating (SIGTERM)
01:02:56.710581 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5928): Terminating (SIGTERM)
01:02:56.720464 --- ParallelTerminationStrategy: PID 5912 died...
01:02:56.777317 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5912): Terminated: ok {rc=0}
01:02:56.787627 --- ParallelTerminationStrategy: PID 5913 died...
01:02:56.843533 run iperf3-srv_10.42.42.10(pid=5913): Terminated {rc=256}
01:02:56.854004 --- ParallelTerminationStrategy: PID 5914 died...
01:02:56.909915 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5914): Terminated: ok {rc=0}
01:02:56.920083 --- ParallelTerminationStrategy: PID 5915 died...
01:02:56.975779 run iperf3-srv_10.42.42.10(pid=5915): Terminated {rc=256}
01:02:56.985792 --- ParallelTerminationStrategy: PID 5916 died...
01:02:57.041459 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5916): Terminated: ok {rc=0}
01:02:57.051582 --- ParallelTerminationStrategy: PID 5917 died...
01:02:57.107249 run iperf3-srv_10.42.42.10(pid=5917): Terminated {rc=256}
01:02:57.117491 --- ParallelTerminationStrategy: PID 5918 died...
01:02:57.173505 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=5918): Terminated: ok {rc=0}
01:02:57.183731 --- ParallelTerminationStrategy: PID 5919 died...
01:02:57.239107 run iperf3-srv_10.42.42.10(pid=5919): Terminated {rc=256}
01:02:57.249381 --- ParallelTerminationStrategy: PID 5922 died...
01:02:57.305076 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5922): Terminated: ok {rc=0}
01:02:57.315262 --- ParallelTerminationStrategy: PID 5925 died...
01:02:57.371237 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5925): Terminated: ok {rc=0}
01:02:57.381550 --- ParallelTerminationStrategy: PID 5928 died...
01:02:57.437539 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5928): Terminated: ok {rc=0}
01:02:57.525906 bus /gobi_4: Setting Powered False
01:02:58.630463 bus /gobi_4: Setting Powered False
01:02:59.743000 bus /gobi_1: Setting Powered False
01:03:00.856767 bus /gobi_1: Setting Powered False
01:03:01.996585 bus /gobi_0: Setting Powered False
01:03:03.109710 bus /gobi_0: Setting Powered False
01:03:04.223497 bus /gobi_6: Setting Powered False
182gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=5937): Process ended prematurely: osmo-stp_10.42.42.5(pid=5937) [trial-2367↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5937)]2186ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=5937): Process ended prematurely: osmo-stp_10.42.42.5(pid=5937) [trial-2367↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5937)]
----------------------------------------------
trial-2367 gprs:trx-b200 ping.py
----------------------------------------------
01:03:05.292405 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:05.303782 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
01:03:05.438244 tst ping.py: using LAC 24902
01:03:05.557088 tst ping.py: using RAC 167
01:03:05.676046 tst ping.py: using CellId 24902
01:03:05.797550 tst ping.py: using BVCI 24903
01:03:05.820252 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:05.830858 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:05.841409 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:05.852097 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:05.862704 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:05.873271 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:05.884041 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:05.894872 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
01:03:06.187437 tst ping.py:19: start network...
01:03:06.321068 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:03:06.831650 run create_hlr_db(pid=5931): Launched
01:03:07.016887 bus /gobi_6: Setting Powered False
01:03:08.035496 run create_hlr_db(pid=5931): Terminated: ok {rc=0}
01:03:08.437689 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5933): Launched
01:03:08.738230 run osmo-hlr_10.42.42.2(pid=5934): Launched
01:03:08.852914 run osmo-stp_10.42.42.5: Starting osmo-stp
01:03:09.382769 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5936): Launched
01:03:09.681645 run osmo-stp_10.42.42.5(pid=5937): Launched
01:03:09.789504 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:03:10.307260 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5939): Launched
01:03:10.412323 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:03:10.672576 run patchelf(pid=5940): Launched
01:03:10.847283 run osmo-stp_10.42.42.5(pid=5937): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5937)]
01:03:10.975646 run osmo-stp_10.42.42.5(pid=5937): stdout:
| (launched: 2022-12-10_01:03:09.534398)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:03:11.104083 run osmo-stp_10.42.42.5(pid=5937): stderr:
| �[0;m�[38;5;43m20221210010309705 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010309705 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010309705 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010309706 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010309708 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010309708 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010309708 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:03:11.228191 run osmo-stp_10.42.42.5(pid=5937): stdout:
| (launched: 2022-12-10_01:03:09.534398)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:03:11.351586 run osmo-stp_10.42.42.5(pid=5937): stderr:
| �[0;m�[38;5;43m20221210010309705 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010309705 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010309705 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010309706 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010309708 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010309708 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010309708 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:03:11.493937 run patchelf(pid=5940): Terminating (SIGINT)
01:03:11.660676 run patchelf(pid=5940): Terminated: ok {rc=0}
01:03:11.691023 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=5937): Process ended prematurely: osmo-stp_10.42.42.5(pid=5937) [trial-2367↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5937)] [trial-2367↪gprs:trx-b200↪ping.py:22]
01:03:11.699621 tst ping.py:22: Test FAILED (6.4 sec)
01:03:11.760226 run osmo-hlr_10.42.42.2(pid=5934): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-b200↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=5934)]
01:03:11.809363 run osmo-hlr_10.42.42.2(pid=5934): stdout:
| (launched: 2022-12-10_01:03:08.589273)
01:03:11.859352 run osmo-hlr_10.42.42.2(pid=5934): stderr:
| �[0;m�[1;31m20221210010308764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210010308764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210010308764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210010308764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210010308764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210010308764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210010308776 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-b200/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210010308785 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210010308785 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210010308785 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:03:11.901456 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5933): Terminating (SIGTERM)
01:03:11.941519 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5936): Terminating (SIGTERM)
01:03:11.981063 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5939): Terminating (SIGTERM)
01:03:11.990502 --- ParallelTerminationStrategy: PID 5933 died...
01:03:12.043742 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5933): Terminated: ok {rc=0}
01:03:12.053616 --- ParallelTerminationStrategy: PID 5936 died...
01:03:12.106865 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5936): Terminated: ok {rc=0}
01:03:12.116871 --- ParallelTerminationStrategy: PID 5939 died...
01:03:12.169992 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5939): Terminated: ok {rc=0}
01:03:12.257774 bus /gobi_6: Setting Powered False
182gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=5947): Process ended prematurely: osmo-stp_10.42.42.5(pid=5947) [trial-2367↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5947)]2186ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=5947): Process ended prematurely: osmo-stp_10.42.42.5(pid=5947) [trial-2367↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5947)]
----------------------------------------------
trial-2367 gprs:trx-b200 ping_idle_ping.py
----------------------------------------------
01:03:13.310374 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:13.321411 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
01:03:13.456017 tst ping_idle_ping.py: using LAC 24903
01:03:13.570486 tst ping_idle_ping.py: using RAC 168
01:03:13.684771 tst ping_idle_ping.py: using CellId 24903
01:03:13.798645 tst ping_idle_ping.py: using BVCI 24904
01:03:13.820642 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:13.831137 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:13.841457 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:13.852054 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:13.862640 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:13.873193 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:13.883773 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
01:03:13.894118 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
01:03:14.162221 tst ping_idle_ping.py:24: start network...
01:03:14.283414 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:03:14.744376 run create_hlr_db(pid=5941): Launched
01:03:14.912731 bus /gobi_6: Setting Powered False
01:03:15.974054 run create_hlr_db(pid=5941): Terminated: ok {rc=0}
01:03:16.426190 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5943): Launched
01:03:16.878249 run osmo-hlr_10.42.42.2(pid=5944): Launched
01:03:17.053092 run osmo-stp_10.42.42.5: Starting osmo-stp
01:03:17.605294 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5946): Launched
01:03:17.919200 run osmo-stp_10.42.42.5(pid=5947): Launched
01:03:18.033388 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:03:18.580940 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5949): Launched
01:03:18.693388 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:03:18.968007 run patchelf(pid=5950): Launched
01:03:19.152744 run osmo-stp_10.42.42.5(pid=5947): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5947)]
01:03:19.287946 run osmo-stp_10.42.42.5(pid=5947): stdout:
| (launched: 2022-12-10_01:03:17.765142)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:03:19.423243 run osmo-stp_10.42.42.5(pid=5947): stderr:
| �[0;m�[38;5;43m20221210010317955 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010317955 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010317955 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010317956 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010317956 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010317956 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010317956 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:03:19.554441 run osmo-stp_10.42.42.5(pid=5947): stdout:
| (launched: 2022-12-10_01:03:17.765142)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:03:19.684362 run osmo-stp_10.42.42.5(pid=5947): stderr:
| �[0;m�[38;5;43m20221210010317955 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210010317955 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210010317955 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210010317956 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210010317956 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210010317956 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210010317956 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:03:19.834857 run patchelf(pid=5950): Terminating (SIGINT)
01:03:20.011043 run patchelf(pid=5950): Terminated: ok {rc=0}
01:03:20.042955 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=5947): Process ended prematurely: osmo-stp_10.42.42.5(pid=5947) [trial-2367↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=5947)] [trial-2367↪gprs:trx-b200↪ping_idle_ping.py:27]
01:03:20.052019 tst ping_idle_ping.py:27: Test FAILED (6.7 sec)
01:03:20.114873 run osmo-hlr_10.42.42.2(pid=5944): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-b200↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=5944)]
01:03:20.166417 run osmo-hlr_10.42.42.2(pid=5944): stdout:
| (launched: 2022-12-10_01:03:16.592542)
01:03:20.218641 run osmo-hlr_10.42.42.2(pid=5944): stderr:
| �[0;m�[1;31m20221210010316820 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210010316820 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210010316820 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210010316820 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210010316820 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210010316821 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210010316836 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-b200/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210010316864 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210010316864 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210010316864 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:03:20.262755 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5943): Terminating (SIGTERM)
01:03:20.304637 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5946): Terminating (SIGTERM)
01:03:20.346417 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5949): Terminating (SIGTERM)
01:03:20.356063 --- ParallelTerminationStrategy: PID 5943 died...
01:03:20.411745 run pcap-recorder_any(filters='host 10.42.42.2')(pid=5943): Terminated: ok {rc=0}
01:03:20.421887 --- ParallelTerminationStrategy: PID 5946 died...
01:03:20.477495 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=5946): Terminated: ok {rc=0}
01:03:20.487506 --- ParallelTerminationStrategy: PID 5949 died...
01:03:20.542799 run pcap-recorder_any(filters='host 10.42.42.6')(pid=5949): Terminated: ok {rc=0}
01:03:20.629457 bus /gobi_6: Setting Powered False
5gprs:trx-b20000