229gprs:trx-lms9.0osmo-stp_10.42.42.5(pid=8787): Process ended prematurely: osmo-stp_10.42.42.5(pid=8787) [trial-2367↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8787)]2139cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8787): Process ended prematurely: osmo-stp_10.42.42.5(pid=8787) [trial-2367↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8787)]
----------------------------------------------
trial-2367 gprs:trx-lms cs_paging_gprs_active.py
----------------------------------------------
01:28:22.592658 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:22.603621 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
01:28:22.747012 tst cs_paging_gprs_active.py: using LAC 25037
01:28:22.863487 tst cs_paging_gprs_active.py: using RAC 47
01:28:22.987016 tst cs_paging_gprs_active.py: using CellId 25037
01:28:23.105585 tst cs_paging_gprs_active.py: using BVCI 25038
01:28:23.128552 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:23.139300 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:23.149669 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:23.160058 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:23.170719 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:23.181188 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:23.191813 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:23.202304 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:23.212937 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
01:28:23.331743 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
01:28:23.611195 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
01:28:23.737558 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:28:24.049277 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8777): Launched
01:28:24.309643 run iperf3-srv_10.42.42.10(pid=8778): Launched
01:28:24.493001 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
01:28:24.618353 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:28:24.929662 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8779): Launched
01:28:25.189125 run iperf3-srv_10.42.42.10(pid=8780): Launched
01:28:25.371827 tst cs_paging_gprs_active.py:38: start network...
01:28:25.497086 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:28:25.971400 run create_hlr_db(pid=8781): Launched
01:28:26.144373 bus /sierra_2: Setting Powered False
01:28:27.166629 run create_hlr_db(pid=8781): Terminated: ok {rc=0}
01:28:27.570729 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8783): Launched
01:28:27.873120 run osmo-hlr_10.42.42.2(pid=8784): Launched
01:28:27.987422 run osmo-stp_10.42.42.5: Starting osmo-stp
01:28:28.520270 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8786): Launched
01:28:28.822272 run osmo-stp_10.42.42.5(pid=8787): Launched
01:28:28.932513 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:28:29.458107 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8789): Launched
01:28:29.565656 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:28:29.828934 run patchelf(pid=8790): Launched
01:28:30.005852 run osmo-stp_10.42.42.5(pid=8787): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8787)]
01:28:30.135123 run osmo-stp_10.42.42.5(pid=8787): stdout:
| (launched: 2022-12-10_01:28:28.673105)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:28:30.264527 run osmo-stp_10.42.42.5(pid=8787): stderr:
| �[0;m�[38;5;43m20221210012828845 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012828845 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012828845 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012828846 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012828846 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012828846 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012828846 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:28:30.389508 run osmo-stp_10.42.42.5(pid=8787): stdout:
| (launched: 2022-12-10_01:28:28.673105)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:28:30.513712 run osmo-stp_10.42.42.5(pid=8787): stderr:
| �[0;m�[38;5;43m20221210012828845 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012828845 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012828845 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012828846 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012828846 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012828846 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012828846 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:28:30.657643 run patchelf(pid=8790): Terminating (SIGINT)
01:28:30.826751 run patchelf(pid=8790): Terminated: ok {rc=0}
01:28:30.869449 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=8787): Process ended prematurely: osmo-stp_10.42.42.5(pid=8787) [trial-2367↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8787)] [trial-2367↪gprs:trx-lms↪cs_paging_gprs_active.py:38]
01:28:30.882062 tst cs_paging_gprs_active.py:38: Test FAILED (8.3 sec)
01:28:30.964084 run osmo-hlr_10.42.42.2(pid=8784): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-lms↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8784)]
01:28:31.016930 run osmo-hlr_10.42.42.2(pid=8784): stdout:
| (launched: 2022-12-10_01:28:27.723831)
01:28:31.076315 run osmo-hlr_10.42.42.2(pid=8784): stderr:
| �[0;m�[1;31m20221210012827902 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210012827902 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210012827902 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210012827902 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210012827902 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210012827902 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210012827914 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-lms/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210012827923 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210012827923 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210012827923 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:28:31.121891 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8777): Terminating (SIGTERM)
01:28:31.166603 run iperf3-srv_10.42.42.10(pid=8778): Terminating (SIGTERM)
01:28:31.211113 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8779): Terminating (SIGTERM)
01:28:31.254449 run iperf3-srv_10.42.42.10(pid=8780): Terminating (SIGTERM)
01:28:31.301963 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8783): Terminating (SIGTERM)
01:28:31.347103 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8786): Terminating (SIGTERM)
01:28:31.392203 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8789): Terminating (SIGTERM)
01:28:31.402932 --- ParallelTerminationStrategy: PID 8777 died...
01:28:31.461401 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8777): Terminated: ok {rc=0}
01:28:31.473371 --- ParallelTerminationStrategy: PID 8778 died...
01:28:31.530570 run iperf3-srv_10.42.42.10(pid=8778): Terminated {rc=256}
01:28:31.541234 --- ParallelTerminationStrategy: PID 8779 died...
01:28:31.598521 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8779): Terminated: ok {rc=0}
01:28:31.609320 --- ParallelTerminationStrategy: PID 8780 died...
01:28:31.667908 run iperf3-srv_10.42.42.10(pid=8780): Terminated {rc=256}
01:28:31.678556 --- ParallelTerminationStrategy: PID 8783 died...
01:28:31.735482 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8783): Terminated: ok {rc=0}
01:28:31.746258 --- ParallelTerminationStrategy: PID 8786 died...
01:28:31.806887 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8786): Terminated: ok {rc=0}
01:28:31.817634 --- ParallelTerminationStrategy: PID 8789 died...
01:28:31.875651 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8789): Terminated: ok {rc=0}
01:28:31.985252 bus /gobi_0: Setting Powered False
01:28:33.092158 bus /gobi_0: Setting Powered False
01:28:34.201133 bus /gobi_6: Setting Powered False
226gprs:trx-lms8.0osmo-stp_10.42.42.5(pid=8799): Process ended prematurely: osmo-stp_10.42.42.5(pid=8799) [trial-2367↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8799)]2142iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8799): Process ended prematurely: osmo-stp_10.42.42.5(pid=8799) [trial-2367↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8799)]
----------------------------------------------
trial-2367 gprs:trx-lms iperf3.py
----------------------------------------------
01:28:35.311007 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:35.322411 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
01:28:35.468572 tst iperf3.py: using LAC 25038
01:28:35.596737 tst iperf3.py: using RAC 48
01:28:35.723180 tst iperf3.py: using CellId 25038
01:28:35.849766 tst iperf3.py: using BVCI 25039
01:28:35.874503 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:35.885715 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:35.896812 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:35.907736 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:35.919856 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:35.930669 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:35.941870 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:35.952514 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:35.963476 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
01:28:36.255793 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
01:28:36.390756 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:28:36.726333 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8791): Launched
01:28:37.006144 run iperf3-srv_10.42.42.10(pid=8792): Launched
01:28:37.203637 tst iperf3.py:8: start network...
01:28:37.338372 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:28:37.857889 run create_hlr_db(pid=8793): Launched
01:28:38.045094 bus /gobi_6: Setting Powered False
01:28:39.066545 run create_hlr_db(pid=8793): Terminated: ok {rc=0}
01:28:39.473232 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8795): Launched
01:28:39.777515 run osmo-hlr_10.42.42.2(pid=8796): Launched
01:28:39.896473 run osmo-stp_10.42.42.5: Starting osmo-stp
01:28:40.432422 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8798): Launched
01:28:40.736754 run osmo-stp_10.42.42.5(pid=8799): Launched
01:28:40.847071 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:28:41.374255 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8801): Launched
01:28:41.481865 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:28:41.745826 run patchelf(pid=8802): Launched
01:28:41.923286 run osmo-stp_10.42.42.5(pid=8799): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8799)]
01:28:42.052864 run osmo-stp_10.42.42.5(pid=8799): stdout:
| (launched: 2022-12-10_01:28:40.587222)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:28:42.182719 run osmo-stp_10.42.42.5(pid=8799): stderr:
| �[0;m�[38;5;43m20221210012840759 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012840760 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012840760 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012840760 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012840761 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012840761 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012840761 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:28:42.308029 run osmo-stp_10.42.42.5(pid=8799): stdout:
| (launched: 2022-12-10_01:28:40.587222)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:28:42.432776 run osmo-stp_10.42.42.5(pid=8799): stderr:
| �[0;m�[38;5;43m20221210012840759 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012840760 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012840760 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012840760 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012840761 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012840761 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012840761 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:28:42.577317 run patchelf(pid=8802): Terminating (SIGINT)
01:28:42.746692 run patchelf(pid=8802): Terminated: ok {rc=0}
01:28:42.777868 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=8799): Process ended prematurely: osmo-stp_10.42.42.5(pid=8799) [trial-2367↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8799)] [trial-2367↪gprs:trx-lms↪iperf3.py:8]
01:28:42.787035 tst iperf3.py:8: Test FAILED (7.5 sec)
01:28:42.848394 run osmo-hlr_10.42.42.2(pid=8796): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-lms↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8796)]
01:28:42.898047 run osmo-hlr_10.42.42.2(pid=8796): stdout:
| (launched: 2022-12-10_01:28:39.627506)
01:28:42.948253 run osmo-hlr_10.42.42.2(pid=8796): stderr:
| �[0;m�[1;31m20221210012839802 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210012839802 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210012839802 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210012839802 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210012839802 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210012839802 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210012839814 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-lms/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210012839824 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210012839824 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210012839824 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:28:42.990833 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8791): Terminating (SIGTERM)
01:28:43.031315 run iperf3-srv_10.42.42.10(pid=8792): Terminating (SIGTERM)
01:28:43.071684 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8795): Terminating (SIGTERM)
01:28:43.112064 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8798): Terminating (SIGTERM)
01:28:43.152571 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8801): Terminating (SIGTERM)
01:28:43.162078 --- ParallelTerminationStrategy: PID 8791 died...
01:28:43.216226 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8791): Terminated: ok {rc=0}
01:28:43.225990 --- ParallelTerminationStrategy: PID 8792 died...
01:28:43.279261 run iperf3-srv_10.42.42.10(pid=8792): Terminated {rc=256}
01:28:43.289092 --- ParallelTerminationStrategy: PID 8795 died...
01:28:43.342435 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8795): Terminated: ok {rc=0}
01:28:43.352380 --- ParallelTerminationStrategy: PID 8798 died...
01:28:43.405641 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8798): Terminated: ok {rc=0}
01:28:43.415569 --- ParallelTerminationStrategy: PID 8801 died...
01:28:43.469148 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8801): Terminated: ok {rc=0}
01:28:43.557351 bus /gobi_6: Setting Powered False
229gprs:trx-lms11.0osmo-stp_10.42.42.5(pid=8817): Process ended prematurely: osmo-stp_10.42.42.5(pid=8817) [trial-2367↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8817)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8817): Process ended prematurely: osmo-stp_10.42.42.5(pid=8817) [trial-2367↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8817)]
----------------------------------------------
trial-2367 gprs:trx-lms iperf3m4.py
----------------------------------------------
01:28:44.667129 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:44.678610 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
01:28:44.822886 tst iperf3m4.py: using LAC 25039
01:28:44.954742 tst iperf3m4.py: using RAC 49
01:28:45.075123 tst iperf3m4.py: using CellId 25039
01:28:45.191689 tst iperf3m4.py: using BVCI 25040
01:28:45.214833 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:45.225512 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:45.235975 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:45.246887 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:45.257608 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:45.268144 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:45.279071 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:45.289563 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:28:45.300236 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
01:28:45.419399 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
01:28:45.541987 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
01:28:45.658019 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
01:28:45.957876 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
01:28:46.099749 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:28:46.435359 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8803): Launched
01:28:46.722328 run iperf3-srv_10.42.42.10(pid=8804): Launched
01:28:46.925198 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
01:28:47.062959 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:28:47.402620 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8805): Launched
01:28:47.682517 run iperf3-srv_10.42.42.10(pid=8806): Launched
01:28:47.880043 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
01:28:48.015238 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:28:48.348348 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8807): Launched
01:28:48.625834 run iperf3-srv_10.42.42.10(pid=8808): Launched
01:28:48.823145 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
01:28:48.957689 run iperf3-srv_10.42.42.10: Starting iperf3-srv
01:28:49.292276 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8809): Launched
01:28:49.572316 run iperf3-srv_10.42.42.10(pid=8810): Launched
01:28:49.769001 tst iperf3m4.py:8: start network...
01:28:49.903174 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:28:50.422251 run create_hlr_db(pid=8811): Launched
01:28:50.608704 bus /gobi_6: Setting Powered False
01:28:51.630187 run create_hlr_db(pid=8811): Terminated: ok {rc=0}
01:28:52.035450 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8813): Launched
01:28:52.338855 run osmo-hlr_10.42.42.2(pid=8814): Launched
01:28:52.457012 run osmo-stp_10.42.42.5: Starting osmo-stp
01:28:52.991910 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8816): Launched
01:28:53.295452 run osmo-stp_10.42.42.5(pid=8817): Launched
01:28:53.406350 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:28:53.932911 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8819): Launched
01:28:54.039718 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:28:54.302196 run patchelf(pid=8820): Launched
01:28:54.479040 run osmo-stp_10.42.42.5(pid=8817): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8817)]
01:28:54.607529 run osmo-stp_10.42.42.5(pid=8817): stdout:
| (launched: 2022-12-10_01:28:53.146014)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:28:54.736277 run osmo-stp_10.42.42.5(pid=8817): stderr:
| �[0;m�[38;5;43m20221210012853316 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012853317 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012853317 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012853317 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012853318 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012853318 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012853318 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:28:54.861062 run osmo-stp_10.42.42.5(pid=8817): stdout:
| (launched: 2022-12-10_01:28:53.146014)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:28:54.984576 run osmo-stp_10.42.42.5(pid=8817): stderr:
| �[0;m�[38;5;43m20221210012853316 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012853317 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012853317 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012853317 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012853318 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012853318 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012853318 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:28:55.128266 run patchelf(pid=8820): Terminating (SIGINT)
01:28:55.296581 run patchelf(pid=8820): Terminated: ok {rc=0}
01:28:55.327518 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=8817): Process ended prematurely: osmo-stp_10.42.42.5(pid=8817) [trial-2367↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8817)] [trial-2367↪gprs:trx-lms↪iperf3m4.py:8]
01:28:55.336717 tst iperf3m4.py:8: Test FAILED (10.7 sec)
01:28:55.397580 run osmo-hlr_10.42.42.2(pid=8814): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-lms↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8814)]
01:28:55.446445 run osmo-hlr_10.42.42.2(pid=8814): stdout:
| (launched: 2022-12-10_01:28:52.188915)
01:28:55.496258 run osmo-hlr_10.42.42.2(pid=8814): stderr:
| �[0;m�[1;31m20221210012852367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210012852367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210012852367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210012852367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210012852367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210012852368 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210012852380 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-lms/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210012852389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210012852389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210012852389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:28:55.537995 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8803): Terminating (SIGTERM)
01:28:55.577743 run iperf3-srv_10.42.42.10(pid=8804): Terminating (SIGTERM)
01:28:55.617414 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8805): Terminating (SIGTERM)
01:28:55.656827 run iperf3-srv_10.42.42.10(pid=8806): Terminating (SIGTERM)
01:28:55.696490 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8807): Terminating (SIGTERM)
01:28:55.736165 run iperf3-srv_10.42.42.10(pid=8808): Terminating (SIGTERM)
01:28:55.775326 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8809): Terminating (SIGTERM)
01:28:55.814728 run iperf3-srv_10.42.42.10(pid=8810): Terminating (SIGTERM)
01:28:55.854322 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8813): Terminating (SIGTERM)
01:28:55.894091 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8816): Terminating (SIGTERM)
01:28:55.933510 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8819): Terminating (SIGTERM)
01:28:55.942897 --- ParallelTerminationStrategy: PID 8803 died...
01:28:55.995768 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8803): Terminated: ok {rc=0}
01:28:56.005270 --- ParallelTerminationStrategy: PID 8804 died...
01:28:56.057355 run iperf3-srv_10.42.42.10(pid=8804): Terminated {rc=256}
01:28:56.067303 --- ParallelTerminationStrategy: PID 8805 died...
01:28:56.119963 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8805): Terminated: ok {rc=0}
01:28:56.129886 --- ParallelTerminationStrategy: PID 8806 died...
01:28:56.182477 run iperf3-srv_10.42.42.10(pid=8806): Terminated {rc=256}
01:28:56.191881 --- ParallelTerminationStrategy: PID 8807 died...
01:28:56.244779 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8807): Terminated: ok {rc=0}
01:28:56.254233 --- ParallelTerminationStrategy: PID 8808 died...
01:28:56.306470 run iperf3-srv_10.42.42.10(pid=8808): Terminated {rc=256}
01:28:56.316134 --- ParallelTerminationStrategy: PID 8809 died...
01:28:56.368723 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8809): Terminated: ok {rc=0}
01:28:56.378189 --- ParallelTerminationStrategy: PID 8810 died...
01:28:56.430760 run iperf3-srv_10.42.42.10(pid=8810): Terminated {rc=256}
01:28:56.440247 --- ParallelTerminationStrategy: PID 8813 died...
01:28:56.492696 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8813): Terminated: ok {rc=0}
01:28:56.502506 --- ParallelTerminationStrategy: PID 8816 died...
01:28:56.555229 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8816): Terminated: ok {rc=0}
01:28:56.565098 --- ParallelTerminationStrategy: PID 8819 died...
01:28:56.617431 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8819): Terminated: ok {rc=0}
01:28:56.705942 bus /gobi_4: Setting Powered False
01:28:57.810545 bus /gobi_4: Setting Powered False
01:28:58.915792 bus /gobi_1: Setting Powered False
01:29:00.017856 bus /gobi_1: Setting Powered False
01:29:01.140847 bus /gobi_0: Setting Powered False
01:29:02.247507 bus /gobi_0: Setting Powered False
01:29:03.357788 bus /gobi_6: Setting Powered False
41gprs:trx-lms7.0osmo-stp_10.42.42.5(pid=8827): Process ended prematurely: osmo-stp_10.42.42.5(pid=8827) [trial-2367↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8827)]2327ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8827): Process ended prematurely: osmo-stp_10.42.42.5(pid=8827) [trial-2367↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8827)]
----------------------------------------------
trial-2367 gprs:trx-lms ping.py
----------------------------------------------
01:29:04.425173 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:04.436484 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
01:29:04.571922 tst ping.py: using LAC 25040
01:29:04.687302 tst ping.py: using RAC 50
01:29:04.802665 tst ping.py: using CellId 25040
01:29:04.917523 tst ping.py: using BVCI 25041
01:29:04.940091 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:04.950620 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:04.960854 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:04.971329 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:04.981831 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:04.992268 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:05.003146 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:05.014102 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
01:29:05.303571 tst ping.py:19: start network...
01:29:05.435315 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:29:05.942074 run create_hlr_db(pid=8821): Launched
01:29:06.124730 bus /gobi_6: Setting Powered False
01:29:07.145610 run create_hlr_db(pid=8821): Terminated: ok {rc=0}
01:29:07.551470 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8823): Launched
01:29:07.869457 run osmo-hlr_10.42.42.2(pid=8824): Launched
01:29:07.985167 run osmo-stp_10.42.42.5: Starting osmo-stp
01:29:08.542109 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8826): Launched
01:29:08.859250 run osmo-stp_10.42.42.5(pid=8827): Launched
01:29:08.973783 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:29:09.525577 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8829): Launched
01:29:09.638025 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:29:09.914488 run patchelf(pid=8830): Launched
01:29:10.100174 run osmo-stp_10.42.42.5(pid=8827): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8827)]
01:29:10.235594 run osmo-stp_10.42.42.5(pid=8827): stdout:
| (launched: 2022-12-10_01:29:08.703086)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:29:10.371114 run osmo-stp_10.42.42.5(pid=8827): stderr:
| �[0;m�[38;5;43m20221210012908895 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012908895 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012908895 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012908896 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012908898 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012908898 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012908898 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:29:10.502394 run osmo-stp_10.42.42.5(pid=8827): stdout:
| (launched: 2022-12-10_01:29:08.703086)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:29:10.632390 run osmo-stp_10.42.42.5(pid=8827): stderr:
| �[0;m�[38;5;43m20221210012908895 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012908895 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012908895 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012908896 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012908898 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012908898 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012908898 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:29:10.783049 run patchelf(pid=8830): Terminating (SIGINT)
01:29:10.960137 run patchelf(pid=8830): Terminated: ok {rc=0}
01:29:10.992391 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=8827): Process ended prematurely: osmo-stp_10.42.42.5(pid=8827) [trial-2367↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8827)] [trial-2367↪gprs:trx-lms↪ping.py:22]
01:29:11.001625 tst ping.py:22: Test FAILED (6.6 sec)
01:29:11.064326 run osmo-hlr_10.42.42.2(pid=8824): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-lms↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8824)]
01:29:11.115502 run osmo-hlr_10.42.42.2(pid=8824): stdout:
| (launched: 2022-12-10_01:29:07.712864)
01:29:11.167517 run osmo-hlr_10.42.42.2(pid=8824): stderr:
| �[0;m�[1;31m20221210012907913 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210012907913 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210012907913 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210012907913 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210012907914 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210012907914 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210012907927 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-lms/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210012907937 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210012907938 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210012907938 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:29:11.211542 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8823): Terminating (SIGTERM)
01:29:11.253502 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8826): Terminating (SIGTERM)
01:29:11.295285 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8829): Terminating (SIGTERM)
01:29:11.305078 --- ParallelTerminationStrategy: PID 8823 died...
01:29:11.360721 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8823): Terminated: ok {rc=0}
01:29:11.370781 --- ParallelTerminationStrategy: PID 8826 died...
01:29:11.425806 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8826): Terminated: ok {rc=0}
01:29:11.435839 --- ParallelTerminationStrategy: PID 8829 died...
01:29:11.490907 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8829): Terminated: ok {rc=0}
01:29:11.578231 bus /gobi_6: Setting Powered False
48gprs:trx-lms7.0osmo-stp_10.42.42.5(pid=8837): Process ended prematurely: osmo-stp_10.42.42.5(pid=8837) [trial-2367↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8837)]2320ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8837): Process ended prematurely: osmo-stp_10.42.42.5(pid=8837) [trial-2367↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8837)]
----------------------------------------------
trial-2367 gprs:trx-lms ping_idle_ping.py
----------------------------------------------
01:29:12.632496 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:12.644622 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
01:29:12.788821 tst ping_idle_ping.py: using LAC 25041
01:29:12.915874 tst ping_idle_ping.py: using RAC 51
01:29:13.040700 tst ping_idle_ping.py: using CellId 25041
01:29:13.165203 tst ping_idle_ping.py: using BVCI 25042
01:29:13.189667 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:13.200617 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:13.211556 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:13.222447 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:13.233303 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:13.244110 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:13.255060 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
01:29:13.266218 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
01:29:13.539465 tst ping_idle_ping.py:24: start network...
01:29:13.663286 run osmo-hlr_10.42.42.2: Starting osmo-hlr
01:29:14.134769 run create_hlr_db(pid=8831): Launched
01:29:14.306549 bus /gobi_6: Setting Powered False
01:29:15.328465 run create_hlr_db(pid=8831): Terminated: ok {rc=0}
01:29:15.725451 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8833): Launched
01:29:16.130543 run osmo-hlr_10.42.42.2(pid=8834): Launched
01:29:16.286460 run osmo-stp_10.42.42.5: Starting osmo-stp
01:29:16.964464 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8836): Launched
01:29:17.289624 run osmo-stp_10.42.42.5(pid=8837): Launched
01:29:17.430166 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
01:29:17.959831 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8839): Launched
01:29:18.065848 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
01:29:18.328362 run patchelf(pid=8840): Launched
01:29:18.502441 run osmo-stp_10.42.42.5(pid=8837): ERR: Terminated: ERROR {rc=1} [trial-2367↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8837)]
01:29:18.629663 run osmo-stp_10.42.42.5(pid=8837): stdout:
| (launched: 2022-12-10_01:29:17.116560)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:29:18.757507 run osmo-stp_10.42.42.5(pid=8837): stderr:
| �[0;m�[38;5;43m20221210012917390 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012917390 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012917390 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012917390 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012917391 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012917391 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012917391 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:29:18.880926 run osmo-stp_10.42.42.5(pid=8837): stdout:
| (launched: 2022-12-10_01:29:17.116560)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
01:29:19.004180 run osmo-stp_10.42.42.5(pid=8837): stderr:
| �[0;m�[38;5;43m20221210012917390 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210012917390 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210012917390 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210012917390 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210012917391 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210012917391 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210012917391 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
01:29:19.146563 run patchelf(pid=8840): Terminating (SIGINT)
01:29:19.313253 run patchelf(pid=8840): Terminated: ok {rc=0}
01:29:19.343856 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=8837): Process ended prematurely: osmo-stp_10.42.42.5(pid=8837) [trial-2367↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8837)] [trial-2367↪gprs:trx-lms↪ping_idle_ping.py:27]
01:29:19.352649 tst ping_idle_ping.py:27: Test FAILED (6.7 sec)
01:29:19.413463 run osmo-hlr_10.42.42.2(pid=8834): ERR: Terminated: ERROR {rc=237} [trial-2367↪gprs:trx-lms↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8834)]
01:29:19.462881 run osmo-hlr_10.42.42.2(pid=8834): stdout:
| (launched: 2022-12-10_01:29:15.883668)
01:29:19.512642 run osmo-hlr_10.42.42.2(pid=8834): stderr:
| �[0;m�[1;31m20221210012916244 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210012916244 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210012916244 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210012916244 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210012916244 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210012916244 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210012916272 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2367/run.2022-12-10_00-40-00/gprs:trx-lms/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210012916290 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210012916290 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210012916290 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
01:29:19.555199 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8833): Terminating (SIGTERM)
01:29:19.595518 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8836): Terminating (SIGTERM)
01:29:19.635720 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8839): Terminating (SIGTERM)
01:29:19.645356 --- ParallelTerminationStrategy: PID 8833 died...
01:29:19.698737 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8833): Terminated: ok {rc=0}
01:29:19.708719 --- ParallelTerminationStrategy: PID 8836 died...
01:29:19.762550 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8836): Terminated: ok {rc=0}
01:29:19.772173 --- ParallelTerminationStrategy: PID 8839 died...
01:29:19.825365 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8839): Terminated: ok {rc=0}
01:29:19.914186 bus /gobi_6: Setting Powered False
5gprs:trx-lms00