77gprs:trx-b2009.0osmo-stp_10.42.42.5(pid=28167): Process ended prematurely: osmo-stp_10.42.42.5(pid=28167) [trial-2394↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28167)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=28167): Process ended prematurely: osmo-stp_10.42.42.5(pid=28167) [trial-2394↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28167)]
----------------------------------------------
trial-2394 gprs:trx-b200 cs_paging_gprs_active.py
----------------------------------------------
20:28:36.924506 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:36.935125 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
20:28:37.071361 tst cs_paging_gprs_active.py: using LAC 36077
20:28:37.193471 tst cs_paging_gprs_active.py: using RAC 122
20:28:37.315459 tst cs_paging_gprs_active.py: using CellId 36077
20:28:37.437761 tst cs_paging_gprs_active.py: using BVCI 36078
20:28:37.461271 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:37.471832 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:37.482815 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:37.493301 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:37.504417 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:37.515627 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:37.527133 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:37.538248 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:37.549585 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
20:28:37.672160 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
20:28:37.955340 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
20:28:38.083978 run iperf3-srv_10.42.42.10: Starting iperf3-srv
20:28:38.402134 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28157): Launched
20:28:38.669614 run iperf3-srv_10.42.42.10(pid=28158): Launched
20:28:38.856831 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
20:28:38.985251 run iperf3-srv_10.42.42.10: Starting iperf3-srv
20:28:39.305719 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28159): Launched
20:28:39.573655 run iperf3-srv_10.42.42.10(pid=28160): Launched
20:28:39.760057 tst cs_paging_gprs_active.py:38: start network...
20:28:39.887712 run osmo-hlr_10.42.42.2: Starting osmo-hlr
20:28:40.370819 run create_hlr_db(pid=28161): Launched
20:28:40.547345 bus /sierra_2: Setting Powered False
20:28:41.569511 run create_hlr_db(pid=28161): Terminated: ok {rc=0}
20:28:42.007851 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28163): Launched
20:28:42.338232 run osmo-hlr_10.42.42.2(pid=28164): Launched
20:28:42.461771 run osmo-stp_10.42.42.5: Starting osmo-stp
20:28:43.035105 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28166): Launched
20:28:43.366684 run osmo-stp_10.42.42.5(pid=28167): Launched
20:28:43.486392 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
20:28:44.062342 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28169): Launched
20:28:44.180544 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
20:28:44.470580 run patchelf(pid=28170): Launched
20:28:44.663321 run osmo-stp_10.42.42.5(pid=28167): ERR: Terminated: ERROR {rc=1} [trial-2394↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28167)]
20:28:44.803545 run osmo-stp_10.42.42.5(pid=28167): stdout:
| (launched: 2022-12-13_20:28:43.203456)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:28:44.943891 run osmo-stp_10.42.42.5(pid=28167): stderr:
| �[0;m�[38;5;43m20221213202843393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202843393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202843393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202843394 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202843394 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202843394 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202843394 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:28:45.080928 run osmo-stp_10.42.42.5(pid=28167): stdout:
| (launched: 2022-12-13_20:28:43.203456)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:28:45.216385 run osmo-stp_10.42.42.5(pid=28167): stderr:
| �[0;m�[38;5;43m20221213202843393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202843393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202843393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202843394 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202843394 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202843394 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202843394 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:28:45.375125 run patchelf(pid=28170): Terminating (SIGINT)
20:28:45.560596 run patchelf(pid=28170): Terminated: ok {rc=0}
20:28:45.594472 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=28167): Process ended prematurely: osmo-stp_10.42.42.5(pid=28167) [trial-2394↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28167)] [trial-2394↪gprs:trx-b200↪cs_paging_gprs_active.py:38]
20:28:45.604356 tst cs_paging_gprs_active.py:38: Test FAILED (8.7 sec)
20:28:45.667360 run osmo-hlr_10.42.42.2(pid=28164): ERR: Terminated: ERROR {rc=237} [trial-2394↪gprs:trx-b200↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=28164)]
20:28:45.718945 run osmo-hlr_10.42.42.2(pid=28164): stdout:
| (launched: 2022-12-13_20:28:42.175522)
20:28:45.771267 run osmo-hlr_10.42.42.2(pid=28164): stderr:
| �[0;m�[1;31m20221213202842372 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213202842372 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213202842372 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213202842372 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213202842372 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213202842373 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213202842387 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2394/run.2022-12-13_20-06-05/gprs:trx-b200/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213202842398 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213202842398 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213202842398 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
20:28:45.815974 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28157): Terminating (SIGTERM)
20:28:45.860002 run iperf3-srv_10.42.42.10(pid=28158): Terminating (SIGTERM)
20:28:45.918427 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28159): Terminating (SIGTERM)
20:28:45.963061 run iperf3-srv_10.42.42.10(pid=28160): Terminating (SIGTERM)
20:28:46.007760 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28163): Terminating (SIGTERM)
20:28:46.051772 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28166): Terminating (SIGTERM)
20:28:46.097885 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28169): Terminating (SIGTERM)
20:28:46.107970 --- ParallelTerminationStrategy: PID 28157 died...
20:28:46.165415 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28157): Terminated: ok {rc=0}
20:28:46.176016 --- ParallelTerminationStrategy: PID 28158 died...
20:28:46.233153 run iperf3-srv_10.42.42.10(pid=28158): Terminated {rc=256}
20:28:46.243676 --- ParallelTerminationStrategy: PID 28159 died...
20:28:46.300200 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28159): Terminated: ok {rc=0}
20:28:46.310606 --- ParallelTerminationStrategy: PID 28160 died...
20:28:46.367617 run iperf3-srv_10.42.42.10(pid=28160): Terminated {rc=256}
20:28:46.378279 --- ParallelTerminationStrategy: PID 28163 died...
20:28:46.434856 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28163): Terminated: ok {rc=0}
20:28:46.445511 --- ParallelTerminationStrategy: PID 28166 died...
20:28:46.502304 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28166): Terminated: ok {rc=0}
20:28:46.513054 --- ParallelTerminationStrategy: PID 28169 died...
20:28:46.569935 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28169): Terminated: ok {rc=0}
20:28:46.661083 bus /gobi_0: Setting Powered False
20:28:47.764069 bus /gobi_0: Setting Powered False
20:28:48.876475 bus /gobi_6: Setting Powered False
77gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=28179): Process ended prematurely: osmo-stp_10.42.42.5(pid=28179) [trial-2394↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28179)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=28179): Process ended prematurely: osmo-stp_10.42.42.5(pid=28179) [trial-2394↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28179)]
----------------------------------------------
trial-2394 gprs:trx-b200 iperf3.py
----------------------------------------------
20:28:49.987330 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:49.998511 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
20:28:50.146232 tst iperf3.py: using LAC 36078
20:28:50.274739 tst iperf3.py: using RAC 123
20:28:50.402968 tst iperf3.py: using CellId 36078
20:28:50.530755 tst iperf3.py: using BVCI 36079
20:28:50.554125 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:50.564831 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:50.575286 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:50.585781 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:50.596787 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:50.607528 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:50.618418 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:50.629098 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:50.639845 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
20:28:50.938661 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
20:28:51.075835 run iperf3-srv_10.42.42.10: Starting iperf3-srv
20:28:51.415956 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28171): Launched
20:28:51.700195 run iperf3-srv_10.42.42.10(pid=28172): Launched
20:28:51.901519 tst iperf3.py:8: start network...
20:28:52.038833 run osmo-hlr_10.42.42.2: Starting osmo-hlr
20:28:52.565340 run create_hlr_db(pid=28173): Launched
20:28:52.754058 bus /gobi_6: Setting Powered False
20:28:53.774839 run create_hlr_db(pid=28173): Terminated: ok {rc=0}
20:28:54.185327 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28175): Launched
20:28:54.492502 run osmo-hlr_10.42.42.2(pid=28176): Launched
20:28:54.613320 run osmo-stp_10.42.42.5: Starting osmo-stp
20:28:55.154017 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28178): Launched
20:28:55.462253 run osmo-stp_10.42.42.5(pid=28179): Launched
20:28:55.575350 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
20:28:56.109918 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28181): Launched
20:28:56.219099 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
20:28:56.486971 run patchelf(pid=28182): Launched
20:28:56.667775 run osmo-stp_10.42.42.5(pid=28179): ERR: Terminated: ERROR {rc=1} [trial-2394↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28179)]
20:28:56.799474 run osmo-stp_10.42.42.5(pid=28179): stdout:
| (launched: 2022-12-13_20:28:55.310240)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:28:56.931625 run osmo-stp_10.42.42.5(pid=28179): stderr:
| �[0;m�[38;5;43m20221213202855478 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202855478 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202855478 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202855479 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202855479 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202855479 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202855479 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:28:57.059710 run osmo-stp_10.42.42.5(pid=28179): stdout:
| (launched: 2022-12-13_20:28:55.310240)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:28:57.186467 run osmo-stp_10.42.42.5(pid=28179): stderr:
| �[0;m�[38;5;43m20221213202855478 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202855478 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202855478 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202855479 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202855479 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202855479 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202855479 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:28:57.334047 run patchelf(pid=28182): Terminating (SIGINT)
20:28:57.508269 run patchelf(pid=28182): Terminated: ok {rc=0}
20:28:57.540159 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=28179): Process ended prematurely: osmo-stp_10.42.42.5(pid=28179) [trial-2394↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28179)] [trial-2394↪gprs:trx-b200↪iperf3.py:8]
20:28:57.549601 tst iperf3.py:8: Test FAILED (7.6 sec)
20:28:57.610557 run osmo-hlr_10.42.42.2(pid=28176): ERR: Terminated: ERROR {rc=237} [trial-2394↪gprs:trx-b200↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=28176)]
20:28:57.660173 run osmo-hlr_10.42.42.2(pid=28176): stdout:
| (launched: 2022-12-13_20:28:54.341115)
20:28:57.710501 run osmo-hlr_10.42.42.2(pid=28176): stderr:
| �[0;m�[1;31m20221213202854515 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213202854515 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213202854515 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213202854515 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213202854515 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213202854515 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213202854527 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2394/run.2022-12-13_20-06-05/gprs:trx-b200/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213202854536 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213202854536 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213202854536 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
20:28:57.753544 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28171): Terminating (SIGTERM)
20:28:57.794120 run iperf3-srv_10.42.42.10(pid=28172): Terminating (SIGTERM)
20:28:57.834685 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28175): Terminating (SIGTERM)
20:28:57.875070 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28178): Terminating (SIGTERM)
20:28:57.916210 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28181): Terminating (SIGTERM)
20:28:57.925592 --- ParallelTerminationStrategy: PID 28171 died...
20:28:57.980161 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28171): Terminated: ok {rc=0}
20:28:57.989837 --- ParallelTerminationStrategy: PID 28172 died...
20:28:58.044037 run iperf3-srv_10.42.42.10(pid=28172): Terminated {rc=256}
20:28:58.053674 --- ParallelTerminationStrategy: PID 28175 died...
20:28:58.108301 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28175): Terminated: ok {rc=0}
20:28:58.117963 --- ParallelTerminationStrategy: PID 28178 died...
20:28:58.172002 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28178): Terminated: ok {rc=0}
20:28:58.181650 --- ParallelTerminationStrategy: PID 28181 died...
20:28:58.235454 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28181): Terminated: ok {rc=0}
20:28:58.320068 bus /gobi_6: Setting Powered False
209gprs:trx-b20012.0osmo-stp_10.42.42.5(pid=28197): Process ended prematurely: osmo-stp_10.42.42.5(pid=28197) [trial-2394↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28197)]2186iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=28197): Process ended prematurely: osmo-stp_10.42.42.5(pid=28197) [trial-2394↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28197)]
----------------------------------------------
trial-2394 gprs:trx-b200 iperf3m4.py
----------------------------------------------
20:28:59.436439 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:28:59.448232 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
20:28:59.595089 tst iperf3m4.py: using LAC 36079
20:28:59.725032 tst iperf3m4.py: using RAC 124
20:28:59.855077 tst iperf3m4.py: using CellId 36079
20:28:59.981136 tst iperf3m4.py: using BVCI 36080
20:29:00.006335 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:00.017714 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:00.028692 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:00.039772 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:00.050694 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:00.061614 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:00.072768 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:00.083826 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:00.095067 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
20:29:00.214784 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
20:29:00.338136 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
20:29:00.455361 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
20:29:00.753233 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
20:29:00.895877 run iperf3-srv_10.42.42.10: Starting iperf3-srv
20:29:01.239119 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28183): Launched
20:29:01.559988 run iperf3-srv_10.42.42.10(pid=28184): Launched
20:29:01.787490 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
20:29:01.927374 run iperf3-srv_10.42.42.10: Starting iperf3-srv
20:29:02.282458 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28185): Launched
20:29:02.567843 run iperf3-srv_10.42.42.10(pid=28186): Launched
20:29:02.770514 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
20:29:02.908578 run iperf3-srv_10.42.42.10: Starting iperf3-srv
20:29:03.249171 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28187): Launched
20:29:03.532531 run iperf3-srv_10.42.42.10(pid=28188): Launched
20:29:03.733734 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
20:29:03.871077 run iperf3-srv_10.42.42.10: Starting iperf3-srv
20:29:04.212812 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28189): Launched
20:29:04.498147 run iperf3-srv_10.42.42.10(pid=28190): Launched
20:29:04.699296 tst iperf3m4.py:8: start network...
20:29:04.836292 run osmo-hlr_10.42.42.2: Starting osmo-hlr
20:29:05.365258 run create_hlr_db(pid=28191): Launched
20:29:05.555434 bus /gobi_6: Setting Powered False
20:29:06.577144 run create_hlr_db(pid=28191): Terminated: ok {rc=0}
20:29:07.014961 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28193): Launched
20:29:07.347174 run osmo-hlr_10.42.42.2(pid=28194): Launched
20:29:07.470969 run osmo-stp_10.42.42.5: Starting osmo-stp
20:29:08.047219 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28196): Launched
20:29:08.377953 run osmo-stp_10.42.42.5(pid=28197): Launched
20:29:08.497278 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
20:29:09.071478 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28199): Launched
20:29:09.189270 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
20:29:09.479452 run patchelf(pid=28200): Launched
20:29:09.671119 run osmo-stp_10.42.42.5(pid=28197): ERR: Terminated: ERROR {rc=1} [trial-2394↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28197)]
20:29:09.811717 run osmo-stp_10.42.42.5(pid=28197): stdout:
| (launched: 2022-12-13_20:29:08.214807)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:29:09.952123 run osmo-stp_10.42.42.5(pid=28197): stderr:
| �[0;m�[38;5;43m20221213202908403 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202908403 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202908403 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202908404 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202908404 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202908404 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202908405 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:29:10.089063 run osmo-stp_10.42.42.5(pid=28197): stdout:
| (launched: 2022-12-13_20:29:08.214807)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:29:10.224390 run osmo-stp_10.42.42.5(pid=28197): stderr:
| �[0;m�[38;5;43m20221213202908403 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202908403 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202908403 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202908404 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202908404 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202908404 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202908405 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:29:10.382368 run patchelf(pid=28200): Terminating (SIGINT)
20:29:10.567412 run patchelf(pid=28200): Terminated: ok {rc=0}
20:29:10.601332 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=28197): Process ended prematurely: osmo-stp_10.42.42.5(pid=28197) [trial-2394↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28197)] [trial-2394↪gprs:trx-b200↪iperf3m4.py:8]
20:29:10.611286 tst iperf3m4.py:8: Test FAILED (11.2 sec)
20:29:10.674656 run osmo-hlr_10.42.42.2(pid=28194): ERR: Terminated: ERROR {rc=237} [trial-2394↪gprs:trx-b200↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=28194)]
20:29:10.726327 run osmo-hlr_10.42.42.2(pid=28194): stdout:
| (launched: 2022-12-13_20:29:07.182884)
20:29:10.778354 run osmo-hlr_10.42.42.2(pid=28194): stderr:
| �[0;m�[1;31m20221213202907379 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213202907379 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213202907380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213202907380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213202907380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213202907380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213202907392 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2394/run.2022-12-13_20-06-05/gprs:trx-b200/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213202907401 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213202907401 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213202907401 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
20:29:10.822895 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28183): Terminating (SIGTERM)
20:29:10.865199 run iperf3-srv_10.42.42.10(pid=28184): Terminating (SIGTERM)
20:29:10.907586 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28185): Terminating (SIGTERM)
20:29:10.949495 run iperf3-srv_10.42.42.10(pid=28186): Terminating (SIGTERM)
20:29:10.991266 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28187): Terminating (SIGTERM)
20:29:11.033233 run iperf3-srv_10.42.42.10(pid=28188): Terminating (SIGTERM)
20:29:11.075473 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28189): Terminating (SIGTERM)
20:29:11.117318 run iperf3-srv_10.42.42.10(pid=28190): Terminating (SIGTERM)
20:29:11.159235 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28193): Terminating (SIGTERM)
20:29:11.201249 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28196): Terminating (SIGTERM)
20:29:11.243587 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28199): Terminating (SIGTERM)
20:29:11.253461 --- ParallelTerminationStrategy: PID 28183 died...
20:29:11.310221 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28183): Terminated: ok {rc=0}
20:29:11.320393 --- ParallelTerminationStrategy: PID 28184 died...
20:29:11.377218 run iperf3-srv_10.42.42.10(pid=28184): Terminated {rc=256}
20:29:11.387428 --- ParallelTerminationStrategy: PID 28185 died...
20:29:11.443574 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28185): Terminated: ok {rc=0}
20:29:11.453734 --- ParallelTerminationStrategy: PID 28186 died...
20:29:11.509695 run iperf3-srv_10.42.42.10(pid=28186): Terminated {rc=256}
20:29:11.519765 --- ParallelTerminationStrategy: PID 28187 died...
20:29:11.575895 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28187): Terminated: ok {rc=0}
20:29:11.586028 --- ParallelTerminationStrategy: PID 28188 died...
20:29:11.641920 run iperf3-srv_10.42.42.10(pid=28188): Terminated {rc=256}
20:29:11.651977 --- ParallelTerminationStrategy: PID 28189 died...
20:29:11.708097 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=28189): Terminated: ok {rc=0}
20:29:11.718239 --- ParallelTerminationStrategy: PID 28190 died...
20:29:11.774242 run iperf3-srv_10.42.42.10(pid=28190): Terminated {rc=256}
20:29:11.784401 --- ParallelTerminationStrategy: PID 28193 died...
20:29:11.840451 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28193): Terminated: ok {rc=0}
20:29:11.850538 --- ParallelTerminationStrategy: PID 28196 died...
20:29:11.906811 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28196): Terminated: ok {rc=0}
20:29:11.917052 --- ParallelTerminationStrategy: PID 28199 died...
20:29:11.973430 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28199): Terminated: ok {rc=0}
20:29:12.060743 bus /gobi_4: Setting Powered False
20:29:13.169274 bus /gobi_4: Setting Powered False
20:29:14.279347 bus /gobi_1: Setting Powered False
20:29:15.386947 bus /gobi_1: Setting Powered False
20:29:16.497243 bus /gobi_0: Setting Powered False
20:29:17.603315 bus /gobi_0: Setting Powered False
20:29:18.714019 bus /gobi_6: Setting Powered False
209gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=28207): Process ended prematurely: osmo-stp_10.42.42.5(pid=28207) [trial-2394↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28207)]2186ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=28207): Process ended prematurely: osmo-stp_10.42.42.5(pid=28207) [trial-2394↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28207)]
----------------------------------------------
trial-2394 gprs:trx-b200 ping.py
----------------------------------------------
20:29:19.784057 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:19.795716 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
20:29:19.939962 tst ping.py: using LAC 36080
20:29:20.066531 tst ping.py: using RAC 125
20:29:20.192658 tst ping.py: using CellId 36080
20:29:20.318982 tst ping.py: using BVCI 36081
20:29:20.344200 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:20.355456 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:20.366553 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:20.377685 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:20.388704 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:20.399752 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:20.410916 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:20.421980 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
20:29:20.715859 tst ping.py:19: start network...
20:29:20.849995 run osmo-hlr_10.42.42.2: Starting osmo-hlr
20:29:21.366731 run create_hlr_db(pid=28201): Launched
20:29:21.553853 bus /gobi_6: Setting Powered False
20:29:22.574040 run create_hlr_db(pid=28201): Terminated: ok {rc=0}
20:29:22.981391 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28203): Launched
20:29:23.283804 run osmo-hlr_10.42.42.2(pid=28204): Launched
20:29:23.399640 run osmo-stp_10.42.42.5: Starting osmo-stp
20:29:23.935036 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28206): Launched
20:29:24.238280 run osmo-stp_10.42.42.5(pid=28207): Launched
20:29:24.348427 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
20:29:24.873390 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28209): Launched
20:29:24.979884 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
20:29:25.243285 run patchelf(pid=28210): Launched
20:29:25.420332 run osmo-stp_10.42.42.5(pid=28207): ERR: Terminated: ERROR {rc=1} [trial-2394↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28207)]
20:29:25.549215 run osmo-stp_10.42.42.5(pid=28207): stdout:
| (launched: 2022-12-13_20:29:24.088080)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:29:25.678600 run osmo-stp_10.42.42.5(pid=28207): stderr:
| �[0;m�[38;5;43m20221213202924255 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202924255 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202924255 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202924255 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202924256 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202924256 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202924256 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:29:25.804192 run osmo-stp_10.42.42.5(pid=28207): stdout:
| (launched: 2022-12-13_20:29:24.088080)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:29:25.928622 run osmo-stp_10.42.42.5(pid=28207): stderr:
| �[0;m�[38;5;43m20221213202924255 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202924255 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202924255 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202924255 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202924256 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202924256 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202924256 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:29:26.073793 run patchelf(pid=28210): Terminating (SIGINT)
20:29:26.244459 run patchelf(pid=28210): Terminated: ok {rc=0}
20:29:26.275334 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=28207): Process ended prematurely: osmo-stp_10.42.42.5(pid=28207) [trial-2394↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28207)] [trial-2394↪gprs:trx-b200↪ping.py:22]
20:29:26.284026 tst ping.py:22: Test FAILED (6.5 sec)
20:29:26.345252 run osmo-hlr_10.42.42.2(pid=28204): ERR: Terminated: ERROR {rc=237} [trial-2394↪gprs:trx-b200↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=28204)]
20:29:26.395185 run osmo-hlr_10.42.42.2(pid=28204): stdout:
| (launched: 2022-12-13_20:29:23.134944)
20:29:26.445728 run osmo-hlr_10.42.42.2(pid=28204): stderr:
| �[0;m�[1;31m20221213202923307 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213202923307 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213202923307 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213202923307 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213202923307 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213202923307 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213202923320 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2394/run.2022-12-13_20-06-05/gprs:trx-b200/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213202923328 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213202923329 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213202923329 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
20:29:26.488564 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28203): Terminating (SIGTERM)
20:29:26.529181 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28206): Terminating (SIGTERM)
20:29:26.569749 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28209): Terminating (SIGTERM)
20:29:26.579131 --- ParallelTerminationStrategy: PID 28203 died...
20:29:26.633768 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28203): Terminated: ok {rc=0}
20:29:26.643356 --- ParallelTerminationStrategy: PID 28206 died...
20:29:26.697789 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28206): Terminated: ok {rc=0}
20:29:26.707471 --- ParallelTerminationStrategy: PID 28209 died...
20:29:26.761597 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28209): Terminated: ok {rc=0}
20:29:26.850473 bus /gobi_6: Setting Powered False
209gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=28217): Process ended prematurely: osmo-stp_10.42.42.5(pid=28217) [trial-2394↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28217)]2186ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=28217): Process ended prematurely: osmo-stp_10.42.42.5(pid=28217) [trial-2394↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28217)]
----------------------------------------------
trial-2394 gprs:trx-b200 ping_idle_ping.py
----------------------------------------------
20:29:27.903765 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:27.915417 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
20:29:28.054479 tst ping_idle_ping.py: using LAC 36081
20:29:28.173716 tst ping_idle_ping.py: using RAC 126
20:29:28.292100 tst ping_idle_ping.py: using CellId 36081
20:29:28.413119 tst ping_idle_ping.py: using BVCI 36082
20:29:28.436196 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:28.446917 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:28.457556 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:28.468090 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:28.479428 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:28.491200 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:28.502785 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
20:29:28.514287 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
20:29:28.794258 tst ping_idle_ping.py:24: start network...
20:29:28.920385 run osmo-hlr_10.42.42.2: Starting osmo-hlr
20:29:29.402261 run create_hlr_db(pid=28211): Launched
20:29:29.577202 bus /gobi_6: Setting Powered False
20:29:30.599197 run create_hlr_db(pid=28211): Terminated: ok {rc=0}
20:29:31.044431 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28213): Launched
20:29:31.423308 run osmo-hlr_10.42.42.2(pid=28214): Launched
20:29:31.670778 run osmo-stp_10.42.42.5: Starting osmo-stp
20:29:32.358303 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28216): Launched
20:29:32.661518 run osmo-stp_10.42.42.5(pid=28217): Launched
20:29:32.770953 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
20:29:33.299132 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28219): Launched
20:29:33.406706 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
20:29:33.673710 run patchelf(pid=28220): Launched
20:29:33.851139 run osmo-stp_10.42.42.5(pid=28217): ERR: Terminated: ERROR {rc=1} [trial-2394↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28217)]
20:29:33.980612 run osmo-stp_10.42.42.5(pid=28217): stdout:
| (launched: 2022-12-13_20:29:32.511649)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:29:34.110538 run osmo-stp_10.42.42.5(pid=28217): stderr:
| �[0;m�[38;5;43m20221213202932680 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202932680 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202932680 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202932681 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202932681 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202932681 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202932681 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:29:34.236090 run osmo-stp_10.42.42.5(pid=28217): stdout:
| (launched: 2022-12-13_20:29:32.511649)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
20:29:34.360864 run osmo-stp_10.42.42.5(pid=28217): stderr:
| �[0;m�[38;5;43m20221213202932680 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213202932680 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213202932680 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213202932681 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213202932681 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213202932681 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213202932681 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
20:29:34.506700 run patchelf(pid=28220): Terminating (SIGINT)
20:29:34.677749 run patchelf(pid=28220): Terminated: ok {rc=0}
20:29:34.708655 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=28217): Process ended prematurely: osmo-stp_10.42.42.5(pid=28217) [trial-2394↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=28217)] [trial-2394↪gprs:trx-b200↪ping_idle_ping.py:27]
20:29:34.717455 tst ping_idle_ping.py:27: Test FAILED (6.8 sec)
20:29:34.778496 run osmo-hlr_10.42.42.2(pid=28214): ERR: Terminated: ERROR {rc=237} [trial-2394↪gprs:trx-b200↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=28214)]
20:29:34.828481 run osmo-hlr_10.42.42.2(pid=28214): stdout:
| (launched: 2022-12-13_20:29:31.214684)
20:29:34.879335 run osmo-hlr_10.42.42.2(pid=28214): stderr:
| �[0;m�[1;31m20221213202931598 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213202931598 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213202931598 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213202931598 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213202931598 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213202931598 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213202931611 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2394/run.2022-12-13_20-06-05/gprs:trx-b200/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213202931621 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213202931621 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213202931622 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
20:29:34.923034 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28213): Terminating (SIGTERM)
20:29:34.963577 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28216): Terminating (SIGTERM)
20:29:35.004032 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28219): Terminating (SIGTERM)
20:29:35.013495 --- ParallelTerminationStrategy: PID 28213 died...
20:29:35.068355 run pcap-recorder_any(filters='host 10.42.42.2')(pid=28213): Terminated: ok {rc=0}
20:29:35.078193 --- ParallelTerminationStrategy: PID 28216 died...
20:29:35.132722 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=28216): Terminated: ok {rc=0}
20:29:35.142607 --- ParallelTerminationStrategy: PID 28219 died...
20:29:35.197213 run pcap-recorder_any(filters='host 10.42.42.6')(pid=28219): Terminated: ok {rc=0}
20:29:35.286441 bus /gobi_6: Setting Powered False
5gprs:trx-b20000