84gprs:trx-umtrx9.0osmo-stp_10.42.42.5(pid=23328): Process ended prematurely: osmo-stp_10.42.42.5(pid=23328) [trial-2401↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23328)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=23328): Process ended prematurely: osmo-stp_10.42.42.5(pid=23328) [trial-2401↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23328)]
----------------------------------------------
trial-2401 gprs:trx-umtrx cs_paging_gprs_active.py
----------------------------------------------
07:36:51.138157 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:36:51.148810 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
07:36:51.287219 tst cs_paging_gprs_active.py: using LAC 39395
07:36:51.407340 tst cs_paging_gprs_active.py: using RAC 125
07:36:51.536025 tst cs_paging_gprs_active.py: using CellId 39395
07:36:51.666605 tst cs_paging_gprs_active.py: using BVCI 39396
07:36:51.692000 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:36:51.703182 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:36:51.714060 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:36:51.725038 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:36:51.736072 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:36:51.747233 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:36:51.758676 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:36:51.769736 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:36:51.781158 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
07:36:51.900334 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
07:36:52.184090 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
07:36:52.312890 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:36:52.634406 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23318): Launched
07:36:52.902818 run iperf3-srv_10.42.42.10(pid=23319): Launched
07:36:53.091645 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
07:36:53.220753 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:36:53.541884 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23320): Launched
07:36:53.810493 run iperf3-srv_10.42.42.10(pid=23321): Launched
07:36:53.999164 tst cs_paging_gprs_active.py:38: start network...
07:36:54.127921 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:36:54.624428 run create_hlr_db(pid=23322): Launched
07:36:54.802328 bus /sierra_2: Setting Powered False
07:36:55.823039 run create_hlr_db(pid=23322): Terminated: ok {rc=0}
07:36:56.267068 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23324): Launched
07:36:56.603425 run osmo-hlr_10.42.42.2(pid=23325): Launched
07:36:56.730010 run osmo-stp_10.42.42.5: Starting osmo-stp
07:36:57.308641 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23327): Launched
07:36:57.643172 run osmo-stp_10.42.42.5(pid=23328): Launched
07:36:57.763990 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:36:58.340296 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23330): Launched
07:36:58.458432 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:36:58.749906 run patchelf(pid=23331): Launched
07:36:58.944828 run osmo-stp_10.42.42.5(pid=23328): ERR: Terminated: ERROR {rc=1} [trial-2401↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23328)]
07:36:59.087431 run osmo-stp_10.42.42.5(pid=23328): stdout:
| (launched: 2022-12-14_07:36:57.478113)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:36:59.229848 run osmo-stp_10.42.42.5(pid=23328): stderr:
| �[0;m�[38;5;43m20221214073657667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073657667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073657667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073657668 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073657668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073657668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073657668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:36:59.367853 run osmo-stp_10.42.42.5(pid=23328): stdout:
| (launched: 2022-12-14_07:36:57.478113)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:36:59.504712 run osmo-stp_10.42.42.5(pid=23328): stderr:
| �[0;m�[38;5;43m20221214073657667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073657667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073657667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073657668 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073657668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073657668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073657668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:36:59.665104 run patchelf(pid=23331): Terminating (SIGINT)
07:36:59.853742 run patchelf(pid=23331): Terminated: ok {rc=0}
07:36:59.888668 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=23328): Process ended prematurely: osmo-stp_10.42.42.5(pid=23328) [trial-2401↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23328)] [trial-2401↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38]
07:36:59.898848 tst cs_paging_gprs_active.py:38: Test FAILED (8.8 sec)
07:36:59.962551 run osmo-hlr_10.42.42.2(pid=23325): ERR: Terminated: ERROR {rc=237} [trial-2401↪gprs:trx-umtrx↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=23325)]
07:37:00.014177 run osmo-hlr_10.42.42.2(pid=23325): stdout:
| (launched: 2022-12-14_07:36:56.436384)
07:37:00.066318 run osmo-hlr_10.42.42.2(pid=23325): stderr:
| �[0;m�[1;31m20221214073656631 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214073656631 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214073656631 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214073656631 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214073656631 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214073656631 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214073656643 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2401/run.2022-12-14_06-57-06/gprs:trx-umtrx/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214073656651 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214073656651 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214073656651 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:37:00.111568 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23318): Terminating (SIGTERM)
07:37:00.154051 run iperf3-srv_10.42.42.10(pid=23319): Terminating (SIGTERM)
07:37:00.196472 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23320): Terminating (SIGTERM)
07:37:00.238888 run iperf3-srv_10.42.42.10(pid=23321): Terminating (SIGTERM)
07:37:00.281483 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23324): Terminating (SIGTERM)
07:37:00.324010 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23327): Terminating (SIGTERM)
07:37:00.366713 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23330): Terminating (SIGTERM)
07:37:00.376462 --- ParallelTerminationStrategy: PID 23318 died...
07:37:00.433401 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23318): Terminated: ok {rc=0}
07:37:00.443674 --- ParallelTerminationStrategy: PID 23319 died...
07:37:00.499651 run iperf3-srv_10.42.42.10(pid=23319): Terminated {rc=256}
07:37:00.509864 --- ParallelTerminationStrategy: PID 23320 died...
07:37:00.566145 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23320): Terminated: ok {rc=0}
07:37:00.576202 --- ParallelTerminationStrategy: PID 23321 died...
07:37:00.632554 run iperf3-srv_10.42.42.10(pid=23321): Terminated {rc=256}
07:37:00.642827 --- ParallelTerminationStrategy: PID 23324 died...
07:37:00.699072 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23324): Terminated: ok {rc=0}
07:37:00.709386 --- ParallelTerminationStrategy: PID 23327 died...
07:37:00.765590 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23327): Terminated: ok {rc=0}
07:37:00.775783 --- ParallelTerminationStrategy: PID 23330 died...
07:37:00.835605 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23330): Terminated: ok {rc=0}
07:37:00.939270 bus /gobi_0: Setting Powered False
07:37:02.050853 bus /gobi_0: Setting Powered False
07:37:03.161854 bus /gobi_6: Setting Powered False
84gprs:trx-umtrx8.0osmo-stp_10.42.42.5(pid=23340): Process ended prematurely: osmo-stp_10.42.42.5(pid=23340) [trial-2401↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23340)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=23340): Process ended prematurely: osmo-stp_10.42.42.5(pid=23340) [trial-2401↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23340)]
----------------------------------------------
trial-2401 gprs:trx-umtrx iperf3.py
----------------------------------------------
07:37:04.274107 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:04.285435 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
07:37:04.424351 tst iperf3.py: using LAC 39396
07:37:04.547738 tst iperf3.py: using RAC 126
07:37:04.669216 tst iperf3.py: using CellId 39396
07:37:04.792783 tst iperf3.py: using BVCI 39397
07:37:04.816605 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:04.827389 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:04.838058 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:04.848531 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:04.858892 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:04.869233 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:04.879715 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:04.890076 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:04.900495 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
07:37:05.182755 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
07:37:05.311102 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:37:05.634201 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23332): Launched
07:37:05.900535 run iperf3-srv_10.42.42.10(pid=23333): Launched
07:37:06.088373 tst iperf3.py:8: start network...
07:37:06.216501 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:37:06.711456 run create_hlr_db(pid=23334): Launched
07:37:06.888505 bus /gobi_6: Setting Powered False
07:37:07.909691 run create_hlr_db(pid=23334): Terminated: ok {rc=0}
07:37:08.351432 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23336): Launched
07:37:08.685552 run osmo-hlr_10.42.42.2(pid=23337): Launched
07:37:08.808370 run osmo-stp_10.42.42.5: Starting osmo-stp
07:37:09.389196 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23339): Launched
07:37:09.722257 run osmo-stp_10.42.42.5(pid=23340): Launched
07:37:09.842400 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:37:10.420877 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23342): Launched
07:37:10.539227 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:37:10.831092 run patchelf(pid=23343): Launched
07:37:11.024154 run osmo-stp_10.42.42.5(pid=23340): ERR: Terminated: ERROR {rc=1} [trial-2401↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23340)]
07:37:11.165911 run osmo-stp_10.42.42.5(pid=23340): stdout:
| (launched: 2022-12-14_07:37:09.558515)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:37:11.307917 run osmo-stp_10.42.42.5(pid=23340): stderr:
| �[0;m�[38;5;43m20221214073709749 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073709750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073709750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073709750 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073709751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073709751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073709751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:37:11.445082 run osmo-stp_10.42.42.5(pid=23340): stdout:
| (launched: 2022-12-14_07:37:09.558515)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:37:11.580886 run osmo-stp_10.42.42.5(pid=23340): stderr:
| �[0;m�[38;5;43m20221214073709749 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073709750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073709750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073709750 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073709751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073709751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073709751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:37:11.740171 run patchelf(pid=23343): Terminating (SIGINT)
07:37:11.927387 run patchelf(pid=23343): Terminated: ok {rc=0}
07:37:11.961544 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=23340): Process ended prematurely: osmo-stp_10.42.42.5(pid=23340) [trial-2401↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23340)] [trial-2401↪gprs:trx-umtrx↪iperf3.py:8]
07:37:11.971359 tst iperf3.py:8: Test FAILED (7.7 sec)
07:37:12.034858 run osmo-hlr_10.42.42.2(pid=23337): ERR: Terminated: ERROR {rc=237} [trial-2401↪gprs:trx-umtrx↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=23337)]
07:37:12.086477 run osmo-hlr_10.42.42.2(pid=23337): stdout:
| (launched: 2022-12-14_07:37:08.520410)
07:37:12.138757 run osmo-hlr_10.42.42.2(pid=23337): stderr:
| �[0;m�[1;31m20221214073708717 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214073708717 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214073708717 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214073708717 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214073708717 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214073708717 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214073708730 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2401/run.2022-12-14_06-57-06/gprs:trx-umtrx/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214073708739 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214073708739 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214073708739 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:37:12.183608 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23332): Terminating (SIGTERM)
07:37:12.226193 run iperf3-srv_10.42.42.10(pid=23333): Terminating (SIGTERM)
07:37:12.268606 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23336): Terminating (SIGTERM)
07:37:12.311035 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23339): Terminating (SIGTERM)
07:37:12.353382 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23342): Terminating (SIGTERM)
07:37:12.363184 --- ParallelTerminationStrategy: PID 23332 died...
07:37:12.419876 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23332): Terminated: ok {rc=0}
07:37:12.430102 --- ParallelTerminationStrategy: PID 23333 died...
07:37:12.486247 run iperf3-srv_10.42.42.10(pid=23333): Terminated {rc=256}
07:37:12.496306 --- ParallelTerminationStrategy: PID 23336 died...
07:37:12.552669 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23336): Terminated: ok {rc=0}
07:37:12.562891 --- ParallelTerminationStrategy: PID 23339 died...
07:37:12.619542 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23339): Terminated: ok {rc=0}
07:37:12.629828 --- ParallelTerminationStrategy: PID 23342 died...
07:37:12.686341 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23342): Terminated: ok {rc=0}
07:37:12.774585 bus /gobi_6: Setting Powered False
227gprs:trx-umtrx11.0osmo-stp_10.42.42.5(pid=23358): Process ended prematurely: osmo-stp_10.42.42.5(pid=23358) [trial-2401↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23358)]2175iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=23358): Process ended prematurely: osmo-stp_10.42.42.5(pid=23358) [trial-2401↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23358)]
----------------------------------------------
trial-2401 gprs:trx-umtrx iperf3m4.py
----------------------------------------------
07:37:13.892920 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:13.905046 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
07:37:14.052523 tst iperf3m4.py: using LAC 39397
07:37:14.176214 tst iperf3m4.py: using RAC 127
07:37:14.301657 tst iperf3m4.py: using CellId 39397
07:37:14.421375 tst iperf3m4.py: using BVCI 39398
07:37:14.445156 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:14.455850 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:14.466364 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:14.476756 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:14.487101 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:14.497900 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:14.508440 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:14.518850 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:14.529438 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
07:37:14.651800 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
07:37:14.777594 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
07:37:14.900738 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
07:37:15.202600 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
07:37:15.341790 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:37:15.687377 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23344): Launched
07:37:15.992149 run iperf3-srv_10.42.42.10(pid=23345): Launched
07:37:16.203237 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
07:37:16.347225 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:37:16.708787 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23346): Launched
07:37:17.037120 run iperf3-srv_10.42.42.10(pid=23347): Launched
07:37:17.278251 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
07:37:17.410338 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:37:17.734599 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23348): Launched
07:37:18.001591 run iperf3-srv_10.42.42.10(pid=23349): Launched
07:37:18.190260 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
07:37:18.318469 run iperf3-srv_10.42.42.10: Starting iperf3-srv
07:37:18.640902 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23350): Launched
07:37:18.910224 run iperf3-srv_10.42.42.10(pid=23351): Launched
07:37:19.098240 tst iperf3m4.py:8: start network...
07:37:19.226249 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:37:19.723325 run create_hlr_db(pid=23352): Launched
07:37:19.900565 bus /gobi_6: Setting Powered False
07:37:20.922160 run create_hlr_db(pid=23352): Terminated: ok {rc=0}
07:37:21.365601 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23354): Launched
07:37:21.700644 run osmo-hlr_10.42.42.2(pid=23355): Launched
07:37:21.822314 run osmo-stp_10.42.42.5: Starting osmo-stp
07:37:22.383374 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23357): Launched
07:37:22.694675 run osmo-stp_10.42.42.5(pid=23358): Launched
07:37:22.807512 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:37:23.347648 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23360): Launched
07:37:23.457661 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:37:23.730183 run patchelf(pid=23361): Launched
07:37:23.910719 run osmo-stp_10.42.42.5(pid=23358): ERR: Terminated: ERROR {rc=1} [trial-2401↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23358)]
07:37:24.042720 run osmo-stp_10.42.42.5(pid=23358): stdout:
| (launched: 2022-12-14_07:37:22.541323)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:37:24.174981 run osmo-stp_10.42.42.5(pid=23358): stderr:
| �[0;m�[38;5;43m20221214073722710 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073722710 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073722710 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073722711 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073722711 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073722711 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073722711 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:37:24.301960 run osmo-stp_10.42.42.5(pid=23358): stdout:
| (launched: 2022-12-14_07:37:22.541323)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:37:24.427934 run osmo-stp_10.42.42.5(pid=23358): stderr:
| �[0;m�[38;5;43m20221214073722710 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073722710 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073722710 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073722711 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073722711 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073722711 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073722711 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:37:24.576570 run patchelf(pid=23361): Terminating (SIGINT)
07:37:24.751539 run patchelf(pid=23361): Terminated: ok {rc=0}
07:37:24.783523 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=23358): Process ended prematurely: osmo-stp_10.42.42.5(pid=23358) [trial-2401↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23358)] [trial-2401↪gprs:trx-umtrx↪iperf3m4.py:8]
07:37:24.792767 tst iperf3m4.py:8: Test FAILED (10.9 sec)
07:37:24.852698 run osmo-hlr_10.42.42.2(pid=23355): ERR: Terminated: ERROR {rc=237} [trial-2401↪gprs:trx-umtrx↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=23355)]
07:37:24.901647 run osmo-hlr_10.42.42.2(pid=23355): stdout:
| (launched: 2022-12-14_07:37:21.534612)
07:37:24.951301 run osmo-hlr_10.42.42.2(pid=23355): stderr:
| �[0;m�[1;31m20221214073721730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214073721730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214073721730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214073721730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214073721730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214073721730 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214073721742 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2401/run.2022-12-14_06-57-06/gprs:trx-umtrx/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214073721752 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214073721752 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214073721752 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:37:24.993449 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23344): Terminating (SIGTERM)
07:37:25.033168 run iperf3-srv_10.42.42.10(pid=23345): Terminating (SIGTERM)
07:37:25.073146 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23346): Terminating (SIGTERM)
07:37:25.113014 run iperf3-srv_10.42.42.10(pid=23347): Terminating (SIGTERM)
07:37:25.152912 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23348): Terminating (SIGTERM)
07:37:25.192600 run iperf3-srv_10.42.42.10(pid=23349): Terminating (SIGTERM)
07:37:25.232277 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23350): Terminating (SIGTERM)
07:37:25.271848 run iperf3-srv_10.42.42.10(pid=23351): Terminating (SIGTERM)
07:37:25.311814 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23354): Terminating (SIGTERM)
07:37:25.351524 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23357): Terminating (SIGTERM)
07:37:25.391252 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23360): Terminating (SIGTERM)
07:37:25.400479 --- ParallelTerminationStrategy: PID 23344 died...
07:37:25.453848 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23344): Terminated: ok {rc=0}
07:37:25.463509 --- ParallelTerminationStrategy: PID 23345 died...
07:37:25.516865 run iperf3-srv_10.42.42.10(pid=23345): Terminated {rc=256}
07:37:25.526458 --- ParallelTerminationStrategy: PID 23346 died...
07:37:25.579281 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23346): Terminated: ok {rc=0}
07:37:25.588857 --- ParallelTerminationStrategy: PID 23347 died...
07:37:25.641600 run iperf3-srv_10.42.42.10(pid=23347): Terminated {rc=256}
07:37:25.651108 --- ParallelTerminationStrategy: PID 23348 died...
07:37:25.703717 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23348): Terminated: ok {rc=0}
07:37:25.713254 --- ParallelTerminationStrategy: PID 23349 died...
07:37:25.765807 run iperf3-srv_10.42.42.10(pid=23349): Terminated {rc=256}
07:37:25.775338 --- ParallelTerminationStrategy: PID 23350 died...
07:37:25.828220 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23350): Terminated: ok {rc=0}
07:37:25.837814 --- ParallelTerminationStrategy: PID 23351 died...
07:37:25.890317 run iperf3-srv_10.42.42.10(pid=23351): Terminated {rc=256}
07:37:25.899903 --- ParallelTerminationStrategy: PID 23354 died...
07:37:25.952850 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23354): Terminated: ok {rc=0}
07:37:25.962514 --- ParallelTerminationStrategy: PID 23357 died...
07:37:26.015342 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23357): Terminated: ok {rc=0}
07:37:26.024864 --- ParallelTerminationStrategy: PID 23360 died...
07:37:26.077719 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23360): Terminated: ok {rc=0}
07:37:26.165578 bus /gobi_4: Setting Powered False
07:37:27.268314 bus /gobi_4: Setting Powered False
07:37:28.379449 bus /gobi_1: Setting Powered False
07:37:29.482387 bus /gobi_1: Setting Powered False
07:37:30.595964 bus /gobi_0: Setting Powered False
07:37:31.710875 bus /gobi_0: Setting Powered False
07:37:32.819820 bus /gobi_6: Setting Powered False
202gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=23369): Process ended prematurely: osmo-stp_10.42.42.5(pid=23369) [trial-2401↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23369)]2200ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=23369): Process ended prematurely: osmo-stp_10.42.42.5(pid=23369) [trial-2401↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23369)]
----------------------------------------------
trial-2401 gprs:trx-umtrx ping.py
----------------------------------------------
07:37:33.886091 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:33.897257 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
07:37:34.041617 tst ping.py: using LAC 39398
07:37:34.171475 tst ping.py: using RAC 128
07:37:34.301262 tst ping.py: using CellId 39398
07:37:34.430742 tst ping.py: using BVCI 39399
07:37:34.455959 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:34.467130 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:34.478419 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:34.489449 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:34.500585 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:34.511764 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:34.523220 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:34.534426 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
07:37:34.831384 tst ping.py:19: start network...
07:37:34.967758 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:37:35.479773 run create_hlr_db(pid=23362): Launched
07:37:35.670556 bus /gobi_6: Setting Powered False
07:37:36.689972 run create_hlr_db(pid=23362): Terminated: ok {rc=0}
07:37:37.099252 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23364): Launched
07:37:37.404537 run osmo-hlr_10.42.42.2(pid=23366): Launched
07:37:37.527132 run osmo-stp_10.42.42.5: Starting osmo-stp
07:37:38.063198 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23368): Launched
07:37:38.366435 run osmo-stp_10.42.42.5(pid=23369): Launched
07:37:38.476208 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:37:39.006229 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23371): Launched
07:37:39.113623 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:37:39.378874 run patchelf(pid=23372): Launched
07:37:39.556135 run osmo-stp_10.42.42.5(pid=23369): ERR: Terminated: ERROR {rc=1} [trial-2401↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23369)]
07:37:39.686309 run osmo-stp_10.42.42.5(pid=23369): stdout:
| (launched: 2022-12-14_07:37:38.217269)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:37:39.816037 run osmo-stp_10.42.42.5(pid=23369): stderr:
| �[0;m�[38;5;43m20221214073738384 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073738384 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073738384 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073738385 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073738385 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073738385 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073738385 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:37:39.940735 run osmo-stp_10.42.42.5(pid=23369): stdout:
| (launched: 2022-12-14_07:37:38.217269)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:37:40.064200 run osmo-stp_10.42.42.5(pid=23369): stderr:
| �[0;m�[38;5;43m20221214073738384 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073738384 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073738384 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073738385 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073738385 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073738385 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073738385 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:37:40.209986 run patchelf(pid=23372): Terminating (SIGINT)
07:37:40.381660 run patchelf(pid=23372): Terminated: ok {rc=0}
07:37:40.412812 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=23369): Process ended prematurely: osmo-stp_10.42.42.5(pid=23369) [trial-2401↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23369)] [trial-2401↪gprs:trx-umtrx↪ping.py:22]
07:37:40.421352 tst ping.py:22: Test FAILED (6.5 sec)
07:37:40.481275 run osmo-hlr_10.42.42.2(pid=23366): ERR: Terminated: ERROR {rc=237} [trial-2401↪gprs:trx-umtrx↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=23366)]
07:37:40.530105 run osmo-hlr_10.42.42.2(pid=23366): stdout:
| (launched: 2022-12-14_07:37:37.253180)
07:37:40.580112 run osmo-hlr_10.42.42.2(pid=23366): stderr:
| �[0;m�[1;31m20221214073737428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214073737428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214073737428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214073737428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214073737428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214073737428 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214073737441 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2401/run.2022-12-14_06-57-06/gprs:trx-umtrx/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214073737451 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214073737451 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214073737451 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:37:40.622380 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23364): Terminating (SIGTERM)
07:37:40.662857 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23368): Terminating (SIGTERM)
07:37:40.703192 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23371): Terminating (SIGTERM)
07:37:40.712476 --- ParallelTerminationStrategy: PID 23364 died...
07:37:40.765638 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23364): Terminated: ok {rc=0}
07:37:40.775240 --- ParallelTerminationStrategy: PID 23368 died...
07:37:40.828622 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23368): Terminated: ok {rc=0}
07:37:40.838241 --- ParallelTerminationStrategy: PID 23371 died...
07:37:40.891234 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23371): Terminated: ok {rc=0}
07:37:40.975394 bus /gobi_6: Setting Powered False
263gprs:trx-umtrx8.0osmo-stp_10.42.42.5(pid=23379): Process ended prematurely: osmo-stp_10.42.42.5(pid=23379) [trial-2401↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23379)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=23379): Process ended prematurely: osmo-stp_10.42.42.5(pid=23379) [trial-2401↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23379)]
----------------------------------------------
trial-2401 gprs:trx-umtrx ping_idle_ping.py
----------------------------------------------
07:37:42.028902 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:42.041485 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
07:37:42.189229 tst ping_idle_ping.py: using LAC 39399
07:37:42.319832 tst ping_idle_ping.py: using RAC 129
07:37:42.448872 tst ping_idle_ping.py: using CellId 39399
07:37:42.576152 tst ping_idle_ping.py: using BVCI 39400
07:37:42.601245 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:42.612794 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:42.623879 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:42.634950 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:42.646478 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:42.657954 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:42.669796 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
07:37:42.681302 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
07:37:42.962926 tst ping_idle_ping.py:24: start network...
07:37:43.090200 run osmo-hlr_10.42.42.2: Starting osmo-hlr
07:37:43.582816 run create_hlr_db(pid=23373): Launched
07:37:43.759544 bus /gobi_6: Setting Powered False
07:37:44.779294 run create_hlr_db(pid=23373): Terminated: ok {rc=0}
07:37:45.217032 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23375): Launched
07:37:45.544558 run osmo-hlr_10.42.42.2(pid=23376): Launched
07:37:45.665470 run osmo-stp_10.42.42.5: Starting osmo-stp
07:37:46.284156 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23378): Launched
07:37:46.802765 run osmo-stp_10.42.42.5(pid=23379): Launched
07:37:46.949259 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
07:37:47.528651 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23381): Launched
07:37:47.647261 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
07:37:47.936325 run patchelf(pid=23382): Launched
07:37:48.129436 run osmo-stp_10.42.42.5(pid=23379): ERR: Terminated: ERROR {rc=1} [trial-2401↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23379)]
07:37:48.270810 run osmo-stp_10.42.42.5(pid=23379): stdout:
| (launched: 2022-12-14_07:37:46.452801)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:37:48.412332 run osmo-stp_10.42.42.5(pid=23379): stderr:
| �[0;m�[38;5;43m20221214073746677 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073746677 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073746678 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073746678 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073746678 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073746679 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073746679 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:37:48.548107 run osmo-stp_10.42.42.5(pid=23379): stdout:
| (launched: 2022-12-14_07:37:46.452801)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
07:37:48.682931 run osmo-stp_10.42.42.5(pid=23379): stderr:
| �[0;m�[38;5;43m20221214073746677 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214073746677 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214073746678 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214073746678 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214073746678 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214073746679 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214073746679 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
07:37:48.840577 run patchelf(pid=23382): Terminating (SIGINT)
07:37:49.026181 run patchelf(pid=23382): Terminated: ok {rc=0}
07:37:49.059759 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=23379): Process ended prematurely: osmo-stp_10.42.42.5(pid=23379) [trial-2401↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23379)] [trial-2401↪gprs:trx-umtrx↪ping_idle_ping.py:27]
07:37:49.069161 tst ping_idle_ping.py:27: Test FAILED (7.0 sec)
07:37:49.133034 run osmo-hlr_10.42.42.2(pid=23376): ERR: Terminated: ERROR {rc=237} [trial-2401↪gprs:trx-umtrx↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=23376)]
07:37:49.184823 run osmo-hlr_10.42.42.2(pid=23376): stdout:
| (launched: 2022-12-14_07:37:45.383295)
07:37:49.237306 run osmo-hlr_10.42.42.2(pid=23376): stderr:
| �[0;m�[1;31m20221214073745577 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214073745577 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214073745577 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214073745577 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214073745577 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214073745577 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214073745590 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2401/run.2022-12-14_06-57-06/gprs:trx-umtrx/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214073745600 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214073745601 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214073745601 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
07:37:49.282358 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23375): Terminating (SIGTERM)
07:37:49.324874 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23378): Terminating (SIGTERM)
07:37:49.367450 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23381): Terminating (SIGTERM)
07:37:49.377424 --- ParallelTerminationStrategy: PID 23375 died...
07:37:49.434205 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23375): Terminated: ok {rc=0}
07:37:49.444440 --- ParallelTerminationStrategy: PID 23378 died...
07:37:49.500734 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23378): Terminated: ok {rc=0}
07:37:49.510993 --- ParallelTerminationStrategy: PID 23381 died...
07:37:49.567448 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23381): Terminated: ok {rc=0}
07:37:49.655120 bus /gobi_6: Setting Powered False
5gprs:trx-umtrx00