70gprs:trx-umtrx9.0osmo-stp_10.42.42.5(pid=30336): Process ended prematurely: osmo-stp_10.42.42.5(pid=30336) [trial-2387↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30336)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=30336): Process ended prematurely: osmo-stp_10.42.42.5(pid=30336) [trial-2387↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30336)]
----------------------------------------------
trial-2387 gprs:trx-umtrx cs_paging_gprs_active.py
----------------------------------------------
09:12:07.577440 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:07.588246 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:12:07.743364 tst cs_paging_gprs_active.py: using LAC 32941
09:12:07.866379 tst cs_paging_gprs_active.py: using RAC 46
09:12:07.981724 tst cs_paging_gprs_active.py: using CellId 32941
09:12:08.096759 tst cs_paging_gprs_active.py: using BVCI 32942
09:12:08.119696 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:08.130430 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:08.140887 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:08.151393 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:08.162077 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:08.172719 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:08.183922 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:08.194563 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:08.205406 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:12:08.325479 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:12:08.605821 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
09:12:08.730892 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:12:09.039834 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30326): Launched
09:12:09.296050 run iperf3-srv_10.42.42.10(pid=30327): Launched
09:12:09.478595 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
09:12:09.603498 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:12:09.911134 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30328): Launched
09:12:10.167087 run iperf3-srv_10.42.42.10(pid=30329): Launched
09:12:10.349043 tst cs_paging_gprs_active.py:38: start network...
09:12:10.473333 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:12:10.942837 run create_hlr_db(pid=30330): Launched
09:12:11.114974 bus /sierra_2: Setting Powered False
09:12:12.137277 run create_hlr_db(pid=30330): Terminated: ok {rc=0}
09:12:12.562062 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30332): Launched
09:12:12.882532 run osmo-hlr_10.42.42.2(pid=30333): Launched
09:12:12.999800 run osmo-stp_10.42.42.5: Starting osmo-stp
09:12:13.554542 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30335): Launched
09:12:13.875133 run osmo-stp_10.42.42.5(pid=30336): Launched
09:12:13.991806 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:12:14.549417 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30338): Launched
09:12:14.663474 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:12:14.941598 run patchelf(pid=30339): Launched
09:12:15.128273 run osmo-stp_10.42.42.5(pid=30336): ERR: Terminated: ERROR {rc=1} [trial-2387↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30336)]
09:12:15.265317 run osmo-stp_10.42.42.5(pid=30336): stdout:
| (launched: 2022-12-13_09:12:13.717834)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:12:15.403210 run osmo-stp_10.42.42.5(pid=30336): stderr:
| �[0;m�[38;5;43m20221213091213909 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091213909 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091213909 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091213910 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091213912 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091213912 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091213912 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:12:15.535421 run osmo-stp_10.42.42.5(pid=30336): stdout:
| (launched: 2022-12-13_09:12:13.717834)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:12:15.666711 run osmo-stp_10.42.42.5(pid=30336): stderr:
| �[0;m�[38;5;43m20221213091213909 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091213909 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091213909 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091213910 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091213912 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091213912 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091213912 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:12:15.819219 run patchelf(pid=30339): Terminating (SIGINT)
09:12:16.016015 run patchelf(pid=30339): Terminated: ok {rc=0}
09:12:16.049598 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=30336): Process ended prematurely: osmo-stp_10.42.42.5(pid=30336) [trial-2387↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30336)] [trial-2387↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38]
09:12:16.059853 tst cs_paging_gprs_active.py:38: Test FAILED (8.5 sec)
09:12:16.124249 run osmo-hlr_10.42.42.2(pid=30333): ERR: Terminated: ERROR {rc=237} [trial-2387↪gprs:trx-umtrx↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=30333)]
09:12:16.176156 run osmo-hlr_10.42.42.2(pid=30333): stdout:
| (launched: 2022-12-13_09:12:12.725010)
09:12:16.230608 run osmo-hlr_10.42.42.2(pid=30333): stderr:
| �[0;m�[1;31m20221213091212925 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213091212925 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213091212925 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213091212925 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213091212925 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213091212925 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213091212947 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2387/run.2022-12-13_08-32-52/gprs:trx-umtrx/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213091212957 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213091212957 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213091212957 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:12:16.276007 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30326): Terminating (SIGTERM)
09:12:16.318542 run iperf3-srv_10.42.42.10(pid=30327): Terminating (SIGTERM)
09:12:16.361299 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30328): Terminating (SIGTERM)
09:12:16.403406 run iperf3-srv_10.42.42.10(pid=30329): Terminating (SIGTERM)
09:12:16.449209 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30332): Terminating (SIGTERM)
09:12:16.493678 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30335): Terminating (SIGTERM)
09:12:16.537653 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30338): Terminating (SIGTERM)
09:12:16.549648 --- ParallelTerminationStrategy: PID 30326 died...
09:12:16.608833 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30326): Terminated: ok {rc=0}
09:12:16.619619 --- ParallelTerminationStrategy: PID 30327 died...
09:12:16.677732 run iperf3-srv_10.42.42.10(pid=30327): Terminated {rc=256}
09:12:16.688750 --- ParallelTerminationStrategy: PID 30328 died...
09:12:16.745461 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30328): Terminated: ok {rc=0}
09:12:16.755995 --- ParallelTerminationStrategy: PID 30329 died...
09:12:16.812935 run iperf3-srv_10.42.42.10(pid=30329): Terminated {rc=256}
09:12:16.824498 --- ParallelTerminationStrategy: PID 30332 died...
09:12:16.881884 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30332): Terminated: ok {rc=0}
09:12:16.892472 --- ParallelTerminationStrategy: PID 30335 died...
09:12:16.949689 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30335): Terminated: ok {rc=0}
09:12:16.960275 --- ParallelTerminationStrategy: PID 30338 died...
09:12:17.017332 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30338): Terminated: ok {rc=0}
09:12:17.106125 bus /gobi_0: Setting Powered False
09:12:18.207189 bus /gobi_0: Setting Powered False
09:12:19.318670 bus /gobi_6: Setting Powered False
70gprs:trx-umtrx8.0osmo-stp_10.42.42.5(pid=30348): Process ended prematurely: osmo-stp_10.42.42.5(pid=30348) [trial-2387↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30348)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=30348): Process ended prematurely: osmo-stp_10.42.42.5(pid=30348) [trial-2387↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30348)]
----------------------------------------------
trial-2387 gprs:trx-umtrx iperf3.py
----------------------------------------------
09:12:20.428662 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:20.439845 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:12:20.583963 tst iperf3.py: using LAC 32942
09:12:20.710576 tst iperf3.py: using RAC 47
09:12:20.836471 tst iperf3.py: using CellId 32942
09:12:20.962697 tst iperf3.py: using BVCI 32943
09:12:20.987421 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:20.998785 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:21.009711 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:21.020602 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:21.031487 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:21.042274 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:21.053239 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:21.063984 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:21.074948 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:12:21.351290 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
09:12:21.475720 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:12:21.783947 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30340): Launched
09:12:22.041413 run iperf3-srv_10.42.42.10(pid=30341): Launched
09:12:22.223727 tst iperf3.py:8: start network...
09:12:22.348359 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:12:22.816850 run create_hlr_db(pid=30342): Launched
09:12:22.988201 bus /gobi_6: Setting Powered False
09:12:24.009854 run create_hlr_db(pid=30342): Terminated: ok {rc=0}
09:12:24.432382 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30344): Launched
09:12:24.752868 run osmo-hlr_10.42.42.2(pid=30345): Launched
09:12:24.869981 run osmo-stp_10.42.42.5: Starting osmo-stp
09:12:25.425207 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30347): Launched
09:12:25.744185 run osmo-stp_10.42.42.5(pid=30348): Launched
09:12:25.859810 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:12:26.415033 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30350): Launched
09:12:26.528843 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:12:26.805236 run patchelf(pid=30351): Launched
09:12:26.991517 run osmo-stp_10.42.42.5(pid=30348): ERR: Terminated: ERROR {rc=1} [trial-2387↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30348)]
09:12:27.128728 run osmo-stp_10.42.42.5(pid=30348): stdout:
| (launched: 2022-12-13_09:12:25.587106)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:12:27.265713 run osmo-stp_10.42.42.5(pid=30348): stderr:
| �[0;m�[38;5;43m20221213091225777 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091225777 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091225778 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091225778 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091225778 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091225779 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091225779 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:12:27.398480 run osmo-stp_10.42.42.5(pid=30348): stdout:
| (launched: 2022-12-13_09:12:25.587106)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:12:27.529783 run osmo-stp_10.42.42.5(pid=30348): stderr:
| �[0;m�[38;5;43m20221213091225777 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091225777 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091225778 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091225778 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091225778 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091225779 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091225779 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:12:27.681807 run patchelf(pid=30351): Terminating (SIGINT)
09:12:27.860768 run patchelf(pid=30351): Terminated: ok {rc=0}
09:12:27.893590 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=30348): Process ended prematurely: osmo-stp_10.42.42.5(pid=30348) [trial-2387↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30348)] [trial-2387↪gprs:trx-umtrx↪iperf3.py:8]
09:12:27.903346 tst iperf3.py:8: Test FAILED (7.5 sec)
09:12:27.965529 run osmo-hlr_10.42.42.2(pid=30345): ERR: Terminated: ERROR {rc=237} [trial-2387↪gprs:trx-umtrx↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=30345)]
09:12:28.016388 run osmo-hlr_10.42.42.2(pid=30345): stdout:
| (launched: 2022-12-13_09:12:24.595181)
09:12:28.067845 run osmo-hlr_10.42.42.2(pid=30345): stderr:
| �[0;m�[1;31m20221213091224792 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213091224792 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213091224792 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213091224792 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213091224792 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213091224792 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213091224820 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2387/run.2022-12-13_08-32-52/gprs:trx-umtrx/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213091224831 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213091224831 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213091224831 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:12:28.111886 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30340): Terminating (SIGTERM)
09:12:28.153905 run iperf3-srv_10.42.42.10(pid=30341): Terminating (SIGTERM)
09:12:28.195390 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30344): Terminating (SIGTERM)
09:12:28.236775 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30347): Terminating (SIGTERM)
09:12:28.278161 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30350): Terminating (SIGTERM)
09:12:28.287892 --- ParallelTerminationStrategy: PID 30340 died...
09:12:28.343683 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30340): Terminated: ok {rc=0}
09:12:28.353762 --- ParallelTerminationStrategy: PID 30341 died...
09:12:28.408879 run iperf3-srv_10.42.42.10(pid=30341): Terminated {rc=256}
09:12:28.418879 --- ParallelTerminationStrategy: PID 30344 died...
09:12:28.474060 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30344): Terminated: ok {rc=0}
09:12:28.484041 --- ParallelTerminationStrategy: PID 30347 died...
09:12:28.539193 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30347): Terminated: ok {rc=0}
09:12:28.549352 --- ParallelTerminationStrategy: PID 30350 died...
09:12:28.604332 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30350): Terminated: ok {rc=0}
09:12:28.690401 bus /gobi_6: Setting Powered False
213gprs:trx-umtrx11.0osmo-stp_10.42.42.5(pid=30366): Process ended prematurely: osmo-stp_10.42.42.5(pid=30366) [trial-2387↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30366)]2175iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=30366): Process ended prematurely: osmo-stp_10.42.42.5(pid=30366) [trial-2387↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30366)]
----------------------------------------------
trial-2387 gprs:trx-umtrx iperf3m4.py
----------------------------------------------
09:12:29.803041 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:29.818146 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:12:29.957515 tst iperf3m4.py: using LAC 32943
09:12:30.084192 tst iperf3m4.py: using RAC 48
09:12:30.209496 tst iperf3m4.py: using CellId 32943
09:12:30.335556 tst iperf3m4.py: using BVCI 32944
09:12:30.360116 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:30.371656 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:30.382629 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:30.393422 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:30.404203 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:30.415160 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:30.426383 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:30.437319 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:30.448111 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:12:30.567107 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:12:30.691591 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:12:30.807103 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:12:31.138530 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
09:12:31.276189 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:12:31.626982 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30352): Launched
09:12:31.923636 run iperf3-srv_10.42.42.10(pid=30353): Launched
09:12:32.124658 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
09:12:32.259639 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:12:32.591204 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30354): Launched
09:12:32.866002 run iperf3-srv_10.42.42.10(pid=30355): Launched
09:12:33.063251 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
09:12:33.197823 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:12:33.523873 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30356): Launched
09:12:33.781059 run iperf3-srv_10.42.42.10(pid=30357): Launched
09:12:33.963095 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
09:12:34.087594 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:12:34.396072 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30358): Launched
09:12:34.652022 run iperf3-srv_10.42.42.10(pid=30359): Launched
09:12:34.833648 tst iperf3m4.py:8: start network...
09:12:34.957928 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:12:35.427446 run create_hlr_db(pid=30360): Launched
09:12:35.598765 bus /gobi_6: Setting Powered False
09:12:36.621103 run create_hlr_db(pid=30360): Terminated: ok {rc=0}
09:12:37.017443 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30362): Launched
09:12:37.317020 run osmo-hlr_10.42.42.2(pid=30363): Launched
09:12:37.431674 run osmo-stp_10.42.42.5: Starting osmo-stp
09:12:37.949934 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30365): Launched
09:12:38.249987 run osmo-stp_10.42.42.5(pid=30366): Launched
09:12:38.360552 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:12:38.881079 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30368): Launched
09:12:38.987447 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:12:39.247619 run patchelf(pid=30369): Launched
09:12:39.422807 run osmo-stp_10.42.42.5(pid=30366): ERR: Terminated: ERROR {rc=1} [trial-2387↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30366)]
09:12:39.551155 run osmo-stp_10.42.42.5(pid=30366): stdout:
| (launched: 2022-12-13_09:12:38.102299)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:12:39.679469 run osmo-stp_10.42.42.5(pid=30366): stderr:
| �[0;m�[38;5;43m20221213091238274 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091238274 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091238274 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091238274 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091238275 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091238275 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091238275 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:12:39.803227 run osmo-stp_10.42.42.5(pid=30366): stdout:
| (launched: 2022-12-13_09:12:38.102299)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:12:39.925849 run osmo-stp_10.42.42.5(pid=30366): stderr:
| �[0;m�[38;5;43m20221213091238274 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091238274 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091238274 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091238274 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091238275 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091238275 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091238275 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:12:40.069358 run patchelf(pid=30369): Terminating (SIGINT)
09:12:40.237282 run patchelf(pid=30369): Terminated: ok {rc=0}
09:12:40.268062 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=30366): Process ended prematurely: osmo-stp_10.42.42.5(pid=30366) [trial-2387↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30366)] [trial-2387↪gprs:trx-umtrx↪iperf3m4.py:8]
09:12:40.277273 tst iperf3m4.py:8: Test FAILED (10.5 sec)
09:12:40.337696 run osmo-hlr_10.42.42.2(pid=30363): ERR: Terminated: ERROR {rc=237} [trial-2387↪gprs:trx-umtrx↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=30363)]
09:12:40.386942 run osmo-hlr_10.42.42.2(pid=30363): stdout:
| (launched: 2022-12-13_09:12:37.169659)
09:12:40.436699 run osmo-hlr_10.42.42.2(pid=30363): stderr:
| �[0;m�[1;31m20221213091237348 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213091237348 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213091237348 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213091237348 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213091237348 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213091237348 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213091237361 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2387/run.2022-12-13_08-32-52/gprs:trx-umtrx/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213091237369 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213091237369 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213091237369 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:12:40.479143 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30352): Terminating (SIGTERM)
09:12:40.518891 run iperf3-srv_10.42.42.10(pid=30353): Terminating (SIGTERM)
09:12:40.558772 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30354): Terminating (SIGTERM)
09:12:40.598593 run iperf3-srv_10.42.42.10(pid=30355): Terminating (SIGTERM)
09:12:40.638300 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30356): Terminating (SIGTERM)
09:12:40.677900 run iperf3-srv_10.42.42.10(pid=30357): Terminating (SIGTERM)
09:12:40.717961 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30358): Terminating (SIGTERM)
09:12:40.757511 run iperf3-srv_10.42.42.10(pid=30359): Terminating (SIGTERM)
09:12:40.797239 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30362): Terminating (SIGTERM)
09:12:40.836678 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30365): Terminating (SIGTERM)
09:12:40.876125 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30368): Terminating (SIGTERM)
09:12:40.885269 --- ParallelTerminationStrategy: PID 30352 died...
09:12:40.938491 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30352): Terminated: ok {rc=0}
09:12:40.947903 --- ParallelTerminationStrategy: PID 30353 died...
09:12:41.000705 run iperf3-srv_10.42.42.10(pid=30353): Terminated {rc=256}
09:12:41.010073 --- ParallelTerminationStrategy: PID 30354 died...
09:12:41.062952 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30354): Terminated: ok {rc=0}
09:12:41.072337 --- ParallelTerminationStrategy: PID 30355 died...
09:12:41.125085 run iperf3-srv_10.42.42.10(pid=30355): Terminated {rc=256}
09:12:41.134344 --- ParallelTerminationStrategy: PID 30356 died...
09:12:41.186973 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30356): Terminated: ok {rc=0}
09:12:41.196350 --- ParallelTerminationStrategy: PID 30357 died...
09:12:41.248899 run iperf3-srv_10.42.42.10(pid=30357): Terminated {rc=256}
09:12:41.258332 --- ParallelTerminationStrategy: PID 30358 died...
09:12:41.310845 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=30358): Terminated: ok {rc=0}
09:12:41.320260 --- ParallelTerminationStrategy: PID 30359 died...
09:12:41.373121 run iperf3-srv_10.42.42.10(pid=30359): Terminated {rc=256}
09:12:41.382541 --- ParallelTerminationStrategy: PID 30362 died...
09:12:41.435060 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30362): Terminated: ok {rc=0}
09:12:41.444533 --- ParallelTerminationStrategy: PID 30365 died...
09:12:41.497418 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30365): Terminated: ok {rc=0}
09:12:41.506845 --- ParallelTerminationStrategy: PID 30368 died...
09:12:41.559955 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30368): Terminated: ok {rc=0}
09:12:41.643068 bus /gobi_4: Setting Powered False
09:12:42.749376 bus /gobi_4: Setting Powered False
09:12:43.856062 bus /gobi_1: Setting Powered False
09:12:44.961596 bus /gobi_1: Setting Powered False
09:12:46.071193 bus /gobi_0: Setting Powered False
09:12:47.178606 bus /gobi_0: Setting Powered False
09:12:48.283947 bus /gobi_6: Setting Powered False
188gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=30376): Process ended prematurely: osmo-stp_10.42.42.5(pid=30376) [trial-2387↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30376)]2200ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=30376): Process ended prematurely: osmo-stp_10.42.42.5(pid=30376) [trial-2387↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30376)]
----------------------------------------------
trial-2387 gprs:trx-umtrx ping.py
----------------------------------------------
09:12:49.352488 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:49.364145 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:12:49.505230 tst ping.py: using LAC 32944
09:12:49.628319 tst ping.py: using RAC 49
09:12:49.746503 tst ping.py: using CellId 32944
09:12:49.861934 tst ping.py: using BVCI 32945
09:12:49.884744 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:49.895426 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:49.906038 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:49.916503 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:49.927267 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:49.938220 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:49.949333 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:49.960264 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:12:50.247893 tst ping.py:19: start network...
09:12:50.379722 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:12:50.879056 run create_hlr_db(pid=30370): Launched
09:12:51.060778 bus /gobi_6: Setting Powered False
09:12:52.086954 run create_hlr_db(pid=30370): Terminated: ok {rc=0}
09:12:52.502907 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30372): Launched
09:12:52.815860 run osmo-hlr_10.42.42.2(pid=30373): Launched
09:12:52.931021 run osmo-stp_10.42.42.5: Starting osmo-stp
09:12:53.477968 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30375): Launched
09:12:53.790974 run osmo-stp_10.42.42.5(pid=30376): Launched
09:12:53.904622 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:12:54.450531 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30378): Launched
09:12:54.562767 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:12:54.835862 run patchelf(pid=30379): Launched
09:12:55.019766 run osmo-stp_10.42.42.5(pid=30376): ERR: Terminated: ERROR {rc=1} [trial-2387↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30376)]
09:12:55.154317 run osmo-stp_10.42.42.5(pid=30376): stdout:
| (launched: 2022-12-13_09:12:53.637419)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:12:55.289314 run osmo-stp_10.42.42.5(pid=30376): stderr:
| �[0;m�[38;5;43m20221213091253826 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091253826 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091253826 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091253827 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091253828 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091253828 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091253828 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:12:55.420308 run osmo-stp_10.42.42.5(pid=30376): stdout:
| (launched: 2022-12-13_09:12:53.637419)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:12:55.549643 run osmo-stp_10.42.42.5(pid=30376): stderr:
| �[0;m�[38;5;43m20221213091253826 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091253826 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091253826 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091253827 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091253828 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091253828 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091253828 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:12:55.699205 run patchelf(pid=30379): Terminating (SIGINT)
09:12:55.875253 run patchelf(pid=30379): Terminated: ok {rc=0}
09:12:55.907233 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=30376): Process ended prematurely: osmo-stp_10.42.42.5(pid=30376) [trial-2387↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30376)] [trial-2387↪gprs:trx-umtrx↪ping.py:22]
09:12:55.916251 tst ping.py:22: Test FAILED (6.6 sec)
09:12:55.978302 run osmo-hlr_10.42.42.2(pid=30373): ERR: Terminated: ERROR {rc=237} [trial-2387↪gprs:trx-umtrx↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=30373)]
09:12:56.029192 run osmo-hlr_10.42.42.2(pid=30373): stdout:
| (launched: 2022-12-13_09:12:52.662008)
09:12:56.080544 run osmo-hlr_10.42.42.2(pid=30373): stderr:
| �[0;m�[1;31m20221213091252861 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213091252861 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213091252861 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213091252861 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213091252861 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213091252861 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213091252874 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2387/run.2022-12-13_08-32-52/gprs:trx-umtrx/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213091252882 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213091252883 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213091252883 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:12:56.124013 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30372): Terminating (SIGTERM)
09:12:56.165154 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30375): Terminating (SIGTERM)
09:12:56.206238 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30378): Terminating (SIGTERM)
09:12:56.215967 --- ParallelTerminationStrategy: PID 30372 died...
09:12:56.271172 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30372): Terminated: ok {rc=0}
09:12:56.281431 --- ParallelTerminationStrategy: PID 30375 died...
09:12:56.336751 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30375): Terminated: ok {rc=0}
09:12:56.346877 --- ParallelTerminationStrategy: PID 30378 died...
09:12:56.401980 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30378): Terminated: ok {rc=0}
09:12:56.487644 bus /gobi_6: Setting Powered False
249gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=30386): Process ended prematurely: osmo-stp_10.42.42.5(pid=30386) [trial-2387↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30386)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=30386): Process ended prematurely: osmo-stp_10.42.42.5(pid=30386) [trial-2387↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30386)]
----------------------------------------------
trial-2387 gprs:trx-umtrx ping_idle_ping.py
----------------------------------------------
09:12:57.538843 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:57.550492 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:12:57.692653 tst ping_idle_ping.py: using LAC 32945
09:12:57.817858 tst ping_idle_ping.py: using RAC 50
09:12:57.943679 tst ping_idle_ping.py: using CellId 32945
09:12:58.073984 tst ping_idle_ping.py: using BVCI 32946
09:12:58.098207 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:58.109440 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:58.120307 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:58.131205 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:58.142155 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:58.152992 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:58.163930 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:12:58.174988 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:12:58.446096 tst ping_idle_ping.py:24: start network...
09:12:58.568752 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:12:59.033317 run create_hlr_db(pid=30380): Launched
09:12:59.203211 bus /gobi_6: Setting Powered False
09:13:00.223863 run create_hlr_db(pid=30380): Terminated: ok {rc=0}
09:13:00.642216 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30382): Launched
09:13:01.058823 run osmo-hlr_10.42.42.2(pid=30383): Launched
09:13:01.188767 run osmo-stp_10.42.42.5: Starting osmo-stp
09:13:01.874452 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30385): Launched
09:13:02.196928 run osmo-stp_10.42.42.5(pid=30386): Launched
09:13:02.404868 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:13:02.939283 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30388): Launched
09:13:03.044278 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:13:03.300322 run patchelf(pid=30389): Launched
09:13:03.473135 run osmo-stp_10.42.42.5(pid=30386): ERR: Terminated: ERROR {rc=1} [trial-2387↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30386)]
09:13:03.599571 run osmo-stp_10.42.42.5(pid=30386): stdout:
| (launched: 2022-12-13_09:13:02.037500)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:13:03.726462 run osmo-stp_10.42.42.5(pid=30386): stderr:
| �[0;m�[38;5;43m20221213091302376 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091302376 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091302376 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091302376 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091302377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091302377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091302377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:13:03.848913 run osmo-stp_10.42.42.5(pid=30386): stdout:
| (launched: 2022-12-13_09:13:02.037500)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:13:03.970676 run osmo-stp_10.42.42.5(pid=30386): stderr:
| �[0;m�[38;5;43m20221213091302376 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213091302376 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213091302376 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213091302376 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213091302377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213091302377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213091302377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:13:04.111794 run patchelf(pid=30389): Terminating (SIGINT)
09:13:04.276738 run patchelf(pid=30389): Terminated: ok {rc=0}
09:13:04.306899 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=30386): Process ended prematurely: osmo-stp_10.42.42.5(pid=30386) [trial-2387↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=30386)] [trial-2387↪gprs:trx-umtrx↪ping_idle_ping.py:27]
09:13:04.315554 tst ping_idle_ping.py:27: Test FAILED (6.8 sec)
09:13:04.375852 run osmo-hlr_10.42.42.2(pid=30383): ERR: Terminated: ERROR {rc=237} [trial-2387↪gprs:trx-umtrx↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=30383)]
09:13:04.425190 run osmo-hlr_10.42.42.2(pid=30383): stdout:
| (launched: 2022-12-13_09:13:00.802661)
09:13:04.474921 run osmo-hlr_10.42.42.2(pid=30383): stderr:
| �[0;m�[1;31m20221213091301099 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213091301099 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213091301099 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213091301099 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213091301099 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213091301099 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213091301124 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2387/run.2022-12-13_08-32-52/gprs:trx-umtrx/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213091301137 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213091301137 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213091301137 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:13:04.517507 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30382): Terminating (SIGTERM)
09:13:04.557627 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30385): Terminating (SIGTERM)
09:13:04.597880 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30388): Terminating (SIGTERM)
09:13:04.607281 --- ParallelTerminationStrategy: PID 30382 died...
09:13:04.661633 run pcap-recorder_any(filters='host 10.42.42.2')(pid=30382): Terminated: ok {rc=0}
09:13:04.671288 --- ParallelTerminationStrategy: PID 30385 died...
09:13:04.725028 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=30385): Terminated: ok {rc=0}
09:13:04.734748 --- ParallelTerminationStrategy: PID 30388 died...
09:13:04.788517 run pcap-recorder_any(filters='host 10.42.42.6')(pid=30388): Terminated: ok {rc=0}
09:13:04.876127 bus /gobi_6: Setting Powered False
5gprs:trx-umtrx00