55gprs:sysmo11.0osmo-stp_10.42.42.5(pid=12508): Process ended prematurely: osmo-stp_10.42.42.5(pid=12508) [trial-2368↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12508)]2314cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=12508): Process ended prematurely: osmo-stp_10.42.42.5(pid=12508) [trial-2368↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12508)]
----------------------------------------------
trial-2368 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
02:13:00.873923 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:00.885838 tst gprs:sysmo: Using 1 x bts (candidates: 1)
02:13:01.026650 tst cs_paging_gprs_active.py: using LAC 25272
02:13:01.170375 tst cs_paging_gprs_active.py: using RAC 27
02:13:01.299619 tst cs_paging_gprs_active.py: using CellId 25272
02:13:01.454836 tst cs_paging_gprs_active.py: using BVCI 25273
02:13:01.480128 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:01.491926 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:01.504207 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:01.516793 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:01.529742 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:01.541370 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:01.553967 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:01.566864 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:01.581056 tst gprs:sysmo: Using 1 x modem (candidates: 4)
02:13:01.706430 tst gprs:sysmo: Using 1 x modem (candidates: 4)
02:13:02.017706 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
02:13:02.177703 run iperf3-srv_10.42.42.10: Starting iperf3-srv
02:13:02.532851 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12498): Launched
02:13:02.824114 run iperf3-srv_10.42.42.10(pid=12499): Launched
02:13:03.030027 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
02:13:03.170755 run iperf3-srv_10.42.42.10: Starting iperf3-srv
02:13:03.519371 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12500): Launched
02:13:03.810033 run iperf3-srv_10.42.42.10(pid=12501): Launched
02:13:04.016229 tst cs_paging_gprs_active.py:38: start network...
02:13:04.157506 run osmo-hlr_10.42.42.2: Starting osmo-hlr
02:13:04.702102 run create_hlr_db(pid=12502): Launched
02:13:04.896048 bus /sierra_2: Setting Powered False
02:13:05.917219 run create_hlr_db(pid=12502): Terminated: ok {rc=0}
02:13:06.366312 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12504): Launched
02:13:06.705130 run osmo-hlr_10.42.42.2(pid=12505): Launched
02:13:06.827702 run osmo-stp_10.42.42.5: Starting osmo-stp
02:13:07.412497 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12507): Launched
02:13:07.751162 run osmo-stp_10.42.42.5(pid=12508): Launched
02:13:07.873719 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
02:13:09.471180 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12510): Launched
02:13:09.590934 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
02:13:09.886203 run patchelf(pid=12511): Launched
02:13:10.081813 run osmo-stp_10.42.42.5(pid=12508): ERR: Terminated: ERROR {rc=1} [trial-2368↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12508)]
02:13:10.225554 run osmo-stp_10.42.42.5(pid=12508): stdout:
| (launched: 2022-12-10_02:13:07.584014)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:10.368627 run osmo-stp_10.42.42.5(pid=12508): stderr:
| �[0;m�[38;5;43m20221210021307779 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021307779 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021307779 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021307779 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021307783 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021307783 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021307783 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:13:10.506991 run osmo-stp_10.42.42.5(pid=12508): stdout:
| (launched: 2022-12-10_02:13:07.584014)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:10.643964 run osmo-stp_10.42.42.5(pid=12508): stderr:
| �[0;m�[38;5;43m20221210021307779 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021307779 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021307779 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021307779 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021307783 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021307783 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021307783 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:13:10.803756 run patchelf(pid=12511): Terminating (SIGINT)
02:13:10.991200 run patchelf(pid=12511): Terminated: ok {rc=0}
02:13:11.025381 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=12508): Process ended prematurely: osmo-stp_10.42.42.5(pid=12508) [trial-2368↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12508)] [trial-2368↪gprs:sysmo↪cs_paging_gprs_active.py:38]
02:13:11.035156 tst cs_paging_gprs_active.py:38: Test FAILED (10.2 sec)
02:13:11.098468 run osmo-hlr_10.42.42.2(pid=12505): ERR: Terminated: ERROR {rc=237} [trial-2368↪gprs:sysmo↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=12505)]
02:13:11.150083 run osmo-hlr_10.42.42.2(pid=12505): stdout:
| (launched: 2022-12-10_02:13:06.537833)
02:13:11.202653 run osmo-hlr_10.42.42.2(pid=12505): stderr:
| �[0;m�[1;31m20221210021306739 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210021306739 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210021306739 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210021306739 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210021306739 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210021306739 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210021306764 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2368/run.2022-12-10_02-07-09/gprs:sysmo/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210021306773 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210021306773 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210021306773 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
02:13:11.247291 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12498): Terminating (SIGTERM)
02:13:11.288941 run iperf3-srv_10.42.42.10(pid=12499): Terminating (SIGTERM)
02:13:11.330913 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12500): Terminating (SIGTERM)
02:13:11.372563 run iperf3-srv_10.42.42.10(pid=12501): Terminating (SIGTERM)
02:13:11.414549 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12504): Terminating (SIGTERM)
02:13:11.456426 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12507): Terminating (SIGTERM)
02:13:11.498362 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12510): Terminating (SIGTERM)
02:13:11.507970 --- ParallelTerminationStrategy: PID 12498 died...
02:13:11.564033 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12498): Terminated: ok {rc=0}
02:13:11.574110 --- ParallelTerminationStrategy: PID 12499 died...
02:13:11.630043 run iperf3-srv_10.42.42.10(pid=12499): Terminated {rc=256}
02:13:11.640018 --- ParallelTerminationStrategy: PID 12500 died...
02:13:11.696341 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12500): Terminated: ok {rc=0}
02:13:11.706487 --- ParallelTerminationStrategy: PID 12501 died...
02:13:11.763442 run iperf3-srv_10.42.42.10(pid=12501): Terminated {rc=256}
02:13:11.773851 --- ParallelTerminationStrategy: PID 12504 died...
02:13:11.829926 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12504): Terminated: ok {rc=0}
02:13:11.840032 --- ParallelTerminationStrategy: PID 12507 died...
02:13:11.896094 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12507): Terminated: ok {rc=0}
02:13:11.906222 --- ParallelTerminationStrategy: PID 12510 died...
02:13:11.962658 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12510): Terminated: ok {rc=0}
02:13:12.052100 bus /gobi_0: Setting Powered False
02:13:13.159089 bus /gobi_0: Setting Powered False
02:13:14.272734 bus /gobi_6: Setting Powered False
55gprs:sysmo8.0osmo-stp_10.42.42.5(pid=12520): Process ended prematurely: osmo-stp_10.42.42.5(pid=12520) [trial-2368↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12520)]2314iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=12520): Process ended prematurely: osmo-stp_10.42.42.5(pid=12520) [trial-2368↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12520)]
----------------------------------------------
trial-2368 gprs:sysmo iperf3.py
----------------------------------------------
02:13:15.390364 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:15.402142 tst gprs:sysmo: Using 1 x bts (candidates: 1)
02:13:15.537842 tst iperf3.py: using LAC 25273
02:13:15.665141 tst iperf3.py: using RAC 28
02:13:15.795429 tst iperf3.py: using CellId 25273
02:13:15.937505 tst iperf3.py: using BVCI 25274
02:13:15.964151 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:15.975983 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:15.987985 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:15.999758 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:16.011265 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:16.022969 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:16.034801 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:16.046502 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:16.058314 tst gprs:sysmo: Using 1 x modem (candidates: 4)
02:13:16.388097 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
02:13:16.531325 run iperf3-srv_10.42.42.10: Starting iperf3-srv
02:13:16.896764 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12512): Launched
02:13:17.195149 run iperf3-srv_10.42.42.10(pid=12513): Launched
02:13:17.401344 tst iperf3.py:8: start network...
02:13:17.541261 run osmo-hlr_10.42.42.2: Starting osmo-hlr
02:13:18.080343 run create_hlr_db(pid=12514): Launched
02:13:18.275699 bus /gobi_6: Setting Powered False
02:13:19.295186 run create_hlr_db(pid=12514): Terminated: ok {rc=0}
02:13:19.718271 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12516): Launched
02:13:20.034617 run osmo-hlr_10.42.42.2(pid=12517): Launched
02:13:20.158457 run osmo-stp_10.42.42.5: Starting osmo-stp
02:13:20.705721 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12519): Launched
02:13:21.020539 run osmo-stp_10.42.42.5(pid=12520): Launched
02:13:21.134373 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
02:13:21.681365 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12522): Launched
02:13:21.793157 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
02:13:22.068176 run patchelf(pid=12523): Launched
02:13:22.251898 run osmo-stp_10.42.42.5(pid=12520): ERR: Terminated: ERROR {rc=1} [trial-2368↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12520)]
02:13:22.386232 run osmo-stp_10.42.42.5(pid=12520): stdout:
| (launched: 2022-12-10_02:13:20.864474)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:22.520228 run osmo-stp_10.42.42.5(pid=12520): stderr:
| �[0;m�[38;5;43m20221210021321038 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021321038 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021321039 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021321039 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021321041 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021321042 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021321042 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:13:22.650119 run osmo-stp_10.42.42.5(pid=12520): stdout:
| (launched: 2022-12-10_02:13:20.864474)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:22.778936 run osmo-stp_10.42.42.5(pid=12520): stderr:
| �[0;m�[38;5;43m20221210021321038 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021321038 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021321039 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021321039 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021321041 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021321042 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021321042 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:13:22.928878 run patchelf(pid=12523): Terminating (SIGINT)
02:13:23.104384 run patchelf(pid=12523): Terminated: ok {rc=0}
02:13:23.136582 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=12520): Process ended prematurely: osmo-stp_10.42.42.5(pid=12520) [trial-2368↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12520)] [trial-2368↪gprs:sysmo↪iperf3.py:8]
02:13:23.145817 tst iperf3.py:8: Test FAILED (7.8 sec)
02:13:23.207058 run osmo-hlr_10.42.42.2(pid=12517): ERR: Terminated: ERROR {rc=237} [trial-2368↪gprs:sysmo↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=12517)]
02:13:23.256783 run osmo-hlr_10.42.42.2(pid=12517): stdout:
| (launched: 2022-12-10_02:13:19.878071)
02:13:23.308002 run osmo-hlr_10.42.42.2(pid=12517): stderr:
| �[0;m�[1;31m20221210021320061 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210021320061 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210021320061 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210021320061 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210021320061 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210021320061 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210021320074 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2368/run.2022-12-10_02-07-09/gprs:sysmo/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210021320083 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210021320083 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210021320083 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
02:13:23.350745 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12512): Terminating (SIGTERM)
02:13:23.391098 run iperf3-srv_10.42.42.10(pid=12513): Terminating (SIGTERM)
02:13:23.431943 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12516): Terminating (SIGTERM)
02:13:23.472110 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12519): Terminating (SIGTERM)
02:13:23.512431 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12522): Terminating (SIGTERM)
02:13:23.522040 --- ParallelTerminationStrategy: PID 12512 died...
02:13:23.576181 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12512): Terminated: ok {rc=0}
02:13:23.586006 --- ParallelTerminationStrategy: PID 12513 died...
02:13:23.640058 run iperf3-srv_10.42.42.10(pid=12513): Terminated {rc=256}
02:13:23.649907 --- ParallelTerminationStrategy: PID 12516 died...
02:13:23.703503 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12516): Terminated: ok {rc=0}
02:13:23.713518 --- ParallelTerminationStrategy: PID 12519 died...
02:13:23.767688 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12519): Terminated: ok {rc=0}
02:13:23.777507 --- ParallelTerminationStrategy: PID 12522 died...
02:13:23.831457 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12522): Terminated: ok {rc=0}
02:13:23.919827 bus /gobi_6: Setting Powered False
56gprs:sysmo12.0osmo-stp_10.42.42.5(pid=12538): Process ended prematurely: osmo-stp_10.42.42.5(pid=12538) [trial-2368↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12538)]2313iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=12538): Process ended prematurely: osmo-stp_10.42.42.5(pid=12538) [trial-2368↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12538)]
----------------------------------------------
trial-2368 gprs:sysmo iperf3m4.py
----------------------------------------------
02:13:25.034741 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:25.046332 tst gprs:sysmo: Using 1 x bts (candidates: 1)
02:13:25.179583 tst iperf3m4.py: using LAC 25274
02:13:25.309092 tst iperf3m4.py: using RAC 29
02:13:25.439108 tst iperf3m4.py: using CellId 25274
02:13:25.568634 tst iperf3m4.py: using BVCI 25275
02:13:25.594564 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:25.605749 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:25.616753 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:25.627782 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:25.638754 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:25.649639 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:25.660737 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:25.671695 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:25.682974 tst gprs:sysmo: Using 1 x modem (candidates: 4)
02:13:25.801513 tst gprs:sysmo: Using 1 x modem (candidates: 4)
02:13:25.927081 tst gprs:sysmo: Using 1 x modem (candidates: 4)
02:13:26.045628 tst gprs:sysmo: Using 1 x modem (candidates: 4)
02:13:26.336373 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
02:13:26.467895 run iperf3-srv_10.42.42.10: Starting iperf3-srv
02:13:26.794478 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12524): Launched
02:13:27.067568 run iperf3-srv_10.42.42.10(pid=12525): Launched
02:13:27.260240 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
02:13:27.391664 run iperf3-srv_10.42.42.10: Starting iperf3-srv
02:13:27.718827 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12526): Launched
02:13:27.991214 run iperf3-srv_10.42.42.10(pid=12527): Launched
02:13:28.182999 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
02:13:28.313865 run iperf3-srv_10.42.42.10: Starting iperf3-srv
02:13:28.640376 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12528): Launched
02:13:28.912048 run iperf3-srv_10.42.42.10(pid=12529): Launched
02:13:29.102573 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
02:13:29.232654 run iperf3-srv_10.42.42.10: Starting iperf3-srv
02:13:29.558663 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12530): Launched
02:13:29.846183 run iperf3-srv_10.42.42.10(pid=12531): Launched
02:13:30.037285 tst iperf3m4.py:8: start network...
02:13:30.168276 run osmo-hlr_10.42.42.2: Starting osmo-hlr
02:13:30.665199 run create_hlr_db(pid=12532): Launched
02:13:30.855142 bus /gobi_6: Setting Powered False
02:13:31.885647 run create_hlr_db(pid=12532): Terminated: ok {rc=0}
02:13:32.363660 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12534): Launched
02:13:32.702356 run osmo-hlr_10.42.42.2(pid=12535): Launched
02:13:32.828020 run osmo-stp_10.42.42.5: Starting osmo-stp
02:13:33.418203 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12537): Launched
02:13:33.755550 run osmo-stp_10.42.42.5(pid=12538): Launched
02:13:33.879438 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
02:13:34.467170 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12540): Launched
02:13:34.586937 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
02:13:34.881060 run patchelf(pid=12541): Launched
02:13:35.077490 run osmo-stp_10.42.42.5(pid=12538): ERR: Terminated: ERROR {rc=1} [trial-2368↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12538)]
02:13:35.221191 run osmo-stp_10.42.42.5(pid=12538): stdout:
| (launched: 2022-12-10_02:13:33.589392)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:35.364703 run osmo-stp_10.42.42.5(pid=12538): stderr:
| �[0;m�[38;5;43m20221210021333784 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021333784 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021333785 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021333785 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021333786 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021333786 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021333786 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:13:35.503365 run osmo-stp_10.42.42.5(pid=12538): stdout:
| (launched: 2022-12-10_02:13:33.589392)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:35.641429 run osmo-stp_10.42.42.5(pid=12538): stderr:
| �[0;m�[38;5;43m20221210021333784 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021333784 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021333785 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021333785 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021333786 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021333786 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021333786 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:13:35.801646 run patchelf(pid=12541): Terminating (SIGINT)
02:13:35.990340 run patchelf(pid=12541): Terminated: ok {rc=0}
02:13:36.024591 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=12538): Process ended prematurely: osmo-stp_10.42.42.5(pid=12538) [trial-2368↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12538)] [trial-2368↪gprs:sysmo↪iperf3m4.py:8]
02:13:36.034504 tst iperf3m4.py:8: Test FAILED (11.0 sec)
02:13:36.098349 run osmo-hlr_10.42.42.2(pid=12535): ERR: Terminated: ERROR {rc=237} [trial-2368↪gprs:sysmo↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=12535)]
02:13:36.150245 run osmo-hlr_10.42.42.2(pid=12535): stdout:
| (launched: 2022-12-10_02:13:32.535135)
02:13:36.202691 run osmo-hlr_10.42.42.2(pid=12535): stderr:
| �[0;m�[1;31m20221210021332736 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210021332736 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210021332736 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210021332736 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210021332736 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210021332736 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210021332748 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2368/run.2022-12-10_02-07-09/gprs:sysmo/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210021332759 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210021332759 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210021332759 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
02:13:36.247748 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12524): Terminating (SIGTERM)
02:13:36.290093 run iperf3-srv_10.42.42.10(pid=12525): Terminating (SIGTERM)
02:13:36.332336 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12526): Terminating (SIGTERM)
02:13:36.374573 run iperf3-srv_10.42.42.10(pid=12527): Terminating (SIGTERM)
02:13:36.419597 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12528): Terminating (SIGTERM)
02:13:36.462066 run iperf3-srv_10.42.42.10(pid=12529): Terminating (SIGTERM)
02:13:36.504229 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12530): Terminating (SIGTERM)
02:13:36.546440 run iperf3-srv_10.42.42.10(pid=12531): Terminating (SIGTERM)
02:13:36.588521 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12534): Terminating (SIGTERM)
02:13:36.630748 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12537): Terminating (SIGTERM)
02:13:36.673065 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12540): Terminating (SIGTERM)
02:13:36.682853 --- ParallelTerminationStrategy: PID 12524 died...
02:13:36.739322 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12524): Terminated: ok {rc=0}
02:13:36.749705 --- ParallelTerminationStrategy: PID 12525 died...
02:13:36.805808 run iperf3-srv_10.42.42.10(pid=12525): Terminated {rc=256}
02:13:36.816006 --- ParallelTerminationStrategy: PID 12526 died...
02:13:36.872064 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12526): Terminated: ok {rc=0}
02:13:36.882292 --- ParallelTerminationStrategy: PID 12527 died...
02:13:36.938566 run iperf3-srv_10.42.42.10(pid=12527): Terminated {rc=256}
02:13:36.948694 --- ParallelTerminationStrategy: PID 12528 died...
02:13:37.005400 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12528): Terminated: ok {rc=0}
02:13:37.015707 --- ParallelTerminationStrategy: PID 12529 died...
02:13:37.071784 run iperf3-srv_10.42.42.10(pid=12529): Terminated {rc=256}
02:13:37.082094 --- ParallelTerminationStrategy: PID 12530 died...
02:13:37.138234 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=12530): Terminated: ok {rc=0}
02:13:37.148451 --- ParallelTerminationStrategy: PID 12531 died...
02:13:37.204302 run iperf3-srv_10.42.42.10(pid=12531): Terminated {rc=256}
02:13:37.214868 --- ParallelTerminationStrategy: PID 12534 died...
02:13:37.271110 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12534): Terminated: ok {rc=0}
02:13:37.281473 --- ParallelTerminationStrategy: PID 12537 died...
02:13:37.338097 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12537): Terminated: ok {rc=0}
02:13:37.348489 --- ParallelTerminationStrategy: PID 12540 died...
02:13:37.405441 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12540): Terminated: ok {rc=0}
02:13:37.494313 bus /gobi_4: Setting Powered False
02:13:38.602290 bus /gobi_4: Setting Powered False
02:13:39.714927 bus /gobi_1: Setting Powered False
02:13:40.820106 bus /gobi_1: Setting Powered False
02:13:41.929632 bus /gobi_0: Setting Powered False
02:13:43.036399 bus /gobi_0: Setting Powered False
02:13:44.148016 bus /gobi_6: Setting Powered False
38gprs:sysmo7.0osmo-stp_10.42.42.5(pid=12548): Process ended prematurely: osmo-stp_10.42.42.5(pid=12548) [trial-2368↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12548)]2331ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=12548): Process ended prematurely: osmo-stp_10.42.42.5(pid=12548) [trial-2368↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12548)]
----------------------------------------------
trial-2368 gprs:sysmo ping.py
----------------------------------------------
02:13:45.217317 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:45.228939 tst gprs:sysmo: Using 1 x bts (candidates: 1)
02:13:45.367864 tst ping.py: using LAC 25275
02:13:45.495516 tst ping.py: using RAC 30
02:13:45.622054 tst ping.py: using CellId 25275
02:13:45.749335 tst ping.py: using BVCI 25276
02:13:45.774671 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:45.786054 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:45.797188 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:45.808293 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:45.819161 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:45.830014 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:45.841651 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:45.853221 tst gprs:sysmo: Using 1 x modem (candidates: 4)
02:13:46.207515 tst ping.py:19: start network...
02:13:46.386603 run osmo-hlr_10.42.42.2: Starting osmo-hlr
02:13:47.000030 run create_hlr_db(pid=12542): Launched
02:13:47.195386 bus /gobi_6: Setting Powered False
02:13:48.212169 run create_hlr_db(pid=12542): Terminated: ok {rc=0}
02:13:48.629688 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12544): Launched
02:13:48.939260 run osmo-hlr_10.42.42.2(pid=12545): Launched
02:13:49.057216 run osmo-stp_10.42.42.5: Starting osmo-stp
02:13:49.597802 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12547): Launched
02:13:49.905903 run osmo-stp_10.42.42.5(pid=12548): Launched
02:13:50.017975 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
02:13:50.555944 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12550): Launched
02:13:50.665648 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
02:13:50.936728 run patchelf(pid=12551): Launched
02:13:51.117723 run osmo-stp_10.42.42.5(pid=12548): ERR: Terminated: ERROR {rc=1} [trial-2368↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12548)]
02:13:51.249663 run osmo-stp_10.42.42.5(pid=12548): stdout:
| (launched: 2022-12-10_02:13:49.753144)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:51.382012 run osmo-stp_10.42.42.5(pid=12548): stderr:
| �[0;m�[38;5;43m20221210021349924 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021349924 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021349924 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021349925 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021349927 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021349927 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021349927 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:13:51.509676 run osmo-stp_10.42.42.5(pid=12548): stdout:
| (launched: 2022-12-10_02:13:49.753144)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:51.635905 run osmo-stp_10.42.42.5(pid=12548): stderr:
| �[0;m�[38;5;43m20221210021349924 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021349924 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021349924 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021349925 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021349927 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021349927 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021349927 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:13:51.783054 run patchelf(pid=12551): Terminating (SIGINT)
02:13:51.955908 run patchelf(pid=12551): Terminated: ok {rc=0}
02:13:51.986968 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=12548): Process ended prematurely: osmo-stp_10.42.42.5(pid=12548) [trial-2368↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12548)] [trial-2368↪gprs:sysmo↪ping.py:22]
02:13:51.995675 tst ping.py:22: Test FAILED (6.8 sec)
02:13:52.057620 run osmo-hlr_10.42.42.2(pid=12545): ERR: Terminated: ERROR {rc=237} [trial-2368↪gprs:sysmo↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=12545)]
02:13:52.108344 run osmo-hlr_10.42.42.2(pid=12545): stdout:
| (launched: 2022-12-10_02:13:48.785719)
02:13:52.159484 run osmo-hlr_10.42.42.2(pid=12545): stderr:
| �[0;m�[1;31m20221210021348962 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210021348962 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210021348962 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210021348962 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210021348962 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210021348962 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210021348974 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2368/run.2022-12-10_02-07-09/gprs:sysmo/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210021348985 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210021348985 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210021348985 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
02:13:52.202726 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12544): Terminating (SIGTERM)
02:13:52.244267 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12547): Terminating (SIGTERM)
02:13:52.285312 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12550): Terminating (SIGTERM)
02:13:52.295093 --- ParallelTerminationStrategy: PID 12544 died...
02:13:52.350561 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12544): Terminated: ok {rc=0}
02:13:52.360564 --- ParallelTerminationStrategy: PID 12547 died...
02:13:52.415224 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12547): Terminated: ok {rc=0}
02:13:52.425408 --- ParallelTerminationStrategy: PID 12550 died...
02:13:52.480238 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12550): Terminated: ok {rc=0}
02:13:52.571827 bus /gobi_6: Setting Powered False
38gprs:sysmo7.0osmo-stp_10.42.42.5(pid=12558): Process ended prematurely: osmo-stp_10.42.42.5(pid=12558) [trial-2368↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12558)]2331ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=12558): Process ended prematurely: osmo-stp_10.42.42.5(pid=12558) [trial-2368↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12558)]
----------------------------------------------
trial-2368 gprs:sysmo ping_idle_ping.py
----------------------------------------------
02:13:53.624626 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:53.636450 tst gprs:sysmo: Using 1 x bts (candidates: 1)
02:13:53.771115 tst ping_idle_ping.py: using LAC 25276
02:13:53.901629 tst ping_idle_ping.py: using RAC 31
02:13:54.034333 tst ping_idle_ping.py: using CellId 25276
02:13:54.159148 tst ping_idle_ping.py: using BVCI 25277
02:13:54.183335 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:54.194370 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:54.205152 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:54.216291 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:54.227493 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:54.238614 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:54.249778 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
02:13:54.261036 tst gprs:sysmo: Using 1 x modem (candidates: 4)
02:13:54.572405 tst ping_idle_ping.py:24: start network...
02:13:54.710789 run osmo-hlr_10.42.42.2: Starting osmo-hlr
02:13:55.232414 run create_hlr_db(pid=12552): Launched
02:13:55.411016 bus /gobi_6: Setting Powered False
02:13:56.431827 run create_hlr_db(pid=12552): Terminated: ok {rc=0}
02:13:56.845854 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12554): Launched
02:13:57.156507 run osmo-hlr_10.42.42.2(pid=12555): Launched
02:13:57.274014 run osmo-stp_10.42.42.5: Starting osmo-stp
02:13:57.820307 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12557): Launched
02:13:58.129714 run osmo-stp_10.42.42.5(pid=12558): Launched
02:13:58.242356 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
02:13:58.784227 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12560): Launched
02:13:58.894260 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
02:13:59.165914 run patchelf(pid=12561): Launched
02:13:59.346999 run osmo-stp_10.42.42.5(pid=12558): ERR: Terminated: ERROR {rc=1} [trial-2368↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12558)]
02:13:59.479754 run osmo-stp_10.42.42.5(pid=12558): stdout:
| (launched: 2022-12-10_02:13:57.976407)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:59.612572 run osmo-stp_10.42.42.5(pid=12558): stderr:
| �[0;m�[38;5;43m20221210021358150 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021358151 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021358151 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021358151 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021358152 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021358152 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021358152 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:13:59.741443 run osmo-stp_10.42.42.5(pid=12558): stdout:
| (launched: 2022-12-10_02:13:57.976407)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
02:13:59.868369 run osmo-stp_10.42.42.5(pid=12558): stderr:
| �[0;m�[38;5;43m20221210021358150 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210021358151 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210021358151 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210021358151 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210021358152 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210021358152 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210021358152 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
02:14:00.016264 run patchelf(pid=12561): Terminating (SIGINT)
02:14:00.189982 run patchelf(pid=12561): Terminated: ok {rc=0}
02:14:00.221679 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=12558): Process ended prematurely: osmo-stp_10.42.42.5(pid=12558) [trial-2368↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=12558)] [trial-2368↪gprs:sysmo↪ping_idle_ping.py:27]
02:14:00.230294 tst ping_idle_ping.py:27: Test FAILED (6.6 sec)
02:14:00.291842 run osmo-hlr_10.42.42.2(pid=12555): ERR: Terminated: ERROR {rc=237} [trial-2368↪gprs:sysmo↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=12555)]
02:14:00.341762 run osmo-hlr_10.42.42.2(pid=12555): stdout:
| (launched: 2022-12-10_02:13:57.002450)
02:14:00.392394 run osmo-hlr_10.42.42.2(pid=12555): stderr:
| �[0;m�[1;31m20221210021357183 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210021357183 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210021357183 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210021357183 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210021357183 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210021357183 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210021357199 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2368/run.2022-12-10_02-07-09/gprs:sysmo/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210021357209 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210021357209 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210021357209 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
02:14:00.435132 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12554): Terminating (SIGTERM)
02:14:00.476031 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12557): Terminating (SIGTERM)
02:14:00.516769 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12560): Terminating (SIGTERM)
02:14:00.526514 --- ParallelTerminationStrategy: PID 12554 died...
02:14:00.581110 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12554): Terminated: ok {rc=0}
02:14:00.591310 --- ParallelTerminationStrategy: PID 12557 died...
02:14:00.645586 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12557): Terminated: ok {rc=0}
02:14:00.655221 --- ParallelTerminationStrategy: PID 12560 died...
02:14:00.709239 run pcap-recorder_any(filters='host 10.42.42.6')(pid=12560): Terminated: ok {rc=0}
02:14:00.798212 bus /gobi_6: Setting Powered False
5gprs:sysmo00