46gprs:sysmo10.0osmo-stp_10.42.42.5(pid=31696): Process ended prematurely: osmo-stp_10.42.42.5(pid=31696) [trial-2359↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31696)]2314cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31696): Process ended prematurely: osmo-stp_10.42.42.5(pid=31696) [trial-2359↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31696)]
----------------------------------------------
trial-2359 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
11:23:36.397804 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:36.408735 tst gprs:sysmo: Using 1 x bts (candidates: 1)
11:23:36.544552 tst cs_paging_gprs_active.py: using LAC 21123
11:23:36.678486 tst cs_paging_gprs_active.py: using RAC 213
11:23:36.814917 tst cs_paging_gprs_active.py: using CellId 21123
11:23:36.944493 tst cs_paging_gprs_active.py: using BVCI 21124
11:23:36.970404 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:36.981607 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:36.992473 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:37.003576 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:37.014861 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:37.025821 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:37.037215 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:37.048002 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:37.059428 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:23:37.178446 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:23:37.468278 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
11:23:37.601081 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:23:37.928376 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31686): Launched
11:23:38.200602 run iperf3-srv_10.42.42.10(pid=31687): Launched
11:23:38.392287 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
11:23:38.523322 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:23:38.850467 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31688): Launched
11:23:39.121359 run iperf3-srv_10.42.42.10(pid=31689): Launched
11:23:39.311757 tst cs_paging_gprs_active.py:38: start network...
11:23:39.441430 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:23:39.945838 run create_hlr_db(pid=31690): Launched
11:23:40.126005 bus /sierra_2: Setting Powered False
11:23:41.146801 run create_hlr_db(pid=31690): Terminated: ok {rc=0}
11:23:41.564321 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31692): Launched
11:23:41.878886 run osmo-hlr_10.42.42.2(pid=31693): Launched
11:23:41.993414 run osmo-stp_10.42.42.5: Starting osmo-stp
11:23:42.533640 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31695): Launched
11:23:42.847598 run osmo-stp_10.42.42.5(pid=31696): Launched
11:23:42.961820 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:23:44.480260 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31698): Launched
11:23:44.591573 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:23:44.868619 run patchelf(pid=31699): Launched
11:23:45.050944 run osmo-stp_10.42.42.5(pid=31696): ERR: Terminated: ERROR {rc=1} [trial-2359↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31696)]
11:23:45.183692 run osmo-stp_10.42.42.5(pid=31696): stdout:
| (launched: 2022-12-09_11:23:42.692872)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:23:45.316741 run osmo-stp_10.42.42.5(pid=31696): stderr:
| �[0;m�[38;5;43m20221209112342867 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112342867 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112342868 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112342868 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112342869 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112342869 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112342869 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:23:45.445950 run osmo-stp_10.42.42.5(pid=31696): stdout:
| (launched: 2022-12-09_11:23:42.692872)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:23:45.574120 run osmo-stp_10.42.42.5(pid=31696): stderr:
| �[0;m�[38;5;43m20221209112342867 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112342867 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112342868 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112342868 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112342869 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112342869 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112342869 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:23:45.724332 run patchelf(pid=31699): Terminating (SIGINT)
11:23:45.912462 run patchelf(pid=31699): Terminated: ok {rc=0}
11:23:45.946203 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=31696): Process ended prematurely: osmo-stp_10.42.42.5(pid=31696) [trial-2359↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31696)] [trial-2359↪gprs:sysmo↪cs_paging_gprs_active.py:38]
11:23:45.955968 tst cs_paging_gprs_active.py:38: Test FAILED (9.6 sec)
11:23:46.018763 run osmo-hlr_10.42.42.2(pid=31693): ERR: Terminated: ERROR {rc=237} [trial-2359↪gprs:sysmo↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31693)]
11:23:46.069726 run osmo-hlr_10.42.42.2(pid=31693): stdout:
| (launched: 2022-12-09_11:23:41.722676)
11:23:46.121413 run osmo-hlr_10.42.42.2(pid=31693): stderr:
| �[0;m�[1;31m20221209112341904 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209112341904 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209112341904 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209112341904 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209112341904 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209112341904 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209112341917 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2359/run.2022-12-09_11-17-44/gprs:sysmo/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209112341926 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209112341926 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209112341926 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:23:46.165423 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31686): Terminating (SIGTERM)
11:23:46.206780 run iperf3-srv_10.42.42.10(pid=31687): Terminating (SIGTERM)
11:23:46.250102 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31688): Terminating (SIGTERM)
11:23:46.292412 run iperf3-srv_10.42.42.10(pid=31689): Terminating (SIGTERM)
11:23:46.335354 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31692): Terminating (SIGTERM)
11:23:46.376670 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31695): Terminating (SIGTERM)
11:23:46.418127 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31698): Terminating (SIGTERM)
11:23:46.427536 --- ParallelTerminationStrategy: PID 31686 died...
11:23:46.483067 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31686): Terminated: ok {rc=0}
11:23:46.493339 --- ParallelTerminationStrategy: PID 31687 died...
11:23:46.548457 run iperf3-srv_10.42.42.10(pid=31687): Terminated {rc=256}
11:23:46.558760 --- ParallelTerminationStrategy: PID 31688 died...
11:23:46.614785 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31688): Terminated: ok {rc=0}
11:23:46.625163 --- ParallelTerminationStrategy: PID 31689 died...
11:23:46.681180 run iperf3-srv_10.42.42.10(pid=31689): Terminated {rc=256}
11:23:46.691521 --- ParallelTerminationStrategy: PID 31692 died...
11:23:46.747191 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31692): Terminated: ok {rc=0}
11:23:46.757590 --- ParallelTerminationStrategy: PID 31695 died...
11:23:46.813255 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31695): Terminated: ok {rc=0}
11:23:46.823612 --- ParallelTerminationStrategy: PID 31698 died...
11:23:46.879496 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31698): Terminated: ok {rc=0}
11:23:46.968407 bus /gobi_0: Setting Powered False
11:23:48.076139 bus /gobi_0: Setting Powered False
11:23:49.185408 bus /gobi_6: Setting Powered False
46gprs:sysmo8.0osmo-stp_10.42.42.5(pid=31708): Process ended prematurely: osmo-stp_10.42.42.5(pid=31708) [trial-2359↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31708)]2314iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31708): Process ended prematurely: osmo-stp_10.42.42.5(pid=31708) [trial-2359↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31708)]
----------------------------------------------
trial-2359 gprs:sysmo iperf3.py
----------------------------------------------
11:23:50.304608 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:50.316692 tst gprs:sysmo: Using 1 x bts (candidates: 1)
11:23:50.445946 tst iperf3.py: using LAC 21124
11:23:50.572428 tst iperf3.py: using RAC 214
11:23:50.698094 tst iperf3.py: using CellId 21124
11:23:50.823223 tst iperf3.py: using BVCI 21125
11:23:50.847270 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:50.857958 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:50.868427 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:50.879840 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:50.890911 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:50.902052 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:50.913245 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:50.924316 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:23:50.935497 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:23:51.220221 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
11:23:51.350566 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:23:51.675924 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31700): Launched
11:23:51.947448 run iperf3-srv_10.42.42.10(pid=31701): Launched
11:23:52.137190 tst iperf3.py:8: start network...
11:23:52.266481 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:23:52.766789 run create_hlr_db(pid=31702): Launched
11:23:52.945377 bus /gobi_6: Setting Powered False
11:23:53.966718 run create_hlr_db(pid=31702): Terminated: ok {rc=0}
11:23:54.411916 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31704): Launched
11:23:54.747693 run osmo-hlr_10.42.42.2(pid=31705): Launched
11:23:54.874141 run osmo-stp_10.42.42.5: Starting osmo-stp
11:23:55.456523 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31707): Launched
11:23:55.791953 run osmo-stp_10.42.42.5(pid=31708): Launched
11:23:55.913209 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:23:56.499462 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Launched
11:23:56.619944 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:23:56.913959 run patchelf(pid=31711): Launched
11:23:57.108827 run osmo-stp_10.42.42.5(pid=31708): ERR: Terminated: ERROR {rc=1} [trial-2359↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31708)]
11:23:57.251284 run osmo-stp_10.42.42.5(pid=31708): stdout:
| (launched: 2022-12-09_11:23:55.626576)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:23:57.393864 run osmo-stp_10.42.42.5(pid=31708): stderr:
| �[0;m�[38;5;43m20221209112355823 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112355823 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112355824 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112355824 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112355825 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112355825 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112355825 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:23:57.531859 run osmo-stp_10.42.42.5(pid=31708): stdout:
| (launched: 2022-12-09_11:23:55.626576)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:23:57.669346 run osmo-stp_10.42.42.5(pid=31708): stderr:
| �[0;m�[38;5;43m20221209112355823 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112355823 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112355824 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112355824 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112355825 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112355825 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112355825 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:23:57.829306 run patchelf(pid=31711): Terminating (SIGINT)
11:23:58.017765 run patchelf(pid=31711): Terminated: ok {rc=0}
11:23:58.052098 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=31708): Process ended prematurely: osmo-stp_10.42.42.5(pid=31708) [trial-2359↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31708)] [trial-2359↪gprs:sysmo↪iperf3.py:8]
11:23:58.061972 tst iperf3.py:8: Test FAILED (7.8 sec)
11:23:58.126351 run osmo-hlr_10.42.42.2(pid=31705): ERR: Terminated: ERROR {rc=237} [trial-2359↪gprs:sysmo↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31705)]
11:23:58.178906 run osmo-hlr_10.42.42.2(pid=31705): stdout:
| (launched: 2022-12-09_11:23:54.581987)
11:23:58.232141 run osmo-hlr_10.42.42.2(pid=31705): stderr:
| �[0;m�[1;31m20221209112354788 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209112354788 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209112354788 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209112354789 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209112354789 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209112354789 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209112354799 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2359/run.2022-12-09_11-17-44/gprs:sysmo/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209112354808 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209112354808 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209112354808 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:23:58.277195 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31700): Terminating (SIGTERM)
11:23:58.320475 run iperf3-srv_10.42.42.10(pid=31701): Terminating (SIGTERM)
11:23:58.365144 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31704): Terminating (SIGTERM)
11:23:58.408011 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31707): Terminating (SIGTERM)
11:23:58.451329 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Terminating (SIGTERM)
11:23:58.461213 --- ParallelTerminationStrategy: PID 31700 died...
11:23:58.519066 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31700): Terminated: ok {rc=0}
11:23:58.529473 --- ParallelTerminationStrategy: PID 31701 died...
11:23:58.586817 run iperf3-srv_10.42.42.10(pid=31701): Terminated {rc=256}
11:23:58.597112 --- ParallelTerminationStrategy: PID 31704 died...
11:23:58.654391 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31704): Terminated: ok {rc=0}
11:23:58.664730 --- ParallelTerminationStrategy: PID 31707 died...
11:23:58.721841 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31707): Terminated: ok {rc=0}
11:23:58.732390 --- ParallelTerminationStrategy: PID 31710 died...
11:23:58.790000 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Terminated: ok {rc=0}
11:23:58.878116 bus /gobi_6: Setting Powered False
47gprs:sysmo12.0osmo-stp_10.42.42.5(pid=31726): Process ended prematurely: osmo-stp_10.42.42.5(pid=31726) [trial-2359↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31726)]2313iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31726): Process ended prematurely: osmo-stp_10.42.42.5(pid=31726) [trial-2359↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31726)]
----------------------------------------------
trial-2359 gprs:sysmo iperf3m4.py
----------------------------------------------
11:23:59.990406 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:00.001807 tst gprs:sysmo: Using 1 x bts (candidates: 1)
11:24:00.130573 tst iperf3m4.py: using LAC 21125
11:24:00.255830 tst iperf3m4.py: using RAC 215
11:24:00.380874 tst iperf3m4.py: using CellId 21125
11:24:00.505549 tst iperf3m4.py: using BVCI 21126
11:24:00.529720 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:00.540282 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:00.550788 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:00.561772 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:00.572481 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:00.583405 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:00.594579 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:00.605567 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:00.616443 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:24:00.735820 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:24:00.871830 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:24:01.015359 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:24:01.350091 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
11:24:01.483099 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:24:01.829868 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31712): Launched
11:24:02.125345 run iperf3-srv_10.42.42.10(pid=31713): Launched
11:24:02.319102 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
11:24:02.448882 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:24:02.774066 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31714): Launched
11:24:03.045275 run iperf3-srv_10.42.42.10(pid=31715): Launched
11:24:03.234721 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
11:24:03.363793 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:24:03.688111 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31716): Launched
11:24:03.958210 run iperf3-srv_10.42.42.10(pid=31717): Launched
11:24:04.147440 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
11:24:04.276068 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:24:04.600272 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31718): Launched
11:24:04.886698 run iperf3-srv_10.42.42.10(pid=31719): Launched
11:24:05.075680 tst iperf3m4.py:8: start network...
11:24:05.205015 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:24:05.706322 run create_hlr_db(pid=31720): Launched
11:24:05.885271 bus /gobi_6: Setting Powered False
11:24:06.907307 run create_hlr_db(pid=31720): Terminated: ok {rc=0}
11:24:07.354186 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31722): Launched
11:24:07.692110 run osmo-hlr_10.42.42.2(pid=31723): Launched
11:24:07.811865 run osmo-stp_10.42.42.5: Starting osmo-stp
11:24:08.398895 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31725): Launched
11:24:08.735851 run osmo-stp_10.42.42.5(pid=31726): Launched
11:24:08.857393 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:24:09.442842 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31728): Launched
11:24:09.562085 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:24:09.855178 run patchelf(pid=31729): Launched
11:24:10.050323 run osmo-stp_10.42.42.5(pid=31726): ERR: Terminated: ERROR {rc=1} [trial-2359↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31726)]
11:24:10.192816 run osmo-stp_10.42.42.5(pid=31726): stdout:
| (launched: 2022-12-09_11:24:08.567786)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:24:10.335533 run osmo-stp_10.42.42.5(pid=31726): stderr:
| �[0;m�[38;5;43m20221209112408763 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112408763 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112408763 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112408764 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112408766 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112408766 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112408766 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:24:10.473526 run osmo-stp_10.42.42.5(pid=31726): stdout:
| (launched: 2022-12-09_11:24:08.567786)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:24:10.610913 run osmo-stp_10.42.42.5(pid=31726): stderr:
| �[0;m�[38;5;43m20221209112408763 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112408763 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112408763 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112408764 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112408766 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112408766 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112408766 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:24:10.771191 run patchelf(pid=31729): Terminating (SIGINT)
11:24:10.960023 run patchelf(pid=31729): Terminated: ok {rc=0}
11:24:10.994469 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=31726): Process ended prematurely: osmo-stp_10.42.42.5(pid=31726) [trial-2359↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31726)] [trial-2359↪gprs:sysmo↪iperf3m4.py:8]
11:24:11.004316 tst iperf3m4.py:8: Test FAILED (11.0 sec)
11:24:11.069100 run osmo-hlr_10.42.42.2(pid=31723): ERR: Terminated: ERROR {rc=237} [trial-2359↪gprs:sysmo↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31723)]
11:24:11.122026 run osmo-hlr_10.42.42.2(pid=31723): stdout:
| (launched: 2022-12-09_11:24:07.524279)
11:24:11.175170 run osmo-hlr_10.42.42.2(pid=31723): stderr:
| �[0;m�[1;31m20221209112407729 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209112407729 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209112407729 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209112407729 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209112407729 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209112407729 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209112407742 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2359/run.2022-12-09_11-17-44/gprs:sysmo/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209112407751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209112407751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209112407751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:24:11.219908 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31712): Terminating (SIGTERM)
11:24:11.262834 run iperf3-srv_10.42.42.10(pid=31713): Terminating (SIGTERM)
11:24:11.305683 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31714): Terminating (SIGTERM)
11:24:11.348024 run iperf3-srv_10.42.42.10(pid=31715): Terminating (SIGTERM)
11:24:11.391042 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31716): Terminating (SIGTERM)
11:24:11.433778 run iperf3-srv_10.42.42.10(pid=31717): Terminating (SIGTERM)
11:24:11.476309 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31718): Terminating (SIGTERM)
11:24:11.518786 run iperf3-srv_10.42.42.10(pid=31719): Terminating (SIGTERM)
11:24:11.561884 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31722): Terminating (SIGTERM)
11:24:11.604781 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31725): Terminating (SIGTERM)
11:24:11.647622 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31728): Terminating (SIGTERM)
11:24:11.657759 --- ParallelTerminationStrategy: PID 31712 died...
11:24:11.715009 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31712): Terminated: ok {rc=0}
11:24:11.725327 --- ParallelTerminationStrategy: PID 31713 died...
11:24:11.782171 run iperf3-srv_10.42.42.10(pid=31713): Terminated {rc=256}
11:24:11.792367 --- ParallelTerminationStrategy: PID 31714 died...
11:24:11.849499 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31714): Terminated: ok {rc=0}
11:24:11.859706 --- ParallelTerminationStrategy: PID 31715 died...
11:24:11.917187 run iperf3-srv_10.42.42.10(pid=31715): Terminated {rc=256}
11:24:11.927391 --- ParallelTerminationStrategy: PID 31716 died...
11:24:11.984489 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31716): Terminated: ok {rc=0}
11:24:11.994763 --- ParallelTerminationStrategy: PID 31717 died...
11:24:12.051494 run iperf3-srv_10.42.42.10(pid=31717): Terminated {rc=256}
11:24:12.061769 --- ParallelTerminationStrategy: PID 31718 died...
11:24:12.118646 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31718): Terminated: ok {rc=0}
11:24:12.128833 --- ParallelTerminationStrategy: PID 31719 died...
11:24:12.185622 run iperf3-srv_10.42.42.10(pid=31719): Terminated {rc=256}
11:24:12.195855 --- ParallelTerminationStrategy: PID 31722 died...
11:24:12.252761 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31722): Terminated: ok {rc=0}
11:24:12.262954 --- ParallelTerminationStrategy: PID 31725 died...
11:24:12.319722 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31725): Terminated: ok {rc=0}
11:24:12.330070 --- ParallelTerminationStrategy: PID 31728 died...
11:24:12.386744 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31728): Terminated: ok {rc=0}
11:24:12.475628 bus /gobi_4: Setting Powered False
11:24:13.583601 bus /gobi_4: Setting Powered False
11:24:14.691956 bus /gobi_1: Setting Powered False
11:24:15.799464 bus /gobi_1: Setting Powered False
11:24:16.911044 bus /gobi_0: Setting Powered False
11:24:18.011871 bus /gobi_0: Setting Powered False
11:24:19.124876 bus /gobi_6: Setting Powered False
29gprs:sysmo7.0osmo-stp_10.42.42.5(pid=31737): Process ended prematurely: osmo-stp_10.42.42.5(pid=31737) [trial-2359↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31737)]2331ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31737): Process ended prematurely: osmo-stp_10.42.42.5(pid=31737) [trial-2359↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31737)]
----------------------------------------------
trial-2359 gprs:sysmo ping.py
----------------------------------------------
11:24:20.191345 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:20.202869 tst gprs:sysmo: Using 1 x bts (candidates: 1)
11:24:20.330404 tst ping.py: using LAC 21126
11:24:20.452573 tst ping.py: using RAC 216
11:24:20.574758 tst ping.py: using CellId 21126
11:24:20.694247 tst ping.py: using BVCI 21127
11:24:20.717879 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:20.728470 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:20.739013 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:20.749765 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:20.760243 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:20.770847 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:20.781546 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:20.792247 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:24:21.073547 tst ping.py:19: start network...
11:24:21.200631 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:24:21.692498 run create_hlr_db(pid=31731): Launched
11:24:21.869582 bus /gobi_6: Setting Powered False
11:24:22.891420 run create_hlr_db(pid=31731): Terminated: ok {rc=0}
11:24:23.330941 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31733): Launched
11:24:23.661389 run osmo-hlr_10.42.42.2(pid=31734): Launched
11:24:23.782629 run osmo-stp_10.42.42.5: Starting osmo-stp
11:24:24.354155 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31736): Launched
11:24:24.686311 run osmo-stp_10.42.42.5(pid=31737): Launched
11:24:24.804783 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:24:25.382520 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31739): Launched
11:24:25.499368 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:24:25.787964 run patchelf(pid=31740): Launched
11:24:25.981280 run osmo-stp_10.42.42.5(pid=31737): ERR: Terminated: ERROR {rc=1} [trial-2359↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31737)]
11:24:26.121697 run osmo-stp_10.42.42.5(pid=31737): stdout:
| (launched: 2022-12-09_11:24:24.520172)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:24:26.262501 run osmo-stp_10.42.42.5(pid=31737): stderr:
| �[0;m�[38;5;43m20221209112424715 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112424715 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112424716 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112424716 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112424716 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112424717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112424717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:24:26.398274 run osmo-stp_10.42.42.5(pid=31737): stdout:
| (launched: 2022-12-09_11:24:24.520172)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:24:26.532614 run osmo-stp_10.42.42.5(pid=31737): stderr:
| �[0;m�[38;5;43m20221209112424715 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112424715 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112424716 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112424716 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112424716 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112424717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112424717 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:24:26.690539 run patchelf(pid=31740): Terminating (SIGINT)
11:24:26.876036 run patchelf(pid=31740): Terminated: ok {rc=0}
11:24:26.909365 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=31737): Process ended prematurely: osmo-stp_10.42.42.5(pid=31737) [trial-2359↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31737)] [trial-2359↪gprs:sysmo↪ping.py:22]
11:24:26.918621 tst ping.py:22: Test FAILED (6.7 sec)
11:24:26.982960 run osmo-hlr_10.42.42.2(pid=31734): ERR: Terminated: ERROR {rc=237} [trial-2359↪gprs:sysmo↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31734)]
11:24:27.035695 run osmo-hlr_10.42.42.2(pid=31734): stdout:
| (launched: 2022-12-09_11:24:23.497682)
11:24:27.088696 run osmo-hlr_10.42.42.2(pid=31734): stderr:
| �[0;m�[1;31m20221209112423699 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209112423699 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209112423699 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209112423699 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209112423699 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209112423699 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209112423711 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2359/run.2022-12-09_11-17-44/gprs:sysmo/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209112423719 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209112423720 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209112423720 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:24:27.133798 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31733): Terminating (SIGTERM)
11:24:27.176337 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31736): Terminating (SIGTERM)
11:24:27.219823 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31739): Terminating (SIGTERM)
11:24:27.229918 --- ParallelTerminationStrategy: PID 31733 died...
11:24:27.287177 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31733): Terminated: ok {rc=0}
11:24:27.297463 --- ParallelTerminationStrategy: PID 31736 died...
11:24:27.354544 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31736): Terminated: ok {rc=0}
11:24:27.364726 --- ParallelTerminationStrategy: PID 31739 died...
11:24:27.421682 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31739): Terminated: ok {rc=0}
11:24:27.510113 bus /gobi_6: Setting Powered False
29gprs:sysmo7.0osmo-stp_10.42.42.5(pid=31747): Process ended prematurely: osmo-stp_10.42.42.5(pid=31747) [trial-2359↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31747)]2331ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31747): Process ended prematurely: osmo-stp_10.42.42.5(pid=31747) [trial-2359↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31747)]
----------------------------------------------
trial-2359 gprs:sysmo ping_idle_ping.py
----------------------------------------------
11:24:28.562368 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:28.574161 tst gprs:sysmo: Using 1 x bts (candidates: 1)
11:24:28.709987 tst ping_idle_ping.py: using LAC 21127
11:24:28.840183 tst ping_idle_ping.py: using RAC 217
11:24:28.968673 tst ping_idle_ping.py: using CellId 21127
11:24:29.101535 tst ping_idle_ping.py: using BVCI 21128
11:24:29.126796 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:29.137869 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:29.148727 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:29.160576 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:29.171884 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:29.183524 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:29.195571 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:24:29.207031 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:24:29.499988 tst ping_idle_ping.py:24: start network...
11:24:29.638771 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:24:30.142572 run create_hlr_db(pid=31741): Launched
11:24:30.328892 bus /gobi_6: Setting Powered False
11:24:31.360251 run create_hlr_db(pid=31741): Terminated: ok {rc=0}
11:24:31.806840 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31743): Launched
11:24:32.218094 run osmo-hlr_10.42.42.2(pid=31744): Launched
11:24:32.380483 run osmo-stp_10.42.42.5: Starting osmo-stp
11:24:32.901945 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31746): Launched
11:24:33.202588 run osmo-stp_10.42.42.5(pid=31747): Launched
11:24:33.310719 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:24:33.833209 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31749): Launched
11:24:33.939162 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:24:34.202501 run patchelf(pid=31750): Launched
11:24:34.378714 run osmo-stp_10.42.42.5(pid=31747): ERR: Terminated: ERROR {rc=1} [trial-2359↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31747)]
11:24:34.507614 run osmo-stp_10.42.42.5(pid=31747): stdout:
| (launched: 2022-12-09_11:24:33.054151)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:24:34.636884 run osmo-stp_10.42.42.5(pid=31747): stderr:
| �[0;m�[38;5;43m20221209112433223 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112433224 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112433224 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112433224 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112433226 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112433227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112433227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:24:34.762100 run osmo-stp_10.42.42.5(pid=31747): stdout:
| (launched: 2022-12-09_11:24:33.054151)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:24:34.885644 run osmo-stp_10.42.42.5(pid=31747): stderr:
| �[0;m�[38;5;43m20221209112433223 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209112433224 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209112433224 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209112433224 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209112433226 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209112433227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209112433227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:24:35.030345 run patchelf(pid=31750): Terminating (SIGINT)
11:24:35.201606 run patchelf(pid=31750): Terminated: ok {rc=0}
11:24:35.232364 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=31747): Process ended prematurely: osmo-stp_10.42.42.5(pid=31747) [trial-2359↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31747)] [trial-2359↪gprs:sysmo↪ping_idle_ping.py:27]
11:24:35.240717 tst ping_idle_ping.py:27: Test FAILED (6.7 sec)
11:24:35.300195 run osmo-hlr_10.42.42.2(pid=31744): ERR: Terminated: ERROR {rc=237} [trial-2359↪gprs:sysmo↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31744)]
11:24:35.349487 run osmo-hlr_10.42.42.2(pid=31744): stdout:
| (launched: 2022-12-09_11:24:31.972352)
11:24:35.398849 run osmo-hlr_10.42.42.2(pid=31744): stderr:
| �[0;m�[1;31m20221209112432281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209112432281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209112432281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209112432281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209112432281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209112432281 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209112432300 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2359/run.2022-12-09_11-17-44/gprs:sysmo/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209112432316 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209112432316 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209112432316 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:24:35.441232 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31743): Terminating (SIGTERM)
11:24:35.481042 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31746): Terminating (SIGTERM)
11:24:35.520784 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31749): Terminating (SIGTERM)
11:24:35.530136 --- ParallelTerminationStrategy: PID 31743 died...
11:24:35.583638 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31743): Terminated: ok {rc=0}
11:24:35.593243 --- ParallelTerminationStrategy: PID 31746 died...
11:24:35.646364 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31746): Terminated: ok {rc=0}
11:24:35.655945 --- ParallelTerminationStrategy: PID 31749 died...
11:24:35.709191 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31749): Terminated: ok {rc=0}
11:24:35.798368 bus /gobi_6: Setting Powered False
5gprs:sysmo00