41gprs:sysmo10.0osmo-stp_10.42.42.5(pid=19818): Process ended prematurely: osmo-stp_10.42.42.5(pid=19818) [trial-2354↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19818)]2314cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=19818): Process ended prematurely: osmo-stp_10.42.42.5(pid=19818) [trial-2354↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19818)]
----------------------------------------------
trial-2354 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
03:40:57.806491 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:57.817003 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:40:57.943159 tst cs_paging_gprs_active.py: using LAC 18818
03:40:58.065501 tst cs_paging_gprs_active.py: using RAC 203
03:40:58.191904 tst cs_paging_gprs_active.py: using CellId 18818
03:40:58.313467 tst cs_paging_gprs_active.py: using BVCI 18819
03:40:58.337267 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:58.347681 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:58.358131 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:58.368431 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:58.378743 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:58.389031 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:58.399374 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:58.409648 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:58.420076 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:40:58.538477 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:40:58.838684 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
03:40:58.979356 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:40:59.323757 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19808): Launched
03:40:59.611518 run iperf3-srv_10.42.42.10(pid=19809): Launched
03:40:59.814950 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
03:40:59.953883 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:00.299859 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19810): Launched
03:41:00.587190 run iperf3-srv_10.42.42.10(pid=19811): Launched
03:41:00.790086 tst cs_paging_gprs_active.py:38: start network...
03:41:00.933290 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:41:01.495447 run create_hlr_db(pid=19812): Launched
03:41:01.691197 bus /sierra_2: Setting Powered False
03:41:02.709997 run create_hlr_db(pid=19812): Terminated: ok {rc=0}
03:41:03.128326 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19814): Launched
03:41:03.440799 run osmo-hlr_10.42.42.2(pid=19815): Launched
03:41:03.554452 run osmo-stp_10.42.42.5: Starting osmo-stp
03:41:04.093351 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19817): Launched
03:41:04.405098 run osmo-stp_10.42.42.5(pid=19818): Launched
03:41:04.518408 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:41:06.033596 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19820): Launched
03:41:06.144045 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:41:06.417528 run patchelf(pid=19821): Launched
03:41:06.598015 run osmo-stp_10.42.42.5(pid=19818): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19818)]
03:41:06.729612 run osmo-stp_10.42.42.5(pid=19818): stdout:
| (launched: 2022-12-09_03:41:04.250545)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:06.861476 run osmo-stp_10.42.42.5(pid=19818): stderr:
| �[0;m�[38;5;43m20221209034104422 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034104422 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034104423 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034104423 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034104424 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034104424 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034104424 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:06.989463 run osmo-stp_10.42.42.5(pid=19818): stdout:
| (launched: 2022-12-09_03:41:04.250545)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:07.116418 run osmo-stp_10.42.42.5(pid=19818): stderr:
| �[0;m�[38;5;43m20221209034104422 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034104422 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034104423 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034104423 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034104424 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034104424 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034104424 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:07.264381 run patchelf(pid=19821): Terminating (SIGINT)
03:41:07.438788 run patchelf(pid=19821): Terminated: ok {rc=0}
03:41:07.470664 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=19818): Process ended prematurely: osmo-stp_10.42.42.5(pid=19818) [trial-2354↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19818)] [trial-2354↪gprs:sysmo↪cs_paging_gprs_active.py:38]
03:41:07.479820 tst cs_paging_gprs_active.py:38: Test FAILED (9.7 sec)
03:41:07.540650 run osmo-hlr_10.42.42.2(pid=19815): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:sysmo↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=19815)]
03:41:07.590121 run osmo-hlr_10.42.42.2(pid=19815): stdout:
| (launched: 2022-12-09_03:41:03.286207)
03:41:07.640131 run osmo-hlr_10.42.42.2(pid=19815): stderr:
| �[0;m�[1;31m20221209034103465 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209034103466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209034103466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209034103466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209034103466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209034103466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209034103479 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:sysmo/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209034103488 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209034103488 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209034103488 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:41:07.682729 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19808): Terminating (SIGTERM)
03:41:07.722755 run iperf3-srv_10.42.42.10(pid=19809): Terminating (SIGTERM)
03:41:07.762966 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19810): Terminating (SIGTERM)
03:41:07.802944 run iperf3-srv_10.42.42.10(pid=19811): Terminating (SIGTERM)
03:41:07.843235 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19814): Terminating (SIGTERM)
03:41:07.883275 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19817): Terminating (SIGTERM)
03:41:07.923359 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19820): Terminating (SIGTERM)
03:41:07.932622 --- ParallelTerminationStrategy: PID 19808 died...
03:41:07.986319 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19808): Terminated: ok {rc=0}
03:41:07.995868 --- ParallelTerminationStrategy: PID 19809 died...
03:41:08.049239 run iperf3-srv_10.42.42.10(pid=19809): Terminated {rc=256}
03:41:08.058853 --- ParallelTerminationStrategy: PID 19810 died...
03:41:08.112592 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19810): Terminated: ok {rc=0}
03:41:08.122324 --- ParallelTerminationStrategy: PID 19811 died...
03:41:08.175964 run iperf3-srv_10.42.42.10(pid=19811): Terminated {rc=256}
03:41:08.185874 --- ParallelTerminationStrategy: PID 19814 died...
03:41:08.239506 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19814): Terminated: ok {rc=0}
03:41:08.249194 --- ParallelTerminationStrategy: PID 19817 died...
03:41:08.302853 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19817): Terminated: ok {rc=0}
03:41:08.312524 --- ParallelTerminationStrategy: PID 19820 died...
03:41:08.366731 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19820): Terminated: ok {rc=0}
03:41:08.456815 bus /gobi_0: Setting Powered False
03:41:09.560364 bus /gobi_0: Setting Powered False
03:41:10.673786 bus /gobi_6: Setting Powered False
41gprs:sysmo9.0osmo-stp_10.42.42.5(pid=19830): Process ended prematurely: osmo-stp_10.42.42.5(pid=19830) [trial-2354↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19830)]2314iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=19830): Process ended prematurely: osmo-stp_10.42.42.5(pid=19830) [trial-2354↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19830)]
----------------------------------------------
trial-2354 gprs:sysmo iperf3.py
----------------------------------------------
03:41:11.787041 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:11.798218 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:41:11.923936 tst iperf3.py: using LAC 18819
03:41:12.046024 tst iperf3.py: using RAC 204
03:41:12.178361 tst iperf3.py: using CellId 18819
03:41:12.302150 tst iperf3.py: using BVCI 18820
03:41:12.325899 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:12.336353 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:12.346860 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:12.357296 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:12.367547 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:12.377998 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:12.388388 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:12.399099 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:12.409753 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:12.706040 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
03:41:12.845569 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:13.190669 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19822): Launched
03:41:13.477014 run iperf3-srv_10.42.42.10(pid=19823): Launched
03:41:13.679996 tst iperf3.py:8: start network...
03:41:13.818849 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:41:14.348682 run create_hlr_db(pid=19824): Launched
03:41:14.540686 bus /gobi_6: Setting Powered False
03:41:15.562037 run create_hlr_db(pid=19824): Terminated: ok {rc=0}
03:41:16.002709 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19826): Launched
03:41:16.449122 run osmo-hlr_10.42.42.2(pid=19827): Launched
03:41:16.615440 run osmo-stp_10.42.42.5: Starting osmo-stp
03:41:17.331671 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19829): Launched
03:41:17.669956 run osmo-stp_10.42.42.5(pid=19830): Launched
03:41:17.793304 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:41:18.372753 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19832): Launched
03:41:18.492841 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:41:18.778791 run patchelf(pid=19833): Launched
03:41:18.959248 run osmo-stp_10.42.42.5(pid=19830): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19830)]
03:41:19.090960 run osmo-stp_10.42.42.5(pid=19830): stdout:
| (launched: 2022-12-09_03:41:17.502471)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:19.223026 run osmo-stp_10.42.42.5(pid=19830): stderr:
| �[0;m�[38;5;43m20221209034117696 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034117696 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034117697 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034117697 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034117697 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034117697 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034117698 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:19.350440 run osmo-stp_10.42.42.5(pid=19830): stdout:
| (launched: 2022-12-09_03:41:17.502471)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:19.476595 run osmo-stp_10.42.42.5(pid=19830): stderr:
| �[0;m�[38;5;43m20221209034117696 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034117696 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034117697 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034117697 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034117697 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034117697 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034117698 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:19.624804 run patchelf(pid=19833): Terminating (SIGINT)
03:41:19.799047 run patchelf(pid=19833): Terminated: ok {rc=0}
03:41:19.830939 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=19830): Process ended prematurely: osmo-stp_10.42.42.5(pid=19830) [trial-2354↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19830)] [trial-2354↪gprs:sysmo↪iperf3.py:8]
03:41:19.840075 tst iperf3.py:8: Test FAILED (8.1 sec)
03:41:19.900703 run osmo-hlr_10.42.42.2(pid=19827): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:sysmo↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=19827)]
03:41:19.950307 run osmo-hlr_10.42.42.2(pid=19827): stdout:
| (launched: 2022-12-09_03:41:16.175251)
03:41:20.000801 run osmo-hlr_10.42.42.2(pid=19827): stderr:
| �[0;m�[1;31m20221209034116489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209034116489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209034116489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209034116489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209034116489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209034116489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209034116511 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:sysmo/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209034116525 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209034116525 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209034116525 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:41:20.043559 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19822): Terminating (SIGTERM)
03:41:20.083671 run iperf3-srv_10.42.42.10(pid=19823): Terminating (SIGTERM)
03:41:20.124053 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19826): Terminating (SIGTERM)
03:41:20.164184 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19829): Terminating (SIGTERM)
03:41:20.204288 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19832): Terminating (SIGTERM)
03:41:20.213678 --- ParallelTerminationStrategy: PID 19822 died...
03:41:20.267552 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19822): Terminated: ok {rc=0}
03:41:20.277275 --- ParallelTerminationStrategy: PID 19823 died...
03:41:20.331354 run iperf3-srv_10.42.42.10(pid=19823): Terminated {rc=256}
03:41:20.341091 --- ParallelTerminationStrategy: PID 19826 died...
03:41:20.395110 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19826): Terminated: ok {rc=0}
03:41:20.404712 --- ParallelTerminationStrategy: PID 19829 died...
03:41:20.458906 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19829): Terminated: ok {rc=0}
03:41:20.468649 --- ParallelTerminationStrategy: PID 19832 died...
03:41:20.522707 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19832): Terminated: ok {rc=0}
03:41:20.612580 bus /gobi_6: Setting Powered False
42gprs:sysmo11.0osmo-stp_10.42.42.5(pid=19848): Process ended prematurely: osmo-stp_10.42.42.5(pid=19848) [trial-2354↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19848)]2313iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=19848): Process ended prematurely: osmo-stp_10.42.42.5(pid=19848) [trial-2354↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19848)]
----------------------------------------------
trial-2354 gprs:sysmo iperf3m4.py
----------------------------------------------
03:41:21.727693 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:21.739028 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:41:21.862867 tst iperf3m4.py: using LAC 18820
03:41:21.986707 tst iperf3m4.py: using RAC 205
03:41:22.110072 tst iperf3m4.py: using CellId 18820
03:41:22.233360 tst iperf3m4.py: using BVCI 18821
03:41:22.257063 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:22.267593 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:22.278204 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:22.289154 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:22.299876 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:22.310688 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:22.321769 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:22.332596 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:22.343579 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:22.466602 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:22.590377 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:22.711059 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:22.993089 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
03:41:23.122240 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:23.445601 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19834): Launched
03:41:23.715024 run iperf3-srv_10.42.42.10(pid=19835): Launched
03:41:23.903419 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
03:41:24.032397 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:24.355342 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19836): Launched
03:41:24.625319 run iperf3-srv_10.42.42.10(pid=19837): Launched
03:41:24.813296 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
03:41:24.941488 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:25.263258 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19838): Launched
03:41:25.531075 run iperf3-srv_10.42.42.10(pid=19839): Launched
03:41:25.718992 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
03:41:25.847356 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:26.169433 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19840): Launched
03:41:26.452650 run iperf3-srv_10.42.42.10(pid=19841): Launched
03:41:26.641708 tst iperf3m4.py:8: start network...
03:41:26.770257 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:41:27.258055 run create_hlr_db(pid=19842): Launched
03:41:27.434576 bus /gobi_6: Setting Powered False
03:41:28.457191 run create_hlr_db(pid=19842): Terminated: ok {rc=0}
03:41:28.898987 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19844): Launched
03:41:29.232741 run osmo-hlr_10.42.42.2(pid=19845): Launched
03:41:29.356241 run osmo-stp_10.42.42.5: Starting osmo-stp
03:41:29.933302 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19847): Launched
03:41:30.268392 run osmo-stp_10.42.42.5(pid=19848): Launched
03:41:30.390006 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:41:30.982504 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19850): Launched
03:41:31.115083 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:41:31.441744 run patchelf(pid=19851): Launched
03:41:31.642767 run osmo-stp_10.42.42.5(pid=19848): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19848)]
03:41:31.783804 run osmo-stp_10.42.42.5(pid=19848): stdout:
| (launched: 2022-12-09_03:41:30.101485)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:31.923713 run osmo-stp_10.42.42.5(pid=19848): stderr:
| �[0;m�[38;5;43m20221209034130298 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034130298 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034130299 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034130299 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034130299 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034130299 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034130299 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:32.055936 run osmo-stp_10.42.42.5(pid=19848): stdout:
| (launched: 2022-12-09_03:41:30.101485)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:32.190583 run osmo-stp_10.42.42.5(pid=19848): stderr:
| �[0;m�[38;5;43m20221209034130298 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034130298 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034130299 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034130299 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034130299 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034130299 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034130299 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:32.350800 run patchelf(pid=19851): Terminating (SIGINT)
03:41:32.538405 run patchelf(pid=19851): Terminated: ok {rc=0}
03:41:32.572749 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=19848): Process ended prematurely: osmo-stp_10.42.42.5(pid=19848) [trial-2354↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19848)] [trial-2354↪gprs:sysmo↪iperf3m4.py:8]
03:41:32.583524 tst iperf3m4.py:8: Test FAILED (10.9 sec)
03:41:32.649996 run osmo-hlr_10.42.42.2(pid=19845): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:sysmo↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=19845)]
03:41:32.703626 run osmo-hlr_10.42.42.2(pid=19845): stdout:
| (launched: 2022-12-09_03:41:29.067766)
03:41:32.757680 run osmo-hlr_10.42.42.2(pid=19845): stderr:
| �[0;m�[1;31m20221209034129271 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209034129271 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209034129271 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209034129271 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209034129271 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209034129271 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209034129283 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:sysmo/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209034129292 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209034129292 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209034129292 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:41:32.803761 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19834): Terminating (SIGTERM)
03:41:32.846866 run iperf3-srv_10.42.42.10(pid=19835): Terminating (SIGTERM)
03:41:32.890673 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19836): Terminating (SIGTERM)
03:41:32.934140 run iperf3-srv_10.42.42.10(pid=19837): Terminating (SIGTERM)
03:41:32.977473 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19838): Terminating (SIGTERM)
03:41:33.021308 run iperf3-srv_10.42.42.10(pid=19839): Terminating (SIGTERM)
03:41:33.064579 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19840): Terminating (SIGTERM)
03:41:33.107856 run iperf3-srv_10.42.42.10(pid=19841): Terminating (SIGTERM)
03:41:33.151369 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19844): Terminating (SIGTERM)
03:41:33.195100 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19847): Terminating (SIGTERM)
03:41:33.238277 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19850): Terminating (SIGTERM)
03:41:33.247916 --- ParallelTerminationStrategy: PID 19834 died...
03:41:33.305872 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19834): Terminated: ok {rc=0}
03:41:33.316278 --- ParallelTerminationStrategy: PID 19835 died...
03:41:33.373825 run iperf3-srv_10.42.42.10(pid=19835): Terminated {rc=256}
03:41:33.384223 --- ParallelTerminationStrategy: PID 19836 died...
03:41:33.441924 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19836): Terminated: ok {rc=0}
03:41:33.452346 --- ParallelTerminationStrategy: PID 19837 died...
03:41:33.510444 run iperf3-srv_10.42.42.10(pid=19837): Terminated {rc=256}
03:41:33.520942 --- ParallelTerminationStrategy: PID 19838 died...
03:41:33.578710 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19838): Terminated: ok {rc=0}
03:41:33.589297 --- ParallelTerminationStrategy: PID 19839 died...
03:41:33.646864 run iperf3-srv_10.42.42.10(pid=19839): Terminated {rc=256}
03:41:33.657294 --- ParallelTerminationStrategy: PID 19840 died...
03:41:33.714906 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=19840): Terminated: ok {rc=0}
03:41:33.725449 --- ParallelTerminationStrategy: PID 19841 died...
03:41:33.783176 run iperf3-srv_10.42.42.10(pid=19841): Terminated {rc=256}
03:41:33.793960 --- ParallelTerminationStrategy: PID 19844 died...
03:41:33.851567 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19844): Terminated: ok {rc=0}
03:41:33.861875 --- ParallelTerminationStrategy: PID 19847 died...
03:41:33.919167 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19847): Terminated: ok {rc=0}
03:41:33.930024 --- ParallelTerminationStrategy: PID 19850 died...
03:41:33.987704 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19850): Terminated: ok {rc=0}
03:41:34.077990 bus /gobi_4: Setting Powered False
03:41:35.187784 bus /gobi_4: Setting Powered False
03:41:36.298051 bus /gobi_1: Setting Powered False
03:41:37.406339 bus /gobi_1: Setting Powered False
03:41:38.515618 bus /gobi_0: Setting Powered False
03:41:39.625602 bus /gobi_0: Setting Powered False
03:41:40.741749 bus /gobi_6: Setting Powered False
24gprs:sysmo7.0osmo-stp_10.42.42.5(pid=19858): Process ended prematurely: osmo-stp_10.42.42.5(pid=19858) [trial-2354↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19858)]2331ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=19858): Process ended prematurely: osmo-stp_10.42.42.5(pid=19858) [trial-2354↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19858)]
----------------------------------------------
trial-2354 gprs:sysmo ping.py
----------------------------------------------
03:41:41.811785 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:41.823645 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:41:41.954622 tst ping.py: using LAC 18821
03:41:42.082447 tst ping.py: using RAC 206
03:41:42.209957 tst ping.py: using CellId 18821
03:41:42.338818 tst ping.py: using BVCI 18822
03:41:42.363955 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:42.375165 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:42.386229 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:42.397437 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:42.408356 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:42.419449 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:42.430880 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:42.442273 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:42.736210 tst ping.py:19: start network...
03:41:42.872043 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:41:43.385900 run create_hlr_db(pid=19852): Launched
03:41:43.574510 bus /gobi_6: Setting Powered False
03:41:44.595997 run create_hlr_db(pid=19852): Terminated: ok {rc=0}
03:41:45.007242 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19854): Launched
03:41:45.313761 run osmo-hlr_10.42.42.2(pid=19855): Launched
03:41:45.429803 run osmo-stp_10.42.42.5: Starting osmo-stp
03:41:46.005101 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19857): Launched
03:41:46.429457 run osmo-stp_10.42.42.5(pid=19858): Launched
03:41:46.594865 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:41:47.263880 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19860): Launched
03:41:47.384260 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:41:47.671509 run patchelf(pid=19861): Launched
03:41:47.862281 run osmo-stp_10.42.42.5(pid=19858): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19858)]
03:41:48.002207 run osmo-stp_10.42.42.5(pid=19858): stdout:
| (launched: 2022-12-09_03:41:46.176010)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:48.142427 run osmo-stp_10.42.42.5(pid=19858): stderr:
| �[0;m�[38;5;43m20221209034146439 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034146439 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034146440 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034146440 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034146441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034146441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034146441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:48.277582 run osmo-stp_10.42.42.5(pid=19858): stdout:
| (launched: 2022-12-09_03:41:46.176010)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:48.411355 run osmo-stp_10.42.42.5(pid=19858): stderr:
| �[0;m�[38;5;43m20221209034146439 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034146439 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034146440 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034146440 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034146441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034146441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034146441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:48.567748 run patchelf(pid=19861): Terminating (SIGINT)
03:41:48.751158 run patchelf(pid=19861): Terminated: ok {rc=0}
03:41:48.784042 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=19858): Process ended prematurely: osmo-stp_10.42.42.5(pid=19858) [trial-2354↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19858)] [trial-2354↪gprs:sysmo↪ping.py:22]
03:41:48.793264 tst ping.py:22: Test FAILED (7.0 sec)
03:41:48.858509 run osmo-hlr_10.42.42.2(pid=19855): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:sysmo↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=19855)]
03:41:48.911696 run osmo-hlr_10.42.42.2(pid=19855): stdout:
| (launched: 2022-12-09_03:41:45.161579)
03:41:48.965534 run osmo-hlr_10.42.42.2(pid=19855): stderr:
| �[0;m�[1;31m20221209034145340 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209034145340 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209034145340 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209034145340 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209034145340 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209034145340 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209034145352 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:sysmo/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209034145361 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209034145361 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209034145361 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:41:49.010835 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19854): Terminating (SIGTERM)
03:41:49.054508 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19857): Terminating (SIGTERM)
03:41:49.097597 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19860): Terminating (SIGTERM)
03:41:49.107730 --- ParallelTerminationStrategy: PID 19854 died...
03:41:49.165695 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19854): Terminated: ok {rc=0}
03:41:49.176148 --- ParallelTerminationStrategy: PID 19857 died...
03:41:49.233843 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19857): Terminated: ok {rc=0}
03:41:49.244285 --- ParallelTerminationStrategy: PID 19860 died...
03:41:49.301823 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19860): Terminated: ok {rc=0}
03:41:49.391461 bus /gobi_6: Setting Powered False
24gprs:sysmo7.0osmo-stp_10.42.42.5(pid=19868): Process ended prematurely: osmo-stp_10.42.42.5(pid=19868) [trial-2354↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19868)]2331ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=19868): Process ended prematurely: osmo-stp_10.42.42.5(pid=19868) [trial-2354↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19868)]
----------------------------------------------
trial-2354 gprs:sysmo ping_idle_ping.py
----------------------------------------------
03:41:50.444640 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:50.455871 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:41:50.584080 tst ping_idle_ping.py: using LAC 18822
03:41:50.704445 tst ping_idle_ping.py: using RAC 207
03:41:50.822178 tst ping_idle_ping.py: using CellId 18822
03:41:50.940786 tst ping_idle_ping.py: using BVCI 18823
03:41:50.964102 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:50.974656 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:50.985139 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:50.995524 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:51.005890 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:51.016280 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:51.026803 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:51.037348 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:51.339993 tst ping_idle_ping.py:24: start network...
03:41:51.477474 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:41:51.994956 run create_hlr_db(pid=19862): Launched
03:41:52.183924 bus /gobi_6: Setting Powered False
03:41:53.204460 run create_hlr_db(pid=19862): Terminated: ok {rc=0}
03:41:53.615126 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19864): Launched
03:41:53.921632 run osmo-hlr_10.42.42.2(pid=19865): Launched
03:41:54.033416 run osmo-stp_10.42.42.5: Starting osmo-stp
03:41:54.562837 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19867): Launched
03:41:54.869609 run osmo-stp_10.42.42.5(pid=19868): Launched
03:41:54.980068 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:41:55.513901 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19870): Launched
03:41:55.622753 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:41:55.890569 run patchelf(pid=19871): Launched
03:41:56.068291 run osmo-stp_10.42.42.5(pid=19868): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19868)]
03:41:56.198822 run osmo-stp_10.42.42.5(pid=19868): stdout:
| (launched: 2022-12-09_03:41:54.716510)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:56.329193 run osmo-stp_10.42.42.5(pid=19868): stderr:
| �[0;m�[38;5;43m20221209034154891 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034154891 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034154891 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034154891 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034154892 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034154892 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034154892 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:56.455194 run osmo-stp_10.42.42.5(pid=19868): stdout:
| (launched: 2022-12-09_03:41:54.716510)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:56.579940 run osmo-stp_10.42.42.5(pid=19868): stderr:
| �[0;m�[38;5;43m20221209034154891 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209034154891 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209034154891 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209034154891 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209034154892 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209034154892 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209034154892 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:56.726095 run patchelf(pid=19871): Terminating (SIGINT)
03:41:56.898696 run patchelf(pid=19871): Terminated: ok {rc=0}
03:41:56.929867 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=19868): Process ended prematurely: osmo-stp_10.42.42.5(pid=19868) [trial-2354↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=19868)] [trial-2354↪gprs:sysmo↪ping_idle_ping.py:27]
03:41:56.938456 tst ping_idle_ping.py:27: Test FAILED (6.5 sec)
03:41:56.998812 run osmo-hlr_10.42.42.2(pid=19865): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:sysmo↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=19865)]
03:41:57.048253 run osmo-hlr_10.42.42.2(pid=19865): stdout:
| (launched: 2022-12-09_03:41:53.769281)
03:41:57.098550 run osmo-hlr_10.42.42.2(pid=19865): stderr:
| �[0;m�[1;31m20221209034153947 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209034153947 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209034153947 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209034153947 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209034153947 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209034153948 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209034153960 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:sysmo/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209034153969 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209034153969 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209034153969 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:41:57.141124 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19864): Terminating (SIGTERM)
03:41:57.181391 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19867): Terminating (SIGTERM)
03:41:57.221336 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19870): Terminating (SIGTERM)
03:41:57.230562 --- ParallelTerminationStrategy: PID 19864 died...
03:41:57.283858 run pcap-recorder_any(filters='host 10.42.42.2')(pid=19864): Terminated: ok {rc=0}
03:41:57.293427 --- ParallelTerminationStrategy: PID 19867 died...
03:41:57.346903 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=19867): Terminated: ok {rc=0}
03:41:57.356511 --- ParallelTerminationStrategy: PID 19870 died...
03:41:57.409783 run pcap-recorder_any(filters='host 10.42.42.6')(pid=19870): Terminated: ok {rc=0}
03:41:57.499996 bus /gobi_6: Setting Powered False
5gprs:sysmo00