86gprs:sysmo10.0osmo-stp_10.42.42.5(pid=4516): Process ended prematurely: osmo-stp_10.42.42.5(pid=4516) [trial-2399↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4516)]2314cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=4516): Process ended prematurely: osmo-stp_10.42.42.5(pid=4516) [trial-2399↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4516)]
----------------------------------------------
trial-2399 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
03:40:41.801425 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:41.812362 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:40:41.944589 tst cs_paging_gprs_active.py: using LAC 38294
03:40:42.066692 tst cs_paging_gprs_active.py: using RAC 44
03:40:42.191258 tst cs_paging_gprs_active.py: using CellId 38294
03:40:42.308919 tst cs_paging_gprs_active.py: using BVCI 38295
03:40:42.332252 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:42.342788 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:42.353136 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:42.363368 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:42.373576 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:42.383776 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:42.394160 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:42.404380 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:42.415047 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:40:42.537795 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:40:42.820673 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
03:40:42.949164 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:40:43.267495 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4506): Launched
03:40:43.532505 run iperf3-srv_10.42.42.10(pid=4507): Launched
03:40:43.718698 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
03:40:43.845524 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:40:44.163570 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4508): Launched
03:40:44.428686 run iperf3-srv_10.42.42.10(pid=4509): Launched
03:40:44.614179 tst cs_paging_gprs_active.py:38: start network...
03:40:44.740711 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:40:45.236673 run create_hlr_db(pid=4510): Launched
03:40:45.412063 bus /sierra_2: Setting Powered False
03:40:46.455368 run create_hlr_db(pid=4510): Terminated: ok {rc=0}
03:40:46.967867 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4512): Launched
03:40:47.342234 run osmo-hlr_10.42.42.2(pid=4513): Launched
03:40:47.466927 run osmo-stp_10.42.42.5: Starting osmo-stp
03:40:48.043910 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4515): Launched
03:40:48.376434 run osmo-stp_10.42.42.5(pid=4516): Launched
03:40:48.499256 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:40:50.073744 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4518): Launched
03:40:50.191024 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:40:50.480257 run patchelf(pid=4519): Launched
03:40:50.672713 run osmo-stp_10.42.42.5(pid=4516): ERR: Terminated: ERROR {rc=1} [trial-2399↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4516)]
03:40:50.813408 run osmo-stp_10.42.42.5(pid=4516): stdout:
| (launched: 2022-12-14_03:40:48.211064)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:40:50.954620 run osmo-stp_10.42.42.5(pid=4516): stderr:
| �[0;m�[38;5;43m20221214034048407 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034048407 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034048408 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034048408 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034048409 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034048409 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034048409 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:40:51.090908 run osmo-stp_10.42.42.5(pid=4516): stdout:
| (launched: 2022-12-14_03:40:48.211064)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:40:51.226087 run osmo-stp_10.42.42.5(pid=4516): stderr:
| �[0;m�[38;5;43m20221214034048407 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034048407 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034048408 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034048408 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034048409 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034048409 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034048409 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:40:51.383548 run patchelf(pid=4519): Terminating (SIGINT)
03:40:51.567551 run patchelf(pid=4519): Terminated: ok {rc=0}
03:40:51.601251 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=4516): Process ended prematurely: osmo-stp_10.42.42.5(pid=4516) [trial-2399↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4516)] [trial-2399↪gprs:sysmo↪cs_paging_gprs_active.py:38]
03:40:51.611008 tst cs_paging_gprs_active.py:38: Test FAILED (9.8 sec)
03:40:51.674566 run osmo-hlr_10.42.42.2(pid=4513): ERR: Terminated: ERROR {rc=237} [trial-2399↪gprs:sysmo↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=4513)]
03:40:51.725729 run osmo-hlr_10.42.42.2(pid=4513): stdout:
| (launched: 2022-12-14_03:40:47.165190)
03:40:51.777386 run osmo-hlr_10.42.42.2(pid=4513): stderr:
| �[0;m�[1;31m20221214034047380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214034047380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214034047380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214034047380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214034047380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214034047380 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214034047392 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2399/run.2022-12-14_03-34-50/gprs:sysmo/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214034047400 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214034047400 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214034047400 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:40:51.822091 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4506): Terminating (SIGTERM)
03:40:51.864809 run iperf3-srv_10.42.42.10(pid=4507): Terminating (SIGTERM)
03:40:51.908804 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4508): Terminating (SIGTERM)
03:40:51.953644 run iperf3-srv_10.42.42.10(pid=4509): Terminating (SIGTERM)
03:40:51.994077 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4512): Terminating (SIGTERM)
03:40:52.037534 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4515): Terminating (SIGTERM)
03:40:52.079808 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4518): Terminating (SIGTERM)
03:40:52.089116 --- ParallelTerminationStrategy: PID 4506 died...
03:40:52.146094 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4506): Terminated: ok {rc=0}
03:40:52.156305 --- ParallelTerminationStrategy: PID 4507 died...
03:40:52.212979 run iperf3-srv_10.42.42.10(pid=4507): Terminated {rc=256}
03:40:52.223050 --- ParallelTerminationStrategy: PID 4508 died...
03:40:52.279913 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4508): Terminated: ok {rc=0}
03:40:52.290207 --- ParallelTerminationStrategy: PID 4509 died...
03:40:52.346863 run iperf3-srv_10.42.42.10(pid=4509): Terminated {rc=256}
03:40:52.357045 --- ParallelTerminationStrategy: PID 4512 died...
03:40:52.413623 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4512): Terminated: ok {rc=0}
03:40:52.423780 --- ParallelTerminationStrategy: PID 4515 died...
03:40:52.480679 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4515): Terminated: ok {rc=0}
03:40:52.490984 --- ParallelTerminationStrategy: PID 4518 died...
03:40:52.547958 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4518): Terminated: ok {rc=0}
03:40:52.635475 bus /gobi_0: Setting Powered False
03:40:53.742278 bus /gobi_0: Setting Powered False
03:40:54.851708 bus /gobi_6: Setting Powered False
86gprs:sysmo8.0osmo-stp_10.42.42.5(pid=4529): Process ended prematurely: osmo-stp_10.42.42.5(pid=4529) [trial-2399↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4529)]2314iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=4529): Process ended prematurely: osmo-stp_10.42.42.5(pid=4529) [trial-2399↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4529)]
----------------------------------------------
trial-2399 gprs:sysmo iperf3.py
----------------------------------------------
03:40:55.968784 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:55.980558 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:40:56.107170 tst iperf3.py: using LAC 38295
03:40:56.229413 tst iperf3.py: using RAC 45
03:40:56.357442 tst iperf3.py: using CellId 38295
03:40:56.487332 tst iperf3.py: using BVCI 38296
03:40:56.512649 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:56.523816 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:56.534901 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:56.545839 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:56.556803 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:56.567807 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:56.579017 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:56.590189 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:40:56.601385 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:40:56.883589 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
03:40:57.011292 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:40:57.329449 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4521): Launched
03:40:57.594445 run iperf3-srv_10.42.42.10(pid=4522): Launched
03:40:57.782143 tst iperf3.py:8: start network...
03:40:57.908323 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:40:58.403109 run create_hlr_db(pid=4523): Launched
03:40:58.577892 bus /gobi_6: Setting Powered False
03:40:59.600108 run create_hlr_db(pid=4523): Terminated: ok {rc=0}
03:41:00.008171 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4525): Launched
03:41:00.315673 run osmo-hlr_10.42.42.2(pid=4526): Launched
03:41:00.426259 run osmo-stp_10.42.42.5: Starting osmo-stp
03:41:01.001174 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4528): Launched
03:41:01.408652 run osmo-stp_10.42.42.5(pid=4529): Launched
03:41:01.623395 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:41:02.328795 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4531): Launched
03:41:02.449191 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:41:02.737952 run patchelf(pid=4532): Launched
03:41:02.931181 run osmo-stp_10.42.42.5(pid=4529): ERR: Terminated: ERROR {rc=1} [trial-2399↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4529)]
03:41:03.073111 run osmo-stp_10.42.42.5(pid=4529): stdout:
| (launched: 2022-12-14_03:41:01.175606)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:03.213891 run osmo-stp_10.42.42.5(pid=4529): stderr:
| �[0;m�[38;5;43m20221214034101508 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034101509 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034101509 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034101509 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034101511 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034101512 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034101512 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:03.350476 run osmo-stp_10.42.42.5(pid=4529): stdout:
| (launched: 2022-12-14_03:41:01.175606)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:03.485406 run osmo-stp_10.42.42.5(pid=4529): stderr:
| �[0;m�[38;5;43m20221214034101508 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034101509 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034101509 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034101509 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034101511 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034101512 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034101512 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:03.642113 run patchelf(pid=4532): Terminating (SIGINT)
03:41:03.825395 run patchelf(pid=4532): Terminated: ok {rc=0}
03:41:03.859120 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=4529): Process ended prematurely: osmo-stp_10.42.42.5(pid=4529) [trial-2399↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4529)] [trial-2399↪gprs:sysmo↪iperf3.py:8]
03:41:03.868770 tst iperf3.py:8: Test FAILED (7.9 sec)
03:41:03.932563 run osmo-hlr_10.42.42.2(pid=4526): ERR: Terminated: ERROR {rc=237} [trial-2399↪gprs:sysmo↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=4526)]
03:41:03.984357 run osmo-hlr_10.42.42.2(pid=4526): stdout:
| (launched: 2022-12-14_03:41:00.163417)
03:41:04.037084 run osmo-hlr_10.42.42.2(pid=4526): stderr:
| �[0;m�[1;31m20221214034100345 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214034100345 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214034100345 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214034100345 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214034100345 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214034100346 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214034100377 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2399/run.2022-12-14_03-34-50/gprs:sysmo/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214034100387 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214034100387 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214034100387 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:41:04.081960 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4521): Terminating (SIGTERM)
03:41:04.124583 run iperf3-srv_10.42.42.10(pid=4522): Terminating (SIGTERM)
03:41:04.166898 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4525): Terminating (SIGTERM)
03:41:04.209026 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4528): Terminating (SIGTERM)
03:41:04.251475 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4531): Terminating (SIGTERM)
03:41:04.261203 --- ParallelTerminationStrategy: PID 4521 died...
03:41:04.317983 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4521): Terminated: ok {rc=0}
03:41:04.327939 --- ParallelTerminationStrategy: PID 4522 died...
03:41:04.384539 run iperf3-srv_10.42.42.10(pid=4522): Terminated {rc=256}
03:41:04.394640 --- ParallelTerminationStrategy: PID 4525 died...
03:41:04.451093 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4525): Terminated: ok {rc=0}
03:41:04.461218 --- ParallelTerminationStrategy: PID 4528 died...
03:41:04.517754 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4528): Terminated: ok {rc=0}
03:41:04.527862 --- ParallelTerminationStrategy: PID 4531 died...
03:41:04.584289 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4531): Terminated: ok {rc=0}
03:41:04.671229 bus /gobi_6: Setting Powered False
87gprs:sysmo11.0osmo-stp_10.42.42.5(pid=4547): Process ended prematurely: osmo-stp_10.42.42.5(pid=4547) [trial-2399↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4547)]2313iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=4547): Process ended prematurely: osmo-stp_10.42.42.5(pid=4547) [trial-2399↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4547)]
----------------------------------------------
trial-2399 gprs:sysmo iperf3m4.py
----------------------------------------------
03:41:05.788242 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:05.800196 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:41:05.924129 tst iperf3m4.py: using LAC 38296
03:41:06.045282 tst iperf3m4.py: using RAC 46
03:41:06.173475 tst iperf3m4.py: using CellId 38296
03:41:06.304741 tst iperf3m4.py: using BVCI 38297
03:41:06.330213 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:06.341547 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:06.352602 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:06.363819 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:06.375074 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:06.386772 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:06.398493 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:06.410037 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:06.421803 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:06.543733 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:06.666505 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:06.785046 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:07.083032 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
03:41:07.220941 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:07.564304 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4533): Launched
03:41:07.850341 run iperf3-srv_10.42.42.10(pid=4534): Launched
03:41:08.052613 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
03:41:08.190875 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:08.534157 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4535): Launched
03:41:08.819365 run iperf3-srv_10.42.42.10(pid=4536): Launched
03:41:09.021545 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
03:41:09.159409 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:09.502588 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4537): Launched
03:41:09.787395 run iperf3-srv_10.42.42.10(pid=4538): Launched
03:41:09.988587 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
03:41:10.125975 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:41:10.468637 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4539): Launched
03:41:10.764432 run iperf3-srv_10.42.42.10(pid=4540): Launched
03:41:10.950700 tst iperf3m4.py:8: start network...
03:41:11.078422 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:41:11.575261 run create_hlr_db(pid=4541): Launched
03:41:11.750797 bus /gobi_6: Setting Powered False
03:41:12.772772 run create_hlr_db(pid=4541): Terminated: ok {rc=0}
03:41:13.182620 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4543): Launched
03:41:13.490637 run osmo-hlr_10.42.42.2(pid=4544): Launched
03:41:13.607859 run osmo-stp_10.42.42.5: Starting osmo-stp
03:41:14.150014 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4546): Launched
03:41:14.457764 run osmo-stp_10.42.42.5(pid=4547): Launched
03:41:14.567830 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:41:15.101384 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4549): Launched
03:41:15.209952 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:41:15.478365 run patchelf(pid=4550): Launched
03:41:15.657411 run osmo-stp_10.42.42.5(pid=4547): ERR: Terminated: ERROR {rc=1} [trial-2399↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4547)]
03:41:15.787935 run osmo-stp_10.42.42.5(pid=4547): stdout:
| (launched: 2022-12-14_03:41:14.304491)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:15.953977 run osmo-stp_10.42.42.5(pid=4547): stderr:
| �[0;m�[38;5;43m20221214034114479 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034114480 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034114480 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034114480 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034114481 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034114481 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034114481 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:16.096016 run osmo-stp_10.42.42.5(pid=4547): stdout:
| (launched: 2022-12-14_03:41:14.304491)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:16.236133 run osmo-stp_10.42.42.5(pid=4547): stderr:
| �[0;m�[38;5;43m20221214034114479 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034114480 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034114480 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034114480 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034114481 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034114481 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034114481 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:16.399573 run patchelf(pid=4550): Terminating (SIGINT)
03:41:16.602868 run patchelf(pid=4550): Terminated: ok {rc=0}
03:41:16.639126 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=4547): Process ended prematurely: osmo-stp_10.42.42.5(pid=4547) [trial-2399↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4547)] [trial-2399↪gprs:sysmo↪iperf3m4.py:8]
03:41:16.650584 tst iperf3m4.py:8: Test FAILED (10.9 sec)
03:41:16.719967 run osmo-hlr_10.42.42.2(pid=4544): ERR: Terminated: ERROR {rc=237} [trial-2399↪gprs:sysmo↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=4544)]
03:41:16.775422 run osmo-hlr_10.42.42.2(pid=4544): stdout:
| (launched: 2022-12-14_03:41:13.338013)
03:41:16.831074 run osmo-hlr_10.42.42.2(pid=4544): stderr:
| �[0;m�[1;31m20221214034113520 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214034113520 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214034113520 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214034113520 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214034113520 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214034113520 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214034113533 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2399/run.2022-12-14_03-34-50/gprs:sysmo/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214034113542 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214034113542 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214034113542 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:41:16.879637 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4533): Terminating (SIGTERM)
03:41:16.926158 run iperf3-srv_10.42.42.10(pid=4534): Terminating (SIGTERM)
03:41:16.970617 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4535): Terminating (SIGTERM)
03:41:17.014901 run iperf3-srv_10.42.42.10(pid=4536): Terminating (SIGTERM)
03:41:17.059133 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4537): Terminating (SIGTERM)
03:41:17.104279 run iperf3-srv_10.42.42.10(pid=4538): Terminating (SIGTERM)
03:41:17.149423 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4539): Terminating (SIGTERM)
03:41:17.193984 run iperf3-srv_10.42.42.10(pid=4540): Terminating (SIGTERM)
03:41:17.236868 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4543): Terminating (SIGTERM)
03:41:17.279944 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4546): Terminating (SIGTERM)
03:41:17.322769 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4549): Terminating (SIGTERM)
03:41:17.332252 --- ParallelTerminationStrategy: PID 4533 died...
03:41:17.390828 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4533): Terminated: ok {rc=0}
03:41:17.401210 --- ParallelTerminationStrategy: PID 4534 died...
03:41:17.459276 run iperf3-srv_10.42.42.10(pid=4534): Terminated {rc=256}
03:41:17.469609 --- ParallelTerminationStrategy: PID 4535 died...
03:41:17.527813 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4535): Terminated: ok {rc=0}
03:41:17.538171 --- ParallelTerminationStrategy: PID 4536 died...
03:41:17.596216 run iperf3-srv_10.42.42.10(pid=4536): Terminated {rc=256}
03:41:17.606464 --- ParallelTerminationStrategy: PID 4537 died...
03:41:17.664219 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4537): Terminated: ok {rc=0}
03:41:17.674520 --- ParallelTerminationStrategy: PID 4538 died...
03:41:17.731896 run iperf3-srv_10.42.42.10(pid=4538): Terminated {rc=256}
03:41:17.741983 --- ParallelTerminationStrategy: PID 4539 died...
03:41:17.799443 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=4539): Terminated: ok {rc=0}
03:41:17.809958 --- ParallelTerminationStrategy: PID 4540 died...
03:41:17.867222 run iperf3-srv_10.42.42.10(pid=4540): Terminated {rc=256}
03:41:17.877341 --- ParallelTerminationStrategy: PID 4543 died...
03:41:17.934913 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4543): Terminated: ok {rc=0}
03:41:17.945138 --- ParallelTerminationStrategy: PID 4546 died...
03:41:18.003117 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4546): Terminated: ok {rc=0}
03:41:18.013404 --- ParallelTerminationStrategy: PID 4549 died...
03:41:18.071256 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4549): Terminated: ok {rc=0}
03:41:18.158551 bus /gobi_4: Setting Powered False
03:41:19.265830 bus /gobi_4: Setting Powered False
03:41:20.375334 bus /gobi_1: Setting Powered False
03:41:21.481928 bus /gobi_1: Setting Powered False
03:41:22.590035 bus /gobi_0: Setting Powered False
03:41:23.702453 bus /gobi_0: Setting Powered False
03:41:24.816161 bus /gobi_6: Setting Powered False
69gprs:sysmo7.0osmo-stp_10.42.42.5(pid=4557): Process ended prematurely: osmo-stp_10.42.42.5(pid=4557) [trial-2399↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4557)]2331ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=4557): Process ended prematurely: osmo-stp_10.42.42.5(pid=4557) [trial-2399↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4557)]
----------------------------------------------
trial-2399 gprs:sysmo ping.py
----------------------------------------------
03:41:25.883713 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:25.895152 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:41:26.076534 tst ping.py: using LAC 38297
03:41:26.198578 tst ping.py: using RAC 47
03:41:26.315618 tst ping.py: using CellId 38297
03:41:26.437216 tst ping.py: using BVCI 38298
03:41:26.460212 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:26.470908 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:26.481503 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:26.491971 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:26.502621 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:26.513422 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:26.524091 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:26.534695 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:26.811417 tst ping.py:19: start network...
03:41:26.936247 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:41:27.422233 run create_hlr_db(pid=4551): Launched
03:41:27.595592 bus /gobi_6: Setting Powered False
03:41:28.616812 run create_hlr_db(pid=4551): Terminated: ok {rc=0}
03:41:29.048800 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4553): Launched
03:41:29.374833 run osmo-hlr_10.42.42.2(pid=4554): Launched
03:41:29.494516 run osmo-stp_10.42.42.5: Starting osmo-stp
03:41:30.050218 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4556): Launched
03:41:30.351334 run osmo-stp_10.42.42.5(pid=4557): Launched
03:41:30.459021 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:41:31.052015 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4559): Launched
03:41:31.191211 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:41:31.511912 run patchelf(pid=4560): Launched
03:41:31.691371 run osmo-stp_10.42.42.5(pid=4557): ERR: Terminated: ERROR {rc=1} [trial-2399↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4557)]
03:41:31.822810 run osmo-stp_10.42.42.5(pid=4557): stdout:
| (launched: 2022-12-14_03:41:30.202460)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:31.955098 run osmo-stp_10.42.42.5(pid=4557): stderr:
| �[0;m�[38;5;43m20221214034130375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034130375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034130375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034130376 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034130378 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034130378 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034130378 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:32.082066 run osmo-stp_10.42.42.5(pid=4557): stdout:
| (launched: 2022-12-14_03:41:30.202460)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:32.207488 run osmo-stp_10.42.42.5(pid=4557): stderr:
| �[0;m�[38;5;43m20221214034130375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034130375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034130375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034130376 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034130378 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034130378 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034130378 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:32.353226 run patchelf(pid=4560): Terminating (SIGINT)
03:41:32.522238 run patchelf(pid=4560): Terminated: ok {rc=0}
03:41:32.553115 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=4557): Process ended prematurely: osmo-stp_10.42.42.5(pid=4557) [trial-2399↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4557)] [trial-2399↪gprs:sysmo↪ping.py:22]
03:41:32.562033 tst ping.py:22: Test FAILED (6.7 sec)
03:41:32.623069 run osmo-hlr_10.42.42.2(pid=4554): ERR: Terminated: ERROR {rc=237} [trial-2399↪gprs:sysmo↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=4554)]
03:41:32.673014 run osmo-hlr_10.42.42.2(pid=4554): stdout:
| (launched: 2022-12-14_03:41:29.213483)
03:41:32.723307 run osmo-hlr_10.42.42.2(pid=4554): stderr:
| �[0;m�[1;31m20221214034129412 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214034129412 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214034129412 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214034129412 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214034129412 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214034129412 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214034129426 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2399/run.2022-12-14_03-34-50/gprs:sysmo/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214034129434 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214034129435 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214034129435 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:41:32.766582 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4553): Terminating (SIGTERM)
03:41:32.807094 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4556): Terminating (SIGTERM)
03:41:32.847675 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4559): Terminating (SIGTERM)
03:41:32.857154 --- ParallelTerminationStrategy: PID 4553 died...
03:41:32.911932 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4553): Terminated: ok {rc=0}
03:41:32.921871 --- ParallelTerminationStrategy: PID 4556 died...
03:41:32.976147 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4556): Terminated: ok {rc=0}
03:41:32.986062 --- ParallelTerminationStrategy: PID 4559 died...
03:41:33.040334 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4559): Terminated: ok {rc=0}
03:41:33.129662 bus /gobi_6: Setting Powered False
69gprs:sysmo7.0osmo-stp_10.42.42.5(pid=4567): Process ended prematurely: osmo-stp_10.42.42.5(pid=4567) [trial-2399↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4567)]2331ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=4567): Process ended prematurely: osmo-stp_10.42.42.5(pid=4567) [trial-2399↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4567)]
----------------------------------------------
trial-2399 gprs:sysmo ping_idle_ping.py
----------------------------------------------
03:41:34.183108 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:34.194894 tst gprs:sysmo: Using 1 x bts (candidates: 1)
03:41:34.319890 tst ping_idle_ping.py: using LAC 38298
03:41:34.447761 tst ping_idle_ping.py: using RAC 48
03:41:34.574972 tst ping_idle_ping.py: using CellId 38298
03:41:34.702280 tst ping_idle_ping.py: using BVCI 38299
03:41:34.727223 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:34.738245 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:34.749099 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:34.759850 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:34.771009 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:34.782018 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:34.793275 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
03:41:34.804420 tst gprs:sysmo: Using 1 x modem (candidates: 4)
03:41:35.088931 tst ping_idle_ping.py:24: start network...
03:41:35.215059 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:41:35.702103 run create_hlr_db(pid=4561): Launched
03:41:35.875208 bus /gobi_6: Setting Powered False
03:41:36.896171 run create_hlr_db(pid=4561): Terminated: ok {rc=0}
03:41:37.299356 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4563): Launched
03:41:37.601389 run osmo-hlr_10.42.42.2(pid=4564): Launched
03:41:37.710752 run osmo-stp_10.42.42.5: Starting osmo-stp
03:41:38.237389 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4566): Launched
03:41:38.539484 run osmo-stp_10.42.42.5(pid=4567): Launched
03:41:38.647842 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:41:39.177964 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4569): Launched
03:41:39.285941 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:41:39.551203 run patchelf(pid=4570): Launched
03:41:39.727597 run osmo-stp_10.42.42.5(pid=4567): ERR: Terminated: ERROR {rc=1} [trial-2399↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4567)]
03:41:39.856675 run osmo-stp_10.42.42.5(pid=4567): stdout:
| (launched: 2022-12-14_03:41:38.389448)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:39.986306 run osmo-stp_10.42.42.5(pid=4567): stderr:
| �[0;m�[38;5;43m20221214034138564 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034138564 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034138564 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034138565 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034138565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034138565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034138565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:40.111186 run osmo-stp_10.42.42.5(pid=4567): stdout:
| (launched: 2022-12-14_03:41:38.389448)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:41:40.234562 run osmo-stp_10.42.42.5(pid=4567): stderr:
| �[0;m�[38;5;43m20221214034138564 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214034138564 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214034138564 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214034138565 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214034138565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214034138565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214034138565 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:41:40.378475 run patchelf(pid=4570): Terminating (SIGINT)
03:41:40.547121 run patchelf(pid=4570): Terminated: ok {rc=0}
03:41:40.577993 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=4567): Process ended prematurely: osmo-stp_10.42.42.5(pid=4567) [trial-2399↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=4567)] [trial-2399↪gprs:sysmo↪ping_idle_ping.py:27]
03:41:40.586557 tst ping_idle_ping.py:27: Test FAILED (6.4 sec)
03:41:40.645896 run osmo-hlr_10.42.42.2(pid=4564): ERR: Terminated: ERROR {rc=237} [trial-2399↪gprs:sysmo↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=4564)]
03:41:40.694181 run osmo-hlr_10.42.42.2(pid=4564): stdout:
| (launched: 2022-12-14_03:41:37.451642)
03:41:40.743247 run osmo-hlr_10.42.42.2(pid=4564): stderr:
| �[0;m�[1;31m20221214034137627 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214034137627 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214034137628 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214034137628 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214034137628 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214034137628 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214034137639 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2399/run.2022-12-14_03-34-50/gprs:sysmo/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214034137648 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214034137648 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214034137648 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:41:40.784810 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4563): Terminating (SIGTERM)
03:41:40.823886 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4566): Terminating (SIGTERM)
03:41:40.863109 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4569): Terminating (SIGTERM)
03:41:40.872181 --- ParallelTerminationStrategy: PID 4563 died...
03:41:40.924803 run pcap-recorder_any(filters='host 10.42.42.2')(pid=4563): Terminated: ok {rc=0}
03:41:40.934369 --- ParallelTerminationStrategy: PID 4566 died...
03:41:40.986592 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=4566): Terminated: ok {rc=0}
03:41:40.995956 --- ParallelTerminationStrategy: PID 4569 died...
03:41:41.048249 run pcap-recorder_any(filters='host 10.42.42.6')(pid=4569): Terminated: ok {rc=0}
03:41:41.134818 bus /gobi_6: Setting Powered False
5gprs:sysmo00