48gprs:sysmo10.0osmo-stp_10.42.42.5(pid=18550): Process ended prematurely: osmo-stp_10.42.42.5(pid=18550) [trial-2361↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18550)]2314cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18550): Process ended prematurely: osmo-stp_10.42.42.5(pid=18550) [trial-2361↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18550)]
----------------------------------------------
trial-2361 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
15:42:42.112654 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:42.123490 tst gprs:sysmo: Using 1 x bts (candidates: 1)
15:42:42.250973 tst cs_paging_gprs_active.py: using LAC 22045
15:42:42.374875 tst cs_paging_gprs_active.py: using RAC 115
15:42:42.498222 tst cs_paging_gprs_active.py: using CellId 22045
15:42:42.616086 tst cs_paging_gprs_active.py: using BVCI 22046
15:42:42.638766 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:42.649206 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:42.659487 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:42.669914 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:42.680108 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:42.690323 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:42.700568 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:42.710799 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:42.721206 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:42:42.837040 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:42:43.130086 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
15:42:43.264400 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:42:43.596773 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18540): Launched
15:42:43.872318 run iperf3-srv_10.42.42.10(pid=18541): Launched
15:42:44.069146 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
15:42:44.202602 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:42:44.535084 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18542): Launched
15:42:44.810483 run iperf3-srv_10.42.42.10(pid=18543): Launched
15:42:45.006581 tst cs_paging_gprs_active.py:38: start network...
15:42:45.140748 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:42:45.649636 run create_hlr_db(pid=18544): Launched
15:42:45.821446 bus /sierra_2: Setting Powered False
15:42:46.847542 run create_hlr_db(pid=18544): Terminated: ok {rc=0}
15:42:47.277927 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18546): Launched
15:42:47.577487 run osmo-hlr_10.42.42.2(pid=18547): Launched
15:42:47.690740 run osmo-stp_10.42.42.5: Starting osmo-stp
15:42:48.218451 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18549): Launched
15:42:48.515143 run osmo-stp_10.42.42.5(pid=18550): Launched
15:42:48.623975 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:42:50.112751 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18552): Launched
15:42:50.219219 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:42:50.482683 run patchelf(pid=18553): Launched
15:42:50.657411 run osmo-stp_10.42.42.5(pid=18550): ERR: Terminated: ERROR {rc=1} [trial-2361↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18550)]
15:42:50.785202 run osmo-stp_10.42.42.5(pid=18550): stdout:
| (launched: 2022-12-09_15:42:48.369265)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:42:50.913311 run osmo-stp_10.42.42.5(pid=18550): stderr:
| �[0;m�[38;5;43m20221209154248539 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154248539 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154248539 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154248540 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154248542 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154248542 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154248542 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:42:51.036335 run osmo-stp_10.42.42.5(pid=18550): stdout:
| (launched: 2022-12-09_15:42:48.369265)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:42:51.158190 run osmo-stp_10.42.42.5(pid=18550): stderr:
| �[0;m�[38;5;43m20221209154248539 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154248539 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154248539 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154248540 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154248542 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154248542 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154248542 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:42:51.300921 run patchelf(pid=18553): Terminating (SIGINT)
15:42:51.468917 run patchelf(pid=18553): Terminated: ok {rc=0}
15:42:51.499789 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=18550): Process ended prematurely: osmo-stp_10.42.42.5(pid=18550) [trial-2361↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18550)] [trial-2361↪gprs:sysmo↪cs_paging_gprs_active.py:38]
15:42:51.508640 tst cs_paging_gprs_active.py:38: Test FAILED (9.4 sec)
15:42:51.567709 run osmo-hlr_10.42.42.2(pid=18547): ERR: Terminated: ERROR {rc=237} [trial-2361↪gprs:sysmo↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18547)]
15:42:51.615948 run osmo-hlr_10.42.42.2(pid=18547): stdout:
| (launched: 2022-12-09_15:42:47.429690)
15:42:51.664922 run osmo-hlr_10.42.42.2(pid=18547): stderr:
| �[0;m�[1;31m20221209154247607 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209154247607 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209154247607 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209154247607 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209154247607 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209154247607 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209154247620 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2361/run.2022-12-09_15-36-58/gprs:sysmo/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209154247629 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209154247629 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209154247629 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:42:51.706553 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18540): Terminating (SIGTERM)
15:42:51.745677 run iperf3-srv_10.42.42.10(pid=18541): Terminating (SIGTERM)
15:42:51.784761 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18542): Terminating (SIGTERM)
15:42:51.823727 run iperf3-srv_10.42.42.10(pid=18543): Terminating (SIGTERM)
15:42:51.862893 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18546): Terminating (SIGTERM)
15:42:51.902168 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18549): Terminating (SIGTERM)
15:42:51.942048 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18552): Terminating (SIGTERM)
15:42:51.951077 --- ParallelTerminationStrategy: PID 18540 died...
15:42:52.003752 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18540): Terminated: ok {rc=0}
15:42:52.013246 --- ParallelTerminationStrategy: PID 18541 died...
15:42:52.066039 run iperf3-srv_10.42.42.10(pid=18541): Terminated {rc=256}
15:42:52.075451 --- ParallelTerminationStrategy: PID 18542 died...
15:42:52.128006 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18542): Terminated: ok {rc=0}
15:42:52.137779 --- ParallelTerminationStrategy: PID 18543 died...
15:42:52.190204 run iperf3-srv_10.42.42.10(pid=18543): Terminated {rc=256}
15:42:52.199582 --- ParallelTerminationStrategy: PID 18546 died...
15:42:52.251888 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18546): Terminated: ok {rc=0}
15:42:52.261766 --- ParallelTerminationStrategy: PID 18549 died...
15:42:52.314273 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18549): Terminated: ok {rc=0}
15:42:52.323716 --- ParallelTerminationStrategy: PID 18552 died...
15:42:52.375895 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18552): Terminated: ok {rc=0}
15:42:52.464051 bus /gobi_0: Setting Powered False
15:42:53.567300 bus /gobi_0: Setting Powered False
15:42:54.676598 bus /gobi_6: Setting Powered False
48gprs:sysmo8.0osmo-stp_10.42.42.5(pid=18562): Process ended prematurely: osmo-stp_10.42.42.5(pid=18562) [trial-2361↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18562)]2314iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18562): Process ended prematurely: osmo-stp_10.42.42.5(pid=18562) [trial-2361↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18562)]
----------------------------------------------
trial-2361 gprs:sysmo iperf3.py
----------------------------------------------
15:42:55.785956 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:55.797007 tst gprs:sysmo: Using 1 x bts (candidates: 1)
15:42:55.918996 tst iperf3.py: using LAC 22046
15:42:56.035003 tst iperf3.py: using RAC 116
15:42:56.152466 tst iperf3.py: using CellId 22046
15:42:56.271486 tst iperf3.py: using BVCI 22047
15:42:56.294324 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:56.304679 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:56.315031 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:56.325357 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:56.335533 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:56.345782 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:56.356117 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:56.366429 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:42:56.376717 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:42:56.668159 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
15:42:56.802089 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:42:57.135165 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18554): Launched
15:42:57.411459 run iperf3-srv_10.42.42.10(pid=18555): Launched
15:42:57.608186 tst iperf3.py:8: start network...
15:42:57.742018 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:42:58.245348 run create_hlr_db(pid=18556): Launched
15:42:58.431532 bus /gobi_6: Setting Powered False
15:42:59.451058 run create_hlr_db(pid=18556): Terminated: ok {rc=0}
15:42:59.852573 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18558): Launched
15:43:00.151074 run osmo-hlr_10.42.42.2(pid=18559): Launched
15:43:00.259962 run osmo-stp_10.42.42.5: Starting osmo-stp
15:43:00.777960 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18561): Launched
15:43:01.191354 run osmo-stp_10.42.42.5(pid=18562): Launched
15:43:01.353248 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:43:01.997966 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18564): Launched
15:43:02.116293 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:43:02.400592 run patchelf(pid=18565): Launched
15:43:02.588863 run osmo-stp_10.42.42.5(pid=18562): ERR: Terminated: ERROR {rc=1} [trial-2361↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18562)]
15:43:02.726145 run osmo-stp_10.42.42.5(pid=18562): stdout:
| (launched: 2022-12-09_15:43:00.936625)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:43:02.863533 run osmo-stp_10.42.42.5(pid=18562): stderr:
| �[0;m�[38;5;43m20221209154301273 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154301273 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154301273 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154301274 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154301274 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154301274 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154301274 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:43:02.996273 run osmo-stp_10.42.42.5(pid=18562): stdout:
| (launched: 2022-12-09_15:43:00.936625)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:43:03.127926 run osmo-stp_10.42.42.5(pid=18562): stderr:
| �[0;m�[38;5;43m20221209154301273 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154301273 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154301273 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154301274 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154301274 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154301274 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154301274 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:43:03.281021 run patchelf(pid=18565): Terminating (SIGINT)
15:43:03.461489 run patchelf(pid=18565): Terminated: ok {rc=0}
15:43:03.494366 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=18562): Process ended prematurely: osmo-stp_10.42.42.5(pid=18562) [trial-2361↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18562)] [trial-2361↪gprs:sysmo↪iperf3.py:8]
15:43:03.503991 tst iperf3.py:8: Test FAILED (7.7 sec)
15:43:03.567239 run osmo-hlr_10.42.42.2(pid=18559): ERR: Terminated: ERROR {rc=237} [trial-2361↪gprs:sysmo↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18559)]
15:43:03.618744 run osmo-hlr_10.42.42.2(pid=18559): stdout:
| (launched: 2022-12-09_15:43:00.003055)
15:43:03.670994 run osmo-hlr_10.42.42.2(pid=18559): stderr:
| �[0;m�[1;31m20221209154300177 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209154300177 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209154300177 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209154300177 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209154300177 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209154300177 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209154300192 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2361/run.2022-12-09_15-36-58/gprs:sysmo/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209154300202 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209154300203 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209154300203 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:43:03.715462 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18554): Terminating (SIGTERM)
15:43:03.757150 run iperf3-srv_10.42.42.10(pid=18555): Terminating (SIGTERM)
15:43:03.799197 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18558): Terminating (SIGTERM)
15:43:03.841271 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18561): Terminating (SIGTERM)
15:43:03.883466 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18564): Terminating (SIGTERM)
15:43:03.893234 --- ParallelTerminationStrategy: PID 18554 died...
15:43:03.949746 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18554): Terminated: ok {rc=0}
15:43:03.959797 --- ParallelTerminationStrategy: PID 18555 died...
15:43:04.015852 run iperf3-srv_10.42.42.10(pid=18555): Terminated {rc=256}
15:43:04.026040 --- ParallelTerminationStrategy: PID 18558 died...
15:43:04.082256 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18558): Terminated: ok {rc=0}
15:43:04.092445 --- ParallelTerminationStrategy: PID 18561 died...
15:43:04.148661 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18561): Terminated: ok {rc=0}
15:43:04.158778 --- ParallelTerminationStrategy: PID 18564 died...
15:43:04.214869 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18564): Terminated: ok {rc=0}
15:43:04.301342 bus /gobi_6: Setting Powered False
49gprs:sysmo11.0osmo-stp_10.42.42.5(pid=18580): Process ended prematurely: osmo-stp_10.42.42.5(pid=18580) [trial-2361↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18580)]2313iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18580): Process ended prematurely: osmo-stp_10.42.42.5(pid=18580) [trial-2361↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18580)]
----------------------------------------------
trial-2361 gprs:sysmo iperf3m4.py
----------------------------------------------
15:43:05.417140 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:05.428722 tst gprs:sysmo: Using 1 x bts (candidates: 1)
15:43:05.560686 tst iperf3m4.py: using LAC 22047
15:43:05.685578 tst iperf3m4.py: using RAC 117
15:43:05.808080 tst iperf3m4.py: using CellId 22047
15:43:05.931117 tst iperf3m4.py: using BVCI 22048
15:43:05.955617 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:05.966532 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:05.977283 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:05.988201 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:05.999077 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:06.009957 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:06.021043 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:06.031874 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:06.043122 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:06.162336 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:06.279514 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:06.394166 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:06.668210 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
15:43:06.791728 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:07.102926 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18566): Launched
15:43:07.360331 run iperf3-srv_10.42.42.10(pid=18567): Launched
15:43:07.542678 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
15:43:07.666470 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:07.976471 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18568): Launched
15:43:08.234779 run iperf3-srv_10.42.42.10(pid=18569): Launched
15:43:08.417115 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
15:43:08.541072 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:08.852262 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18570): Launched
15:43:09.108554 run iperf3-srv_10.42.42.10(pid=18571): Launched
15:43:09.289707 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
15:43:09.413329 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:43:09.722552 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18572): Launched
15:43:09.995777 run iperf3-srv_10.42.42.10(pid=18573): Launched
15:43:10.177226 tst iperf3m4.py:8: start network...
15:43:10.301616 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:43:10.781349 run create_hlr_db(pid=18574): Launched
15:43:10.951510 bus /gobi_6: Setting Powered False
15:43:11.974072 run create_hlr_db(pid=18574): Terminated: ok {rc=0}
15:43:12.401972 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18576): Launched
15:43:12.724485 run osmo-hlr_10.42.42.2(pid=18577): Launched
15:43:12.842038 run osmo-stp_10.42.42.5: Starting osmo-stp
15:43:13.398236 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18579): Launched
15:43:13.719157 run osmo-stp_10.42.42.5(pid=18580): Launched
15:43:13.837776 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:43:14.398723 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18582): Launched
15:43:14.513520 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:43:14.793307 run patchelf(pid=18583): Launched
15:43:14.981250 run osmo-stp_10.42.42.5(pid=18580): ERR: Terminated: ERROR {rc=1} [trial-2361↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18580)]
15:43:15.118862 run osmo-stp_10.42.42.5(pid=18580): stdout:
| (launched: 2022-12-09_15:43:13.560803)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:43:15.256422 run osmo-stp_10.42.42.5(pid=18580): stderr:
| �[0;m�[38;5;43m20221209154313751 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154313752 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154313752 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154313752 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154313753 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154313753 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154313753 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:43:15.389424 run osmo-stp_10.42.42.5(pid=18580): stdout:
| (launched: 2022-12-09_15:43:13.560803)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:43:15.520562 run osmo-stp_10.42.42.5(pid=18580): stderr:
| �[0;m�[38;5;43m20221209154313751 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154313752 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154313752 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154313752 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154313753 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154313753 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154313753 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:43:15.673615 run patchelf(pid=18583): Terminating (SIGINT)
15:43:15.860049 run patchelf(pid=18583): Terminated: ok {rc=0}
15:43:15.895024 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=18580): Process ended prematurely: osmo-stp_10.42.42.5(pid=18580) [trial-2361↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18580)] [trial-2361↪gprs:sysmo↪iperf3m4.py:8]
15:43:15.910499 tst iperf3m4.py:8: Test FAILED (10.5 sec)
15:43:15.976363 run osmo-hlr_10.42.42.2(pid=18577): ERR: Terminated: ERROR {rc=237} [trial-2361↪gprs:sysmo↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18577)]
15:43:16.029432 run osmo-hlr_10.42.42.2(pid=18577): stdout:
| (launched: 2022-12-09_15:43:12.565267)
15:43:16.083493 run osmo-hlr_10.42.42.2(pid=18577): stderr:
| �[0;m�[1;31m20221209154312764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209154312764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209154312764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209154312764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209154312764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209154312764 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209154312776 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2361/run.2022-12-09_15-36-58/gprs:sysmo/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209154312784 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209154312785 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209154312785 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:43:16.130114 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18566): Terminating (SIGTERM)
15:43:16.175662 run iperf3-srv_10.42.42.10(pid=18567): Terminating (SIGTERM)
15:43:16.221326 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18568): Terminating (SIGTERM)
15:43:16.267002 run iperf3-srv_10.42.42.10(pid=18569): Terminating (SIGTERM)
15:43:16.311523 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18570): Terminating (SIGTERM)
15:43:16.356319 run iperf3-srv_10.42.42.10(pid=18571): Terminating (SIGTERM)
15:43:16.402826 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18572): Terminating (SIGTERM)
15:43:16.448257 run iperf3-srv_10.42.42.10(pid=18573): Terminating (SIGTERM)
15:43:16.494023 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18576): Terminating (SIGTERM)
15:43:16.538855 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18579): Terminating (SIGTERM)
15:43:16.582699 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18582): Terminating (SIGTERM)
15:43:16.592885 --- ParallelTerminationStrategy: PID 18566 died...
15:43:16.651512 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18566): Terminated: ok {rc=0}
15:43:16.662690 --- ParallelTerminationStrategy: PID 18567 died...
15:43:16.721241 run iperf3-srv_10.42.42.10(pid=18567): Terminated {rc=256}
15:43:16.732066 --- ParallelTerminationStrategy: PID 18568 died...
15:43:16.790628 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18568): Terminated: ok {rc=0}
15:43:16.801531 --- ParallelTerminationStrategy: PID 18569 died...
15:43:16.860263 run iperf3-srv_10.42.42.10(pid=18569): Terminated {rc=256}
15:43:16.871052 --- ParallelTerminationStrategy: PID 18570 died...
15:43:16.929800 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18570): Terminated: ok {rc=0}
15:43:16.940686 --- ParallelTerminationStrategy: PID 18571 died...
15:43:16.999129 run iperf3-srv_10.42.42.10(pid=18571): Terminated {rc=256}
15:43:17.010031 --- ParallelTerminationStrategy: PID 18572 died...
15:43:17.068449 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18572): Terminated: ok {rc=0}
15:43:17.079361 --- ParallelTerminationStrategy: PID 18573 died...
15:43:17.137550 run iperf3-srv_10.42.42.10(pid=18573): Terminated {rc=256}
15:43:17.148532 --- ParallelTerminationStrategy: PID 18576 died...
15:43:17.205613 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18576): Terminated: ok {rc=0}
15:43:17.215607 --- ParallelTerminationStrategy: PID 18579 died...
15:43:17.271251 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18579): Terminated: ok {rc=0}
15:43:17.281389 --- ParallelTerminationStrategy: PID 18582 died...
15:43:17.337311 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18582): Terminated: ok {rc=0}
15:43:17.424652 bus /gobi_4: Setting Powered False
15:43:18.531248 bus /gobi_4: Setting Powered False
15:43:19.639103 bus /gobi_1: Setting Powered False
15:43:20.740743 bus /gobi_1: Setting Powered False
15:43:21.851502 bus /gobi_0: Setting Powered False
15:43:22.953743 bus /gobi_0: Setting Powered False
15:43:24.064513 bus /gobi_6: Setting Powered False
31gprs:sysmo7.0osmo-stp_10.42.42.5(pid=18591): Process ended prematurely: osmo-stp_10.42.42.5(pid=18591) [trial-2361↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18591)]2331ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18591): Process ended prematurely: osmo-stp_10.42.42.5(pid=18591) [trial-2361↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18591)]
----------------------------------------------
trial-2361 gprs:sysmo ping.py
----------------------------------------------
15:43:25.133227 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:25.144793 tst gprs:sysmo: Using 1 x bts (candidates: 1)
15:43:25.278431 tst ping.py: using LAC 22048
15:43:25.399643 tst ping.py: using RAC 118
15:43:25.513509 tst ping.py: using CellId 22048
15:43:25.628046 tst ping.py: using BVCI 22049
15:43:25.650695 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:25.661412 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:25.672028 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:25.683166 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:25.694273 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:25.705294 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:25.716145 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:25.727066 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:25.998904 tst ping.py:19: start network...
15:43:26.121681 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:43:26.593855 run create_hlr_db(pid=18585): Launched
15:43:26.763018 bus /gobi_6: Setting Powered False
15:43:27.785394 run create_hlr_db(pid=18585): Terminated: ok {rc=0}
15:43:28.202312 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18587): Launched
15:43:28.495648 run osmo-hlr_10.42.42.2(pid=18588): Launched
15:43:28.607788 run osmo-stp_10.42.42.5: Starting osmo-stp
15:43:29.125063 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18590): Launched
15:43:29.416857 run osmo-stp_10.42.42.5(pid=18591): Launched
15:43:29.521482 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:43:30.028926 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18593): Launched
15:43:30.132618 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:43:30.389077 run patchelf(pid=18594): Launched
15:43:30.560527 run osmo-stp_10.42.42.5(pid=18591): ERR: Terminated: ERROR {rc=1} [trial-2361↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18591)]
15:43:30.686276 run osmo-stp_10.42.42.5(pid=18591): stdout:
| (launched: 2022-12-09_15:43:29.273034)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:43:30.812202 run osmo-stp_10.42.42.5(pid=18591): stderr:
| �[0;m�[38;5;43m20221209154329444 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154329444 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154329444 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154329445 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154329445 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154329445 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154329446 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:43:30.965306 run osmo-stp_10.42.42.5(pid=18591): stdout:
| (launched: 2022-12-09_15:43:29.273034)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:43:31.087487 run osmo-stp_10.42.42.5(pid=18591): stderr:
| �[0;m�[38;5;43m20221209154329444 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154329444 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154329444 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154329445 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154329445 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154329445 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154329446 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:43:31.229814 run patchelf(pid=18594): Terminating (SIGINT)
15:43:31.397473 run patchelf(pid=18594): Terminated: ok {rc=0}
15:43:31.428227 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=18591): Process ended prematurely: osmo-stp_10.42.42.5(pid=18591) [trial-2361↪gprs:sysmo↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18591)] [trial-2361↪gprs:sysmo↪ping.py:22]
15:43:31.437213 tst ping.py:22: Test FAILED (6.3 sec)
15:43:31.497315 run osmo-hlr_10.42.42.2(pid=18588): ERR: Terminated: ERROR {rc=237} [trial-2361↪gprs:sysmo↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18588)]
15:43:31.546760 run osmo-hlr_10.42.42.2(pid=18588): stdout:
| (launched: 2022-12-09_15:43:28.351263)
15:43:31.597759 run osmo-hlr_10.42.42.2(pid=18588): stderr:
| �[0;m�[1;31m20221209154328529 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209154328529 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209154328529 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209154328530 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209154328530 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209154328530 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209154328542 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2361/run.2022-12-09_15-36-58/gprs:sysmo/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209154328550 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209154328550 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209154328551 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:43:31.640923 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18587): Terminating (SIGTERM)
15:43:31.682915 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18590): Terminating (SIGTERM)
15:43:31.725235 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18593): Terminating (SIGTERM)
15:43:31.735440 --- ParallelTerminationStrategy: PID 18587 died...
15:43:31.791288 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18587): Terminated: ok {rc=0}
15:43:31.801409 --- ParallelTerminationStrategy: PID 18590 died...
15:43:31.855303 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18590): Terminated: ok {rc=0}
15:43:31.865302 --- ParallelTerminationStrategy: PID 18593 died...
15:43:31.926266 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18593): Terminated: ok {rc=0}
15:43:32.016331 bus /gobi_6: Setting Powered False
31gprs:sysmo7.0osmo-stp_10.42.42.5(pid=18601): Process ended prematurely: osmo-stp_10.42.42.5(pid=18601) [trial-2361↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18601)]2331ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18601): Process ended prematurely: osmo-stp_10.42.42.5(pid=18601) [trial-2361↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18601)]
----------------------------------------------
trial-2361 gprs:sysmo ping_idle_ping.py
----------------------------------------------
15:43:33.067126 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:33.078158 tst gprs:sysmo: Using 1 x bts (candidates: 1)
15:43:33.212484 tst ping_idle_ping.py: using LAC 22049
15:43:33.333784 tst ping_idle_ping.py: using RAC 119
15:43:33.455762 tst ping_idle_ping.py: using CellId 22049
15:43:33.578990 tst ping_idle_ping.py: using BVCI 22050
15:43:33.603200 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:33.614026 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:33.625019 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:33.636534 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:33.648042 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:33.659407 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:33.671047 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
15:43:33.682659 tst gprs:sysmo: Using 1 x modem (candidates: 4)
15:43:33.964317 tst ping_idle_ping.py:24: start network...
15:43:34.086243 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:43:34.556150 run create_hlr_db(pid=18595): Launched
15:43:34.724803 bus /gobi_6: Setting Powered False
15:43:35.745729 run create_hlr_db(pid=18595): Terminated: ok {rc=0}
15:43:36.135906 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18597): Launched
15:43:36.430229 run osmo-hlr_10.42.42.2(pid=18598): Launched
15:43:36.536340 run osmo-stp_10.42.42.5: Starting osmo-stp
15:43:37.043060 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18600): Launched
15:43:37.336155 run osmo-stp_10.42.42.5(pid=18601): Launched
15:43:37.442219 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:43:37.952875 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18603): Launched
15:43:38.057804 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:43:38.313759 run patchelf(pid=18604): Launched
15:43:38.485519 run osmo-stp_10.42.42.5(pid=18601): ERR: Terminated: ERROR {rc=1} [trial-2361↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18601)]
15:43:38.610982 run osmo-stp_10.42.42.5(pid=18601): stdout:
| (launched: 2022-12-09_15:43:37.191358)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:43:38.736450 run osmo-stp_10.42.42.5(pid=18601): stderr:
| �[0;m�[38;5;43m20221209154337361 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154337361 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154337362 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154337362 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154337363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154337363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154337363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:43:38.857819 run osmo-stp_10.42.42.5(pid=18601): stdout:
| (launched: 2022-12-09_15:43:37.191358)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:43:38.978143 run osmo-stp_10.42.42.5(pid=18601): stderr:
| �[0;m�[38;5;43m20221209154337361 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209154337361 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209154337362 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209154337362 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209154337363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209154337363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209154337363 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:43:39.117895 run patchelf(pid=18604): Terminating (SIGINT)
15:43:39.282088 run patchelf(pid=18604): Terminated: ok {rc=0}
15:43:39.311980 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=18601): Process ended prematurely: osmo-stp_10.42.42.5(pid=18601) [trial-2361↪gprs:sysmo↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18601)] [trial-2361↪gprs:sysmo↪ping_idle_ping.py:27]
15:43:39.320220 tst ping_idle_ping.py:27: Test FAILED (6.2 sec)
15:43:39.378808 run osmo-hlr_10.42.42.2(pid=18598): ERR: Terminated: ERROR {rc=237} [trial-2361↪gprs:sysmo↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18598)]
15:43:39.426605 run osmo-hlr_10.42.42.2(pid=18598): stdout:
| (launched: 2022-12-09_15:43:36.285618)
15:43:39.475365 run osmo-hlr_10.42.42.2(pid=18598): stderr:
| �[0;m�[1;31m20221209154336461 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209154336461 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209154336461 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209154336461 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209154336461 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209154336461 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209154336476 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2361/run.2022-12-09_15-36-58/gprs:sysmo/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209154336485 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209154336485 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209154336485 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:43:39.516410 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18597): Terminating (SIGTERM)
15:43:39.555406 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18600): Terminating (SIGTERM)
15:43:39.594507 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18603): Terminating (SIGTERM)
15:43:39.603591 --- ParallelTerminationStrategy: PID 18597 died...
15:43:39.655899 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18597): Terminated: ok {rc=0}
15:43:39.665912 --- ParallelTerminationStrategy: PID 18600 died...
15:43:39.718089 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18600): Terminated: ok {rc=0}
15:43:39.727542 --- ParallelTerminationStrategy: PID 18603 died...
15:43:39.779073 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18603): Terminated: ok {rc=0}
15:43:39.860751 bus /gobi_6: Setting Powered False
5gprs:sysmo00