83gprs:trx-b2009.0osmo-stp_10.42.42.5(pid=14305): Process ended prematurely: osmo-stp_10.42.42.5(pid=14305) [trial-2400↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14305)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=14305): Process ended prematurely: osmo-stp_10.42.42.5(pid=14305) [trial-2400↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14305)]
----------------------------------------------
trial-2400 gprs:trx-b200 cs_paging_gprs_active.py
----------------------------------------------
05:51:00.706684 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:00.717219 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
05:51:00.868640 tst cs_paging_gprs_active.py: using LAC 38843
05:51:01.011223 tst cs_paging_gprs_active.py: using RAC 83
05:51:01.145426 tst cs_paging_gprs_active.py: using CellId 38843
05:51:01.300193 tst cs_paging_gprs_active.py: using BVCI 38844
05:51:01.333910 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:01.348567 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:01.363287 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:01.378343 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:01.392857 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:01.406840 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:01.418868 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:01.431116 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:01.443759 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
05:51:01.589651 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
05:51:01.899446 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
05:51:02.031921 run iperf3-srv_10.42.42.10: Starting iperf3-srv
05:51:02.369290 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14295): Launched
05:51:02.633384 run iperf3-srv_10.42.42.10(pid=14296): Launched
05:51:02.820575 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
05:51:02.949718 run iperf3-srv_10.42.42.10: Starting iperf3-srv
05:51:03.268272 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14297): Launched
05:51:03.532580 run iperf3-srv_10.42.42.10(pid=14298): Launched
05:51:03.720290 tst cs_paging_gprs_active.py:38: start network...
05:51:03.848273 run osmo-hlr_10.42.42.2: Starting osmo-hlr
05:51:04.332094 run create_hlr_db(pid=14299): Launched
05:51:04.508925 bus /sierra_2: Setting Powered False
05:51:05.531199 run create_hlr_db(pid=14299): Terminated: ok {rc=0}
05:51:05.948031 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14301): Launched
05:51:06.275365 run osmo-hlr_10.42.42.2(pid=14302): Launched
05:51:06.399721 run osmo-stp_10.42.42.5: Starting osmo-stp
05:51:06.968557 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14304): Launched
05:51:07.295073 run osmo-stp_10.42.42.5(pid=14305): Launched
05:51:07.414939 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
05:51:07.986902 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14307): Launched
05:51:08.104246 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
05:51:08.389831 run patchelf(pid=14308): Launched
05:51:08.582198 run osmo-stp_10.42.42.5(pid=14305): ERR: Terminated: ERROR {rc=1} [trial-2400↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14305)]
05:51:08.722533 run osmo-stp_10.42.42.5(pid=14305): stdout:
| (launched: 2022-12-14_05:51:07.135864)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:08.863158 run osmo-stp_10.42.42.5(pid=14305): stderr:
| �[0;m�[38;5;43m20221214055107324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055107324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055107324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055107325 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055107325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055107325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055107325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:08.999126 run osmo-stp_10.42.42.5(pid=14305): stdout:
| (launched: 2022-12-14_05:51:07.135864)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:09.132892 run osmo-stp_10.42.42.5(pid=14305): stderr:
| �[0;m�[38;5;43m20221214055107324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055107324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055107324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055107325 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055107325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055107325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055107325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:09.288516 run patchelf(pid=14308): Terminating (SIGINT)
05:51:09.471988 run patchelf(pid=14308): Terminated: ok {rc=0}
05:51:09.505775 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=14305): Process ended prematurely: osmo-stp_10.42.42.5(pid=14305) [trial-2400↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14305)] [trial-2400↪gprs:trx-b200↪cs_paging_gprs_active.py:38]
05:51:09.515963 tst cs_paging_gprs_active.py:38: Test FAILED (8.8 sec)
05:51:09.577906 run osmo-hlr_10.42.42.2(pid=14302): ERR: Terminated: ERROR {rc=237} [trial-2400↪gprs:trx-b200↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=14302)]
05:51:09.628516 run osmo-hlr_10.42.42.2(pid=14302): stdout:
| (launched: 2022-12-14_05:51:06.115188)
05:51:09.679914 run osmo-hlr_10.42.42.2(pid=14302): stderr:
| �[0;m�[1;31m20221214055106311 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214055106311 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214055106311 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214055106311 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214055106311 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214055106311 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214055106324 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2400/run.2022-12-14_05-28-31/gprs:trx-b200/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214055106332 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214055106333 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214055106333 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
05:51:09.724036 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14295): Terminating (SIGTERM)
05:51:09.765560 run iperf3-srv_10.42.42.10(pid=14296): Terminating (SIGTERM)
05:51:09.806884 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14297): Terminating (SIGTERM)
05:51:09.848160 run iperf3-srv_10.42.42.10(pid=14298): Terminating (SIGTERM)
05:51:09.889732 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14301): Terminating (SIGTERM)
05:51:09.931154 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14304): Terminating (SIGTERM)
05:51:09.972937 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14307): Terminating (SIGTERM)
05:51:09.982669 --- ParallelTerminationStrategy: PID 14295 died...
05:51:10.038623 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14295): Terminated: ok {rc=0}
05:51:10.048794 --- ParallelTerminationStrategy: PID 14296 died...
05:51:10.103983 run iperf3-srv_10.42.42.10(pid=14296): Terminated {rc=256}
05:51:10.113959 --- ParallelTerminationStrategy: PID 14297 died...
05:51:10.168756 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14297): Terminated: ok {rc=0}
05:51:10.178617 --- ParallelTerminationStrategy: PID 14298 died...
05:51:10.233625 run iperf3-srv_10.42.42.10(pid=14298): Terminated {rc=256}
05:51:10.243590 --- ParallelTerminationStrategy: PID 14301 died...
05:51:10.298737 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14301): Terminated: ok {rc=0}
05:51:10.308615 --- ParallelTerminationStrategy: PID 14304 died...
05:51:10.363657 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14304): Terminated: ok {rc=0}
05:51:10.373709 --- ParallelTerminationStrategy: PID 14307 died...
05:51:10.429119 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14307): Terminated: ok {rc=0}
05:51:10.516129 bus /gobi_0: Setting Powered False
05:51:11.622375 bus /gobi_0: Setting Powered False
05:51:12.731290 bus /gobi_6: Setting Powered False
83gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=14317): Process ended prematurely: osmo-stp_10.42.42.5(pid=14317) [trial-2400↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14317)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=14317): Process ended prematurely: osmo-stp_10.42.42.5(pid=14317) [trial-2400↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14317)]
----------------------------------------------
trial-2400 gprs:trx-b200 iperf3.py
----------------------------------------------
05:51:13.846558 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:13.858315 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
05:51:14.006043 tst iperf3.py: using LAC 38844
05:51:14.136462 tst iperf3.py: using RAC 84
05:51:14.264359 tst iperf3.py: using CellId 38844
05:51:14.402101 tst iperf3.py: using BVCI 38845
05:51:14.427233 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:14.438523 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:14.449434 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:14.460255 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:14.471599 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:14.482639 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:14.493929 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:14.504868 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:14.515989 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
05:51:14.810685 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
05:51:14.948511 run iperf3-srv_10.42.42.10: Starting iperf3-srv
05:51:15.287390 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14309): Launched
05:51:15.567662 run iperf3-srv_10.42.42.10(pid=14310): Launched
05:51:15.768297 tst iperf3.py:8: start network...
05:51:15.910818 run osmo-hlr_10.42.42.2: Starting osmo-hlr
05:51:16.473864 run create_hlr_db(pid=14311): Launched
05:51:16.665619 bus /gobi_6: Setting Powered False
05:51:17.683720 run create_hlr_db(pid=14311): Terminated: ok {rc=0}
05:51:18.095845 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14313): Launched
05:51:18.401684 run osmo-hlr_10.42.42.2(pid=14314): Launched
05:51:18.521819 run osmo-stp_10.42.42.5: Starting osmo-stp
05:51:19.061899 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14316): Launched
05:51:19.368369 run osmo-stp_10.42.42.5(pid=14317): Launched
05:51:19.480911 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
05:51:20.014250 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14319): Launched
05:51:20.124123 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
05:51:20.391595 run patchelf(pid=14320): Launched
05:51:20.572165 run osmo-stp_10.42.42.5(pid=14317): ERR: Terminated: ERROR {rc=1} [trial-2400↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14317)]
05:51:20.704122 run osmo-stp_10.42.42.5(pid=14317): stdout:
| (launched: 2022-12-14_05:51:19.217809)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:20.835889 run osmo-stp_10.42.42.5(pid=14317): stderr:
| �[0;m�[38;5;43m20221214055119387 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055119388 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055119388 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055119388 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055119389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055119389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055119389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:20.963091 run osmo-stp_10.42.42.5(pid=14317): stdout:
| (launched: 2022-12-14_05:51:19.217809)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:21.089363 run osmo-stp_10.42.42.5(pid=14317): stderr:
| �[0;m�[38;5;43m20221214055119387 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055119388 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055119388 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055119388 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055119389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055119389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055119389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:21.235986 run patchelf(pid=14320): Terminating (SIGINT)
05:51:21.408074 run patchelf(pid=14320): Terminated: ok {rc=0}
05:51:21.439975 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=14317): Process ended prematurely: osmo-stp_10.42.42.5(pid=14317) [trial-2400↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14317)] [trial-2400↪gprs:trx-b200↪iperf3.py:8]
05:51:21.449457 tst iperf3.py:8: Test FAILED (7.6 sec)
05:51:21.509252 run osmo-hlr_10.42.42.2(pid=14314): ERR: Terminated: ERROR {rc=237} [trial-2400↪gprs:trx-b200↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=14314)]
05:51:21.557914 run osmo-hlr_10.42.42.2(pid=14314): stdout:
| (launched: 2022-12-14_05:51:18.251575)
05:51:21.607228 run osmo-hlr_10.42.42.2(pid=14314): stderr:
| �[0;m�[1;31m20221214055118423 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214055118423 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214055118423 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214055118423 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214055118423 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214055118423 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214055118435 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2400/run.2022-12-14_05-28-31/gprs:trx-b200/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214055118443 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214055118444 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214055118444 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
05:51:21.649677 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14309): Terminating (SIGTERM)
05:51:21.689748 run iperf3-srv_10.42.42.10(pid=14310): Terminating (SIGTERM)
05:51:21.729944 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14313): Terminating (SIGTERM)
05:51:21.769742 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14316): Terminating (SIGTERM)
05:51:21.809815 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14319): Terminating (SIGTERM)
05:51:21.819077 --- ParallelTerminationStrategy: PID 14309 died...
05:51:21.872664 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14309): Terminated: ok {rc=0}
05:51:21.882381 --- ParallelTerminationStrategy: PID 14310 died...
05:51:21.935520 run iperf3-srv_10.42.42.10(pid=14310): Terminated {rc=256}
05:51:21.944906 --- ParallelTerminationStrategy: PID 14313 died...
05:51:21.998000 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14313): Terminated: ok {rc=0}
05:51:22.007516 --- ParallelTerminationStrategy: PID 14316 died...
05:51:22.060626 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14316): Terminated: ok {rc=0}
05:51:22.070185 --- ParallelTerminationStrategy: PID 14319 died...
05:51:22.123281 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14319): Terminated: ok {rc=0}
05:51:22.210290 bus /gobi_6: Setting Powered False
215gprs:trx-b20011.0osmo-stp_10.42.42.5(pid=14335): Process ended prematurely: osmo-stp_10.42.42.5(pid=14335) [trial-2400↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14335)]2186iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=14335): Process ended prematurely: osmo-stp_10.42.42.5(pid=14335) [trial-2400↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14335)]
----------------------------------------------
trial-2400 gprs:trx-b200 iperf3m4.py
----------------------------------------------
05:51:23.325136 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:23.336674 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
05:51:23.483280 tst iperf3m4.py: using LAC 38845
05:51:23.614056 tst iperf3m4.py: using RAC 85
05:51:23.733820 tst iperf3m4.py: using CellId 38845
05:51:23.854032 tst iperf3m4.py: using BVCI 38846
05:51:23.877584 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:23.888463 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:23.899226 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:23.909860 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:23.920565 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:23.931239 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:23.942235 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:23.952922 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:23.963880 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
05:51:24.079862 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
05:51:24.201114 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
05:51:24.318891 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
05:51:24.598013 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
05:51:24.726506 run iperf3-srv_10.42.42.10: Starting iperf3-srv
05:51:25.045973 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14321): Launched
05:51:25.310353 run iperf3-srv_10.42.42.10(pid=14322): Launched
05:51:25.498099 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
05:51:25.626165 run iperf3-srv_10.42.42.10: Starting iperf3-srv
05:51:25.943391 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14323): Launched
05:51:26.207256 run iperf3-srv_10.42.42.10(pid=14324): Launched
05:51:26.394624 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
05:51:26.522713 run iperf3-srv_10.42.42.10: Starting iperf3-srv
05:51:26.841312 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14325): Launched
05:51:27.105830 run iperf3-srv_10.42.42.10(pid=14326): Launched
05:51:27.293366 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
05:51:27.421948 run iperf3-srv_10.42.42.10: Starting iperf3-srv
05:51:27.739930 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14327): Launched
05:51:28.003333 run iperf3-srv_10.42.42.10(pid=14328): Launched
05:51:28.190759 tst iperf3m4.py:8: start network...
05:51:28.319155 run osmo-hlr_10.42.42.2: Starting osmo-hlr
05:51:28.800676 run create_hlr_db(pid=14329): Launched
05:51:28.977428 bus /gobi_6: Setting Powered False
05:51:29.998633 run create_hlr_db(pid=14329): Terminated: ok {rc=0}
05:51:30.407143 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14331): Launched
05:51:30.713819 run osmo-hlr_10.42.42.2(pid=14332): Launched
05:51:30.851581 run osmo-stp_10.42.42.5: Starting osmo-stp
05:51:31.516448 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14334): Launched
05:51:31.999843 run osmo-stp_10.42.42.5(pid=14335): Launched
05:51:32.133192 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
05:51:32.712199 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14337): Launched
05:51:32.830438 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
05:51:33.119682 run patchelf(pid=14338): Launched
05:51:33.315236 run osmo-stp_10.42.42.5(pid=14335): ERR: Terminated: ERROR {rc=1} [trial-2400↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14335)]
05:51:33.458745 run osmo-stp_10.42.42.5(pid=14335): stdout:
| (launched: 2022-12-14_05:51:31.675719)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:33.602876 run osmo-stp_10.42.42.5(pid=14335): stderr:
| �[0;m�[38;5;43m20221214055131892 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055131892 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055131892 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055131893 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055131893 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055131893 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055131893 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:33.741448 run osmo-stp_10.42.42.5(pid=14335): stdout:
| (launched: 2022-12-14_05:51:31.675719)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:33.878789 run osmo-stp_10.42.42.5(pid=14335): stderr:
| �[0;m�[38;5;43m20221214055131892 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055131892 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055131892 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055131893 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055131893 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055131893 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055131893 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:34.038369 run patchelf(pid=14338): Terminating (SIGINT)
05:51:34.225827 run patchelf(pid=14338): Terminated: ok {rc=0}
05:51:34.260120 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=14335): Process ended prematurely: osmo-stp_10.42.42.5(pid=14335) [trial-2400↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14335)] [trial-2400↪gprs:trx-b200↪iperf3m4.py:8]
05:51:34.270374 tst iperf3m4.py:8: Test FAILED (11.0 sec)
05:51:34.334389 run osmo-hlr_10.42.42.2(pid=14332): ERR: Terminated: ERROR {rc=237} [trial-2400↪gprs:trx-b200↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=14332)]
05:51:34.386595 run osmo-hlr_10.42.42.2(pid=14332): stdout:
| (launched: 2022-12-14_05:51:30.562826)
05:51:34.439273 run osmo-hlr_10.42.42.2(pid=14332): stderr:
| �[0;m�[1;31m20221214055130735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214055130735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214055130735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214055130735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214055130735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214055130735 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214055130747 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2400/run.2022-12-14_05-28-31/gprs:trx-b200/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214055130755 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214055130755 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214055130756 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
05:51:34.484704 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14321): Terminating (SIGTERM)
05:51:34.527703 run iperf3-srv_10.42.42.10(pid=14322): Terminating (SIGTERM)
05:51:34.570479 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14323): Terminating (SIGTERM)
05:51:34.613943 run iperf3-srv_10.42.42.10(pid=14324): Terminating (SIGTERM)
05:51:34.656999 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14325): Terminating (SIGTERM)
05:51:34.699891 run iperf3-srv_10.42.42.10(pid=14326): Terminating (SIGTERM)
05:51:34.742498 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14327): Terminating (SIGTERM)
05:51:34.785281 run iperf3-srv_10.42.42.10(pid=14328): Terminating (SIGTERM)
05:51:34.827860 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14331): Terminating (SIGTERM)
05:51:34.870834 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14334): Terminating (SIGTERM)
05:51:34.913607 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14337): Terminating (SIGTERM)
05:51:34.923523 --- ParallelTerminationStrategy: PID 14321 died...
05:51:34.980624 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14321): Terminated: ok {rc=0}
05:51:34.990834 --- ParallelTerminationStrategy: PID 14322 died...
05:51:35.047468 run iperf3-srv_10.42.42.10(pid=14322): Terminated {rc=256}
05:51:35.057802 --- ParallelTerminationStrategy: PID 14323 died...
05:51:35.114793 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14323): Terminated: ok {rc=0}
05:51:35.125087 --- ParallelTerminationStrategy: PID 14324 died...
05:51:35.181583 run iperf3-srv_10.42.42.10(pid=14324): Terminated {rc=256}
05:51:35.191876 --- ParallelTerminationStrategy: PID 14325 died...
05:51:35.248526 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14325): Terminated: ok {rc=0}
05:51:35.258885 --- ParallelTerminationStrategy: PID 14326 died...
05:51:35.315566 run iperf3-srv_10.42.42.10(pid=14326): Terminated {rc=256}
05:51:35.325948 --- ParallelTerminationStrategy: PID 14327 died...
05:51:35.382543 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=14327): Terminated: ok {rc=0}
05:51:35.392822 --- ParallelTerminationStrategy: PID 14328 died...
05:51:35.449430 run iperf3-srv_10.42.42.10(pid=14328): Terminated {rc=256}
05:51:35.459617 --- ParallelTerminationStrategy: PID 14331 died...
05:51:35.516373 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14331): Terminated: ok {rc=0}
05:51:35.526792 --- ParallelTerminationStrategy: PID 14334 died...
05:51:35.583439 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14334): Terminated: ok {rc=0}
05:51:35.593857 --- ParallelTerminationStrategy: PID 14337 died...
05:51:35.650415 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14337): Terminated: ok {rc=0}
05:51:35.738391 bus /gobi_4: Setting Powered False
05:51:36.843471 bus /gobi_4: Setting Powered False
05:51:37.955321 bus /gobi_1: Setting Powered False
05:51:39.059294 bus /gobi_1: Setting Powered False
05:51:40.170583 bus /gobi_0: Setting Powered False
05:51:41.273559 bus /gobi_0: Setting Powered False
05:51:42.386413 bus /gobi_6: Setting Powered False
215gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=14346): Process ended prematurely: osmo-stp_10.42.42.5(pid=14346) [trial-2400↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14346)]2186ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=14346): Process ended prematurely: osmo-stp_10.42.42.5(pid=14346) [trial-2400↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14346)]
----------------------------------------------
trial-2400 gprs:trx-b200 ping.py
----------------------------------------------
05:51:43.453185 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:43.464220 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
05:51:43.604923 tst ping.py: using LAC 38846
05:51:43.726650 tst ping.py: using RAC 86
05:51:43.850200 tst ping.py: using CellId 38846
05:51:43.971410 tst ping.py: using BVCI 38847
05:51:43.995298 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:44.006062 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:44.016675 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:44.027266 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:44.038593 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:44.049516 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:44.060778 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:44.071735 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
05:51:44.370805 tst ping.py:19: start network...
05:51:44.509445 run osmo-hlr_10.42.42.2: Starting osmo-hlr
05:51:45.034982 run create_hlr_db(pid=14340): Launched
05:51:45.226456 bus /gobi_6: Setting Powered False
05:51:46.248824 run create_hlr_db(pid=14340): Terminated: ok {rc=0}
05:51:46.682620 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14342): Launched
05:51:47.014182 run osmo-hlr_10.42.42.2(pid=14343): Launched
05:51:47.167133 run osmo-stp_10.42.42.5: Starting osmo-stp
05:51:47.751347 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14345): Launched
05:51:48.081866 run osmo-stp_10.42.42.5(pid=14346): Launched
05:51:48.201900 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
05:51:48.775114 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14348): Launched
05:51:48.892708 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
05:51:49.178790 run patchelf(pid=14349): Launched
05:51:49.371376 run osmo-stp_10.42.42.5(pid=14346): ERR: Terminated: ERROR {rc=1} [trial-2400↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14346)]
05:51:49.512399 run osmo-stp_10.42.42.5(pid=14346): stdout:
| (launched: 2022-12-14_05:51:47.918864)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:49.653411 run osmo-stp_10.42.42.5(pid=14346): stderr:
| �[0;m�[38;5;43m20221214055148114 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055148114 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055148114 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055148114 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055148115 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055148115 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055148115 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:49.789255 run osmo-stp_10.42.42.5(pid=14346): stdout:
| (launched: 2022-12-14_05:51:47.918864)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:49.923991 run osmo-stp_10.42.42.5(pid=14346): stderr:
| �[0;m�[38;5;43m20221214055148114 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055148114 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055148114 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055148114 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055148115 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055148115 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055148115 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:50.081060 run patchelf(pid=14349): Terminating (SIGINT)
05:51:50.265032 run patchelf(pid=14349): Terminated: ok {rc=0}
05:51:50.298203 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=14346): Process ended prematurely: osmo-stp_10.42.42.5(pid=14346) [trial-2400↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14346)] [trial-2400↪gprs:trx-b200↪ping.py:22]
05:51:50.307493 tst ping.py:22: Test FAILED (6.8 sec)
05:51:50.370955 run osmo-hlr_10.42.42.2(pid=14343): ERR: Terminated: ERROR {rc=237} [trial-2400↪gprs:trx-b200↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=14343)]
05:51:50.422976 run osmo-hlr_10.42.42.2(pid=14343): stdout:
| (launched: 2022-12-14_05:51:46.845254)
05:51:50.475486 run osmo-hlr_10.42.42.2(pid=14343): stderr:
| �[0;m�[1;31m20221214055147249 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214055147249 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214055147249 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214055147249 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214055147249 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214055147249 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214055147273 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2400/run.2022-12-14_05-28-31/gprs:trx-b200/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214055147281 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214055147281 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214055147282 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
05:51:50.520636 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14342): Terminating (SIGTERM)
05:51:50.563759 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14345): Terminating (SIGTERM)
05:51:50.606063 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14348): Terminating (SIGTERM)
05:51:50.615835 --- ParallelTerminationStrategy: PID 14342 died...
05:51:50.672312 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14342): Terminated: ok {rc=0}
05:51:50.682440 --- ParallelTerminationStrategy: PID 14345 died...
05:51:50.738864 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14345): Terminated: ok {rc=0}
05:51:50.748894 --- ParallelTerminationStrategy: PID 14348 died...
05:51:50.805066 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14348): Terminated: ok {rc=0}
05:51:50.892386 bus /gobi_6: Setting Powered False
215gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=14356): Process ended prematurely: osmo-stp_10.42.42.5(pid=14356) [trial-2400↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14356)]2186ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=14356): Process ended prematurely: osmo-stp_10.42.42.5(pid=14356) [trial-2400↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14356)]
----------------------------------------------
trial-2400 gprs:trx-b200 ping_idle_ping.py
----------------------------------------------
05:51:51.946695 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:51.958567 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
05:51:52.111283 tst ping_idle_ping.py: using LAC 38847
05:51:52.239384 tst ping_idle_ping.py: using RAC 87
05:51:52.369476 tst ping_idle_ping.py: using CellId 38847
05:51:52.502892 tst ping_idle_ping.py: using BVCI 38848
05:51:52.528410 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:52.539652 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:52.550796 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:52.561890 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:52.573478 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:52.584892 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:52.596448 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
05:51:52.607963 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
05:51:52.905351 tst ping_idle_ping.py:24: start network...
05:51:53.043756 run osmo-hlr_10.42.42.2: Starting osmo-hlr
05:51:53.573758 run create_hlr_db(pid=14350): Launched
05:51:53.765501 bus /gobi_6: Setting Powered False
05:51:54.785975 run create_hlr_db(pid=14350): Terminated: ok {rc=0}
05:51:55.202894 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14352): Launched
05:51:55.513753 run osmo-hlr_10.42.42.2(pid=14353): Launched
05:51:55.633999 run osmo-stp_10.42.42.5: Starting osmo-stp
05:51:56.176485 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14355): Launched
05:51:56.486311 run osmo-stp_10.42.42.5(pid=14356): Launched
05:51:56.601515 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
05:51:57.140349 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14358): Launched
05:51:57.251180 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
05:51:57.523049 run patchelf(pid=14359): Launched
05:51:57.704653 run osmo-stp_10.42.42.5(pid=14356): ERR: Terminated: ERROR {rc=1} [trial-2400↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14356)]
05:51:57.837498 run osmo-stp_10.42.42.5(pid=14356): stdout:
| (launched: 2022-12-14_05:51:56.333616)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:57.970442 run osmo-stp_10.42.42.5(pid=14356): stderr:
| �[0;m�[38;5;43m20221214055156504 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055156504 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055156504 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055156505 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055156505 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055156505 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055156505 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:58.098627 run osmo-stp_10.42.42.5(pid=14356): stdout:
| (launched: 2022-12-14_05:51:56.333616)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
05:51:58.225714 run osmo-stp_10.42.42.5(pid=14356): stderr:
| �[0;m�[38;5;43m20221214055156504 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214055156504 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214055156504 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214055156505 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214055156505 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214055156505 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214055156505 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
05:51:58.374128 run patchelf(pid=14359): Terminating (SIGINT)
05:51:58.547424 run patchelf(pid=14359): Terminated: ok {rc=0}
05:51:58.578798 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=14356): Process ended prematurely: osmo-stp_10.42.42.5(pid=14356) [trial-2400↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=14356)] [trial-2400↪gprs:trx-b200↪ping_idle_ping.py:27]
05:51:58.587772 tst ping_idle_ping.py:27: Test FAILED (6.6 sec)
05:51:58.648850 run osmo-hlr_10.42.42.2(pid=14353): ERR: Terminated: ERROR {rc=237} [trial-2400↪gprs:trx-b200↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=14353)]
05:51:58.699290 run osmo-hlr_10.42.42.2(pid=14353): stdout:
| (launched: 2022-12-14_05:51:55.361232)
05:51:58.749949 run osmo-hlr_10.42.42.2(pid=14353): stderr:
| �[0;m�[1;31m20221214055155538 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214055155538 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214055155538 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214055155538 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214055155538 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214055155538 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214055155550 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2400/run.2022-12-14_05-28-31/gprs:trx-b200/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214055155558 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214055155558 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214055155558 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
05:51:58.793255 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14352): Terminating (SIGTERM)
05:51:58.834018 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14355): Terminating (SIGTERM)
05:51:58.874606 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14358): Terminating (SIGTERM)
05:51:58.883970 --- ParallelTerminationStrategy: PID 14352 died...
05:51:58.939162 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14352): Terminated: ok {rc=0}
05:51:58.949006 --- ParallelTerminationStrategy: PID 14355 died...
05:51:59.003353 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14355): Terminated: ok {rc=0}
05:51:59.013193 --- ParallelTerminationStrategy: PID 14358 died...
05:51:59.067409 run pcap-recorder_any(filters='host 10.42.42.6')(pid=14358): Terminated: ok {rc=0}
05:51:59.152511 bus /gobi_6: Setting Powered False
5gprs:trx-b20000