213gprs:trx-lms10.0osmo-stp_10.42.42.5(pid=31293): Process ended prematurely: osmo-stp_10.42.42.5(pid=31293) [trial-2351↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31293)]2139cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31293): Process ended prematurely: osmo-stp_10.42.42.5(pid=31293) [trial-2351↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31293)]
----------------------------------------------
trial-2351 gprs:trx-lms cs_paging_gprs_active.py
----------------------------------------------
23:59:59.080615 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:59.091493 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
23:59:59.229320 tst cs_paging_gprs_active.py: using LAC 17661
23:59:59.348149 tst cs_paging_gprs_active.py: using RAC 66
23:59:59.469594 tst cs_paging_gprs_active.py: using CellId 17661
23:59:59.589095 tst cs_paging_gprs_active.py: using BVCI 17662
23:59:59.612159 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:59.622794 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:59.633102 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:59.643348 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:59.653804 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:59.664463 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:59.675060 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:59.685967 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:59.696675 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
23:59:59.811100 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:00.087108 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
00:00:00.213434 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:00.527235 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31242): Launched
00:00:00.788421 run iperf3-srv_10.42.42.10(pid=31243): Launched
00:00:01.045095 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
00:00:01.185342 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:01.546419 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31251): Launched
00:00:01.882310 run iperf3-srv_10.42.42.10(pid=31252): Launched
00:00:02.145248 tst cs_paging_gprs_active.py:38: start network...
00:00:02.297939 run osmo-hlr_10.42.42.2: Starting osmo-hlr
00:00:02.952138 run create_hlr_db(pid=31264): Launched
00:00:03.153041 bus /sierra_2: Setting Powered False
00:00:04.208599 run create_hlr_db(pid=31264): Terminated: ok {rc=0}
00:00:04.697329 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31281): Launched
00:00:05.133735 run osmo-hlr_10.42.42.2(pid=31282): Launched
00:00:05.394031 run osmo-stp_10.42.42.5: Starting osmo-stp
00:00:06.004295 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31291): Launched
00:00:06.574107 run osmo-stp_10.42.42.5(pid=31293): Launched
00:00:06.787740 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
00:00:07.410441 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31303): Launched
00:00:07.529201 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
00:00:07.834045 run patchelf(pid=31304): Launched
00:00:08.027357 run osmo-stp_10.42.42.5(pid=31293): ERR: Terminated: ERROR {rc=1} [trial-2351↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31293)]
00:00:08.168620 run osmo-stp_10.42.42.5(pid=31293): stdout:
| (launched: 2022-12-09_00:00:06.203749)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:08.309610 run osmo-stp_10.42.42.5(pid=31293): stderr:
| �[0;m�[38;5;43m20221209000006599 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000006599 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000006600 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000006600 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000006600 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000006600 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000006605 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:08.446827 run osmo-stp_10.42.42.5(pid=31293): stdout:
| (launched: 2022-12-09_00:00:06.203749)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:08.583764 run osmo-stp_10.42.42.5(pid=31293): stderr:
| �[0;m�[38;5;43m20221209000006599 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000006599 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000006600 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000006600 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000006600 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000006600 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000006605 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:08.743276 run patchelf(pid=31304): Terminating (SIGINT)
00:00:08.926901 run patchelf(pid=31304): Terminated: ok {rc=0}
00:00:08.963121 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=31293): Process ended prematurely: osmo-stp_10.42.42.5(pid=31293) [trial-2351↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31293)] [trial-2351↪gprs:trx-lms↪cs_paging_gprs_active.py:38]
00:00:08.973987 tst cs_paging_gprs_active.py:38: Test FAILED (9.9 sec)
00:00:09.041323 run osmo-hlr_10.42.42.2(pid=31282): ERR: Terminated: ERROR {rc=237} [trial-2351↪gprs:trx-lms↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31282)]
00:00:09.093588 run osmo-hlr_10.42.42.2(pid=31282): stdout:
| (launched: 2022-12-09_00:00:04.867451)
00:00:09.149204 run osmo-hlr_10.42.42.2(pid=31282): stderr:
| �[0;m�[1;31m20221209000005181 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209000005181 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209000005181 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209000005181 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209000005181 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209000005181 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209000005208 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2351/run.2022-12-08_23-11-41/gprs:trx-lms/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209000005217 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209000005217 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209000005217 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
00:00:09.196100 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31242): Terminating (SIGTERM)
00:00:09.241945 run iperf3-srv_10.42.42.10(pid=31243): Terminating (SIGTERM)
00:00:09.287711 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31251): Terminating (SIGTERM)
00:00:09.334772 run iperf3-srv_10.42.42.10(pid=31252): Terminating (SIGTERM)
00:00:09.380214 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31281): Terminating (SIGTERM)
00:00:09.424726 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31291): Terminating (SIGTERM)
00:00:09.472051 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31303): Terminating (SIGTERM)
00:00:09.482861 --- ParallelTerminationStrategy: PID 31242 died...
00:00:09.543778 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31242): Terminated: ok {rc=0}
00:00:09.554668 --- ParallelTerminationStrategy: PID 31243 died...
00:00:09.614203 run iperf3-srv_10.42.42.10(pid=31243): Terminated {rc=256}
00:00:09.625097 --- ParallelTerminationStrategy: PID 31251 died...
00:00:09.683920 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31251): Terminated: ok {rc=0}
00:00:09.695843 --- ParallelTerminationStrategy: PID 31252 died...
00:00:09.754505 run iperf3-srv_10.42.42.10(pid=31252): Terminated {rc=256}
00:00:09.766406 --- ParallelTerminationStrategy: PID 31281 died...
00:00:09.826045 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31281): Terminated: ok {rc=0}
00:00:09.837968 --- ParallelTerminationStrategy: PID 31291 died...
00:00:09.895631 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31291): Terminated: ok {rc=0}
00:00:09.906509 --- ParallelTerminationStrategy: PID 31303 died...
00:00:09.965562 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31303): Terminated: ok {rc=0}
00:00:10.051803 bus /gobi_0: Setting Powered False
00:00:11.161475 bus /gobi_0: Setting Powered False
00:00:12.275322 bus /gobi_6: Setting Powered False
210gprs:trx-lms9.0osmo-stp_10.42.42.5(pid=31320): Process ended prematurely: osmo-stp_10.42.42.5(pid=31320) [trial-2351↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31320)]2142iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31320): Process ended prematurely: osmo-stp_10.42.42.5(pid=31320) [trial-2351↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31320)]
----------------------------------------------
trial-2351 gprs:trx-lms iperf3.py
----------------------------------------------
00:00:13.389240 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:13.401475 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
00:00:13.541522 tst iperf3.py: using LAC 17662
00:00:13.663169 tst iperf3.py: using RAC 67
00:00:13.787637 tst iperf3.py: using CellId 17662
00:00:13.906561 tst iperf3.py: using BVCI 17663
00:00:13.931217 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:13.942146 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:13.952866 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:13.964143 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:13.975039 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:13.985820 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:13.996766 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:14.007692 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:14.018582 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:14.302369 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
00:00:14.432849 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:14.770950 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31310): Launched
00:00:15.045051 run iperf3-srv_10.42.42.10(pid=31311): Launched
00:00:15.236112 tst iperf3.py:8: start network...
00:00:15.365606 run osmo-hlr_10.42.42.2: Starting osmo-hlr
00:00:15.914631 run create_hlr_db(pid=31313): Launched
00:00:16.109914 bus /gobi_6: Setting Powered False
00:00:17.188857 run create_hlr_db(pid=31313): Terminated: ok {rc=0}
00:00:17.701519 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31316): Launched
00:00:18.154153 run osmo-hlr_10.42.42.2(pid=31317): Launched
00:00:18.392845 run osmo-stp_10.42.42.5: Starting osmo-stp
00:00:18.978441 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31319): Launched
00:00:19.400495 run osmo-stp_10.42.42.5(pid=31320): Launched
00:00:19.626514 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
00:00:20.212021 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31322): Launched
00:00:20.330500 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
00:00:20.635195 run patchelf(pid=31323): Launched
00:00:20.829109 run osmo-stp_10.42.42.5(pid=31320): ERR: Terminated: ERROR {rc=1} [trial-2351↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31320)]
00:00:20.971241 run osmo-stp_10.42.42.5(pid=31320): stdout:
| (launched: 2022-12-09_00:00:19.143801)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:21.111624 run osmo-stp_10.42.42.5(pid=31320): stderr:
| �[0;m�[38;5;43m20221209000019448 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000019448 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000019448 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000019449 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000019449 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000019449 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000019449 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:21.248137 run osmo-stp_10.42.42.5(pid=31320): stdout:
| (launched: 2022-12-09_00:00:19.143801)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:21.388731 run osmo-stp_10.42.42.5(pid=31320): stderr:
| �[0;m�[38;5;43m20221209000019448 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000019448 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000019448 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000019449 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000019449 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000019449 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000019449 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:21.545405 run patchelf(pid=31323): Terminating (SIGINT)
00:00:21.726622 run patchelf(pid=31323): Terminated: ok {rc=0}
00:00:21.759843 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=31320): Process ended prematurely: osmo-stp_10.42.42.5(pid=31320) [trial-2351↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31320)] [trial-2351↪gprs:trx-lms↪iperf3.py:8]
00:00:21.769901 tst iperf3.py:8: Test FAILED (8.4 sec)
00:00:21.833710 run osmo-hlr_10.42.42.2(pid=31317): ERR: Terminated: ERROR {rc=237} [trial-2351↪gprs:trx-lms↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31317)]
00:00:21.885099 run osmo-hlr_10.42.42.2(pid=31317): stdout:
| (launched: 2022-12-09_00:00:17.886585)
00:00:21.937325 run osmo-hlr_10.42.42.2(pid=31317): stderr:
| �[0;m�[1;31m20221209000018191 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209000018191 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209000018191 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209000018191 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209000018191 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209000018191 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209000018214 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2351/run.2022-12-08_23-11-41/gprs:trx-lms/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209000018222 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209000018222 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209000018222 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
00:00:21.982388 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31310): Terminating (SIGTERM)
00:00:22.024361 run iperf3-srv_10.42.42.10(pid=31311): Terminating (SIGTERM)
00:00:22.066418 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31316): Terminating (SIGTERM)
00:00:22.108218 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31319): Terminating (SIGTERM)
00:00:22.150764 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31322): Terminating (SIGTERM)
00:00:22.160446 --- ParallelTerminationStrategy: PID 31310 died...
00:00:22.216933 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31310): Terminated: ok {rc=0}
00:00:22.227152 --- ParallelTerminationStrategy: PID 31311 died...
00:00:22.283116 run iperf3-srv_10.42.42.10(pid=31311): Terminated {rc=256}
00:00:22.293529 --- ParallelTerminationStrategy: PID 31316 died...
00:00:22.349186 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31316): Terminated: ok {rc=0}
00:00:22.359648 --- ParallelTerminationStrategy: PID 31319 died...
00:00:22.415425 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31319): Terminated: ok {rc=0}
00:00:22.425595 --- ParallelTerminationStrategy: PID 31322 died...
00:00:22.481713 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31322): Terminated: ok {rc=0}
00:00:22.567717 bus /gobi_6: Setting Powered False
213gprs:trx-lms12.0osmo-stp_10.42.42.5(pid=31343): Process ended prematurely: osmo-stp_10.42.42.5(pid=31343) [trial-2351↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31343)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31343): Process ended prematurely: osmo-stp_10.42.42.5(pid=31343) [trial-2351↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31343)]
----------------------------------------------
trial-2351 gprs:trx-lms iperf3m4.py
----------------------------------------------
00:00:23.684739 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:23.696429 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
00:00:23.835154 tst iperf3m4.py: using LAC 17663
00:00:23.953238 tst iperf3m4.py: using RAC 68
00:00:24.072118 tst iperf3m4.py: using CellId 17663
00:00:24.190717 tst iperf3m4.py: using BVCI 17664
00:00:24.213978 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:24.224587 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:24.235086 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:24.245436 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:24.255749 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:24.266233 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:24.276908 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:24.287077 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:24.298115 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:24.413591 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:24.537653 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:24.656584 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:24.949689 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
00:00:25.084501 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:25.417137 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31329): Launched
00:00:25.694050 run iperf3-srv_10.42.42.10(pid=31330): Launched
00:00:25.891910 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
00:00:26.027181 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:26.360311 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31331): Launched
00:00:26.635930 run iperf3-srv_10.42.42.10(pid=31332): Launched
00:00:26.833694 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
00:00:26.968673 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:27.301375 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31333): Launched
00:00:27.578954 run iperf3-srv_10.42.42.10(pid=31334): Launched
00:00:27.774581 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
00:00:27.900627 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:28.212378 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31335): Launched
00:00:28.472375 run iperf3-srv_10.42.42.10(pid=31336): Launched
00:00:28.656619 tst iperf3m4.py:8: start network...
00:00:28.782185 run osmo-hlr_10.42.42.2: Starting osmo-hlr
00:00:29.264571 run create_hlr_db(pid=31337): Launched
00:00:29.439531 bus /gobi_6: Setting Powered False
00:00:30.459369 run create_hlr_db(pid=31337): Terminated: ok {rc=0}
00:00:30.870488 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31339): Launched
00:00:31.327503 run osmo-hlr_10.42.42.2(pid=31340): Launched
00:00:31.468101 run osmo-stp_10.42.42.5: Starting osmo-stp
00:00:32.069672 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31342): Launched
00:00:32.490586 run osmo-stp_10.42.42.5(pid=31343): Launched
00:00:32.610199 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
00:00:33.171205 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31345): Launched
00:00:33.286548 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
00:00:33.567370 run patchelf(pid=31346): Launched
00:00:33.756442 run osmo-stp_10.42.42.5(pid=31343): ERR: Terminated: ERROR {rc=1} [trial-2351↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31343)]
00:00:33.894580 run osmo-stp_10.42.42.5(pid=31343): stdout:
| (launched: 2022-12-09_00:00:32.225439)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:34.033378 run osmo-stp_10.42.42.5(pid=31343): stderr:
| �[0;m�[38;5;43m20221209000032440 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000032440 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000032440 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000032441 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000032441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000032441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000032441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:34.166834 run osmo-stp_10.42.42.5(pid=31343): stdout:
| (launched: 2022-12-09_00:00:32.225439)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:34.298449 run osmo-stp_10.42.42.5(pid=31343): stderr:
| �[0;m�[38;5;43m20221209000032440 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000032440 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000032440 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000032441 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000032441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000032441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000032441 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:34.451722 run patchelf(pid=31346): Terminating (SIGINT)
00:00:34.631850 run patchelf(pid=31346): Terminated: ok {rc=0}
00:00:34.664995 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=31343): Process ended prematurely: osmo-stp_10.42.42.5(pid=31343) [trial-2351↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31343)] [trial-2351↪gprs:trx-lms↪iperf3m4.py:8]
00:00:34.674877 tst iperf3m4.py:8: Test FAILED (11.0 sec)
00:00:34.738703 run osmo-hlr_10.42.42.2(pid=31340): ERR: Terminated: ERROR {rc=237} [trial-2351↪gprs:trx-lms↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31340)]
00:00:34.790186 run osmo-hlr_10.42.42.2(pid=31340): stdout:
| (launched: 2022-12-09_00:00:31.053931)
00:00:34.841405 run osmo-hlr_10.42.42.2(pid=31340): stderr:
| �[0;m�[1;31m20221209000031266 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209000031266 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209000031267 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209000031267 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209000031267 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209000031267 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209000031279 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2351/run.2022-12-08_23-11-41/gprs:trx-lms/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209000031299 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209000031299 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209000031299 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
00:00:34.885599 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31329): Terminating (SIGTERM)
00:00:34.927074 run iperf3-srv_10.42.42.10(pid=31330): Terminating (SIGTERM)
00:00:34.968941 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31331): Terminating (SIGTERM)
00:00:35.010622 run iperf3-srv_10.42.42.10(pid=31332): Terminating (SIGTERM)
00:00:35.052352 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31333): Terminating (SIGTERM)
00:00:35.094088 run iperf3-srv_10.42.42.10(pid=31334): Terminating (SIGTERM)
00:00:35.135873 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31335): Terminating (SIGTERM)
00:00:35.177358 run iperf3-srv_10.42.42.10(pid=31336): Terminating (SIGTERM)
00:00:35.219034 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31339): Terminating (SIGTERM)
00:00:35.260508 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31342): Terminating (SIGTERM)
00:00:35.302106 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31345): Terminating (SIGTERM)
00:00:35.311736 --- ParallelTerminationStrategy: PID 31329 died...
00:00:35.367956 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31329): Terminated: ok {rc=0}
00:00:35.378234 --- ParallelTerminationStrategy: PID 31330 died...
00:00:35.433908 run iperf3-srv_10.42.42.10(pid=31330): Terminated {rc=256}
00:00:35.443727 --- ParallelTerminationStrategy: PID 31331 died...
00:00:35.499317 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31331): Terminated: ok {rc=0}
00:00:35.509814 --- ParallelTerminationStrategy: PID 31332 died...
00:00:35.565191 run iperf3-srv_10.42.42.10(pid=31332): Terminated {rc=256}
00:00:35.575211 --- ParallelTerminationStrategy: PID 31333 died...
00:00:35.631199 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31333): Terminated: ok {rc=0}
00:00:35.641726 --- ParallelTerminationStrategy: PID 31334 died...
00:00:35.696807 run iperf3-srv_10.42.42.10(pid=31334): Terminated {rc=256}
00:00:35.707247 --- ParallelTerminationStrategy: PID 31335 died...
00:00:35.762742 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31335): Terminated: ok {rc=0}
00:00:35.773186 --- ParallelTerminationStrategy: PID 31336 died...
00:00:35.828527 run iperf3-srv_10.42.42.10(pid=31336): Terminated {rc=256}
00:00:35.839117 --- ParallelTerminationStrategy: PID 31339 died...
00:00:35.894523 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31339): Terminated: ok {rc=0}
00:00:35.904875 --- ParallelTerminationStrategy: PID 31342 died...
00:00:35.960765 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31342): Terminated: ok {rc=0}
00:00:35.970983 --- ParallelTerminationStrategy: PID 31345 died...
00:00:36.026800 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31345): Terminated: ok {rc=0}
00:00:36.111887 bus /gobi_4: Setting Powered False
00:00:37.217904 bus /gobi_4: Setting Powered False
00:00:38.323835 bus /gobi_1: Setting Powered False
00:00:39.423969 bus /gobi_1: Setting Powered False
00:00:40.532052 bus /gobi_0: Setting Powered False
00:00:41.632017 bus /gobi_0: Setting Powered False
00:00:42.741381 bus /gobi_6: Setting Powered False
25gprs:trx-lms7.0osmo-stp_10.42.42.5(pid=31353): Process ended prematurely: osmo-stp_10.42.42.5(pid=31353) [trial-2351↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31353)]2327ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31353): Process ended prematurely: osmo-stp_10.42.42.5(pid=31353) [trial-2351↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31353)]
----------------------------------------------
trial-2351 gprs:trx-lms ping.py
----------------------------------------------
00:00:43.808304 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:43.819885 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
00:00:43.963136 tst ping.py: using LAC 17664
00:00:44.086335 tst ping.py: using RAC 69
00:00:44.215292 tst ping.py: using CellId 17664
00:00:44.336890 tst ping.py: using BVCI 17665
00:00:44.361172 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:44.371969 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:44.382856 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:44.393656 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:44.404821 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:44.416850 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:44.428377 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:44.439437 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:44.712433 tst ping.py:19: start network...
00:00:44.835128 run osmo-hlr_10.42.42.2: Starting osmo-hlr
00:00:45.308561 run create_hlr_db(pid=31347): Launched
00:00:45.481582 bus /gobi_6: Setting Powered False
00:00:46.500146 run create_hlr_db(pid=31347): Terminated: ok {rc=0}
00:00:46.918271 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31349): Launched
00:00:47.224728 run osmo-hlr_10.42.42.2(pid=31350): Launched
00:00:47.353809 run osmo-stp_10.42.42.5: Starting osmo-stp
00:00:47.902735 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31352): Launched
00:00:48.199480 run osmo-stp_10.42.42.5(pid=31353): Launched
00:00:48.307106 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
00:00:48.825460 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31355): Launched
00:00:48.930930 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
00:00:49.189818 run patchelf(pid=31356): Launched
00:00:49.364288 run osmo-stp_10.42.42.5(pid=31353): ERR: Terminated: ERROR {rc=1} [trial-2351↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31353)]
00:00:49.493303 run osmo-stp_10.42.42.5(pid=31353): stdout:
| (launched: 2022-12-09_00:00:48.053851)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:49.621583 run osmo-stp_10.42.42.5(pid=31353): stderr:
| �[0;m�[38;5;43m20221209000048225 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000048225 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000048226 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000048226 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000048226 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000048227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000048227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:49.744706 run osmo-stp_10.42.42.5(pid=31353): stdout:
| (launched: 2022-12-09_00:00:48.053851)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:49.866850 run osmo-stp_10.42.42.5(pid=31353): stderr:
| �[0;m�[38;5;43m20221209000048225 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000048225 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000048226 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000048226 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000048226 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000048227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000048227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:50.008719 run patchelf(pid=31356): Terminating (SIGINT)
00:00:50.175231 run patchelf(pid=31356): Terminated: ok {rc=0}
00:00:50.205702 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=31353): Process ended prematurely: osmo-stp_10.42.42.5(pid=31353) [trial-2351↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31353)] [trial-2351↪gprs:trx-lms↪ping.py:22]
00:00:50.214526 tst ping.py:22: Test FAILED (6.4 sec)
00:00:50.273406 run osmo-hlr_10.42.42.2(pid=31350): ERR: Terminated: ERROR {rc=237} [trial-2351↪gprs:trx-lms↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31350)]
00:00:50.321564 run osmo-hlr_10.42.42.2(pid=31350): stdout:
| (launched: 2022-12-09_00:00:47.071096)
00:00:50.370427 run osmo-hlr_10.42.42.2(pid=31350): stderr:
| �[0;m�[1;31m20221209000047366 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209000047367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209000047367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209000047367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209000047367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209000047367 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209000047379 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2351/run.2022-12-08_23-11-41/gprs:trx-lms/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209000047389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209000047389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209000047389 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
00:00:50.411680 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31349): Terminating (SIGTERM)
00:00:50.450706 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31352): Terminating (SIGTERM)
00:00:50.489791 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31355): Terminating (SIGTERM)
00:00:50.498977 --- ParallelTerminationStrategy: PID 31349 died...
00:00:50.551369 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31349): Terminated: ok {rc=0}
00:00:50.560811 --- ParallelTerminationStrategy: PID 31352 died...
00:00:50.613487 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31352): Terminated: ok {rc=0}
00:00:50.623034 --- ParallelTerminationStrategy: PID 31355 died...
00:00:50.674796 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31355): Terminated: ok {rc=0}
00:00:50.761958 bus /gobi_6: Setting Powered False
32gprs:trx-lms7.0osmo-stp_10.42.42.5(pid=31363): Process ended prematurely: osmo-stp_10.42.42.5(pid=31363) [trial-2351↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31363)]2320ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=31363): Process ended prematurely: osmo-stp_10.42.42.5(pid=31363) [trial-2351↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31363)]
----------------------------------------------
trial-2351 gprs:trx-lms ping_idle_ping.py
----------------------------------------------
00:00:51.813101 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:51.824195 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
00:00:51.960615 tst ping_idle_ping.py: using LAC 17665
00:00:52.085848 tst ping_idle_ping.py: using RAC 70
00:00:52.205085 tst ping_idle_ping.py: using CellId 17665
00:00:52.327326 tst ping_idle_ping.py: using BVCI 17666
00:00:52.350292 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:52.360888 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:52.371644 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:52.382578 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:52.392834 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:52.404249 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:52.415395 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:52.425972 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:52.715316 tst ping_idle_ping.py:24: start network...
00:00:52.847909 run osmo-hlr_10.42.42.2: Starting osmo-hlr
00:00:53.348145 run create_hlr_db(pid=31357): Launched
00:00:53.533830 bus /gobi_6: Setting Powered False
00:00:54.554059 run create_hlr_db(pid=31357): Terminated: ok {rc=0}
00:00:54.957732 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31359): Launched
00:00:55.256726 run osmo-hlr_10.42.42.2(pid=31360): Launched
00:00:55.367510 run osmo-stp_10.42.42.5: Starting osmo-stp
00:00:55.886270 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31362): Launched
00:00:56.183747 run osmo-stp_10.42.42.5(pid=31363): Launched
00:00:56.292818 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
00:00:56.812353 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31365): Launched
00:00:56.917927 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
00:00:57.177712 run patchelf(pid=31366): Launched
00:00:57.351563 run osmo-stp_10.42.42.5(pid=31363): ERR: Terminated: ERROR {rc=1} [trial-2351↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31363)]
00:00:57.479387 run osmo-stp_10.42.42.5(pid=31363): stdout:
| (launched: 2022-12-09_00:00:56.037265)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:57.607578 run osmo-stp_10.42.42.5(pid=31363): stderr:
| �[0;m�[38;5;43m20221209000056207 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000056207 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000056207 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000056208 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000056210 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000056210 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000056210 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:57.730538 run osmo-stp_10.42.42.5(pid=31363): stdout:
| (launched: 2022-12-09_00:00:56.037265)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:57.851837 run osmo-stp_10.42.42.5(pid=31363): stderr:
| �[0;m�[38;5;43m20221209000056207 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209000056207 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209000056207 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209000056208 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209000056210 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209000056210 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209000056210 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:57.993577 run patchelf(pid=31366): Terminating (SIGINT)
00:00:58.159856 run patchelf(pid=31366): Terminated: ok {rc=0}
00:00:58.190536 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=31363): Process ended prematurely: osmo-stp_10.42.42.5(pid=31363) [trial-2351↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=31363)] [trial-2351↪gprs:trx-lms↪ping_idle_ping.py:27]
00:00:58.199090 tst ping_idle_ping.py:27: Test FAILED (6.4 sec)
00:00:58.257764 run osmo-hlr_10.42.42.2(pid=31360): ERR: Terminated: ERROR {rc=237} [trial-2351↪gprs:trx-lms↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=31360)]
00:00:58.305838 run osmo-hlr_10.42.42.2(pid=31360): stdout:
| (launched: 2022-12-09_00:00:55.109496)
00:00:58.354544 run osmo-hlr_10.42.42.2(pid=31360): stderr:
| �[0;m�[1;31m20221209000055285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209000055285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209000055285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209000055285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209000055285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209000055285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209000055300 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2351/run.2022-12-08_23-11-41/gprs:trx-lms/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209000055311 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209000055311 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209000055311 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
00:00:58.395970 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31359): Terminating (SIGTERM)
00:00:58.435077 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31362): Terminating (SIGTERM)
00:00:58.474418 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31365): Terminating (SIGTERM)
00:00:58.483489 --- ParallelTerminationStrategy: PID 31359 died...
00:00:58.535899 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31359): Terminated: ok {rc=0}
00:00:58.545568 --- ParallelTerminationStrategy: PID 31362 died...
00:00:58.597667 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31362): Terminated: ok {rc=0}
00:00:58.607093 --- ParallelTerminationStrategy: PID 31365 died...
00:00:58.659272 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31365): Terminated: ok {rc=0}
00:00:58.741854 bus /gobi_6: Setting Powered False
5gprs:trx-lms00