225gprs:trx-lms9.0osmo-stp_10.42.42.5(pid=6597): Process ended prematurely: osmo-stp_10.42.42.5(pid=6597) [trial-2363↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6597)]2139cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=6597): Process ended prematurely: osmo-stp_10.42.42.5(pid=6597) [trial-2363↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6597)]
----------------------------------------------
trial-2363 gprs:trx-lms cs_paging_gprs_active.py
----------------------------------------------
19:25:41.727770 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:41.738763 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
19:25:41.880228 tst cs_paging_gprs_active.py: using LAC 23193
19:25:42.015312 tst cs_paging_gprs_active.py: using RAC 243
19:25:42.144821 tst cs_paging_gprs_active.py: using CellId 23193
19:25:42.269014 tst cs_paging_gprs_active.py: using BVCI 23194
19:25:42.293276 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:42.304079 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:42.314773 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:42.325668 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:42.336451 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:42.347230 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:42.358311 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:42.369212 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:42.380123 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
19:25:42.494827 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
19:25:42.786061 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
19:25:42.919920 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:25:43.249511 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6587): Launched
19:25:43.522392 run iperf3-srv_10.42.42.10(pid=6588): Launched
19:25:43.717423 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
19:25:43.850539 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:25:44.178933 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6589): Launched
19:25:44.451093 run iperf3-srv_10.42.42.10(pid=6590): Launched
19:25:44.646279 tst cs_paging_gprs_active.py:38: start network...
19:25:44.779307 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:25:45.278716 run create_hlr_db(pid=6591): Launched
19:25:45.463015 bus /sierra_2: Setting Powered False
19:25:46.485886 run create_hlr_db(pid=6591): Terminated: ok {rc=0}
19:25:46.923496 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6593): Launched
19:25:47.259578 run osmo-hlr_10.42.42.2(pid=6594): Launched
19:25:47.371187 run osmo-stp_10.42.42.5: Starting osmo-stp
19:25:47.887613 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6596): Launched
19:25:48.183683 run osmo-stp_10.42.42.5(pid=6597): Launched
19:25:48.292185 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:25:48.807880 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6599): Launched
19:25:48.913836 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:25:49.172405 run patchelf(pid=6600): Launched
19:25:49.346090 run osmo-stp_10.42.42.5(pid=6597): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6597)]
19:25:49.473266 run osmo-stp_10.42.42.5(pid=6597): stdout:
| (launched: 2022-12-09_19:25:48.038556)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:25:49.600682 run osmo-stp_10.42.42.5(pid=6597): stderr:
| �[0;m�[38;5;43m20221209192548206 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192548206 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192548206 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192548207 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192548207 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192548207 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192548207 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:25:49.723526 run osmo-stp_10.42.42.5(pid=6597): stdout:
| (launched: 2022-12-09_19:25:48.038556)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:25:49.845319 run osmo-stp_10.42.42.5(pid=6597): stderr:
| �[0;m�[38;5;43m20221209192548206 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192548206 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192548206 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192548207 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192548207 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192548207 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192548207 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:25:49.987031 run patchelf(pid=6600): Terminating (SIGINT)
19:25:50.153760 run patchelf(pid=6600): Terminated: ok {rc=0}
19:25:50.184359 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=6597): Process ended prematurely: osmo-stp_10.42.42.5(pid=6597) [trial-2363↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6597)] [trial-2363↪gprs:trx-lms↪cs_paging_gprs_active.py:38]
19:25:50.193454 tst cs_paging_gprs_active.py:38: Test FAILED (8.5 sec)
19:25:50.252159 run osmo-hlr_10.42.42.2(pid=6594): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:trx-lms↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=6594)]
19:25:50.299767 run osmo-hlr_10.42.42.2(pid=6594): stdout:
| (launched: 2022-12-09_19:25:47.088278)
19:25:50.347908 run osmo-hlr_10.42.42.2(pid=6594): stderr:
| �[0;m�[1;31m20221209192547285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209192547285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209192547285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209192547285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209192547285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209192547285 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209192547297 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:trx-lms/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209192547306 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209192547307 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209192547307 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:25:50.388974 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6587): Terminating (SIGTERM)
19:25:50.427375 run iperf3-srv_10.42.42.10(pid=6588): Terminating (SIGTERM)
19:25:50.466185 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6589): Terminating (SIGTERM)
19:25:50.504583 run iperf3-srv_10.42.42.10(pid=6590): Terminating (SIGTERM)
19:25:50.542909 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6593): Terminating (SIGTERM)
19:25:50.581352 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6596): Terminating (SIGTERM)
19:25:50.619792 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6599): Terminating (SIGTERM)
19:25:50.628760 --- ParallelTerminationStrategy: PID 6587 died...
19:25:50.680913 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6587): Terminated: ok {rc=0}
19:25:50.690525 --- ParallelTerminationStrategy: PID 6588 died...
19:25:50.741937 run iperf3-srv_10.42.42.10(pid=6588): Terminated {rc=256}
19:25:50.751294 --- ParallelTerminationStrategy: PID 6589 died...
19:25:50.803142 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6589): Terminated: ok {rc=0}
19:25:50.812622 --- ParallelTerminationStrategy: PID 6590 died...
19:25:50.864342 run iperf3-srv_10.42.42.10(pid=6590): Terminated {rc=256}
19:25:50.873683 --- ParallelTerminationStrategy: PID 6593 died...
19:25:50.925251 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6593): Terminated: ok {rc=0}
19:25:50.934662 --- ParallelTerminationStrategy: PID 6596 died...
19:25:50.986105 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6596): Terminated: ok {rc=0}
19:25:50.995431 --- ParallelTerminationStrategy: PID 6599 died...
19:25:51.047317 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6599): Terminated: ok {rc=0}
19:25:51.134373 bus /gobi_0: Setting Powered False
19:25:52.236679 bus /gobi_0: Setting Powered False
19:25:53.346918 bus /gobi_6: Setting Powered False
222gprs:trx-lms8.0osmo-stp_10.42.42.5(pid=6609): Process ended prematurely: osmo-stp_10.42.42.5(pid=6609) [trial-2363↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6609)]2142iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=6609): Process ended prematurely: osmo-stp_10.42.42.5(pid=6609) [trial-2363↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6609)]
----------------------------------------------
trial-2363 gprs:trx-lms iperf3.py
----------------------------------------------
19:25:54.454716 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:54.465578 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
19:25:54.603881 tst iperf3.py: using LAC 23194
19:25:54.718351 tst iperf3.py: using RAC 244
19:25:54.833427 tst iperf3.py: using CellId 23194
19:25:54.949366 tst iperf3.py: using BVCI 23195
19:25:54.971835 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:54.982302 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:54.992375 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:55.002523 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:55.012503 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:55.022584 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:55.032892 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:55.042943 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:25:55.053107 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
19:25:55.330459 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
19:25:55.455689 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:25:55.763471 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6601): Launched
19:25:56.018486 run iperf3-srv_10.42.42.10(pid=6602): Launched
19:25:56.199848 tst iperf3.py:8: start network...
19:25:56.324068 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:25:56.799193 run create_hlr_db(pid=6603): Launched
19:25:56.970906 bus /gobi_6: Setting Powered False
19:25:57.991957 run create_hlr_db(pid=6603): Terminated: ok {rc=0}
19:25:58.414916 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6605): Launched
19:25:58.734685 run osmo-hlr_10.42.42.2(pid=6606): Launched
19:25:58.851195 run osmo-stp_10.42.42.5: Starting osmo-stp
19:25:59.406567 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6608): Launched
19:25:59.726057 run osmo-stp_10.42.42.5(pid=6609): Launched
19:25:59.844286 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:26:00.398520 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6611): Launched
19:26:00.512372 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:26:00.788276 run patchelf(pid=6612): Launched
19:26:00.982736 run osmo-stp_10.42.42.5(pid=6609): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6609)]
19:26:01.122552 run osmo-stp_10.42.42.5(pid=6609): stdout:
| (launched: 2022-12-09_19:25:59.569224)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:26:01.262683 run osmo-stp_10.42.42.5(pid=6609): stderr:
| �[0;m�[38;5;43m20221209192559756 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192559756 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192559757 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192559757 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192559757 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192559758 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192559758 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:26:01.398053 run osmo-stp_10.42.42.5(pid=6609): stdout:
| (launched: 2022-12-09_19:25:59.569224)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:26:01.532602 run osmo-stp_10.42.42.5(pid=6609): stderr:
| �[0;m�[38;5;43m20221209192559756 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192559756 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192559757 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192559757 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192559757 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192559758 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192559758 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:26:01.688354 run patchelf(pid=6612): Terminating (SIGINT)
19:26:01.872351 run patchelf(pid=6612): Terminated: ok {rc=0}
19:26:01.906657 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=6609): Process ended prematurely: osmo-stp_10.42.42.5(pid=6609) [trial-2363↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6609)] [trial-2363↪gprs:trx-lms↪iperf3.py:8]
19:26:01.917209 tst iperf3.py:8: Test FAILED (7.5 sec)
19:26:01.983251 run osmo-hlr_10.42.42.2(pid=6606): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:trx-lms↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=6606)]
19:26:02.035844 run osmo-hlr_10.42.42.2(pid=6606): stdout:
| (launched: 2022-12-09_19:25:58.577633)
19:26:02.089694 run osmo-hlr_10.42.42.2(pid=6606): stderr:
| �[0;m�[1;31m20221209192558770 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209192558770 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209192558770 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209192558770 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209192558770 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209192558770 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209192558782 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:trx-lms/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209192558791 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209192558791 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209192558791 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:26:02.135155 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6601): Terminating (SIGTERM)
19:26:02.179575 run iperf3-srv_10.42.42.10(pid=6602): Terminating (SIGTERM)
19:26:02.223093 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6605): Terminating (SIGTERM)
19:26:02.265174 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6608): Terminating (SIGTERM)
19:26:02.305927 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6611): Terminating (SIGTERM)
19:26:02.316111 --- ParallelTerminationStrategy: PID 6601 died...
19:26:02.371222 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6601): Terminated: ok {rc=0}
19:26:02.381843 --- ParallelTerminationStrategy: PID 6602 died...
19:26:02.437026 run iperf3-srv_10.42.42.10(pid=6602): Terminated {rc=256}
19:26:02.446856 --- ParallelTerminationStrategy: PID 6605 died...
19:26:02.502358 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6605): Terminated: ok {rc=0}
19:26:02.512188 --- ParallelTerminationStrategy: PID 6608 died...
19:26:02.566932 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6608): Terminated: ok {rc=0}
19:26:02.577885 --- ParallelTerminationStrategy: PID 6611 died...
19:26:02.633387 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6611): Terminated: ok {rc=0}
19:26:02.717740 bus /gobi_6: Setting Powered False
225gprs:trx-lms11.0osmo-stp_10.42.42.5(pid=6627): Process ended prematurely: osmo-stp_10.42.42.5(pid=6627) [trial-2363↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6627)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=6627): Process ended prematurely: osmo-stp_10.42.42.5(pid=6627) [trial-2363↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6627)]
----------------------------------------------
trial-2363 gprs:trx-lms iperf3m4.py
----------------------------------------------
19:26:03.826776 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:03.837855 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
19:26:03.974868 tst iperf3m4.py: using LAC 23195
19:26:04.090615 tst iperf3m4.py: using RAC 245
19:26:04.206051 tst iperf3m4.py: using CellId 23195
19:26:04.323897 tst iperf3m4.py: using BVCI 23196
19:26:04.347388 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:04.358160 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:04.368504 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:04.379121 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:04.389601 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:04.399756 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:04.410668 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:04.421107 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:04.431699 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
19:26:04.544483 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
19:26:04.663795 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
19:26:04.781640 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
19:26:05.070701 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
19:26:05.204389 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:26:05.532685 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6613): Launched
19:26:05.805202 run iperf3-srv_10.42.42.10(pid=6614): Launched
19:26:06.000893 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
19:26:06.135033 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:26:06.463755 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6615): Launched
19:26:06.736504 run iperf3-srv_10.42.42.10(pid=6616): Launched
19:26:06.932513 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
19:26:07.066630 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:26:07.395350 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6617): Launched
19:26:07.666942 run iperf3-srv_10.42.42.10(pid=6618): Launched
19:26:07.862114 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
19:26:07.994999 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:26:08.323092 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6619): Launched
19:26:08.594957 run iperf3-srv_10.42.42.10(pid=6620): Launched
19:26:08.790208 tst iperf3m4.py:8: start network...
19:26:08.923643 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:26:09.422730 run create_hlr_db(pid=6621): Launched
19:26:09.606989 bus /gobi_6: Setting Powered False
19:26:10.627939 run create_hlr_db(pid=6621): Terminated: ok {rc=0}
19:26:11.025087 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6623): Launched
19:26:11.320801 run osmo-hlr_10.42.42.2(pid=6624): Launched
19:26:11.429882 run osmo-stp_10.42.42.5: Starting osmo-stp
19:26:11.944929 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6626): Launched
19:26:12.242581 run osmo-stp_10.42.42.5(pid=6627): Launched
19:26:12.351802 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:26:12.874138 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6629): Launched
19:26:12.980821 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:26:13.242834 run patchelf(pid=6630): Launched
19:26:13.419064 run osmo-stp_10.42.42.5(pid=6627): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6627)]
19:26:13.547780 run osmo-stp_10.42.42.5(pid=6627): stdout:
| (launched: 2022-12-09_19:26:12.096219)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:26:13.677289 run osmo-stp_10.42.42.5(pid=6627): stderr:
| �[0;m�[38;5;43m20221209192612262 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192612262 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192612262 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192612262 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192612263 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192612263 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192612263 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:26:13.801619 run osmo-stp_10.42.42.5(pid=6627): stdout:
| (launched: 2022-12-09_19:26:12.096219)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:26:13.925220 run osmo-stp_10.42.42.5(pid=6627): stderr:
| �[0;m�[38;5;43m20221209192612262 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192612262 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192612262 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192612262 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192612263 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192612263 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192612263 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:26:14.068272 run patchelf(pid=6630): Terminating (SIGINT)
19:26:14.236784 run patchelf(pid=6630): Terminated: ok {rc=0}
19:26:14.267890 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=6627): Process ended prematurely: osmo-stp_10.42.42.5(pid=6627) [trial-2363↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6627)] [trial-2363↪gprs:trx-lms↪iperf3m4.py:8]
19:26:14.277094 tst iperf3m4.py:8: Test FAILED (10.5 sec)
19:26:14.335821 run osmo-hlr_10.42.42.2(pid=6624): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:trx-lms↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=6624)]
19:26:14.383656 run osmo-hlr_10.42.42.2(pid=6624): stdout:
| (launched: 2022-12-09_19:26:11.176390)
19:26:14.432386 run osmo-hlr_10.42.42.2(pid=6624): stderr:
| �[0;m�[1;31m20221209192611346 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209192611346 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209192611346 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209192611347 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209192611347 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209192611347 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209192611358 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:trx-lms/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209192611367 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209192611367 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209192611368 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:26:14.473549 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6613): Terminating (SIGTERM)
19:26:14.512347 run iperf3-srv_10.42.42.10(pid=6614): Terminating (SIGTERM)
19:26:14.550894 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6615): Terminating (SIGTERM)
19:26:14.589604 run iperf3-srv_10.42.42.10(pid=6616): Terminating (SIGTERM)
19:26:14.628908 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6617): Terminating (SIGTERM)
19:26:14.667750 run iperf3-srv_10.42.42.10(pid=6618): Terminating (SIGTERM)
19:26:14.706525 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6619): Terminating (SIGTERM)
19:26:14.745020 run iperf3-srv_10.42.42.10(pid=6620): Terminating (SIGTERM)
19:26:14.783687 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6623): Terminating (SIGTERM)
19:26:14.822351 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6626): Terminating (SIGTERM)
19:26:14.861054 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6629): Terminating (SIGTERM)
19:26:14.870067 --- ParallelTerminationStrategy: PID 6613 died...
19:26:14.921915 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6613): Terminated: ok {rc=0}
19:26:14.931305 --- ParallelTerminationStrategy: PID 6614 died...
19:26:14.982638 run iperf3-srv_10.42.42.10(pid=6614): Terminated {rc=256}
19:26:14.991800 --- ParallelTerminationStrategy: PID 6615 died...
19:26:15.043263 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6615): Terminated: ok {rc=0}
19:26:15.052550 --- ParallelTerminationStrategy: PID 6616 died...
19:26:15.104302 run iperf3-srv_10.42.42.10(pid=6616): Terminated {rc=256}
19:26:15.113794 --- ParallelTerminationStrategy: PID 6617 died...
19:26:15.164992 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6617): Terminated: ok {rc=0}
19:26:15.174301 --- ParallelTerminationStrategy: PID 6618 died...
19:26:15.226091 run iperf3-srv_10.42.42.10(pid=6618): Terminated {rc=256}
19:26:15.235603 --- ParallelTerminationStrategy: PID 6619 died...
19:26:15.286918 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6619): Terminated: ok {rc=0}
19:26:15.296302 --- ParallelTerminationStrategy: PID 6620 died...
19:26:15.347700 run iperf3-srv_10.42.42.10(pid=6620): Terminated {rc=256}
19:26:15.357124 --- ParallelTerminationStrategy: PID 6623 died...
19:26:15.408535 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6623): Terminated: ok {rc=0}
19:26:15.417965 --- ParallelTerminationStrategy: PID 6626 died...
19:26:15.469501 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6626): Terminated: ok {rc=0}
19:26:15.478957 --- ParallelTerminationStrategy: PID 6629 died...
19:26:15.530429 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6629): Terminated: ok {rc=0}
19:26:15.617357 bus /gobi_4: Setting Powered False
19:26:16.718558 bus /gobi_4: Setting Powered False
19:26:17.823062 bus /gobi_1: Setting Powered False
19:26:18.925406 bus /gobi_1: Setting Powered False
19:26:20.030564 bus /gobi_0: Setting Powered False
19:26:21.135959 bus /gobi_0: Setting Powered False
19:26:22.246781 bus /gobi_6: Setting Powered False
37gprs:trx-lms7.0osmo-stp_10.42.42.5(pid=6637): Process ended prematurely: osmo-stp_10.42.42.5(pid=6637) [trial-2363↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6637)]2327ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=6637): Process ended prematurely: osmo-stp_10.42.42.5(pid=6637) [trial-2363↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6637)]
----------------------------------------------
trial-2363 gprs:trx-lms ping.py
----------------------------------------------
19:26:23.315276 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:23.326953 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
19:26:23.464347 tst ping.py: using LAC 23196
19:26:23.580858 tst ping.py: using RAC 246
19:26:23.699167 tst ping.py: using CellId 23196
19:26:23.815446 tst ping.py: using BVCI 23197
19:26:23.838442 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:23.848857 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:23.859336 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:23.869539 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:23.879527 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:23.889693 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:23.899883 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:23.910101 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
19:26:24.181825 tst ping.py:19: start network...
19:26:24.304707 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:26:24.779089 run create_hlr_db(pid=6631): Launched
19:26:24.950044 bus /gobi_6: Setting Powered False
19:26:25.971892 run create_hlr_db(pid=6631): Terminated: ok {rc=0}
19:26:26.366231 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6633): Launched
19:26:26.663972 run osmo-hlr_10.42.42.2(pid=6634): Launched
19:26:26.779941 run osmo-stp_10.42.42.5: Starting osmo-stp
19:26:27.306299 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6636): Launched
19:26:27.604089 run osmo-stp_10.42.42.5(pid=6637): Launched
19:26:27.711362 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:26:28.228848 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6639): Launched
19:26:28.334267 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:26:28.594065 run patchelf(pid=6640): Launched
19:26:28.767893 run osmo-stp_10.42.42.5(pid=6637): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6637)]
19:26:28.895317 run osmo-stp_10.42.42.5(pid=6637): stdout:
| (launched: 2022-12-09_19:26:27.457349)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:26:29.022817 run osmo-stp_10.42.42.5(pid=6637): stderr:
| �[0;m�[38;5;43m20221209192627630 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192627630 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192627630 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192627630 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192627631 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192627631 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192627631 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:26:29.146366 run osmo-stp_10.42.42.5(pid=6637): stdout:
| (launched: 2022-12-09_19:26:27.457349)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:26:29.268537 run osmo-stp_10.42.42.5(pid=6637): stderr:
| �[0;m�[38;5;43m20221209192627630 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192627630 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192627630 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192627630 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192627631 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192627631 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192627631 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:26:29.410356 run patchelf(pid=6640): Terminating (SIGINT)
19:26:29.576767 run patchelf(pid=6640): Terminated: ok {rc=0}
19:26:29.607268 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=6637): Process ended prematurely: osmo-stp_10.42.42.5(pid=6637) [trial-2363↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6637)] [trial-2363↪gprs:trx-lms↪ping.py:22]
19:26:29.615966 tst ping.py:22: Test FAILED (6.3 sec)
19:26:29.675088 run osmo-hlr_10.42.42.2(pid=6634): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:trx-lms↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=6634)]
19:26:29.723371 run osmo-hlr_10.42.42.2(pid=6634): stdout:
| (launched: 2022-12-09_19:26:26.517646)
19:26:29.772225 run osmo-hlr_10.42.42.2(pid=6634): stderr:
| �[0;m�[1;31m20221209192626694 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209192626694 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209192626694 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209192626694 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209192626694 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209192626694 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209192626706 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:trx-lms/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209192626715 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209192626715 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209192626715 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:26:29.813783 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6633): Terminating (SIGTERM)
19:26:29.852895 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6636): Terminating (SIGTERM)
19:26:29.891762 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6639): Terminating (SIGTERM)
19:26:29.900848 --- ParallelTerminationStrategy: PID 6633 died...
19:26:29.953253 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6633): Terminated: ok {rc=0}
19:26:29.962785 --- ParallelTerminationStrategy: PID 6636 died...
19:26:30.014740 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6636): Terminated: ok {rc=0}
19:26:30.024261 --- ParallelTerminationStrategy: PID 6639 died...
19:26:30.076241 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6639): Terminated: ok {rc=0}
19:26:30.163123 bus /gobi_6: Setting Powered False
44gprs:trx-lms7.0osmo-stp_10.42.42.5(pid=6647): Process ended prematurely: osmo-stp_10.42.42.5(pid=6647) [trial-2363↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6647)]2320ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=6647): Process ended prematurely: osmo-stp_10.42.42.5(pid=6647) [trial-2363↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6647)]
----------------------------------------------
trial-2363 gprs:trx-lms ping_idle_ping.py
----------------------------------------------
19:26:31.215111 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:31.227903 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
19:26:31.407961 tst ping_idle_ping.py: using LAC 23197
19:26:31.565290 tst ping_idle_ping.py: using RAC 247
19:26:31.716432 tst ping_idle_ping.py: using CellId 23197
19:26:31.850773 tst ping_idle_ping.py: using BVCI 23198
19:26:31.877383 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:31.889888 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:31.902071 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:31.915473 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:31.927752 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:31.940645 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:31.953730 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
19:26:31.966789 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
19:26:32.265936 tst ping_idle_ping.py:24: start network...
19:26:32.390654 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:26:32.872479 run create_hlr_db(pid=6641): Launched
19:26:33.045207 bus /gobi_6: Setting Powered False
19:26:34.066323 run create_hlr_db(pid=6641): Terminated: ok {rc=0}
19:26:34.491416 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6643): Launched
19:26:34.811298 run osmo-hlr_10.42.42.2(pid=6644): Launched
19:26:34.927410 run osmo-stp_10.42.42.5: Starting osmo-stp
19:26:35.482351 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6646): Launched
19:26:35.801927 run osmo-stp_10.42.42.5(pid=6647): Launched
19:26:35.918031 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:26:36.474611 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6649): Launched
19:26:36.588278 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:26:36.866526 run patchelf(pid=6650): Launched
19:26:37.052852 run osmo-stp_10.42.42.5(pid=6647): ERR: Terminated: ERROR {rc=1} [trial-2363↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6647)]
19:26:37.189166 run osmo-stp_10.42.42.5(pid=6647): stdout:
| (launched: 2022-12-09_19:26:35.644579)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:26:37.326121 run osmo-stp_10.42.42.5(pid=6647): stderr:
| �[0;m�[38;5;43m20221209192635836 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192635836 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192635836 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192635837 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192635837 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192635837 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192635837 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:26:37.458108 run osmo-stp_10.42.42.5(pid=6647): stdout:
| (launched: 2022-12-09_19:26:35.644579)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:26:37.588918 run osmo-stp_10.42.42.5(pid=6647): stderr:
| �[0;m�[38;5;43m20221209192635836 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209192635836 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209192635836 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209192635837 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209192635837 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209192635837 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209192635837 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:26:37.740754 run patchelf(pid=6650): Terminating (SIGINT)
19:26:37.918571 run patchelf(pid=6650): Terminated: ok {rc=0}
19:26:37.951156 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=6647): Process ended prematurely: osmo-stp_10.42.42.5(pid=6647) [trial-2363↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6647)] [trial-2363↪gprs:trx-lms↪ping_idle_ping.py:27]
19:26:37.960479 tst ping_idle_ping.py:27: Test FAILED (6.7 sec)
19:26:38.023707 run osmo-hlr_10.42.42.2(pid=6644): ERR: Terminated: ERROR {rc=237} [trial-2363↪gprs:trx-lms↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=6644)]
19:26:38.074549 run osmo-hlr_10.42.42.2(pid=6644): stdout:
| (launched: 2022-12-09_19:26:34.654362)
19:26:38.126157 run osmo-hlr_10.42.42.2(pid=6644): stderr:
| �[0;m�[1;31m20221209192634854 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209192634854 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209192634855 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209192634855 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209192634855 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209192634855 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209192634868 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2363/run.2022-12-09_18-37-22/gprs:trx-lms/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209192634878 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209192634878 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209192634878 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:26:38.169613 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6643): Terminating (SIGTERM)
19:26:38.211685 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6646): Terminating (SIGTERM)
19:26:38.253318 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6649): Terminating (SIGTERM)
19:26:38.263016 --- ParallelTerminationStrategy: PID 6643 died...
19:26:38.318936 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6643): Terminated: ok {rc=0}
19:26:38.329065 --- ParallelTerminationStrategy: PID 6646 died...
19:26:38.384462 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6646): Terminated: ok {rc=0}
19:26:38.394802 --- ParallelTerminationStrategy: PID 6649 died...
19:26:38.450047 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6649): Terminated: ok {rc=0}
19:26:38.535820 bus /gobi_6: Setting Powered False
5gprs:trx-lms00