258gprs:trx-lms9.0osmo-stp_10.42.42.5(pid=15937): Process ended prematurely: osmo-stp_10.42.42.5(pid=15937) [trial-2396↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=15937)]2139cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=15937): Process ended prematurely: osmo-stp_10.42.42.5(pid=15937) [trial-2396↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=15937)]
----------------------------------------------
trial-2396 gprs:trx-lms cs_paging_gprs_active.py
----------------------------------------------
23:59:48.864331 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:48.874884 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
23:59:49.014953 tst cs_paging_gprs_active.py: using LAC 37137
23:59:49.140143 tst cs_paging_gprs_active.py: using RAC 162
23:59:49.265806 tst cs_paging_gprs_active.py: using CellId 37137
23:59:49.390118 tst cs_paging_gprs_active.py: using BVCI 37138
23:59:49.414445 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:49.425669 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:49.436827 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:49.447707 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:49.458944 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:49.470097 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:49.481430 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:49.492341 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
23:59:49.503460 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
23:59:49.620605 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
23:59:49.910719 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
23:59:50.044294 run iperf3-srv_10.42.42.10: Starting iperf3-srv
23:59:50.373630 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15927): Launched
23:59:50.646440 run iperf3-srv_10.42.42.10(pid=15928): Launched
23:59:50.841678 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
23:59:50.974965 run iperf3-srv_10.42.42.10: Starting iperf3-srv
23:59:51.303808 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15929): Launched
23:59:51.576705 run iperf3-srv_10.42.42.10(pid=15930): Launched
23:59:51.772701 tst cs_paging_gprs_active.py:38: start network...
23:59:51.906266 run osmo-hlr_10.42.42.2: Starting osmo-hlr
23:59:52.406044 run create_hlr_db(pid=15931): Launched
23:59:52.590225 bus /sierra_2: Setting Powered False
23:59:53.612187 run create_hlr_db(pid=15931): Terminated: ok {rc=0}
23:59:54.011303 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15933): Launched
23:59:54.308741 run osmo-hlr_10.42.42.2(pid=15934): Launched
23:59:54.418807 run osmo-stp_10.42.42.5: Starting osmo-stp
23:59:54.933345 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15936): Launched
23:59:55.230536 run osmo-stp_10.42.42.5(pid=15937): Launched
23:59:55.339510 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
23:59:55.858884 run pcap-recorder_any(filters='host 10.42.42.6')(pid=15939): Launched
23:59:55.965672 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
23:59:56.225126 run patchelf(pid=15940): Launched
23:59:56.399180 run osmo-stp_10.42.42.5(pid=15937): ERR: Terminated: ERROR {rc=1} [trial-2396↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=15937)]
23:59:56.526512 run osmo-stp_10.42.42.5(pid=15937): stdout:
| (launched: 2022-12-13_23:59:55.084321)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
23:59:56.654294 run osmo-stp_10.42.42.5(pid=15937): stderr:
| �[0;m�[38;5;43m20221213235955253 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213235955253 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213235955254 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213235955254 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213235955254 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213235955255 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213235955255 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
23:59:56.777641 run osmo-stp_10.42.42.5(pid=15937): stdout:
| (launched: 2022-12-13_23:59:55.084321)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
23:59:56.899692 run osmo-stp_10.42.42.5(pid=15937): stderr:
| �[0;m�[38;5;43m20221213235955253 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213235955253 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213235955254 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213235955254 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213235955254 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213235955255 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213235955255 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
23:59:57.041897 run patchelf(pid=15940): Terminating (SIGINT)
23:59:57.209081 run patchelf(pid=15940): Terminated: ok {rc=0}
23:59:57.239746 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=15937): Process ended prematurely: osmo-stp_10.42.42.5(pid=15937) [trial-2396↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=15937)] [trial-2396↪gprs:trx-lms↪cs_paging_gprs_active.py:38]
23:59:57.248839 tst cs_paging_gprs_active.py:38: Test FAILED (8.4 sec)
23:59:57.308844 run osmo-hlr_10.42.42.2(pid=15934): ERR: Terminated: ERROR {rc=237} [trial-2396↪gprs:trx-lms↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=15934)]
23:59:57.357777 run osmo-hlr_10.42.42.2(pid=15934): stdout:
| (launched: 2022-12-13_23:59:54.163656)
23:59:57.407291 run osmo-hlr_10.42.42.2(pid=15934): stderr:
| �[0;m�[1;31m20221213235954338 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213235954338 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213235954338 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213235954338 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213235954339 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213235954339 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213235954351 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2396/run.2022-12-13_23-11-51/gprs:trx-lms/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213235954360 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213235954360 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213235954360 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
23:59:57.449608 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15927): Terminating (SIGTERM)
23:59:57.489325 run iperf3-srv_10.42.42.10(pid=15928): Terminating (SIGTERM)
23:59:57.529098 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15929): Terminating (SIGTERM)
23:59:57.568763 run iperf3-srv_10.42.42.10(pid=15930): Terminating (SIGTERM)
23:59:57.608328 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15933): Terminating (SIGTERM)
23:59:57.648412 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15936): Terminating (SIGTERM)
23:59:57.688105 run pcap-recorder_any(filters='host 10.42.42.6')(pid=15939): Terminating (SIGTERM)
23:59:57.697373 --- ParallelTerminationStrategy: PID 15927 died...
23:59:57.750549 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15927): Terminated: ok {rc=0}
23:59:57.760060 --- ParallelTerminationStrategy: PID 15928 died...
23:59:57.812909 run iperf3-srv_10.42.42.10(pid=15928): Terminated {rc=256}
23:59:57.822421 --- ParallelTerminationStrategy: PID 15929 died...
23:59:57.875357 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15929): Terminated: ok {rc=0}
23:59:57.884818 --- ParallelTerminationStrategy: PID 15930 died...
23:59:57.937495 run iperf3-srv_10.42.42.10(pid=15930): Terminated {rc=256}
23:59:57.947010 --- ParallelTerminationStrategy: PID 15933 died...
23:59:57.999778 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15933): Terminated: ok {rc=0}
23:59:58.009367 --- ParallelTerminationStrategy: PID 15936 died...
23:59:58.062070 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15936): Terminated: ok {rc=0}
23:59:58.071616 --- ParallelTerminationStrategy: PID 15939 died...
23:59:58.124372 run pcap-recorder_any(filters='host 10.42.42.6')(pid=15939): Terminated: ok {rc=0}
23:59:58.212103 bus /gobi_0: Setting Powered False
23:59:59.314165 bus /gobi_0: Setting Powered False
00:00:00.420846 bus /gobi_6: Setting Powered False
255gprs:trx-lms9.0osmo-stp_10.42.42.5(pid=15988): Process ended prematurely: osmo-stp_10.42.42.5(pid=15988) [trial-2396↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=15988)]2142iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=15988): Process ended prematurely: osmo-stp_10.42.42.5(pid=15988) [trial-2396↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=15988)]
----------------------------------------------
trial-2396 gprs:trx-lms iperf3.py
----------------------------------------------
00:00:01.570102 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:01.586438 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
00:00:01.753726 tst iperf3.py: using LAC 37138
00:00:01.907857 tst iperf3.py: using RAC 163
00:00:02.077353 tst iperf3.py: using CellId 37138
00:00:02.259413 tst iperf3.py: using BVCI 37139
00:00:02.307621 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:02.323480 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:02.338750 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:02.355866 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:02.370074 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:02.384060 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:02.397756 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:02.413630 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:02.426702 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:02.799772 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
00:00:02.953710 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:03.375372 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15971): Launched
00:00:03.720866 run iperf3-srv_10.42.42.10(pid=15973): Launched
00:00:03.947564 tst iperf3.py:8: start network...
00:00:04.094059 run osmo-hlr_10.42.42.2: Starting osmo-hlr
00:00:04.630815 run create_hlr_db(pid=15975): Launched
00:00:04.839098 bus /gobi_6: Setting Powered False
00:00:05.827712 run create_hlr_db(pid=15975): Terminated: ok {rc=0}
00:00:06.281485 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15984): Launched
00:00:06.600553 run osmo-hlr_10.42.42.2(pid=15985): Launched
00:00:06.721928 run osmo-stp_10.42.42.5: Starting osmo-stp
00:00:07.246315 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15987): Launched
00:00:07.543592 run osmo-stp_10.42.42.5(pid=15988): Launched
00:00:07.651483 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
00:00:08.167334 run pcap-recorder_any(filters='host 10.42.42.6')(pid=15990): Launched
00:00:08.272923 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
00:00:08.531916 run patchelf(pid=15991): Launched
00:00:08.706321 run osmo-stp_10.42.42.5(pid=15988): ERR: Terminated: ERROR {rc=1} [trial-2396↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=15988)]
00:00:08.835912 run osmo-stp_10.42.42.5(pid=15988): stdout:
| (launched: 2022-12-14_00:00:07.397229)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:08.963223 run osmo-stp_10.42.42.5(pid=15988): stderr:
| �[0;m�[38;5;43m20221214000007570 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214000007570 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214000007570 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214000007571 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214000007571 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214000007571 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214000007571 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:09.086121 run osmo-stp_10.42.42.5(pid=15988): stdout:
| (launched: 2022-12-14_00:00:07.397229)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:09.208096 run osmo-stp_10.42.42.5(pid=15988): stderr:
| �[0;m�[38;5;43m20221214000007570 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214000007570 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214000007570 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214000007571 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214000007571 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214000007571 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214000007571 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:09.350295 run patchelf(pid=15991): Terminating (SIGINT)
00:00:09.517774 run patchelf(pid=15991): Terminated: ok {rc=0}
00:00:09.548340 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=15988): Process ended prematurely: osmo-stp_10.42.42.5(pid=15988) [trial-2396↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=15988)] [trial-2396↪gprs:trx-lms↪iperf3.py:8]
00:00:09.557401 tst iperf3.py:8: Test FAILED (8.0 sec)
00:00:09.616887 run osmo-hlr_10.42.42.2(pid=15985): ERR: Terminated: ERROR {rc=237} [trial-2396↪gprs:trx-lms↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=15985)]
00:00:09.665438 run osmo-hlr_10.42.42.2(pid=15985): stdout:
| (launched: 2022-12-14_00:00:06.444010)
00:00:09.714814 run osmo-hlr_10.42.42.2(pid=15985): stderr:
| �[0;m�[1;31m20221214000006643 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214000006643 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214000006643 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214000006643 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214000006643 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214000006643 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214000006655 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2396/run.2022-12-13_23-11-51/gprs:trx-lms/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214000006664 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214000006664 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214000006664 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
00:00:09.757669 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15971): Terminating (SIGTERM)
00:00:09.797319 run iperf3-srv_10.42.42.10(pid=15973): Terminating (SIGTERM)
00:00:09.837389 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15984): Terminating (SIGTERM)
00:00:09.877010 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15987): Terminating (SIGTERM)
00:00:09.916433 run pcap-recorder_any(filters='host 10.42.42.6')(pid=15990): Terminating (SIGTERM)
00:00:09.925673 --- ParallelTerminationStrategy: PID 15971 died...
00:00:09.978633 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15971): Terminated: ok {rc=0}
00:00:09.988272 --- ParallelTerminationStrategy: PID 15973 died...
00:00:10.040831 run iperf3-srv_10.42.42.10(pid=15973): Terminated {rc=256}
00:00:10.050284 --- ParallelTerminationStrategy: PID 15984 died...
00:00:10.102964 run pcap-recorder_any(filters='host 10.42.42.2')(pid=15984): Terminated: ok {rc=0}
00:00:10.112537 --- ParallelTerminationStrategy: PID 15987 died...
00:00:10.165200 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=15987): Terminated: ok {rc=0}
00:00:10.174804 --- ParallelTerminationStrategy: PID 15990 died...
00:00:10.227376 run pcap-recorder_any(filters='host 10.42.42.6')(pid=15990): Terminated: ok {rc=0}
00:00:10.310172 bus /gobi_6: Setting Powered False
258gprs:trx-lms11.0osmo-stp_10.42.42.5(pid=16008): Process ended prematurely: osmo-stp_10.42.42.5(pid=16008) [trial-2396↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16008)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=16008): Process ended prematurely: osmo-stp_10.42.42.5(pid=16008) [trial-2396↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16008)]
----------------------------------------------
trial-2396 gprs:trx-lms iperf3m4.py
----------------------------------------------
00:00:11.424099 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:11.435625 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
00:00:11.590220 tst iperf3m4.py: using LAC 37139
00:00:11.718482 tst iperf3m4.py: using RAC 164
00:00:11.845290 tst iperf3m4.py: using CellId 37139
00:00:11.970800 tst iperf3m4.py: using BVCI 37140
00:00:11.994942 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:12.006012 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:12.016786 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:12.027587 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:12.038677 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:12.049886 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:12.061168 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:12.072171 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:12.083512 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:12.203353 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:12.324862 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:12.444719 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:12.720788 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
00:00:12.844142 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:13.150283 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15993): Launched
00:00:13.404599 run iperf3-srv_10.42.42.10(pid=15994): Launched
00:00:13.585174 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
00:00:13.708651 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:14.016761 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15996): Launched
00:00:14.271173 run iperf3-srv_10.42.42.10(pid=15997): Launched
00:00:14.452334 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
00:00:14.576021 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:14.882602 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15998): Launched
00:00:15.137723 run iperf3-srv_10.42.42.10(pid=15999): Launched
00:00:15.318004 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
00:00:15.441204 run iperf3-srv_10.42.42.10: Starting iperf3-srv
00:00:15.748077 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=16000): Launched
00:00:16.049278 run iperf3-srv_10.42.42.10(pid=16001): Launched
00:00:16.234667 tst iperf3m4.py:8: start network...
00:00:16.361125 run osmo-hlr_10.42.42.2: Starting osmo-hlr
00:00:16.890404 run create_hlr_db(pid=16002): Launched
00:00:17.064368 bus /gobi_6: Setting Powered False
00:00:18.084201 run create_hlr_db(pid=16002): Terminated: ok {rc=0}
00:00:18.506177 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16004): Launched
00:00:18.825175 run osmo-hlr_10.42.42.2(pid=16005): Launched
00:00:18.943871 run osmo-stp_10.42.42.5: Starting osmo-stp
00:00:19.495955 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16007): Launched
00:00:19.814894 run osmo-stp_10.42.42.5(pid=16008): Launched
00:00:19.930649 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
00:00:20.483319 run pcap-recorder_any(filters='host 10.42.42.6')(pid=16010): Launched
00:00:20.596709 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
00:00:20.874446 run patchelf(pid=16011): Launched
00:00:21.060665 run osmo-stp_10.42.42.5(pid=16008): ERR: Terminated: ERROR {rc=1} [trial-2396↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16008)]
00:00:21.196703 run osmo-stp_10.42.42.5(pid=16008): stdout:
| (launched: 2022-12-14_00:00:19.657599)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:21.332883 run osmo-stp_10.42.42.5(pid=16008): stderr:
| �[0;m�[38;5;43m20221214000019848 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214000019848 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214000019848 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214000019849 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214000019851 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214000019851 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214000019851 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:21.464612 run osmo-stp_10.42.42.5(pid=16008): stdout:
| (launched: 2022-12-14_00:00:19.657599)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:21.595106 run osmo-stp_10.42.42.5(pid=16008): stderr:
| �[0;m�[38;5;43m20221214000019848 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214000019848 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214000019848 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214000019849 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214000019851 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214000019851 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214000019851 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:21.747590 run patchelf(pid=16011): Terminating (SIGINT)
00:00:21.926152 run patchelf(pid=16011): Terminated: ok {rc=0}
00:00:21.958928 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=16008): Process ended prematurely: osmo-stp_10.42.42.5(pid=16008) [trial-2396↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16008)] [trial-2396↪gprs:trx-lms↪iperf3m4.py:8]
00:00:21.968821 tst iperf3m4.py:8: Test FAILED (10.6 sec)
00:00:22.032479 run osmo-hlr_10.42.42.2(pid=16005): ERR: Terminated: ERROR {rc=237} [trial-2396↪gprs:trx-lms↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=16005)]
00:00:22.084130 run osmo-hlr_10.42.42.2(pid=16005): stdout:
| (launched: 2022-12-14_00:00:18.668081)
00:00:22.136190 run osmo-hlr_10.42.42.2(pid=16005): stderr:
| �[0;m�[1;31m20221214000018866 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214000018866 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214000018866 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214000018866 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214000018866 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214000018866 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214000018880 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2396/run.2022-12-13_23-11-51/gprs:trx-lms/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214000018889 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214000018890 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214000018890 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
00:00:22.181060 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15993): Terminating (SIGTERM)
00:00:22.223169 run iperf3-srv_10.42.42.10(pid=15994): Terminating (SIGTERM)
00:00:22.265151 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15996): Terminating (SIGTERM)
00:00:22.306853 run iperf3-srv_10.42.42.10(pid=15997): Terminating (SIGTERM)
00:00:22.348686 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15998): Terminating (SIGTERM)
00:00:22.390498 run iperf3-srv_10.42.42.10(pid=15999): Terminating (SIGTERM)
00:00:22.432447 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=16000): Terminating (SIGTERM)
00:00:22.474211 run iperf3-srv_10.42.42.10(pid=16001): Terminating (SIGTERM)
00:00:22.516065 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16004): Terminating (SIGTERM)
00:00:22.557950 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16007): Terminating (SIGTERM)
00:00:22.599855 run pcap-recorder_any(filters='host 10.42.42.6')(pid=16010): Terminating (SIGTERM)
00:00:22.609517 --- ParallelTerminationStrategy: PID 15993 died...
00:00:22.665874 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15993): Terminated: ok {rc=0}
00:00:22.676051 --- ParallelTerminationStrategy: PID 15994 died...
00:00:22.732359 run iperf3-srv_10.42.42.10(pid=15994): Terminated {rc=256}
00:00:22.742360 --- ParallelTerminationStrategy: PID 15996 died...
00:00:22.798551 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15996): Terminated: ok {rc=0}
00:00:22.808542 --- ParallelTerminationStrategy: PID 15997 died...
00:00:22.864502 run iperf3-srv_10.42.42.10(pid=15997): Terminated {rc=256}
00:00:22.874496 --- ParallelTerminationStrategy: PID 15998 died...
00:00:22.930410 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=15998): Terminated: ok {rc=0}
00:00:22.940371 --- ParallelTerminationStrategy: PID 15999 died...
00:00:22.996131 run iperf3-srv_10.42.42.10(pid=15999): Terminated {rc=256}
00:00:23.006078 --- ParallelTerminationStrategy: PID 16000 died...
00:00:23.062008 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=16000): Terminated: ok {rc=0}
00:00:23.071962 --- ParallelTerminationStrategy: PID 16001 died...
00:00:23.127939 run iperf3-srv_10.42.42.10(pid=16001): Terminated {rc=256}
00:00:23.138044 --- ParallelTerminationStrategy: PID 16004 died...
00:00:23.194159 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16004): Terminated: ok {rc=0}
00:00:23.204114 --- ParallelTerminationStrategy: PID 16007 died...
00:00:23.260271 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16007): Terminated: ok {rc=0}
00:00:23.270366 --- ParallelTerminationStrategy: PID 16010 died...
00:00:23.326583 run pcap-recorder_any(filters='host 10.42.42.6')(pid=16010): Terminated: ok {rc=0}
00:00:23.413751 bus /gobi_4: Setting Powered False
00:00:24.521039 bus /gobi_4: Setting Powered False
00:00:25.629312 bus /gobi_1: Setting Powered False
00:00:26.735296 bus /gobi_1: Setting Powered False
00:00:27.843229 bus /gobi_0: Setting Powered False
00:00:28.948330 bus /gobi_0: Setting Powered False
00:00:30.059774 bus /gobi_6: Setting Powered False
70gprs:trx-lms7.0osmo-stp_10.42.42.5(pid=16018): Process ended prematurely: osmo-stp_10.42.42.5(pid=16018) [trial-2396↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16018)]2327ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=16018): Process ended prematurely: osmo-stp_10.42.42.5(pid=16018) [trial-2396↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16018)]
----------------------------------------------
trial-2396 gprs:trx-lms ping.py
----------------------------------------------
00:00:31.125337 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:31.137176 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
00:00:31.304808 tst ping.py: using LAC 37140
00:00:31.431409 tst ping.py: using RAC 165
00:00:31.583613 tst ping.py: using CellId 37140
00:00:31.715167 tst ping.py: using BVCI 37141
00:00:31.739769 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:31.751132 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:31.762505 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:31.774057 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:31.787243 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:31.800739 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:31.813959 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:31.826767 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:32.121401 tst ping.py:19: start network...
00:00:32.255166 run osmo-hlr_10.42.42.2: Starting osmo-hlr
00:00:32.744559 run create_hlr_db(pid=16012): Launched
00:00:32.926203 bus /gobi_6: Setting Powered False
00:00:33.948319 run create_hlr_db(pid=16012): Terminated: ok {rc=0}
00:00:34.347142 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16014): Launched
00:00:34.657464 run osmo-hlr_10.42.42.2(pid=16015): Launched
00:00:34.771949 run osmo-stp_10.42.42.5: Starting osmo-stp
00:00:35.316753 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16017): Launched
00:00:35.628646 run osmo-stp_10.42.42.5(pid=16018): Launched
00:00:35.742865 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
00:00:36.285958 run pcap-recorder_any(filters='host 10.42.42.6')(pid=16020): Launched
00:00:36.397661 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
00:00:36.670211 run patchelf(pid=16021): Launched
00:00:36.853826 run osmo-stp_10.42.42.5(pid=16018): ERR: Terminated: ERROR {rc=1} [trial-2396↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16018)]
00:00:36.987935 run osmo-stp_10.42.42.5(pid=16018): stdout:
| (launched: 2022-12-14_00:00:35.475141)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:37.122579 run osmo-stp_10.42.42.5(pid=16018): stderr:
| �[0;m�[38;5;43m20221214000035667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214000035667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214000035667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214000035667 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214000035668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214000035668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214000035668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:37.252176 run osmo-stp_10.42.42.5(pid=16018): stdout:
| (launched: 2022-12-14_00:00:35.475141)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:37.380926 run osmo-stp_10.42.42.5(pid=16018): stderr:
| �[0;m�[38;5;43m20221214000035667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214000035667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214000035667 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214000035667 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214000035668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214000035668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214000035668 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:37.530817 run patchelf(pid=16021): Terminating (SIGINT)
00:00:37.706668 run patchelf(pid=16021): Terminated: ok {rc=0}
00:00:37.738807 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=16018): Process ended prematurely: osmo-stp_10.42.42.5(pid=16018) [trial-2396↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16018)] [trial-2396↪gprs:trx-lms↪ping.py:22]
00:00:37.747992 tst ping.py:22: Test FAILED (6.6 sec)
00:00:37.811783 run osmo-hlr_10.42.42.2(pid=16015): ERR: Terminated: ERROR {rc=237} [trial-2396↪gprs:trx-lms↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=16015)]
00:00:37.863565 run osmo-hlr_10.42.42.2(pid=16015): stdout:
| (launched: 2022-12-14_00:00:34.505130)
00:00:37.916059 run osmo-hlr_10.42.42.2(pid=16015): stderr:
| �[0;m�[1;31m20221214000034703 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214000034703 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214000034703 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214000034703 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214000034703 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214000034703 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214000034716 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2396/run.2022-12-13_23-11-51/gprs:trx-lms/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214000034724 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214000034724 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214000034725 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
00:00:37.960657 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16014): Terminating (SIGTERM)
00:00:38.002878 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16017): Terminating (SIGTERM)
00:00:38.045223 run pcap-recorder_any(filters='host 10.42.42.6')(pid=16020): Terminating (SIGTERM)
00:00:38.055014 --- ParallelTerminationStrategy: PID 16014 died...
00:00:38.112031 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16014): Terminated: ok {rc=0}
00:00:38.122290 --- ParallelTerminationStrategy: PID 16017 died...
00:00:38.179183 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16017): Terminated: ok {rc=0}
00:00:38.189458 --- ParallelTerminationStrategy: PID 16020 died...
00:00:38.246282 run pcap-recorder_any(filters='host 10.42.42.6')(pid=16020): Terminated: ok {rc=0}
00:00:38.333508 bus /gobi_6: Setting Powered False
77gprs:trx-lms7.0osmo-stp_10.42.42.5(pid=16028): Process ended prematurely: osmo-stp_10.42.42.5(pid=16028) [trial-2396↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16028)]2320ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=16028): Process ended prematurely: osmo-stp_10.42.42.5(pid=16028) [trial-2396↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16028)]
----------------------------------------------
trial-2396 gprs:trx-lms ping_idle_ping.py
----------------------------------------------
00:00:39.387236 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:39.398880 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
00:00:39.535868 tst ping_idle_ping.py: using LAC 37141
00:00:39.651002 tst ping_idle_ping.py: using RAC 166
00:00:39.766643 tst ping_idle_ping.py: using CellId 37141
00:00:39.881683 tst ping_idle_ping.py: using BVCI 37142
00:00:39.903981 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:39.914542 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:39.925191 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:39.935432 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:39.946765 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:39.958435 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:39.969692 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
00:00:39.980651 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
00:00:40.254474 tst ping_idle_ping.py:24: start network...
00:00:40.376438 run osmo-hlr_10.42.42.2: Starting osmo-hlr
00:00:40.846505 run create_hlr_db(pid=16022): Launched
00:00:41.015565 bus /gobi_6: Setting Powered False
00:00:42.038849 run create_hlr_db(pid=16022): Terminated: ok {rc=0}
00:00:42.454684 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16024): Launched
00:00:42.769318 run osmo-hlr_10.42.42.2(pid=16025): Launched
00:00:42.886007 run osmo-stp_10.42.42.5: Starting osmo-stp
00:00:43.427206 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16027): Launched
00:00:43.739677 run osmo-stp_10.42.42.5(pid=16028): Launched
00:00:43.854371 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
00:00:44.399380 run pcap-recorder_any(filters='host 10.42.42.6')(pid=16030): Launched
00:00:44.511385 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
00:00:44.783669 run patchelf(pid=16031): Launched
00:00:44.967085 run osmo-stp_10.42.42.5(pid=16028): ERR: Terminated: ERROR {rc=1} [trial-2396↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16028)]
00:00:45.101314 run osmo-stp_10.42.42.5(pid=16028): stdout:
| (launched: 2022-12-14_00:00:43.585952)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:45.235963 run osmo-stp_10.42.42.5(pid=16028): stderr:
| �[0;m�[38;5;43m20221214000043775 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214000043775 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214000043776 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214000043776 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214000043778 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214000043778 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214000043778 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:45.366033 run osmo-stp_10.42.42.5(pid=16028): stdout:
| (launched: 2022-12-14_00:00:43.585952)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
00:00:45.495042 run osmo-stp_10.42.42.5(pid=16028): stderr:
| �[0;m�[38;5;43m20221214000043775 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214000043775 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214000043776 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214000043776 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214000043778 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214000043778 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214000043778 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
00:00:45.644672 run patchelf(pid=16031): Terminating (SIGINT)
00:00:45.820999 run patchelf(pid=16031): Terminated: ok {rc=0}
00:00:45.854863 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=16028): Process ended prematurely: osmo-stp_10.42.42.5(pid=16028) [trial-2396↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=16028)] [trial-2396↪gprs:trx-lms↪ping_idle_ping.py:27]
00:00:45.865186 tst ping_idle_ping.py:27: Test FAILED (6.5 sec)
00:00:45.944414 run osmo-hlr_10.42.42.2(pid=16025): ERR: Terminated: ERROR {rc=237} [trial-2396↪gprs:trx-lms↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=16025)]
00:00:45.998041 run osmo-hlr_10.42.42.2(pid=16025): stdout:
| (launched: 2022-12-14_00:00:42.614340)
00:00:46.052462 run osmo-hlr_10.42.42.2(pid=16025): stderr:
| �[0;m�[1;31m20221214000042810 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214000042810 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214000042810 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214000042810 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214000042810 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214000042810 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214000042822 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2396/run.2022-12-13_23-11-51/gprs:trx-lms/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214000042833 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214000042833 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214000042833 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
00:00:46.096484 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16024): Terminating (SIGTERM)
00:00:46.140656 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16027): Terminating (SIGTERM)
00:00:46.183191 run pcap-recorder_any(filters='host 10.42.42.6')(pid=16030): Terminating (SIGTERM)
00:00:46.194788 --- ParallelTerminationStrategy: PID 16024 died...
00:00:46.251913 run pcap-recorder_any(filters='host 10.42.42.2')(pid=16024): Terminated: ok {rc=0}
00:00:46.262607 --- ParallelTerminationStrategy: PID 16027 died...
00:00:46.317928 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=16027): Terminated: ok {rc=0}
00:00:46.328048 --- ParallelTerminationStrategy: PID 16030 died...
00:00:46.383991 run pcap-recorder_any(filters='host 10.42.42.6')(pid=16030): Terminated: ok {rc=0}
00:00:46.470353 bus /gobi_6: Setting Powered False
5gprs:trx-lms00