239gprs:oc2g9.0osmo-stp_10.42.42.5(pid=18440): Process ended prematurely: osmo-stp_10.42.42.5(pid=18440) [trial-2381↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18440)]2143cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18440): Process ended prematurely: osmo-stp_10.42.42.5(pid=18440) [trial-2381↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18440)]
----------------------------------------------
trial-2381 gprs:oc2g cs_paging_gprs_active.py
----------------------------------------------
19:22:49.891641 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:22:49.902164 tst gprs:oc2g: Using 1 x bts (candidates: 1)
19:22:50.033479 tst cs_paging_gprs_active.py: using LAC 30960
19:22:50.163434 tst cs_paging_gprs_active.py: using RAC 105
19:22:50.292535 tst cs_paging_gprs_active.py: using CellId 30960
19:22:50.421473 tst cs_paging_gprs_active.py: using BVCI 30961
19:22:50.446694 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:22:50.457764 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:22:50.468556 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:22:50.479720 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:22:50.490730 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:22:50.501776 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:22:50.513321 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:22:50.524424 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:22:50.535652 tst gprs:oc2g: Using 1 x modem (candidates: 4)
19:22:50.652553 tst gprs:oc2g: Using 1 x modem (candidates: 4)
19:22:50.951234 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
19:22:51.089159 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:22:51.431368 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18430): Launched
19:22:51.716032 run iperf3-srv_10.42.42.10(pid=18431): Launched
19:22:51.918791 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
19:22:52.056484 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:22:52.399000 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18432): Launched
19:22:52.683508 run iperf3-srv_10.42.42.10(pid=18433): Launched
19:22:52.885139 tst cs_paging_gprs_active.py:38: start network...
19:22:53.022379 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:22:53.538473 run create_hlr_db(pid=18434): Launched
19:22:53.728924 bus /sierra_2: Setting Powered False
19:22:54.750446 run create_hlr_db(pid=18434): Terminated: ok {rc=0}
19:22:55.189106 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18436): Launched
19:22:55.519574 run osmo-hlr_10.42.42.2(pid=18437): Launched
19:22:55.642035 run osmo-stp_10.42.42.5: Starting osmo-stp
19:22:56.215156 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18439): Launched
19:22:56.544297 run osmo-stp_10.42.42.5(pid=18440): Launched
19:22:56.663480 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:22:57.237902 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18442): Launched
19:22:57.355465 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:22:57.643763 run patchelf(pid=18443): Launched
19:22:57.836905 run osmo-stp_10.42.42.5(pid=18440): ERR: Terminated: ERROR {rc=1} [trial-2381↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18440)]
19:22:57.977876 run osmo-stp_10.42.42.5(pid=18440): stdout:
| (launched: 2022-12-12_19:22:56.382135)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:22:58.119948 run osmo-stp_10.42.42.5(pid=18440): stderr:
| �[0;m�[38;5;43m20221212192256574 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192256574 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192256574 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192256574 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192256575 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192256575 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192256575 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:22:58.255963 run osmo-stp_10.42.42.5(pid=18440): stdout:
| (launched: 2022-12-12_19:22:56.382135)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:22:58.391313 run osmo-stp_10.42.42.5(pid=18440): stderr:
| �[0;m�[38;5;43m20221212192256574 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192256574 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192256574 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192256574 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192256575 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192256575 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192256575 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:22:58.549397 run patchelf(pid=18443): Terminating (SIGINT)
19:22:58.734619 run patchelf(pid=18443): Terminated: ok {rc=0}
19:22:58.768463 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=18440): Process ended prematurely: osmo-stp_10.42.42.5(pid=18440) [trial-2381↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18440)] [trial-2381↪gprs:oc2g↪cs_paging_gprs_active.py:38]
19:22:58.778301 tst cs_paging_gprs_active.py:38: Test FAILED (8.9 sec)
19:22:58.842931 run osmo-hlr_10.42.42.2(pid=18437): ERR: Terminated: ERROR {rc=237} [trial-2381↪gprs:oc2g↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18437)]
19:22:58.895255 run osmo-hlr_10.42.42.2(pid=18437): stdout:
| (launched: 2022-12-12_19:22:55.356499)
19:22:58.948175 run osmo-hlr_10.42.42.2(pid=18437): stderr:
| �[0;m�[1;31m20221212192255554 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212192255554 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212192255554 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212192255554 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212192255554 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212192255554 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212192255567 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2381/run.2022-12-12_19-09-05/gprs:oc2g/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212192255577 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212192255577 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212192255578 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:22:58.992801 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18430): Terminating (SIGTERM)
19:22:59.034997 run iperf3-srv_10.42.42.10(pid=18431): Terminating (SIGTERM)
19:22:59.077887 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18432): Terminating (SIGTERM)
19:22:59.120299 run iperf3-srv_10.42.42.10(pid=18433): Terminating (SIGTERM)
19:22:59.162369 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18436): Terminating (SIGTERM)
19:22:59.204859 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18439): Terminating (SIGTERM)
19:22:59.246939 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18442): Terminating (SIGTERM)
19:22:59.257069 --- ParallelTerminationStrategy: PID 18430 died...
19:22:59.314175 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18430): Terminated: ok {rc=0}
19:22:59.324284 --- ParallelTerminationStrategy: PID 18431 died...
19:22:59.380181 run iperf3-srv_10.42.42.10(pid=18431): Terminated {rc=256}
19:22:59.390510 --- ParallelTerminationStrategy: PID 18432 died...
19:22:59.446825 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18432): Terminated: ok {rc=0}
19:22:59.457135 --- ParallelTerminationStrategy: PID 18433 died...
19:22:59.513758 run iperf3-srv_10.42.42.10(pid=18433): Terminated {rc=256}
19:22:59.524057 --- ParallelTerminationStrategy: PID 18436 died...
19:22:59.580277 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18436): Terminated: ok {rc=0}
19:22:59.590779 --- ParallelTerminationStrategy: PID 18439 died...
19:22:59.646947 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18439): Terminated: ok {rc=0}
19:22:59.657888 --- ParallelTerminationStrategy: PID 18442 died...
19:22:59.714804 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18442): Terminated: ok {rc=0}
19:22:59.802558 bus /gobi_0: Setting Powered False
19:23:00.923038 bus /gobi_0: Setting Powered False
19:23:02.037759 bus /gobi_6: Setting Powered False
239gprs:oc2g8.0osmo-stp_10.42.42.5(pid=18452): Process ended prematurely: osmo-stp_10.42.42.5(pid=18452) [trial-2381↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18452)]2143iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18452): Process ended prematurely: osmo-stp_10.42.42.5(pid=18452) [trial-2381↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18452)]
----------------------------------------------
trial-2381 gprs:oc2g iperf3.py
----------------------------------------------
19:23:03.152808 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:03.164436 tst gprs:oc2g: Using 1 x bts (candidates: 1)
19:23:03.295130 tst iperf3.py: using LAC 30961
19:23:03.423801 tst iperf3.py: using RAC 106
19:23:03.554081 tst iperf3.py: using CellId 30961
19:23:03.684549 tst iperf3.py: using BVCI 30962
19:23:03.709852 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:03.720701 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:03.731890 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:03.743627 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:03.755156 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:03.766801 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:03.778580 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:03.790284 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:03.802086 tst gprs:oc2g: Using 1 x modem (candidates: 4)
19:23:04.090853 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
19:23:04.218840 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:23:04.538928 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18444): Launched
19:23:04.804408 run iperf3-srv_10.42.42.10(pid=18445): Launched
19:23:04.991430 tst iperf3.py:8: start network...
19:23:05.118978 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:23:05.613785 run create_hlr_db(pid=18446): Launched
19:23:05.790652 bus /gobi_6: Setting Powered False
19:23:06.812492 run create_hlr_db(pid=18446): Terminated: ok {rc=0}
19:23:07.222903 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18448): Launched
19:23:07.531714 run osmo-hlr_10.42.42.2(pid=18449): Launched
19:23:07.647865 run osmo-stp_10.42.42.5: Starting osmo-stp
19:23:08.181475 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18451): Launched
19:23:08.488575 run osmo-stp_10.42.42.5(pid=18452): Launched
19:23:08.599534 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:23:09.137649 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18454): Launched
19:23:09.246746 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:23:09.515052 run patchelf(pid=18455): Launched
19:23:09.695350 run osmo-stp_10.42.42.5(pid=18452): ERR: Terminated: ERROR {rc=1} [trial-2381↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18452)]
19:23:09.827140 run osmo-stp_10.42.42.5(pid=18452): stdout:
| (launched: 2022-12-12_19:23:08.337081)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:23:09.959454 run osmo-stp_10.42.42.5(pid=18452): stderr:
| �[0;m�[38;5;43m20221212192308509 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192308509 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192308510 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192308510 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192308510 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192308511 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192308511 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:23:10.086748 run osmo-stp_10.42.42.5(pid=18452): stdout:
| (launched: 2022-12-12_19:23:08.337081)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:23:10.212837 run osmo-stp_10.42.42.5(pid=18452): stderr:
| �[0;m�[38;5;43m20221212192308509 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192308509 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192308510 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192308510 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192308510 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192308511 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192308511 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:23:10.359496 run patchelf(pid=18455): Terminating (SIGINT)
19:23:10.531791 run patchelf(pid=18455): Terminated: ok {rc=0}
19:23:10.563220 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=18452): Process ended prematurely: osmo-stp_10.42.42.5(pid=18452) [trial-2381↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18452)] [trial-2381↪gprs:oc2g↪iperf3.py:8]
19:23:10.573029 tst iperf3.py:8: Test FAILED (7.5 sec)
19:23:10.633546 run osmo-hlr_10.42.42.2(pid=18449): ERR: Terminated: ERROR {rc=237} [trial-2381↪gprs:oc2g↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18449)]
19:23:10.682868 run osmo-hlr_10.42.42.2(pid=18449): stdout:
| (launched: 2022-12-12_19:23:07.379041)
19:23:10.733167 run osmo-hlr_10.42.42.2(pid=18449): stderr:
| �[0;m�[1;31m20221212192307556 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212192307556 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212192307556 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212192307556 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212192307556 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212192307556 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212192307569 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2381/run.2022-12-12_19-09-05/gprs:oc2g/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212192307578 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212192307578 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212192307578 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:23:10.775372 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18444): Terminating (SIGTERM)
19:23:10.815158 run iperf3-srv_10.42.42.10(pid=18445): Terminating (SIGTERM)
19:23:10.855186 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18448): Terminating (SIGTERM)
19:23:10.894885 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18451): Terminating (SIGTERM)
19:23:10.934607 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18454): Terminating (SIGTERM)
19:23:10.943936 --- ParallelTerminationStrategy: PID 18444 died...
19:23:10.997606 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18444): Terminated: ok {rc=0}
19:23:11.007537 --- ParallelTerminationStrategy: PID 18445 died...
19:23:11.060789 run iperf3-srv_10.42.42.10(pid=18445): Terminated {rc=256}
19:23:11.070620 --- ParallelTerminationStrategy: PID 18448 died...
19:23:11.124088 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18448): Terminated: ok {rc=0}
19:23:11.134080 --- ParallelTerminationStrategy: PID 18451 died...
19:23:11.187272 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18451): Terminated: ok {rc=0}
19:23:11.197236 --- ParallelTerminationStrategy: PID 18454 died...
19:23:11.250239 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18454): Terminated: ok {rc=0}
19:23:11.338168 bus /gobi_6: Setting Powered False
243gprs:oc2g11.0osmo-stp_10.42.42.5(pid=18470): Process ended prematurely: osmo-stp_10.42.42.5(pid=18470) [trial-2381↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18470)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18470): Process ended prematurely: osmo-stp_10.42.42.5(pid=18470) [trial-2381↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18470)]
----------------------------------------------
trial-2381 gprs:oc2g iperf3m4.py
----------------------------------------------
19:23:12.450978 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:12.462378 tst gprs:oc2g: Using 1 x bts (candidates: 1)
19:23:12.588136 tst iperf3m4.py: using LAC 30962
19:23:12.717431 tst iperf3m4.py: using RAC 107
19:23:12.847394 tst iperf3m4.py: using CellId 30962
19:23:12.971267 tst iperf3m4.py: using BVCI 30963
19:23:12.994802 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:13.005439 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:13.016094 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:13.027311 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:13.038426 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:13.049431 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:13.060305 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:13.071199 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:13.082315 tst gprs:oc2g: Using 1 x modem (candidates: 4)
19:23:13.198291 tst gprs:oc2g: Using 1 x modem (candidates: 4)
19:23:13.317265 tst gprs:oc2g: Using 1 x modem (candidates: 4)
19:23:13.437430 tst gprs:oc2g: Using 1 x modem (candidates: 4)
19:23:13.718798 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
19:23:13.846275 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:23:14.167798 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18456): Launched
19:23:14.433535 run iperf3-srv_10.42.42.10(pid=18457): Launched
19:23:14.620034 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
19:23:14.747379 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:23:15.096453 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18458): Launched
19:23:15.381416 run iperf3-srv_10.42.42.10(pid=18459): Launched
19:23:15.584409 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
19:23:15.721824 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:23:16.074466 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18460): Launched
19:23:16.361206 run iperf3-srv_10.42.42.10(pid=18461): Launched
19:23:16.552090 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
19:23:16.682597 run iperf3-srv_10.42.42.10: Starting iperf3-srv
19:23:17.008763 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18462): Launched
19:23:17.292240 run iperf3-srv_10.42.42.10(pid=18463): Launched
19:23:17.478140 tst iperf3m4.py:8: start network...
19:23:17.605692 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:23:18.100772 run create_hlr_db(pid=18464): Launched
19:23:18.278789 bus /gobi_6: Setting Powered False
19:23:19.299297 run create_hlr_db(pid=18464): Terminated: ok {rc=0}
19:23:19.738310 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18466): Launched
19:23:20.068680 run osmo-hlr_10.42.42.2(pid=18467): Launched
19:23:20.190273 run osmo-stp_10.42.42.5: Starting osmo-stp
19:23:20.760691 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18469): Launched
19:23:21.090600 run osmo-stp_10.42.42.5(pid=18470): Launched
19:23:21.210062 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:23:21.787215 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18472): Launched
19:23:21.905227 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:23:22.193751 run patchelf(pid=18473): Launched
19:23:22.387186 run osmo-stp_10.42.42.5(pid=18470): ERR: Terminated: ERROR {rc=1} [trial-2381↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18470)]
19:23:22.528741 run osmo-stp_10.42.42.5(pid=18470): stdout:
| (launched: 2022-12-12_19:23:20.927913)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:23:22.670891 run osmo-stp_10.42.42.5(pid=18470): stderr:
| �[0;m�[38;5;43m20221212192321120 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192321120 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192321120 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192321121 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192321121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192321121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192321121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:23:22.807453 run osmo-stp_10.42.42.5(pid=18470): stdout:
| (launched: 2022-12-12_19:23:20.927913)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:23:22.942656 run osmo-stp_10.42.42.5(pid=18470): stderr:
| �[0;m�[38;5;43m20221212192321120 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192321120 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192321120 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192321121 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192321121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192321121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192321121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:23:23.100211 run patchelf(pid=18473): Terminating (SIGINT)
19:23:23.285076 run patchelf(pid=18473): Terminated: ok {rc=0}
19:23:23.319000 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=18470): Process ended prematurely: osmo-stp_10.42.42.5(pid=18470) [trial-2381↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18470)] [trial-2381↪gprs:oc2g↪iperf3m4.py:8]
19:23:23.329140 tst iperf3m4.py:8: Test FAILED (10.9 sec)
19:23:23.394416 run osmo-hlr_10.42.42.2(pid=18467): ERR: Terminated: ERROR {rc=237} [trial-2381↪gprs:oc2g↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18467)]
19:23:23.446944 run osmo-hlr_10.42.42.2(pid=18467): stdout:
| (launched: 2022-12-12_19:23:19.905781)
19:23:23.500244 run osmo-hlr_10.42.42.2(pid=18467): stderr:
| �[0;m�[1;31m20221212192320105 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212192320105 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212192320105 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212192320105 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212192320105 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212192320105 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212192320119 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2381/run.2022-12-12_19-09-05/gprs:oc2g/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212192320127 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212192320127 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212192320127 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:23:23.546027 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18456): Terminating (SIGTERM)
19:23:23.588562 run iperf3-srv_10.42.42.10(pid=18457): Terminating (SIGTERM)
19:23:23.631501 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18458): Terminating (SIGTERM)
19:23:23.674123 run iperf3-srv_10.42.42.10(pid=18459): Terminating (SIGTERM)
19:23:23.716894 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18460): Terminating (SIGTERM)
19:23:23.759544 run iperf3-srv_10.42.42.10(pid=18461): Terminating (SIGTERM)
19:23:23.802310 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18462): Terminating (SIGTERM)
19:23:23.844843 run iperf3-srv_10.42.42.10(pid=18463): Terminating (SIGTERM)
19:23:23.887611 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18466): Terminating (SIGTERM)
19:23:23.930597 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18469): Terminating (SIGTERM)
19:23:23.973063 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18472): Terminating (SIGTERM)
19:23:23.983321 --- ParallelTerminationStrategy: PID 18456 died...
19:23:24.041118 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18456): Terminated: ok {rc=0}
19:23:24.051523 --- ParallelTerminationStrategy: PID 18457 died...
19:23:24.108604 run iperf3-srv_10.42.42.10(pid=18457): Terminated {rc=256}
19:23:24.119231 --- ParallelTerminationStrategy: PID 18458 died...
19:23:24.176788 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18458): Terminated: ok {rc=0}
19:23:24.187238 --- ParallelTerminationStrategy: PID 18459 died...
19:23:24.244706 run iperf3-srv_10.42.42.10(pid=18459): Terminated {rc=256}
19:23:24.255068 --- ParallelTerminationStrategy: PID 18460 died...
19:23:24.312157 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18460): Terminated: ok {rc=0}
19:23:24.322924 --- ParallelTerminationStrategy: PID 18461 died...
19:23:24.380487 run iperf3-srv_10.42.42.10(pid=18461): Terminated {rc=256}
19:23:24.390817 --- ParallelTerminationStrategy: PID 18462 died...
19:23:24.447884 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18462): Terminated: ok {rc=0}
19:23:24.458604 --- ParallelTerminationStrategy: PID 18463 died...
19:23:24.515708 run iperf3-srv_10.42.42.10(pid=18463): Terminated {rc=256}
19:23:24.526842 --- ParallelTerminationStrategy: PID 18466 died...
19:23:24.584124 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18466): Terminated: ok {rc=0}
19:23:24.594834 --- ParallelTerminationStrategy: PID 18469 died...
19:23:24.652225 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18469): Terminated: ok {rc=0}
19:23:24.662691 --- ParallelTerminationStrategy: PID 18472 died...
19:23:24.719790 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18472): Terminated: ok {rc=0}
19:23:24.807849 bus /gobi_4: Setting Powered False
19:23:25.917274 bus /gobi_4: Setting Powered False
19:23:27.030848 bus /gobi_1: Setting Powered False
19:23:28.135513 bus /gobi_1: Setting Powered False
19:23:29.249206 bus /gobi_0: Setting Powered False
19:23:30.352848 bus /gobi_0: Setting Powered False
19:23:31.479971 bus /gobi_6: Setting Powered False
243gprs:oc2g7.0osmo-stp_10.42.42.5(pid=18480): Process ended prematurely: osmo-stp_10.42.42.5(pid=18480) [trial-2381↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18480)]2139ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18480): Process ended prematurely: osmo-stp_10.42.42.5(pid=18480) [trial-2381↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18480)]
----------------------------------------------
trial-2381 gprs:oc2g ping.py
----------------------------------------------
19:23:32.552723 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:32.564668 tst gprs:oc2g: Using 1 x bts (candidates: 1)
19:23:32.697441 tst ping.py: using LAC 30963
19:23:32.825467 tst ping.py: using RAC 108
19:23:32.952109 tst ping.py: using CellId 30963
19:23:33.080916 tst ping.py: using BVCI 30964
19:23:33.106028 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:33.117205 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:33.128329 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:33.139844 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:33.151158 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:33.162636 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:33.174162 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:33.185483 tst gprs:oc2g: Using 1 x modem (candidates: 4)
19:23:33.464172 tst ping.py:19: start network...
19:23:33.590802 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:23:34.074430 run create_hlr_db(pid=18474): Launched
19:23:34.249368 bus /gobi_6: Setting Powered False
19:23:35.270287 run create_hlr_db(pid=18474): Terminated: ok {rc=0}
19:23:35.702979 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18476): Launched
19:23:36.027677 run osmo-hlr_10.42.42.2(pid=18477): Launched
19:23:36.149504 run osmo-stp_10.42.42.5: Starting osmo-stp
19:23:36.718095 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18479): Launched
19:23:37.043361 run osmo-stp_10.42.42.5(pid=18480): Launched
19:23:37.161579 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:23:37.731597 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18482): Launched
19:23:37.847848 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:23:38.130833 run patchelf(pid=18483): Launched
19:23:38.321085 run osmo-stp_10.42.42.5(pid=18480): ERR: Terminated: ERROR {rc=1} [trial-2381↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18480)]
19:23:38.460561 run osmo-stp_10.42.42.5(pid=18480): stdout:
| (launched: 2022-12-12_19:23:36.883141)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:23:38.600086 run osmo-stp_10.42.42.5(pid=18480): stderr:
| �[0;m�[38;5;43m20221212192337074 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192337074 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192337074 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192337075 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192337075 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192337075 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192337075 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:23:38.734668 run osmo-stp_10.42.42.5(pid=18480): stdout:
| (launched: 2022-12-12_19:23:36.883141)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:23:38.867952 run osmo-stp_10.42.42.5(pid=18480): stderr:
| �[0;m�[38;5;43m20221212192337074 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192337074 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192337074 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192337075 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192337075 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192337075 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192337075 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:23:39.022821 run patchelf(pid=18483): Terminating (SIGINT)
19:23:39.203955 run patchelf(pid=18483): Terminated: ok {rc=0}
19:23:39.236743 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=18480): Process ended prematurely: osmo-stp_10.42.42.5(pid=18480) [trial-2381↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18480)] [trial-2381↪gprs:oc2g↪ping.py:22]
19:23:39.246134 tst ping.py:22: Test FAILED (6.7 sec)
19:23:39.311454 run osmo-hlr_10.42.42.2(pid=18477): ERR: Terminated: ERROR {rc=237} [trial-2381↪gprs:oc2g↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18477)]
19:23:39.364197 run osmo-hlr_10.42.42.2(pid=18477): stdout:
| (launched: 2022-12-12_19:23:35.868139)
19:23:39.417436 run osmo-hlr_10.42.42.2(pid=18477): stderr:
| �[0;m�[1;31m20221212192336067 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212192336067 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212192336067 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212192336067 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212192336067 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212192336067 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212192336081 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2381/run.2022-12-12_19-09-05/gprs:oc2g/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212192336090 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212192336091 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212192336091 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:23:39.463036 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18476): Terminating (SIGTERM)
19:23:39.506234 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18479): Terminating (SIGTERM)
19:23:39.549754 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18482): Terminating (SIGTERM)
19:23:39.559731 --- ParallelTerminationStrategy: PID 18476 died...
19:23:39.617890 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18476): Terminated: ok {rc=0}
19:23:39.628359 --- ParallelTerminationStrategy: PID 18479 died...
19:23:39.686161 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18479): Terminated: ok {rc=0}
19:23:39.696732 --- ParallelTerminationStrategy: PID 18482 died...
19:23:39.753808 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18482): Terminated: ok {rc=0}
19:23:39.843248 bus /gobi_6: Setting Powered False
243gprs:oc2g7.0osmo-stp_10.42.42.5(pid=18490): Process ended prematurely: osmo-stp_10.42.42.5(pid=18490) [trial-2381↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18490)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18490): Process ended prematurely: osmo-stp_10.42.42.5(pid=18490) [trial-2381↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18490)]
----------------------------------------------
trial-2381 gprs:oc2g ping_idle_ping.py
----------------------------------------------
19:23:40.895426 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:40.906905 tst gprs:oc2g: Using 1 x bts (candidates: 1)
19:23:41.042740 tst ping_idle_ping.py: using LAC 30964
19:23:41.169693 tst ping_idle_ping.py: using RAC 109
19:23:41.292888 tst ping_idle_ping.py: using CellId 30964
19:23:41.416052 tst ping_idle_ping.py: using BVCI 30965
19:23:41.440037 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:41.450774 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:41.461447 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:41.472064 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:41.482471 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:41.492869 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:41.503505 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
19:23:41.514078 tst gprs:oc2g: Using 1 x modem (candidates: 4)
19:23:41.808558 tst ping_idle_ping.py:24: start network...
19:23:41.944423 run osmo-hlr_10.42.42.2: Starting osmo-hlr
19:23:42.458847 run create_hlr_db(pid=18484): Launched
19:23:42.648403 bus /gobi_6: Setting Powered False
19:23:43.668889 run create_hlr_db(pid=18484): Terminated: ok {rc=0}
19:23:44.101517 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18486): Launched
19:23:44.427762 run osmo-hlr_10.42.42.2(pid=18487): Launched
19:23:44.547835 run osmo-stp_10.42.42.5: Starting osmo-stp
19:23:45.113525 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18489): Launched
19:23:45.438725 run osmo-stp_10.42.42.5(pid=18490): Launched
19:23:45.556713 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
19:23:46.181547 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18492): Launched
19:23:46.340215 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
19:23:46.682474 run patchelf(pid=18521): Launched
19:23:46.882760 run osmo-stp_10.42.42.5(pid=18490): ERR: Terminated: ERROR {rc=1} [trial-2381↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18490)]
19:23:47.017333 run osmo-stp_10.42.42.5(pid=18490): stdout:
| (launched: 2022-12-12_19:23:45.278660)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:23:47.174310 run osmo-stp_10.42.42.5(pid=18490): stderr:
| �[0;m�[38;5;43m20221212192345471 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192345471 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192345472 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192345472 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192345473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192345473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192345473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:23:47.315185 run osmo-stp_10.42.42.5(pid=18490): stdout:
| (launched: 2022-12-12_19:23:45.278660)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
19:23:47.444897 run osmo-stp_10.42.42.5(pid=18490): stderr:
| �[0;m�[38;5;43m20221212192345471 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212192345471 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212192345472 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212192345472 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212192345473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212192345473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212192345473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
19:23:47.599834 run patchelf(pid=18521): Terminating (SIGINT)
19:23:47.797873 run patchelf(pid=18521): Terminated: ok {rc=0}
19:23:47.836266 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=18490): Process ended prematurely: osmo-stp_10.42.42.5(pid=18490) [trial-2381↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18490)] [trial-2381↪gprs:oc2g↪ping_idle_ping.py:27]
19:23:47.847346 tst ping_idle_ping.py:27: Test FAILED (6.9 sec)
19:23:47.925252 run osmo-hlr_10.42.42.2(pid=18487): ERR: Terminated: ERROR {rc=237} [trial-2381↪gprs:oc2g↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18487)]
19:23:47.979599 run osmo-hlr_10.42.42.2(pid=18487): stdout:
| (launched: 2022-12-12_19:23:44.266694)
19:23:48.034458 run osmo-hlr_10.42.42.2(pid=18487): stderr:
| �[0;m�[1;31m20221212192344466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212192344466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212192344466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212192344466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212192344466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212192344466 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212192344479 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2381/run.2022-12-12_19-09-05/gprs:oc2g/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212192344487 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212192344487 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212192344487 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
19:23:48.086490 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18486): Terminating (SIGTERM)
19:23:48.130462 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18489): Terminating (SIGTERM)
19:23:48.181387 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18492): Terminating (SIGTERM)
19:23:48.194337 --- ParallelTerminationStrategy: PID 18486 died...
19:23:48.255891 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18486): Terminated: ok {rc=0}
19:23:48.266869 --- ParallelTerminationStrategy: PID 18489 died...
19:23:48.331126 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18489): Terminated: ok {rc=0}
19:23:48.342004 --- ParallelTerminationStrategy: PID 18492 died...
19:23:48.413556 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18492): Terminated: ok {rc=0}
19:23:48.516273 bus /gobi_6: Setting Powered False
5gprs:oc2g00