232gprs:oc2g9.0osmo-stp_10.42.42.5(pid=2255): Process ended prematurely: osmo-stp_10.42.42.5(pid=2255) [trial-2374↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2255)]2143cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2255): Process ended prematurely: osmo-stp_10.42.42.5(pid=2255) [trial-2374↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2255)]
----------------------------------------------
trial-2374 gprs:oc2g cs_paging_gprs_active.py
----------------------------------------------
11:28:59.728348 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:28:59.739291 tst gprs:oc2g: Using 1 x bts (candidates: 1)
11:28:59.867960 tst cs_paging_gprs_active.py: using LAC 28080
11:28:59.995250 tst cs_paging_gprs_active.py: using RAC 30
11:29:00.122462 tst cs_paging_gprs_active.py: using CellId 28080
11:29:00.250095 tst cs_paging_gprs_active.py: using BVCI 28081
11:29:00.274847 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:00.285609 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:00.296262 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:00.307488 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:00.318564 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:00.329717 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:00.341109 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:00.352291 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:00.364077 tst gprs:oc2g: Using 1 x modem (candidates: 4)
11:29:00.482073 tst gprs:oc2g: Using 1 x modem (candidates: 4)
11:29:00.764683 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
11:29:00.901293 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:29:01.235877 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2245): Launched
11:29:01.522465 run iperf3-srv_10.42.42.10(pid=2246): Launched
11:29:01.715194 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
11:29:01.843704 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:29:02.196041 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2247): Launched
11:29:02.476684 run iperf3-srv_10.42.42.10(pid=2248): Launched
11:29:02.676257 tst cs_paging_gprs_active.py:38: start network...
11:29:02.811839 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:29:03.331443 run create_hlr_db(pid=2249): Launched
11:29:03.519494 bus /sierra_2: Setting Powered False
11:29:04.539988 run create_hlr_db(pid=2249): Terminated: ok {rc=0}
11:29:04.946250 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2251): Launched
11:29:05.249901 run osmo-hlr_10.42.42.2(pid=2252): Launched
11:29:05.366965 run osmo-stp_10.42.42.5: Starting osmo-stp
11:29:05.900484 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2254): Launched
11:29:06.203625 run osmo-stp_10.42.42.5(pid=2255): Launched
11:29:06.314793 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:29:06.843285 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2257): Launched
11:29:06.951687 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:29:07.215742 run patchelf(pid=2258): Launched
11:29:07.393113 run osmo-stp_10.42.42.5(pid=2255): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2255)]
11:29:07.522915 run osmo-stp_10.42.42.5(pid=2255): stdout:
| (launched: 2022-12-10_11:29:06.054080)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:07.652564 run osmo-stp_10.42.42.5(pid=2255): stderr:
| �[0;m�[38;5;43m20221210112906225 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112906226 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112906226 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112906226 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112906227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112906227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112906227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:07.777689 run osmo-stp_10.42.42.5(pid=2255): stdout:
| (launched: 2022-12-10_11:29:06.054080)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:07.901517 run osmo-stp_10.42.42.5(pid=2255): stderr:
| �[0;m�[38;5;43m20221210112906225 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112906226 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112906226 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112906226 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112906227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112906227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112906227 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:08.046612 run patchelf(pid=2258): Terminating (SIGINT)
11:29:08.216548 run patchelf(pid=2258): Terminated: ok {rc=0}
11:29:08.247684 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=2255): Process ended prematurely: osmo-stp_10.42.42.5(pid=2255) [trial-2374↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2255)] [trial-2374↪gprs:oc2g↪cs_paging_gprs_active.py:38]
11:29:08.256751 tst cs_paging_gprs_active.py:38: Test FAILED (8.6 sec)
11:29:08.316915 run osmo-hlr_10.42.42.2(pid=2252): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:oc2g↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2252)]
11:29:08.366314 run osmo-hlr_10.42.42.2(pid=2252): stdout:
| (launched: 2022-12-10_11:29:05.099959)
11:29:08.416154 run osmo-hlr_10.42.42.2(pid=2252): stderr:
| �[0;m�[1;31m20221210112905275 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210112905275 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210112905275 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210112905275 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210112905275 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210112905275 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210112905288 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:oc2g/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210112905298 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210112905298 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210112905298 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:29:08.458201 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2245): Terminating (SIGTERM)
11:29:08.498130 run iperf3-srv_10.42.42.10(pid=2246): Terminating (SIGTERM)
11:29:08.538201 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2247): Terminating (SIGTERM)
11:29:08.577946 run iperf3-srv_10.42.42.10(pid=2248): Terminating (SIGTERM)
11:29:08.617496 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2251): Terminating (SIGTERM)
11:29:08.657057 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2254): Terminating (SIGTERM)
11:29:08.697488 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2257): Terminating (SIGTERM)
11:29:08.706936 --- ParallelTerminationStrategy: PID 2245 died...
11:29:08.760371 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2245): Terminated: ok {rc=0}
11:29:08.770326 --- ParallelTerminationStrategy: PID 2246 died...
11:29:08.822883 run iperf3-srv_10.42.42.10(pid=2246): Terminated {rc=256}
11:29:08.832608 --- ParallelTerminationStrategy: PID 2247 died...
11:29:08.886364 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2247): Terminated: ok {rc=0}
11:29:08.896076 --- ParallelTerminationStrategy: PID 2248 died...
11:29:08.948935 run iperf3-srv_10.42.42.10(pid=2248): Terminated {rc=256}
11:29:08.958890 --- ParallelTerminationStrategy: PID 2251 died...
11:29:09.011291 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2251): Terminated: ok {rc=0}
11:29:09.021106 --- ParallelTerminationStrategy: PID 2254 died...
11:29:09.074268 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2254): Terminated: ok {rc=0}
11:29:09.083814 --- ParallelTerminationStrategy: PID 2257 died...
11:29:09.136667 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2257): Terminated: ok {rc=0}
11:29:09.221111 bus /gobi_0: Setting Powered False
11:29:10.327704 bus /gobi_0: Setting Powered False
11:29:11.433863 bus /gobi_6: Setting Powered False
232gprs:oc2g8.0osmo-stp_10.42.42.5(pid=2267): Process ended prematurely: osmo-stp_10.42.42.5(pid=2267) [trial-2374↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2267)]2143iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2267): Process ended prematurely: osmo-stp_10.42.42.5(pid=2267) [trial-2374↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2267)]
----------------------------------------------
trial-2374 gprs:oc2g iperf3.py
----------------------------------------------
11:29:12.548477 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:12.560008 tst gprs:oc2g: Using 1 x bts (candidates: 1)
11:29:12.693539 tst iperf3.py: using LAC 28081
11:29:12.818686 tst iperf3.py: using RAC 31
11:29:12.945301 tst iperf3.py: using CellId 28081
11:29:13.065896 tst iperf3.py: using BVCI 28082
11:29:13.088856 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:13.099363 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:13.109645 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:13.120123 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:13.130615 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:13.140939 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:13.151842 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:13.162433 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:13.174017 tst gprs:oc2g: Using 1 x modem (candidates: 4)
11:29:13.454136 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
11:29:13.580467 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:29:13.893599 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2259): Launched
11:29:14.153409 run iperf3-srv_10.42.42.10(pid=2260): Launched
11:29:14.337515 tst iperf3.py:8: start network...
11:29:14.463024 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:29:14.937984 run create_hlr_db(pid=2261): Launched
11:29:15.113304 bus /gobi_6: Setting Powered False
11:29:16.151425 run create_hlr_db(pid=2261): Terminated: ok {rc=0}
11:29:16.609283 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2263): Launched
11:29:17.048665 run osmo-hlr_10.42.42.2(pid=2264): Launched
11:29:17.253152 run osmo-stp_10.42.42.5: Starting osmo-stp
11:29:17.813277 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2266): Launched
11:29:18.138075 run osmo-stp_10.42.42.5(pid=2267): Launched
11:29:18.256168 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:29:18.822151 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2269): Launched
11:29:18.937701 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:29:19.221814 run patchelf(pid=2270): Launched
11:29:19.411138 run osmo-stp_10.42.42.5(pid=2267): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2267)]
11:29:19.549694 run osmo-stp_10.42.42.5(pid=2267): stdout:
| (launched: 2022-12-10_11:29:17.978479)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:19.688792 run osmo-stp_10.42.42.5(pid=2267): stderr:
| �[0;m�[38;5;43m20221210112918172 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112918172 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112918173 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112918173 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112918174 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112918174 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112918174 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:19.823150 run osmo-stp_10.42.42.5(pid=2267): stdout:
| (launched: 2022-12-10_11:29:17.978479)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:19.955886 run osmo-stp_10.42.42.5(pid=2267): stderr:
| �[0;m�[38;5;43m20221210112918172 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112918172 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112918173 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112918173 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112918174 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112918174 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112918174 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:20.112013 run patchelf(pid=2270): Terminating (SIGINT)
11:29:20.294228 run patchelf(pid=2270): Terminated: ok {rc=0}
11:29:20.327601 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=2267): Process ended prematurely: osmo-stp_10.42.42.5(pid=2267) [trial-2374↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2267)] [trial-2374↪gprs:oc2g↪iperf3.py:8]
11:29:20.337436 tst iperf3.py:8: Test FAILED (7.8 sec)
11:29:20.400335 run osmo-hlr_10.42.42.2(pid=2264): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:oc2g↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2264)]
11:29:20.451715 run osmo-hlr_10.42.42.2(pid=2264): stdout:
| (launched: 2022-12-10_11:29:16.777699)
11:29:20.503733 run osmo-hlr_10.42.42.2(pid=2264): stderr:
| �[0;m�[1;31m20221210112917085 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210112917085 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210112917085 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210112917085 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210112917085 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210112917085 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210112917104 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:oc2g/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210112917113 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210112917113 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210112917113 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:29:20.548079 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2259): Terminating (SIGTERM)
11:29:20.589779 run iperf3-srv_10.42.42.10(pid=2260): Terminating (SIGTERM)
11:29:20.631482 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2263): Terminating (SIGTERM)
11:29:20.673314 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2266): Terminating (SIGTERM)
11:29:20.715142 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2269): Terminating (SIGTERM)
11:29:20.724888 --- ParallelTerminationStrategy: PID 2259 died...
11:29:20.781001 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2259): Terminated: ok {rc=0}
11:29:20.791069 --- ParallelTerminationStrategy: PID 2260 died...
11:29:20.846581 run iperf3-srv_10.42.42.10(pid=2260): Terminated {rc=256}
11:29:20.856806 --- ParallelTerminationStrategy: PID 2263 died...
11:29:20.912277 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2263): Terminated: ok {rc=0}
11:29:20.922453 --- ParallelTerminationStrategy: PID 2266 died...
11:29:20.978157 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2266): Terminated: ok {rc=0}
11:29:20.988358 --- ParallelTerminationStrategy: PID 2269 died...
11:29:21.044548 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2269): Terminated: ok {rc=0}
11:29:21.130654 bus /gobi_6: Setting Powered False
236gprs:oc2g11.0osmo-stp_10.42.42.5(pid=2285): Process ended prematurely: osmo-stp_10.42.42.5(pid=2285) [trial-2374↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2285)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2285): Process ended prematurely: osmo-stp_10.42.42.5(pid=2285) [trial-2374↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2285)]
----------------------------------------------
trial-2374 gprs:oc2g iperf3m4.py
----------------------------------------------
11:29:22.246905 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:22.258468 tst gprs:oc2g: Using 1 x bts (candidates: 1)
11:29:22.389651 tst iperf3m4.py: using LAC 28082
11:29:22.518402 tst iperf3m4.py: using RAC 32
11:29:22.645470 tst iperf3m4.py: using CellId 28082
11:29:22.773856 tst iperf3m4.py: using BVCI 28083
11:29:22.798653 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:22.809421 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:22.820030 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:22.830694 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:22.841216 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:22.851767 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:22.862436 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:22.872924 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:22.883729 tst gprs:oc2g: Using 1 x modem (candidates: 4)
11:29:23.000811 tst gprs:oc2g: Using 1 x modem (candidates: 4)
11:29:23.119925 tst gprs:oc2g: Using 1 x modem (candidates: 4)
11:29:23.238312 tst gprs:oc2g: Using 1 x modem (candidates: 4)
11:29:23.515669 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
11:29:23.642122 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:29:23.957427 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2271): Launched
11:29:24.218858 run iperf3-srv_10.42.42.10(pid=2272): Launched
11:29:24.403752 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
11:29:24.529995 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:29:24.843185 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2273): Launched
11:29:25.104288 run iperf3-srv_10.42.42.10(pid=2274): Launched
11:29:25.289234 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
11:29:25.415004 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:29:25.728922 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2275): Launched
11:29:25.988592 run iperf3-srv_10.42.42.10(pid=2276): Launched
11:29:26.172747 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
11:29:26.298295 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:29:26.611671 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2277): Launched
11:29:26.871118 run iperf3-srv_10.42.42.10(pid=2278): Launched
11:29:27.055196 tst iperf3m4.py:8: start network...
11:29:27.181560 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:29:27.656865 run create_hlr_db(pid=2279): Launched
11:29:27.832041 bus /gobi_6: Setting Powered False
11:29:28.853209 run create_hlr_db(pid=2279): Terminated: ok {rc=0}
11:29:29.256493 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2281): Launched
11:29:29.559316 run osmo-hlr_10.42.42.2(pid=2282): Launched
11:29:29.677433 run osmo-stp_10.42.42.5: Starting osmo-stp
11:29:30.207474 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2284): Launched
11:29:30.511318 run osmo-stp_10.42.42.5(pid=2285): Launched
11:29:30.622086 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:29:31.227656 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2287): Launched
11:29:31.379347 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:29:31.699124 run patchelf(pid=2288): Launched
11:29:31.894731 run osmo-stp_10.42.42.5(pid=2285): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2285)]
11:29:32.038848 run osmo-stp_10.42.42.5(pid=2285): stdout:
| (launched: 2022-12-10_11:29:30.361330)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:32.182090 run osmo-stp_10.42.42.5(pid=2285): stderr:
| �[0;m�[38;5;43m20221210112930531 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112930532 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112930532 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112930532 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112930533 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112930533 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112930533 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:32.318597 run osmo-stp_10.42.42.5(pid=2285): stdout:
| (launched: 2022-12-10_11:29:30.361330)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:32.451810 run osmo-stp_10.42.42.5(pid=2285): stderr:
| �[0;m�[38;5;43m20221210112930531 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112930532 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112930532 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112930532 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112930533 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112930533 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112930533 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:32.607700 run patchelf(pid=2288): Terminating (SIGINT)
11:29:32.789719 run patchelf(pid=2288): Terminated: ok {rc=0}
11:29:32.823250 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=2285): Process ended prematurely: osmo-stp_10.42.42.5(pid=2285) [trial-2374↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2285)] [trial-2374↪gprs:oc2g↪iperf3m4.py:8]
11:29:32.833324 tst iperf3m4.py:8: Test FAILED (10.6 sec)
11:29:32.896632 run osmo-hlr_10.42.42.2(pid=2282): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:oc2g↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2282)]
11:29:32.948263 run osmo-hlr_10.42.42.2(pid=2282): stdout:
| (launched: 2022-12-10_11:29:29.410035)
11:29:33.000379 run osmo-hlr_10.42.42.2(pid=2282): stderr:
| �[0;m�[1;31m20221210112929589 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210112929589 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210112929589 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210112929589 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210112929589 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210112929589 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210112929601 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:oc2g/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210112929609 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210112929610 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210112929610 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:29:33.045096 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2271): Terminating (SIGTERM)
11:29:33.086994 run iperf3-srv_10.42.42.10(pid=2272): Terminating (SIGTERM)
11:29:33.129115 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2273): Terminating (SIGTERM)
11:29:33.171339 run iperf3-srv_10.42.42.10(pid=2274): Terminating (SIGTERM)
11:29:33.213592 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2275): Terminating (SIGTERM)
11:29:33.255515 run iperf3-srv_10.42.42.10(pid=2276): Terminating (SIGTERM)
11:29:33.297709 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2277): Terminating (SIGTERM)
11:29:33.339735 run iperf3-srv_10.42.42.10(pid=2278): Terminating (SIGTERM)
11:29:33.381804 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2281): Terminating (SIGTERM)
11:29:33.423925 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2284): Terminating (SIGTERM)
11:29:33.466068 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2287): Terminating (SIGTERM)
11:29:33.475860 --- ParallelTerminationStrategy: PID 2271 died...
11:29:33.532227 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2271): Terminated: ok {rc=0}
11:29:33.542618 --- ParallelTerminationStrategy: PID 2272 died...
11:29:33.598772 run iperf3-srv_10.42.42.10(pid=2272): Terminated {rc=256}
11:29:33.609109 --- ParallelTerminationStrategy: PID 2273 died...
11:29:33.665055 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2273): Terminated: ok {rc=0}
11:29:33.675193 --- ParallelTerminationStrategy: PID 2274 died...
11:29:33.731182 run iperf3-srv_10.42.42.10(pid=2274): Terminated {rc=256}
11:29:33.741488 --- ParallelTerminationStrategy: PID 2275 died...
11:29:33.797638 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2275): Terminated: ok {rc=0}
11:29:33.807912 --- ParallelTerminationStrategy: PID 2276 died...
11:29:33.864104 run iperf3-srv_10.42.42.10(pid=2276): Terminated {rc=256}
11:29:33.874425 --- ParallelTerminationStrategy: PID 2277 died...
11:29:33.930500 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=2277): Terminated: ok {rc=0}
11:29:33.940756 --- ParallelTerminationStrategy: PID 2278 died...
11:29:33.997274 run iperf3-srv_10.42.42.10(pid=2278): Terminated {rc=256}
11:29:34.007575 --- ParallelTerminationStrategy: PID 2281 died...
11:29:34.064050 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2281): Terminated: ok {rc=0}
11:29:34.074387 --- ParallelTerminationStrategy: PID 2284 died...
11:29:34.130681 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2284): Terminated: ok {rc=0}
11:29:34.140889 --- ParallelTerminationStrategy: PID 2287 died...
11:29:34.196926 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2287): Terminated: ok {rc=0}
11:29:34.284592 bus /gobi_4: Setting Powered False
11:29:35.389311 bus /gobi_4: Setting Powered False
11:29:36.500454 bus /gobi_1: Setting Powered False
11:29:37.605677 bus /gobi_1: Setting Powered False
11:29:38.712782 bus /gobi_0: Setting Powered False
11:29:39.813963 bus /gobi_0: Setting Powered False
11:29:40.926126 bus /gobi_6: Setting Powered False
236gprs:oc2g7.0osmo-stp_10.42.42.5(pid=2296): Process ended prematurely: osmo-stp_10.42.42.5(pid=2296) [trial-2374↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2296)]2139ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2296): Process ended prematurely: osmo-stp_10.42.42.5(pid=2296) [trial-2374↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2296)]
----------------------------------------------
trial-2374 gprs:oc2g ping.py
----------------------------------------------
11:29:41.992854 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:42.003994 tst gprs:oc2g: Using 1 x bts (candidates: 1)
11:29:42.126267 tst ping.py: using LAC 28083
11:29:42.244445 tst ping.py: using RAC 33
11:29:42.362624 tst ping.py: using CellId 28083
11:29:42.478680 tst ping.py: using BVCI 28084
11:29:42.501340 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:42.511712 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:42.522358 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:42.533125 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:42.543633 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:42.554324 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:42.565320 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:42.576019 tst gprs:oc2g: Using 1 x modem (candidates: 4)
11:29:42.849554 tst ping.py:19: start network...
11:29:42.973993 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:29:43.440700 run create_hlr_db(pid=2290): Launched
11:29:43.613055 bus /gobi_6: Setting Powered False
11:29:44.633817 run create_hlr_db(pid=2290): Terminated: ok {rc=0}
11:29:45.057829 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2292): Launched
11:29:45.379059 run osmo-hlr_10.42.42.2(pid=2293): Launched
11:29:45.497578 run osmo-stp_10.42.42.5: Starting osmo-stp
11:29:46.075911 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2295): Launched
11:29:46.520598 run osmo-stp_10.42.42.5(pid=2296): Launched
11:29:46.713343 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:29:47.292656 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2298): Launched
11:29:47.399603 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:29:47.659757 run patchelf(pid=2299): Launched
11:29:47.834672 run osmo-stp_10.42.42.5(pid=2296): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2296)]
11:29:47.961876 run osmo-stp_10.42.42.5(pid=2296): stdout:
| (launched: 2022-12-10_11:29:46.242066)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:48.089498 run osmo-stp_10.42.42.5(pid=2296): stderr:
| �[0;m�[38;5;43m20221210112946520 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112946520 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112946520 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112946532 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112946532 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112946532 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112946532 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:48.212851 run osmo-stp_10.42.42.5(pid=2296): stdout:
| (launched: 2022-12-10_11:29:46.242066)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:48.335451 run osmo-stp_10.42.42.5(pid=2296): stderr:
| �[0;m�[38;5;43m20221210112946520 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112946520 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112946520 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112946532 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112946532 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112946532 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112946532 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:48.478106 run patchelf(pid=2299): Terminating (SIGINT)
11:29:48.645004 run patchelf(pid=2299): Terminated: ok {rc=0}
11:29:48.675190 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=2296): Process ended prematurely: osmo-stp_10.42.42.5(pid=2296) [trial-2374↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2296)] [trial-2374↪gprs:oc2g↪ping.py:22]
11:29:48.683805 tst ping.py:22: Test FAILED (6.7 sec)
11:29:48.745330 run osmo-hlr_10.42.42.2(pid=2293): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:oc2g↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2293)]
11:29:48.794618 run osmo-hlr_10.42.42.2(pid=2293): stdout:
| (launched: 2022-12-10_11:29:45.220578)
11:29:48.845306 run osmo-hlr_10.42.42.2(pid=2293): stderr:
| �[0;m�[1;31m20221210112945418 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210112945418 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210112945418 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210112945418 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210112945418 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210112945418 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210112945430 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:oc2g/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210112945438 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210112945439 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210112945439 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:29:48.887580 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2292): Terminating (SIGTERM)
11:29:48.928138 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2295): Terminating (SIGTERM)
11:29:48.968578 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2298): Terminating (SIGTERM)
11:29:48.978307 --- ParallelTerminationStrategy: PID 2292 died...
11:29:49.032499 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2292): Terminated: ok {rc=0}
11:29:49.042443 --- ParallelTerminationStrategy: PID 2295 died...
11:29:49.095919 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2295): Terminated: ok {rc=0}
11:29:49.105954 --- ParallelTerminationStrategy: PID 2298 died...
11:29:49.159455 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2298): Terminated: ok {rc=0}
11:29:49.249092 bus /gobi_6: Setting Powered False
236gprs:oc2g7.0osmo-stp_10.42.42.5(pid=2306): Process ended prematurely: osmo-stp_10.42.42.5(pid=2306) [trial-2374↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2306)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=2306): Process ended prematurely: osmo-stp_10.42.42.5(pid=2306) [trial-2374↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2306)]
----------------------------------------------
trial-2374 gprs:oc2g ping_idle_ping.py
----------------------------------------------
11:29:50.300548 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:50.311761 tst gprs:oc2g: Using 1 x bts (candidates: 1)
11:29:50.433053 tst ping_idle_ping.py: using LAC 28084
11:29:50.557166 tst ping_idle_ping.py: using RAC 34
11:29:50.682359 tst ping_idle_ping.py: using CellId 28084
11:29:50.809531 tst ping_idle_ping.py: using BVCI 28085
11:29:50.833969 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:50.844850 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:50.855859 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:50.867271 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:50.878614 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:50.889808 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:50.901194 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
11:29:50.912417 tst gprs:oc2g: Using 1 x modem (candidates: 4)
11:29:51.205508 tst ping_idle_ping.py:24: start network...
11:29:51.339920 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:29:51.850266 run create_hlr_db(pid=2300): Launched
11:29:52.023387 bus /gobi_6: Setting Powered False
11:29:53.044062 run create_hlr_db(pid=2300): Terminated: ok {rc=0}
11:29:53.441130 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2302): Launched
11:29:53.741091 run osmo-hlr_10.42.42.2(pid=2303): Launched
11:29:53.857366 run osmo-stp_10.42.42.5: Starting osmo-stp
11:29:54.382251 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2305): Launched
11:29:54.681188 run osmo-stp_10.42.42.5(pid=2306): Launched
11:29:54.789772 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:29:55.309496 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2308): Launched
11:29:55.416136 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:29:55.676929 run patchelf(pid=2309): Launched
11:29:55.851816 run osmo-stp_10.42.42.5(pid=2306): ERR: Terminated: ERROR {rc=1} [trial-2374↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2306)]
11:29:55.979458 run osmo-stp_10.42.42.5(pid=2306): stdout:
| (launched: 2022-12-10_11:29:54.533495)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:56.107773 run osmo-stp_10.42.42.5(pid=2306): stderr:
| �[0;m�[38;5;43m20221210112954702 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112954702 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112954702 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112954703 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112954703 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112954703 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112954703 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:56.231179 run osmo-stp_10.42.42.5(pid=2306): stdout:
| (launched: 2022-12-10_11:29:54.533495)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
11:29:56.353868 run osmo-stp_10.42.42.5(pid=2306): stderr:
| �[0;m�[38;5;43m20221210112954702 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210112954702 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210112954702 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210112954703 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210112954703 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210112954703 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210112954703 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
11:29:56.496421 run patchelf(pid=2309): Terminating (SIGINT)
11:29:56.664058 run patchelf(pid=2309): Terminated: ok {rc=0}
11:29:56.694431 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=2306): Process ended prematurely: osmo-stp_10.42.42.5(pid=2306) [trial-2374↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=2306)] [trial-2374↪gprs:oc2g↪ping_idle_ping.py:27]
11:29:56.702888 tst ping_idle_ping.py:27: Test FAILED (6.4 sec)
11:29:56.763389 run osmo-hlr_10.42.42.2(pid=2303): ERR: Terminated: ERROR {rc=237} [trial-2374↪gprs:oc2g↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=2303)]
11:29:56.812258 run osmo-hlr_10.42.42.2(pid=2303): stdout:
| (launched: 2022-12-10_11:29:53.592796)
11:29:56.861591 run osmo-hlr_10.42.42.2(pid=2303): stderr:
| �[0;m�[1;31m20221210112953767 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210112953767 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210112953767 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210112953767 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210112953768 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210112953768 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210112953781 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2374/run.2022-12-10_11-15-10/gprs:oc2g/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210112953789 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210112953789 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210112953789 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
11:29:56.903679 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2302): Terminating (SIGTERM)
11:29:56.943816 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2305): Terminating (SIGTERM)
11:29:56.983508 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2308): Terminating (SIGTERM)
11:29:56.993169 --- ParallelTerminationStrategy: PID 2302 died...
11:29:57.046497 run pcap-recorder_any(filters='host 10.42.42.2')(pid=2302): Terminated: ok {rc=0}
11:29:57.056549 --- ParallelTerminationStrategy: PID 2305 died...
11:29:57.109685 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=2305): Terminated: ok {rc=0}
11:29:57.119663 --- ParallelTerminationStrategy: PID 2308 died...
11:29:57.172565 run pcap-recorder_any(filters='host 10.42.42.6')(pid=2308): Terminated: ok {rc=0}
11:29:57.255978 bus /gobi_6: Setting Powered False
5gprs:oc2g00