234gprs:oc2g9.0osmo-stp_10.42.42.5(pid=20481): Process ended prematurely: osmo-stp_10.42.42.5(pid=20481) [trial-2376↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20481)]2143cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=20481): Process ended prematurely: osmo-stp_10.42.42.5(pid=20481) [trial-2376↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20481)]
----------------------------------------------
trial-2376 gprs:oc2g cs_paging_gprs_active.py
----------------------------------------------
15:50:43.847082 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:43.857992 tst gprs:oc2g: Using 1 x bts (candidates: 1)
15:50:43.981677 tst cs_paging_gprs_active.py: using LAC 29002
15:50:44.101458 tst cs_paging_gprs_active.py: using RAC 187
15:50:44.220362 tst cs_paging_gprs_active.py: using CellId 29002
15:50:44.339539 tst cs_paging_gprs_active.py: using BVCI 29003
15:50:44.362523 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:44.372919 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:44.383399 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:44.394478 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:44.404814 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:44.415271 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:44.425813 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:44.436195 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:44.446762 tst gprs:oc2g: Using 1 x modem (candidates: 4)
15:50:44.561515 tst gprs:oc2g: Using 1 x modem (candidates: 4)
15:50:44.834755 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
15:50:44.959041 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:50:45.267332 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20471): Launched
15:50:45.522950 run iperf3-srv_10.42.42.10(pid=20472): Launched
15:50:45.704636 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
15:50:45.828847 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:50:46.165601 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20473): Launched
15:50:46.455437 run iperf3-srv_10.42.42.10(pid=20474): Launched
15:50:46.640695 tst cs_paging_gprs_active.py:38: start network...
15:50:46.767847 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:50:47.280185 run create_hlr_db(pid=20475): Launched
15:50:47.451233 bus /sierra_2: Setting Powered False
15:50:48.474084 run create_hlr_db(pid=20475): Terminated: ok {rc=0}
15:50:48.896255 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20477): Launched
15:50:49.214427 run osmo-hlr_10.42.42.2(pid=20478): Launched
15:50:49.330885 run osmo-stp_10.42.42.5: Starting osmo-stp
15:50:49.885621 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20480): Launched
15:50:50.203250 run osmo-stp_10.42.42.5(pid=20481): Launched
15:50:50.318748 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:50:50.871738 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20483): Launched
15:50:50.985488 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:50:51.263064 run patchelf(pid=20484): Launched
15:50:51.449908 run osmo-stp_10.42.42.5(pid=20481): ERR: Terminated: ERROR {rc=1} [trial-2376↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20481)]
15:50:51.587649 run osmo-stp_10.42.42.5(pid=20481): stdout:
| (launched: 2022-12-10_15:50:50.048097)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:50:51.725210 run osmo-stp_10.42.42.5(pid=20481): stderr:
| �[0;m�[38;5;43m20221210155050238 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155050238 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155050238 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155050239 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155050239 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155050239 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155050239 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:50:51.857804 run osmo-stp_10.42.42.5(pid=20481): stdout:
| (launched: 2022-12-10_15:50:50.048097)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:50:51.989070 run osmo-stp_10.42.42.5(pid=20481): stderr:
| �[0;m�[38;5;43m20221210155050238 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155050238 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155050238 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155050239 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155050239 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155050239 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155050239 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:50:52.142140 run patchelf(pid=20484): Terminating (SIGINT)
15:50:52.321647 run patchelf(pid=20484): Terminated: ok {rc=0}
15:50:52.354568 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=20481): Process ended prematurely: osmo-stp_10.42.42.5(pid=20481) [trial-2376↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20481)] [trial-2376↪gprs:oc2g↪cs_paging_gprs_active.py:38]
15:50:52.364211 tst cs_paging_gprs_active.py:38: Test FAILED (8.6 sec)
15:50:52.427182 run osmo-hlr_10.42.42.2(pid=20478): ERR: Terminated: ERROR {rc=237} [trial-2376↪gprs:oc2g↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=20478)]
15:50:52.478631 run osmo-hlr_10.42.42.2(pid=20478): stdout:
| (launched: 2022-12-10_15:50:49.058988)
15:50:52.530505 run osmo-hlr_10.42.42.2(pid=20478): stderr:
| �[0;m�[1;31m20221210155049256 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210155049256 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210155049256 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210155049256 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210155049256 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210155049256 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210155049268 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2376/run.2022-12-10_15-37-08/gprs:oc2g/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210155049276 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210155049277 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210155049277 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:50:52.574668 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20471): Terminating (SIGTERM)
15:50:52.616277 run iperf3-srv_10.42.42.10(pid=20472): Terminating (SIGTERM)
15:50:52.658007 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20473): Terminating (SIGTERM)
15:50:52.699733 run iperf3-srv_10.42.42.10(pid=20474): Terminating (SIGTERM)
15:50:52.741586 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20477): Terminating (SIGTERM)
15:50:52.783234 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20480): Terminating (SIGTERM)
15:50:52.825021 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20483): Terminating (SIGTERM)
15:50:52.834728 --- ParallelTerminationStrategy: PID 20471 died...
15:50:52.890598 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20471): Terminated: ok {rc=0}
15:50:52.900571 --- ParallelTerminationStrategy: PID 20472 died...
15:50:52.955749 run iperf3-srv_10.42.42.10(pid=20472): Terminated {rc=256}
15:50:52.965821 --- ParallelTerminationStrategy: PID 20473 died...
15:50:53.021884 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20473): Terminated: ok {rc=0}
15:50:53.031864 --- ParallelTerminationStrategy: PID 20474 died...
15:50:53.087075 run iperf3-srv_10.42.42.10(pid=20474): Terminated {rc=256}
15:50:53.097221 --- ParallelTerminationStrategy: PID 20477 died...
15:50:53.152510 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20477): Terminated: ok {rc=0}
15:50:53.162604 --- ParallelTerminationStrategy: PID 20480 died...
15:50:53.217816 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20480): Terminated: ok {rc=0}
15:50:53.227834 --- ParallelTerminationStrategy: PID 20483 died...
15:50:53.283585 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20483): Terminated: ok {rc=0}
15:50:53.369165 bus /gobi_0: Setting Powered False
15:50:54.476354 bus /gobi_0: Setting Powered False
15:50:55.584663 bus /gobi_6: Setting Powered False
234gprs:oc2g8.0osmo-stp_10.42.42.5(pid=20494): Process ended prematurely: osmo-stp_10.42.42.5(pid=20494) [trial-2376↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20494)]2143iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=20494): Process ended prematurely: osmo-stp_10.42.42.5(pid=20494) [trial-2376↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20494)]
----------------------------------------------
trial-2376 gprs:oc2g iperf3.py
----------------------------------------------
15:50:56.692835 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:56.703851 tst gprs:oc2g: Using 1 x bts (candidates: 1)
15:50:56.826364 tst iperf3.py: using LAC 29003
15:50:56.942353 tst iperf3.py: using RAC 188
15:50:57.058481 tst iperf3.py: using CellId 29003
15:50:57.176749 tst iperf3.py: using BVCI 29004
15:50:57.199539 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:57.210075 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:57.220421 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:57.231020 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:57.241613 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:57.252067 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:57.262816 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:57.273448 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:50:57.284047 tst gprs:oc2g: Using 1 x modem (candidates: 4)
15:50:57.559002 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
15:50:57.684202 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:50:57.992108 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20486): Launched
15:50:58.247647 run iperf3-srv_10.42.42.10(pid=20487): Launched
15:50:58.429356 tst iperf3.py:8: start network...
15:50:58.553399 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:50:59.019327 run create_hlr_db(pid=20488): Launched
15:50:59.191270 bus /gobi_6: Setting Powered False
15:51:00.213066 run create_hlr_db(pid=20488): Terminated: ok {rc=0}
15:51:00.636362 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20490): Launched
15:51:01.064865 run osmo-hlr_10.42.42.2(pid=20491): Launched
15:51:01.344721 run osmo-stp_10.42.42.5: Starting osmo-stp
15:51:01.967357 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20493): Launched
15:51:02.314822 run osmo-stp_10.42.42.5(pid=20494): Launched
15:51:02.430842 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:51:02.990879 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20496): Launched
15:51:03.106147 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:51:03.383395 run patchelf(pid=20497): Launched
15:51:03.570957 run osmo-stp_10.42.42.5(pid=20494): ERR: Terminated: ERROR {rc=1} [trial-2376↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20494)]
15:51:03.708735 run osmo-stp_10.42.42.5(pid=20494): stdout:
| (launched: 2022-12-10_15:51:02.124110)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:51:03.846385 run osmo-stp_10.42.42.5(pid=20494): stderr:
| �[0;m�[38;5;43m20221210155102341 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155102341 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155102342 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155102342 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155102342 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155102343 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155102343 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:51:03.979046 run osmo-stp_10.42.42.5(pid=20494): stdout:
| (launched: 2022-12-10_15:51:02.124110)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:51:04.111116 run osmo-stp_10.42.42.5(pid=20494): stderr:
| �[0;m�[38;5;43m20221210155102341 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155102341 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155102342 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155102342 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155102342 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155102343 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155102343 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:51:04.264073 run patchelf(pid=20497): Terminating (SIGINT)
15:51:04.444139 run patchelf(pid=20497): Terminated: ok {rc=0}
15:51:04.477274 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=20494): Process ended prematurely: osmo-stp_10.42.42.5(pid=20494) [trial-2376↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20494)] [trial-2376↪gprs:oc2g↪iperf3.py:8]
15:51:04.487067 tst iperf3.py:8: Test FAILED (7.8 sec)
15:51:04.550089 run osmo-hlr_10.42.42.2(pid=20491): ERR: Terminated: ERROR {rc=237} [trial-2376↪gprs:oc2g↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=20491)]
15:51:04.601671 run osmo-hlr_10.42.42.2(pid=20491): stdout:
| (launched: 2022-12-10_15:51:00.798821)
15:51:04.653879 run osmo-hlr_10.42.42.2(pid=20491): stderr:
| �[0;m�[1;31m20221210155101220 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210155101220 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210155101221 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210155101221 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210155101221 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210155101222 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210155101247 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2376/run.2022-12-10_15-37-08/gprs:oc2g/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210155101265 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210155101265 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210155101265 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:51:04.698936 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20486): Terminating (SIGTERM)
15:51:04.741010 run iperf3-srv_10.42.42.10(pid=20487): Terminating (SIGTERM)
15:51:04.783466 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20490): Terminating (SIGTERM)
15:51:04.825378 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20493): Terminating (SIGTERM)
15:51:04.867312 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20496): Terminating (SIGTERM)
15:51:04.877157 --- ParallelTerminationStrategy: PID 20486 died...
15:51:04.933638 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20486): Terminated: ok {rc=0}
15:51:04.943779 --- ParallelTerminationStrategy: PID 20487 died...
15:51:04.999730 run iperf3-srv_10.42.42.10(pid=20487): Terminated {rc=256}
15:51:05.010050 --- ParallelTerminationStrategy: PID 20490 died...
15:51:05.065955 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20490): Terminated: ok {rc=0}
15:51:05.076039 --- ParallelTerminationStrategy: PID 20493 died...
15:51:05.131818 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20493): Terminated: ok {rc=0}
15:51:05.142006 --- ParallelTerminationStrategy: PID 20496 died...
15:51:05.197780 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20496): Terminated: ok {rc=0}
15:51:05.285448 bus /gobi_6: Setting Powered False
238gprs:oc2g11.0osmo-stp_10.42.42.5(pid=20512): Process ended prematurely: osmo-stp_10.42.42.5(pid=20512) [trial-2376↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20512)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=20512): Process ended prematurely: osmo-stp_10.42.42.5(pid=20512) [trial-2376↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20512)]
----------------------------------------------
trial-2376 gprs:oc2g iperf3m4.py
----------------------------------------------
15:51:06.400779 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:06.412356 tst gprs:oc2g: Using 1 x bts (candidates: 1)
15:51:06.539631 tst iperf3m4.py: using LAC 29004
15:51:06.666598 tst iperf3m4.py: using RAC 189
15:51:06.788669 tst iperf3m4.py: using CellId 29004
15:51:06.905647 tst iperf3m4.py: using BVCI 29005
15:51:06.928498 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:06.939150 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:06.949765 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:06.960734 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:06.971409 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:06.982452 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:06.993241 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:07.003902 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:07.014741 tst gprs:oc2g: Using 1 x modem (candidates: 4)
15:51:07.129217 tst gprs:oc2g: Using 1 x modem (candidates: 4)
15:51:07.250975 tst gprs:oc2g: Using 1 x modem (candidates: 4)
15:51:07.364847 tst gprs:oc2g: Using 1 x modem (candidates: 4)
15:51:07.652558 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
15:51:07.785920 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:51:08.116203 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20498): Launched
15:51:08.389430 run iperf3-srv_10.42.42.10(pid=20499): Launched
15:51:08.585175 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
15:51:08.718092 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:51:09.047665 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20500): Launched
15:51:09.320254 run iperf3-srv_10.42.42.10(pid=20501): Launched
15:51:09.514811 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
15:51:09.647149 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:51:09.976118 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20502): Launched
15:51:10.249462 run iperf3-srv_10.42.42.10(pid=20503): Launched
15:51:10.444344 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
15:51:10.577039 run iperf3-srv_10.42.42.10: Starting iperf3-srv
15:51:10.905683 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20504): Launched
15:51:11.178320 run iperf3-srv_10.42.42.10(pid=20505): Launched
15:51:11.372871 tst iperf3m4.py:8: start network...
15:51:11.505319 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:51:12.014966 run create_hlr_db(pid=20506): Launched
15:51:12.199765 bus /gobi_6: Setting Powered False
15:51:13.210645 run create_hlr_db(pid=20506): Terminated: ok {rc=0}
15:51:13.641156 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20508): Launched
15:51:13.959224 run osmo-hlr_10.42.42.2(pid=20509): Launched
15:51:14.077671 run osmo-stp_10.42.42.5: Starting osmo-stp
15:51:14.631654 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20511): Launched
15:51:14.948098 run osmo-stp_10.42.42.5(pid=20512): Launched
15:51:15.068603 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:51:15.622362 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20514): Launched
15:51:15.735956 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:51:16.046527 run patchelf(pid=20515): Launched
15:51:16.229321 run osmo-stp_10.42.42.5(pid=20512): ERR: Terminated: ERROR {rc=1} [trial-2376↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20512)]
15:51:16.365359 run osmo-stp_10.42.42.5(pid=20512): stdout:
| (launched: 2022-12-10_15:51:14.793162)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:51:16.499934 run osmo-stp_10.42.42.5(pid=20512): stderr:
| �[0;m�[38;5;43m20221210155114979 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155114979 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155114980 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155114980 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155114981 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155114981 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155114981 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:51:16.630806 run osmo-stp_10.42.42.5(pid=20512): stdout:
| (launched: 2022-12-10_15:51:14.793162)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:51:16.760541 run osmo-stp_10.42.42.5(pid=20512): stderr:
| �[0;m�[38;5;43m20221210155114979 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155114979 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155114980 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155114980 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155114981 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155114981 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155114981 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:51:16.910381 run patchelf(pid=20515): Terminating (SIGINT)
15:51:17.086293 run patchelf(pid=20515): Terminated: ok {rc=0}
15:51:17.119273 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=20512): Process ended prematurely: osmo-stp_10.42.42.5(pid=20512) [trial-2376↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20512)] [trial-2376↪gprs:oc2g↪iperf3m4.py:8]
15:51:17.130606 tst iperf3m4.py:8: Test FAILED (10.8 sec)
15:51:17.195890 run osmo-hlr_10.42.42.2(pid=20509): ERR: Terminated: ERROR {rc=237} [trial-2376↪gprs:oc2g↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=20509)]
15:51:17.246696 run osmo-hlr_10.42.42.2(pid=20509): stdout:
| (launched: 2022-12-10_15:51:13.804038)
15:51:17.297603 run osmo-hlr_10.42.42.2(pid=20509): stderr:
| �[0;m�[1;31m20221210155113999 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210155113999 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210155113999 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210155113999 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210155113999 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210155113999 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210155114011 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2376/run.2022-12-10_15-37-08/gprs:oc2g/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210155114020 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210155114020 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210155114020 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:51:17.340125 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20498): Terminating (SIGTERM)
15:51:17.380770 run iperf3-srv_10.42.42.10(pid=20499): Terminating (SIGTERM)
15:51:17.421533 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20500): Terminating (SIGTERM)
15:51:17.462188 run iperf3-srv_10.42.42.10(pid=20501): Terminating (SIGTERM)
15:51:17.502947 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20502): Terminating (SIGTERM)
15:51:17.543583 run iperf3-srv_10.42.42.10(pid=20503): Terminating (SIGTERM)
15:51:17.584176 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20504): Terminating (SIGTERM)
15:51:17.625457 run iperf3-srv_10.42.42.10(pid=20505): Terminating (SIGTERM)
15:51:17.665441 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20508): Terminating (SIGTERM)
15:51:17.706095 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20511): Terminating (SIGTERM)
15:51:17.747161 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20514): Terminating (SIGTERM)
15:51:17.756898 --- ParallelTerminationStrategy: PID 20498 died...
15:51:17.811313 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20498): Terminated: ok {rc=0}
15:51:17.821354 --- ParallelTerminationStrategy: PID 20499 died...
15:51:17.874685 run iperf3-srv_10.42.42.10(pid=20499): Terminated {rc=256}
15:51:17.884626 --- ParallelTerminationStrategy: PID 20500 died...
15:51:17.937906 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20500): Terminated: ok {rc=0}
15:51:17.947853 --- ParallelTerminationStrategy: PID 20501 died...
15:51:18.000933 run iperf3-srv_10.42.42.10(pid=20501): Terminated {rc=256}
15:51:18.010923 --- ParallelTerminationStrategy: PID 20502 died...
15:51:18.064220 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20502): Terminated: ok {rc=0}
15:51:18.074270 --- ParallelTerminationStrategy: PID 20503 died...
15:51:18.127684 run iperf3-srv_10.42.42.10(pid=20503): Terminated {rc=256}
15:51:18.137845 --- ParallelTerminationStrategy: PID 20504 died...
15:51:18.192040 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20504): Terminated: ok {rc=0}
15:51:18.201927 --- ParallelTerminationStrategy: PID 20505 died...
15:51:18.255544 run iperf3-srv_10.42.42.10(pid=20505): Terminated {rc=256}
15:51:18.265750 --- ParallelTerminationStrategy: PID 20508 died...
15:51:18.318885 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20508): Terminated: ok {rc=0}
15:51:18.328898 --- ParallelTerminationStrategy: PID 20511 died...
15:51:18.382569 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20511): Terminated: ok {rc=0}
15:51:18.392313 --- ParallelTerminationStrategy: PID 20514 died...
15:51:18.446322 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20514): Terminated: ok {rc=0}
15:51:18.530273 bus /gobi_4: Setting Powered False
15:51:19.637231 bus /gobi_4: Setting Powered False
15:51:20.748818 bus /gobi_1: Setting Powered False
15:51:21.852740 bus /gobi_1: Setting Powered False
15:51:22.959563 bus /gobi_0: Setting Powered False
15:51:24.066163 bus /gobi_0: Setting Powered False
15:51:25.179111 bus /gobi_6: Setting Powered False
238gprs:oc2g7.0osmo-stp_10.42.42.5(pid=20522): Process ended prematurely: osmo-stp_10.42.42.5(pid=20522) [trial-2376↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20522)]2139ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=20522): Process ended prematurely: osmo-stp_10.42.42.5(pid=20522) [trial-2376↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20522)]
----------------------------------------------
trial-2376 gprs:oc2g ping.py
----------------------------------------------
15:51:26.245554 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:26.256481 tst gprs:oc2g: Using 1 x bts (candidates: 1)
15:51:26.378634 tst ping.py: using LAC 29005
15:51:26.494237 tst ping.py: using RAC 190
15:51:26.609701 tst ping.py: using CellId 29005
15:51:26.726293 tst ping.py: using BVCI 29006
15:51:26.748917 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:26.759430 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:26.769810 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:26.780365 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:26.790860 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:26.801427 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:26.812377 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:26.822998 tst gprs:oc2g: Using 1 x modem (candidates: 4)
15:51:27.095134 tst ping.py:19: start network...
15:51:27.219555 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:51:27.683838 run create_hlr_db(pid=20516): Launched
15:51:27.855142 bus /gobi_6: Setting Powered False
15:51:28.876189 run create_hlr_db(pid=20516): Terminated: ok {rc=0}
15:51:29.297734 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20518): Launched
15:51:29.616294 run osmo-hlr_10.42.42.2(pid=20519): Launched
15:51:29.733421 run osmo-stp_10.42.42.5: Starting osmo-stp
15:51:30.288688 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20521): Launched
15:51:30.608104 run osmo-stp_10.42.42.5(pid=20522): Launched
15:51:30.723430 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:51:31.291727 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20524): Launched
15:51:31.403316 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:51:31.684283 run patchelf(pid=20525): Launched
15:51:31.866059 run osmo-stp_10.42.42.5(pid=20522): ERR: Terminated: ERROR {rc=1} [trial-2376↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20522)]
15:51:31.999179 run osmo-stp_10.42.42.5(pid=20522): stdout:
| (launched: 2022-12-10_15:51:30.451249)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:51:32.132256 run osmo-stp_10.42.42.5(pid=20522): stderr:
| �[0;m�[38;5;43m20221210155130643 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155130643 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155130643 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155130644 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155130644 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155130644 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155130644 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:51:32.257052 run osmo-stp_10.42.42.5(pid=20522): stdout:
| (launched: 2022-12-10_15:51:30.451249)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:51:32.380037 run osmo-stp_10.42.42.5(pid=20522): stderr:
| �[0;m�[38;5;43m20221210155130643 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155130643 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155130643 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155130644 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155130644 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155130644 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155130644 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:51:32.522536 run patchelf(pid=20525): Terminating (SIGINT)
15:51:32.690027 run patchelf(pid=20525): Terminated: ok {rc=0}
15:51:32.720381 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=20522): Process ended prematurely: osmo-stp_10.42.42.5(pid=20522) [trial-2376↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20522)] [trial-2376↪gprs:oc2g↪ping.py:22]
15:51:32.729295 tst ping.py:22: Test FAILED (6.5 sec)
15:51:32.790840 run osmo-hlr_10.42.42.2(pid=20519): ERR: Terminated: ERROR {rc=237} [trial-2376↪gprs:oc2g↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=20519)]
15:51:32.840275 run osmo-hlr_10.42.42.2(pid=20519): stdout:
| (launched: 2022-12-10_15:51:29.459210)
15:51:32.890410 run osmo-hlr_10.42.42.2(pid=20519): stderr:
| �[0;m�[1;31m20221210155129658 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210155129658 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210155129658 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210155129658 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210155129658 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210155129658 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210155129673 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2376/run.2022-12-10_15-37-08/gprs:oc2g/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210155129682 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210155129683 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210155129683 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:51:32.933000 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20518): Terminating (SIGTERM)
15:51:32.973597 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20521): Terminating (SIGTERM)
15:51:33.014134 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20524): Terminating (SIGTERM)
15:51:33.023802 --- ParallelTerminationStrategy: PID 20518 died...
15:51:33.077796 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20518): Terminated: ok {rc=0}
15:51:33.088007 --- ParallelTerminationStrategy: PID 20521 died...
15:51:33.142493 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20521): Terminated: ok {rc=0}
15:51:33.152219 --- ParallelTerminationStrategy: PID 20524 died...
15:51:33.206349 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20524): Terminated: ok {rc=0}
15:51:33.296299 bus /gobi_6: Setting Powered False
238gprs:oc2g7.0osmo-stp_10.42.42.5(pid=20532): Process ended prematurely: osmo-stp_10.42.42.5(pid=20532) [trial-2376↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20532)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=20532): Process ended prematurely: osmo-stp_10.42.42.5(pid=20532) [trial-2376↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20532)]
----------------------------------------------
trial-2376 gprs:oc2g ping_idle_ping.py
----------------------------------------------
15:51:34.348533 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:34.360123 tst gprs:oc2g: Using 1 x bts (candidates: 1)
15:51:34.489806 tst ping_idle_ping.py: using LAC 29006
15:51:34.619649 tst ping_idle_ping.py: using RAC 191
15:51:34.742802 tst ping_idle_ping.py: using CellId 29006
15:51:34.865882 tst ping_idle_ping.py: using BVCI 29007
15:51:34.890463 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:34.901610 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:34.912491 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:34.923823 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:34.934849 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:34.945851 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:34.957002 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
15:51:34.967954 tst gprs:oc2g: Using 1 x modem (candidates: 4)
15:51:35.242152 tst ping_idle_ping.py:24: start network...
15:51:35.365951 run osmo-hlr_10.42.42.2: Starting osmo-hlr
15:51:35.837409 run create_hlr_db(pid=20526): Launched
15:51:36.010406 bus /gobi_6: Setting Powered False
15:51:37.031002 run create_hlr_db(pid=20526): Terminated: ok {rc=0}
15:51:37.454844 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20528): Launched
15:51:37.773154 run osmo-hlr_10.42.42.2(pid=20529): Launched
15:51:37.893049 run osmo-stp_10.42.42.5: Starting osmo-stp
15:51:38.446074 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20531): Launched
15:51:38.764497 run osmo-stp_10.42.42.5(pid=20532): Launched
15:51:38.881398 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
15:51:39.437353 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20534): Launched
15:51:39.550848 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
15:51:39.828110 run patchelf(pid=20535): Launched
15:51:40.013846 run osmo-stp_10.42.42.5(pid=20532): ERR: Terminated: ERROR {rc=1} [trial-2376↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20532)]
15:51:40.150331 run osmo-stp_10.42.42.5(pid=20532): stdout:
| (launched: 2022-12-10_15:51:38.608021)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:51:40.286537 run osmo-stp_10.42.42.5(pid=20532): stderr:
| �[0;m�[38;5;43m20221210155138801 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155138801 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155138801 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155138802 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155138802 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155138802 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155138802 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:51:40.418152 run osmo-stp_10.42.42.5(pid=20532): stdout:
| (launched: 2022-12-10_15:51:38.608021)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
15:51:40.548446 run osmo-stp_10.42.42.5(pid=20532): stderr:
| �[0;m�[38;5;43m20221210155138801 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210155138801 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210155138801 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210155138802 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210155138802 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210155138802 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210155138802 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
15:51:40.700710 run patchelf(pid=20535): Terminating (SIGINT)
15:51:40.879020 run patchelf(pid=20535): Terminated: ok {rc=0}
15:51:40.911236 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=20532): Process ended prematurely: osmo-stp_10.42.42.5(pid=20532) [trial-2376↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=20532)] [trial-2376↪gprs:oc2g↪ping_idle_ping.py:27]
15:51:40.920288 tst ping_idle_ping.py:27: Test FAILED (6.6 sec)
15:51:40.983112 run osmo-hlr_10.42.42.2(pid=20529): ERR: Terminated: ERROR {rc=237} [trial-2376↪gprs:oc2g↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=20529)]
15:51:41.034646 run osmo-hlr_10.42.42.2(pid=20529): stdout:
| (launched: 2022-12-10_15:51:37.617312)
15:51:41.086591 run osmo-hlr_10.42.42.2(pid=20529): stderr:
| �[0;m�[1;31m20221210155137815 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210155137815 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210155137815 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210155137815 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210155137815 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210155137815 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210155137830 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2376/run.2022-12-10_15-37-08/gprs:oc2g/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210155137838 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210155137839 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210155137839 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
15:51:41.130808 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20528): Terminating (SIGTERM)
15:51:41.172498 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20531): Terminating (SIGTERM)
15:51:41.214288 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20534): Terminating (SIGTERM)
15:51:41.224029 --- ParallelTerminationStrategy: PID 20528 died...
15:51:41.279775 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20528): Terminated: ok {rc=0}
15:51:41.289933 --- ParallelTerminationStrategy: PID 20531 died...
15:51:41.345635 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20531): Terminated: ok {rc=0}
15:51:41.355831 --- ParallelTerminationStrategy: PID 20534 died...
15:51:41.411793 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20534): Terminated: ok {rc=0}
15:51:41.500546 bus /gobi_6: Setting Powered False
5gprs:oc2g00