208gprs:oc2g9.0osmo-stp_10.42.42.5(pid=18679): Process ended prematurely: osmo-stp_10.42.42.5(pid=18679) [trial-2350↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18679)]2143cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18679): Process ended prematurely: osmo-stp_10.42.42.5(pid=18679) [trial-2350↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18679)]
----------------------------------------------
trial-2350 gprs:oc2g cs_paging_gprs_active.py
----------------------------------------------
21:58:08.337538 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:08.347844 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:58:08.481296 tst cs_paging_gprs_active.py: using LAC 17016
21:58:08.607402 tst cs_paging_gprs_active.py: using RAC 186
21:58:08.733757 tst cs_paging_gprs_active.py: using CellId 17016
21:58:08.853365 tst cs_paging_gprs_active.py: using BVCI 17017
21:58:08.876857 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:08.887278 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:08.897536 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:08.907697 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:08.917784 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:08.927787 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:08.937960 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:08.948059 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:08.958355 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:09.078979 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:09.376437 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
21:58:09.514218 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:09.855330 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18669): Launched
21:58:10.139016 run iperf3-srv_10.42.42.10(pid=18670): Launched
21:58:10.341582 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
21:58:10.479255 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:10.818895 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18671): Launched
21:58:11.102835 run iperf3-srv_10.42.42.10(pid=18672): Launched
21:58:11.305063 tst cs_paging_gprs_active.py:38: start network...
21:58:11.443685 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:11.971179 run create_hlr_db(pid=18673): Launched
21:58:12.161789 bus /sierra_2: Setting Powered False
21:58:13.184354 run create_hlr_db(pid=18673): Terminated: ok {rc=0}
21:58:13.596539 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18675): Launched
21:58:13.903621 run osmo-hlr_10.42.42.2(pid=18676): Launched
21:58:14.015876 run osmo-stp_10.42.42.5: Starting osmo-stp
21:58:14.549023 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18678): Launched
21:58:14.855102 run osmo-stp_10.42.42.5(pid=18679): Launched
21:58:14.965572 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:58:15.500266 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18681): Launched
21:58:15.609924 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:58:15.885025 run patchelf(pid=18682): Launched
21:58:16.079569 run osmo-stp_10.42.42.5(pid=18679): ERR: Terminated: ERROR {rc=1} [trial-2350↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18679)]
21:58:16.215203 run osmo-stp_10.42.42.5(pid=18679): stdout:
| (launched: 2022-12-08_21:58:14.705275)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:16.351109 run osmo-stp_10.42.42.5(pid=18679): stderr:
| �[0;m�[38;5;43m20221208215814878 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215814878 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215814878 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215814879 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215814879 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215814879 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215814879 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:16.482537 run osmo-stp_10.42.42.5(pid=18679): stdout:
| (launched: 2022-12-08_21:58:14.705275)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:16.613076 run osmo-stp_10.42.42.5(pid=18679): stderr:
| �[0;m�[38;5;43m20221208215814878 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215814878 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215814878 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215814879 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215814879 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215814879 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215814879 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:16.764316 run patchelf(pid=18682): Terminating (SIGINT)
21:58:16.944110 run patchelf(pid=18682): Terminated: ok {rc=0}
21:58:16.977607 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=18679): Process ended prematurely: osmo-stp_10.42.42.5(pid=18679) [trial-2350↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18679)] [trial-2350↪gprs:oc2g↪cs_paging_gprs_active.py:38]
21:58:16.987893 tst cs_paging_gprs_active.py:38: Test FAILED (8.7 sec)
21:58:17.050518 run osmo-hlr_10.42.42.2(pid=18676): ERR: Terminated: ERROR {rc=237} [trial-2350↪gprs:oc2g↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18676)]
21:58:17.101890 run osmo-hlr_10.42.42.2(pid=18676): stdout:
| (launched: 2022-12-08_21:58:13.753311)
21:58:17.153668 run osmo-hlr_10.42.42.2(pid=18676): stderr:
| �[0;m�[1;31m20221208215813934 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221208215813934 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221208215813934 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221208215813934 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221208215813934 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221208215813934 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221208215813947 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2350/run.2022-12-08_21-44-24/gprs:oc2g/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221208215813958 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221208215813958 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221208215813958 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:58:17.196347 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18669): Terminating (SIGTERM)
21:58:17.236221 run iperf3-srv_10.42.42.10(pid=18670): Terminating (SIGTERM)
21:58:17.275985 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18671): Terminating (SIGTERM)
21:58:17.315722 run iperf3-srv_10.42.42.10(pid=18672): Terminating (SIGTERM)
21:58:17.355784 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18675): Terminating (SIGTERM)
21:58:17.395764 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18678): Terminating (SIGTERM)
21:58:17.435907 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18681): Terminating (SIGTERM)
21:58:17.445365 --- ParallelTerminationStrategy: PID 18669 died...
21:58:17.499483 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18669): Terminated: ok {rc=0}
21:58:17.509319 --- ParallelTerminationStrategy: PID 18670 died...
21:58:17.563341 run iperf3-srv_10.42.42.10(pid=18670): Terminated {rc=256}
21:58:17.573147 --- ParallelTerminationStrategy: PID 18671 died...
21:58:17.626755 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18671): Terminated: ok {rc=0}
21:58:17.636424 --- ParallelTerminationStrategy: PID 18672 died...
21:58:17.690019 run iperf3-srv_10.42.42.10(pid=18672): Terminated {rc=256}
21:58:17.699690 --- ParallelTerminationStrategy: PID 18675 died...
21:58:17.753078 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18675): Terminated: ok {rc=0}
21:58:17.762854 --- ParallelTerminationStrategy: PID 18678 died...
21:58:17.816331 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18678): Terminated: ok {rc=0}
21:58:17.826141 --- ParallelTerminationStrategy: PID 18681 died...
21:58:17.880020 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18681): Terminated: ok {rc=0}
21:58:17.964453 bus /gobi_0: Setting Powered False
21:58:19.073068 bus /gobi_0: Setting Powered False
21:58:20.186027 bus /gobi_6: Setting Powered False
208gprs:oc2g8.0osmo-stp_10.42.42.5(pid=18691): Process ended prematurely: osmo-stp_10.42.42.5(pid=18691) [trial-2350↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18691)]2143iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18691): Process ended prematurely: osmo-stp_10.42.42.5(pid=18691) [trial-2350↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18691)]
----------------------------------------------
trial-2350 gprs:oc2g iperf3.py
----------------------------------------------
21:58:21.297977 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:21.308762 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:58:21.431835 tst iperf3.py: using LAC 17017
21:58:21.553993 tst iperf3.py: using RAC 187
21:58:21.681868 tst iperf3.py: using CellId 17017
21:58:21.812151 tst iperf3.py: using BVCI 17018
21:58:21.837678 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:21.848677 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:21.859674 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:21.870978 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:21.882509 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:21.893733 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:21.905060 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:21.916340 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:21.927771 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:22.209264 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
21:58:22.336730 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:22.653618 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18683): Launched
21:58:22.918305 run iperf3-srv_10.42.42.10(pid=18684): Launched
21:58:23.104482 tst iperf3.py:8: start network...
21:58:23.232756 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:23.724680 run create_hlr_db(pid=18685): Launched
21:58:23.900588 bus /gobi_6: Setting Powered False
21:58:24.922445 run create_hlr_db(pid=18685): Terminated: ok {rc=0}
21:58:25.360074 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18687): Launched
21:58:25.689020 run osmo-hlr_10.42.42.2(pid=18688): Launched
21:58:25.811627 run osmo-stp_10.42.42.5: Starting osmo-stp
21:58:26.384847 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18690): Launched
21:58:26.713777 run osmo-stp_10.42.42.5(pid=18691): Launched
21:58:26.833882 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:58:27.410133 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18693): Launched
21:58:27.528715 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:58:27.816333 run patchelf(pid=18694): Launched
21:58:28.009066 run osmo-stp_10.42.42.5(pid=18691): ERR: Terminated: ERROR {rc=1} [trial-2350↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18691)]
21:58:28.149998 run osmo-stp_10.42.42.5(pid=18691): stdout:
| (launched: 2022-12-08_21:58:26.552312)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:28.291237 run osmo-stp_10.42.42.5(pid=18691): stderr:
| �[0;m�[38;5;43m20221208215826750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215826750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215826750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215826750 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215826751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215826751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215826751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:28.428813 run osmo-stp_10.42.42.5(pid=18691): stdout:
| (launched: 2022-12-08_21:58:26.552312)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:28.564392 run osmo-stp_10.42.42.5(pid=18691): stderr:
| �[0;m�[38;5;43m20221208215826750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215826750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215826750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215826750 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215826751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215826751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215826751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:28.721965 run patchelf(pid=18694): Terminating (SIGINT)
21:58:28.906952 run patchelf(pid=18694): Terminated: ok {rc=0}
21:58:28.940596 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=18691): Process ended prematurely: osmo-stp_10.42.42.5(pid=18691) [trial-2350↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18691)] [trial-2350↪gprs:oc2g↪iperf3.py:8]
21:58:28.950695 tst iperf3.py:8: Test FAILED (7.7 sec)
21:58:29.015617 run osmo-hlr_10.42.42.2(pid=18688): ERR: Terminated: ERROR {rc=237} [trial-2350↪gprs:oc2g↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18688)]
21:58:29.068738 run osmo-hlr_10.42.42.2(pid=18688): stdout:
| (launched: 2022-12-08_21:58:25.527695)
21:58:29.122470 run osmo-hlr_10.42.42.2(pid=18688): stderr:
| �[0;m�[1;31m20221208215825727 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221208215825727 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221208215825727 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221208215825727 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221208215825727 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221208215825727 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221208215825740 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2350/run.2022-12-08_21-44-24/gprs:oc2g/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221208215825748 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221208215825748 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221208215825749 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:58:29.167663 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18683): Terminating (SIGTERM)
21:58:29.210945 run iperf3-srv_10.42.42.10(pid=18684): Terminating (SIGTERM)
21:58:29.253775 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18687): Terminating (SIGTERM)
21:58:29.296685 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18690): Terminating (SIGTERM)
21:58:29.340213 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18693): Terminating (SIGTERM)
21:58:29.350524 --- ParallelTerminationStrategy: PID 18683 died...
21:58:29.408132 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18683): Terminated: ok {rc=0}
21:58:29.418916 --- ParallelTerminationStrategy: PID 18684 died...
21:58:29.476191 run iperf3-srv_10.42.42.10(pid=18684): Terminated {rc=256}
21:58:29.486708 --- ParallelTerminationStrategy: PID 18687 died...
21:58:29.543998 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18687): Terminated: ok {rc=0}
21:58:29.554352 --- ParallelTerminationStrategy: PID 18690 died...
21:58:29.611624 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18690): Terminated: ok {rc=0}
21:58:29.622125 --- ParallelTerminationStrategy: PID 18693 died...
21:58:29.679472 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18693): Terminated: ok {rc=0}
21:58:29.767466 bus /gobi_6: Setting Powered False
212gprs:oc2g11.0osmo-stp_10.42.42.5(pid=18709): Process ended prematurely: osmo-stp_10.42.42.5(pid=18709) [trial-2350↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18709)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18709): Process ended prematurely: osmo-stp_10.42.42.5(pid=18709) [trial-2350↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18709)]
----------------------------------------------
trial-2350 gprs:oc2g iperf3m4.py
----------------------------------------------
21:58:30.891360 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:30.903519 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:58:31.043193 tst iperf3m4.py: using LAC 17018
21:58:31.173858 tst iperf3m4.py: using RAC 188
21:58:31.299098 tst iperf3m4.py: using CellId 17018
21:58:31.429696 tst iperf3m4.py: using BVCI 17019
21:58:31.456229 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:31.468198 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:31.480156 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:31.494089 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:31.507179 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:31.519571 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:31.532263 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:31.545023 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:31.557738 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:31.680527 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:31.809360 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:31.963454 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:32.305067 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
21:58:32.433847 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:32.754199 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18695): Launched
21:58:33.019309 run iperf3-srv_10.42.42.10(pid=18696): Launched
21:58:33.205895 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
21:58:33.334067 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:33.651637 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18697): Launched
21:58:33.914961 run iperf3-srv_10.42.42.10(pid=18698): Launched
21:58:34.101627 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
21:58:34.229248 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:34.546926 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18699): Launched
21:58:34.810459 run iperf3-srv_10.42.42.10(pid=18700): Launched
21:58:34.996329 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
21:58:35.123406 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:58:35.441412 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18701): Launched
21:58:35.704928 run iperf3-srv_10.42.42.10(pid=18702): Launched
21:58:35.891339 tst iperf3m4.py:8: start network...
21:58:36.018849 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:36.510682 run create_hlr_db(pid=18703): Launched
21:58:36.687100 bus /gobi_6: Setting Powered False
21:58:37.710270 run create_hlr_db(pid=18703): Terminated: ok {rc=0}
21:58:38.147005 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18705): Launched
21:58:38.476578 run osmo-hlr_10.42.42.2(pid=18706): Launched
21:58:38.599000 run osmo-stp_10.42.42.5: Starting osmo-stp
21:58:39.172578 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18708): Launched
21:58:39.502349 run osmo-stp_10.42.42.5(pid=18709): Launched
21:58:39.622743 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:58:40.197649 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18711): Launched
21:58:40.315422 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:58:40.602619 run patchelf(pid=18712): Launched
21:58:40.795244 run osmo-stp_10.42.42.5(pid=18709): ERR: Terminated: ERROR {rc=1} [trial-2350↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18709)]
21:58:40.936020 run osmo-stp_10.42.42.5(pid=18709): stdout:
| (launched: 2022-12-08_21:58:39.339938)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:41.077639 run osmo-stp_10.42.42.5(pid=18709): stderr:
| �[0;m�[38;5;43m20221208215839533 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215839533 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215839533 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215839533 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215839536 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215839536 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215839536 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:41.214164 run osmo-stp_10.42.42.5(pid=18709): stdout:
| (launched: 2022-12-08_21:58:39.339938)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:41.350637 run osmo-stp_10.42.42.5(pid=18709): stderr:
| �[0;m�[38;5;43m20221208215839533 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215839533 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215839533 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215839533 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215839536 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215839536 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215839536 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:41.508367 run patchelf(pid=18712): Terminating (SIGINT)
21:58:41.692808 run patchelf(pid=18712): Terminated: ok {rc=0}
21:58:41.726584 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=18709): Process ended prematurely: osmo-stp_10.42.42.5(pid=18709) [trial-2350↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18709)] [trial-2350↪gprs:oc2g↪iperf3m4.py:8]
21:58:41.736428 tst iperf3m4.py:8: Test FAILED (10.9 sec)
21:58:41.801650 run osmo-hlr_10.42.42.2(pid=18706): ERR: Terminated: ERROR {rc=237} [trial-2350↪gprs:oc2g↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18706)]
21:58:41.854243 run osmo-hlr_10.42.42.2(pid=18706): stdout:
| (launched: 2022-12-08_21:58:38.314293)
21:58:41.907468 run osmo-hlr_10.42.42.2(pid=18706): stderr:
| �[0;m�[1;31m20221208215838513 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221208215838513 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221208215838513 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221208215838513 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221208215838513 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221208215838513 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221208215838526 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2350/run.2022-12-08_21-44-24/gprs:oc2g/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221208215838534 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221208215838534 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221208215838534 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:58:41.952387 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18695): Terminating (SIGTERM)
21:58:41.995108 run iperf3-srv_10.42.42.10(pid=18696): Terminating (SIGTERM)
21:58:42.038068 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18697): Terminating (SIGTERM)
21:58:42.081008 run iperf3-srv_10.42.42.10(pid=18698): Terminating (SIGTERM)
21:58:42.123615 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18699): Terminating (SIGTERM)
21:58:42.166287 run iperf3-srv_10.42.42.10(pid=18700): Terminating (SIGTERM)
21:58:42.209250 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18701): Terminating (SIGTERM)
21:58:42.251820 run iperf3-srv_10.42.42.10(pid=18702): Terminating (SIGTERM)
21:58:42.294604 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18705): Terminating (SIGTERM)
21:58:42.337064 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18708): Terminating (SIGTERM)
21:58:42.380319 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18711): Terminating (SIGTERM)
21:58:42.390295 --- ParallelTerminationStrategy: PID 18695 died...
21:58:42.447814 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18695): Terminated: ok {rc=0}
21:58:42.458176 --- ParallelTerminationStrategy: PID 18696 died...
21:58:42.515286 run iperf3-srv_10.42.42.10(pid=18696): Terminated {rc=256}
21:58:42.525681 --- ParallelTerminationStrategy: PID 18697 died...
21:58:42.582679 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18697): Terminated: ok {rc=0}
21:58:42.592921 --- ParallelTerminationStrategy: PID 18698 died...
21:58:42.649663 run iperf3-srv_10.42.42.10(pid=18698): Terminated {rc=256}
21:58:42.660139 --- ParallelTerminationStrategy: PID 18699 died...
21:58:42.716901 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18699): Terminated: ok {rc=0}
21:58:42.727508 --- ParallelTerminationStrategy: PID 18700 died...
21:58:42.784472 run iperf3-srv_10.42.42.10(pid=18700): Terminated {rc=256}
21:58:42.795088 --- ParallelTerminationStrategy: PID 18701 died...
21:58:42.852237 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18701): Terminated: ok {rc=0}
21:58:42.862567 --- ParallelTerminationStrategy: PID 18702 died...
21:58:42.919499 run iperf3-srv_10.42.42.10(pid=18702): Terminated {rc=256}
21:58:42.930002 --- ParallelTerminationStrategy: PID 18705 died...
21:58:42.987031 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18705): Terminated: ok {rc=0}
21:58:42.997677 --- ParallelTerminationStrategy: PID 18708 died...
21:58:43.054859 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18708): Terminated: ok {rc=0}
21:58:43.065374 --- ParallelTerminationStrategy: PID 18711 died...
21:58:43.122536 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18711): Terminated: ok {rc=0}
21:58:43.210541 bus /gobi_4: Setting Powered False
21:58:44.319415 bus /gobi_4: Setting Powered False
21:58:45.429096 bus /gobi_1: Setting Powered False
21:58:46.534136 bus /gobi_1: Setting Powered False
21:58:47.642497 bus /gobi_0: Setting Powered False
21:58:48.749575 bus /gobi_0: Setting Powered False
21:58:49.862643 bus /gobi_6: Setting Powered False
212gprs:oc2g7.0osmo-stp_10.42.42.5(pid=18719): Process ended prematurely: osmo-stp_10.42.42.5(pid=18719) [trial-2350↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18719)]2139ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18719): Process ended prematurely: osmo-stp_10.42.42.5(pid=18719) [trial-2350↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18719)]
----------------------------------------------
trial-2350 gprs:oc2g ping.py
----------------------------------------------
21:58:50.932533 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:50.943826 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:58:51.075545 tst ping.py: using LAC 17019
21:58:51.202390 tst ping.py: using RAC 189
21:58:51.329244 tst ping.py: using CellId 17019
21:58:51.458373 tst ping.py: using BVCI 17020
21:58:51.483272 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:51.494214 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:51.504917 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:51.515943 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:51.526949 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:51.537931 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:51.548867 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:51.559948 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:58:51.835729 tst ping.py:19: start network...
21:58:51.962653 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:58:52.438314 run create_hlr_db(pid=18713): Launched
21:58:52.613815 bus /gobi_6: Setting Powered False
21:58:53.633953 run create_hlr_db(pid=18713): Terminated: ok {rc=0}
21:58:54.063539 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18715): Launched
21:58:54.388504 run osmo-hlr_10.42.42.2(pid=18716): Launched
21:58:54.506871 run osmo-stp_10.42.42.5: Starting osmo-stp
21:58:55.074069 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18718): Launched
21:58:55.397654 run osmo-stp_10.42.42.5(pid=18719): Launched
21:58:55.514619 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:58:56.079307 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18721): Launched
21:58:56.194985 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:58:56.478201 run patchelf(pid=18722): Launched
21:58:56.669128 run osmo-stp_10.42.42.5(pid=18719): ERR: Terminated: ERROR {rc=1} [trial-2350↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18719)]
21:58:56.808249 run osmo-stp_10.42.42.5(pid=18719): stdout:
| (launched: 2022-12-08_21:58:55.239048)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:56.947511 run osmo-stp_10.42.42.5(pid=18719): stderr:
| �[0;m�[38;5;43m20221208215855433 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215855434 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215855434 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215855434 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215855435 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215855435 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215855435 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:57.082636 run osmo-stp_10.42.42.5(pid=18719): stdout:
| (launched: 2022-12-08_21:58:55.239048)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:58:57.216126 run osmo-stp_10.42.42.5(pid=18719): stderr:
| �[0;m�[38;5;43m20221208215855433 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215855434 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215855434 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215855434 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215855435 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215855435 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215855435 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:58:57.371837 run patchelf(pid=18722): Terminating (SIGINT)
21:58:57.554458 run patchelf(pid=18722): Terminated: ok {rc=0}
21:58:57.587283 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=18719): Process ended prematurely: osmo-stp_10.42.42.5(pid=18719) [trial-2350↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18719)] [trial-2350↪gprs:oc2g↪ping.py:22]
21:58:57.596488 tst ping.py:22: Test FAILED (6.7 sec)
21:58:57.661129 run osmo-hlr_10.42.42.2(pid=18716): ERR: Terminated: ERROR {rc=237} [trial-2350↪gprs:oc2g↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18716)]
21:58:57.714077 run osmo-hlr_10.42.42.2(pid=18716): stdout:
| (launched: 2022-12-08_21:58:54.228427)
21:58:57.767459 run osmo-hlr_10.42.42.2(pid=18716): stderr:
| �[0;m�[1;31m20221208215854426 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221208215854426 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221208215854426 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221208215854426 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221208215854426 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221208215854426 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221208215854438 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2350/run.2022-12-08_21-44-24/gprs:oc2g/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221208215854446 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221208215854446 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221208215854446 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:58:57.812830 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18715): Terminating (SIGTERM)
21:58:57.855724 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18718): Terminating (SIGTERM)
21:58:57.898590 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18721): Terminating (SIGTERM)
21:58:57.908576 --- ParallelTerminationStrategy: PID 18715 died...
21:58:57.965906 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18715): Terminated: ok {rc=0}
21:58:57.976319 --- ParallelTerminationStrategy: PID 18718 died...
21:58:58.033613 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18718): Terminated: ok {rc=0}
21:58:58.044011 --- ParallelTerminationStrategy: PID 18721 died...
21:58:58.100890 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18721): Terminated: ok {rc=0}
21:58:58.189568 bus /gobi_6: Setting Powered False
212gprs:oc2g7.0osmo-stp_10.42.42.5(pid=18729): Process ended prematurely: osmo-stp_10.42.42.5(pid=18729) [trial-2350↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18729)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18729): Process ended prematurely: osmo-stp_10.42.42.5(pid=18729) [trial-2350↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18729)]
----------------------------------------------
trial-2350 gprs:oc2g ping_idle_ping.py
----------------------------------------------
21:58:59.241980 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:59.253285 tst gprs:oc2g: Using 1 x bts (candidates: 1)
21:58:59.381805 tst ping_idle_ping.py: using LAC 17020
21:58:59.501564 tst ping_idle_ping.py: using RAC 190
21:58:59.625905 tst ping_idle_ping.py: using CellId 17020
21:58:59.745924 tst ping_idle_ping.py: using BVCI 17021
21:58:59.769525 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:59.780077 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:59.790755 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:59.801685 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:59.812514 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:59.823460 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:59.834659 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
21:58:59.845660 tst gprs:oc2g: Using 1 x modem (candidates: 4)
21:59:00.120551 tst ping_idle_ping.py:24: start network...
21:59:00.246248 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:59:00.724163 run create_hlr_db(pid=18723): Launched
21:59:00.914140 bus /gobi_6: Setting Powered False
21:59:01.934900 run create_hlr_db(pid=18723): Terminated: ok {rc=0}
21:59:02.382966 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18725): Launched
21:59:02.707865 run osmo-hlr_10.42.42.2(pid=18726): Launched
21:59:02.826736 run osmo-stp_10.42.42.5: Starting osmo-stp
21:59:03.391114 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18728): Launched
21:59:03.715725 run osmo-stp_10.42.42.5(pid=18729): Launched
21:59:03.833980 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:59:04.400853 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18731): Launched
21:59:04.516934 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:59:04.800284 run patchelf(pid=18732): Launched
21:59:04.990810 run osmo-stp_10.42.42.5(pid=18729): ERR: Terminated: ERROR {rc=1} [trial-2350↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18729)]
21:59:05.130481 run osmo-stp_10.42.42.5(pid=18729): stdout:
| (launched: 2022-12-08_21:59:03.556823)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:59:05.270347 run osmo-stp_10.42.42.5(pid=18729): stderr:
| �[0;m�[38;5;43m20221208215903751 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215903751 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215903751 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215903752 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215903752 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215903752 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215903752 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:59:05.405515 run osmo-stp_10.42.42.5(pid=18729): stdout:
| (launched: 2022-12-08_21:59:03.556823)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:59:05.539373 run osmo-stp_10.42.42.5(pid=18729): stderr:
| �[0;m�[38;5;43m20221208215903751 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221208215903751 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221208215903751 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221208215903752 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221208215903752 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221208215903752 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221208215903752 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:59:05.694908 run patchelf(pid=18732): Terminating (SIGINT)
21:59:05.877679 run patchelf(pid=18732): Terminated: ok {rc=0}
21:59:05.910748 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=18729): Process ended prematurely: osmo-stp_10.42.42.5(pid=18729) [trial-2350↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18729)] [trial-2350↪gprs:oc2g↪ping_idle_ping.py:27]
21:59:05.920024 tst ping_idle_ping.py:27: Test FAILED (6.7 sec)
21:59:05.983695 run osmo-hlr_10.42.42.2(pid=18726): ERR: Terminated: ERROR {rc=237} [trial-2350↪gprs:oc2g↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18726)]
21:59:06.036063 run osmo-hlr_10.42.42.2(pid=18726): stdout:
| (launched: 2022-12-08_21:59:02.548670)
21:59:06.088942 run osmo-hlr_10.42.42.2(pid=18726): stderr:
| �[0;m�[1;31m20221208215902748 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221208215902748 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221208215902748 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221208215902748 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221208215902748 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221208215902748 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221208215902765 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2350/run.2022-12-08_21-44-24/gprs:oc2g/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221208215902773 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221208215902774 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221208215902774 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:59:06.134248 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18725): Terminating (SIGTERM)
21:59:06.177039 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18728): Terminating (SIGTERM)
21:59:06.219998 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18731): Terminating (SIGTERM)
21:59:06.230331 --- ParallelTerminationStrategy: PID 18725 died...
21:59:06.288030 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18725): Terminated: ok {rc=0}
21:59:06.298620 --- ParallelTerminationStrategy: PID 18728 died...
21:59:06.356117 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18728): Terminated: ok {rc=0}
21:59:06.366784 --- ParallelTerminationStrategy: PID 18731 died...
21:59:06.424345 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18731): Terminated: ok {rc=0}
21:59:06.512497 bus /gobi_6: Setting Powered False
5gprs:oc2g00