231gprs:oc2g9.0osmo-stp_10.42.42.5(pid=26619): Process ended prematurely: osmo-stp_10.42.42.5(pid=26619) [trial-2373↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26619)]2143cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=26619): Process ended prematurely: osmo-stp_10.42.42.5(pid=26619) [trial-2373↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26619)]
----------------------------------------------
trial-2373 gprs:oc2g cs_paging_gprs_active.py
----------------------------------------------
10:01:52.137827 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:01:52.148673 tst gprs:oc2g: Using 1 x bts (candidates: 1)
10:01:52.277710 tst cs_paging_gprs_active.py: using LAC 27619
10:01:52.397696 tst cs_paging_gprs_active.py: using RAC 79
10:01:52.516007 tst cs_paging_gprs_active.py: using CellId 27619
10:01:52.633949 tst cs_paging_gprs_active.py: using BVCI 27620
10:01:52.657464 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:01:52.667878 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:01:52.678253 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:01:52.688426 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:01:52.698628 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:01:52.708700 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:01:52.718981 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:01:52.729131 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:01:52.739400 tst gprs:oc2g: Using 1 x modem (candidates: 4)
10:01:52.855191 tst gprs:oc2g: Using 1 x modem (candidates: 4)
10:01:53.132469 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
10:01:53.259084 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:01:53.574740 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26609): Launched
10:01:53.836453 run iperf3-srv_10.42.42.10(pid=26610): Launched
10:01:54.022239 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
10:01:54.149105 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:01:54.464885 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26611): Launched
10:01:54.726945 run iperf3-srv_10.42.42.10(pid=26612): Launched
10:01:54.912922 tst cs_paging_gprs_active.py:38: start network...
10:01:55.039442 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:01:55.526892 run create_hlr_db(pid=26613): Launched
10:01:55.702369 bus /sierra_2: Setting Powered False
10:01:56.723437 run create_hlr_db(pid=26613): Terminated: ok {rc=0}
10:01:57.157233 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26615): Launched
10:01:57.482518 run osmo-hlr_10.42.42.2(pid=26616): Launched
10:01:57.601978 run osmo-stp_10.42.42.5: Starting osmo-stp
10:01:58.165459 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26618): Launched
10:01:58.490242 run osmo-stp_10.42.42.5(pid=26619): Launched
10:01:58.609518 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:01:59.177334 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26621): Launched
10:01:59.294148 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:01:59.579489 run patchelf(pid=26622): Launched
10:01:59.770864 run osmo-stp_10.42.42.5(pid=26619): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26619)]
10:01:59.911244 run osmo-stp_10.42.42.5(pid=26619): stdout:
| (launched: 2022-12-10_10:01:58.330883)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:00.051596 run osmo-stp_10.42.42.5(pid=26619): stderr:
| �[0;m�[38;5;43m20221210100158525 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100158525 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100158525 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100158526 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100158526 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100158526 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100158526 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:00.187084 run osmo-stp_10.42.42.5(pid=26619): stdout:
| (launched: 2022-12-10_10:01:58.330883)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:00.321324 run osmo-stp_10.42.42.5(pid=26619): stderr:
| �[0;m�[38;5;43m20221210100158525 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100158525 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100158525 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100158526 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100158526 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100158526 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100158526 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:00.478822 run patchelf(pid=26622): Terminating (SIGINT)
10:02:00.663849 run patchelf(pid=26622): Terminated: ok {rc=0}
10:02:00.697490 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=26619): Process ended prematurely: osmo-stp_10.42.42.5(pid=26619) [trial-2373↪gprs:oc2g↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26619)] [trial-2373↪gprs:oc2g↪cs_paging_gprs_active.py:38]
10:02:00.707127 tst cs_paging_gprs_active.py:38: Test FAILED (8.6 sec)
10:02:00.771044 run osmo-hlr_10.42.42.2(pid=26616): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:oc2g↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=26616)]
10:02:00.822494 run osmo-hlr_10.42.42.2(pid=26616): stdout:
| (launched: 2022-12-10_10:01:57.323196)
10:02:00.883131 run osmo-hlr_10.42.42.2(pid=26616): stderr:
| �[0;m�[1;31m20221210100157523 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210100157523 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210100157523 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210100157523 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210100157523 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210100157523 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210100157535 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:oc2g/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210100157545 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210100157546 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210100157546 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:02:00.945671 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26609): Terminating (SIGTERM)
10:02:00.989913 run iperf3-srv_10.42.42.10(pid=26610): Terminating (SIGTERM)
10:02:01.033921 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26611): Terminating (SIGTERM)
10:02:01.077951 run iperf3-srv_10.42.42.10(pid=26612): Terminating (SIGTERM)
10:02:01.120695 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26615): Terminating (SIGTERM)
10:02:01.165810 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26618): Terminating (SIGTERM)
10:02:01.210879 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26621): Terminating (SIGTERM)
10:02:01.221510 --- ParallelTerminationStrategy: PID 26609 died...
10:02:01.281331 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26609): Terminated: ok {rc=0}
10:02:01.291962 --- ParallelTerminationStrategy: PID 26610 died...
10:02:01.347567 run iperf3-srv_10.42.42.10(pid=26610): Terminated {rc=256}
10:02:01.359381 --- ParallelTerminationStrategy: PID 26611 died...
10:02:01.416202 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26611): Terminated: ok {rc=0}
10:02:01.427939 --- ParallelTerminationStrategy: PID 26612 died...
10:02:01.485053 run iperf3-srv_10.42.42.10(pid=26612): Terminated {rc=256}
10:02:01.495593 --- ParallelTerminationStrategy: PID 26615 died...
10:02:01.551742 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26615): Terminated: ok {rc=0}
10:02:01.562767 --- ParallelTerminationStrategy: PID 26618 died...
10:02:01.618149 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26618): Terminated: ok {rc=0}
10:02:01.630658 --- ParallelTerminationStrategy: PID 26621 died...
10:02:01.687148 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26621): Terminated: ok {rc=0}
10:02:01.772613 bus /gobi_0: Setting Powered False
10:02:02.876206 bus /gobi_0: Setting Powered False
10:02:03.987063 bus /gobi_6: Setting Powered False
231gprs:oc2g8.0osmo-stp_10.42.42.5(pid=26631): Process ended prematurely: osmo-stp_10.42.42.5(pid=26631) [trial-2373↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26631)]2143iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=26631): Process ended prematurely: osmo-stp_10.42.42.5(pid=26631) [trial-2373↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26631)]
----------------------------------------------
trial-2373 gprs:oc2g iperf3.py
----------------------------------------------
10:02:05.096379 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:05.107406 tst gprs:oc2g: Using 1 x bts (candidates: 1)
10:02:05.229714 tst iperf3.py: using LAC 27620
10:02:05.360809 tst iperf3.py: using RAC 80
10:02:05.488271 tst iperf3.py: using CellId 27620
10:02:05.618888 tst iperf3.py: using BVCI 27621
10:02:05.643940 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:05.655077 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:05.666249 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:05.677378 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:05.688256 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:05.699336 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:05.710533 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:05.721553 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:05.732665 tst gprs:oc2g: Using 1 x modem (candidates: 4)
10:02:06.030359 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
10:02:06.166561 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:02:06.503219 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26623): Launched
10:02:06.782828 run iperf3-srv_10.42.42.10(pid=26624): Launched
10:02:06.982785 tst iperf3.py:8: start network...
10:02:07.119276 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:02:07.632877 run create_hlr_db(pid=26625): Launched
10:02:07.823477 bus /gobi_6: Setting Powered False
10:02:08.843056 run create_hlr_db(pid=26625): Terminated: ok {rc=0}
10:02:09.251478 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26627): Launched
10:02:09.556392 run osmo-hlr_10.42.42.2(pid=26628): Launched
10:02:09.673319 run osmo-stp_10.42.42.5: Starting osmo-stp
10:02:10.199490 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26630): Launched
10:02:10.502700 run osmo-stp_10.42.42.5(pid=26631): Launched
10:02:10.615030 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:02:11.147491 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26633): Launched
10:02:11.255926 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:02:11.520657 run patchelf(pid=26634): Launched
10:02:11.699702 run osmo-stp_10.42.42.5(pid=26631): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26631)]
10:02:11.829773 run osmo-stp_10.42.42.5(pid=26631): stdout:
| (launched: 2022-12-10_10:02:10.354243)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:11.960583 run osmo-stp_10.42.42.5(pid=26631): stderr:
| �[0;m�[38;5;43m20221210100210526 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100210526 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100210526 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100210527 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100210527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100210527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100210527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:12.086135 run osmo-stp_10.42.42.5(pid=26631): stdout:
| (launched: 2022-12-10_10:02:10.354243)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:12.210586 run osmo-stp_10.42.42.5(pid=26631): stderr:
| �[0;m�[38;5;43m20221210100210526 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100210526 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100210526 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100210527 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100210527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100210527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100210527 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:12.355927 run patchelf(pid=26634): Terminating (SIGINT)
10:02:12.527505 run patchelf(pid=26634): Terminated: ok {rc=0}
10:02:12.558760 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=26631): Process ended prematurely: osmo-stp_10.42.42.5(pid=26631) [trial-2373↪gprs:oc2g↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26631)] [trial-2373↪gprs:oc2g↪iperf3.py:8]
10:02:12.567833 tst iperf3.py:8: Test FAILED (7.5 sec)
10:02:12.627592 run osmo-hlr_10.42.42.2(pid=26628): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:oc2g↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=26628)]
10:02:12.676553 run osmo-hlr_10.42.42.2(pid=26628): stdout:
| (launched: 2022-12-10_10:02:09.406549)
10:02:12.726095 run osmo-hlr_10.42.42.2(pid=26628): stderr:
| �[0;m�[1;31m20221210100209583 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210100209583 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210100209583 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210100209583 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210100209583 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210100209583 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210100209595 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:oc2g/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210100209604 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210100209604 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210100209604 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:02:12.768079 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26623): Terminating (SIGTERM)
10:02:12.807711 run iperf3-srv_10.42.42.10(pid=26624): Terminating (SIGTERM)
10:02:12.847347 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26627): Terminating (SIGTERM)
10:02:12.886798 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26630): Terminating (SIGTERM)
10:02:12.926259 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26633): Terminating (SIGTERM)
10:02:12.935477 --- ParallelTerminationStrategy: PID 26623 died...
10:02:12.988741 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26623): Terminated: ok {rc=0}
10:02:12.998373 --- ParallelTerminationStrategy: PID 26624 died...
10:02:13.051261 run iperf3-srv_10.42.42.10(pid=26624): Terminated {rc=256}
10:02:13.060828 --- ParallelTerminationStrategy: PID 26627 died...
10:02:13.113970 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26627): Terminated: ok {rc=0}
10:02:13.123539 --- ParallelTerminationStrategy: PID 26630 died...
10:02:13.176543 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26630): Terminated: ok {rc=0}
10:02:13.186230 --- ParallelTerminationStrategy: PID 26633 died...
10:02:13.239774 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26633): Terminated: ok {rc=0}
10:02:13.322389 bus /gobi_6: Setting Powered False
235gprs:oc2g11.0osmo-stp_10.42.42.5(pid=26649): Process ended prematurely: osmo-stp_10.42.42.5(pid=26649) [trial-2373↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26649)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=26649): Process ended prematurely: osmo-stp_10.42.42.5(pid=26649) [trial-2373↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26649)]
----------------------------------------------
trial-2373 gprs:oc2g iperf3m4.py
----------------------------------------------
10:02:14.438225 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:14.450361 tst gprs:oc2g: Using 1 x bts (candidates: 1)
10:02:14.584120 tst iperf3m4.py: using LAC 27621
10:02:14.711690 tst iperf3m4.py: using RAC 81
10:02:14.839353 tst iperf3m4.py: using CellId 27621
10:02:14.966890 tst iperf3m4.py: using BVCI 27622
10:02:14.991887 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:15.002860 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:15.013691 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:15.024781 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:15.035754 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:15.046951 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:15.058167 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:15.069113 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:15.080368 tst gprs:oc2g: Using 1 x modem (candidates: 4)
10:02:15.200619 tst gprs:oc2g: Using 1 x modem (candidates: 4)
10:02:15.322015 tst gprs:oc2g: Using 1 x modem (candidates: 4)
10:02:15.438698 tst gprs:oc2g: Using 1 x modem (candidates: 4)
10:02:15.716356 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
10:02:15.858888 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:02:16.252513 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26635): Launched
10:02:16.558147 run iperf3-srv_10.42.42.10(pid=26636): Launched
10:02:16.766302 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
10:02:16.908505 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:02:17.265641 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26637): Launched
10:02:17.547076 run iperf3-srv_10.42.42.10(pid=26638): Launched
10:02:17.747783 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
10:02:17.884771 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:02:18.222419 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26639): Launched
10:02:18.503476 run iperf3-srv_10.42.42.10(pid=26640): Launched
10:02:18.703142 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
10:02:18.839284 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:02:19.176254 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26641): Launched
10:02:19.457659 run iperf3-srv_10.42.42.10(pid=26642): Launched
10:02:19.658476 tst iperf3m4.py:8: start network...
10:02:19.795130 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:02:20.309314 run create_hlr_db(pid=26643): Launched
10:02:20.500038 bus /gobi_6: Setting Powered False
10:02:21.519782 run create_hlr_db(pid=26643): Terminated: ok {rc=0}
10:02:21.930976 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26645): Launched
10:02:22.235632 run osmo-hlr_10.42.42.2(pid=26646): Launched
10:02:22.356802 run osmo-stp_10.42.42.5: Starting osmo-stp
10:02:22.894950 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26648): Launched
10:02:23.199029 run osmo-stp_10.42.42.5(pid=26649): Launched
10:02:23.309267 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:02:23.842330 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26651): Launched
10:02:23.950843 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:02:24.216115 run patchelf(pid=26652): Launched
10:02:24.394275 run osmo-stp_10.42.42.5(pid=26649): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26649)]
10:02:24.524897 run osmo-stp_10.42.42.5(pid=26649): stdout:
| (launched: 2022-12-10_10:02:23.049598)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:24.655760 run osmo-stp_10.42.42.5(pid=26649): stderr:
| �[0;m�[38;5;43m20221210100223222 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100223222 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100223223 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100223223 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100223223 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100223224 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100223224 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:24.781779 run osmo-stp_10.42.42.5(pid=26649): stdout:
| (launched: 2022-12-10_10:02:23.049598)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:24.906807 run osmo-stp_10.42.42.5(pid=26649): stderr:
| �[0;m�[38;5;43m20221210100223222 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100223222 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100223223 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100223223 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100223223 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100223224 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100223224 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:25.051902 run patchelf(pid=26652): Terminating (SIGINT)
10:02:25.222955 run patchelf(pid=26652): Terminated: ok {rc=0}
10:02:25.254660 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=26649): Process ended prematurely: osmo-stp_10.42.42.5(pid=26649) [trial-2373↪gprs:oc2g↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26649)] [trial-2373↪gprs:oc2g↪iperf3m4.py:8]
10:02:25.263682 tst iperf3m4.py:8: Test FAILED (10.9 sec)
10:02:25.323674 run osmo-hlr_10.42.42.2(pid=26646): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:oc2g↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=26646)]
10:02:25.372657 run osmo-hlr_10.42.42.2(pid=26646): stdout:
| (launched: 2022-12-10_10:02:22.085556)
10:02:25.422148 run osmo-hlr_10.42.42.2(pid=26646): stderr:
| �[0;m�[1;31m20221210100222265 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210100222265 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210100222265 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210100222265 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210100222265 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210100222265 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210100222278 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:oc2g/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210100222286 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210100222287 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210100222287 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:02:25.464363 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26635): Terminating (SIGTERM)
10:02:25.504041 run iperf3-srv_10.42.42.10(pid=26636): Terminating (SIGTERM)
10:02:25.543793 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26637): Terminating (SIGTERM)
10:02:25.583415 run iperf3-srv_10.42.42.10(pid=26638): Terminating (SIGTERM)
10:02:25.623166 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26639): Terminating (SIGTERM)
10:02:25.663292 run iperf3-srv_10.42.42.10(pid=26640): Terminating (SIGTERM)
10:02:25.702928 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26641): Terminating (SIGTERM)
10:02:25.742362 run iperf3-srv_10.42.42.10(pid=26642): Terminating (SIGTERM)
10:02:25.781993 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26645): Terminating (SIGTERM)
10:02:25.821554 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26648): Terminating (SIGTERM)
10:02:25.861228 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26651): Terminating (SIGTERM)
10:02:25.870138 --- ParallelTerminationStrategy: PID 26635 died...
10:02:25.923352 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26635): Terminated: ok {rc=0}
10:02:25.933082 --- ParallelTerminationStrategy: PID 26636 died...
10:02:25.985908 run iperf3-srv_10.42.42.10(pid=26636): Terminated {rc=256}
10:02:25.995531 --- ParallelTerminationStrategy: PID 26637 died...
10:02:26.048796 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26637): Terminated: ok {rc=0}
10:02:26.058429 --- ParallelTerminationStrategy: PID 26638 died...
10:02:26.110949 run iperf3-srv_10.42.42.10(pid=26638): Terminated {rc=256}
10:02:26.120447 --- ParallelTerminationStrategy: PID 26639 died...
10:02:26.173262 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26639): Terminated: ok {rc=0}
10:02:26.182833 --- ParallelTerminationStrategy: PID 26640 died...
10:02:26.236062 run iperf3-srv_10.42.42.10(pid=26640): Terminated {rc=256}
10:02:26.245790 --- ParallelTerminationStrategy: PID 26641 died...
10:02:26.298627 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26641): Terminated: ok {rc=0}
10:02:26.308221 --- ParallelTerminationStrategy: PID 26642 died...
10:02:26.361175 run iperf3-srv_10.42.42.10(pid=26642): Terminated {rc=256}
10:02:26.370864 --- ParallelTerminationStrategy: PID 26645 died...
10:02:26.424352 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26645): Terminated: ok {rc=0}
10:02:26.434063 --- ParallelTerminationStrategy: PID 26648 died...
10:02:26.486923 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26648): Terminated: ok {rc=0}
10:02:26.496608 --- ParallelTerminationStrategy: PID 26651 died...
10:02:26.549512 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26651): Terminated: ok {rc=0}
10:02:26.638278 bus /gobi_4: Setting Powered False
10:02:27.742200 bus /gobi_4: Setting Powered False
10:02:28.853685 bus /gobi_1: Setting Powered False
10:02:29.956401 bus /gobi_1: Setting Powered False
10:02:31.068684 bus /gobi_0: Setting Powered False
10:02:32.185544 bus /gobi_0: Setting Powered False
10:02:33.297086 bus /gobi_6: Setting Powered False
235gprs:oc2g7.0osmo-stp_10.42.42.5(pid=26660): Process ended prematurely: osmo-stp_10.42.42.5(pid=26660) [trial-2373↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26660)]2139ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=26660): Process ended prematurely: osmo-stp_10.42.42.5(pid=26660) [trial-2373↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26660)]
----------------------------------------------
trial-2373 gprs:oc2g ping.py
----------------------------------------------
10:02:34.363252 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:34.374217 tst gprs:oc2g: Using 1 x bts (candidates: 1)
10:02:34.497843 tst ping.py: using LAC 27622
10:02:34.620133 tst ping.py: using RAC 82
10:02:34.746046 tst ping.py: using CellId 27622
10:02:34.861887 tst ping.py: using BVCI 27623
10:02:34.884567 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:34.895202 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:34.905654 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:34.915980 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:34.926287 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:34.936599 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:34.947001 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:34.957230 tst gprs:oc2g: Using 1 x modem (candidates: 4)
10:02:35.230138 tst ping.py:19: start network...
10:02:35.354700 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:02:35.834655 run create_hlr_db(pid=26653): Launched
10:02:36.015233 bus /gobi_6: Setting Powered False
10:02:37.040459 run create_hlr_db(pid=26653): Terminated: ok {rc=0}
10:02:37.467721 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26656): Launched
10:02:37.788597 run osmo-hlr_10.42.42.2(pid=26657): Launched
10:02:37.908652 run osmo-stp_10.42.42.5: Starting osmo-stp
10:02:38.467972 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26659): Launched
10:02:38.788680 run osmo-stp_10.42.42.5(pid=26660): Launched
10:02:38.906359 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:02:39.468413 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26662): Launched
10:02:39.583584 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:02:39.862093 run patchelf(pid=26663): Launched
10:02:40.049785 run osmo-stp_10.42.42.5(pid=26660): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26660)]
10:02:40.186856 run osmo-stp_10.42.42.5(pid=26660): stdout:
| (launched: 2022-12-10_10:02:38.631336)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:40.324488 run osmo-stp_10.42.42.5(pid=26660): stderr:
| �[0;m�[38;5;43m20221210100238822 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100238822 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100238823 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100238823 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100238824 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100238824 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100238824 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:40.457142 run osmo-stp_10.42.42.5(pid=26660): stdout:
| (launched: 2022-12-10_10:02:38.631336)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:40.589068 run osmo-stp_10.42.42.5(pid=26660): stderr:
| �[0;m�[38;5;43m20221210100238822 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100238822 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100238823 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100238823 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100238824 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100238824 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100238824 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:40.742710 run patchelf(pid=26663): Terminating (SIGINT)
10:02:40.922527 run patchelf(pid=26663): Terminated: ok {rc=0}
10:02:40.955085 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=26660): Process ended prematurely: osmo-stp_10.42.42.5(pid=26660) [trial-2373↪gprs:oc2g↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26660)] [trial-2373↪gprs:oc2g↪ping.py:22]
10:02:40.964194 tst ping.py:22: Test FAILED (6.6 sec)
10:02:41.028181 run osmo-hlr_10.42.42.2(pid=26657): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:oc2g↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=26657)]
10:02:41.079944 run osmo-hlr_10.42.42.2(pid=26657): stdout:
| (launched: 2022-12-10_10:02:37.631401)
10:02:41.132677 run osmo-hlr_10.42.42.2(pid=26657): stderr:
| �[0;m�[1;31m20221210100237830 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210100237830 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210100237830 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210100237830 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210100237830 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210100237830 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210100237842 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:oc2g/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210100237851 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210100237851 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210100237851 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:02:41.177272 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26656): Terminating (SIGTERM)
10:02:41.219601 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26659): Terminating (SIGTERM)
10:02:41.261952 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26662): Terminating (SIGTERM)
10:02:41.271727 --- ParallelTerminationStrategy: PID 26656 died...
10:02:41.328317 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26656): Terminated: ok {rc=0}
10:02:41.338570 --- ParallelTerminationStrategy: PID 26659 died...
10:02:41.394918 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26659): Terminated: ok {rc=0}
10:02:41.405296 --- ParallelTerminationStrategy: PID 26662 died...
10:02:41.461781 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26662): Terminated: ok {rc=0}
10:02:41.549364 bus /gobi_6: Setting Powered False
235gprs:oc2g7.0osmo-stp_10.42.42.5(pid=26670): Process ended prematurely: osmo-stp_10.42.42.5(pid=26670) [trial-2373↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26670)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=26670): Process ended prematurely: osmo-stp_10.42.42.5(pid=26670) [trial-2373↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26670)]
----------------------------------------------
trial-2373 gprs:oc2g ping_idle_ping.py
----------------------------------------------
10:02:42.601928 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:42.613661 tst gprs:oc2g: Using 1 x bts (candidates: 1)
10:02:42.737964 tst ping_idle_ping.py: using LAC 27623
10:02:42.856247 tst ping_idle_ping.py: using RAC 83
10:02:42.975382 tst ping_idle_ping.py: using CellId 27623
10:02:43.095057 tst ping_idle_ping.py: using BVCI 27624
10:02:43.118140 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:43.128571 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:43.138989 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:43.149386 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:43.159625 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:43.169957 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:43.180385 tst gprs:oc2g: Using 1 x ip_address (candidates: 9)
10:02:43.190951 tst gprs:oc2g: Using 1 x modem (candidates: 4)
10:02:43.463919 tst ping_idle_ping.py:24: start network...
10:02:43.588429 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:02:44.070240 run create_hlr_db(pid=26664): Launched
10:02:44.243655 bus /gobi_6: Setting Powered False
10:02:45.265372 run create_hlr_db(pid=26664): Terminated: ok {rc=0}
10:02:45.693674 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26666): Launched
10:02:46.147920 run osmo-hlr_10.42.42.2(pid=26667): Launched
10:02:46.350035 run osmo-stp_10.42.42.5: Starting osmo-stp
10:02:46.986814 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26669): Launched
10:02:47.376660 run osmo-stp_10.42.42.5(pid=26670): Launched
10:02:47.545766 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:02:48.106713 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26672): Launched
10:02:48.221856 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:02:48.501309 run patchelf(pid=26673): Launched
10:02:48.688537 run osmo-stp_10.42.42.5(pid=26670): ERR: Terminated: ERROR {rc=1} [trial-2373↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26670)]
10:02:48.825808 run osmo-stp_10.42.42.5(pid=26670): stdout:
| (launched: 2022-12-10_10:02:47.153506)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:48.963286 run osmo-stp_10.42.42.5(pid=26670): stderr:
| �[0;m�[38;5;43m20221210100247424 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100247424 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100247425 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100247425 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100247425 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100247425 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100247426 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:49.095815 run osmo-stp_10.42.42.5(pid=26670): stdout:
| (launched: 2022-12-10_10:02:47.153506)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:02:49.227177 run osmo-stp_10.42.42.5(pid=26670): stderr:
| �[0;m�[38;5;43m20221210100247424 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221210100247424 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221210100247425 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221210100247425 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221210100247425 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221210100247425 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221210100247426 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:02:49.380631 run patchelf(pid=26673): Terminating (SIGINT)
10:02:49.561460 run patchelf(pid=26673): Terminated: ok {rc=0}
10:02:49.593996 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=26670): Process ended prematurely: osmo-stp_10.42.42.5(pid=26670) [trial-2373↪gprs:oc2g↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26670)] [trial-2373↪gprs:oc2g↪ping_idle_ping.py:27]
10:02:49.603300 tst ping_idle_ping.py:27: Test FAILED (7.0 sec)
10:02:49.666409 run osmo-hlr_10.42.42.2(pid=26667): ERR: Terminated: ERROR {rc=237} [trial-2373↪gprs:oc2g↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=26667)]
10:02:49.717249 run osmo-hlr_10.42.42.2(pid=26667): stdout:
| (launched: 2022-12-10_10:02:45.868301)
10:02:49.768919 run osmo-hlr_10.42.42.2(pid=26667): stderr:
| �[0;m�[1;31m20221210100246136 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221210100246136 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221210100246136 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221210100246136 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221210100246136 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221210100246136 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221210100246158 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2373/run.2022-12-10_09-48-07/gprs:oc2g/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221210100246178 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221210100246178 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221210100246178 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:02:49.813117 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26666): Terminating (SIGTERM)
10:02:49.854845 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26669): Terminating (SIGTERM)
10:02:49.896868 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26672): Terminating (SIGTERM)
10:02:49.906758 --- ParallelTerminationStrategy: PID 26666 died...
10:02:49.962894 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26666): Terminated: ok {rc=0}
10:02:49.973176 --- ParallelTerminationStrategy: PID 26669 died...
10:02:50.028872 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26669): Terminated: ok {rc=0}
10:02:50.039482 --- ParallelTerminationStrategy: PID 26672 died...
10:02:50.095597 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26672): Terminated: ok {rc=0}
10:02:50.182479 bus /gobi_6: Setting Powered False
5gprs:oc2g00