40gprs:trx-umtrx9.0osmo-stp_10.42.42.5(pid=18024): Process ended prematurely: osmo-stp_10.42.42.5(pid=18024) [trial-2357↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18024)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18024): Process ended prematurely: osmo-stp_10.42.42.5(pid=18024) [trial-2357↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18024)]
----------------------------------------------
trial-2357 gprs:trx-umtrx cs_paging_gprs_active.py
----------------------------------------------
09:02:32.855627 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:32.867007 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:02:33.007600 tst cs_paging_gprs_active.py: using LAC 20380
09:02:33.126725 tst cs_paging_gprs_active.py: using RAC 235
09:02:33.246328 tst cs_paging_gprs_active.py: using CellId 20380
09:02:33.366120 tst cs_paging_gprs_active.py: using BVCI 20381
09:02:33.390020 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:33.401021 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:33.411618 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:33.422685 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:33.433375 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:33.443990 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:33.454762 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:33.465290 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:33.475983 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:02:33.597493 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:02:33.900019 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
09:02:34.039648 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:02:34.381305 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18011): Launched
09:02:34.666527 run iperf3-srv_10.42.42.10(pid=18012): Launched
09:02:34.870781 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
09:02:35.009889 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:02:35.351175 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18013): Launched
09:02:35.636529 run iperf3-srv_10.42.42.10(pid=18014): Launched
09:02:35.842428 tst cs_paging_gprs_active.py:38: start network...
09:02:35.980879 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:02:36.504824 run create_hlr_db(pid=18018): Launched
09:02:36.697110 bus /sierra_2: Setting Powered False
09:02:37.718538 run create_hlr_db(pid=18018): Terminated: ok {rc=0}
09:02:38.159196 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18020): Launched
09:02:38.491603 run osmo-hlr_10.42.42.2(pid=18021): Launched
09:02:38.617290 run osmo-stp_10.42.42.5: Starting osmo-stp
09:02:39.190950 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18023): Launched
09:02:39.522924 run osmo-stp_10.42.42.5(pid=18024): Launched
09:02:39.643993 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:02:40.222641 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18026): Launched
09:02:40.340889 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:02:40.630238 run patchelf(pid=18027): Launched
09:02:40.823235 run osmo-stp_10.42.42.5(pid=18024): ERR: Terminated: ERROR {rc=1} [trial-2357↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18024)]
09:02:40.964034 run osmo-stp_10.42.42.5(pid=18024): stdout:
| (launched: 2022-12-09_09:02:39.359417)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:02:41.105252 run osmo-stp_10.42.42.5(pid=18024): stderr:
| �[0;m�[38;5;43m20221209090239550 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090239551 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090239551 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090239551 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090239552 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090239552 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090239552 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:02:41.241954 run osmo-stp_10.42.42.5(pid=18024): stdout:
| (launched: 2022-12-09_09:02:39.359417)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:02:41.378128 run osmo-stp_10.42.42.5(pid=18024): stderr:
| �[0;m�[38;5;43m20221209090239550 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090239551 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090239551 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090239551 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090239552 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090239552 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090239552 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:02:41.536124 run patchelf(pid=18027): Terminating (SIGINT)
09:02:41.721279 run patchelf(pid=18027): Terminated: ok {rc=0}
09:02:41.755101 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=18024): Process ended prematurely: osmo-stp_10.42.42.5(pid=18024) [trial-2357↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18024)] [trial-2357↪gprs:trx-umtrx↪cs_paging_gprs_active.py:38]
09:02:41.764860 tst cs_paging_gprs_active.py:38: Test FAILED (8.9 sec)
09:02:41.828928 run osmo-hlr_10.42.42.2(pid=18021): ERR: Terminated: ERROR {rc=237} [trial-2357↪gprs:trx-umtrx↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18021)]
09:02:41.880845 run osmo-hlr_10.42.42.2(pid=18021): stdout:
| (launched: 2022-12-09_09:02:38.327986)
09:02:41.933334 run osmo-hlr_10.42.42.2(pid=18021): stderr:
| �[0;m�[1;31m20221209090238521 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209090238521 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209090238521 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209090238522 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209090238522 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209090238522 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209090238534 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2357/run.2022-12-09_08-22-41/gprs:trx-umtrx/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209090238543 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209090238543 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209090238543 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:02:41.977927 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18011): Terminating (SIGTERM)
09:02:42.020202 run iperf3-srv_10.42.42.10(pid=18012): Terminating (SIGTERM)
09:02:42.062239 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18013): Terminating (SIGTERM)
09:02:42.104014 run iperf3-srv_10.42.42.10(pid=18014): Terminating (SIGTERM)
09:02:42.146376 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18020): Terminating (SIGTERM)
09:02:42.188351 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18023): Terminating (SIGTERM)
09:02:42.230954 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18026): Terminating (SIGTERM)
09:02:42.240736 --- ParallelTerminationStrategy: PID 18011 died...
09:02:42.297250 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18011): Terminated: ok {rc=0}
09:02:42.307305 --- ParallelTerminationStrategy: PID 18012 died...
09:02:42.363272 run iperf3-srv_10.42.42.10(pid=18012): Terminated {rc=256}
09:02:42.373411 --- ParallelTerminationStrategy: PID 18013 died...
09:02:42.429380 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18013): Terminated: ok {rc=0}
09:02:42.439372 --- ParallelTerminationStrategy: PID 18014 died...
09:02:42.495292 run iperf3-srv_10.42.42.10(pid=18014): Terminated {rc=256}
09:02:42.505390 --- ParallelTerminationStrategy: PID 18020 died...
09:02:42.561508 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18020): Terminated: ok {rc=0}
09:02:42.571844 --- ParallelTerminationStrategy: PID 18023 died...
09:02:42.628084 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18023): Terminated: ok {rc=0}
09:02:42.638227 --- ParallelTerminationStrategy: PID 18026 died...
09:02:42.694261 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18026): Terminated: ok {rc=0}
09:02:42.784100 bus /gobi_0: Setting Powered False
09:02:43.889415 bus /gobi_0: Setting Powered False
09:02:45.000874 bus /gobi_6: Setting Powered False
40gprs:trx-umtrx8.0osmo-stp_10.42.42.5(pid=18036): Process ended prematurely: osmo-stp_10.42.42.5(pid=18036) [trial-2357↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18036)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18036): Process ended prematurely: osmo-stp_10.42.42.5(pid=18036) [trial-2357↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18036)]
----------------------------------------------
trial-2357 gprs:trx-umtrx iperf3.py
----------------------------------------------
09:02:46.130694 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:46.143933 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:02:46.311767 tst iperf3.py: using LAC 20381
09:02:46.444289 tst iperf3.py: using RAC 236
09:02:46.600304 tst iperf3.py: using CellId 20381
09:02:46.754872 tst iperf3.py: using BVCI 20382
09:02:46.782407 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:46.795164 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:46.809074 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:46.822267 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:46.835767 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:46.849364 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:46.862025 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:46.875521 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:46.890288 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:02:47.197922 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
09:02:47.338754 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:02:47.682321 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18028): Launched
09:02:47.969128 run iperf3-srv_10.42.42.10(pid=18029): Launched
09:02:48.173585 tst iperf3.py:8: start network...
09:02:48.312873 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:02:48.837152 run create_hlr_db(pid=18030): Launched
09:02:49.015298 bus /gobi_6: Setting Powered False
09:02:50.036339 run create_hlr_db(pid=18030): Terminated: ok {rc=0}
09:02:50.475957 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18032): Launched
09:02:50.805614 run osmo-hlr_10.42.42.2(pid=18033): Launched
09:02:50.920596 run osmo-stp_10.42.42.5: Starting osmo-stp
09:02:51.460937 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18035): Launched
09:02:51.769833 run osmo-stp_10.42.42.5(pid=18036): Launched
09:02:51.882011 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:02:52.420050 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18038): Launched
09:02:52.530209 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:02:52.812389 run patchelf(pid=18039): Launched
09:02:53.005856 run osmo-stp_10.42.42.5(pid=18036): ERR: Terminated: ERROR {rc=1} [trial-2357↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18036)]
09:02:53.147281 run osmo-stp_10.42.42.5(pid=18036): stdout:
| (launched: 2022-12-09_09:02:51.618791)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:02:53.288854 run osmo-stp_10.42.42.5(pid=18036): stderr:
| �[0;m�[38;5;43m20221209090251788 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090251788 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090251788 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090251789 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090251789 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090251789 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090251789 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:02:53.426245 run osmo-stp_10.42.42.5(pid=18036): stdout:
| (launched: 2022-12-09_09:02:51.618791)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:02:53.562217 run osmo-stp_10.42.42.5(pid=18036): stderr:
| �[0;m�[38;5;43m20221209090251788 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090251788 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090251788 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090251789 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090251789 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090251789 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090251789 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:02:53.720375 run patchelf(pid=18039): Terminating (SIGINT)
09:02:53.905832 run patchelf(pid=18039): Terminated: ok {rc=0}
09:02:53.939839 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=18036): Process ended prematurely: osmo-stp_10.42.42.5(pid=18036) [trial-2357↪gprs:trx-umtrx↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18036)] [trial-2357↪gprs:trx-umtrx↪iperf3.py:8]
09:02:53.949955 tst iperf3.py:8: Test FAILED (7.9 sec)
09:02:54.014050 run osmo-hlr_10.42.42.2(pid=18033): ERR: Terminated: ERROR {rc=237} [trial-2357↪gprs:trx-umtrx↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18033)]
09:02:54.065909 run osmo-hlr_10.42.42.2(pid=18033): stdout:
| (launched: 2022-12-09_09:02:50.644859)
09:02:54.118457 run osmo-hlr_10.42.42.2(pid=18033): stderr:
| �[0;m�[1;31m20221209090250821 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209090250821 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209090250821 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209090250821 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209090250821 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209090250822 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209090250835 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2357/run.2022-12-09_08-22-41/gprs:trx-umtrx/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209090250844 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209090250844 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209090250844 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:02:54.162819 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18028): Terminating (SIGTERM)
09:02:54.205531 run iperf3-srv_10.42.42.10(pid=18029): Terminating (SIGTERM)
09:02:54.248197 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18032): Terminating (SIGTERM)
09:02:54.290577 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18035): Terminating (SIGTERM)
09:02:54.333182 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18038): Terminating (SIGTERM)
09:02:54.343113 --- ParallelTerminationStrategy: PID 18028 died...
09:02:54.400063 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18028): Terminated: ok {rc=0}
09:02:54.410389 --- ParallelTerminationStrategy: PID 18029 died...
09:02:54.466979 run iperf3-srv_10.42.42.10(pid=18029): Terminated {rc=256}
09:02:54.477198 --- ParallelTerminationStrategy: PID 18032 died...
09:02:54.533706 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18032): Terminated: ok {rc=0}
09:02:54.543959 --- ParallelTerminationStrategy: PID 18035 died...
09:02:54.600244 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18035): Terminated: ok {rc=0}
09:02:54.610618 --- ParallelTerminationStrategy: PID 18038 died...
09:02:54.666933 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18038): Terminated: ok {rc=0}
09:02:54.754264 bus /gobi_6: Setting Powered False
183gprs:trx-umtrx11.0osmo-stp_10.42.42.5(pid=18055): Process ended prematurely: osmo-stp_10.42.42.5(pid=18055) [trial-2357↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18055)]2175iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18055): Process ended prematurely: osmo-stp_10.42.42.5(pid=18055) [trial-2357↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18055)]
----------------------------------------------
trial-2357 gprs:trx-umtrx iperf3m4.py
----------------------------------------------
09:02:55.874854 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:55.886757 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:02:56.033425 tst iperf3m4.py: using LAC 20382
09:02:56.163866 tst iperf3m4.py: using RAC 237
09:02:56.293571 tst iperf3m4.py: using CellId 20382
09:02:56.428863 tst iperf3m4.py: using BVCI 20383
09:02:56.454352 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:56.465726 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:56.476723 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:56.487967 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:56.499232 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:56.510433 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:56.522012 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:56.533916 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:02:56.545471 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:02:56.668183 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:02:56.793171 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:02:56.916450 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:02:57.202865 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
09:02:57.332584 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:02:57.653099 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18040): Launched
09:02:57.918852 run iperf3-srv_10.42.42.10(pid=18041): Launched
09:02:58.107980 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
09:02:58.237184 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:02:58.556759 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18042): Launched
09:02:58.822157 run iperf3-srv_10.42.42.10(pid=18043): Launched
09:02:59.010716 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
09:02:59.139294 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:02:59.459745 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18044): Launched
09:02:59.725323 run iperf3-srv_10.42.42.10(pid=18045): Launched
09:02:59.914180 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
09:03:00.042762 run iperf3-srv_10.42.42.10: Starting iperf3-srv
09:03:00.362384 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18046): Launched
09:03:00.628105 run iperf3-srv_10.42.42.10(pid=18047): Launched
09:03:00.816304 tst iperf3m4.py:8: start network...
09:03:00.975867 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:03:01.547857 run create_hlr_db(pid=18049): Launched
09:03:01.729628 bus /gobi_6: Setting Powered False
09:03:02.751087 run create_hlr_db(pid=18049): Terminated: ok {rc=0}
09:03:03.164822 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18051): Launched
09:03:03.475275 run osmo-hlr_10.42.42.2(pid=18052): Launched
09:03:03.588637 run osmo-stp_10.42.42.5: Starting osmo-stp
09:03:04.128067 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18054): Launched
09:03:04.438736 run osmo-stp_10.42.42.5(pid=18055): Launched
09:03:04.551093 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:03:05.087864 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18057): Launched
09:03:05.197909 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:03:05.466320 run patchelf(pid=18058): Launched
09:03:05.646463 run osmo-stp_10.42.42.5(pid=18055): ERR: Terminated: ERROR {rc=1} [trial-2357↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18055)]
09:03:05.778826 run osmo-stp_10.42.42.5(pid=18055): stdout:
| (launched: 2022-12-09_09:03:04.286267)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:03:05.910819 run osmo-stp_10.42.42.5(pid=18055): stderr:
| �[0;m�[38;5;43m20221209090304453 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090304453 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090304453 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090304454 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090304454 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090304454 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090304454 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:03:06.039078 run osmo-stp_10.42.42.5(pid=18055): stdout:
| (launched: 2022-12-09_09:03:04.286267)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:03:06.165751 run osmo-stp_10.42.42.5(pid=18055): stderr:
| �[0;m�[38;5;43m20221209090304453 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090304453 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090304453 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090304454 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090304454 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090304454 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090304454 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:03:06.312760 run patchelf(pid=18058): Terminating (SIGINT)
09:03:06.485638 run patchelf(pid=18058): Terminated: ok {rc=0}
09:03:06.517411 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=18055): Process ended prematurely: osmo-stp_10.42.42.5(pid=18055) [trial-2357↪gprs:trx-umtrx↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18055)] [trial-2357↪gprs:trx-umtrx↪iperf3m4.py:8]
09:03:06.526762 tst iperf3m4.py:8: Test FAILED (10.7 sec)
09:03:06.586409 run osmo-hlr_10.42.42.2(pid=18052): ERR: Terminated: ERROR {rc=237} [trial-2357↪gprs:trx-umtrx↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18052)]
09:03:06.635067 run osmo-hlr_10.42.42.2(pid=18052): stdout:
| (launched: 2022-12-09_09:03:03.323682)
09:03:06.684320 run osmo-hlr_10.42.42.2(pid=18052): stderr:
| �[0;m�[1;31m20221209090303497 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209090303497 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209090303497 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209090303497 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209090303497 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209090303497 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209090303512 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2357/run.2022-12-09_08-22-41/gprs:trx-umtrx/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209090303523 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209090303523 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209090303523 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:03:06.726111 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18040): Terminating (SIGTERM)
09:03:06.765562 run iperf3-srv_10.42.42.10(pid=18041): Terminating (SIGTERM)
09:03:06.805169 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18042): Terminating (SIGTERM)
09:03:06.844680 run iperf3-srv_10.42.42.10(pid=18043): Terminating (SIGTERM)
09:03:06.884247 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18044): Terminating (SIGTERM)
09:03:06.923641 run iperf3-srv_10.42.42.10(pid=18045): Terminating (SIGTERM)
09:03:06.963203 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18046): Terminating (SIGTERM)
09:03:07.002453 run iperf3-srv_10.42.42.10(pid=18047): Terminating (SIGTERM)
09:03:07.042149 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18051): Terminating (SIGTERM)
09:03:07.081655 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18054): Terminating (SIGTERM)
09:03:07.121241 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18057): Terminating (SIGTERM)
09:03:07.130387 --- ParallelTerminationStrategy: PID 18040 died...
09:03:07.183888 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18040): Terminated: ok {rc=0}
09:03:07.193679 --- ParallelTerminationStrategy: PID 18041 died...
09:03:07.246782 run iperf3-srv_10.42.42.10(pid=18041): Terminated {rc=256}
09:03:07.256433 --- ParallelTerminationStrategy: PID 18042 died...
09:03:07.310028 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18042): Terminated: ok {rc=0}
09:03:07.319678 --- ParallelTerminationStrategy: PID 18043 died...
09:03:07.372833 run iperf3-srv_10.42.42.10(pid=18043): Terminated {rc=256}
09:03:07.382566 --- ParallelTerminationStrategy: PID 18044 died...
09:03:07.435921 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18044): Terminated: ok {rc=0}
09:03:07.445736 --- ParallelTerminationStrategy: PID 18045 died...
09:03:07.498609 run iperf3-srv_10.42.42.10(pid=18045): Terminated {rc=256}
09:03:07.508261 --- ParallelTerminationStrategy: PID 18046 died...
09:03:07.561963 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18046): Terminated: ok {rc=0}
09:03:07.571564 --- ParallelTerminationStrategy: PID 18047 died...
09:03:07.624886 run iperf3-srv_10.42.42.10(pid=18047): Terminated {rc=256}
09:03:07.634729 --- ParallelTerminationStrategy: PID 18051 died...
09:03:07.688093 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18051): Terminated: ok {rc=0}
09:03:07.698338 --- ParallelTerminationStrategy: PID 18054 died...
09:03:07.751961 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18054): Terminated: ok {rc=0}
09:03:07.761685 --- ParallelTerminationStrategy: PID 18057 died...
09:03:07.815182 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18057): Terminated: ok {rc=0}
09:03:07.905267 bus /gobi_4: Setting Powered False
09:03:09.008452 bus /gobi_4: Setting Powered False
09:03:10.120707 bus /gobi_1: Setting Powered False
09:03:11.227804 bus /gobi_1: Setting Powered False
09:03:12.335901 bus /gobi_0: Setting Powered False
09:03:13.442621 bus /gobi_0: Setting Powered False
09:03:14.551900 bus /gobi_6: Setting Powered False
158gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=18065): Process ended prematurely: osmo-stp_10.42.42.5(pid=18065) [trial-2357↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18065)]2200ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18065): Process ended prematurely: osmo-stp_10.42.42.5(pid=18065) [trial-2357↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18065)]
----------------------------------------------
trial-2357 gprs:trx-umtrx ping.py
----------------------------------------------
09:03:15.621361 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:15.633447 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:03:15.779475 tst ping.py: using LAC 20383
09:03:15.925973 tst ping.py: using RAC 238
09:03:16.062755 tst ping.py: using CellId 20383
09:03:16.221598 tst ping.py: using BVCI 20384
09:03:16.256210 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:16.271545 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:16.284321 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:16.302263 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:16.319865 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:16.337486 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:16.353864 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:16.371100 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:03:16.715608 tst ping.py:19: start network...
09:03:16.847795 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:03:17.352787 run create_hlr_db(pid=18059): Launched
09:03:17.542238 bus /gobi_6: Setting Powered False
09:03:18.562019 run create_hlr_db(pid=18059): Terminated: ok {rc=0}
09:03:18.992709 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18061): Launched
09:03:19.319278 run osmo-hlr_10.42.42.2(pid=18062): Launched
09:03:19.437338 run osmo-stp_10.42.42.5: Starting osmo-stp
09:03:20.003958 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18064): Launched
09:03:20.331128 run osmo-stp_10.42.42.5(pid=18065): Launched
09:03:20.454596 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:03:21.023688 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18067): Launched
09:03:21.140504 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:03:21.426556 run patchelf(pid=18068): Launched
09:03:21.617606 run osmo-stp_10.42.42.5(pid=18065): ERR: Terminated: ERROR {rc=1} [trial-2357↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18065)]
09:03:21.756773 run osmo-stp_10.42.42.5(pid=18065): stdout:
| (launched: 2022-12-09_09:03:20.169433)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:03:21.896499 run osmo-stp_10.42.42.5(pid=18065): stderr:
| �[0;m�[38;5;43m20221209090320358 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090320358 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090320358 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090320358 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090320359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090320359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090320359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:03:22.031530 run osmo-stp_10.42.42.5(pid=18065): stdout:
| (launched: 2022-12-09_09:03:20.169433)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:03:22.165375 run osmo-stp_10.42.42.5(pid=18065): stderr:
| �[0;m�[38;5;43m20221209090320358 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090320358 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090320358 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090320358 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090320359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090320359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090320359 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:03:22.320567 run patchelf(pid=18068): Terminating (SIGINT)
09:03:22.503893 run patchelf(pid=18068): Terminated: ok {rc=0}
09:03:22.536853 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=18065): Process ended prematurely: osmo-stp_10.42.42.5(pid=18065) [trial-2357↪gprs:trx-umtrx↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18065)] [trial-2357↪gprs:trx-umtrx↪ping.py:22]
09:03:22.546182 tst ping.py:22: Test FAILED (6.9 sec)
09:03:22.610282 run osmo-hlr_10.42.42.2(pid=18062): ERR: Terminated: ERROR {rc=237} [trial-2357↪gprs:trx-umtrx↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18062)]
09:03:22.662216 run osmo-hlr_10.42.42.2(pid=18062): stdout:
| (launched: 2022-12-09_09:03:19.158006)
09:03:22.714936 run osmo-hlr_10.42.42.2(pid=18062): stderr:
| �[0;m�[1;31m20221209090319354 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209090319354 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209090319354 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209090319354 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209090319354 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209090319355 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209090319381 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2357/run.2022-12-09_08-22-41/gprs:trx-umtrx/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209090319390 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209090319391 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209090319391 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:03:22.759722 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18061): Terminating (SIGTERM)
09:03:22.803080 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18064): Terminating (SIGTERM)
09:03:22.845749 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18067): Terminating (SIGTERM)
09:03:22.855898 --- ParallelTerminationStrategy: PID 18061 died...
09:03:22.913150 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18061): Terminated: ok {rc=0}
09:03:22.923600 --- ParallelTerminationStrategy: PID 18064 died...
09:03:22.980127 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18064): Terminated: ok {rc=0}
09:03:22.990749 --- ParallelTerminationStrategy: PID 18067 died...
09:03:23.047192 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18067): Terminated: ok {rc=0}
09:03:23.134637 bus /gobi_6: Setting Powered False
219gprs:trx-umtrx7.0osmo-stp_10.42.42.5(pid=18075): Process ended prematurely: osmo-stp_10.42.42.5(pid=18075) [trial-2357↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18075)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18075): Process ended prematurely: osmo-stp_10.42.42.5(pid=18075) [trial-2357↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18075)]
----------------------------------------------
trial-2357 gprs:trx-umtrx ping_idle_ping.py
----------------------------------------------
09:03:24.188118 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:24.200381 tst gprs:trx-umtrx: Using 1 x bts (candidates: 1)
09:03:24.346057 tst ping_idle_ping.py: using LAC 20384
09:03:24.474571 tst ping_idle_ping.py: using RAC 239
09:03:24.600790 tst ping_idle_ping.py: using CellId 20384
09:03:24.726931 tst ping_idle_ping.py: using BVCI 20385
09:03:24.751890 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:24.763400 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:24.774530 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:24.785872 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:24.797272 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:24.808489 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:24.820044 tst gprs:trx-umtrx: Using 1 x ip_address (candidates: 9)
09:03:24.831742 tst gprs:trx-umtrx: Using 1 x modem (candidates: 4)
09:03:25.113370 tst ping_idle_ping.py:24: start network...
09:03:25.241196 run osmo-hlr_10.42.42.2: Starting osmo-hlr
09:03:25.734334 run create_hlr_db(pid=18069): Launched
09:03:25.910959 bus /gobi_6: Setting Powered False
09:03:26.931167 run create_hlr_db(pid=18069): Terminated: ok {rc=0}
09:03:27.365778 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18071): Launched
09:03:27.694522 run osmo-hlr_10.42.42.2(pid=18072): Launched
09:03:27.814442 run osmo-stp_10.42.42.5: Starting osmo-stp
09:03:28.380790 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18074): Launched
09:03:28.708867 run osmo-stp_10.42.42.5(pid=18075): Launched
09:03:28.827673 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:03:29.399314 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18077): Launched
09:03:29.515911 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:03:29.800613 run patchelf(pid=18078): Launched
09:03:29.991567 run osmo-stp_10.42.42.5(pid=18075): ERR: Terminated: ERROR {rc=1} [trial-2357↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18075)]
09:03:30.131346 run osmo-stp_10.42.42.5(pid=18075): stdout:
| (launched: 2022-12-09_09:03:28.547307)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:03:30.271518 run osmo-stp_10.42.42.5(pid=18075): stderr:
| �[0;m�[38;5;43m20221209090328735 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090328736 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090328737 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090328737 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090328738 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090328738 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090328738 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:03:30.407463 run osmo-stp_10.42.42.5(pid=18075): stdout:
| (launched: 2022-12-09_09:03:28.547307)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
09:03:30.541680 run osmo-stp_10.42.42.5(pid=18075): stderr:
| �[0;m�[38;5;43m20221209090328735 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090328736 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090328737 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090328737 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090328738 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090328738 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090328738 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
09:03:30.697383 run patchelf(pid=18078): Terminating (SIGINT)
09:03:30.880902 run patchelf(pid=18078): Terminated: ok {rc=0}
09:03:30.915851 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=18075): Process ended prematurely: osmo-stp_10.42.42.5(pid=18075) [trial-2357↪gprs:trx-umtrx↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18075)] [trial-2357↪gprs:trx-umtrx↪ping_idle_ping.py:27]
09:03:30.925777 tst ping_idle_ping.py:27: Test FAILED (6.7 sec)
09:03:30.997706 run osmo-hlr_10.42.42.2(pid=18072): ERR: Terminated: ERROR {rc=237} [trial-2357↪gprs:trx-umtrx↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18072)]
09:03:31.051748 run osmo-hlr_10.42.42.2(pid=18072): stdout:
| (launched: 2022-12-09_09:03:27.533241)
09:03:31.104651 run osmo-hlr_10.42.42.2(pid=18072): stderr:
| �[0;m�[1;31m20221209090327728 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209090327728 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209090327728 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209090327728 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209090327728 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209090327729 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209090327741 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2357/run.2022-12-09_08-22-41/gprs:trx-umtrx/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209090327751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209090327751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209090327751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
09:03:31.149709 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18071): Terminating (SIGTERM)
09:03:31.193991 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18074): Terminating (SIGTERM)
09:03:31.238931 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18077): Terminating (SIGTERM)
09:03:31.249905 --- ParallelTerminationStrategy: PID 18071 died...
09:03:31.307062 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18071): Terminated: ok {rc=0}
09:03:31.317932 --- ParallelTerminationStrategy: PID 18074 died...
09:03:31.374497 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18074): Terminated: ok {rc=0}
09:03:31.384747 --- ParallelTerminationStrategy: PID 18077 died...
09:03:31.440215 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18077): Terminated: ok {rc=0}
09:03:31.532220 bus /gobi_6: Setting Powered False
5gprs:trx-umtrx00