71gprs:trx-b20010.0osmo-stp_10.42.42.5(pid=8902): Process ended prematurely: osmo-stp_10.42.42.5(pid=8902) [trial-2388↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8902)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8902): Process ended prematurely: osmo-stp_10.42.42.5(pid=8902) [trial-2388↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8902)]
----------------------------------------------
trial-2388 gprs:trx-b200 cs_paging_gprs_active.py
----------------------------------------------
10:27:39.529914 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:39.540942 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
10:27:39.687981 tst cs_paging_gprs_active.py: using LAC 33311
10:27:39.816233 tst cs_paging_gprs_active.py: using RAC 161
10:27:39.938931 tst cs_paging_gprs_active.py: using CellId 33311
10:27:40.063139 tst cs_paging_gprs_active.py: using BVCI 33312
10:27:40.087455 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:40.098311 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:40.108855 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:40.119385 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:40.130220 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:40.141136 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:40.152184 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:40.163070 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:40.174294 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
10:27:40.298141 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
10:27:40.602824 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
10:27:40.743631 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:27:41.092133 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8892): Launched
10:27:41.383278 run iperf3-srv_10.42.42.10(pid=8893): Launched
10:27:41.589623 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
10:27:41.729737 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:27:42.078744 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8894): Launched
10:27:42.367732 run iperf3-srv_10.42.42.10(pid=8895): Launched
10:27:42.573602 tst cs_paging_gprs_active.py:38: start network...
10:27:42.713727 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:27:43.243125 run create_hlr_db(pid=8896): Launched
10:27:43.435735 bus /sierra_2: Setting Powered False
10:27:44.458263 run create_hlr_db(pid=8896): Terminated: ok {rc=0}
10:27:44.906117 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8898): Launched
10:27:45.243678 run osmo-hlr_10.42.42.2(pid=8899): Launched
10:27:45.366916 run osmo-stp_10.42.42.5: Starting osmo-stp
10:27:45.961492 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8901): Launched
10:27:46.324738 run osmo-stp_10.42.42.5(pid=8902): Launched
10:27:46.502795 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:27:47.155380 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8904): Launched
10:27:47.271119 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:27:47.550097 run patchelf(pid=8905): Launched
10:27:47.732635 run osmo-stp_10.42.42.5(pid=8902): ERR: Terminated: ERROR {rc=1} [trial-2388↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8902)]
10:27:47.865827 run osmo-stp_10.42.42.5(pid=8902): stdout:
| (launched: 2022-12-13_10:27:46.125215)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:27:47.998863 run osmo-stp_10.42.42.5(pid=8902): stderr:
| �[0;m�[38;5;43m20221213102746459 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102746460 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102746460 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102746460 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102746473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102746473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102746473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:27:48.127648 run osmo-stp_10.42.42.5(pid=8902): stdout:
| (launched: 2022-12-13_10:27:46.125215)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:27:48.255318 run osmo-stp_10.42.42.5(pid=8902): stderr:
| �[0;m�[38;5;43m20221213102746459 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102746460 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102746460 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102746460 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102746473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102746473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102746473 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:27:48.404735 run patchelf(pid=8905): Terminating (SIGINT)
10:27:48.580596 run patchelf(pid=8905): Terminated: ok {rc=0}
10:27:48.612859 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=8902): Process ended prematurely: osmo-stp_10.42.42.5(pid=8902) [trial-2388↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8902)] [trial-2388↪gprs:trx-b200↪cs_paging_gprs_active.py:38]
10:27:48.622266 tst cs_paging_gprs_active.py:38: Test FAILED (9.1 sec)
10:27:48.682585 run osmo-hlr_10.42.42.2(pid=8899): ERR: Terminated: ERROR {rc=237} [trial-2388↪gprs:trx-b200↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8899)]
10:27:48.731971 run osmo-hlr_10.42.42.2(pid=8899): stdout:
| (launched: 2022-12-13_10:27:45.075944)
10:27:48.781883 run osmo-hlr_10.42.42.2(pid=8899): stderr:
| �[0;m�[1;31m20221213102745273 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213102745273 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213102745273 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213102745273 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213102745273 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213102745273 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213102745286 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2388/run.2022-12-13_10-04-59/gprs:trx-b200/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213102745294 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213102745294 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213102745294 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:27:48.824422 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8892): Terminating (SIGTERM)
10:27:48.864603 run iperf3-srv_10.42.42.10(pid=8893): Terminating (SIGTERM)
10:27:48.905699 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8894): Terminating (SIGTERM)
10:27:48.946309 run iperf3-srv_10.42.42.10(pid=8895): Terminating (SIGTERM)
10:27:48.987014 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8898): Terminating (SIGTERM)
10:27:49.026874 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8901): Terminating (SIGTERM)
10:27:49.066733 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8904): Terminating (SIGTERM)
10:27:49.075943 --- ParallelTerminationStrategy: PID 8892 died...
10:27:49.129690 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8892): Terminated: ok {rc=0}
10:27:49.139247 --- ParallelTerminationStrategy: PID 8893 died...
10:27:49.192290 run iperf3-srv_10.42.42.10(pid=8893): Terminated {rc=256}
10:27:49.201859 --- ParallelTerminationStrategy: PID 8894 died...
10:27:49.255173 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8894): Terminated: ok {rc=0}
10:27:49.264729 --- ParallelTerminationStrategy: PID 8895 died...
10:27:49.317891 run iperf3-srv_10.42.42.10(pid=8895): Terminated {rc=256}
10:27:49.327384 --- ParallelTerminationStrategy: PID 8898 died...
10:27:49.380570 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8898): Terminated: ok {rc=0}
10:27:49.390148 --- ParallelTerminationStrategy: PID 8901 died...
10:27:49.443289 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8901): Terminated: ok {rc=0}
10:27:49.452842 --- ParallelTerminationStrategy: PID 8904 died...
10:27:49.506246 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8904): Terminated: ok {rc=0}
10:27:49.591345 bus /gobi_0: Setting Powered False
10:27:50.700356 bus /gobi_0: Setting Powered False
10:27:51.810953 bus /gobi_6: Setting Powered False
71gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=8914): Process ended prematurely: osmo-stp_10.42.42.5(pid=8914) [trial-2388↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8914)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8914): Process ended prematurely: osmo-stp_10.42.42.5(pid=8914) [trial-2388↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8914)]
----------------------------------------------
trial-2388 gprs:trx-b200 iperf3.py
----------------------------------------------
10:27:52.928546 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:52.940486 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
10:27:53.091803 tst iperf3.py: using LAC 33312
10:27:53.224146 tst iperf3.py: using RAC 162
10:27:53.356771 tst iperf3.py: using CellId 33312
10:27:53.489112 tst iperf3.py: using BVCI 33313
10:27:53.514670 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:53.526233 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:53.537284 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:53.548270 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:53.559513 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:53.570734 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:53.582615 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:53.593789 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:27:53.605305 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
10:27:53.908279 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
10:27:54.048779 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:27:54.396620 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8906): Launched
10:27:54.685858 run iperf3-srv_10.42.42.10(pid=8907): Launched
10:27:54.890027 tst iperf3.py:8: start network...
10:27:55.029772 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:27:55.557777 run create_hlr_db(pid=8908): Launched
10:27:55.750786 bus /gobi_6: Setting Powered False
10:27:56.771716 run create_hlr_db(pid=8908): Terminated: ok {rc=0}
10:27:57.192939 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8910): Launched
10:27:57.508446 run osmo-hlr_10.42.42.2(pid=8911): Launched
10:27:57.623891 run osmo-stp_10.42.42.5: Starting osmo-stp
10:27:58.169254 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8913): Launched
10:27:58.484282 run osmo-stp_10.42.42.5(pid=8914): Launched
10:27:58.599201 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:27:59.145124 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8916): Launched
10:27:59.255934 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:27:59.529452 run patchelf(pid=8917): Launched
10:27:59.711237 run osmo-stp_10.42.42.5(pid=8914): ERR: Terminated: ERROR {rc=1} [trial-2388↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8914)]
10:27:59.844223 run osmo-stp_10.42.42.5(pid=8914): stdout:
| (launched: 2022-12-13_10:27:58.327955)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:27:59.977533 run osmo-stp_10.42.42.5(pid=8914): stderr:
| �[0;m�[38;5;43m20221213102758499 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102758499 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102758499 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102758500 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102758500 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102758500 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102758501 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:00.105914 run osmo-stp_10.42.42.5(pid=8914): stdout:
| (launched: 2022-12-13_10:27:58.327955)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:00.232932 run osmo-stp_10.42.42.5(pid=8914): stderr:
| �[0;m�[38;5;43m20221213102758499 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102758499 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102758499 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102758500 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102758500 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102758500 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102758501 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:00.381757 run patchelf(pid=8917): Terminating (SIGINT)
10:28:00.557605 run patchelf(pid=8917): Terminated: ok {rc=0}
10:28:00.589898 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=8914): Process ended prematurely: osmo-stp_10.42.42.5(pid=8914) [trial-2388↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8914)] [trial-2388↪gprs:trx-b200↪iperf3.py:8]
10:28:00.598980 tst iperf3.py:8: Test FAILED (7.7 sec)
10:28:00.659546 run osmo-hlr_10.42.42.2(pid=8911): ERR: Terminated: ERROR {rc=237} [trial-2388↪gprs:trx-b200↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8911)]
10:28:00.708931 run osmo-hlr_10.42.42.2(pid=8911): stdout:
| (launched: 2022-12-13_10:27:57.351317)
10:28:00.759354 run osmo-hlr_10.42.42.2(pid=8911): stderr:
| �[0;m�[1;31m20221213102757530 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213102757530 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213102757530 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213102757530 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213102757530 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213102757530 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213102757542 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2388/run.2022-12-13_10-04-59/gprs:trx-b200/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213102757551 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213102757551 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213102757551 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:28:00.801765 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8906): Terminating (SIGTERM)
10:28:00.841707 run iperf3-srv_10.42.42.10(pid=8907): Terminating (SIGTERM)
10:28:00.884339 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8910): Terminating (SIGTERM)
10:28:00.926959 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8913): Terminating (SIGTERM)
10:28:00.975060 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8916): Terminating (SIGTERM)
10:28:00.984802 --- ParallelTerminationStrategy: PID 8906 died...
10:28:01.041006 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8906): Terminated: ok {rc=0}
10:28:01.051488 --- ParallelTerminationStrategy: PID 8907 died...
10:28:01.106738 run iperf3-srv_10.42.42.10(pid=8907): Terminated {rc=256}
10:28:01.116781 --- ParallelTerminationStrategy: PID 8910 died...
10:28:01.171068 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8910): Terminated: ok {rc=0}
10:28:01.181156 --- ParallelTerminationStrategy: PID 8913 died...
10:28:01.235547 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8913): Terminated: ok {rc=0}
10:28:01.245679 --- ParallelTerminationStrategy: PID 8916 died...
10:28:01.300268 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8916): Terminated: ok {rc=0}
10:28:01.386538 bus /gobi_6: Setting Powered False
203gprs:trx-b20011.0osmo-stp_10.42.42.5(pid=8932): Process ended prematurely: osmo-stp_10.42.42.5(pid=8932) [trial-2388↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8932)]2186iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8932): Process ended prematurely: osmo-stp_10.42.42.5(pid=8932) [trial-2388↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8932)]
----------------------------------------------
trial-2388 gprs:trx-b200 iperf3m4.py
----------------------------------------------
10:28:02.496189 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:02.507703 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
10:28:02.646884 tst iperf3m4.py: using LAC 33313
10:28:02.770434 tst iperf3m4.py: using RAC 163
10:28:02.894694 tst iperf3m4.py: using CellId 33313
10:28:03.019551 tst iperf3m4.py: using BVCI 33314
10:28:03.043612 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:03.054634 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:03.065485 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:03.076495 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:03.087289 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:03.097963 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:03.108633 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:03.119430 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:03.130262 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
10:28:03.248196 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
10:28:03.373274 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
10:28:03.493483 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
10:28:03.798844 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
10:28:03.939024 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:28:04.287312 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8918): Launched
10:28:04.577782 run iperf3-srv_10.42.42.10(pid=8919): Launched
10:28:04.783213 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
10:28:04.922842 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:28:05.272791 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8920): Launched
10:28:05.563804 run iperf3-srv_10.42.42.10(pid=8921): Launched
10:28:05.768928 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
10:28:05.908871 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:28:06.256246 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8922): Launched
10:28:06.546751 run iperf3-srv_10.42.42.10(pid=8923): Launched
10:28:06.751440 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
10:28:06.890587 run iperf3-srv_10.42.42.10: Starting iperf3-srv
10:28:07.238746 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8924): Launched
10:28:07.527752 run iperf3-srv_10.42.42.10(pid=8925): Launched
10:28:07.733524 tst iperf3m4.py:8: start network...
10:28:07.873554 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:28:08.402954 run create_hlr_db(pid=8926): Launched
10:28:08.596925 bus /gobi_6: Setting Powered False
10:28:09.618804 run create_hlr_db(pid=8926): Terminated: ok {rc=0}
10:28:10.040793 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8928): Launched
10:28:10.357661 run osmo-hlr_10.42.42.2(pid=8929): Launched
10:28:10.472301 run osmo-stp_10.42.42.5: Starting osmo-stp
10:28:11.016511 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8931): Launched
10:28:11.331867 run osmo-stp_10.42.42.5(pid=8932): Launched
10:28:11.445480 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:28:11.990047 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8934): Launched
10:28:12.101247 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:28:12.375292 run patchelf(pid=8935): Launched
10:28:12.558133 run osmo-stp_10.42.42.5(pid=8932): ERR: Terminated: ERROR {rc=1} [trial-2388↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8932)]
10:28:12.691259 run osmo-stp_10.42.42.5(pid=8932): stdout:
| (launched: 2022-12-13_10:28:11.175254)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:12.824300 run osmo-stp_10.42.42.5(pid=8932): stderr:
| �[0;m�[38;5;43m20221213102811346 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102811346 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102811346 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102811347 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102811347 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102811348 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102811348 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:12.952641 run osmo-stp_10.42.42.5(pid=8932): stdout:
| (launched: 2022-12-13_10:28:11.175254)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:13.080642 run osmo-stp_10.42.42.5(pid=8932): stderr:
| �[0;m�[38;5;43m20221213102811346 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102811346 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102811346 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102811347 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102811347 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102811348 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102811348 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:13.230129 run patchelf(pid=8935): Terminating (SIGINT)
10:28:13.405660 run patchelf(pid=8935): Terminated: ok {rc=0}
10:28:13.438039 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=8932): Process ended prematurely: osmo-stp_10.42.42.5(pid=8932) [trial-2388↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8932)] [trial-2388↪gprs:trx-b200↪iperf3m4.py:8]
10:28:13.447258 tst iperf3m4.py:8: Test FAILED (11.0 sec)
10:28:13.508074 run osmo-hlr_10.42.42.2(pid=8929): ERR: Terminated: ERROR {rc=237} [trial-2388↪gprs:trx-b200↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8929)]
10:28:13.557697 run osmo-hlr_10.42.42.2(pid=8929): stdout:
| (launched: 2022-12-13_10:28:10.200251)
10:28:13.607812 run osmo-hlr_10.42.42.2(pid=8929): stderr:
| �[0;m�[1;31m20221213102810376 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213102810376 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213102810376 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213102810376 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213102810376 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213102810376 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213102810389 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2388/run.2022-12-13_10-04-59/gprs:trx-b200/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213102810397 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213102810397 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213102810397 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:28:13.650284 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8918): Terminating (SIGTERM)
10:28:13.690239 run iperf3-srv_10.42.42.10(pid=8919): Terminating (SIGTERM)
10:28:13.730384 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8920): Terminating (SIGTERM)
10:28:13.770447 run iperf3-srv_10.42.42.10(pid=8921): Terminating (SIGTERM)
10:28:13.810645 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8922): Terminating (SIGTERM)
10:28:13.850928 run iperf3-srv_10.42.42.10(pid=8923): Terminating (SIGTERM)
10:28:13.891169 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8924): Terminating (SIGTERM)
10:28:13.931376 run iperf3-srv_10.42.42.10(pid=8925): Terminating (SIGTERM)
10:28:13.971587 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8928): Terminating (SIGTERM)
10:28:14.011817 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8931): Terminating (SIGTERM)
10:28:14.052010 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8934): Terminating (SIGTERM)
10:28:14.061406 --- ParallelTerminationStrategy: PID 8918 died...
10:28:14.115530 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8918): Terminated: ok {rc=0}
10:28:14.125389 --- ParallelTerminationStrategy: PID 8919 died...
10:28:14.178701 run iperf3-srv_10.42.42.10(pid=8919): Terminated {rc=256}
10:28:14.188371 --- ParallelTerminationStrategy: PID 8920 died...
10:28:14.241886 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8920): Terminated: ok {rc=0}
10:28:14.251555 --- ParallelTerminationStrategy: PID 8921 died...
10:28:14.304830 run iperf3-srv_10.42.42.10(pid=8921): Terminated {rc=256}
10:28:14.314530 --- ParallelTerminationStrategy: PID 8922 died...
10:28:14.368092 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8922): Terminated: ok {rc=0}
10:28:14.377736 --- ParallelTerminationStrategy: PID 8923 died...
10:28:14.430961 run iperf3-srv_10.42.42.10(pid=8923): Terminated {rc=256}
10:28:14.440582 --- ParallelTerminationStrategy: PID 8924 died...
10:28:14.493849 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8924): Terminated: ok {rc=0}
10:28:14.503548 --- ParallelTerminationStrategy: PID 8925 died...
10:28:14.556723 run iperf3-srv_10.42.42.10(pid=8925): Terminated {rc=256}
10:28:14.566466 --- ParallelTerminationStrategy: PID 8928 died...
10:28:14.619873 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8928): Terminated: ok {rc=0}
10:28:14.629527 --- ParallelTerminationStrategy: PID 8931 died...
10:28:14.682765 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8931): Terminated: ok {rc=0}
10:28:14.692450 --- ParallelTerminationStrategy: PID 8934 died...
10:28:14.746246 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8934): Terminated: ok {rc=0}
10:28:14.835532 bus /gobi_4: Setting Powered False
10:28:15.987740 bus /gobi_4: Setting Powered False
10:28:17.122072 bus /gobi_1: Setting Powered False
10:28:18.230823 bus /gobi_1: Setting Powered False
10:28:19.340721 bus /gobi_0: Setting Powered False
10:28:20.447871 bus /gobi_0: Setting Powered False
10:28:21.558443 bus /gobi_6: Setting Powered False
203gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=8942): Process ended prematurely: osmo-stp_10.42.42.5(pid=8942) [trial-2388↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8942)]2186ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8942): Process ended prematurely: osmo-stp_10.42.42.5(pid=8942) [trial-2388↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8942)]
----------------------------------------------
trial-2388 gprs:trx-b200 ping.py
----------------------------------------------
10:28:22.625406 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:22.636699 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
10:28:22.774195 tst ping.py: using LAC 33314
10:28:22.894342 tst ping.py: using RAC 164
10:28:23.016351 tst ping.py: using CellId 33314
10:28:23.138033 tst ping.py: using BVCI 33315
10:28:23.162251 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:23.172894 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:23.183450 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:23.194026 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:23.204369 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:23.214729 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:23.225214 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:23.235715 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
10:28:23.516364 tst ping.py:19: start network...
10:28:23.644041 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:28:24.137585 run create_hlr_db(pid=8936): Launched
10:28:24.313832 bus /gobi_6: Setting Powered False
10:28:25.335880 run create_hlr_db(pid=8936): Terminated: ok {rc=0}
10:28:25.777193 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8938): Launched
10:28:26.107302 run osmo-hlr_10.42.42.2(pid=8939): Launched
10:28:26.226742 run osmo-stp_10.42.42.5: Starting osmo-stp
10:28:26.804150 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8941): Launched
10:28:27.135817 run osmo-stp_10.42.42.5(pid=8942): Launched
10:28:27.256249 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:28:27.832855 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8944): Launched
10:28:27.950267 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:28:28.239658 run patchelf(pid=8945): Launched
10:28:28.431968 run osmo-stp_10.42.42.5(pid=8942): ERR: Terminated: ERROR {rc=1} [trial-2388↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8942)]
10:28:28.572680 run osmo-stp_10.42.42.5(pid=8942): stdout:
| (launched: 2022-12-13_10:28:26.971988)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:28.713458 run osmo-stp_10.42.42.5(pid=8942): stderr:
| �[0;m�[38;5;43m20221213102827164 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102827164 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102827164 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102827164 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102827165 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102827165 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102827165 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:28.849660 run osmo-stp_10.42.42.5(pid=8942): stdout:
| (launched: 2022-12-13_10:28:26.971988)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:28.984759 run osmo-stp_10.42.42.5(pid=8942): stderr:
| �[0;m�[38;5;43m20221213102827164 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102827164 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102827164 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102827164 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102827165 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102827165 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102827165 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:29.141976 run patchelf(pid=8945): Terminating (SIGINT)
10:28:29.326566 run patchelf(pid=8945): Terminated: ok {rc=0}
10:28:29.360044 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=8942): Process ended prematurely: osmo-stp_10.42.42.5(pid=8942) [trial-2388↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8942)] [trial-2388↪gprs:trx-b200↪ping.py:22]
10:28:29.369578 tst ping.py:22: Test FAILED (6.7 sec)
10:28:29.434140 run osmo-hlr_10.42.42.2(pid=8939): ERR: Terminated: ERROR {rc=237} [trial-2388↪gprs:trx-b200↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8939)]
10:28:29.486537 run osmo-hlr_10.42.42.2(pid=8939): stdout:
| (launched: 2022-12-13_10:28:25.944446)
10:28:29.539912 run osmo-hlr_10.42.42.2(pid=8939): stderr:
| �[0;m�[1;31m20221213102826142 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213102826142 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213102826142 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213102826142 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213102826142 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213102826142 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213102826155 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2388/run.2022-12-13_10-04-59/gprs:trx-b200/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213102826164 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213102826164 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213102826164 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:28:29.585304 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8938): Terminating (SIGTERM)
10:28:29.628466 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8941): Terminating (SIGTERM)
10:28:29.671335 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8944): Terminating (SIGTERM)
10:28:29.681320 --- ParallelTerminationStrategy: PID 8938 died...
10:28:29.739483 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8938): Terminated: ok {rc=0}
10:28:29.749949 --- ParallelTerminationStrategy: PID 8941 died...
10:28:29.807532 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8941): Terminated: ok {rc=0}
10:28:29.817813 --- ParallelTerminationStrategy: PID 8944 died...
10:28:29.874902 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8944): Terminated: ok {rc=0}
10:28:29.964262 bus /gobi_6: Setting Powered False
203gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=8952): Process ended prematurely: osmo-stp_10.42.42.5(pid=8952) [trial-2388↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8952)]2186ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=8952): Process ended prematurely: osmo-stp_10.42.42.5(pid=8952) [trial-2388↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8952)]
----------------------------------------------
trial-2388 gprs:trx-b200 ping_idle_ping.py
----------------------------------------------
10:28:31.018743 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:31.031420 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
10:28:31.185615 tst ping_idle_ping.py: using LAC 33315
10:28:31.329070 tst ping_idle_ping.py: using RAC 165
10:28:31.484411 tst ping_idle_ping.py: using CellId 33315
10:28:31.614398 tst ping_idle_ping.py: using BVCI 33316
10:28:31.639548 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:31.651102 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:31.662209 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:31.673702 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:31.685442 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:31.696423 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:31.708024 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
10:28:31.719114 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
10:28:32.038053 tst ping_idle_ping.py:24: start network...
10:28:32.179734 run osmo-hlr_10.42.42.2: Starting osmo-hlr
10:28:32.706993 run create_hlr_db(pid=8946): Launched
10:28:32.898019 bus /gobi_6: Setting Powered False
10:28:33.918796 run create_hlr_db(pid=8946): Terminated: ok {rc=0}
10:28:34.333175 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8948): Launched
10:28:34.643141 run osmo-hlr_10.42.42.2(pid=8949): Launched
10:28:34.755864 run osmo-stp_10.42.42.5: Starting osmo-stp
10:28:35.289895 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8951): Launched
10:28:35.598571 run osmo-stp_10.42.42.5(pid=8952): Launched
10:28:35.709665 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
10:28:36.246720 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8954): Launched
10:28:36.355509 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
10:28:36.626065 run patchelf(pid=8955): Launched
10:28:36.805776 run osmo-stp_10.42.42.5(pid=8952): ERR: Terminated: ERROR {rc=1} [trial-2388↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8952)]
10:28:36.937206 run osmo-stp_10.42.42.5(pid=8952): stdout:
| (launched: 2022-12-13_10:28:35.445481)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:37.068426 run osmo-stp_10.42.42.5(pid=8952): stderr:
| �[0;m�[38;5;43m20221213102835617 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102835617 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102835617 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102835618 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102835618 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102835618 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102835618 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:37.195109 run osmo-stp_10.42.42.5(pid=8952): stdout:
| (launched: 2022-12-13_10:28:35.445481)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
10:28:37.321737 run osmo-stp_10.42.42.5(pid=8952): stderr:
| �[0;m�[38;5;43m20221213102835617 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221213102835617 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221213102835617 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221213102835618 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221213102835618 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221213102835618 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221213102835618 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
10:28:37.468289 run patchelf(pid=8955): Terminating (SIGINT)
10:28:37.640851 run patchelf(pid=8955): Terminated: ok {rc=0}
10:28:37.672194 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=8952): Process ended prematurely: osmo-stp_10.42.42.5(pid=8952) [trial-2388↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=8952)] [trial-2388↪gprs:trx-b200↪ping_idle_ping.py:27]
10:28:37.680886 tst ping_idle_ping.py:27: Test FAILED (6.7 sec)
10:28:37.740819 run osmo-hlr_10.42.42.2(pid=8949): ERR: Terminated: ERROR {rc=237} [trial-2388↪gprs:trx-b200↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=8949)]
10:28:37.789811 run osmo-hlr_10.42.42.2(pid=8949): stdout:
| (launched: 2022-12-13_10:28:34.488507)
10:28:37.839499 run osmo-hlr_10.42.42.2(pid=8949): stderr:
| �[0;m�[1;31m20221213102834664 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221213102834665 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221213102834665 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221213102834665 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221213102834665 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221213102834665 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221213102834676 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2388/run.2022-12-13_10-04-59/gprs:trx-b200/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221213102834685 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221213102834685 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221213102834685 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
10:28:37.881801 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8948): Terminating (SIGTERM)
10:28:37.921726 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8951): Terminating (SIGTERM)
10:28:37.961947 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8954): Terminating (SIGTERM)
10:28:37.971252 --- ParallelTerminationStrategy: PID 8948 died...
10:28:38.024545 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8948): Terminated: ok {rc=0}
10:28:38.034267 --- ParallelTerminationStrategy: PID 8951 died...
10:28:38.087474 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8951): Terminated: ok {rc=0}
10:28:38.097161 --- ParallelTerminationStrategy: PID 8954 died...
10:28:38.150265 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8954): Terminated: ok {rc=0}
10:28:38.239375 bus /gobi_6: Setting Powered False
5gprs:trx-b20000