88gprs:trx-b2009.0osmo-stp_10.42.42.5(pid=25988): Process ended prematurely: osmo-stp_10.42.42.5(pid=25988) [trial-2405↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=25988)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=25988): Process ended prematurely: osmo-stp_10.42.42.5(pid=25988) [trial-2405↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=25988)]
----------------------------------------------
trial-2405 gprs:trx-b200 cs_paging_gprs_active.py
----------------------------------------------
13:15:33.818261 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:33.829179 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
13:15:33.977527 tst cs_paging_gprs_active.py: using LAC 41148
13:15:34.102269 tst cs_paging_gprs_active.py: using RAC 93
13:15:34.227062 tst cs_paging_gprs_active.py: using CellId 41148
13:15:34.353608 tst cs_paging_gprs_active.py: using BVCI 41149
13:15:34.378306 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:34.389246 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:34.399844 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:34.410600 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:34.421553 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:34.432343 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:34.443443 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:34.454422 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:34.465590 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
13:15:34.583790 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
13:15:34.879309 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
13:15:35.015828 run iperf3-srv_10.42.42.10: Starting iperf3-srv
13:15:35.352221 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=25977): Launched
13:15:35.631005 run iperf3-srv_10.42.42.10(pid=25978): Launched
13:15:35.831459 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
13:15:35.967733 run iperf3-srv_10.42.42.10: Starting iperf3-srv
13:15:36.303608 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=25979): Launched
13:15:36.581912 run iperf3-srv_10.42.42.10(pid=25980): Launched
13:15:36.782102 tst cs_paging_gprs_active.py:38: start network...
13:15:36.918133 run osmo-hlr_10.42.42.2: Starting osmo-hlr
13:15:37.429322 run create_hlr_db(pid=25981): Launched
13:15:37.618357 bus /sierra_2: Setting Powered False
13:15:38.628251 run create_hlr_db(pid=25981): Terminated: ok {rc=0}
13:15:39.039342 run pcap-recorder_any(filters='host 10.42.42.2')(pid=25984): Launched
13:15:39.341786 run osmo-hlr_10.42.42.2(pid=25985): Launched
13:15:39.461724 run osmo-stp_10.42.42.5: Starting osmo-stp
13:15:40.000777 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=25987): Launched
13:15:40.325624 run osmo-stp_10.42.42.5(pid=25988): Launched
13:15:40.443847 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
13:15:41.010705 run pcap-recorder_any(filters='host 10.42.42.6')(pid=25990): Launched
13:15:41.127132 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
13:15:41.409970 run patchelf(pid=25991): Launched
13:15:41.600243 run osmo-stp_10.42.42.5(pid=25988): ERR: Terminated: ERROR {rc=1} [trial-2405↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=25988)]
13:15:41.739678 run osmo-stp_10.42.42.5(pid=25988): stdout:
| (launched: 2022-12-14_13:15:40.166898)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:15:41.879602 run osmo-stp_10.42.42.5(pid=25988): stderr:
| �[0;m�[38;5;43m20221214131540359 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131540359 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131540360 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131540360 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131540360 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131540361 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131540361 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:15:42.015664 run osmo-stp_10.42.42.5(pid=25988): stdout:
| (launched: 2022-12-14_13:15:40.166898)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:15:42.141313 run osmo-stp_10.42.42.5(pid=25988): stderr:
| �[0;m�[38;5;43m20221214131540359 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131540359 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131540360 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131540360 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131540360 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131540361 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131540361 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:15:42.287234 run patchelf(pid=25991): Terminating (SIGINT)
13:15:42.457968 run patchelf(pid=25991): Terminated: ok {rc=0}
13:15:42.489576 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=25988): Process ended prematurely: osmo-stp_10.42.42.5(pid=25988) [trial-2405↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=25988)] [trial-2405↪gprs:trx-b200↪cs_paging_gprs_active.py:38]
13:15:42.499435 tst cs_paging_gprs_active.py:38: Test FAILED (8.7 sec)
13:15:42.559645 run osmo-hlr_10.42.42.2(pid=25985): ERR: Terminated: ERROR {rc=237} [trial-2405↪gprs:trx-b200↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=25985)]
13:15:42.609618 run osmo-hlr_10.42.42.2(pid=25985): stdout:
| (launched: 2022-12-14_13:15:39.193160)
13:15:42.659654 run osmo-hlr_10.42.42.2(pid=25985): stderr:
| �[0;m�[1;31m20221214131539369 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214131539369 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214131539369 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214131539369 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214131539369 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214131539370 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214131539381 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2405/run.2022-12-14_12-53-19/gprs:trx-b200/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214131539390 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214131539390 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214131539390 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
13:15:42.702449 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=25977): Terminating (SIGTERM)
13:15:42.742564 run iperf3-srv_10.42.42.10(pid=25978): Terminating (SIGTERM)
13:15:42.782722 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=25979): Terminating (SIGTERM)
13:15:42.822561 run iperf3-srv_10.42.42.10(pid=25980): Terminating (SIGTERM)
13:15:42.862633 run pcap-recorder_any(filters='host 10.42.42.2')(pid=25984): Terminating (SIGTERM)
13:15:42.902801 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=25987): Terminating (SIGTERM)
13:15:42.943185 run pcap-recorder_any(filters='host 10.42.42.6')(pid=25990): Terminating (SIGTERM)
13:15:42.952465 --- ParallelTerminationStrategy: PID 25977 died...
13:15:43.006945 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=25977): Terminated: ok {rc=0}
13:15:43.016702 --- ParallelTerminationStrategy: PID 25978 died...
13:15:43.070566 run iperf3-srv_10.42.42.10(pid=25978): Terminated {rc=256}
13:15:43.080247 --- ParallelTerminationStrategy: PID 25979 died...
13:15:43.133827 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=25979): Terminated: ok {rc=0}
13:15:43.143464 --- ParallelTerminationStrategy: PID 25980 died...
13:15:43.197189 run iperf3-srv_10.42.42.10(pid=25980): Terminated {rc=256}
13:15:43.206871 --- ParallelTerminationStrategy: PID 25984 died...
13:15:43.260624 run pcap-recorder_any(filters='host 10.42.42.2')(pid=25984): Terminated: ok {rc=0}
13:15:43.270357 --- ParallelTerminationStrategy: PID 25987 died...
13:15:43.324077 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=25987): Terminated: ok {rc=0}
13:15:43.333880 --- ParallelTerminationStrategy: PID 25990 died...
13:15:43.387390 run pcap-recorder_any(filters='host 10.42.42.6')(pid=25990): Terminated: ok {rc=0}
13:15:43.475192 bus /gobi_0: Setting Powered False
13:15:44.578686 bus /gobi_0: Setting Powered False
13:15:45.688673 bus /gobi_6: Setting Powered False
88gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=26000): Process ended prematurely: osmo-stp_10.42.42.5(pid=26000) [trial-2405↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26000)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=26000): Process ended prematurely: osmo-stp_10.42.42.5(pid=26000) [trial-2405↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26000)]
----------------------------------------------
trial-2405 gprs:trx-b200 iperf3.py
----------------------------------------------
13:15:46.807584 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:46.820188 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
13:15:46.974038 tst iperf3.py: using LAC 41149
13:15:47.105238 tst iperf3.py: using RAC 94
13:15:47.235630 tst iperf3.py: using CellId 41149
13:15:47.356291 tst iperf3.py: using BVCI 41150
13:15:47.379880 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:47.390600 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:47.401048 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:47.411402 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:47.422112 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:47.432710 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:47.443175 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:47.453776 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:47.464025 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
13:15:47.759639 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
13:15:47.895977 run iperf3-srv_10.42.42.10: Starting iperf3-srv
13:15:48.230922 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=25992): Launched
13:15:48.508415 run iperf3-srv_10.42.42.10(pid=25993): Launched
13:15:48.708276 tst iperf3.py:8: start network...
13:15:48.844061 run osmo-hlr_10.42.42.2: Starting osmo-hlr
13:15:49.351179 run create_hlr_db(pid=25994): Launched
13:15:49.538534 bus /gobi_6: Setting Powered False
13:15:50.559764 run create_hlr_db(pid=25994): Terminated: ok {rc=0}
13:15:50.965857 run pcap-recorder_any(filters='host 10.42.42.2')(pid=25996): Launched
13:15:51.267889 run osmo-hlr_10.42.42.2(pid=25997): Launched
13:15:51.387322 run osmo-stp_10.42.42.5: Starting osmo-stp
13:15:51.921290 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=25999): Launched
13:15:52.244693 run osmo-stp_10.42.42.5(pid=26000): Launched
13:15:52.362948 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
13:15:52.929057 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26002): Launched
13:15:53.045118 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
13:15:53.325674 run patchelf(pid=26003): Launched
13:15:53.514627 run osmo-stp_10.42.42.5(pid=26000): ERR: Terminated: ERROR {rc=1} [trial-2405↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26000)]
13:15:53.653363 run osmo-stp_10.42.42.5(pid=26000): stdout:
| (launched: 2022-12-14_13:15:52.085460)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:15:53.792033 run osmo-stp_10.42.42.5(pid=26000): stderr:
| �[0;m�[38;5;43m20221214131552276 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131552277 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131552277 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131552277 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131552278 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131552278 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131552278 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:15:53.926290 run osmo-stp_10.42.42.5(pid=26000): stdout:
| (launched: 2022-12-14_13:15:52.085460)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:15:54.059061 run osmo-stp_10.42.42.5(pid=26000): stderr:
| �[0;m�[38;5;43m20221214131552276 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131552277 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131552277 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131552277 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131552278 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131552278 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131552278 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:15:54.213792 run patchelf(pid=26003): Terminating (SIGINT)
13:15:54.395012 run patchelf(pid=26003): Terminated: ok {rc=0}
13:15:54.428092 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=26000): Process ended prematurely: osmo-stp_10.42.42.5(pid=26000) [trial-2405↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26000)] [trial-2405↪gprs:trx-b200↪iperf3.py:8]
13:15:54.437999 tst iperf3.py:8: Test FAILED (7.7 sec)
13:15:54.500666 run osmo-hlr_10.42.42.2(pid=25997): ERR: Terminated: ERROR {rc=237} [trial-2405↪gprs:trx-b200↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=25997)]
13:15:54.551546 run osmo-hlr_10.42.42.2(pid=25997): stdout:
| (launched: 2022-12-14_13:15:51.119064)
13:15:54.602888 run osmo-hlr_10.42.42.2(pid=25997): stderr:
| �[0;m�[1;31m20221214131551297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214131551297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214131551297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214131551297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214131551297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214131551297 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214131551309 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2405/run.2022-12-14_12-53-19/gprs:trx-b200/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214131551319 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214131551319 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214131551319 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
13:15:54.647351 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=25992): Terminating (SIGTERM)
13:15:54.689932 run iperf3-srv_10.42.42.10(pid=25993): Terminating (SIGTERM)
13:15:54.731945 run pcap-recorder_any(filters='host 10.42.42.2')(pid=25996): Terminating (SIGTERM)
13:15:54.774119 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=25999): Terminating (SIGTERM)
13:15:54.815855 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26002): Terminating (SIGTERM)
13:15:54.825551 --- ParallelTerminationStrategy: PID 25992 died...
13:15:54.881886 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=25992): Terminated: ok {rc=0}
13:15:54.891818 --- ParallelTerminationStrategy: PID 25993 died...
13:15:54.947609 run iperf3-srv_10.42.42.10(pid=25993): Terminated {rc=256}
13:15:54.957900 --- ParallelTerminationStrategy: PID 25996 died...
13:15:55.013835 run pcap-recorder_any(filters='host 10.42.42.2')(pid=25996): Terminated: ok {rc=0}
13:15:55.023783 --- ParallelTerminationStrategy: PID 25999 died...
13:15:55.079983 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=25999): Terminated: ok {rc=0}
13:15:55.090127 --- ParallelTerminationStrategy: PID 26002 died...
13:15:55.146460 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26002): Terminated: ok {rc=0}
13:15:55.233790 bus /gobi_6: Setting Powered False
220gprs:trx-b20011.0osmo-stp_10.42.42.5(pid=26018): Process ended prematurely: osmo-stp_10.42.42.5(pid=26018) [trial-2405↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26018)]2186iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=26018): Process ended prematurely: osmo-stp_10.42.42.5(pid=26018) [trial-2405↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26018)]
----------------------------------------------
trial-2405 gprs:trx-b200 iperf3m4.py
----------------------------------------------
13:15:56.348869 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:56.360459 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
13:15:56.503074 tst iperf3m4.py: using LAC 41150
13:15:56.629181 tst iperf3m4.py: using RAC 95
13:15:56.759033 tst iperf3m4.py: using CellId 41150
13:15:56.885432 tst iperf3m4.py: using BVCI 41151
13:15:56.910124 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:56.921239 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:56.932002 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:56.942850 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:56.954323 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:56.965832 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:56.977676 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:56.989212 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:15:57.001150 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
13:15:57.119704 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
13:15:57.241847 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
13:15:57.357814 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
13:15:57.635332 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
13:15:57.761459 run iperf3-srv_10.42.42.10: Starting iperf3-srv
13:15:58.075615 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26004): Launched
13:15:58.334866 run iperf3-srv_10.42.42.10(pid=26005): Launched
13:15:58.518415 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
13:15:58.643973 run iperf3-srv_10.42.42.10: Starting iperf3-srv
13:15:58.957975 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26006): Launched
13:15:59.216714 run iperf3-srv_10.42.42.10(pid=26007): Launched
13:15:59.400414 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
13:15:59.525526 run iperf3-srv_10.42.42.10: Starting iperf3-srv
13:15:59.837480 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26008): Launched
13:16:00.096422 run iperf3-srv_10.42.42.10(pid=26009): Launched
13:16:00.280108 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
13:16:00.405508 run iperf3-srv_10.42.42.10: Starting iperf3-srv
13:16:00.718757 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26010): Launched
13:16:01.020362 run iperf3-srv_10.42.42.10(pid=26011): Launched
13:16:01.233071 tst iperf3m4.py:8: start network...
13:16:01.373917 run osmo-hlr_10.42.42.2: Starting osmo-hlr
13:16:01.929714 run create_hlr_db(pid=26012): Launched
13:16:02.106476 bus /gobi_6: Setting Powered False
13:16:03.126115 run create_hlr_db(pid=26012): Terminated: ok {rc=0}
13:16:03.556696 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26014): Launched
13:16:03.881416 run osmo-hlr_10.42.42.2(pid=26015): Launched
13:16:03.999510 run osmo-stp_10.42.42.5: Starting osmo-stp
13:16:04.563897 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26017): Launched
13:16:04.889585 run osmo-stp_10.42.42.5(pid=26018): Launched
13:16:05.007724 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
13:16:05.571502 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26020): Launched
13:16:05.687497 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
13:16:05.969491 run patchelf(pid=26021): Launched
13:16:06.158922 run osmo-stp_10.42.42.5(pid=26018): ERR: Terminated: ERROR {rc=1} [trial-2405↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26018)]
13:16:06.297787 run osmo-stp_10.42.42.5(pid=26018): stdout:
| (launched: 2022-12-14_13:16:04.730124)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:16:06.437099 run osmo-stp_10.42.42.5(pid=26018): stderr:
| �[0;m�[38;5;43m20221214131604923 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131604923 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131604924 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131604924 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131604925 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131604925 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131604925 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:16:06.571807 run osmo-stp_10.42.42.5(pid=26018): stdout:
| (launched: 2022-12-14_13:16:04.730124)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:16:06.705859 run osmo-stp_10.42.42.5(pid=26018): stderr:
| �[0;m�[38;5;43m20221214131604923 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131604923 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131604924 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131604924 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131604925 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131604925 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131604925 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:16:06.861469 run patchelf(pid=26021): Terminating (SIGINT)
13:16:07.043446 run patchelf(pid=26021): Terminated: ok {rc=0}
13:16:07.076644 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=26018): Process ended prematurely: osmo-stp_10.42.42.5(pid=26018) [trial-2405↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26018)] [trial-2405↪gprs:trx-b200↪iperf3m4.py:8]
13:16:07.086712 tst iperf3m4.py:8: Test FAILED (10.8 sec)
13:16:07.150342 run osmo-hlr_10.42.42.2(pid=26015): ERR: Terminated: ERROR {rc=237} [trial-2405↪gprs:trx-b200↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=26015)]
13:16:07.202009 run osmo-hlr_10.42.42.2(pid=26015): stdout:
| (launched: 2022-12-14_13:16:03.722983)
13:16:07.254015 run osmo-hlr_10.42.42.2(pid=26015): stderr:
| �[0;m�[1;31m20221214131603922 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214131603922 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214131603922 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214131603922 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214131603922 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214131603922 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214131603941 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2405/run.2022-12-14_12-53-19/gprs:trx-b200/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214131603951 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214131603951 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214131603951 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
13:16:07.298698 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26004): Terminating (SIGTERM)
13:16:07.340697 run iperf3-srv_10.42.42.10(pid=26005): Terminating (SIGTERM)
13:16:07.384367 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26006): Terminating (SIGTERM)
13:16:07.426457 run iperf3-srv_10.42.42.10(pid=26007): Terminating (SIGTERM)
13:16:07.468350 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26008): Terminating (SIGTERM)
13:16:07.510324 run iperf3-srv_10.42.42.10(pid=26009): Terminating (SIGTERM)
13:16:07.552328 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26010): Terminating (SIGTERM)
13:16:07.594205 run iperf3-srv_10.42.42.10(pid=26011): Terminating (SIGTERM)
13:16:07.636147 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26014): Terminating (SIGTERM)
13:16:07.678498 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26017): Terminating (SIGTERM)
13:16:07.720595 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26020): Terminating (SIGTERM)
13:16:07.730363 --- ParallelTerminationStrategy: PID 26004 died...
13:16:07.786962 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26004): Terminated: ok {rc=0}
13:16:07.797092 --- ParallelTerminationStrategy: PID 26005 died...
13:16:07.853265 run iperf3-srv_10.42.42.10(pid=26005): Terminated {rc=256}
13:16:07.863255 --- ParallelTerminationStrategy: PID 26006 died...
13:16:07.919482 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26006): Terminated: ok {rc=0}
13:16:07.929949 --- ParallelTerminationStrategy: PID 26007 died...
13:16:07.986080 run iperf3-srv_10.42.42.10(pid=26007): Terminated {rc=256}
13:16:07.996115 --- ParallelTerminationStrategy: PID 26008 died...
13:16:08.052318 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26008): Terminated: ok {rc=0}
13:16:08.062364 --- ParallelTerminationStrategy: PID 26009 died...
13:16:08.118491 run iperf3-srv_10.42.42.10(pid=26009): Terminated {rc=256}
13:16:08.128483 --- ParallelTerminationStrategy: PID 26010 died...
13:16:08.184722 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26010): Terminated: ok {rc=0}
13:16:08.194773 --- ParallelTerminationStrategy: PID 26011 died...
13:16:08.250870 run iperf3-srv_10.42.42.10(pid=26011): Terminated {rc=256}
13:16:08.261062 --- ParallelTerminationStrategy: PID 26014 died...
13:16:08.317109 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26014): Terminated: ok {rc=0}
13:16:08.327110 --- ParallelTerminationStrategy: PID 26017 died...
13:16:08.383505 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26017): Terminated: ok {rc=0}
13:16:08.393983 --- ParallelTerminationStrategy: PID 26020 died...
13:16:08.450212 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26020): Terminated: ok {rc=0}
13:16:08.537136 bus /gobi_4: Setting Powered False
13:16:09.643888 bus /gobi_4: Setting Powered False
13:16:10.752107 bus /gobi_1: Setting Powered False
13:16:11.858748 bus /gobi_1: Setting Powered False
13:16:12.966564 bus /gobi_0: Setting Powered False
13:16:14.071787 bus /gobi_0: Setting Powered False
13:16:15.184377 bus /gobi_6: Setting Powered False
220gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=26028): Process ended prematurely: osmo-stp_10.42.42.5(pid=26028) [trial-2405↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26028)]2186ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=26028): Process ended prematurely: osmo-stp_10.42.42.5(pid=26028) [trial-2405↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26028)]
----------------------------------------------
trial-2405 gprs:trx-b200 ping.py
----------------------------------------------
13:16:16.254144 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:16.266713 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
13:16:16.409293 tst ping.py: using LAC 41151
13:16:16.535881 tst ping.py: using RAC 96
13:16:16.691870 tst ping.py: using CellId 41151
13:16:16.850738 tst ping.py: using BVCI 41152
13:16:16.885557 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:16.899803 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:16.914774 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:16.932215 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:16.948392 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:16.964312 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:16.981281 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:16.996633 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
13:16:17.314655 tst ping.py:19: start network...
13:16:17.438638 run osmo-hlr_10.42.42.2: Starting osmo-hlr
13:16:17.915850 run create_hlr_db(pid=26022): Launched
13:16:18.086862 bus /gobi_6: Setting Powered False
13:16:19.109167 run create_hlr_db(pid=26022): Terminated: ok {rc=0}
13:16:19.504194 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26024): Launched
13:16:19.800503 run osmo-hlr_10.42.42.2(pid=26025): Launched
13:16:19.908464 run osmo-stp_10.42.42.5: Starting osmo-stp
13:16:20.426261 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26027): Launched
13:16:20.723313 run osmo-stp_10.42.42.5(pid=26028): Launched
13:16:20.832025 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
13:16:21.348895 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26030): Launched
13:16:21.454511 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
13:16:21.713075 run patchelf(pid=26031): Launched
13:16:21.887054 run osmo-stp_10.42.42.5(pid=26028): ERR: Terminated: ERROR {rc=1} [trial-2405↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26028)]
13:16:22.014179 run osmo-stp_10.42.42.5(pid=26028): stdout:
| (launched: 2022-12-14_13:16:20.577208)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:16:22.141879 run osmo-stp_10.42.42.5(pid=26028): stderr:
| �[0;m�[38;5;43m20221214131620748 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131620749 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131620750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131620750 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131620751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131620751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131620751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:16:22.265376 run osmo-stp_10.42.42.5(pid=26028): stdout:
| (launched: 2022-12-14_13:16:20.577208)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:16:22.387926 run osmo-stp_10.42.42.5(pid=26028): stderr:
| �[0;m�[38;5;43m20221214131620748 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131620749 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131620750 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131620750 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131620751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131620751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131620751 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:16:22.530110 run patchelf(pid=26031): Terminating (SIGINT)
13:16:22.696857 run patchelf(pid=26031): Terminated: ok {rc=0}
13:16:22.727016 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=26028): Process ended prematurely: osmo-stp_10.42.42.5(pid=26028) [trial-2405↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26028)] [trial-2405↪gprs:trx-b200↪ping.py:22]
13:16:22.735526 tst ping.py:22: Test FAILED (6.5 sec)
13:16:22.795663 run osmo-hlr_10.42.42.2(pid=26025): ERR: Terminated: ERROR {rc=237} [trial-2405↪gprs:trx-b200↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=26025)]
13:16:22.844570 run osmo-hlr_10.42.42.2(pid=26025): stdout:
| (launched: 2022-12-14_13:16:19.654751)
13:16:22.894334 run osmo-hlr_10.42.42.2(pid=26025): stderr:
| �[0;m�[1;31m20221214131619829 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214131619829 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214131619829 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214131619829 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214131619829 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214131619829 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214131619863 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2405/run.2022-12-14_12-53-19/gprs:trx-b200/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214131619873 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214131619873 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214131619874 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
13:16:22.936830 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26024): Terminating (SIGTERM)
13:16:22.977291 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26027): Terminating (SIGTERM)
13:16:23.017422 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26030): Terminating (SIGTERM)
13:16:23.026621 --- ParallelTerminationStrategy: PID 26024 died...
13:16:23.079867 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26024): Terminated: ok {rc=0}
13:16:23.089827 --- ParallelTerminationStrategy: PID 26027 died...
13:16:23.142973 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26027): Terminated: ok {rc=0}
13:16:23.152471 --- ParallelTerminationStrategy: PID 26030 died...
13:16:23.205418 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26030): Terminated: ok {rc=0}
13:16:23.293699 bus /gobi_6: Setting Powered False
220gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=26038): Process ended prematurely: osmo-stp_10.42.42.5(pid=26038) [trial-2405↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26038)]2186ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=26038): Process ended prematurely: osmo-stp_10.42.42.5(pid=26038) [trial-2405↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26038)]
----------------------------------------------
trial-2405 gprs:trx-b200 ping_idle_ping.py
----------------------------------------------
13:16:24.345949 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:24.357383 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
13:16:24.501069 tst ping_idle_ping.py: using LAC 41152
13:16:24.628381 tst ping_idle_ping.py: using RAC 97
13:16:24.757546 tst ping_idle_ping.py: using CellId 41152
13:16:24.884585 tst ping_idle_ping.py: using BVCI 41153
13:16:24.909407 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:24.920634 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:24.932191 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:24.943322 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:24.954914 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:24.966316 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:24.977930 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
13:16:24.989416 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
13:16:25.261773 tst ping_idle_ping.py:24: start network...
13:16:25.385131 run osmo-hlr_10.42.42.2: Starting osmo-hlr
13:16:25.866079 run create_hlr_db(pid=26032): Launched
13:16:26.037766 bus /gobi_6: Setting Powered False
13:16:27.060008 run create_hlr_db(pid=26032): Terminated: ok {rc=0}
13:16:27.458192 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26034): Launched
13:16:27.757407 run osmo-hlr_10.42.42.2(pid=26035): Launched
13:16:27.865806 run osmo-stp_10.42.42.5: Starting osmo-stp
13:16:28.380181 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26037): Launched
13:16:28.677583 run osmo-stp_10.42.42.5(pid=26038): Launched
13:16:28.786193 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
13:16:29.304910 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26040): Launched
13:16:29.410665 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
13:16:29.669145 run patchelf(pid=26041): Launched
13:16:29.842784 run osmo-stp_10.42.42.5(pid=26038): ERR: Terminated: ERROR {rc=1} [trial-2405↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26038)]
13:16:29.969581 run osmo-stp_10.42.42.5(pid=26038): stdout:
| (launched: 2022-12-14_13:16:28.530712)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:16:30.096784 run osmo-stp_10.42.42.5(pid=26038): stderr:
| �[0;m�[38;5;43m20221214131628700 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131628701 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131628701 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131628701 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131628703 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131628704 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131628704 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:16:30.219830 run osmo-stp_10.42.42.5(pid=26038): stdout:
| (launched: 2022-12-14_13:16:28.530712)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
13:16:30.341926 run osmo-stp_10.42.42.5(pid=26038): stderr:
| �[0;m�[38;5;43m20221214131628700 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214131628701 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214131628701 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214131628701 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214131628703 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214131628704 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214131628704 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
13:16:30.483342 run patchelf(pid=26041): Terminating (SIGINT)
13:16:30.650296 run patchelf(pid=26041): Terminated: ok {rc=0}
13:16:30.680612 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=26038): Process ended prematurely: osmo-stp_10.42.42.5(pid=26038) [trial-2405↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=26038)] [trial-2405↪gprs:trx-b200↪ping_idle_ping.py:27]
13:16:30.689324 tst ping_idle_ping.py:27: Test FAILED (6.3 sec)
13:16:30.748543 run osmo-hlr_10.42.42.2(pid=26035): ERR: Terminated: ERROR {rc=237} [trial-2405↪gprs:trx-b200↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=26035)]
13:16:30.797344 run osmo-hlr_10.42.42.2(pid=26035): stdout:
| (launched: 2022-12-14_13:16:27.609662)
13:16:30.852081 run osmo-hlr_10.42.42.2(pid=26035): stderr:
| �[0;m�[1;31m20221214131627788 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214131627789 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214131627789 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214131627789 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214131627789 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214131627789 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214131627801 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2405/run.2022-12-14_12-53-19/gprs:trx-b200/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214131627810 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214131627810 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214131627810 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
13:16:30.911217 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26034): Terminating (SIGTERM)
13:16:30.951413 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26037): Terminating (SIGTERM)
13:16:30.993561 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26040): Terminating (SIGTERM)
13:16:31.004053 --- ParallelTerminationStrategy: PID 26034 died...
13:16:31.059785 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26034): Terminated: ok {rc=0}
13:16:31.069748 --- ParallelTerminationStrategy: PID 26037 died...
13:16:31.123314 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26037): Terminated: ok {rc=0}
13:16:31.133316 --- ParallelTerminationStrategy: PID 26040 died...
13:16:31.187223 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26040): Terminated: ok {rc=0}
13:16:31.271627 bus /gobi_6: Setting Powered False
5gprs:trx-b20000