37gprs:trx-b2009.0osmo-stp_10.42.42.5(pid=21257): Process ended prematurely: osmo-stp_10.42.42.5(pid=21257) [trial-2354↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21257)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21257): Process ended prematurely: osmo-stp_10.42.42.5(pid=21257) [trial-2354↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21257)]
----------------------------------------------
trial-2354 gprs:trx-b200 cs_paging_gprs_active.py
----------------------------------------------
03:57:27.197746 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:27.209058 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
03:57:27.352283 tst cs_paging_gprs_active.py: using LAC 18906
03:57:27.480588 tst cs_paging_gprs_active.py: using RAC 36
03:57:27.609711 tst cs_paging_gprs_active.py: using CellId 18906
03:57:27.736517 tst cs_paging_gprs_active.py: using BVCI 18907
03:57:27.761511 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:27.772470 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:27.783339 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:27.794385 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:27.805854 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:27.816725 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:27.827969 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:27.839185 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:27.850470 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
03:57:27.969903 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
03:57:28.264877 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
03:57:28.401355 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:57:28.737394 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21247): Launched
03:57:29.018654 run iperf3-srv_10.42.42.10(pid=21248): Launched
03:57:29.218469 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
03:57:29.354516 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:57:29.690148 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21249): Launched
03:57:29.970096 run iperf3-srv_10.42.42.10(pid=21250): Launched
03:57:30.169863 tst cs_paging_gprs_active.py:38: start network...
03:57:30.306149 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:57:30.819516 run create_hlr_db(pid=21251): Launched
03:57:31.020289 bus /sierra_2: Setting Powered False
03:57:32.034851 run create_hlr_db(pid=21251): Terminated: ok {rc=0}
03:57:32.442697 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21253): Launched
03:57:32.748463 run osmo-hlr_10.42.42.2(pid=21254): Launched
03:57:32.861286 run osmo-stp_10.42.42.5: Starting osmo-stp
03:57:33.387474 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21256): Launched
03:57:33.690210 run osmo-stp_10.42.42.5(pid=21257): Launched
03:57:33.799209 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:57:34.331410 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21259): Launched
03:57:34.440291 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:57:34.705529 run patchelf(pid=21260): Launched
03:57:34.882956 run osmo-stp_10.42.42.5(pid=21257): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21257)]
03:57:35.013205 run osmo-stp_10.42.42.5(pid=21257): stdout:
| (launched: 2022-12-09_03:57:33.541439)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:57:35.143026 run osmo-stp_10.42.42.5(pid=21257): stderr:
| �[0;m�[38;5;43m20221209035733714 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035733714 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035733714 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035733715 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035733715 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035733715 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035733715 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:57:35.269652 run osmo-stp_10.42.42.5(pid=21257): stdout:
| (launched: 2022-12-09_03:57:33.541439)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:57:35.394209 run osmo-stp_10.42.42.5(pid=21257): stderr:
| �[0;m�[38;5;43m20221209035733714 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035733714 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035733714 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035733715 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035733715 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035733715 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035733715 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:57:35.539925 run patchelf(pid=21260): Terminating (SIGINT)
03:57:35.711392 run patchelf(pid=21260): Terminated: ok {rc=0}
03:57:35.743043 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=21257): Process ended prematurely: osmo-stp_10.42.42.5(pid=21257) [trial-2354↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21257)] [trial-2354↪gprs:trx-b200↪cs_paging_gprs_active.py:38]
03:57:35.752299 tst cs_paging_gprs_active.py:38: Test FAILED (8.6 sec)
03:57:35.813228 run osmo-hlr_10.42.42.2(pid=21254): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:trx-b200↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21254)]
03:57:35.863199 run osmo-hlr_10.42.42.2(pid=21254): stdout:
| (launched: 2022-12-09_03:57:32.596429)
03:57:35.913464 run osmo-hlr_10.42.42.2(pid=21254): stderr:
| �[0;m�[1;31m20221209035732773 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209035732773 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209035732773 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209035732773 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209035732773 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209035732773 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209035732785 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:trx-b200/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209035732793 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209035732794 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209035732794 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:57:35.955957 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21247): Terminating (SIGTERM)
03:57:35.995954 run iperf3-srv_10.42.42.10(pid=21248): Terminating (SIGTERM)
03:57:36.036150 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21249): Terminating (SIGTERM)
03:57:36.076192 run iperf3-srv_10.42.42.10(pid=21250): Terminating (SIGTERM)
03:57:36.116335 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21253): Terminating (SIGTERM)
03:57:36.156491 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21256): Terminating (SIGTERM)
03:57:36.196738 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21259): Terminating (SIGTERM)
03:57:36.206427 --- ParallelTerminationStrategy: PID 21247 died...
03:57:36.260281 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21247): Terminated: ok {rc=0}
03:57:36.269927 --- ParallelTerminationStrategy: PID 21248 died...
03:57:36.322942 run iperf3-srv_10.42.42.10(pid=21248): Terminated {rc=256}
03:57:36.332435 --- ParallelTerminationStrategy: PID 21249 died...
03:57:36.385588 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21249): Terminated: ok {rc=0}
03:57:36.395100 --- ParallelTerminationStrategy: PID 21250 died...
03:57:36.448300 run iperf3-srv_10.42.42.10(pid=21250): Terminated {rc=256}
03:57:36.457876 --- ParallelTerminationStrategy: PID 21253 died...
03:57:36.511256 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21253): Terminated: ok {rc=0}
03:57:36.520748 --- ParallelTerminationStrategy: PID 21256 died...
03:57:36.574077 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21256): Terminated: ok {rc=0}
03:57:36.583639 --- ParallelTerminationStrategy: PID 21259 died...
03:57:36.636828 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21259): Terminated: ok {rc=0}
03:57:36.719841 bus /gobi_0: Setting Powered False
03:57:37.828536 bus /gobi_0: Setting Powered False
03:57:38.937586 bus /gobi_6: Setting Powered False
37gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=21273): Process ended prematurely: osmo-stp_10.42.42.5(pid=21273) [trial-2354↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21273)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21273): Process ended prematurely: osmo-stp_10.42.42.5(pid=21273) [trial-2354↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21273)]
----------------------------------------------
trial-2354 gprs:trx-b200 iperf3.py
----------------------------------------------
03:57:40.052196 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:40.063879 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
03:57:40.211642 tst iperf3.py: using LAC 18907
03:57:40.337981 tst iperf3.py: using RAC 37
03:57:40.464522 tst iperf3.py: using CellId 18907
03:57:40.603769 tst iperf3.py: using BVCI 18908
03:57:40.628612 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:40.639792 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:40.650720 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:40.661906 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:40.673019 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:40.684036 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:40.695377 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:40.706431 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:40.717728 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
03:57:40.995295 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
03:57:41.121206 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:57:41.435432 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21265): Launched
03:57:41.698158 run iperf3-srv_10.42.42.10(pid=21266): Launched
03:57:41.882704 tst iperf3.py:8: start network...
03:57:42.009121 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:57:42.498416 run create_hlr_db(pid=21267): Launched
03:57:42.670958 bus /gobi_6: Setting Powered False
03:57:43.693795 run create_hlr_db(pid=21267): Terminated: ok {rc=0}
03:57:44.126284 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21269): Launched
03:57:44.453754 run osmo-hlr_10.42.42.2(pid=21270): Launched
03:57:44.571881 run osmo-stp_10.42.42.5: Starting osmo-stp
03:57:45.139535 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21272): Launched
03:57:45.466996 run osmo-stp_10.42.42.5(pid=21273): Launched
03:57:45.586189 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:57:46.180826 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21275): Launched
03:57:46.294352 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:57:46.584472 run patchelf(pid=21276): Launched
03:57:46.767082 run osmo-stp_10.42.42.5(pid=21273): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21273)]
03:57:46.901429 run osmo-stp_10.42.42.5(pid=21273): stdout:
| (launched: 2022-12-09_03:57:45.305586)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:57:47.036221 run osmo-stp_10.42.42.5(pid=21273): stderr:
| �[0;m�[38;5;43m20221209035745500 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035745501 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035745501 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035745501 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035745502 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035745502 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035745502 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:57:47.164274 run osmo-stp_10.42.42.5(pid=21273): stdout:
| (launched: 2022-12-09_03:57:45.305586)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:57:47.288797 run osmo-stp_10.42.42.5(pid=21273): stderr:
| �[0;m�[38;5;43m20221209035745500 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035745501 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035745501 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035745501 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035745502 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035745502 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035745502 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:57:47.434667 run patchelf(pid=21276): Terminating (SIGINT)
03:57:47.606088 run patchelf(pid=21276): Terminated: ok {rc=0}
03:57:47.637820 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=21273): Process ended prematurely: osmo-stp_10.42.42.5(pid=21273) [trial-2354↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21273)] [trial-2354↪gprs:trx-b200↪iperf3.py:8]
03:57:47.647277 tst iperf3.py:8: Test FAILED (7.6 sec)
03:57:47.708201 run osmo-hlr_10.42.42.2(pid=21270): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:trx-b200↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21270)]
03:57:47.757659 run osmo-hlr_10.42.42.2(pid=21270): stdout:
| (launched: 2022-12-09_03:57:44.291494)
03:57:47.807872 run osmo-hlr_10.42.42.2(pid=21270): stderr:
| �[0;m�[1;31m20221209035744489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209035744489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209035744489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209035744489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209035744489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209035744489 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209035744502 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:trx-b200/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209035744511 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209035744511 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209035744511 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:57:47.850850 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21265): Terminating (SIGTERM)
03:57:47.891095 run iperf3-srv_10.42.42.10(pid=21266): Terminating (SIGTERM)
03:57:47.931438 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21269): Terminating (SIGTERM)
03:57:47.971725 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21272): Terminating (SIGTERM)
03:57:48.012250 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21275): Terminating (SIGTERM)
03:57:48.021670 --- ParallelTerminationStrategy: PID 21265 died...
03:57:48.075795 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21265): Terminated: ok {rc=0}
03:57:48.085820 --- ParallelTerminationStrategy: PID 21266 died...
03:57:48.139481 run iperf3-srv_10.42.42.10(pid=21266): Terminated {rc=256}
03:57:48.149137 --- ParallelTerminationStrategy: PID 21269 died...
03:57:48.202676 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21269): Terminated: ok {rc=0}
03:57:48.212289 --- ParallelTerminationStrategy: PID 21272 died...
03:57:48.265973 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21272): Terminated: ok {rc=0}
03:57:48.275629 --- ParallelTerminationStrategy: PID 21275 died...
03:57:48.329877 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21275): Terminated: ok {rc=0}
03:57:48.414263 bus /gobi_6: Setting Powered False
169gprs:trx-b20011.0osmo-stp_10.42.42.5(pid=21291): Process ended prematurely: osmo-stp_10.42.42.5(pid=21291) [trial-2354↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21291)]2186iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21291): Process ended prematurely: osmo-stp_10.42.42.5(pid=21291) [trial-2354↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21291)]
----------------------------------------------
trial-2354 gprs:trx-b200 iperf3m4.py
----------------------------------------------
03:57:49.530539 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:49.542273 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
03:57:49.684345 tst iperf3m4.py: using LAC 18908
03:57:49.810421 tst iperf3m4.py: using RAC 38
03:57:49.936365 tst iperf3m4.py: using CellId 18908
03:57:50.062288 tst iperf3m4.py: using BVCI 18909
03:57:50.087069 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:50.098305 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:50.109266 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:50.120394 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:50.131552 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:50.142621 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:50.153996 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:50.165415 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:57:50.176661 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
03:57:50.293530 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
03:57:50.416468 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
03:57:50.538266 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
03:57:50.834423 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
03:57:50.971384 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:57:51.309933 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21277): Launched
03:57:51.591608 run iperf3-srv_10.42.42.10(pid=21278): Launched
03:57:51.790980 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
03:57:51.927729 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:57:52.266493 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21279): Launched
03:57:52.546831 run iperf3-srv_10.42.42.10(pid=21280): Launched
03:57:52.746311 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
03:57:52.882611 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:57:53.219987 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21281): Launched
03:57:53.501379 run iperf3-srv_10.42.42.10(pid=21282): Launched
03:57:53.700184 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
03:57:53.836250 run iperf3-srv_10.42.42.10: Starting iperf3-srv
03:57:54.174453 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21283): Launched
03:57:54.455794 run iperf3-srv_10.42.42.10(pid=21284): Launched
03:57:54.654448 tst iperf3m4.py:8: start network...
03:57:54.790194 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:57:55.302310 run create_hlr_db(pid=21285): Launched
03:57:55.490160 bus /gobi_6: Setting Powered False
03:57:56.512160 run create_hlr_db(pid=21285): Terminated: ok {rc=0}
03:57:56.920264 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21287): Launched
03:57:57.224680 run osmo-hlr_10.42.42.2(pid=21288): Launched
03:57:57.336500 run osmo-stp_10.42.42.5: Starting osmo-stp
03:57:57.863975 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21290): Launched
03:57:58.167430 run osmo-stp_10.42.42.5(pid=21291): Launched
03:57:58.277749 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:57:58.805584 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21293): Launched
03:57:58.913586 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:57:59.179143 run patchelf(pid=21294): Launched
03:57:59.357490 run osmo-stp_10.42.42.5(pid=21291): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21291)]
03:57:59.487417 run osmo-stp_10.42.42.5(pid=21291): stdout:
| (launched: 2022-12-09_03:57:58.018178)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:57:59.617930 run osmo-stp_10.42.42.5(pid=21291): stderr:
| �[0;m�[38;5;43m20221209035758190 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035758190 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035758190 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035758191 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035758191 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035758191 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035758191 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:57:59.743923 run osmo-stp_10.42.42.5(pid=21291): stdout:
| (launched: 2022-12-09_03:57:58.018178)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:57:59.868903 run osmo-stp_10.42.42.5(pid=21291): stderr:
| �[0;m�[38;5;43m20221209035758190 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035758190 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035758190 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035758191 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035758191 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035758191 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035758191 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:58:00.015015 run patchelf(pid=21294): Terminating (SIGINT)
03:58:00.186491 run patchelf(pid=21294): Terminated: ok {rc=0}
03:58:00.218110 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=21291): Process ended prematurely: osmo-stp_10.42.42.5(pid=21291) [trial-2354↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21291)] [trial-2354↪gprs:trx-b200↪iperf3m4.py:8]
03:58:00.227424 tst iperf3m4.py:8: Test FAILED (10.7 sec)
03:58:00.288172 run osmo-hlr_10.42.42.2(pid=21288): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:trx-b200↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21288)]
03:58:00.337227 run osmo-hlr_10.42.42.2(pid=21288): stdout:
| (launched: 2022-12-09_03:57:57.074509)
03:58:00.386599 run osmo-hlr_10.42.42.2(pid=21288): stderr:
| �[0;m�[1;31m20221209035757253 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209035757253 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209035757253 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209035757254 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209035757254 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209035757254 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209035757269 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:trx-b200/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209035757279 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209035757280 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209035757280 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:58:00.429135 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21277): Terminating (SIGTERM)
03:58:00.468571 run iperf3-srv_10.42.42.10(pid=21278): Terminating (SIGTERM)
03:58:00.508060 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21279): Terminating (SIGTERM)
03:58:00.547483 run iperf3-srv_10.42.42.10(pid=21280): Terminating (SIGTERM)
03:58:00.586955 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21281): Terminating (SIGTERM)
03:58:00.626280 run iperf3-srv_10.42.42.10(pid=21282): Terminating (SIGTERM)
03:58:00.665603 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21283): Terminating (SIGTERM)
03:58:00.704751 run iperf3-srv_10.42.42.10(pid=21284): Terminating (SIGTERM)
03:58:00.744666 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21287): Terminating (SIGTERM)
03:58:00.784035 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21290): Terminating (SIGTERM)
03:58:00.823575 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21293): Terminating (SIGTERM)
03:58:00.832705 --- ParallelTerminationStrategy: PID 21277 died...
03:58:00.896351 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21277): Terminated: ok {rc=0}
03:58:00.907642 --- ParallelTerminationStrategy: PID 21278 died...
03:58:00.961772 run iperf3-srv_10.42.42.10(pid=21278): Terminated {rc=256}
03:58:00.971744 --- ParallelTerminationStrategy: PID 21279 died...
03:58:01.025679 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21279): Terminated: ok {rc=0}
03:58:01.035700 --- ParallelTerminationStrategy: PID 21280 died...
03:58:01.089805 run iperf3-srv_10.42.42.10(pid=21280): Terminated {rc=256}
03:58:01.099760 --- ParallelTerminationStrategy: PID 21281 died...
03:58:01.153807 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21281): Terminated: ok {rc=0}
03:58:01.163753 --- ParallelTerminationStrategy: PID 21282 died...
03:58:01.217725 run iperf3-srv_10.42.42.10(pid=21282): Terminated {rc=256}
03:58:01.227659 --- ParallelTerminationStrategy: PID 21283 died...
03:58:01.281775 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21283): Terminated: ok {rc=0}
03:58:01.291768 --- ParallelTerminationStrategy: PID 21284 died...
03:58:01.345913 run iperf3-srv_10.42.42.10(pid=21284): Terminated {rc=256}
03:58:01.355937 --- ParallelTerminationStrategy: PID 21287 died...
03:58:01.409998 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21287): Terminated: ok {rc=0}
03:58:01.420004 --- ParallelTerminationStrategy: PID 21290 died...
03:58:01.474234 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21290): Terminated: ok {rc=0}
03:58:01.484222 --- ParallelTerminationStrategy: PID 21293 died...
03:58:01.538456 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21293): Terminated: ok {rc=0}
03:58:01.634283 bus /gobi_4: Setting Powered False
03:58:02.737579 bus /gobi_4: Setting Powered False
03:58:03.849977 bus /gobi_1: Setting Powered False
03:58:04.952737 bus /gobi_1: Setting Powered False
03:58:06.059342 bus /gobi_0: Setting Powered False
03:58:07.166672 bus /gobi_0: Setting Powered False
03:58:08.277790 bus /gobi_6: Setting Powered False
169gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=21302): Process ended prematurely: osmo-stp_10.42.42.5(pid=21302) [trial-2354↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21302)]2186ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21302): Process ended prematurely: osmo-stp_10.42.42.5(pid=21302) [trial-2354↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21302)]
----------------------------------------------
trial-2354 gprs:trx-b200 ping.py
----------------------------------------------
03:58:09.343779 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:09.355190 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
03:58:09.491707 tst ping.py: using LAC 18909
03:58:09.611395 tst ping.py: using RAC 39
03:58:09.728880 tst ping.py: using CellId 18909
03:58:09.846811 tst ping.py: using BVCI 18910
03:58:09.869682 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:09.880230 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:09.890745 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:09.901248 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:09.911569 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:09.921979 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:09.932423 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:09.942936 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
03:58:10.220908 tst ping.py:19: start network...
03:58:10.344415 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:58:10.822415 run create_hlr_db(pid=21296): Launched
03:58:10.993134 bus /gobi_6: Setting Powered False
03:58:12.015535 run create_hlr_db(pid=21296): Terminated: ok {rc=0}
03:58:12.440496 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21298): Launched
03:58:12.760739 run osmo-hlr_10.42.42.2(pid=21299): Launched
03:58:12.881168 run osmo-stp_10.42.42.5: Starting osmo-stp
03:58:13.443221 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21301): Launched
03:58:13.764183 run osmo-stp_10.42.42.5(pid=21302): Launched
03:58:13.881012 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:58:14.441539 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21304): Launched
03:58:14.556446 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:58:14.837296 run patchelf(pid=21305): Launched
03:58:15.025028 run osmo-stp_10.42.42.5(pid=21302): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21302)]
03:58:15.161880 run osmo-stp_10.42.42.5(pid=21302): stdout:
| (launched: 2022-12-09_03:58:13.606045)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:58:15.299060 run osmo-stp_10.42.42.5(pid=21302): stderr:
| �[0;m�[38;5;43m20221209035813797 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035813797 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035813798 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035813798 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035813798 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035813799 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035813799 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:58:15.432076 run osmo-stp_10.42.42.5(pid=21302): stdout:
| (launched: 2022-12-09_03:58:13.606045)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:58:15.563997 run osmo-stp_10.42.42.5(pid=21302): stderr:
| �[0;m�[38;5;43m20221209035813797 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035813797 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035813798 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035813798 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035813798 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035813799 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035813799 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:58:15.717743 run patchelf(pid=21305): Terminating (SIGINT)
03:58:15.901544 run patchelf(pid=21305): Terminated: ok {rc=0}
03:58:15.934487 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=21302): Process ended prematurely: osmo-stp_10.42.42.5(pid=21302) [trial-2354↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21302)] [trial-2354↪gprs:trx-b200↪ping.py:22]
03:58:15.943745 tst ping.py:22: Test FAILED (6.6 sec)
03:58:16.006827 run osmo-hlr_10.42.42.2(pid=21299): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:trx-b200↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21299)]
03:58:16.059063 run osmo-hlr_10.42.42.2(pid=21299): stdout:
| (launched: 2022-12-09_03:58:12.603454)
03:58:16.110776 run osmo-hlr_10.42.42.2(pid=21299): stderr:
| �[0;m�[1;31m20221209035812801 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209035812801 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209035812801 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209035812802 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209035812802 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209035812802 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209035812813 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:trx-b200/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209035812823 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209035812823 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209035812823 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:58:16.154141 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21298): Terminating (SIGTERM)
03:58:16.197610 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21301): Terminating (SIGTERM)
03:58:16.241604 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21304): Terminating (SIGTERM)
03:58:16.251516 --- ParallelTerminationStrategy: PID 21298 died...
03:58:16.307579 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21298): Terminated: ok {rc=0}
03:58:16.317578 --- ParallelTerminationStrategy: PID 21301 died...
03:58:16.371945 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21301): Terminated: ok {rc=0}
03:58:16.382409 --- ParallelTerminationStrategy: PID 21304 died...
03:58:16.437889 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21304): Terminated: ok {rc=0}
03:58:16.523292 bus /gobi_6: Setting Powered False
169gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=21312): Process ended prematurely: osmo-stp_10.42.42.5(pid=21312) [trial-2354↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21312)]2186ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=21312): Process ended prematurely: osmo-stp_10.42.42.5(pid=21312) [trial-2354↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21312)]
----------------------------------------------
trial-2354 gprs:trx-b200 ping_idle_ping.py
----------------------------------------------
03:58:17.576204 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:17.588619 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
03:58:17.738814 tst ping_idle_ping.py: using LAC 18910
03:58:17.865604 tst ping_idle_ping.py: using RAC 40
03:58:17.990644 tst ping_idle_ping.py: using CellId 18910
03:58:18.117579 tst ping_idle_ping.py: using BVCI 18911
03:58:18.142241 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:18.153504 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:18.164430 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:18.175795 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:18.187152 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:18.198448 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:18.209896 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
03:58:18.221264 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
03:58:18.497075 tst ping_idle_ping.py:24: start network...
03:58:18.621661 run osmo-hlr_10.42.42.2: Starting osmo-hlr
03:58:19.104592 run create_hlr_db(pid=21306): Launched
03:58:19.275857 bus /gobi_6: Setting Powered False
03:58:20.310581 run create_hlr_db(pid=21306): Terminated: ok {rc=0}
03:58:20.738656 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21308): Launched
03:58:21.061207 run osmo-hlr_10.42.42.2(pid=21309): Launched
03:58:21.178180 run osmo-stp_10.42.42.5: Starting osmo-stp
03:58:21.735626 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21311): Launched
03:58:22.058364 run osmo-stp_10.42.42.5(pid=21312): Launched
03:58:22.175314 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
03:58:22.733834 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21314): Launched
03:58:22.848078 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
03:58:23.127919 run patchelf(pid=21315): Launched
03:58:23.315389 run osmo-stp_10.42.42.5(pid=21312): ERR: Terminated: ERROR {rc=1} [trial-2354↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21312)]
03:58:23.452402 run osmo-stp_10.42.42.5(pid=21312): stdout:
| (launched: 2022-12-09_03:58:21.898339)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:58:23.589830 run osmo-stp_10.42.42.5(pid=21312): stderr:
| �[0;m�[38;5;43m20221209035822093 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035822093 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035822093 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035822094 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035822094 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035822094 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035822094 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:58:23.722582 run osmo-stp_10.42.42.5(pid=21312): stdout:
| (launched: 2022-12-09_03:58:21.898339)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
03:58:23.854409 run osmo-stp_10.42.42.5(pid=21312): stderr:
| �[0;m�[38;5;43m20221209035822093 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209035822093 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209035822093 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209035822094 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209035822094 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209035822094 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209035822094 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
03:58:24.008122 run patchelf(pid=21315): Terminating (SIGINT)
03:58:24.188604 run patchelf(pid=21315): Terminated: ok {rc=0}
03:58:24.221316 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=21312): Process ended prematurely: osmo-stp_10.42.42.5(pid=21312) [trial-2354↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=21312)] [trial-2354↪gprs:trx-b200↪ping_idle_ping.py:27]
03:58:24.230688 tst ping_idle_ping.py:27: Test FAILED (6.6 sec)
03:58:24.295257 run osmo-hlr_10.42.42.2(pid=21309): ERR: Terminated: ERROR {rc=237} [trial-2354↪gprs:trx-b200↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=21309)]
03:58:24.347605 run osmo-hlr_10.42.42.2(pid=21309): stdout:
| (launched: 2022-12-09_03:58:20.901473)
03:58:24.400457 run osmo-hlr_10.42.42.2(pid=21309): stderr:
| �[0;m�[1;31m20221209035821098 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209035821098 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209035821098 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209035821098 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209035821098 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209035821098 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209035821109 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2354/run.2022-12-09_03-35-04/gprs:trx-b200/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209035821118 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209035821118 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209035821118 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
03:58:24.445757 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21308): Terminating (SIGTERM)
03:58:24.488351 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21311): Terminating (SIGTERM)
03:58:24.530625 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21314): Terminating (SIGTERM)
03:58:24.540754 --- ParallelTerminationStrategy: PID 21308 died...
03:58:24.597699 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21308): Terminated: ok {rc=0}
03:58:24.608200 --- ParallelTerminationStrategy: PID 21311 died...
03:58:24.665061 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21311): Terminated: ok {rc=0}
03:58:24.675392 --- ParallelTerminationStrategy: PID 21314 died...
03:58:24.731895 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21314): Terminated: ok {rc=0}
03:58:24.820107 bus /gobi_6: Setting Powered False
5gprs:trx-b20000