63gprs:trx-b2009.0osmo-stp_10.42.42.5(pid=10232): Process ended prematurely: osmo-stp_10.42.42.5(pid=10232) [trial-2380↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10232)]2318cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=10232): Process ended prematurely: osmo-stp_10.42.42.5(pid=10232) [trial-2380↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10232)]
----------------------------------------------
trial-2380 gprs:trx-b200 cs_paging_gprs_active.py
----------------------------------------------
18:03:54.323173 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:03:54.333889 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
18:03:54.471703 tst cs_paging_gprs_active.py: using LAC 30545
18:03:54.592221 tst cs_paging_gprs_active.py: using RAC 200
18:03:54.713183 tst cs_paging_gprs_active.py: using CellId 30545
18:03:54.833251 tst cs_paging_gprs_active.py: using BVCI 30546
18:03:54.856281 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:03:54.866986 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:03:54.877523 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:03:54.888219 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:03:54.898783 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:03:54.909422 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:03:54.920116 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:03:54.930861 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:03:54.941628 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
18:03:55.063959 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
18:03:55.342718 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
18:03:55.469857 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:03:55.787345 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10222): Launched
18:03:56.052633 run iperf3-srv_10.42.42.10(pid=10223): Launched
18:03:56.239019 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
18:03:56.365860 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:03:56.683525 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10224): Launched
18:03:56.946150 run iperf3-srv_10.42.42.10(pid=10225): Launched
18:03:57.131668 tst cs_paging_gprs_active.py:38: start network...
18:03:57.258543 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:03:57.739807 run create_hlr_db(pid=10226): Launched
18:03:57.914269 bus /sierra_2: Setting Powered False
18:03:58.937384 run create_hlr_db(pid=10226): Terminated: ok {rc=0}
18:03:59.343786 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10228): Launched
18:03:59.651085 run osmo-hlr_10.42.42.2(pid=10229): Launched
18:03:59.770052 run osmo-stp_10.42.42.5: Starting osmo-stp
18:04:00.305861 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10231): Launched
18:04:00.611468 run osmo-stp_10.42.42.5(pid=10232): Launched
18:04:00.722635 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:04:01.313348 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10234): Launched
18:04:01.433909 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:04:01.741902 run patchelf(pid=10235): Launched
18:04:01.938093 run osmo-stp_10.42.42.5(pid=10232): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10232)]
18:04:02.081062 run osmo-stp_10.42.42.5(pid=10232): stdout:
| (launched: 2022-12-12_18:04:00.460190)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:02.224754 run osmo-stp_10.42.42.5(pid=10232): stderr:
| �[0;m�[38;5;43m20221212180400630 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180400631 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180400631 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180400631 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180400632 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180400632 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180400632 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:02.360057 run osmo-stp_10.42.42.5(pid=10232): stdout:
| (launched: 2022-12-12_18:04:00.460190)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:02.493527 run osmo-stp_10.42.42.5(pid=10232): stderr:
| �[0;m�[38;5;43m20221212180400630 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180400631 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180400631 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180400631 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180400632 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180400632 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180400632 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:02.648719 run patchelf(pid=10235): Terminating (SIGINT)
18:04:02.829973 run patchelf(pid=10235): Terminated: ok {rc=0}
18:04:02.863796 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=10232): Process ended prematurely: osmo-stp_10.42.42.5(pid=10232) [trial-2380↪gprs:trx-b200↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10232)] [trial-2380↪gprs:trx-b200↪cs_paging_gprs_active.py:38]
18:04:02.874136 tst cs_paging_gprs_active.py:38: Test FAILED (8.6 sec)
18:04:02.937635 run osmo-hlr_10.42.42.2(pid=10229): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-b200↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=10229)]
18:04:02.989299 run osmo-hlr_10.42.42.2(pid=10229): stdout:
| (launched: 2022-12-12_18:03:59.498057)
18:04:03.041570 run osmo-hlr_10.42.42.2(pid=10229): stderr:
| �[0;m�[1;31m20221212180359671 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212180359671 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212180359671 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212180359671 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212180359671 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212180359671 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212180359683 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-b200/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212180359692 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212180359692 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212180359692 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:04:03.086317 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10222): Terminating (SIGTERM)
18:04:03.128648 run iperf3-srv_10.42.42.10(pid=10223): Terminating (SIGTERM)
18:04:03.172314 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10224): Terminating (SIGTERM)
18:04:03.214662 run iperf3-srv_10.42.42.10(pid=10225): Terminating (SIGTERM)
18:04:03.257425 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10228): Terminating (SIGTERM)
18:04:03.299395 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10231): Terminating (SIGTERM)
18:04:03.341378 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10234): Terminating (SIGTERM)
18:04:03.351140 --- ParallelTerminationStrategy: PID 10222 died...
18:04:03.407509 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10222): Terminated: ok {rc=0}
18:04:03.417770 --- ParallelTerminationStrategy: PID 10223 died...
18:04:03.473399 run iperf3-srv_10.42.42.10(pid=10223): Terminated {rc=256}
18:04:03.483355 --- ParallelTerminationStrategy: PID 10224 died...
18:04:03.539390 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10224): Terminated: ok {rc=0}
18:04:03.549574 --- ParallelTerminationStrategy: PID 10225 died...
18:04:03.605192 run iperf3-srv_10.42.42.10(pid=10225): Terminated {rc=256}
18:04:03.615184 --- ParallelTerminationStrategy: PID 10228 died...
18:04:03.671284 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10228): Terminated: ok {rc=0}
18:04:03.681384 --- ParallelTerminationStrategy: PID 10231 died...
18:04:03.737408 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10231): Terminated: ok {rc=0}
18:04:03.747433 --- ParallelTerminationStrategy: PID 10234 died...
18:04:03.803365 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10234): Terminated: ok {rc=0}
18:04:03.891347 bus /gobi_0: Setting Powered False
18:04:04.996040 bus /gobi_0: Setting Powered False
18:04:06.106835 bus /gobi_6: Setting Powered False
63gprs:trx-b2008.0osmo-stp_10.42.42.5(pid=10244): Process ended prematurely: osmo-stp_10.42.42.5(pid=10244) [trial-2380↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10244)]2318iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=10244): Process ended prematurely: osmo-stp_10.42.42.5(pid=10244) [trial-2380↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10244)]
----------------------------------------------
trial-2380 gprs:trx-b200 iperf3.py
----------------------------------------------
18:04:07.222592 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:07.234721 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
18:04:07.383162 tst iperf3.py: using LAC 30546
18:04:07.510093 tst iperf3.py: using RAC 201
18:04:07.642485 tst iperf3.py: using CellId 30546
18:04:07.773723 tst iperf3.py: using BVCI 30547
18:04:07.798556 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:07.809816 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:07.820717 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:07.831687 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:07.842522 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:07.853377 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:07.864435 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:07.875242 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:07.886303 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
18:04:08.162788 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
18:04:08.288933 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:04:08.605493 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10236): Launched
18:04:08.867672 run iperf3-srv_10.42.42.10(pid=10237): Launched
18:04:09.052462 tst iperf3.py:8: start network...
18:04:09.178575 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:04:09.658582 run create_hlr_db(pid=10238): Launched
18:04:09.832530 bus /gobi_6: Setting Powered False
18:04:10.854314 run create_hlr_db(pid=10238): Terminated: ok {rc=0}
18:04:11.284542 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10240): Launched
18:04:11.609083 run osmo-hlr_10.42.42.2(pid=10241): Launched
18:04:11.726996 run osmo-stp_10.42.42.5: Starting osmo-stp
18:04:12.294152 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10243): Launched
18:04:12.620676 run osmo-stp_10.42.42.5(pid=10244): Launched
18:04:12.737995 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:04:13.304081 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10246): Launched
18:04:13.419600 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:04:13.704530 run patchelf(pid=10247): Launched
18:04:13.894162 run osmo-stp_10.42.42.5(pid=10244): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10244)]
18:04:14.032647 run osmo-stp_10.42.42.5(pid=10244): stdout:
| (launched: 2022-12-12_18:04:12.459310)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:14.171275 run osmo-stp_10.42.42.5(pid=10244): stderr:
| �[0;m�[38;5;43m20221212180412649 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180412649 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180412649 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180412650 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180412650 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180412650 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180412650 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:14.306041 run osmo-stp_10.42.42.5(pid=10244): stdout:
| (launched: 2022-12-12_18:04:12.459310)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:14.439129 run osmo-stp_10.42.42.5(pid=10244): stderr:
| �[0;m�[38;5;43m20221212180412649 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180412649 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180412649 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180412650 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180412650 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180412650 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180412650 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:14.594647 run patchelf(pid=10247): Terminating (SIGINT)
18:04:14.775947 run patchelf(pid=10247): Terminated: ok {rc=0}
18:04:14.809631 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=10244): Process ended prematurely: osmo-stp_10.42.42.5(pid=10244) [trial-2380↪gprs:trx-b200↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10244)] [trial-2380↪gprs:trx-b200↪iperf3.py:8]
18:04:14.819524 tst iperf3.py:8: Test FAILED (7.6 sec)
18:04:14.882156 run osmo-hlr_10.42.42.2(pid=10241): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-b200↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=10241)]
18:04:14.933050 run osmo-hlr_10.42.42.2(pid=10241): stdout:
| (launched: 2022-12-12_18:04:11.449283)
18:04:14.984379 run osmo-hlr_10.42.42.2(pid=10241): stderr:
| �[0;m�[1;31m20221212180411644 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212180411644 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212180411644 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212180411644 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212180411644 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212180411644 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212180411656 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-b200/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212180411666 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212180411666 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212180411666 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:04:15.028564 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10236): Terminating (SIGTERM)
18:04:15.070122 run iperf3-srv_10.42.42.10(pid=10237): Terminating (SIGTERM)
18:04:15.111800 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10240): Terminating (SIGTERM)
18:04:15.153512 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10243): Terminating (SIGTERM)
18:04:15.195197 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10246): Terminating (SIGTERM)
18:04:15.204880 --- ParallelTerminationStrategy: PID 10236 died...
18:04:15.260995 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10236): Terminated: ok {rc=0}
18:04:15.271314 --- ParallelTerminationStrategy: PID 10237 died...
18:04:15.326788 run iperf3-srv_10.42.42.10(pid=10237): Terminated {rc=256}
18:04:15.336687 --- ParallelTerminationStrategy: PID 10240 died...
18:04:15.391972 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10240): Terminated: ok {rc=0}
18:04:15.402060 --- ParallelTerminationStrategy: PID 10243 died...
18:04:15.457578 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10243): Terminated: ok {rc=0}
18:04:15.467658 --- ParallelTerminationStrategy: PID 10246 died...
18:04:15.523162 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10246): Terminated: ok {rc=0}
18:04:15.610383 bus /gobi_6: Setting Powered False
195gprs:trx-b20011.0osmo-stp_10.42.42.5(pid=10262): Process ended prematurely: osmo-stp_10.42.42.5(pid=10262) [trial-2380↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10262)]2186iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=10262): Process ended prematurely: osmo-stp_10.42.42.5(pid=10262) [trial-2380↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10262)]
----------------------------------------------
trial-2380 gprs:trx-b200 iperf3m4.py
----------------------------------------------
18:04:16.757881 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:16.774682 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
18:04:16.957383 tst iperf3m4.py: using LAC 30547
18:04:17.100265 tst iperf3m4.py: using RAC 202
18:04:17.228160 tst iperf3m4.py: using CellId 30547
18:04:17.354692 tst iperf3m4.py: using BVCI 30548
18:04:17.379591 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:17.391072 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:17.402172 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:17.413138 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:17.423898 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:17.434722 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:17.445849 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:17.456674 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:17.467721 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
18:04:17.586303 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
18:04:17.706675 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
18:04:17.827094 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
18:04:18.108816 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
18:04:18.236291 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:04:18.553765 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10248): Launched
18:04:18.817209 run iperf3-srv_10.42.42.10(pid=10249): Launched
18:04:19.002874 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
18:04:19.129925 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:04:19.448732 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10250): Launched
18:04:19.713086 run iperf3-srv_10.42.42.10(pid=10251): Launched
18:04:19.898442 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
18:04:20.024832 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:04:20.344145 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10252): Launched
18:04:20.609946 run iperf3-srv_10.42.42.10(pid=10253): Launched
18:04:20.795281 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
18:04:20.921735 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:04:21.237797 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10254): Launched
18:04:21.500531 run iperf3-srv_10.42.42.10(pid=10255): Launched
18:04:21.685865 tst iperf3m4.py:8: start network...
18:04:21.812648 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:04:22.309312 run create_hlr_db(pid=10256): Launched
18:04:22.498556 bus /gobi_6: Setting Powered False
18:04:23.520450 run create_hlr_db(pid=10256): Terminated: ok {rc=0}
18:04:23.955878 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10258): Launched
18:04:24.283872 run osmo-hlr_10.42.42.2(pid=10259): Launched
18:04:24.404177 run osmo-stp_10.42.42.5: Starting osmo-stp
18:04:24.978332 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10261): Launched
18:04:25.307882 run osmo-stp_10.42.42.5(pid=10262): Launched
18:04:25.426048 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:04:25.995724 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10264): Launched
18:04:26.112505 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:04:26.401441 run patchelf(pid=10265): Launched
18:04:26.593456 run osmo-stp_10.42.42.5(pid=10262): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10262)]
18:04:26.734074 run osmo-stp_10.42.42.5(pid=10262): stdout:
| (launched: 2022-12-12_18:04:25.145188)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:26.874170 run osmo-stp_10.42.42.5(pid=10262): stderr:
| �[0;m�[38;5;43m20221212180425334 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180425334 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180425334 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180425334 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180425335 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180425335 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180425335 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:27.009352 run osmo-stp_10.42.42.5(pid=10262): stdout:
| (launched: 2022-12-12_18:04:25.145188)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:27.143537 run osmo-stp_10.42.42.5(pid=10262): stderr:
| �[0;m�[38;5;43m20221212180425334 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180425334 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180425334 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180425334 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180425335 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180425335 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180425335 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:27.300615 run patchelf(pid=10265): Terminating (SIGINT)
18:04:27.484140 run patchelf(pid=10265): Terminated: ok {rc=0}
18:04:27.517935 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=10262): Process ended prematurely: osmo-stp_10.42.42.5(pid=10262) [trial-2380↪gprs:trx-b200↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10262)] [trial-2380↪gprs:trx-b200↪iperf3m4.py:8]
18:04:27.527922 tst iperf3m4.py:8: Test FAILED (10.8 sec)
18:04:27.590711 run osmo-hlr_10.42.42.2(pid=10259): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-b200↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=10259)]
18:04:27.641679 run osmo-hlr_10.42.42.2(pid=10259): stdout:
| (launched: 2022-12-12_18:04:24.122485)
18:04:27.693135 run osmo-hlr_10.42.42.2(pid=10259): stderr:
| �[0;m�[1;31m20221212180424318 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212180424318 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212180424318 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212180424318 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212180424318 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212180424319 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212180424331 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-b200/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212180424340 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212180424341 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212180424341 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:04:27.737009 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10248): Terminating (SIGTERM)
18:04:27.778264 run iperf3-srv_10.42.42.10(pid=10249): Terminating (SIGTERM)
18:04:27.820127 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10250): Terminating (SIGTERM)
18:04:27.862244 run iperf3-srv_10.42.42.10(pid=10251): Terminating (SIGTERM)
18:04:27.903702 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10252): Terminating (SIGTERM)
18:04:27.945138 run iperf3-srv_10.42.42.10(pid=10253): Terminating (SIGTERM)
18:04:27.986634 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10254): Terminating (SIGTERM)
18:04:28.028549 run iperf3-srv_10.42.42.10(pid=10255): Terminating (SIGTERM)
18:04:28.070223 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10258): Terminating (SIGTERM)
18:04:28.111923 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10261): Terminating (SIGTERM)
18:04:28.153864 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10264): Terminating (SIGTERM)
18:04:28.163651 --- ParallelTerminationStrategy: PID 10248 died...
18:04:28.220165 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10248): Terminated: ok {rc=0}
18:04:28.230630 --- ParallelTerminationStrategy: PID 10249 died...
18:04:28.286702 run iperf3-srv_10.42.42.10(pid=10249): Terminated {rc=256}
18:04:28.296728 --- ParallelTerminationStrategy: PID 10250 died...
18:04:28.352812 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10250): Terminated: ok {rc=0}
18:04:28.362861 --- ParallelTerminationStrategy: PID 10251 died...
18:04:28.418746 run iperf3-srv_10.42.42.10(pid=10251): Terminated {rc=256}
18:04:28.428815 --- ParallelTerminationStrategy: PID 10252 died...
18:04:28.485110 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10252): Terminated: ok {rc=0}
18:04:28.495189 --- ParallelTerminationStrategy: PID 10253 died...
18:04:28.551041 run iperf3-srv_10.42.42.10(pid=10253): Terminated {rc=256}
18:04:28.561110 --- ParallelTerminationStrategy: PID 10254 died...
18:04:28.617115 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=10254): Terminated: ok {rc=0}
18:04:28.627457 --- ParallelTerminationStrategy: PID 10255 died...
18:04:28.683516 run iperf3-srv_10.42.42.10(pid=10255): Terminated {rc=256}
18:04:28.693723 --- ParallelTerminationStrategy: PID 10258 died...
18:04:28.749635 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10258): Terminated: ok {rc=0}
18:04:28.759747 --- ParallelTerminationStrategy: PID 10261 died...
18:04:28.815987 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10261): Terminated: ok {rc=0}
18:04:28.826169 --- ParallelTerminationStrategy: PID 10264 died...
18:04:28.882574 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10264): Terminated: ok {rc=0}
18:04:28.970905 bus /gobi_4: Setting Powered False
18:04:30.076553 bus /gobi_4: Setting Powered False
18:04:31.185599 bus /gobi_1: Setting Powered False
18:04:32.294421 bus /gobi_1: Setting Powered False
18:04:33.402279 bus /gobi_0: Setting Powered False
18:04:34.504668 bus /gobi_0: Setting Powered False
18:04:35.614395 bus /gobi_6: Setting Powered False
195gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=10272): Process ended prematurely: osmo-stp_10.42.42.5(pid=10272) [trial-2380↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10272)]2186ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=10272): Process ended prematurely: osmo-stp_10.42.42.5(pid=10272) [trial-2380↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10272)]
----------------------------------------------
trial-2380 gprs:trx-b200 ping.py
----------------------------------------------
18:04:36.684815 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:36.696923 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
18:04:36.841213 tst ping.py: using LAC 30548
18:04:36.968350 tst ping.py: using RAC 203
18:04:37.094133 tst ping.py: using CellId 30548
18:04:37.221947 tst ping.py: using BVCI 30549
18:04:37.247030 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:37.258240 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:37.269391 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:37.280501 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:37.291419 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:37.302399 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:37.313488 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:37.324637 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
18:04:37.616868 tst ping.py:19: start network...
18:04:37.751150 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:04:38.268772 run create_hlr_db(pid=10266): Launched
18:04:38.455380 bus /gobi_6: Setting Powered False
18:04:39.476341 run create_hlr_db(pid=10266): Terminated: ok {rc=0}
18:04:39.882816 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10268): Launched
18:04:40.186814 run osmo-hlr_10.42.42.2(pid=10269): Launched
18:04:40.300544 run osmo-stp_10.42.42.5: Starting osmo-stp
18:04:40.828192 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10271): Launched
18:04:41.133406 run osmo-stp_10.42.42.5(pid=10272): Launched
18:04:41.243138 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:04:41.770404 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10274): Launched
18:04:41.878027 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:04:42.143422 run patchelf(pid=10275): Launched
18:04:42.320353 run osmo-stp_10.42.42.5(pid=10272): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10272)]
18:04:42.449527 run osmo-stp_10.42.42.5(pid=10272): stdout:
| (launched: 2022-12-12_18:04:40.981303)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:42.578814 run osmo-stp_10.42.42.5(pid=10272): stderr:
| �[0;m�[38;5;43m20221212180441148 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180441148 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180441149 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180441149 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180441149 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180441149 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180441150 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:42.704198 run osmo-stp_10.42.42.5(pid=10272): stdout:
| (launched: 2022-12-12_18:04:40.981303)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:42.828250 run osmo-stp_10.42.42.5(pid=10272): stderr:
| �[0;m�[38;5;43m20221212180441148 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180441148 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180441149 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180441149 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180441149 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180441149 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180441150 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:42.971938 run patchelf(pid=10275): Terminating (SIGINT)
18:04:43.140101 run patchelf(pid=10275): Terminated: ok {rc=0}
18:04:43.170662 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=10272): Process ended prematurely: osmo-stp_10.42.42.5(pid=10272) [trial-2380↪gprs:trx-b200↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10272)] [trial-2380↪gprs:trx-b200↪ping.py:22]
18:04:43.179437 tst ping.py:22: Test FAILED (6.5 sec)
18:04:43.241986 run osmo-hlr_10.42.42.2(pid=10269): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-b200↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=10269)]
18:04:43.291650 run osmo-hlr_10.42.42.2(pid=10269): stdout:
| (launched: 2022-12-12_18:04:40.035325)
18:04:43.342175 run osmo-hlr_10.42.42.2(pid=10269): stderr:
| �[0;m�[1;31m20221212180440210 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212180440210 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212180440210 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212180440210 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212180440210 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212180440210 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212180440222 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-b200/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212180440231 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212180440232 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212180440232 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:04:43.384931 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10268): Terminating (SIGTERM)
18:04:43.425836 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10271): Terminating (SIGTERM)
18:04:43.466628 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10274): Terminating (SIGTERM)
18:04:43.476040 --- ParallelTerminationStrategy: PID 10268 died...
18:04:43.530251 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10268): Terminated: ok {rc=0}
18:04:43.540190 --- ParallelTerminationStrategy: PID 10271 died...
18:04:43.594460 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10271): Terminated: ok {rc=0}
18:04:43.604365 --- ParallelTerminationStrategy: PID 10274 died...
18:04:43.658130 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10274): Terminated: ok {rc=0}
18:04:43.748596 bus /gobi_6: Setting Powered False
195gprs:trx-b2007.0osmo-stp_10.42.42.5(pid=10282): Process ended prematurely: osmo-stp_10.42.42.5(pid=10282) [trial-2380↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10282)]2186ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=10282): Process ended prematurely: osmo-stp_10.42.42.5(pid=10282) [trial-2380↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10282)]
----------------------------------------------
trial-2380 gprs:trx-b200 ping_idle_ping.py
----------------------------------------------
18:04:44.801508 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:44.813520 tst gprs:trx-b200: Using 1 x bts (candidates: 1)
18:04:44.951926 tst ping_idle_ping.py: using LAC 30549
18:04:45.071195 tst ping_idle_ping.py: using RAC 204
18:04:45.198612 tst ping_idle_ping.py: using CellId 30549
18:04:45.326116 tst ping_idle_ping.py: using BVCI 30550
18:04:45.350925 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:45.362159 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:45.373422 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:45.385141 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:45.396636 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:45.408110 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:45.419869 tst gprs:trx-b200: Using 1 x ip_address (candidates: 9)
18:04:45.431674 tst gprs:trx-b200: Using 1 x modem (candidates: 4)
18:04:45.710258 tst ping_idle_ping.py:24: start network...
18:04:45.835696 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:04:46.402787 run create_hlr_db(pid=10276): Launched
18:04:46.580299 bus /gobi_6: Setting Powered False
18:04:47.599035 run create_hlr_db(pid=10276): Terminated: ok {rc=0}
18:04:48.031292 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10278): Launched
18:04:48.357841 run osmo-hlr_10.42.42.2(pid=10279): Launched
18:04:48.475147 run osmo-stp_10.42.42.5: Starting osmo-stp
18:04:49.038291 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10281): Launched
18:04:49.367127 run osmo-stp_10.42.42.5(pid=10282): Launched
18:04:49.483528 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:04:50.047816 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10284): Launched
18:04:50.162571 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:04:50.446083 run patchelf(pid=10285): Launched
18:04:50.634555 run osmo-stp_10.42.42.5(pid=10282): ERR: Terminated: ERROR {rc=1} [trial-2380↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10282)]
18:04:50.772099 run osmo-stp_10.42.42.5(pid=10282): stdout:
| (launched: 2022-12-12_18:04:49.202351)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:50.910597 run osmo-stp_10.42.42.5(pid=10282): stderr:
| �[0;m�[38;5;43m20221212180449393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180449393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180449394 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180449394 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180449395 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180449396 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180449396 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:51.043681 run osmo-stp_10.42.42.5(pid=10282): stdout:
| (launched: 2022-12-12_18:04:49.202351)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
18:04:51.175727 run osmo-stp_10.42.42.5(pid=10282): stderr:
| �[0;m�[38;5;43m20221212180449393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212180449393 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212180449394 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212180449394 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212180449395 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212180449396 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212180449396 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
18:04:51.330167 run patchelf(pid=10285): Terminating (SIGINT)
18:04:51.509689 run patchelf(pid=10285): Terminated: ok {rc=0}
18:04:51.542439 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=10282): Process ended prematurely: osmo-stp_10.42.42.5(pid=10282) [trial-2380↪gprs:trx-b200↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=10282)] [trial-2380↪gprs:trx-b200↪ping_idle_ping.py:27]
18:04:51.551820 tst ping_idle_ping.py:27: Test FAILED (6.7 sec)
18:04:51.614348 run osmo-hlr_10.42.42.2(pid=10279): ERR: Terminated: ERROR {rc=237} [trial-2380↪gprs:trx-b200↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=10279)]
18:04:51.665276 run osmo-hlr_10.42.42.2(pid=10279): stdout:
| (launched: 2022-12-12_18:04:48.195489)
18:04:51.716857 run osmo-hlr_10.42.42.2(pid=10279): stderr:
| �[0;m�[1;31m20221212180448392 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212180448392 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212180448392 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212180448392 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212180448392 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212180448392 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212180448406 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2380/run.2022-12-12_17-41-28/gprs:trx-b200/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212180448414 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212180448414 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212180448415 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
18:04:51.760754 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10278): Terminating (SIGTERM)
18:04:51.802418 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10281): Terminating (SIGTERM)
18:04:51.844576 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10284): Terminating (SIGTERM)
18:04:51.854452 --- ParallelTerminationStrategy: PID 10278 died...
18:04:51.911390 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10278): Terminated: ok {rc=0}
18:04:51.921805 --- ParallelTerminationStrategy: PID 10281 died...
18:04:51.977740 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10281): Terminated: ok {rc=0}
18:04:51.987857 --- ParallelTerminationStrategy: PID 10284 died...
18:04:52.043841 run pcap-recorder_any(filters='host 10.42.42.6')(pid=10284): Terminated: ok {rc=0}
18:04:52.131433 bus /gobi_6: Setting Powered False
5gprs:trx-b20000