244gprs:trx-lms11.0osmo-stp_10.42.42.5(pid=29887): Process ended prematurely: osmo-stp_10.42.42.5(pid=29887) [trial-2382↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29887)]2139cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=29887): Process ended prematurely: osmo-stp_10.42.42.5(pid=29887) [trial-2382↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29887)]
----------------------------------------------
trial-2382 gprs:trx-lms cs_paging_gprs_active.py
----------------------------------------------
21:25:59.651091 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:25:59.663847 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
21:25:59.817867 tst cs_paging_gprs_active.py: using LAC 31605
21:25:59.971258 tst cs_paging_gprs_active.py: using RAC 240
21:26:00.108465 tst cs_paging_gprs_active.py: using CellId 31605
21:26:00.252683 tst cs_paging_gprs_active.py: using BVCI 31606
21:26:00.279852 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:00.291853 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:00.304535 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:00.316466 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:00.329304 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:00.342186 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:00.356049 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:00.367590 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:00.381129 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
21:26:00.508883 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
21:26:00.822672 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
21:26:01.020287 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:26:01.384338 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29877): Launched
21:26:01.675974 run iperf3-srv_10.42.42.10(pid=29878): Launched
21:26:01.884510 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
21:26:02.024602 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:26:02.377071 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29879): Launched
21:26:02.700432 run iperf3-srv_10.42.42.10(pid=29880): Launched
21:26:02.911908 tst cs_paging_gprs_active.py:38: start network...
21:26:03.052833 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:26:03.640133 run create_hlr_db(pid=29881): Launched
21:26:03.841575 bus /sierra_2: Setting Powered False
21:26:04.879444 run create_hlr_db(pid=29881): Terminated: ok {rc=0}
21:26:05.368780 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29883): Launched
21:26:05.816740 run osmo-hlr_10.42.42.2(pid=29884): Launched
21:26:06.067038 run osmo-stp_10.42.42.5: Starting osmo-stp
21:26:06.733026 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29886): Launched
21:26:07.167678 run osmo-stp_10.42.42.5(pid=29887): Launched
21:26:07.387847 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:26:08.074013 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29889): Launched
21:26:08.198817 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:26:08.500497 run patchelf(pid=29890): Launched
21:26:08.700722 run osmo-stp_10.42.42.5(pid=29887): ERR: Terminated: ERROR {rc=1} [trial-2382↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29887)]
21:26:08.837073 run osmo-stp_10.42.42.5(pid=29887): stdout:
| (launched: 2022-12-12_21:26:06.906549)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:26:08.973775 run osmo-stp_10.42.42.5(pid=29887): stderr:
| �[0;m�[38;5;43m20221212212607234 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212607234 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212607234 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212607235 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212607235 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212607235 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212607235 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:26:09.110733 run osmo-stp_10.42.42.5(pid=29887): stdout:
| (launched: 2022-12-12_21:26:06.906549)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:26:09.248088 run osmo-stp_10.42.42.5(pid=29887): stderr:
| �[0;m�[38;5;43m20221212212607234 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212607234 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212607234 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212607235 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212607235 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212607235 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212607235 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:26:09.404111 run patchelf(pid=29890): Terminating (SIGINT)
21:26:09.591002 run patchelf(pid=29890): Terminated: ok {rc=0}
21:26:09.624766 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=29887): Process ended prematurely: osmo-stp_10.42.42.5(pid=29887) [trial-2382↪gprs:trx-lms↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29887)] [trial-2382↪gprs:trx-lms↪cs_paging_gprs_active.py:38]
21:26:09.634849 tst cs_paging_gprs_active.py:38: Test FAILED (10.0 sec)
21:26:09.697724 run osmo-hlr_10.42.42.2(pid=29884): ERR: Terminated: ERROR {rc=237} [trial-2382↪gprs:trx-lms↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=29884)]
21:26:09.749358 run osmo-hlr_10.42.42.2(pid=29884): stdout:
| (launched: 2022-12-12_21:26:05.538341)
21:26:09.801611 run osmo-hlr_10.42.42.2(pid=29884): stderr:
| �[0;m�[1;31m20221212212605840 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212212605840 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212212605849 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212212605849 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212212605849 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212212605849 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212212605874 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2382/run.2022-12-12_20-36-34/gprs:trx-lms/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212212605890 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212212605890 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212212605890 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:26:09.845761 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29877): Terminating (SIGTERM)
21:26:09.888520 run iperf3-srv_10.42.42.10(pid=29878): Terminating (SIGTERM)
21:26:09.932052 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29879): Terminating (SIGTERM)
21:26:09.974309 run iperf3-srv_10.42.42.10(pid=29880): Terminating (SIGTERM)
21:26:10.017278 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29883): Terminating (SIGTERM)
21:26:10.060187 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29886): Terminating (SIGTERM)
21:26:10.104437 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29889): Terminating (SIGTERM)
21:26:10.115841 --- ParallelTerminationStrategy: PID 29877 died...
21:26:10.172934 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29877): Terminated: ok {rc=0}
21:26:10.183485 --- ParallelTerminationStrategy: PID 29878 died...
21:26:10.239323 run iperf3-srv_10.42.42.10(pid=29878): Terminated {rc=256}
21:26:10.249959 --- ParallelTerminationStrategy: PID 29879 died...
21:26:10.305525 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29879): Terminated: ok {rc=0}
21:26:10.316236 --- ParallelTerminationStrategy: PID 29880 died...
21:26:10.371590 run iperf3-srv_10.42.42.10(pid=29880): Terminated {rc=256}
21:26:10.382075 --- ParallelTerminationStrategy: PID 29883 died...
21:26:10.438168 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29883): Terminated: ok {rc=0}
21:26:10.448502 --- ParallelTerminationStrategy: PID 29886 died...
21:26:10.504151 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29886): Terminated: ok {rc=0}
21:26:10.514646 --- ParallelTerminationStrategy: PID 29889 died...
21:26:10.570865 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29889): Terminated: ok {rc=0}
21:26:10.659770 bus /gobi_0: Setting Powered False
21:26:11.790135 bus /gobi_0: Setting Powered False
21:26:12.910283 bus /gobi_6: Setting Powered False
241gprs:trx-lms10.0osmo-stp_10.42.42.5(pid=29899): Process ended prematurely: osmo-stp_10.42.42.5(pid=29899) [trial-2382↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29899)]2142iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=29899): Process ended prematurely: osmo-stp_10.42.42.5(pid=29899) [trial-2382↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29899)]
----------------------------------------------
trial-2382 gprs:trx-lms iperf3.py
----------------------------------------------
21:26:14.031565 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:14.047240 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
21:26:14.247238 tst iperf3.py: using LAC 31606
21:26:14.386994 tst iperf3.py: using RAC 241
21:26:14.544623 tst iperf3.py: using CellId 31606
21:26:14.680046 tst iperf3.py: using BVCI 31607
21:26:14.707364 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:14.719170 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:14.730806 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:14.742784 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:14.754211 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:14.765844 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:14.777823 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:14.790331 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:14.801926 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
21:26:15.104154 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
21:26:15.248110 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:26:15.657380 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29891): Launched
21:26:15.995693 run iperf3-srv_10.42.42.10(pid=29892): Launched
21:26:16.208507 tst iperf3.py:8: start network...
21:26:16.344305 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:26:16.923792 run create_hlr_db(pid=29893): Launched
21:26:17.121507 bus /gobi_6: Setting Powered False
21:26:18.151004 run create_hlr_db(pid=29893): Terminated: ok {rc=0}
21:26:18.686086 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29895): Launched
21:26:19.140632 run osmo-hlr_10.42.42.2(pid=29896): Launched
21:26:19.421282 run osmo-stp_10.42.42.5: Starting osmo-stp
21:26:20.052644 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29898): Launched
21:26:20.483408 run osmo-stp_10.42.42.5(pid=29899): Launched
21:26:20.687673 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:26:21.404706 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29901): Launched
21:26:21.530518 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:26:21.846227 run patchelf(pid=29902): Launched
21:26:22.049534 run osmo-stp_10.42.42.5(pid=29899): ERR: Terminated: ERROR {rc=1} [trial-2382↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29899)]
21:26:22.198441 run osmo-stp_10.42.42.5(pid=29899): stdout:
| (launched: 2022-12-12_21:26:20.230531)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:26:22.347381 run osmo-stp_10.42.42.5(pid=29899): stderr:
| �[0;m�[38;5;43m20221212212620544 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212620546 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212620546 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212620547 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212620548 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212620548 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212620548 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:26:22.491406 run osmo-stp_10.42.42.5(pid=29899): stdout:
| (launched: 2022-12-12_21:26:20.230531)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:26:22.635501 run osmo-stp_10.42.42.5(pid=29899): stderr:
| �[0;m�[38;5;43m20221212212620544 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212620546 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212620546 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212620547 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212620548 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212620548 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212620548 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:26:22.802413 run patchelf(pid=29902): Terminating (SIGINT)
21:26:22.997727 run patchelf(pid=29902): Terminated: ok {rc=0}
21:26:23.032762 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=29899): Process ended prematurely: osmo-stp_10.42.42.5(pid=29899) [trial-2382↪gprs:trx-lms↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29899)] [trial-2382↪gprs:trx-lms↪iperf3.py:8]
21:26:23.043103 tst iperf3.py:8: Test FAILED (9.0 sec)
21:26:23.114072 run osmo-hlr_10.42.42.2(pid=29896): ERR: Terminated: ERROR {rc=237} [trial-2382↪gprs:trx-lms↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=29896)]
21:26:23.170393 run osmo-hlr_10.42.42.2(pid=29896): stdout:
| (launched: 2022-12-12_21:26:18.869717)
21:26:23.227518 run osmo-hlr_10.42.42.2(pid=29896): stderr:
| �[0;m�[1;31m20221212212619180 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212212619180 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212212619180 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212212619180 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212212619180 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212212619180 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212212619201 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2382/run.2022-12-12_20-36-34/gprs:trx-lms/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212212619218 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212212619218 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212212619218 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:26:23.276554 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29891): Terminating (SIGTERM)
21:26:23.322528 run iperf3-srv_10.42.42.10(pid=29892): Terminating (SIGTERM)
21:26:23.367813 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29895): Terminating (SIGTERM)
21:26:23.416408 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29898): Terminating (SIGTERM)
21:26:23.464679 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29901): Terminating (SIGTERM)
21:26:23.475842 --- ParallelTerminationStrategy: PID 29891 died...
21:26:23.539447 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29891): Terminated: ok {rc=0}
21:26:23.551769 --- ParallelTerminationStrategy: PID 29892 died...
21:26:23.613048 run iperf3-srv_10.42.42.10(pid=29892): Terminated {rc=256}
21:26:23.624904 --- ParallelTerminationStrategy: PID 29895 died...
21:26:23.686877 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29895): Terminated: ok {rc=0}
21:26:23.698516 --- ParallelTerminationStrategy: PID 29898 died...
21:26:23.759842 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29898): Terminated: ok {rc=0}
21:26:23.771417 --- ParallelTerminationStrategy: PID 29901 died...
21:26:23.832881 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29901): Terminated: ok {rc=0}
21:26:23.928439 bus /gobi_6: Setting Powered False
244gprs:trx-lms13.0osmo-stp_10.42.42.5(pid=29917): Process ended prematurely: osmo-stp_10.42.42.5(pid=29917) [trial-2382↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29917)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=29917): Process ended prematurely: osmo-stp_10.42.42.5(pid=29917) [trial-2382↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29917)]
----------------------------------------------
trial-2382 gprs:trx-lms iperf3m4.py
----------------------------------------------
21:26:25.054257 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:25.067363 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
21:26:25.234330 tst iperf3m4.py: using LAC 31607
21:26:25.377185 tst iperf3m4.py: using RAC 242
21:26:25.514664 tst iperf3m4.py: using CellId 31607
21:26:25.650449 tst iperf3m4.py: using BVCI 31608
21:26:25.679303 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:25.692095 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:25.705208 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:25.717499 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:25.731472 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:25.744834 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:25.758430 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:25.770531 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:25.783989 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
21:26:25.914005 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
21:26:26.051177 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
21:26:26.211149 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
21:26:26.530387 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
21:26:26.678461 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:26:27.119430 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29903): Launched
21:26:27.438866 run iperf3-srv_10.42.42.10(pid=29904): Launched
21:26:27.644203 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
21:26:27.783438 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:26:28.146814 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29905): Launched
21:26:28.454810 run iperf3-srv_10.42.42.10(pid=29906): Launched
21:26:28.663481 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
21:26:28.806164 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:26:29.166145 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29907): Launched
21:26:29.470837 run iperf3-srv_10.42.42.10(pid=29908): Launched
21:26:29.681667 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
21:26:29.823082 run iperf3-srv_10.42.42.10: Starting iperf3-srv
21:26:30.187550 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29909): Launched
21:26:30.490085 run iperf3-srv_10.42.42.10(pid=29910): Launched
21:26:30.697274 tst iperf3m4.py:8: start network...
21:26:30.845295 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:26:31.426245 run create_hlr_db(pid=29911): Launched
21:26:31.614603 bus /gobi_6: Setting Powered False
21:26:32.690269 run create_hlr_db(pid=29911): Terminated: ok {rc=0}
21:26:33.174562 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29913): Launched
21:26:33.673980 run osmo-hlr_10.42.42.2(pid=29914): Launched
21:26:33.874884 run osmo-stp_10.42.42.5: Starting osmo-stp
21:26:34.543879 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29916): Launched
21:26:34.982347 run osmo-stp_10.42.42.5(pid=29917): Launched
21:26:35.246931 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:26:35.860930 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29919): Launched
21:26:35.986522 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:26:36.300261 run patchelf(pid=29920): Launched
21:26:36.503809 run osmo-stp_10.42.42.5(pid=29917): ERR: Terminated: ERROR {rc=1} [trial-2382↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29917)]
21:26:36.653593 run osmo-stp_10.42.42.5(pid=29917): stdout:
| (launched: 2022-12-12_21:26:34.717797)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:26:36.803065 run osmo-stp_10.42.42.5(pid=29917): stderr:
| �[0;m�[38;5;43m20221212212635060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212635060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212635060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212635061 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212635062 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212635062 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212635062 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:26:36.949332 run osmo-stp_10.42.42.5(pid=29917): stdout:
| (launched: 2022-12-12_21:26:34.717797)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:26:37.092505 run osmo-stp_10.42.42.5(pid=29917): stderr:
| �[0;m�[38;5;43m20221212212635060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212635060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212635060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212635061 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212635062 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212635062 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212635062 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:26:37.259706 run patchelf(pid=29920): Terminating (SIGINT)
21:26:37.455223 run patchelf(pid=29920): Terminated: ok {rc=0}
21:26:37.492363 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=29917): Process ended prematurely: osmo-stp_10.42.42.5(pid=29917) [trial-2382↪gprs:trx-lms↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29917)] [trial-2382↪gprs:trx-lms↪iperf3m4.py:8]
21:26:37.503940 tst iperf3m4.py:8: Test FAILED (12.5 sec)
21:26:37.575578 run osmo-hlr_10.42.42.2(pid=29914): ERR: Terminated: ERROR {rc=237} [trial-2382↪gprs:trx-lms↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=29914)]
21:26:37.631425 run osmo-hlr_10.42.42.2(pid=29914): stdout:
| (launched: 2022-12-12_21:26:33.408736)
21:26:37.688537 run osmo-hlr_10.42.42.2(pid=29914): stderr:
| �[0;m�[1;31m20221212212633726 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212212633726 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212212633726 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212212633726 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212212633726 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212212633726 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212212633752 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2382/run.2022-12-12_20-36-34/gprs:trx-lms/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212212633763 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212212633763 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212212633764 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:26:37.735986 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29903): Terminating (SIGTERM)
21:26:37.784512 run iperf3-srv_10.42.42.10(pid=29904): Terminating (SIGTERM)
21:26:37.830335 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29905): Terminating (SIGTERM)
21:26:37.876805 run iperf3-srv_10.42.42.10(pid=29906): Terminating (SIGTERM)
21:26:37.924327 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29907): Terminating (SIGTERM)
21:26:37.968820 run iperf3-srv_10.42.42.10(pid=29908): Terminating (SIGTERM)
21:26:38.015763 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29909): Terminating (SIGTERM)
21:26:38.061223 run iperf3-srv_10.42.42.10(pid=29910): Terminating (SIGTERM)
21:26:38.105631 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29913): Terminating (SIGTERM)
21:26:38.152606 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29916): Terminating (SIGTERM)
21:26:38.197411 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29919): Terminating (SIGTERM)
21:26:38.209671 --- ParallelTerminationStrategy: PID 29903 died...
21:26:38.271980 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29903): Terminated: ok {rc=0}
21:26:38.283465 --- ParallelTerminationStrategy: PID 29904 died...
21:26:38.344624 run iperf3-srv_10.42.42.10(pid=29904): Terminated {rc=256}
21:26:38.356083 --- ParallelTerminationStrategy: PID 29905 died...
21:26:38.417348 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29905): Terminated: ok {rc=0}
21:26:38.429169 --- ParallelTerminationStrategy: PID 29906 died...
21:26:38.490016 run iperf3-srv_10.42.42.10(pid=29906): Terminated {rc=256}
21:26:38.501417 --- ParallelTerminationStrategy: PID 29907 died...
21:26:38.563810 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29907): Terminated: ok {rc=0}
21:26:38.575205 --- ParallelTerminationStrategy: PID 29908 died...
21:26:38.636204 run iperf3-srv_10.42.42.10(pid=29908): Terminated {rc=256}
21:26:38.647560 --- ParallelTerminationStrategy: PID 29909 died...
21:26:38.710218 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=29909): Terminated: ok {rc=0}
21:26:38.721682 --- ParallelTerminationStrategy: PID 29910 died...
21:26:38.794998 run iperf3-srv_10.42.42.10(pid=29910): Terminated {rc=256}
21:26:38.806560 --- ParallelTerminationStrategy: PID 29913 died...
21:26:38.867599 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29913): Terminated: ok {rc=0}
21:26:38.880202 --- ParallelTerminationStrategy: PID 29916 died...
21:26:38.941809 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29916): Terminated: ok {rc=0}
21:26:38.953311 --- ParallelTerminationStrategy: PID 29919 died...
21:26:39.013510 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29919): Terminated: ok {rc=0}
21:26:39.109888 bus /gobi_4: Setting Powered False
21:26:40.229219 bus /gobi_4: Setting Powered False
21:26:41.370205 bus /gobi_1: Setting Powered False
21:26:42.480657 bus /gobi_1: Setting Powered False
21:26:43.622589 bus /gobi_0: Setting Powered False
21:26:44.734312 bus /gobi_0: Setting Powered False
21:26:45.853964 bus /gobi_6: Setting Powered False
56gprs:trx-lms8.0osmo-stp_10.42.42.5(pid=29927): Process ended prematurely: osmo-stp_10.42.42.5(pid=29927) [trial-2382↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29927)]2327ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=29927): Process ended prematurely: osmo-stp_10.42.42.5(pid=29927) [trial-2382↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29927)]
----------------------------------------------
trial-2382 gprs:trx-lms ping.py
----------------------------------------------
21:26:46.942485 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:46.954401 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
21:26:47.138644 tst ping.py: using LAC 31608
21:26:47.292665 tst ping.py: using RAC 243
21:26:47.447666 tst ping.py: using CellId 31608
21:26:47.603313 tst ping.py: using BVCI 31609
21:26:47.633913 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:47.647197 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:47.660306 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:47.672604 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:47.687086 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:47.699803 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:47.713419 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:47.725446 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
21:26:48.016882 tst ping.py:19: start network...
21:26:48.148281 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:26:48.691889 run create_hlr_db(pid=29921): Launched
21:26:48.875168 bus /gobi_6: Setting Powered False
21:26:49.907414 run create_hlr_db(pid=29921): Terminated: ok {rc=0}
21:26:50.340052 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29923): Launched
21:26:50.753440 run osmo-hlr_10.42.42.2(pid=29924): Launched
21:26:51.010992 run osmo-stp_10.42.42.5: Starting osmo-stp
21:26:51.640309 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29926): Launched
21:26:52.056378 run osmo-stp_10.42.42.5(pid=29927): Launched
21:26:52.295490 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:26:52.977505 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29929): Launched
21:26:53.100860 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:26:53.417918 run patchelf(pid=29930): Launched
21:26:53.604701 run osmo-stp_10.42.42.5(pid=29927): ERR: Terminated: ERROR {rc=1} [trial-2382↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29927)]
21:26:53.739886 run osmo-stp_10.42.42.5(pid=29927): stdout:
| (launched: 2022-12-12_21:26:51.801190)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:26:53.875732 run osmo-stp_10.42.42.5(pid=29927): stderr:
| �[0;m�[38;5;43m20221212212652123 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212652123 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212652124 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212652124 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212652124 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212652129 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212652129 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:26:54.007302 run osmo-stp_10.42.42.5(pid=29927): stdout:
| (launched: 2022-12-12_21:26:51.801190)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:26:54.136409 run osmo-stp_10.42.42.5(pid=29927): stderr:
| �[0;m�[38;5;43m20221212212652123 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212652123 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212652124 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212652124 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212652124 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212652129 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212652129 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:26:54.288107 run patchelf(pid=29930): Terminating (SIGINT)
21:26:54.466576 run patchelf(pid=29930): Terminated: ok {rc=0}
21:26:54.498856 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=29927): Process ended prematurely: osmo-stp_10.42.42.5(pid=29927) [trial-2382↪gprs:trx-lms↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29927)] [trial-2382↪gprs:trx-lms↪ping.py:22]
21:26:54.507830 tst ping.py:22: Test FAILED (7.6 sec)
21:26:54.571104 run osmo-hlr_10.42.42.2(pid=29924): ERR: Terminated: ERROR {rc=237} [trial-2382↪gprs:trx-lms↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=29924)]
21:26:54.622624 run osmo-hlr_10.42.42.2(pid=29924): stdout:
| (launched: 2022-12-12_21:26:50.502484)
21:26:54.674540 run osmo-hlr_10.42.42.2(pid=29924): stderr:
| �[0;m�[1;31m20221212212650841 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212212650841 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212212650841 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212212650841 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212212650841 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212212650841 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212212650860 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2382/run.2022-12-12_20-36-34/gprs:trx-lms/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212212650868 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212212650868 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212212650868 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:26:54.718714 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29923): Terminating (SIGTERM)
21:26:54.762590 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29926): Terminating (SIGTERM)
21:26:54.805741 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29929): Terminating (SIGTERM)
21:26:54.817288 --- ParallelTerminationStrategy: PID 29923 died...
21:26:54.874994 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29923): Terminated: ok {rc=0}
21:26:54.885384 --- ParallelTerminationStrategy: PID 29926 died...
21:26:54.940768 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29926): Terminated: ok {rc=0}
21:26:54.951145 --- ParallelTerminationStrategy: PID 29929 died...
21:26:55.007375 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29929): Terminated: ok {rc=0}
21:26:55.096938 bus /gobi_6: Setting Powered False
63gprs:trx-lms8.0osmo-stp_10.42.42.5(pid=29937): Process ended prematurely: osmo-stp_10.42.42.5(pid=29937) [trial-2382↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29937)]2320ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=29937): Process ended prematurely: osmo-stp_10.42.42.5(pid=29937) [trial-2382↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29937)]
----------------------------------------------
trial-2382 gprs:trx-lms ping_idle_ping.py
----------------------------------------------
21:26:56.164455 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:56.178612 tst gprs:trx-lms: Using 1 x bts (candidates: 1)
21:26:56.321940 tst ping_idle_ping.py: using LAC 31609
21:26:56.468393 tst ping_idle_ping.py: using RAC 244
21:26:56.601593 tst ping_idle_ping.py: using CellId 31609
21:26:56.733882 tst ping_idle_ping.py: using BVCI 31610
21:26:56.760688 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:56.772642 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:56.784421 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:56.796702 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:56.810200 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:56.823013 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:56.838327 tst gprs:trx-lms: Using 1 x ip_address (candidates: 9)
21:26:56.851702 tst gprs:trx-lms: Using 1 x modem (candidates: 4)
21:26:57.171305 tst ping_idle_ping.py:24: start network...
21:26:57.314811 run osmo-hlr_10.42.42.2: Starting osmo-hlr
21:26:57.894751 run create_hlr_db(pid=29931): Launched
21:26:58.094276 bus /gobi_6: Setting Powered False
21:26:59.121084 run create_hlr_db(pid=29931): Terminated: ok {rc=0}
21:26:59.588715 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29933): Launched
21:27:00.049677 run osmo-hlr_10.42.42.2(pid=29934): Launched
21:27:00.289844 run osmo-stp_10.42.42.5: Starting osmo-stp
21:27:00.868571 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29936): Launched
21:27:01.313372 run osmo-stp_10.42.42.5(pid=29937): Launched
21:27:01.532743 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
21:27:02.143097 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29939): Launched
21:27:02.257121 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
21:27:02.582832 run patchelf(pid=29940): Launched
21:27:02.772185 run osmo-stp_10.42.42.5(pid=29937): ERR: Terminated: ERROR {rc=1} [trial-2382↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29937)]
21:27:02.907811 run osmo-stp_10.42.42.5(pid=29937): stdout:
| (launched: 2022-12-12_21:27:01.048732)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:27:03.043725 run osmo-stp_10.42.42.5(pid=29937): stderr:
| �[0;m�[38;5;43m20221212212701375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212701375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212701376 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212701376 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212701376 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212701377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212701377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:27:03.175620 run osmo-stp_10.42.42.5(pid=29937): stdout:
| (launched: 2022-12-12_21:27:01.048732)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
21:27:03.305141 run osmo-stp_10.42.42.5(pid=29937): stderr:
| �[0;m�[38;5;43m20221212212701375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221212212701375 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221212212701376 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221212212701376 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221212212701376 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221212212701377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221212212701377 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
21:27:03.456304 run patchelf(pid=29940): Terminating (SIGINT)
21:27:03.635043 run patchelf(pid=29940): Terminated: ok {rc=0}
21:27:03.668432 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=29937): Process ended prematurely: osmo-stp_10.42.42.5(pid=29937) [trial-2382↪gprs:trx-lms↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=29937)] [trial-2382↪gprs:trx-lms↪ping_idle_ping.py:27]
21:27:03.678347 tst ping_idle_ping.py:27: Test FAILED (7.5 sec)
21:27:03.742424 run osmo-hlr_10.42.42.2(pid=29934): ERR: Terminated: ERROR {rc=237} [trial-2382↪gprs:trx-lms↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=29934)]
21:27:03.794728 run osmo-hlr_10.42.42.2(pid=29934): stdout:
| (launched: 2022-12-12_21:26:59.763547)
21:27:03.847515 run osmo-hlr_10.42.42.2(pid=29934): stderr:
| �[0;m�[1;31m20221212212700039 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221212212700039 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221212212700039 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221212212700039 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221212212700039 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221212212700039 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221212212700057 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2382/run.2022-12-12_20-36-34/gprs:trx-lms/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221212212700078 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221212212700078 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221212212700079 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
21:27:03.892241 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29933): Terminating (SIGTERM)
21:27:03.936109 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29936): Terminating (SIGTERM)
21:27:03.979776 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29939): Terminating (SIGTERM)
21:27:03.991209 --- ParallelTerminationStrategy: PID 29933 died...
21:27:04.048909 run pcap-recorder_any(filters='host 10.42.42.2')(pid=29933): Terminated: ok {rc=0}
21:27:04.059667 --- ParallelTerminationStrategy: PID 29936 died...
21:27:04.115738 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=29936): Terminated: ok {rc=0}
21:27:04.126337 --- ParallelTerminationStrategy: PID 29939 died...
21:27:04.182432 run pcap-recorder_any(filters='host 10.42.42.6')(pid=29939): Terminated: ok {rc=0}
21:27:04.306510 bus /gobi_6: Setting Powered False
5gprs:trx-lms00