227gprs:trx-sysmocell50009.0osmo-stp_10.42.42.5(pid=22992): Process ended prematurely: osmo-stp_10.42.42.5(pid=22992) [trial-2365↪gprs:trx-sysmocell5000↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=22992)]2139cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module>
setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=22992): Process ended prematurely: osmo-stp_10.42.42.5(pid=22992) [trial-2365↪gprs:trx-sysmocell5000↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=22992)]
----------------------------------------------
trial-2365 gprs:trx-sysmocell5000 cs_paging_gprs_active.py
----------------------------------------------
22:15:38.622843 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:38.633949 tst gprs:trx-sysmocell5000: Using 1 x bts (candidates: 1)
22:15:38.778461 tst cs_paging_gprs_active.py: using LAC 24024
22:15:38.911331 tst cs_paging_gprs_active.py: using RAC 54
22:15:39.037004 tst cs_paging_gprs_active.py: using CellId 24024
22:15:39.161595 tst cs_paging_gprs_active.py: using BVCI 24025
22:15:39.184546 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:39.195486 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:39.206279 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:39.216863 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:39.227480 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:39.238623 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:39.249712 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:39.260641 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:39.271555 tst gprs:trx-sysmocell5000: Using 1 x modem (candidates: 4)
22:15:39.387830 tst gprs:trx-sysmocell5000: Using 1 x modem (candidates: 4)
22:15:39.680751 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
22:15:39.815375 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:15:40.150449 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=22982): Launched
22:15:40.429268 run iperf3-srv_10.42.42.10(pid=22983): Launched
22:15:40.627529 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
22:15:40.762062 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:15:41.096127 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=22984): Launched
22:15:41.374677 run iperf3-srv_10.42.42.10(pid=22985): Launched
22:15:41.572703 tst cs_paging_gprs_active.py:38: start network...
22:15:41.707179 run osmo-hlr_10.42.42.2: Starting osmo-hlr
22:15:42.224464 run create_hlr_db(pid=22986): Launched
22:15:42.411865 bus /sierra_2: Setting Powered False
22:15:43.433292 run create_hlr_db(pid=22986): Terminated: ok {rc=0}
22:15:43.841192 run pcap-recorder_any(filters='host 10.42.42.2')(pid=22988): Launched
22:15:44.144828 run osmo-hlr_10.42.42.2(pid=22989): Launched
22:15:44.264061 run osmo-stp_10.42.42.5: Starting osmo-stp
22:15:44.796457 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=22991): Launched
22:15:45.099774 run osmo-stp_10.42.42.5(pid=22992): Launched
22:15:45.211191 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
22:15:45.739389 run pcap-recorder_any(filters='host 10.42.42.6')(pid=22994): Launched
22:15:45.867201 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
22:15:46.176572 run patchelf(pid=22995): Launched
22:15:46.375982 run osmo-stp_10.42.42.5(pid=22992): ERR: Terminated: ERROR {rc=1} [trial-2365↪gprs:trx-sysmocell5000↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=22992)]
22:15:46.519972 run osmo-stp_10.42.42.5(pid=22992): stdout:
| (launched: 2022-12-09_22:15:44.950839)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:15:46.665066 run osmo-stp_10.42.42.5(pid=22992): stderr:
| �[0;m�[38;5;43m20221209221545119 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221545119 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221545120 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221545120 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221545121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221545121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221545121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:15:46.803365 run osmo-stp_10.42.42.5(pid=22992): stdout:
| (launched: 2022-12-09_22:15:44.950839)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:15:46.939285 run osmo-stp_10.42.42.5(pid=22992): stderr:
| �[0;m�[38;5;43m20221209221545119 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221545119 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221545120 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221545120 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221545121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221545121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221545121 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:15:47.099035 run patchelf(pid=22995): Terminating (SIGINT)
22:15:47.284441 run patchelf(pid=22995): Terminated: ok {rc=0}
22:15:47.318193 tst cs_paging_gprs_active.py:38: ERR: Error: osmo-stp_10.42.42.5(pid=22992): Process ended prematurely: osmo-stp_10.42.42.5(pid=22992) [trial-2365↪gprs:trx-sysmocell5000↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=22992)] [trial-2365↪gprs:trx-sysmocell5000↪cs_paging_gprs_active.py:38]
22:15:47.328357 tst cs_paging_gprs_active.py:38: Test FAILED (8.7 sec)
22:15:47.392358 run osmo-hlr_10.42.42.2(pid=22989): ERR: Terminated: ERROR {rc=237} [trial-2365↪gprs:trx-sysmocell5000↪cs_paging_gprs_active.py↪cs_paging_gprs_active.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=22989)]
22:15:47.444542 run osmo-hlr_10.42.42.2(pid=22989): stdout:
| (launched: 2022-12-09_22:15:43.994945)
22:15:47.497213 run osmo-hlr_10.42.42.2(pid=22989): stderr:
| �[0;m�[1;31m20221209221544169 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209221544169 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209221544169 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209221544169 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209221544169 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209221544170 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209221544181 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2365/run.2022-12-09_21-44-27/gprs:trx-sysmocell5000/cs_paging_gprs_active.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209221544191 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209221544192 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209221544192 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
22:15:47.542328 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=22982): Terminating (SIGTERM)
22:15:47.584571 run iperf3-srv_10.42.42.10(pid=22983): Terminating (SIGTERM)
22:15:47.626968 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=22984): Terminating (SIGTERM)
22:15:47.669224 run iperf3-srv_10.42.42.10(pid=22985): Terminating (SIGTERM)
22:15:47.711659 run pcap-recorder_any(filters='host 10.42.42.2')(pid=22988): Terminating (SIGTERM)
22:15:47.754029 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=22991): Terminating (SIGTERM)
22:15:47.796416 run pcap-recorder_any(filters='host 10.42.42.6')(pid=22994): Terminating (SIGTERM)
22:15:47.806265 --- ParallelTerminationStrategy: PID 22982 died...
22:15:47.863025 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=22982): Terminated: ok {rc=0}
22:15:47.873186 --- ParallelTerminationStrategy: PID 22983 died...
22:15:47.929373 run iperf3-srv_10.42.42.10(pid=22983): Terminated {rc=256}
22:15:47.939396 --- ParallelTerminationStrategy: PID 22984 died...
22:15:47.995771 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=22984): Terminated: ok {rc=0}
22:15:48.005987 --- ParallelTerminationStrategy: PID 22985 died...
22:15:48.061947 run iperf3-srv_10.42.42.10(pid=22985): Terminated {rc=256}
22:15:48.072062 --- ParallelTerminationStrategy: PID 22988 died...
22:15:48.128180 run pcap-recorder_any(filters='host 10.42.42.2')(pid=22988): Terminated: ok {rc=0}
22:15:48.138318 --- ParallelTerminationStrategy: PID 22991 died...
22:15:48.194877 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=22991): Terminated: ok {rc=0}
22:15:48.205169 --- ParallelTerminationStrategy: PID 22994 died...
22:15:48.261327 run pcap-recorder_any(filters='host 10.42.42.6')(pid=22994): Terminated: ok {rc=0}
22:15:48.348549 bus /gobi_0: Setting Powered False
22:15:49.455010 bus /gobi_0: Setting Powered False
22:15:50.562933 bus /gobi_6: Setting Powered False
227gprs:trx-sysmocell50008.0osmo-stp_10.42.42.5(pid=23004): Process ended prematurely: osmo-stp_10.42.42.5(pid=23004) [trial-2365↪gprs:trx-sysmocell5000↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23004)]2139iperf3.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=23004): Process ended prematurely: osmo-stp_10.42.42.5(pid=23004) [trial-2365↪gprs:trx-sysmocell5000↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23004)]
----------------------------------------------
trial-2365 gprs:trx-sysmocell5000 iperf3.py
----------------------------------------------
22:15:51.677699 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:51.689589 tst gprs:trx-sysmocell5000: Using 1 x bts (candidates: 1)
22:15:51.837048 tst iperf3.py: using LAC 24025
22:15:51.965069 tst iperf3.py: using RAC 55
22:15:52.093712 tst iperf3.py: using CellId 24025
22:15:52.222219 tst iperf3.py: using BVCI 24026
22:15:52.247007 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:52.258221 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:52.269152 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:52.280059 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:52.290977 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:52.301862 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:52.312863 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:52.323639 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:15:52.334741 tst gprs:trx-sysmocell5000: Using 1 x modem (candidates: 4)
22:15:52.614062 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
22:15:52.739507 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:15:53.052473 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=22996): Launched
22:15:53.312812 run iperf3-srv_10.42.42.10(pid=22997): Launched
22:15:53.496771 tst iperf3.py:8: start network...
22:15:53.622224 run osmo-hlr_10.42.42.2: Starting osmo-hlr
22:15:54.098703 run create_hlr_db(pid=22998): Launched
22:15:54.273421 bus /gobi_6: Setting Powered False
22:15:55.294847 run create_hlr_db(pid=22998): Terminated: ok {rc=0}
22:15:55.725580 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23000): Launched
22:15:56.048206 run osmo-hlr_10.42.42.2(pid=23001): Launched
22:15:56.166223 run osmo-stp_10.42.42.5: Starting osmo-stp
22:15:56.729532 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23003): Launched
22:15:57.052235 run osmo-stp_10.42.42.5(pid=23004): Launched
22:15:57.169240 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
22:15:57.734614 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23006): Launched
22:15:57.850794 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
22:15:58.134652 run patchelf(pid=23007): Launched
22:15:58.326562 run osmo-stp_10.42.42.5(pid=23004): ERR: Terminated: ERROR {rc=1} [trial-2365↪gprs:trx-sysmocell5000↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23004)]
22:15:58.466193 run osmo-stp_10.42.42.5(pid=23004): stdout:
| (launched: 2022-12-09_22:15:56.894282)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:15:58.605768 run osmo-stp_10.42.42.5(pid=23004): stderr:
| �[0;m�[38;5;43m20221209221557087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221557087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221557087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221557087 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221557088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221557088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221557088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:15:58.740277 run osmo-stp_10.42.42.5(pid=23004): stdout:
| (launched: 2022-12-09_22:15:56.894282)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:15:58.873697 run osmo-stp_10.42.42.5(pid=23004): stderr:
| �[0;m�[38;5;43m20221209221557087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221557087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221557087 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221557087 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221557088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221557088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221557088 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:15:59.030300 run patchelf(pid=23007): Terminating (SIGINT)
22:15:59.215155 run patchelf(pid=23007): Terminated: ok {rc=0}
22:15:59.249406 tst iperf3.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=23004): Process ended prematurely: osmo-stp_10.42.42.5(pid=23004) [trial-2365↪gprs:trx-sysmocell5000↪iperf3.py:8↪iperf3.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23004)] [trial-2365↪gprs:trx-sysmocell5000↪iperf3.py:8]
22:15:59.259450 tst iperf3.py:8: Test FAILED (7.6 sec)
22:15:59.322824 run osmo-hlr_10.42.42.2(pid=23001): ERR: Terminated: ERROR {rc=237} [trial-2365↪gprs:trx-sysmocell5000↪iperf3.py↪iperf3.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=23001)]
22:15:59.374743 run osmo-hlr_10.42.42.2(pid=23001): stdout:
| (launched: 2022-12-09_22:15:55.889891)
22:15:59.427089 run osmo-hlr_10.42.42.2(pid=23001): stderr:
| �[0;m�[1;31m20221209221556088 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209221556088 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209221556089 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209221556089 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209221556089 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209221556089 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209221556100 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2365/run.2022-12-09_21-44-27/gprs:trx-sysmocell5000/iperf3.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209221556109 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209221556109 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209221556109 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
22:15:59.471707 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=22996): Terminating (SIGTERM)
22:15:59.513853 run iperf3-srv_10.42.42.10(pid=22997): Terminating (SIGTERM)
22:15:59.556045 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23000): Terminating (SIGTERM)
22:15:59.598139 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23003): Terminating (SIGTERM)
22:15:59.640100 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23006): Terminating (SIGTERM)
22:15:59.649842 --- ParallelTerminationStrategy: PID 22996 died...
22:15:59.706203 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=22996): Terminated: ok {rc=0}
22:15:59.716348 --- ParallelTerminationStrategy: PID 22997 died...
22:15:59.772229 run iperf3-srv_10.42.42.10(pid=22997): Terminated {rc=256}
22:15:59.782243 --- ParallelTerminationStrategy: PID 23000 died...
22:15:59.838084 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23000): Terminated: ok {rc=0}
22:15:59.848218 --- ParallelTerminationStrategy: PID 23003 died...
22:15:59.904023 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23003): Terminated: ok {rc=0}
22:15:59.914211 --- ParallelTerminationStrategy: PID 23006 died...
22:15:59.969889 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23006): Terminated: ok {rc=0}
22:16:00.057878 bus /gobi_6: Setting Powered False
227gprs:trx-sysmocell500011.0osmo-stp_10.42.42.5(pid=23022): Process ended prematurely: osmo-stp_10.42.42.5(pid=23022) [trial-2365↪gprs:trx-sysmocell5000↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23022)]2139iperf3m4.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=23022): Process ended prematurely: osmo-stp_10.42.42.5(pid=23022) [trial-2365↪gprs:trx-sysmocell5000↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23022)]
----------------------------------------------
trial-2365 gprs:trx-sysmocell5000 iperf3m4.py
----------------------------------------------
22:16:01.195207 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:01.211779 tst gprs:trx-sysmocell5000: Using 1 x bts (candidates: 1)
22:16:01.387260 tst iperf3m4.py: using LAC 24026
22:16:01.538429 tst iperf3m4.py: using RAC 56
22:16:01.660237 tst iperf3m4.py: using CellId 24026
22:16:01.784285 tst iperf3m4.py: using BVCI 24027
22:16:01.810438 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:01.822548 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:01.834741 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:01.846366 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:01.858061 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:01.869775 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:01.881848 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:01.893411 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:01.905205 tst gprs:trx-sysmocell5000: Using 1 x modem (candidates: 4)
22:16:02.024534 tst gprs:trx-sysmocell5000: Using 1 x modem (candidates: 4)
22:16:02.153036 tst gprs:trx-sysmocell5000: Using 1 x modem (candidates: 4)
22:16:02.278969 tst gprs:trx-sysmocell5000: Using 1 x modem (candidates: 4)
22:16:02.559043 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
22:16:02.684912 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:16:02.997880 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23008): Launched
22:16:03.258486 run iperf3-srv_10.42.42.10(pid=23009): Launched
22:16:03.443105 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
22:16:03.568749 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:16:03.881316 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23010): Launched
22:16:04.140707 run iperf3-srv_10.42.42.10(pid=23011): Launched
22:16:04.324341 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
22:16:04.449736 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:16:04.762508 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23012): Launched
22:16:05.021719 run iperf3-srv_10.42.42.10(pid=23013): Launched
22:16:05.205551 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
22:16:05.331198 run iperf3-srv_10.42.42.10: Starting iperf3-srv
22:16:05.643863 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23014): Launched
22:16:05.902649 run iperf3-srv_10.42.42.10(pid=23015): Launched
22:16:06.086354 tst iperf3m4.py:8: start network...
22:16:06.211700 run osmo-hlr_10.42.42.2: Starting osmo-hlr
22:16:06.687139 run create_hlr_db(pid=23016): Launched
22:16:06.861361 bus /gobi_6: Setting Powered False
22:16:07.882922 run create_hlr_db(pid=23016): Terminated: ok {rc=0}
22:16:08.314701 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23018): Launched
22:16:08.640190 run osmo-hlr_10.42.42.2(pid=23019): Launched
22:16:08.757760 run osmo-stp_10.42.42.5: Starting osmo-stp
22:16:09.320783 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23021): Launched
22:16:09.646076 run osmo-stp_10.42.42.5(pid=23022): Launched
22:16:09.763854 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
22:16:10.328496 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23024): Launched
22:16:10.443958 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
22:16:10.725732 run patchelf(pid=23025): Launched
22:16:10.915804 run osmo-stp_10.42.42.5(pid=23022): ERR: Terminated: ERROR {rc=1} [trial-2365↪gprs:trx-sysmocell5000↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23022)]
22:16:11.054704 run osmo-stp_10.42.42.5(pid=23022): stdout:
| (launched: 2022-12-09_22:16:09.486366)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:16:11.193816 run osmo-stp_10.42.42.5(pid=23022): stderr:
| �[0;m�[38;5;43m20221209221609681 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221609681 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221609681 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221609682 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221609682 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221609682 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221609682 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:16:11.328223 run osmo-stp_10.42.42.5(pid=23022): stdout:
| (launched: 2022-12-09_22:16:09.486366)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:16:11.461504 run osmo-stp_10.42.42.5(pid=23022): stderr:
| �[0;m�[38;5;43m20221209221609681 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221609681 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221609681 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221609682 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221609682 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221609682 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221609682 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:16:11.617262 run patchelf(pid=23025): Terminating (SIGINT)
22:16:11.800466 run patchelf(pid=23025): Terminated: ok {rc=0}
22:16:11.834003 tst iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=23022): Process ended prematurely: osmo-stp_10.42.42.5(pid=23022) [trial-2365↪gprs:trx-sysmocell5000↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23022)] [trial-2365↪gprs:trx-sysmocell5000↪iperf3m4.py:8]
22:16:11.843962 tst iperf3m4.py:8: Test FAILED (10.7 sec)
22:16:11.907225 run osmo-hlr_10.42.42.2(pid=23019): ERR: Terminated: ERROR {rc=237} [trial-2365↪gprs:trx-sysmocell5000↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=23019)]
22:16:11.958967 run osmo-hlr_10.42.42.2(pid=23019): stdout:
| (launched: 2022-12-09_22:16:08.480653)
22:16:12.011183 run osmo-hlr_10.42.42.2(pid=23019): stderr:
| �[0;m�[1;31m20221209221608678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209221608678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209221608678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209221608678 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209221608679 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209221608679 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209221608691 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2365/run.2022-12-09_21-44-27/gprs:trx-sysmocell5000/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209221608699 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209221608699 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209221608699 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
22:16:12.055640 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23008): Terminating (SIGTERM)
22:16:12.097404 run iperf3-srv_10.42.42.10(pid=23009): Terminating (SIGTERM)
22:16:12.139316 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23010): Terminating (SIGTERM)
22:16:12.181338 run iperf3-srv_10.42.42.10(pid=23011): Terminating (SIGTERM)
22:16:12.223567 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23012): Terminating (SIGTERM)
22:16:12.265372 run iperf3-srv_10.42.42.10(pid=23013): Terminating (SIGTERM)
22:16:12.307289 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23014): Terminating (SIGTERM)
22:16:12.349173 run iperf3-srv_10.42.42.10(pid=23015): Terminating (SIGTERM)
22:16:12.391006 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23018): Terminating (SIGTERM)
22:16:12.433208 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23021): Terminating (SIGTERM)
22:16:12.475225 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23024): Terminating (SIGTERM)
22:16:12.485000 --- ParallelTerminationStrategy: PID 23008 died...
22:16:12.541397 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23008): Terminated: ok {rc=0}
22:16:12.551465 --- ParallelTerminationStrategy: PID 23009 died...
22:16:12.607406 run iperf3-srv_10.42.42.10(pid=23009): Terminated {rc=256}
22:16:12.617440 --- ParallelTerminationStrategy: PID 23010 died...
22:16:12.673399 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23010): Terminated: ok {rc=0}
22:16:12.683464 --- ParallelTerminationStrategy: PID 23011 died...
22:16:12.739194 run iperf3-srv_10.42.42.10(pid=23011): Terminated {rc=256}
22:16:12.749403 --- ParallelTerminationStrategy: PID 23012 died...
22:16:12.805248 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23012): Terminated: ok {rc=0}
22:16:12.815396 --- ParallelTerminationStrategy: PID 23013 died...
22:16:12.870910 run iperf3-srv_10.42.42.10(pid=23013): Terminated {rc=256}
22:16:12.881043 --- ParallelTerminationStrategy: PID 23014 died...
22:16:12.936877 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=23014): Terminated: ok {rc=0}
22:16:12.947038 --- ParallelTerminationStrategy: PID 23015 died...
22:16:13.002437 run iperf3-srv_10.42.42.10(pid=23015): Terminated {rc=256}
22:16:13.012414 --- ParallelTerminationStrategy: PID 23018 died...
22:16:13.067919 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23018): Terminated: ok {rc=0}
22:16:13.077931 --- ParallelTerminationStrategy: PID 23021 died...
22:16:13.133267 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23021): Terminated: ok {rc=0}
22:16:13.143321 --- ParallelTerminationStrategy: PID 23024 died...
22:16:13.199726 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23024): Terminated: ok {rc=0}
22:16:13.288026 bus /gobi_4: Setting Powered False
22:16:14.394458 bus /gobi_4: Setting Powered False
22:16:15.503384 bus /gobi_1: Setting Powered False
22:16:16.607785 bus /gobi_1: Setting Powered False
22:16:17.719244 bus /gobi_0: Setting Powered False
22:16:18.822871 bus /gobi_0: Setting Powered False
22:16:19.930526 bus /gobi_6: Setting Powered False
227gprs:trx-sysmocell50007.0osmo-stp_10.42.42.5(pid=23032): Process ended prematurely: osmo-stp_10.42.42.5(pid=23032) [trial-2365↪gprs:trx-sysmocell5000↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23032)]2139ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping.py", line 22, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=23032): Process ended prematurely: osmo-stp_10.42.42.5(pid=23032) [trial-2365↪gprs:trx-sysmocell5000↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23032)]
----------------------------------------------
trial-2365 gprs:trx-sysmocell5000 ping.py
----------------------------------------------
22:16:20.999745 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:21.011735 tst gprs:trx-sysmocell5000: Using 1 x bts (candidates: 1)
22:16:21.154855 tst ping.py: using LAC 24027
22:16:21.274385 tst ping.py: using RAC 57
22:16:21.393241 tst ping.py: using CellId 24027
22:16:21.509017 tst ping.py: using BVCI 24028
22:16:21.531723 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:21.542901 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:21.553938 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:21.564733 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:21.575422 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:21.586253 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:21.597183 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:21.607803 tst gprs:trx-sysmocell5000: Using 1 x modem (candidates: 4)
22:16:21.897748 tst ping.py:19: start network...
22:16:22.030163 run osmo-hlr_10.42.42.2: Starting osmo-hlr
22:16:22.535453 run create_hlr_db(pid=23026): Launched
22:16:22.721048 bus /gobi_6: Setting Powered False
22:16:23.740087 run create_hlr_db(pid=23026): Terminated: ok {rc=0}
22:16:24.138358 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23028): Launched
22:16:24.436507 run osmo-hlr_10.42.42.2(pid=23029): Launched
22:16:24.545350 run osmo-stp_10.42.42.5: Starting osmo-stp
22:16:25.062198 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23031): Launched
22:16:25.358169 run osmo-stp_10.42.42.5(pid=23032): Launched
22:16:25.465731 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
22:16:25.981041 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23034): Launched
22:16:26.086221 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
22:16:26.345186 run patchelf(pid=23035): Launched
22:16:26.520107 run osmo-stp_10.42.42.5(pid=23032): ERR: Terminated: ERROR {rc=1} [trial-2365↪gprs:trx-sysmocell5000↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23032)]
22:16:26.648314 run osmo-stp_10.42.42.5(pid=23032): stdout:
| (launched: 2022-12-09_22:16:25.212775)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:16:26.776419 run osmo-stp_10.42.42.5(pid=23032): stderr:
| �[0;m�[38;5;43m20221209221625381 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221625381 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221625381 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221625382 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221625382 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221625382 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221625382 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:16:26.899983 run osmo-stp_10.42.42.5(pid=23032): stdout:
| (launched: 2022-12-09_22:16:25.212775)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:16:27.021743 run osmo-stp_10.42.42.5(pid=23032): stderr:
| �[0;m�[38;5;43m20221209221625381 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221625381 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221625381 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221625382 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221625382 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221625382 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221625382 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:16:27.164773 run patchelf(pid=23035): Terminating (SIGINT)
22:16:27.332934 run patchelf(pid=23035): Terminated: ok {rc=0}
22:16:27.363468 tst ping.py:22: ERR: Error: osmo-stp_10.42.42.5(pid=23032): Process ended prematurely: osmo-stp_10.42.42.5(pid=23032) [trial-2365↪gprs:trx-sysmocell5000↪ping.py:22↪ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23032)] [trial-2365↪gprs:trx-sysmocell5000↪ping.py:22]
22:16:27.371925 tst ping.py:22: Test FAILED (6.4 sec)
22:16:27.433054 run osmo-hlr_10.42.42.2(pid=23029): ERR: Terminated: ERROR {rc=237} [trial-2365↪gprs:trx-sysmocell5000↪ping.py↪ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=23029)]
22:16:27.482637 run osmo-hlr_10.42.42.2(pid=23029): stdout:
| (launched: 2022-12-09_22:16:24.288654)
22:16:27.533006 run osmo-hlr_10.42.42.2(pid=23029): stderr:
| �[0;m�[1;31m20221209221624465 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209221624465 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209221624465 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209221624465 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209221624465 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209221624465 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209221624478 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2365/run.2022-12-09_21-44-27/gprs:trx-sysmocell5000/ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209221624489 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209221624489 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209221624489 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
22:16:27.575238 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23028): Terminating (SIGTERM)
22:16:27.615417 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23031): Terminating (SIGTERM)
22:16:27.655631 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23034): Terminating (SIGTERM)
22:16:27.665296 --- ParallelTerminationStrategy: PID 23028 died...
22:16:27.719667 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23028): Terminated: ok {rc=0}
22:16:27.729531 --- ParallelTerminationStrategy: PID 23031 died...
22:16:27.783164 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23031): Terminated: ok {rc=0}
22:16:27.793103 --- ParallelTerminationStrategy: PID 23034 died...
22:16:27.846644 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23034): Terminated: ok {rc=0}
22:16:27.935669 bus /gobi_6: Setting Powered False
227gprs:trx-sysmocell50007.0osmo-stp_10.42.42.5(pid=23042): Process ended prematurely: osmo-stp_10.42.42.5(pid=23042) [trial-2365↪gprs:trx-sysmocell5000↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23042)]2139ping_idle_ping.pyfalseFAILEDTraceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 27, in <module>
ggsn.start()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
proc.launch_sync()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
self.wait()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
MainLoop.wait(self.terminated, timeout=timeout)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
self.poll(may_block=True)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
self.deferred_handling.handle_queue()
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
handler(*args, **kwargs)
File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=23042): Process ended prematurely: osmo-stp_10.42.42.5(pid=23042) [trial-2365↪gprs:trx-sysmocell5000↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23042)]
----------------------------------------------
trial-2365 gprs:trx-sysmocell5000 ping_idle_ping.py
----------------------------------------------
22:16:28.988227 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:29.000049 tst gprs:trx-sysmocell5000: Using 1 x bts (candidates: 1)
22:16:29.140604 tst ping_idle_ping.py: using LAC 24028
22:16:29.257683 tst ping_idle_ping.py: using RAC 58
22:16:29.371848 tst ping_idle_ping.py: using CellId 24028
22:16:29.495614 tst ping_idle_ping.py: using BVCI 24029
22:16:29.520640 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:29.532463 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:29.544067 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:29.555704 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:29.566926 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:29.578163 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:29.589740 tst gprs:trx-sysmocell5000: Using 1 x ip_address (candidates: 9)
22:16:29.600500 tst gprs:trx-sysmocell5000: Using 1 x modem (candidates: 4)
22:16:29.874260 tst ping_idle_ping.py:24: start network...
22:16:29.997483 run osmo-hlr_10.42.42.2: Starting osmo-hlr
22:16:30.466047 run create_hlr_db(pid=23036): Launched
22:16:30.637810 bus /gobi_6: Setting Powered False
22:16:31.659484 run create_hlr_db(pid=23036): Terminated: ok {rc=0}
22:16:32.076307 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23038): Launched
22:16:32.376195 run osmo-hlr_10.42.42.2(pid=23039): Launched
22:16:32.488096 run osmo-stp_10.42.42.5: Starting osmo-stp
22:16:33.002700 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23041): Launched
22:16:33.299882 run osmo-stp_10.42.42.5(pid=23042): Launched
22:16:33.407455 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
22:16:33.926485 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23044): Launched
22:16:34.032916 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
22:16:34.293838 run patchelf(pid=23045): Launched
22:16:34.468069 run osmo-stp_10.42.42.5(pid=23042): ERR: Terminated: ERROR {rc=1} [trial-2365↪gprs:trx-sysmocell5000↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23042)]
22:16:34.595617 run osmo-stp_10.42.42.5(pid=23042): stdout:
| (launched: 2022-12-09_22:16:33.153869)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:16:34.723886 run osmo-stp_10.42.42.5(pid=23042): stderr:
| �[0;m�[38;5;43m20221209221633324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221633324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221633324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221633325 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221633325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221633325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221633325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:16:34.847067 run osmo-stp_10.42.42.5(pid=23042): stdout:
| (launched: 2022-12-09_22:16:33.153869)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
|
| Free Software lives by contribution. If you use this, please contribute!
|
22:16:34.968702 run osmo-stp_10.42.42.5(pid=23042): stderr:
| �[0;m�[38;5;43m20221209221633324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209221633324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209221633324 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209221633325 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209221633325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209221633325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209221633325 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use
22:16:35.111579 run patchelf(pid=23045): Terminating (SIGINT)
22:16:35.279813 run patchelf(pid=23045): Terminated: ok {rc=0}
22:16:35.310283 tst ping_idle_ping.py:27: ERR: Error: osmo-stp_10.42.42.5(pid=23042): Process ended prematurely: osmo-stp_10.42.42.5(pid=23042) [trial-2365↪gprs:trx-sysmocell5000↪ping_idle_ping.py:27↪ping_idle_ping.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=23042)] [trial-2365↪gprs:trx-sysmocell5000↪ping_idle_ping.py:27]
22:16:35.318967 tst ping_idle_ping.py:27: Test FAILED (6.3 sec)
22:16:35.379316 run osmo-hlr_10.42.42.2(pid=23039): ERR: Terminated: ERROR {rc=237} [trial-2365↪gprs:trx-sysmocell5000↪ping_idle_ping.py↪ping_idle_ping.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=23039)]
22:16:35.428321 run osmo-hlr_10.42.42.2(pid=23039): stdout:
| (launched: 2022-12-09_22:16:32.229753)
22:16:35.478299 run osmo-hlr_10.42.42.2(pid=23039): stderr:
| �[0;m�[1;31m20221209221632405 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209221632405 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209221632405 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209221632405 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209221632405 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209221632406 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209221632420 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2365/run.2022-12-09_21-44-27/gprs:trx-sysmocell5000/ping_idle_ping.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209221632430 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209221632430 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209221632430 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99)
22:16:35.520549 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23038): Terminating (SIGTERM)
22:16:35.561029 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23041): Terminating (SIGTERM)
22:16:35.601704 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23044): Terminating (SIGTERM)
22:16:35.610982 --- ParallelTerminationStrategy: PID 23038 died...
22:16:35.664706 run pcap-recorder_any(filters='host 10.42.42.2')(pid=23038): Terminated: ok {rc=0}
22:16:35.674410 --- ParallelTerminationStrategy: PID 23041 died...
22:16:35.727716 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=23041): Terminated: ok {rc=0}
22:16:35.737865 --- ParallelTerminationStrategy: PID 23044 died...
22:16:35.791776 run pcap-recorder_any(filters='host 10.42.42.6')(pid=23044): Terminated: ok {rc=0}
22:16:35.875829 bus /gobi_6: Setting Powered False
5gprs:trx-sysmocell500000