Skip to content
[Jenkins]Jenkins
log in | sign up

Failed

gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend.iperf3m4.py (from gprs_trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend)

Failing for the past 219 builds (Since #2139 )
Took 11 sec.

Stacktrace

osmo-stp_10.42.42.5(pid=18333): Process ended prematurely: osmo-stp_10.42.42.5(pid=18333) [trial-2357↪gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18333)]

Standard Output

----------------------------------------------
trial-2357 gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend iperf3m4.py
----------------------------------------------
09:06:56.222381 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x ip_address (candidates: 9)
09:06:56.235091 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x bts (candidates: 1)
09:06:56.383842 tst                      iperf3m4.py: using LAC 20402
09:06:56.510648 tst                      iperf3m4.py: using RAC 2
09:06:56.631572 tst                      iperf3m4.py: using CellId 20402
09:06:56.755448 tst                      iperf3m4.py: using BVCI 20403
09:06:56.779063 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x ip_address (candidates: 9)
09:06:56.790717 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x ip_address (candidates: 9)
09:06:56.802134 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x ip_address (candidates: 9)
09:06:56.813743 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x ip_address (candidates: 9)
09:06:56.825140 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x ip_address (candidates: 9)
09:06:56.837268 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x ip_address (candidates: 9)
09:06:56.848677 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x ip_address (candidates: 9)
09:06:56.859908 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x ip_address (candidates: 9)
09:06:56.871387 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x modem (candidates: 4)
09:06:56.993079 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x modem (candidates: 4)
09:06:57.114541 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x modem (candidates: 4)
09:06:57.235772 tst gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend: Using 1 x modem (candidates: 4)
09:06:57.538511 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
09:06:57.676388 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
09:06:58.016110 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18319): Launched
09:06:58.298995 run iperf3-srv_10.42.42.10(pid=18320): Launched
09:06:58.499717 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
09:06:58.637115 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
09:06:58.975349 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18321): Launched
09:06:59.257685 run iperf3-srv_10.42.42.10(pid=18322): Launched
09:06:59.458799 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
09:06:59.597182 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
09:06:59.935246 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18323): Launched
09:07:00.217502 run iperf3-srv_10.42.42.10(pid=18324): Launched
09:07:00.418000 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
09:07:00.556098 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
09:07:00.912412 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18325): Launched
09:07:01.215733 run iperf3-srv_10.42.42.10(pid=18326): Launched
09:07:01.420683 tst                    iperf3m4.py:8: start network...
09:07:01.561833 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
09:07:02.163518 run         create_hlr_db(pid=18327): Launched
09:07:02.358701 bus                          /gobi_6: Setting Powered False
09:07:03.374997 run         create_hlr_db(pid=18327): Terminated: ok {rc=0}
09:07:03.787006 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18329): Launched
09:07:04.094955 run   osmo-hlr_10.42.42.2(pid=18330): Launched
09:07:04.207821 run              osmo-stp_10.42.42.5: Starting osmo-stp
09:07:04.738833 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18332): Launched
09:07:05.047504 run   osmo-stp_10.42.42.5(pid=18333): Launched
09:07:05.160018 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
09:07:05.695343 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18335): Launched
09:07:05.803973 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
09:07:06.069854 run              patchelf(pid=18336): Launched
09:07:06.248119 run   osmo-stp_10.42.42.5(pid=18333): ERR: Terminated: ERROR {rc=1}  [trial-2357↪gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18333)]
09:07:06.378142 run   osmo-stp_10.42.42.5(pid=18333): stdout: 
| (launched: 2022-12-09_09:07:04.893857)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
| 
| Free Software lives by contribution.  If you use this, please contribute!
|  
09:07:06.508295 run   osmo-stp_10.42.42.5(pid=18333): stderr: 
| �[0;m�[38;5;43m20221209090705060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090705060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090705060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090705061 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090705061 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090705061 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090705061 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use 
09:07:06.634405 run   osmo-stp_10.42.42.5(pid=18333): stdout: 
| (launched: 2022-12-09_09:07:04.893857)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
| 
| Free Software lives by contribution.  If you use this, please contribute!
|  
09:07:06.759444 run   osmo-stp_10.42.42.5(pid=18333): stderr: 
| �[0;m�[38;5;43m20221209090705060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221209090705060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221209090705060 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221209090705061 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221209090705061 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221209090705061 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221209090705061 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use 
09:07:06.905288 run              patchelf(pid=18336): Terminating (SIGINT)
09:07:07.076363 run              patchelf(pid=18336): Terminated: ok {rc=0}
09:07:07.107769 tst                    iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=18333): Process ended prematurely: osmo-stp_10.42.42.5(pid=18333) [trial-2357↪gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18333)]  [trial-2357↪gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend↪iperf3m4.py:8]
09:07:07.117163 tst                    iperf3m4.py:8: Test FAILED (10.9 sec)
09:07:07.177402 run   osmo-hlr_10.42.42.2(pid=18330): ERR: Terminated: ERROR {rc=237}  [trial-2357↪gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=18330)]
09:07:07.226032 run   osmo-hlr_10.42.42.2(pid=18330): stdout: 
| (launched: 2022-12-09_09:07:03.943628) 
09:07:07.275193 run   osmo-hlr_10.42.42.2(pid=18330): stderr: 
| �[0;m�[1;31m20221209090704117 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221209090704117 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221209090704117 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221209090704117 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221209090704117 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221209090704117 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221209090704134 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2357/run.2022-12-09_08-22-41/gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221209090704145 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221209090704146 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221209090704146 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99) 
09:07:07.317453 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18319): Terminating (SIGTERM)
09:07:07.356921 run iperf3-srv_10.42.42.10(pid=18320): Terminating (SIGTERM)
09:07:07.396473 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18321): Terminating (SIGTERM)
09:07:07.435743 run iperf3-srv_10.42.42.10(pid=18322): Terminating (SIGTERM)
09:07:07.475252 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18323): Terminating (SIGTERM)
09:07:07.514643 run iperf3-srv_10.42.42.10(pid=18324): Terminating (SIGTERM)
09:07:07.554038 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18325): Terminating (SIGTERM)
09:07:07.593319 run iperf3-srv_10.42.42.10(pid=18326): Terminating (SIGTERM)
09:07:07.633181 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18329): Terminating (SIGTERM)
09:07:07.672510 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18332): Terminating (SIGTERM)
09:07:07.712005 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18335): Terminating (SIGTERM)
09:07:07.721123 ---      ParallelTerminationStrategy: PID 18319 died...
09:07:07.773888 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18319): Terminated: ok {rc=0}
09:07:07.783349 ---      ParallelTerminationStrategy: PID 18320 died...
09:07:07.835800 run iperf3-srv_10.42.42.10(pid=18320): Terminated {rc=256}
09:07:07.845293 ---      ParallelTerminationStrategy: PID 18321 died...
09:07:07.897823 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18321): Terminated: ok {rc=0}
09:07:07.907277 ---      ParallelTerminationStrategy: PID 18322 died...
09:07:07.960216 run iperf3-srv_10.42.42.10(pid=18322): Terminated {rc=256}
09:07:07.969672 ---      ParallelTerminationStrategy: PID 18323 died...
09:07:08.022100 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18323): Terminated: ok {rc=0}
09:07:08.031553 ---      ParallelTerminationStrategy: PID 18324 died...
09:07:08.083922 run iperf3-srv_10.42.42.10(pid=18324): Terminated {rc=256}
09:07:08.093388 ---      ParallelTerminationStrategy: PID 18325 died...
09:07:08.146142 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18325): Terminated: ok {rc=0}
09:07:08.155573 ---      ParallelTerminationStrategy: PID 18326 died...
09:07:08.208531 run iperf3-srv_10.42.42.10(pid=18326): Terminated {rc=256}
09:07:08.218018 ---      ParallelTerminationStrategy: PID 18329 died...
09:07:08.271149 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18329): Terminated: ok {rc=0}
09:07:08.280641 ---      ParallelTerminationStrategy: PID 18332 died...
09:07:08.333231 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18332): Terminated: ok {rc=0}
09:07:08.342742 ---      ParallelTerminationStrategy: PID 18335 died...
09:07:08.395428 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18335): Terminated: ok {rc=0}
09:07:08.484759 bus                          /gobi_4: Setting Powered False
09:07:09.587778 bus                          /gobi_4: Setting Powered False
09:07:10.699212 bus                          /gobi_1: Setting Powered False
09:07:11.801470 bus                          /gobi_1: Setting Powered False
09:07:12.912727 bus                          /gobi_0: Setting Powered False
09:07:14.015255 bus                          /gobi_0: Setting Powered False
09:07:15.127565 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
    setup_run_iperf3_test_parallel(4)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
    ggsn.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
    util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
    self.wait()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
    MainLoop.wait(self.terminated, timeout=timeout)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
    raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=18333): Process ended prematurely: osmo-stp_10.42.42.5(pid=18333) [trial-2357↪gprs:trx-umtrx+mod-bts0-numtrx2+mod-bts0-chanallocdescend↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=18333)]