Skip to content
[Jenkins]Jenkins
log in | sign up

Failed

gprs:trx-sysmocell5000+mod-bts0-dynts-ipa.iperf3m4.py (from gprs_trx-sysmocell5000+mod-bts0-dynts-ipa)

Failing for the past 261 builds (Since #2139 )
Took 11 sec.

Stacktrace

osmo-stp_10.42.42.5(pid=6726): Process ended prematurely: osmo-stp_10.42.42.5(pid=6726) [trial-2399↪gprs:trx-sysmocell5000+mod-bts0-dynts-ipa↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6726)]

Standard Output

----------------------------------------------
trial-2399 gprs:trx-sysmocell5000+mod-bts0-dynts-ipa iperf3m4.py
----------------------------------------------
04:08:31.508854 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x ip_address (candidates: 9)
04:08:31.521584 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x bts (candidates: 1)
04:08:31.692518 tst                      iperf3m4.py: using LAC 38441
04:08:31.857005 tst                      iperf3m4.py: using RAC 191
04:08:31.996940 tst                      iperf3m4.py: using CellId 38441
04:08:32.129327 tst                      iperf3m4.py: using BVCI 38442
04:08:32.155099 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x ip_address (candidates: 9)
04:08:32.167252 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x ip_address (candidates: 9)
04:08:32.179242 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x ip_address (candidates: 9)
04:08:32.191364 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x ip_address (candidates: 9)
04:08:32.204279 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x ip_address (candidates: 9)
04:08:32.217084 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x ip_address (candidates: 9)
04:08:32.230023 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x ip_address (candidates: 9)
04:08:32.242843 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x ip_address (candidates: 9)
04:08:32.257523 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x modem (candidates: 4)
04:08:32.376520 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x modem (candidates: 4)
04:08:32.498662 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x modem (candidates: 4)
04:08:32.615741 tst gprs:trx-sysmocell5000+mod-bts0-dynts-ipa: Using 1 x modem (candidates: 4)
04:08:32.907555 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
04:08:33.041321 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
04:08:33.372391 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6712): Launched
04:08:33.648851 run iperf3-srv_10.42.42.10(pid=6713): Launched
04:08:33.844761 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
04:08:33.978445 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
04:08:34.311271 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6714): Launched
04:08:34.588471 run iperf3-srv_10.42.42.10(pid=6715): Launched
04:08:34.784610 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
04:08:34.918380 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
04:08:35.249596 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6716): Launched
04:08:35.523090 run iperf3-srv_10.42.42.10(pid=6717): Launched
04:08:35.717819 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
04:08:35.850569 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
04:08:36.181495 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6718): Launched
04:08:36.457431 run iperf3-srv_10.42.42.10(pid=6719): Launched
04:08:36.652125 tst                    iperf3m4.py:8: start network...
04:08:36.784913 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
04:08:37.288937 run          create_hlr_db(pid=6720): Launched
04:08:37.472232 bus                          /gobi_6: Setting Powered False
04:08:38.494112 run          create_hlr_db(pid=6720): Terminated: ok {rc=0}
04:08:38.918479 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6722): Launched
04:08:39.237421 run    osmo-hlr_10.42.42.2(pid=6723): Launched
04:08:39.356664 run              osmo-stp_10.42.42.5: Starting osmo-stp
04:08:39.914550 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6725): Launched
04:08:40.233397 run    osmo-stp_10.42.42.5(pid=6726): Launched
04:08:40.349063 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
04:08:40.906031 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6728): Launched
04:08:41.020175 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
04:08:41.299322 run               patchelf(pid=6729): Launched
04:08:41.486250 run    osmo-stp_10.42.42.5(pid=6726): ERR: Terminated: ERROR {rc=1}  [trial-2399↪gprs:trx-sysmocell5000+mod-bts0-dynts-ipa↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6726)]
04:08:41.623131 run    osmo-stp_10.42.42.5(pid=6726): stdout: 
| (launched: 2022-12-14_04:08:40.076644)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
| 
| Free Software lives by contribution.  If you use this, please contribute!
|  
04:08:41.759889 run    osmo-stp_10.42.42.5(pid=6726): stderr: 
| �[0;m�[38;5;43m20221214040840264 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214040840264 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214040840264 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214040840265 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214040840265 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214040840265 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214040840265 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use 
04:08:41.892424 run    osmo-stp_10.42.42.5(pid=6726): stdout: 
| (launched: 2022-12-14_04:08:40.076644)
| Copyright (C) 2015-2020 by Harald Welte <laforge@gnumonks.org>
| Contributions by Holger Freyther, Neels Hofmeyr, Pau Espin, Vadim Yanitskiy
| License GPLv2+: GNU GPL Version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
| 
| Free Software lives by contribution.  If you use this, please contribute!
|  
04:08:42.023621 run    osmo-stp_10.42.42.5(pid=6726): stderr: 
| �[0;m�[38;5;43m20221214040840264 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Creating m3ua Server (null):2905 (osmo_ss7.c:2189)
| �[0;m�[38;5;43m20221214040840264 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m Created m3ua server on (null):2905 (osmo_ss7.c:2207)
| �[0;m�[38;5;43m20221214040840264 �[1;32mDLSS7�[0;m�[38;5;43m �[1;32mINFO�[0;m�[38;5;43m (Re)binding m3ua Server to 10.42.42.5:2905 (osmo_ss7.c:2235)
| �[0;m20221214040840265 �[1;33mDLGLOBAL�[0;m �[1;33mNOTICE�[0;m unable to bind socket: 10.42.42.5:2905: Address already in use (socket.c:822)
| % Unable to bind xUA server to IP(s)
| 20221214040840265 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.5:4239: Address already in use (socket.c:935)
| 20221214040840265 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.5:4239 (socket.c:946)
| 20221214040840265 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.5 4239 (telnet_interface.c:99)
| Error binding VTY port
| : Address already in use 
04:08:42.176373 run               patchelf(pid=6729): Terminating (SIGINT)
04:08:42.354949 run               patchelf(pid=6729): Terminated: ok {rc=0}
04:08:42.387748 tst                    iperf3m4.py:8: ERR: Error: osmo-stp_10.42.42.5(pid=6726): Process ended prematurely: osmo-stp_10.42.42.5(pid=6726) [trial-2399↪gprs:trx-sysmocell5000+mod-bts0-dynts-ipa↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6726)]  [trial-2399↪gprs:trx-sysmocell5000+mod-bts0-dynts-ipa↪iperf3m4.py:8]
04:08:42.397730 tst                    iperf3m4.py:8: Test FAILED (10.9 sec)
04:08:42.461928 run    osmo-hlr_10.42.42.2(pid=6723): ERR: Terminated: ERROR {rc=237}  [trial-2399↪gprs:trx-sysmocell5000+mod-bts0-dynts-ipa↪iperf3m4.py↪iperf3m4.py↪osmo-hlr_10.42.42.2↪osmo-hlr_10.42.42.2(pid=6723)]
04:08:42.513895 run    osmo-hlr_10.42.42.2(pid=6723): stdout: 
| (launched: 2022-12-14_04:08:39.081078) 
04:08:42.566678 run    osmo-hlr_10.42.42.2(pid=6723): stderr: 
| �[0;m�[1;31m20221214040839274 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SECURE_DELETE' (db.c:579)
| �[0;m�[1;31m20221214040839274 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'SOUNDEX' (db.c:579)
| �[0;m�[1;31m20221214040839274 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'TEMP_STORE=1' (db.c:579)
| �[0;m�[1;31m20221214040839274 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'THREADSAFE=1' (db.c:579)
| �[0;m�[1;31m20221214040839274 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m SQLite3 compiled with 'USE_URI' (db.c:579)
| �[0;m�[1;31m20221214040839274 �[1;34mDDB�[0;m�[1;31m �[1;34mDEBUG�[0;m�[1;31m Not setting SQL log callback: SQLite3 compiled without support for it (db.c:596)
| �[0;m�[1;31m20221214040839287 �[1;33mDDB�[0;m�[1;31m �[1;33mNOTICE�[0;m�[1;31m Database '/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-2399/run.2022-12-14_03-34-50/gprs:trx-sysmocell5000+mod-bts0-dynts-ipa/iperf3m4.py/osmo-hlr_10.42.42.2/hlr.db' has HLR DB schema version 6 (db.c:636)
| �[0;m20221214040839295 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m unable to bind socket:10.42.42.2:4258: Address already in use (socket.c:935)
| 20221214040839295 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m no suitable addr found for: 10.42.42.2:4258 (socket.c:946)
| 20221214040839295 �[1;31mDLGLOBAL�[0;m �[1;31mERROR�[0;m Cannot bind telnet at 10.42.42.2 4258 (telnet_interface.c:99) 
04:08:42.611622 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6712): Terminating (SIGTERM)
04:08:42.654162 run iperf3-srv_10.42.42.10(pid=6713): Terminating (SIGTERM)
04:08:42.696767 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6714): Terminating (SIGTERM)
04:08:42.739281 run iperf3-srv_10.42.42.10(pid=6715): Terminating (SIGTERM)
04:08:42.781909 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6716): Terminating (SIGTERM)
04:08:42.824344 run iperf3-srv_10.42.42.10(pid=6717): Terminating (SIGTERM)
04:08:42.866867 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6718): Terminating (SIGTERM)
04:08:42.909284 run iperf3-srv_10.42.42.10(pid=6719): Terminating (SIGTERM)
04:08:42.951891 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6722): Terminating (SIGTERM)
04:08:42.994187 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6725): Terminating (SIGTERM)
04:08:43.036096 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6728): Terminating (SIGTERM)
04:08:43.045765 ---      ParallelTerminationStrategy: PID 6712 died...
04:08:43.102331 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6712): Terminated: ok {rc=0}
04:08:43.112290 ---      ParallelTerminationStrategy: PID 6713 died...
04:08:43.168703 run iperf3-srv_10.42.42.10(pid=6713): Terminated {rc=256}
04:08:43.178732 ---      ParallelTerminationStrategy: PID 6714 died...
04:08:43.235208 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6714): Terminated: ok {rc=0}
04:08:43.245334 ---      ParallelTerminationStrategy: PID 6715 died...
04:08:43.301476 run iperf3-srv_10.42.42.10(pid=6715): Terminated {rc=256}
04:08:43.311502 ---      ParallelTerminationStrategy: PID 6716 died...
04:08:43.367856 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6716): Terminated: ok {rc=0}
04:08:43.377905 ---      ParallelTerminationStrategy: PID 6717 died...
04:08:43.434232 run iperf3-srv_10.42.42.10(pid=6717): Terminated {rc=256}
04:08:43.444196 ---      ParallelTerminationStrategy: PID 6718 died...
04:08:43.500492 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=6718): Terminated: ok {rc=0}
04:08:43.510514 ---      ParallelTerminationStrategy: PID 6719 died...
04:08:43.566848 run iperf3-srv_10.42.42.10(pid=6719): Terminated {rc=256}
04:08:43.576870 ---      ParallelTerminationStrategy: PID 6722 died...
04:08:43.633446 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6722): Terminated: ok {rc=0}
04:08:43.643648 ---      ParallelTerminationStrategy: PID 6725 died...
04:08:43.700075 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6725): Terminated: ok {rc=0}
04:08:43.710095 ---      ParallelTerminationStrategy: PID 6728 died...
04:08:43.765996 run pcap-recorder_any(filters='host 10.42.42.6')(pid=6728): Terminated: ok {rc=0}
04:08:43.854343 bus                          /gobi_4: Setting Powered False
04:08:44.960156 bus                          /gobi_4: Setting Powered False
04:08:46.110075 bus                          /gobi_1: Setting Powered False
04:08:47.218970 bus                          /gobi_1: Setting Powered False
04:08:48.326692 bus                          /gobi_0: Setting Powered False
04:08:49.433505 bus                          /gobi_0: Setting Powered False
04:08:50.542660 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
    setup_run_iperf3_test_parallel(4)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 67, in setup_run_iperf3_test_parallel
    ggsn.start()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/obj/ggsn_osmo.py", line 57, in start
    util.change_elf_rpath(binary, util.prepend_library_path(lib), self.run_dir.new_dir('patchelf'))
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 57, in change_elf_rpath
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 214, in launch_sync
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 211, in launch_sync
    self.wait()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 421, in wait
    MainLoop.wait(self.terminated, timeout=timeout)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 108, in wait
    if not self.wait_no_raise(condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 98, in wait_no_raise
    self.poll(may_block=True)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 86, in poll
    self.deferred_handling.handle_queue()
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 33, in handle_queue
    handler(*args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod/osmo-gsm-tester/src/osmo_gsm_tester/testenv.py", line 142, in poll
    raise log_module.Error('Process ended prematurely: %s' % proc.name())
osmo_gsm_tester.core.log.Error: osmo-stp_10.42.42.5(pid=6726): Process ended prematurely: osmo-stp_10.42.42.5(pid=6726) [trial-2399↪gprs:trx-sysmocell5000+mod-bts0-dynts-ipa↪iperf3m4.py:8↪iperf3m4.py↪osmo-stp_10.42.42.5↪osmo-stp_10.42.42.5(pid=6726)]