![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3.py (from gprs_sysmo)
Failing for the past 6 builds
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2068 gprs:sysmo iperf3.py
----------------------------------------------
17:10:37.312172 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:10:37.323887 tst gprs:sysmo: Using 1 x bts (candidates: 1)
17:10:37.455565 tst iperf3.py: using LAC 4435
17:10:37.582146 tst iperf3.py: using RAC 100
17:10:37.715369 tst iperf3.py: using CellId 4435
17:10:37.840498 tst iperf3.py: using BVCI 4436
17:10:37.865527 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:10:37.876673 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:10:37.887834 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:10:37.899277 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:10:37.910078 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:10:37.921389 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:10:37.932788 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:10:37.943827 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:10:37.955073 tst gprs:sysmo: Using 1 x modem (candidates: 4)
17:10:38.254573 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
17:10:38.392161 run iperf3-srv_10.42.42.10: Starting iperf3-srv
17:10:38.733983 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31787): Launched
17:10:39.019665 run iperf3-srv_10.42.42.10(pid=31788): Launched
17:10:39.221026 tst iperf3.py:8: start network...
17:10:39.358550 run osmo-hlr_10.42.42.2: Starting osmo-hlr
17:10:39.878890 run create_hlr_db(pid=31790): Launched
17:10:40.068682 bus /gobi_6: Setting Powered False
17:10:41.090088 run create_hlr_db(pid=31790): Terminated: ok {rc=0}
17:10:41.503705 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31792): Launched
17:10:41.813147 run osmo-hlr_10.42.42.2(pid=31793): Launched
17:10:41.923007 run osmo-stp_10.42.42.5: Starting osmo-stp
17:10:42.455123 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31794): Launched
17:10:42.763762 run osmo-stp_10.42.42.5(pid=31795): Launched
17:10:42.871439 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
17:10:43.404809 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31796): Launched
17:10:43.513416 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
17:10:43.785558 run patchelf(pid=31797): Launched
17:10:44.967841 run patchelf(pid=31797): Terminated: ok {rc=0}
17:10:45.080434 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
17:10:45.356487 run osmo-gsm-tester_setcap_net_admin.sh(pid=31798): Launched
17:10:46.538857 run osmo-gsm-tester_setcap_net_admin.sh(pid=31798): Terminated: ok {rc=0}
17:10:46.853823 run osmo-ggsn_10.42.42.6(pid=31801): Launched
17:10:47.120927 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
17:10:47.699288 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31822): Launched
17:10:48.029985 run osmo-sgsn_10.42.42.7(pid=31823): Launched
17:10:48.150851 run osmo-msc_10.42.42.8: Starting osmo-msc
17:10:48.753675 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31824): Launched
17:10:49.086589 run osmo-msc_10.42.42.8(pid=31825): Launched
17:10:52.262955 run osmo-mgw_10.42.42.3: Starting osmo-mgw
17:10:52.833903 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31826): Launched
17:10:53.164956 run osmo-mgw_10.42.42.3(pid=31827): Launched
17:10:53.282408 run osmo-mgw_10.42.42.4: Starting osmo-mgw
17:10:53.841229 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31828): Launched
17:10:54.172140 run osmo-mgw_10.42.42.4(pid=31829): Launched
17:10:54.289196 run osmo-bsc_10.42.42.9: Starting osmo-bsc
17:10:54.975790 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31830): Launched
17:10:55.305688 run osmo-bsc_10.42.42.9(pid=31831): Launched
17:11:01.583998 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
17:11:02.173940 run rm-remote-dir(pid=31832): Launched
17:11:04.378303 run rm-remote-dir(pid=31832): Terminated: ok {rc=0}
17:11:04.733080 run mk-remote-dir(pid=31833): Launched
17:11:05.934473 run mk-remote-dir(pid=31833): Terminated: ok {rc=0}
17:11:06.231835 run scp-inst-to-remote(pid=31834): Launched
17:11:18.439711 run scp-inst-to-remote(pid=31834): Terminated: ok {rc=0}
17:11:18.766448 run mk-remote-dir(pid=31836): Launched
17:11:19.952783 run mk-remote-dir(pid=31836): Terminated: ok {rc=0}
17:11:20.233246 run scp-cfg-to-remote(pid=31837): Launched
17:11:22.420708 run scp-cfg-to-remote(pid=31837): Terminated: ok {rc=0}
17:11:22.742668 run reload-dsp-firmware(pid=31839): Launched
17:11:24.928627 run reload-dsp-firmware(pid=31839): Terminated: ok {rc=0}
17:11:25.241782 run osmo-bts-sysmo(pid=31840): Launched
17:11:29.537005 tst iperf3.py:8: Waiting for bts to be ready...
17:11:31.144743 run rm-remote-dir(pid=31841): Launched
17:11:33.328591 run rm-remote-dir(pid=31841): Terminated: ok {rc=0}
17:11:33.643857 run mk-remote-dir(pid=31846): Launched
17:11:35.836774 run mk-remote-dir(pid=31846): Terminated: ok {rc=0}
17:11:36.127453 run scp-inst-to-bts(pid=31847): Launched
17:11:49.327384 run scp-inst-to-bts(pid=31847): Terminated: ok {rc=0}
17:11:49.644949 run mk-remote-run-dir(pid=31849): Launched
17:11:51.827320 run mk-remote-run-dir(pid=31849): Terminated: ok {rc=0}
17:11:52.099748 run scp-cfg-to-sysmobts(pid=31850): Launched
17:11:54.282497 run scp-cfg-to-sysmobts(pid=31850): Terminated: ok {rc=0}
17:11:54.599728 run osmo-pcu(pid=31852): Launched
17:11:54.724718 tst iperf3.py: using MSISDN 4030
17:11:54.906584 bus /gobi_6: Setting Powered True
17:11:58.405403 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='4030', subscriber_id=1}
17:11:58.695966 bus /gobi_6: Setting Online False
17:11:59.881102 bus /gobi_6: Setting Powered False
17:12:07.230781 bus /gobi_6: Setting Powered True
17:12:08.618759 bus /gobi_6: Setting Online True
17:12:11.132273 tst /gobi_6: Connect to ('901', '70')
17:12:11.395585 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
17:12:11.581999 tst iperf3.py:8: waiting for modems to attach...
17:12:19.910549 tst /gobi_6: Already registered with network ('901', '70')
17:12:30.024583 tst iperf3.py:8: waiting for modems to attach to data services...
17:12:46.187870 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
17:12:46.393081 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
17:12:46.775436 run move_netns(pid=31859): Launched
17:12:47.975185 run move_netns(pid=31859): Terminated: ok {rc=0}
17:12:48.328710 run ifup(pid=31863): Launched
17:12:49.527011 run ifup(pid=31863): Terminated: ok {rc=0}
17:12:49.876521 run dhcp(pid=31867): Launched
17:12:51.073787 run dhcp(pid=31867): Terminated: ok {rc=0}
17:12:51.283907 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
17:12:51.424498 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
17:12:51.830197 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883): Launched
17:12:52.167884 run iperf3-cli_10.42.42.10(pid=31886): Launched
17:14:06.492824 run iperf3-cli_10.42.42.10(pid=31886): ERR: Terminated: ERROR {rc=1} [trial-2068↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=31886)]
17:14:06.622840 run iperf3-cli_10.42.42.10(pid=31886): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
17:14:06.751643 run iperf3-cli_10.42.42.10(pid=31886): stderr:
| (launched: 2022-12-01_17:12:52.050181)
17:14:06.869822 run iperf3-srv_10.42.42.10(pid=31788): Terminating (SIGINT)
17:14:07.202097 run iperf3-srv_10.42.42.10(pid=31788): Terminated {rc=1}
17:14:07.390922 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
17:14:07.446458 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2068↪gprs:sysmo↪iperf3.py:8]
17:14:07.453615 tst iperf3.py:8: Test FAILED (210.2 sec)
17:14:07.491054 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31787): Terminating (SIGTERM)
17:14:07.534913 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31792): Terminating (SIGTERM)
17:14:07.576256 run osmo-hlr_10.42.42.2(pid=31793): Terminating (SIGTERM)
17:14:07.619588 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31794): Terminating (SIGTERM)
17:14:07.664036 run osmo-stp_10.42.42.5(pid=31795): Terminating (SIGTERM)
17:14:07.706263 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31796): Terminating (SIGTERM)
17:14:07.749724 run osmo-ggsn_10.42.42.6(pid=31801): Terminating (SIGTERM)
17:14:07.795794 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31822): Terminating (SIGTERM)
17:14:07.850129 run osmo-sgsn_10.42.42.7(pid=31823): Terminating (SIGTERM)
17:14:07.900371 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31824): Terminating (SIGTERM)
17:14:07.947835 run osmo-msc_10.42.42.8(pid=31825): Terminating (SIGTERM)
17:14:07.997171 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31826): Terminating (SIGTERM)
17:14:08.043397 run osmo-mgw_10.42.42.3(pid=31827): Terminating (SIGTERM)
17:14:08.103204 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31828): Terminating (SIGTERM)
17:14:08.145716 run osmo-mgw_10.42.42.4(pid=31829): Terminating (SIGTERM)
17:14:08.187687 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31830): Terminating (SIGTERM)
17:14:08.230280 run osmo-bsc_10.42.42.9(pid=31831): Terminating (SIGTERM)
17:14:08.273600 run osmo-bts-sysmo(pid=31840): Terminating (SIGTERM)
17:14:08.317237 run osmo-pcu(pid=31852): Terminating (SIGTERM)
17:14:08.360245 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883): Terminating (SIGTERM)
17:14:08.537540 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883)-killSignals.SIGTERM(pid=31909): Launched
17:14:09.634182 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883)-killSignals.SIGTERM(pid=31909): Terminated: ok {rc=0}
17:14:09.648819 --- ParallelTerminationStrategy: PID 31787 died...
17:14:09.706042 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31787): Terminated: ok {rc=0}
17:14:09.716447 --- ParallelTerminationStrategy: PID 31792 died...
17:14:09.773222 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31792): Terminated: ok {rc=0}
17:14:09.783904 --- ParallelTerminationStrategy: PID 31793 died...
17:14:09.840578 run osmo-hlr_10.42.42.2(pid=31793): Terminated: ok {rc=0}
17:14:09.850670 --- ParallelTerminationStrategy: PID 31794 died...
17:14:09.907498 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31794): Terminated: ok {rc=0}
17:14:09.917734 --- ParallelTerminationStrategy: PID 31795 died...
17:14:09.974112 run osmo-stp_10.42.42.5(pid=31795): Terminated: ok {rc=0}
17:14:09.984304 --- ParallelTerminationStrategy: PID 31796 died...
17:14:10.041256 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31796): Terminated: ok {rc=0}
17:14:10.051550 --- ParallelTerminationStrategy: PID 31801 died...
17:14:10.108065 run osmo-ggsn_10.42.42.6(pid=31801): Terminated: ok {rc=0}
17:14:10.118354 --- ParallelTerminationStrategy: PID 31822 died...
17:14:10.174914 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31822): Terminated: ok {rc=0}
17:14:10.185162 --- ParallelTerminationStrategy: PID 31823 died...
17:14:10.241661 run osmo-sgsn_10.42.42.7(pid=31823): Terminated: ok {rc=0}
17:14:10.252090 --- ParallelTerminationStrategy: PID 31824 died...
17:14:10.308695 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31824): Terminated: ok {rc=0}
17:14:10.319018 --- ParallelTerminationStrategy: PID 31825 died...
17:14:10.375776 run osmo-msc_10.42.42.8(pid=31825): Terminated: ok {rc=0}
17:14:10.386027 --- ParallelTerminationStrategy: PID 31826 died...
17:14:10.442771 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31826): Terminated: ok {rc=0}
17:14:10.453098 --- ParallelTerminationStrategy: PID 31827 died...
17:14:10.510050 run osmo-mgw_10.42.42.3(pid=31827): Terminated {rc=15}
17:14:10.520553 --- ParallelTerminationStrategy: PID 31828 died...
17:14:10.577167 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31828): Terminated: ok {rc=0}
17:14:10.587743 --- ParallelTerminationStrategy: PID 31829 died...
17:14:10.644612 run osmo-mgw_10.42.42.4(pid=31829): Terminated {rc=15}
17:14:10.654996 --- ParallelTerminationStrategy: PID 31830 died...
17:14:10.711639 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31830): Terminated: ok {rc=0}
17:14:10.722285 --- ParallelTerminationStrategy: PID 31831 died...
17:14:10.778994 run osmo-bsc_10.42.42.9(pid=31831): Terminated: ok {rc=0}
17:14:10.789549 --- ParallelTerminationStrategy: PID 31840 died...
17:14:10.846642 run osmo-bts-sysmo(pid=31840): Terminated: ok {rc=0}
17:14:10.857206 --- ParallelTerminationStrategy: PID 31852 died...
17:14:10.915850 run osmo-pcu(pid=31852): Terminated: ok {rc=0}
17:14:16.784408 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883): Terminating (SIGINT)
17:14:16.943779 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883)-killSignals.SIGINT(pid=31914): Launched
17:14:18.040413 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883)-killSignals.SIGINT(pid=31914): Terminated: ok {rc=0}
17:14:23.913002 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883): Terminating (SIGKILL)
17:14:24.071672 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883)-killSignals.SIGUSR1(pid=31917): Launched
17:14:25.167551 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31883)-killSignals.SIGUSR1(pid=31917): Terminated: ok {rc=0}
17:14:25.419016 bus /gobi_6: Setting Online False
17:14:28.466901 bus /gobi_6: Setting Powered False
17:14:36.207753 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'