![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3.py (from gprs_sysmo)
Failing for the past 4 builds
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2066 gprs:sysmo iperf3.py
----------------------------------------------
11:09:40.151011 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.162854 tst gprs:sysmo: Using 1 x bts (candidates: 1)
11:09:40.292881 tst iperf3.py: using LAC 3960
11:09:40.414052 tst iperf3.py: using RAC 135
11:09:40.540455 tst iperf3.py: using CellId 3960
11:09:40.667308 tst iperf3.py: using BVCI 3961
11:09:40.690388 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.700999 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.711521 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.722491 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.733377 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.743942 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.754545 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.765076 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.775863 tst gprs:sysmo: Using 1 x modem (candidates: 4)
11:09:41.058385 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
11:09:41.185415 run iperf3-srv_10.42.42.10: Starting iperf3-srv
11:09:41.506837 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18165): Launched
11:09:41.769631 run iperf3-srv_10.42.42.10(pid=18166): Launched
11:09:41.955061 tst iperf3.py:8: start network...
11:09:42.081532 run osmo-hlr_10.42.42.2: Starting osmo-hlr
11:09:42.564308 run create_hlr_db(pid=18167): Launched
11:09:42.739729 bus /gobi_6: Setting Powered False
11:09:43.762830 run create_hlr_db(pid=18167): Terminated: ok {rc=0}
11:09:44.199605 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18169): Launched
11:09:44.531938 run osmo-hlr_10.42.42.2(pid=18170): Launched
11:09:44.649997 run osmo-stp_10.42.42.5: Starting osmo-stp
11:09:45.216929 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18171): Launched
11:09:45.545247 run osmo-stp_10.42.42.5(pid=18172): Launched
11:09:45.661780 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:09:46.230687 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18173): Launched
11:09:46.347869 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:09:46.635775 run patchelf(pid=18174): Launched
11:09:47.829846 run patchelf(pid=18174): Terminated: ok {rc=0}
11:09:47.950444 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
11:09:48.242233 run osmo-gsm-tester_setcap_net_admin.sh(pid=18175): Launched
11:09:49.436422 run osmo-gsm-tester_setcap_net_admin.sh(pid=18175): Terminated: ok {rc=0}
11:09:49.769926 run osmo-ggsn_10.42.42.6(pid=18178): Launched
11:09:50.015955 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
11:09:50.568880 run pcap-recorder_any(filters='host 10.42.42.7')(pid=18199): Launched
11:09:50.877878 run osmo-sgsn_10.42.42.7(pid=18200): Launched
11:09:50.991756 run osmo-msc_10.42.42.8: Starting osmo-msc
11:09:51.555760 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=18201): Launched
11:09:51.866583 run osmo-msc_10.42.42.8(pid=18202): Launched
11:09:55.030595 run osmo-mgw_10.42.42.3: Starting osmo-mgw
11:09:55.561812 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=18203): Launched
11:09:55.869861 run osmo-mgw_10.42.42.3(pid=18204): Launched
11:09:55.979390 run osmo-mgw_10.42.42.4: Starting osmo-mgw
11:09:56.504332 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=18205): Launched
11:09:56.813018 run osmo-mgw_10.42.42.4(pid=18206): Launched
11:09:56.922801 run osmo-bsc_10.42.42.9: Starting osmo-bsc
11:09:57.566072 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=18207): Launched
11:09:57.875600 run osmo-bsc_10.42.42.9(pid=18208): Launched
11:10:04.137256 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
11:10:04.684880 run rm-remote-dir(pid=18212): Launched
11:10:06.876153 run rm-remote-dir(pid=18212): Terminated: ok {rc=0}
11:10:07.206339 run mk-remote-dir(pid=18213): Launched
11:10:08.395439 run mk-remote-dir(pid=18213): Terminated: ok {rc=0}
11:10:08.673794 run scp-inst-to-remote(pid=18214): Launched
11:10:20.884496 run scp-inst-to-remote(pid=18214): Terminated: ok {rc=0}
11:10:21.233752 run mk-remote-dir(pid=18216): Launched
11:10:22.433115 run mk-remote-dir(pid=18216): Terminated: ok {rc=0}
11:10:22.729212 run scp-cfg-to-remote(pid=18217): Launched
11:10:23.927340 run scp-cfg-to-remote(pid=18217): Terminated: ok {rc=0}
11:10:24.268243 run reload-dsp-firmware(pid=18219): Launched
11:10:26.454219 run reload-dsp-firmware(pid=18219): Terminated: ok {rc=0}
11:10:26.765639 run osmo-bts-sysmo(pid=18220): Launched
11:10:31.083253 tst iperf3.py:8: Waiting for bts to be ready...
11:10:32.727433 run rm-remote-dir(pid=18221): Launched
11:10:34.921105 run rm-remote-dir(pid=18221): Terminated: ok {rc=0}
11:10:35.258294 run mk-remote-dir(pid=18222): Launched
11:10:37.451736 run mk-remote-dir(pid=18222): Terminated: ok {rc=0}
11:10:37.739163 run scp-inst-to-bts(pid=18223): Launched
11:10:50.942442 run scp-inst-to-bts(pid=18223): Terminated: ok {rc=0}
11:10:51.258504 run mk-remote-run-dir(pid=18226): Launched
11:10:53.440178 run mk-remote-run-dir(pid=18226): Terminated: ok {rc=0}
11:10:53.710222 run scp-cfg-to-sysmobts(pid=18227): Launched
11:10:55.902001 run scp-cfg-to-sysmobts(pid=18227): Terminated: ok {rc=0}
11:10:56.240722 run osmo-pcu(pid=18229): Launched
11:10:56.370817 tst iperf3.py: using MSISDN 3973
11:10:56.539235 bus /gobi_6: Setting Powered True
11:11:00.041469 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3973', subscriber_id=1}
11:11:00.335661 bus /gobi_6: Setting Online False
11:11:01.523505 bus /gobi_6: Setting Powered False
11:11:08.291484 bus /gobi_6: Setting Powered True
11:11:09.686082 bus /gobi_6: Setting Online True
11:11:12.159195 tst /gobi_6: Connect to ('901', '70')
11:11:12.443572 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
11:11:12.646312 tst iperf3.py:8: waiting for modems to attach...
11:11:19.740207 tst /gobi_6: Already registered with network ('901', '70')
11:11:31.178934 tst iperf3.py:8: waiting for modems to attach to data services...
11:11:49.085649 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
11:11:49.276496 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
11:11:49.641006 run move_netns(pid=18232): Launched
11:11:50.838801 run move_netns(pid=18232): Terminated: ok {rc=0}
11:11:51.192394 run ifup(pid=18235): Launched
11:11:52.390583 run ifup(pid=18235): Terminated: ok {rc=0}
11:11:52.732224 run dhcp(pid=18238): Launched
11:11:53.917242 run dhcp(pid=18238): Terminated: ok {rc=0}
11:11:54.112190 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
11:11:54.243176 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
11:11:54.626346 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254): Launched
11:11:54.941828 run iperf3-cli_10.42.42.10(pid=18257): Launched
11:13:03.261392 run iperf3-cli_10.42.42.10(pid=18257): ERR: Terminated: ERROR {rc=1} [trial-2066↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=18257)]
11:13:03.394757 run iperf3-cli_10.42.42.10(pid=18257): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
11:13:03.527048 run iperf3-cli_10.42.42.10(pid=18257): stderr:
| (launched: 2022-09-07_11:11:54.831390)
11:13:03.647871 run iperf3-srv_10.42.42.10(pid=18166): Terminating (SIGINT)
11:13:03.994111 run iperf3-srv_10.42.42.10(pid=18166): Terminated {rc=1}
11:13:04.198689 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
11:13:04.260102 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2066↪gprs:sysmo↪iperf3.py:8]
11:13:04.267803 tst iperf3.py:8: Test FAILED (204.2 sec)
11:13:04.305776 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18165): Terminating (SIGTERM)
11:13:04.350639 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18169): Terminating (SIGTERM)
11:13:04.393669 run osmo-hlr_10.42.42.2(pid=18170): Terminating (SIGTERM)
11:13:04.437420 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18171): Terminating (SIGTERM)
11:13:04.483368 run osmo-stp_10.42.42.5(pid=18172): Terminating (SIGTERM)
11:13:04.538455 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18173): Terminating (SIGTERM)
11:13:04.586419 run osmo-ggsn_10.42.42.6(pid=18178): Terminating (SIGTERM)
11:13:04.660198 run pcap-recorder_any(filters='host 10.42.42.7')(pid=18199): Terminating (SIGTERM)
11:13:04.749713 run osmo-sgsn_10.42.42.7(pid=18200): Terminating (SIGTERM)
11:13:04.796942 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=18201): Terminating (SIGTERM)
11:13:04.848612 run osmo-msc_10.42.42.8(pid=18202): Terminating (SIGTERM)
11:13:04.896300 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=18203): Terminating (SIGTERM)
11:13:04.940384 run osmo-mgw_10.42.42.3(pid=18204): Terminating (SIGTERM)
11:13:04.983412 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=18205): Terminating (SIGTERM)
11:13:05.031041 run osmo-mgw_10.42.42.4(pid=18206): Terminating (SIGTERM)
11:13:05.075266 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=18207): Terminating (SIGTERM)
11:13:05.121905 run osmo-bsc_10.42.42.9(pid=18208): Terminating (SIGTERM)
11:13:05.166820 run osmo-bts-sysmo(pid=18220): Terminating (SIGTERM)
11:13:05.212297 run osmo-pcu(pid=18229): Terminating (SIGTERM)
11:13:05.260373 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254): Terminating (SIGTERM)
11:13:05.415159 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGTERM(pid=18280): Launched
11:13:06.506836 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGTERM(pid=18280): Terminated: ok {rc=0}
11:13:06.521025 --- ParallelTerminationStrategy: PID 18165 died...
11:13:06.576623 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18165): Terminated: ok {rc=0}
11:13:06.586641 --- ParallelTerminationStrategy: PID 18169 died...
11:13:06.641141 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18169): Terminated: ok {rc=0}
11:13:06.651634 --- ParallelTerminationStrategy: PID 18170 died...
11:13:06.706117 run osmo-hlr_10.42.42.2(pid=18170): Terminated: ok {rc=0}
11:13:06.716243 --- ParallelTerminationStrategy: PID 18171 died...
11:13:06.770648 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18171): Terminated: ok {rc=0}
11:13:06.780959 --- ParallelTerminationStrategy: PID 18172 died...
11:13:06.835508 run osmo-stp_10.42.42.5(pid=18172): Terminated: ok {rc=0}
11:13:06.845538 --- ParallelTerminationStrategy: PID 18173 died...
11:13:06.899703 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18173): Terminated: ok {rc=0}
11:13:06.909685 --- ParallelTerminationStrategy: PID 18178 died...
11:13:06.964652 run osmo-ggsn_10.42.42.6(pid=18178): Terminated: ok {rc=0}
11:13:06.974964 --- ParallelTerminationStrategy: PID 18199 died...
11:13:07.029274 run pcap-recorder_any(filters='host 10.42.42.7')(pid=18199): Terminated: ok {rc=0}
11:13:07.039649 --- ParallelTerminationStrategy: PID 18200 died...
11:13:07.094530 run osmo-sgsn_10.42.42.7(pid=18200): Terminated: ok {rc=0}
11:13:07.104740 --- ParallelTerminationStrategy: PID 18201 died...
11:13:07.159233 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=18201): Terminated: ok {rc=0}
11:13:07.169283 --- ParallelTerminationStrategy: PID 18202 died...
11:13:07.224371 run osmo-msc_10.42.42.8(pid=18202): Terminated: ok {rc=0}
11:13:07.234258 --- ParallelTerminationStrategy: PID 18203 died...
11:13:07.289633 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=18203): Terminated: ok {rc=0}
11:13:07.300058 --- ParallelTerminationStrategy: PID 18204 died...
11:13:07.354817 run osmo-mgw_10.42.42.3(pid=18204): Terminated {rc=15}
11:13:07.365038 --- ParallelTerminationStrategy: PID 18205 died...
11:13:07.419596 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=18205): Terminated: ok {rc=0}
11:13:07.429821 --- ParallelTerminationStrategy: PID 18206 died...
11:13:07.484611 run osmo-mgw_10.42.42.4(pid=18206): Terminated {rc=15}
11:13:07.494912 --- ParallelTerminationStrategy: PID 18207 died...
11:13:07.550140 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=18207): Terminated: ok {rc=0}
11:13:07.560547 --- ParallelTerminationStrategy: PID 18208 died...
11:13:07.615006 run osmo-bsc_10.42.42.9(pid=18208): Terminated: ok {rc=0}
11:13:07.625344 --- ParallelTerminationStrategy: PID 18220 died...
11:13:07.680609 run osmo-bts-sysmo(pid=18220): Terminated: ok {rc=0}
11:13:07.690874 --- ParallelTerminationStrategy: PID 18229 died...
11:13:07.745495 run osmo-pcu(pid=18229): Terminated: ok {rc=0}
11:13:13.611982 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254): Terminating (SIGINT)
11:13:13.763409 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGINT(pid=18284): Launched
11:13:14.855010 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGINT(pid=18284): Terminated: ok {rc=0}
11:13:20.724863 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254): Terminating (SIGKILL)
11:13:20.876485 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGUSR1(pid=18287): Launched
11:13:21.968589 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGUSR1(pid=18287): Terminated: ok {rc=0}
11:13:22.205598 bus /gobi_6: Setting Online False
11:13:25.253697 bus /gobi_6: Setting Powered False
11:13:40.094141 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'