![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3.py (from gprs_sysmo)
Failing for the past 2 builds
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2064 gprs:sysmo iperf3.py
----------------------------------------------
18:40:11.134089 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.145169 tst gprs:sysmo: Using 1 x bts (candidates: 1)
18:40:11.265542 tst iperf3.py: using LAC 3946
18:40:11.384384 tst iperf3.py: using RAC 121
18:40:11.505400 tst iperf3.py: using CellId 3946
18:40:11.628289 tst iperf3.py: using BVCI 3947
18:40:11.652939 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.664234 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.675311 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.686610 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.697842 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.708852 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.719955 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.730864 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.741928 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:40:12.032669 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
18:40:12.166594 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:40:12.503352 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26843): Launched
18:40:12.785344 run iperf3-srv_10.42.42.10(pid=26844): Launched
18:40:12.984549 tst iperf3.py:8: start network...
18:40:13.120299 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:40:13.630669 run create_hlr_db(pid=26845): Launched
18:40:13.818142 bus /gobi_6: Setting Powered False
18:40:14.839412 run create_hlr_db(pid=26845): Terminated: ok {rc=0}
18:40:15.271067 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26847): Launched
18:40:15.596630 run osmo-hlr_10.42.42.2(pid=26848): Launched
18:40:15.712963 run osmo-stp_10.42.42.5: Starting osmo-stp
18:40:16.274415 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26849): Launched
18:40:16.599266 run osmo-stp_10.42.42.5(pid=26850): Launched
18:40:16.714606 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:40:17.280219 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26851): Launched
18:40:17.396390 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:40:17.679758 run patchelf(pid=26852): Launched
18:40:18.873030 run patchelf(pid=26852): Terminated: ok {rc=0}
18:40:18.992460 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
18:40:19.284962 run osmo-gsm-tester_setcap_net_admin.sh(pid=26853): Launched
18:40:20.477585 run osmo-gsm-tester_setcap_net_admin.sh(pid=26853): Terminated: ok {rc=0}
18:40:20.809183 run osmo-ggsn_10.42.42.6(pid=26856): Launched
18:40:21.037983 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
18:40:21.647198 run pcap-recorder_any(filters='host 10.42.42.7')(pid=26879): Launched
18:40:21.970963 run osmo-sgsn_10.42.42.7(pid=26880): Launched
18:40:22.088996 run osmo-msc_10.42.42.8: Starting osmo-msc
18:40:22.681198 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=26881): Launched
18:40:23.006474 run osmo-msc_10.42.42.8(pid=26882): Launched
18:40:26.178450 run osmo-mgw_10.42.42.3: Starting osmo-mgw
18:40:26.739460 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=26883): Launched
18:40:27.067941 run osmo-mgw_10.42.42.3(pid=26884): Launched
18:40:27.184851 run osmo-mgw_10.42.42.4: Starting osmo-mgw
18:40:27.743389 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=26885): Launched
18:40:28.071920 run osmo-mgw_10.42.42.4(pid=26886): Launched
18:40:28.189068 run osmo-bsc_10.42.42.9: Starting osmo-bsc
18:40:28.874191 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=26887): Launched
18:40:29.205099 run osmo-bsc_10.42.42.9(pid=26888): Launched
18:40:35.483562 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
18:40:36.069004 run rm-remote-dir(pid=26889): Launched
18:40:38.273038 run rm-remote-dir(pid=26889): Terminated: ok {rc=0}
18:40:38.625761 run mk-remote-dir(pid=26890): Launched
18:40:39.825418 run mk-remote-dir(pid=26890): Terminated: ok {rc=0}
18:40:40.122122 run scp-inst-to-remote(pid=26891): Launched
18:40:52.330599 run scp-inst-to-remote(pid=26891): Terminated: ok {rc=0}
18:40:52.655173 run mk-remote-dir(pid=26893): Launched
18:40:53.840404 run mk-remote-dir(pid=26893): Terminated: ok {rc=0}
18:40:54.118457 run scp-cfg-to-remote(pid=26894): Launched
18:40:55.304494 run scp-cfg-to-remote(pid=26894): Terminated: ok {rc=0}
18:40:55.624346 run reload-dsp-firmware(pid=26896): Launched
18:40:57.809607 run reload-dsp-firmware(pid=26896): Terminated: ok {rc=0}
18:40:58.121252 run osmo-bts-sysmo(pid=26897): Launched
18:41:02.436979 tst iperf3.py:8: Waiting for bts to be ready...
18:41:04.081416 run rm-remote-dir(pid=26898): Launched
18:41:06.266407 run rm-remote-dir(pid=26898): Terminated: ok {rc=0}
18:41:06.579910 run mk-remote-dir(pid=26899): Launched
18:41:08.760513 run mk-remote-dir(pid=26899): Terminated: ok {rc=0}
18:41:09.028462 run scp-inst-to-bts(pid=26900): Launched
18:41:22.225483 run scp-inst-to-bts(pid=26900): Terminated: ok {rc=0}
18:41:22.539772 run mk-remote-run-dir(pid=26902): Launched
18:41:24.732505 run mk-remote-run-dir(pid=26902): Terminated: ok {rc=0}
18:41:25.020777 run scp-cfg-to-sysmobts(pid=26903): Launched
18:41:27.215568 run scp-cfg-to-sysmobts(pid=26903): Terminated: ok {rc=0}
18:41:27.551475 run osmo-pcu(pid=26905): Launched
18:41:27.684940 tst iperf3.py: using MSISDN 3951
18:41:27.857140 bus /gobi_6: Setting Powered True
18:41:31.369123 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3951', subscriber_id=1}
18:41:31.659124 bus /gobi_6: Setting Online False
18:41:32.845922 bus /gobi_6: Setting Powered False
18:41:40.232766 bus /gobi_6: Setting Powered True
18:41:41.631903 bus /gobi_6: Setting Online True
18:41:44.171967 tst /gobi_6: Connect to ('901', '70')
18:41:44.433578 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
18:41:44.619708 tst iperf3.py:8: waiting for modems to attach...
18:41:50.060913 tst /gobi_6: Already registered with network ('901', '70')
18:42:02.086847 tst iperf3.py:8: waiting for modems to attach to data services...
18:42:06.665173 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
18:42:06.868413 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
18:42:07.246417 run move_netns(pid=26909): Launched
18:42:08.448341 run move_netns(pid=26909): Terminated: ok {rc=0}
18:42:08.802178 run ifup(pid=26912): Launched
18:42:10.001451 run ifup(pid=26912): Terminated: ok {rc=0}
18:42:10.347898 run dhcp(pid=26915): Launched
18:42:11.530357 run dhcp(pid=26915): Terminated: ok {rc=0}
18:42:11.724176 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
18:42:11.854073 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
18:42:12.236255 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931): Launched
18:42:12.551963 run iperf3-cli_10.42.42.10(pid=26934): Launched
18:43:26.886780 run iperf3-cli_10.42.42.10(pid=26934): ERR: Terminated: ERROR {rc=1} [trial-2064↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=26934)]
18:43:27.028931 run iperf3-cli_10.42.42.10(pid=26934): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
18:43:27.168916 run iperf3-cli_10.42.42.10(pid=26934): stderr:
| (launched: 2022-03-29_18:42:12.439797)
18:43:27.295853 run iperf3-srv_10.42.42.10(pid=26844): Terminating (SIGINT)
18:43:27.642126 run iperf3-srv_10.42.42.10(pid=26844): Terminated {rc=1}
18:43:27.852097 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
18:43:27.913617 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2064↪gprs:sysmo↪iperf3.py:8]
18:43:27.921782 tst iperf3.py:8: Test FAILED (196.8 sec)
18:43:27.959836 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26843): Terminating (SIGTERM)
18:43:28.003684 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26847): Terminating (SIGTERM)
18:43:28.046597 run osmo-hlr_10.42.42.2(pid=26848): Terminating (SIGTERM)
18:43:28.091841 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26849): Terminating (SIGTERM)
18:43:28.140227 run osmo-stp_10.42.42.5(pid=26850): Terminating (SIGTERM)
18:43:28.204874 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26851): Terminating (SIGTERM)
18:43:28.250579 run osmo-ggsn_10.42.42.6(pid=26856): Terminating (SIGTERM)
18:43:28.318615 run pcap-recorder_any(filters='host 10.42.42.7')(pid=26879): Terminating (SIGTERM)
18:43:28.385042 run osmo-sgsn_10.42.42.7(pid=26880): Terminating (SIGTERM)
18:43:28.438397 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=26881): Terminating (SIGTERM)
18:43:28.487140 run osmo-msc_10.42.42.8(pid=26882): Terminating (SIGTERM)
18:43:28.532768 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=26883): Terminating (SIGTERM)
18:43:28.579923 run osmo-mgw_10.42.42.3(pid=26884): Terminating (SIGTERM)
18:43:28.624085 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=26885): Terminating (SIGTERM)
18:43:28.667599 run osmo-mgw_10.42.42.4(pid=26886): Terminating (SIGTERM)
18:43:28.712326 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=26887): Terminating (SIGTERM)
18:43:28.754974 run osmo-bsc_10.42.42.9(pid=26888): Terminating (SIGTERM)
18:43:28.799408 run osmo-bts-sysmo(pid=26897): Terminating (SIGTERM)
18:43:28.843627 run osmo-pcu(pid=26905): Terminating (SIGTERM)
18:43:28.887432 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931): Terminating (SIGTERM)
18:43:29.049282 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGTERM(pid=26955): Launched
18:43:30.142577 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGTERM(pid=26955): Terminated: ok {rc=0}
18:43:30.157057 --- ParallelTerminationStrategy: PID 26843 died...
18:43:30.213297 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26843): Terminated: ok {rc=0}
18:43:30.223497 --- ParallelTerminationStrategy: PID 26847 died...
18:43:30.280120 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26847): Terminated: ok {rc=0}
18:43:30.290396 --- ParallelTerminationStrategy: PID 26848 died...
18:43:30.346740 run osmo-hlr_10.42.42.2(pid=26848): Terminated: ok {rc=0}
18:43:30.356865 --- ParallelTerminationStrategy: PID 26849 died...
18:43:30.413129 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26849): Terminated: ok {rc=0}
18:43:30.423692 --- ParallelTerminationStrategy: PID 26850 died...
18:43:30.479888 run osmo-stp_10.42.42.5(pid=26850): Terminated: ok {rc=0}
18:43:30.490018 --- ParallelTerminationStrategy: PID 26851 died...
18:43:30.546265 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26851): Terminated: ok {rc=0}
18:43:30.556455 --- ParallelTerminationStrategy: PID 26856 died...
18:43:30.612642 run osmo-ggsn_10.42.42.6(pid=26856): Terminated: ok {rc=0}
18:43:30.622974 --- ParallelTerminationStrategy: PID 26879 died...
18:43:30.679318 run pcap-recorder_any(filters='host 10.42.42.7')(pid=26879): Terminated: ok {rc=0}
18:43:30.689498 --- ParallelTerminationStrategy: PID 26880 died...
18:43:30.745950 run osmo-sgsn_10.42.42.7(pid=26880): Terminated: ok {rc=0}
18:43:30.756143 --- ParallelTerminationStrategy: PID 26881 died...
18:43:30.812278 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=26881): Terminated: ok {rc=0}
18:43:30.822500 --- ParallelTerminationStrategy: PID 26882 died...
18:43:30.879020 run osmo-msc_10.42.42.8(pid=26882): Terminated: ok {rc=0}
18:43:30.889226 --- ParallelTerminationStrategy: PID 26883 died...
18:43:30.945578 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=26883): Terminated: ok {rc=0}
18:43:30.955819 --- ParallelTerminationStrategy: PID 26884 died...
18:43:31.012099 run osmo-mgw_10.42.42.3(pid=26884): Terminated {rc=15}
18:43:31.022361 --- ParallelTerminationStrategy: PID 26885 died...
18:43:31.078599 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=26885): Terminated: ok {rc=0}
18:43:31.088888 --- ParallelTerminationStrategy: PID 26886 died...
18:43:31.145241 run osmo-mgw_10.42.42.4(pid=26886): Terminated {rc=15}
18:43:31.155553 --- ParallelTerminationStrategy: PID 26887 died...
18:43:31.211931 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=26887): Terminated: ok {rc=0}
18:43:31.222453 --- ParallelTerminationStrategy: PID 26888 died...
18:43:31.278578 run osmo-bsc_10.42.42.9(pid=26888): Terminated: ok {rc=0}
18:43:31.288830 --- ParallelTerminationStrategy: PID 26897 died...
18:43:31.345338 run osmo-bts-sysmo(pid=26897): Terminated: ok {rc=0}
18:43:31.355710 --- ParallelTerminationStrategy: PID 26905 died...
18:43:31.411854 run osmo-pcu(pid=26905): Terminated: ok {rc=0}
18:43:37.278182 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931): Terminating (SIGINT)
18:43:37.432065 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGINT(pid=26960): Launched
18:43:38.523294 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGINT(pid=26960): Terminated: ok {rc=0}
18:43:44.393943 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931): Terminating (SIGKILL)
18:43:44.547767 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGUSR1(pid=26963): Launched
18:43:45.641991 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGUSR1(pid=26963): Terminated: ok {rc=0}
18:43:45.879384 bus /gobi_6: Setting Online False
18:43:48.910062 bus /gobi_6: Setting Powered False
18:44:03.972907 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'