![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3.py (from gprs_sysmo)
Failing for the past 5 builds
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2067 gprs:sysmo iperf3.py
----------------------------------------------
12:07:29.145703 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.157557 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:07:29.296637 tst iperf3.py: using LAC 3967
12:07:29.434199 tst iperf3.py: using RAC 142
12:07:29.561792 tst iperf3.py: using CellId 3967
12:07:29.687413 tst iperf3.py: using BVCI 3968
12:07:29.712348 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.723596 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.734593 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.745843 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.756939 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.768117 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.779890 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.791146 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.802411 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:07:30.110289 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
12:07:30.247539 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:07:30.587851 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21080): Launched
12:07:30.870677 run iperf3-srv_10.42.42.10(pid=21081): Launched
12:07:31.069887 tst iperf3.py:8: start network...
12:07:31.206246 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:07:31.722526 run create_hlr_db(pid=21082): Launched
12:07:31.911420 bus /gobi_6: Setting Powered False
12:07:32.932936 run create_hlr_db(pid=21082): Terminated: ok {rc=0}
12:07:33.344987 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21084): Launched
12:07:33.658625 run osmo-hlr_10.42.42.2(pid=21085): Launched
12:07:33.769519 run osmo-stp_10.42.42.5: Starting osmo-stp
12:07:34.301497 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21086): Launched
12:07:34.609515 run osmo-stp_10.42.42.5(pid=21087): Launched
12:07:34.718457 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:07:35.251825 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21088): Launched
12:07:35.359842 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:07:35.628423 run patchelf(pid=21089): Launched
12:07:36.810507 run patchelf(pid=21089): Terminated: ok {rc=0}
12:07:36.922418 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:07:37.200812 run osmo-gsm-tester_setcap_net_admin.sh(pid=21090): Launched
12:07:38.381919 run osmo-gsm-tester_setcap_net_admin.sh(pid=21090): Terminated: ok {rc=0}
12:07:38.691826 run osmo-ggsn_10.42.42.6(pid=21093): Launched
12:07:38.881853 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:07:39.517255 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21112): Launched
12:07:39.848863 run osmo-sgsn_10.42.42.7(pid=21113): Launched
12:07:39.968604 run osmo-msc_10.42.42.8: Starting osmo-msc
12:07:40.569195 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21114): Launched
12:07:40.900610 run osmo-msc_10.42.42.8(pid=21115): Launched
12:07:44.074331 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:07:44.644470 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21116): Launched
12:07:44.974852 run osmo-mgw_10.42.42.3(pid=21117): Launched
12:07:45.092476 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:07:45.652529 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21118): Launched
12:07:45.983248 run osmo-mgw_10.42.42.4(pid=21119): Launched
12:07:46.100402 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:07:46.786232 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21120): Launched
12:07:47.117596 run osmo-bsc_10.42.42.9(pid=21121): Launched
12:07:53.394793 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:07:53.976672 run rm-remote-dir(pid=21122): Launched
12:07:56.177730 run rm-remote-dir(pid=21122): Terminated: ok {rc=0}
12:07:56.528012 run mk-remote-dir(pid=21123): Launched
12:07:57.727297 run mk-remote-dir(pid=21123): Terminated: ok {rc=0}
12:07:58.023013 run scp-inst-to-remote(pid=21124): Launched
12:08:10.239817 run scp-inst-to-remote(pid=21124): Terminated: ok {rc=0}
12:08:10.585408 run mk-remote-dir(pid=21126): Launched
12:08:11.782785 run mk-remote-dir(pid=21126): Terminated: ok {rc=0}
12:08:12.077989 run scp-cfg-to-remote(pid=21127): Launched
12:08:13.275035 run scp-cfg-to-remote(pid=21127): Terminated: ok {rc=0}
12:08:13.613996 run reload-dsp-firmware(pid=21129): Launched
12:08:15.808861 run reload-dsp-firmware(pid=21129): Terminated: ok {rc=0}
12:08:16.140713 run osmo-bts-sysmo(pid=21130): Launched
12:08:20.453197 tst iperf3.py:8: Waiting for bts to be ready...
12:08:22.059958 run rm-remote-dir(pid=21131): Launched
12:08:24.250338 run rm-remote-dir(pid=21131): Terminated: ok {rc=0}
12:08:24.586611 run mk-remote-dir(pid=21132): Launched
12:08:26.781268 run mk-remote-dir(pid=21132): Terminated: ok {rc=0}
12:08:27.068976 run scp-inst-to-bts(pid=21133): Launched
12:08:40.275783 run scp-inst-to-bts(pid=21133): Terminated: ok {rc=0}
12:08:40.610679 run mk-remote-run-dir(pid=21135): Launched
12:08:42.804368 run mk-remote-run-dir(pid=21135): Terminated: ok {rc=0}
12:08:43.091956 run scp-cfg-to-sysmobts(pid=21136): Launched
12:08:45.285300 run scp-cfg-to-sysmobts(pid=21136): Terminated: ok {rc=0}
12:08:45.620744 run osmo-pcu(pid=21138): Launched
12:08:45.750911 tst iperf3.py: using MSISDN 3984
12:08:45.922960 bus /gobi_6: Setting Powered True
12:08:49.429825 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3984', subscriber_id=1}
12:08:49.717794 bus /gobi_6: Setting Online False
12:08:50.913511 bus /gobi_6: Setting Powered False
12:08:58.203273 bus /gobi_6: Setting Powered True
12:08:59.594063 bus /gobi_6: Setting Online True
12:09:02.073782 tst /gobi_6: Connect to ('901', '70')
12:09:02.340491 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:09:02.526347 tst iperf3.py:8: waiting for modems to attach...
12:09:09.560292 tst /gobi_6: Already registered with network ('901', '70')
12:09:19.999606 tst iperf3.py:8: waiting for modems to attach to data services...
12:09:29.878579 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:09:30.078172 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
12:09:30.449116 run move_netns(pid=21141): Launched
12:09:31.634903 run move_netns(pid=21141): Terminated: ok {rc=0}
12:09:31.963116 run ifup(pid=21144): Launched
12:09:33.144766 run ifup(pid=21144): Terminated: ok {rc=0}
12:09:33.469109 run dhcp(pid=21147): Launched
12:09:34.650991 run dhcp(pid=21147): Terminated: ok {rc=0}
12:09:34.845412 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
12:09:34.976010 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:09:35.362272 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163): Launched
12:09:35.701174 run iperf3-cli_10.42.42.10(pid=21166): Launched
12:10:50.028284 run iperf3-cli_10.42.42.10(pid=21166): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21166)]
12:10:50.159238 run iperf3-cli_10.42.42.10(pid=21166): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:10:50.291191 run iperf3-cli_10.42.42.10(pid=21166): stderr:
| (launched: 2022-09-08_12:09:35.580850)
12:10:50.410857 run iperf3-srv_10.42.42.10(pid=21081): Terminating (SIGINT)
12:10:50.746264 run iperf3-srv_10.42.42.10(pid=21081): Terminated {rc=1}
12:10:50.939216 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
12:10:50.997637 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2067↪gprs:sysmo↪iperf3.py:8]
12:10:51.004954 tst iperf3.py:8: Test FAILED (201.9 sec)
12:10:51.041340 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21080): Terminating (SIGTERM)
12:10:51.083698 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21084): Terminating (SIGTERM)
12:10:51.125351 run osmo-hlr_10.42.42.2(pid=21085): Terminating (SIGTERM)
12:10:51.168488 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21086): Terminating (SIGTERM)
12:10:51.211952 run osmo-stp_10.42.42.5(pid=21087): Terminating (SIGTERM)
12:10:51.255032 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21088): Terminating (SIGTERM)
12:10:51.297651 run osmo-ggsn_10.42.42.6(pid=21093): Terminating (SIGTERM)
12:10:51.345774 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21112): Terminating (SIGTERM)
12:10:51.400532 run osmo-sgsn_10.42.42.7(pid=21113): Terminating (SIGTERM)
12:10:51.444372 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21114): Terminating (SIGTERM)
12:10:51.501156 run osmo-msc_10.42.42.8(pid=21115): Terminating (SIGTERM)
12:10:51.545532 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21116): Terminating (SIGTERM)
12:10:51.591116 run osmo-mgw_10.42.42.3(pid=21117): Terminating (SIGTERM)
12:10:51.633825 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21118): Terminating (SIGTERM)
12:10:51.674793 run osmo-mgw_10.42.42.4(pid=21119): Terminating (SIGTERM)
12:10:51.715960 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21120): Terminating (SIGTERM)
12:10:51.756670 run osmo-bsc_10.42.42.9(pid=21121): Terminating (SIGTERM)
12:10:51.797947 run osmo-bts-sysmo(pid=21130): Terminating (SIGTERM)
12:10:51.838788 run osmo-pcu(pid=21138): Terminating (SIGTERM)
12:10:51.880025 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163): Terminating (SIGTERM)
12:10:52.026290 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGTERM(pid=21189): Launched
12:10:53.117144 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGTERM(pid=21189): Terminated: ok {rc=0}
12:10:53.131054 --- ParallelTerminationStrategy: PID 21080 died...
12:10:53.185326 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21080): Terminated: ok {rc=0}
12:10:53.195214 --- ParallelTerminationStrategy: PID 21084 died...
12:10:53.249133 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21084): Terminated: ok {rc=0}
12:10:53.259072 --- ParallelTerminationStrategy: PID 21085 died...
12:10:53.313380 run osmo-hlr_10.42.42.2(pid=21085): Terminated: ok {rc=0}
12:10:53.323566 --- ParallelTerminationStrategy: PID 21086 died...
12:10:53.377620 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21086): Terminated: ok {rc=0}
12:10:53.387527 --- ParallelTerminationStrategy: PID 21087 died...
12:10:53.441045 run osmo-stp_10.42.42.5(pid=21087): Terminated: ok {rc=0}
12:10:53.450778 --- ParallelTerminationStrategy: PID 21088 died...
12:10:53.504478 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21088): Terminated: ok {rc=0}
12:10:53.514221 --- ParallelTerminationStrategy: PID 21093 died...
12:10:53.568129 run osmo-ggsn_10.42.42.6(pid=21093): Terminated: ok {rc=0}
12:10:53.577920 --- ParallelTerminationStrategy: PID 21112 died...
12:10:53.632376 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21112): Terminated: ok {rc=0}
12:10:53.642114 --- ParallelTerminationStrategy: PID 21113 died...
12:10:53.695818 run osmo-sgsn_10.42.42.7(pid=21113): Terminated: ok {rc=0}
12:10:53.705617 --- ParallelTerminationStrategy: PID 21114 died...
12:10:53.759701 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21114): Terminated: ok {rc=0}
12:10:53.769491 --- ParallelTerminationStrategy: PID 21115 died...
12:10:53.823173 run osmo-msc_10.42.42.8(pid=21115): Terminated: ok {rc=0}
12:10:53.832992 --- ParallelTerminationStrategy: PID 21116 died...
12:10:53.887078 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21116): Terminated: ok {rc=0}
12:10:53.896926 --- ParallelTerminationStrategy: PID 21117 died...
12:10:53.951468 run osmo-mgw_10.42.42.3(pid=21117): Terminated {rc=15}
12:10:53.961232 --- ParallelTerminationStrategy: PID 21118 died...
12:10:54.015091 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21118): Terminated: ok {rc=0}
12:10:54.025043 --- ParallelTerminationStrategy: PID 21119 died...
12:10:54.078882 run osmo-mgw_10.42.42.4(pid=21119): Terminated {rc=15}
12:10:54.088672 --- ParallelTerminationStrategy: PID 21120 died...
12:10:54.142528 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21120): Terminated: ok {rc=0}
12:10:54.152391 --- ParallelTerminationStrategy: PID 21121 died...
12:10:54.206450 run osmo-bsc_10.42.42.9(pid=21121): Terminated: ok {rc=0}
12:10:54.216464 --- ParallelTerminationStrategy: PID 21130 died...
12:10:54.270792 run osmo-bts-sysmo(pid=21130): Terminated: ok {rc=0}
12:10:54.280562 --- ParallelTerminationStrategy: PID 21138 died...
12:10:54.334830 run osmo-pcu(pid=21138): Terminated: ok {rc=0}
12:11:00.201338 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163): Terminating (SIGINT)
12:11:00.353349 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGINT(pid=21195): Launched
12:11:01.443752 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGINT(pid=21195): Terminated: ok {rc=0}
12:11:07.313241 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163): Terminating (SIGKILL)
12:11:07.466405 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGUSR1(pid=21199): Launched
12:11:08.561439 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGUSR1(pid=21199): Terminated: ok {rc=0}
12:11:08.801385 bus /gobi_6: Setting Online False
12:11:11.824484 bus /gobi_6: Setting Powered False
12:11:27.221491 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'