![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3.py (from gprs_sysmo)
Failing for the past 3 builds
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2065 gprs:sysmo iperf3.py
----------------------------------------------
12:18:51.153797 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.165368 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:18:51.294519 tst iperf3.py: using LAC 3953
12:18:51.416572 tst iperf3.py: using RAC 128
12:18:51.540823 tst iperf3.py: using CellId 3953
12:18:51.658910 tst iperf3.py: using BVCI 3954
12:18:51.682124 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.692966 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.703595 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.714878 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.726079 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.737572 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.749109 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.760662 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.772167 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:18:52.068298 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
12:18:52.204122 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:18:52.543601 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31702): Launched
12:18:52.825180 run iperf3-srv_10.42.42.10(pid=31703): Launched
12:18:53.024353 tst iperf3.py:8: start network...
12:18:53.160529 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:18:53.672351 run create_hlr_db(pid=31704): Launched
12:18:53.847555 bus /gobi_6: Setting Powered False
12:18:54.869246 run create_hlr_db(pid=31704): Terminated: ok {rc=0}
12:18:55.277499 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31706): Launched
12:18:55.586029 run osmo-hlr_10.42.42.2(pid=31707): Launched
12:18:55.695860 run osmo-stp_10.42.42.5: Starting osmo-stp
12:18:56.224755 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31708): Launched
12:18:56.532767 run osmo-stp_10.42.42.5(pid=31709): Launched
12:18:56.640857 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:18:57.173462 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Launched
12:18:57.283228 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:18:57.552453 run patchelf(pid=31711): Launched
12:18:58.734617 run patchelf(pid=31711): Terminated: ok {rc=0}
12:18:58.846809 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:18:59.119294 run osmo-gsm-tester_setcap_net_admin.sh(pid=31712): Launched
12:19:00.301194 run osmo-gsm-tester_setcap_net_admin.sh(pid=31712): Terminated: ok {rc=0}
12:19:00.612263 run osmo-ggsn_10.42.42.6(pid=31715): Launched
12:19:00.884053 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:19:01.456971 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31732): Launched
12:19:01.787450 run osmo-sgsn_10.42.42.7(pid=31733): Launched
12:19:01.906356 run osmo-msc_10.42.42.8: Starting osmo-msc
12:19:02.503036 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31734): Launched
12:19:02.830055 run osmo-msc_10.42.42.8(pid=31735): Launched
12:19:06.003792 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:19:06.567176 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31736): Launched
12:19:06.893418 run osmo-mgw_10.42.42.3(pid=31737): Launched
12:19:07.009175 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:19:07.564633 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31738): Launched
12:19:07.893002 run osmo-mgw_10.42.42.4(pid=31739): Launched
12:19:08.008856 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:19:08.691321 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31740): Launched
12:19:09.019609 run osmo-bsc_10.42.42.9(pid=31741): Launched
12:19:15.285085 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:19:15.828571 run rm-remote-dir(pid=31742): Launched
12:19:18.019954 run rm-remote-dir(pid=31742): Terminated: ok {rc=0}
12:19:18.347624 run mk-remote-dir(pid=31743): Launched
12:19:19.534458 run mk-remote-dir(pid=31743): Terminated: ok {rc=0}
12:19:19.809904 run scp-inst-to-remote(pid=31744): Launched
12:19:32.017616 run scp-inst-to-remote(pid=31744): Terminated: ok {rc=0}
12:19:32.361792 run mk-remote-dir(pid=31746): Launched
12:19:33.558084 run mk-remote-dir(pid=31746): Terminated: ok {rc=0}
12:19:33.852709 run scp-cfg-to-remote(pid=31747): Launched
12:19:35.048154 run scp-cfg-to-remote(pid=31747): Terminated: ok {rc=0}
12:19:35.387677 run reload-dsp-firmware(pid=31749): Launched
12:19:37.583762 run reload-dsp-firmware(pid=31749): Terminated: ok {rc=0}
12:19:37.913876 run osmo-bts-sysmo(pid=31750): Launched
12:19:42.227388 tst iperf3.py:8: Waiting for bts to be ready...
12:19:43.869414 run rm-remote-dir(pid=31751): Launched
12:19:46.062837 run rm-remote-dir(pid=31751): Terminated: ok {rc=0}
12:19:46.396659 run mk-remote-dir(pid=31752): Launched
12:19:48.578355 run mk-remote-dir(pid=31752): Terminated: ok {rc=0}
12:19:48.844194 run scp-inst-to-bts(pid=31753): Launched
12:20:02.049550 run scp-inst-to-bts(pid=31753): Terminated: ok {rc=0}
12:20:02.382594 run mk-remote-run-dir(pid=31759): Launched
12:20:04.565993 run mk-remote-run-dir(pid=31759): Terminated: ok {rc=0}
12:20:04.832096 run scp-cfg-to-sysmobts(pid=31760): Launched
12:20:07.012406 run scp-cfg-to-sysmobts(pid=31760): Terminated: ok {rc=0}
12:20:07.326074 run osmo-pcu(pid=31762): Launched
12:20:07.453699 tst iperf3.py: using MSISDN 3962
12:20:07.627424 bus /gobi_6: Setting Powered True
12:20:11.113278 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3962', subscriber_id=1}
12:20:11.401124 bus /gobi_6: Setting Online False
12:20:12.586456 bus /gobi_6: Setting Powered False
12:20:19.313328 bus /gobi_6: Setting Powered True
12:20:20.707048 bus /gobi_6: Setting Online True
12:20:23.127677 tst /gobi_6: Connect to ('901', '70')
12:20:23.398352 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:20:23.582843 tst iperf3.py:8: waiting for modems to attach...
12:20:37.605716 tst /gobi_6: Already registered with network ('901', '70')
12:20:43.025968 tst iperf3.py:8: waiting for modems to attach to data services...
12:20:45.993107 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:20:46.177645 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
12:20:46.529453 run move_netns(pid=31766): Launched
12:20:47.710907 run move_netns(pid=31766): Terminated: ok {rc=0}
12:20:48.036421 run ifup(pid=31769): Launched
12:20:49.218521 run ifup(pid=31769): Terminated: ok {rc=0}
12:20:49.551562 run dhcp(pid=31772): Launched
12:20:50.733333 run dhcp(pid=31772): Terminated: ok {rc=0}
12:20:50.926298 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
12:20:51.055797 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:20:51.429755 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Launched
12:20:51.741394 run iperf3-cli_10.42.42.10(pid=31792): Launched
12:22:00.065028 run iperf3-cli_10.42.42.10(pid=31792): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=31792)]
12:22:00.203671 run iperf3-cli_10.42.42.10(pid=31792): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:22:00.343343 run iperf3-cli_10.42.42.10(pid=31792): stderr:
| (launched: 2022-06-18_12:20:51.632024)
12:22:00.470363 run iperf3-srv_10.42.42.10(pid=31703): Terminating (SIGINT)
12:22:00.814070 run iperf3-srv_10.42.42.10(pid=31703): Terminated {rc=1}
12:22:01.021317 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
12:22:01.082054 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2065↪gprs:sysmo↪iperf3.py:8]
12:22:01.089648 tst iperf3.py:8: Test FAILED (190.0 sec)
12:22:01.127271 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31702): Terminating (SIGTERM)
12:22:01.170450 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31706): Terminating (SIGTERM)
12:22:01.213432 run osmo-hlr_10.42.42.2(pid=31707): Terminating (SIGTERM)
12:22:01.257764 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31708): Terminating (SIGTERM)
12:22:01.303785 run osmo-stp_10.42.42.5(pid=31709): Terminating (SIGTERM)
12:22:01.355729 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Terminating (SIGTERM)
12:22:01.409377 run osmo-ggsn_10.42.42.6(pid=31715): Terminating (SIGTERM)
12:22:01.490349 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31732): Terminating (SIGTERM)
12:22:01.546249 run osmo-sgsn_10.42.42.7(pid=31733): Terminating (SIGTERM)
12:22:01.592368 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31734): Terminating (SIGTERM)
12:22:01.644488 run osmo-msc_10.42.42.8(pid=31735): Terminating (SIGTERM)
12:22:01.690141 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31736): Terminating (SIGTERM)
12:22:01.735610 run osmo-mgw_10.42.42.3(pid=31737): Terminating (SIGTERM)
12:22:01.777393 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31738): Terminating (SIGTERM)
12:22:01.818671 run osmo-mgw_10.42.42.4(pid=31739): Terminating (SIGTERM)
12:22:01.860046 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31740): Terminating (SIGTERM)
12:22:01.900829 run osmo-bsc_10.42.42.9(pid=31741): Terminating (SIGTERM)
12:22:01.941667 run osmo-bts-sysmo(pid=31750): Terminating (SIGTERM)
12:22:01.983059 run osmo-pcu(pid=31762): Terminating (SIGTERM)
12:22:02.025531 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Terminating (SIGTERM)
12:22:02.173577 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGTERM(pid=31812): Launched
12:22:03.264996 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGTERM(pid=31812): Terminated: ok {rc=0}
12:22:03.278834 --- ParallelTerminationStrategy: PID 31702 died...
12:22:03.333412 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31702): Terminated: ok {rc=0}
12:22:03.343504 --- ParallelTerminationStrategy: PID 31706 died...
12:22:03.397860 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31706): Terminated: ok {rc=0}
12:22:03.407953 --- ParallelTerminationStrategy: PID 31707 died...
12:22:03.461989 run osmo-hlr_10.42.42.2(pid=31707): Terminated: ok {rc=0}
12:22:03.471530 --- ParallelTerminationStrategy: PID 31708 died...
12:22:03.525168 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31708): Terminated: ok {rc=0}
12:22:03.535001 --- ParallelTerminationStrategy: PID 31709 died...
12:22:03.588592 run osmo-stp_10.42.42.5(pid=31709): Terminated: ok {rc=0}
12:22:03.598221 --- ParallelTerminationStrategy: PID 31710 died...
12:22:03.652360 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Terminated: ok {rc=0}
12:22:03.661925 --- ParallelTerminationStrategy: PID 31715 died...
12:22:03.715773 run osmo-ggsn_10.42.42.6(pid=31715): Terminated: ok {rc=0}
12:22:03.725438 --- ParallelTerminationStrategy: PID 31732 died...
12:22:03.779965 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31732): Terminated: ok {rc=0}
12:22:03.789671 --- ParallelTerminationStrategy: PID 31733 died...
12:22:03.844136 run osmo-sgsn_10.42.42.7(pid=31733): Terminated: ok {rc=0}
12:22:03.854027 --- ParallelTerminationStrategy: PID 31734 died...
12:22:03.908033 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31734): Terminated: ok {rc=0}
12:22:03.918026 --- ParallelTerminationStrategy: PID 31735 died...
12:22:03.971891 run osmo-msc_10.42.42.8(pid=31735): Terminated: ok {rc=0}
12:22:03.981891 --- ParallelTerminationStrategy: PID 31736 died...
12:22:04.035714 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31736): Terminated: ok {rc=0}
12:22:04.045688 --- ParallelTerminationStrategy: PID 31737 died...
12:22:04.099929 run osmo-mgw_10.42.42.3(pid=31737): Terminated {rc=15}
12:22:04.109914 --- ParallelTerminationStrategy: PID 31738 died...
12:22:04.163849 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31738): Terminated: ok {rc=0}
12:22:04.173707 --- ParallelTerminationStrategy: PID 31739 died...
12:22:04.227438 run osmo-mgw_10.42.42.4(pid=31739): Terminated {rc=15}
12:22:04.237395 --- ParallelTerminationStrategy: PID 31740 died...
12:22:04.291435 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31740): Terminated: ok {rc=0}
12:22:04.301420 --- ParallelTerminationStrategy: PID 31741 died...
12:22:04.355337 run osmo-bsc_10.42.42.9(pid=31741): Terminated: ok {rc=0}
12:22:04.365446 --- ParallelTerminationStrategy: PID 31750 died...
12:22:04.419775 run osmo-bts-sysmo(pid=31750): Terminated: ok {rc=0}
12:22:04.429834 --- ParallelTerminationStrategy: PID 31762 died...
12:22:04.484832 run osmo-pcu(pid=31762): Terminated: ok {rc=0}
12:22:10.351736 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Terminating (SIGINT)
12:22:10.502301 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGINT(pid=31817): Launched
12:22:11.592511 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGINT(pid=31817): Terminated: ok {rc=0}
12:22:17.462515 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Terminating (SIGKILL)
12:22:17.609518 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGUSR1(pid=31820): Launched
12:22:18.700506 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGUSR1(pid=31820): Terminated: ok {rc=0}
12:22:18.932014 bus /gobi_6: Setting Online False
12:22:21.972358 bus /gobi_6: Setting Powered False
12:22:39.046993 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'