![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3.py (from gprs_sysmo)
Failing for the past 1 build
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2063 gprs:sysmo iperf3.py
----------------------------------------------
14:52:59.295047 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.306116 tst gprs:sysmo: Using 1 x bts (candidates: 1)
14:52:59.428640 tst iperf3.py: using LAC 3939
14:52:59.545284 tst iperf3.py: using RAC 114
14:52:59.662409 tst iperf3.py: using CellId 3939
14:52:59.782121 tst iperf3.py: using BVCI 3940
14:52:59.805138 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.815756 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.826118 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.836712 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.847245 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.857680 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.868371 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.878856 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.889541 tst gprs:sysmo: Using 1 x modem (candidates: 4)
14:53:00.164579 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
14:53:00.289703 run iperf3-srv_10.42.42.10: Starting iperf3-srv
14:53:00.603715 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Launched
14:53:00.864326 run iperf3-srv_10.42.42.10(pid=7380): Launched
14:53:01.046990 tst iperf3.py:8: start network...
14:53:01.171095 run osmo-hlr_10.42.42.2: Starting osmo-hlr
14:53:01.655377 run create_hlr_db(pid=7381): Launched
14:53:01.827395 bus /gobi_6: Setting Powered False
14:53:02.849823 run create_hlr_db(pid=7381): Terminated: ok {rc=0}
14:53:03.250893 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Launched
14:53:03.554420 run osmo-hlr_10.42.42.2(pid=7384): Launched
14:53:03.663603 run osmo-stp_10.42.42.5: Starting osmo-stp
14:53:04.185074 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Launched
14:53:04.488975 run osmo-stp_10.42.42.5(pid=7386): Launched
14:53:04.595159 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
14:53:05.118502 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Launched
14:53:05.225061 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
14:53:05.488675 run patchelf(pid=7388): Launched
14:53:06.667431 run patchelf(pid=7388): Terminated: ok {rc=0}
14:53:06.777548 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
14:53:07.057040 run osmo-gsm-tester_setcap_net_admin.sh(pid=7389): Launched
14:53:08.247348 run osmo-gsm-tester_setcap_net_admin.sh(pid=7389): Terminated: ok {rc=0}
14:53:08.575910 run osmo-ggsn_10.42.42.6(pid=7392): Launched
14:53:08.799101 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
14:53:09.379533 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Launched
14:53:09.705996 run osmo-sgsn_10.42.42.7(pid=7410): Launched
14:53:09.825923 run osmo-msc_10.42.42.8: Starting osmo-msc
14:53:10.420843 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Launched
14:53:10.749992 run osmo-msc_10.42.42.8(pid=7412): Launched
14:53:13.923969 run osmo-mgw_10.42.42.3: Starting osmo-mgw
14:53:14.490463 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Launched
14:53:14.818634 run osmo-mgw_10.42.42.3(pid=7414): Launched
14:53:14.935436 run osmo-mgw_10.42.42.4: Starting osmo-mgw
14:53:15.490641 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Launched
14:53:15.817442 run osmo-mgw_10.42.42.4(pid=7416): Launched
14:53:15.934094 run osmo-bsc_10.42.42.9: Starting osmo-bsc
14:53:16.616082 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Launched
14:53:16.945354 run osmo-bsc_10.42.42.9(pid=7418): Launched
14:53:23.223754 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
14:53:23.805496 run rm-remote-dir(pid=7419): Launched
14:53:26.007601 run rm-remote-dir(pid=7419): Terminated: ok {rc=0}
14:53:26.354965 run mk-remote-dir(pid=7420): Launched
14:53:27.553974 run mk-remote-dir(pid=7420): Terminated: ok {rc=0}
14:53:27.848702 run scp-inst-to-remote(pid=7421): Launched
14:53:40.054041 run scp-inst-to-remote(pid=7421): Terminated: ok {rc=0}
14:53:40.375519 run mk-remote-dir(pid=7423): Launched
14:53:41.559384 run mk-remote-dir(pid=7423): Terminated: ok {rc=0}
14:53:41.833619 run scp-cfg-to-remote(pid=7424): Launched
14:53:43.030303 run scp-cfg-to-remote(pid=7424): Terminated: ok {rc=0}
14:53:43.370482 run reload-dsp-firmware(pid=7426): Launched
14:53:45.567725 run reload-dsp-firmware(pid=7426): Terminated: ok {rc=0}
14:53:45.900049 run osmo-bts-sysmo(pid=7427): Launched
14:53:50.214549 tst iperf3.py:8: Waiting for bts to be ready...
14:53:51.857035 run rm-remote-dir(pid=7428): Launched
14:53:54.040240 run rm-remote-dir(pid=7428): Terminated: ok {rc=0}
14:53:54.353906 run mk-remote-dir(pid=7429): Launched
14:53:56.536674 run mk-remote-dir(pid=7429): Terminated: ok {rc=0}
14:53:56.804438 run scp-inst-to-bts(pid=7430): Launched
14:54:10.001885 run scp-inst-to-bts(pid=7430): Terminated: ok {rc=0}
14:54:10.315974 run mk-remote-run-dir(pid=7432): Launched
14:54:12.498382 run mk-remote-run-dir(pid=7432): Terminated: ok {rc=0}
14:54:12.765911 run scp-cfg-to-sysmobts(pid=7433): Launched
14:54:14.947582 run scp-cfg-to-sysmobts(pid=7433): Terminated: ok {rc=0}
14:54:15.261922 run osmo-pcu(pid=7435): Launched
14:54:15.385379 tst iperf3.py: using MSISDN 3940
14:54:15.564610 bus /gobi_6: Setting Powered True
14:54:19.048990 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3940', subscriber_id=1}
14:54:19.322831 bus /gobi_6: Setting Online False
14:54:20.517371 bus /gobi_6: Setting Powered False
14:54:27.288487 bus /gobi_6: Setting Powered True
14:54:28.696759 bus /gobi_6: Setting Online True
14:54:31.166646 tst /gobi_6: Connect to ('901', '70')
14:54:31.444845 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
14:54:31.646027 tst iperf3.py:8: waiting for modems to attach...
14:54:44.244803 tst /gobi_6: Already registered with network ('901', '70')
14:54:50.090265 tst iperf3.py:8: waiting for modems to attach to data services...
14:55:14.104375 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context4', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
14:55:14.305778 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context4'"
14:55:14.682650 run move_netns(pid=7442): Launched
14:55:15.879244 run move_netns(pid=7442): Terminated: ok {rc=0}
14:55:16.225805 run ifup(pid=7445): Launched
14:55:17.421131 run ifup(pid=7445): Terminated: ok {rc=0}
14:55:17.765629 run dhcp(pid=7448): Launched
14:55:18.962892 run dhcp(pid=7448): Terminated: ok {rc=0}
14:55:19.172461 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context4'"
14:55:19.312473 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
14:55:19.714895 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Launched
14:55:20.032224 run iperf3-cli_10.42.42.10(pid=7467): Launched
14:56:28.353069 run iperf3-cli_10.42.42.10(pid=7467): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=7467)]
14:56:28.490608 run iperf3-cli_10.42.42.10(pid=7467): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
14:56:28.626915 run iperf3-cli_10.42.42.10(pid=7467): stderr:
| (launched: 2022-02-24_14:55:19.919964)
14:56:28.752451 run iperf3-srv_10.42.42.10(pid=7380): Terminating (SIGINT)
14:56:28.926316 run iperf3-srv_10.42.42.10(pid=7380): Terminated {rc=1}
14:56:29.129427 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context4'"
14:56:29.188793 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2063↪gprs:sysmo↪iperf3.py:8]
14:56:29.196330 tst iperf3.py:8: Test FAILED (209.9 sec)
14:56:29.234515 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Terminating (SIGTERM)
14:56:29.278421 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Terminating (SIGTERM)
14:56:29.321844 run osmo-hlr_10.42.42.2(pid=7384): Terminating (SIGTERM)
14:56:29.365725 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Terminating (SIGTERM)
14:56:29.410343 run osmo-stp_10.42.42.5(pid=7386): Terminating (SIGTERM)
14:56:29.480269 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Terminating (SIGTERM)
14:56:29.536713 run osmo-ggsn_10.42.42.6(pid=7392): Terminating (SIGTERM)
14:56:29.605153 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Terminating (SIGTERM)
14:56:29.661429 run osmo-sgsn_10.42.42.7(pid=7410): Terminating (SIGTERM)
14:56:29.710880 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Terminating (SIGTERM)
14:56:29.759276 run osmo-msc_10.42.42.8(pid=7412): Terminating (SIGTERM)
14:56:29.805931 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Terminating (SIGTERM)
14:56:29.849720 run osmo-mgw_10.42.42.3(pid=7414): Terminating (SIGTERM)
14:56:29.892989 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Terminating (SIGTERM)
14:56:29.935638 run osmo-mgw_10.42.42.4(pid=7416): Terminating (SIGTERM)
14:56:29.979178 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Terminating (SIGTERM)
14:56:30.021566 run osmo-bsc_10.42.42.9(pid=7418): Terminating (SIGTERM)
14:56:30.064127 run osmo-bts-sysmo(pid=7427): Terminating (SIGTERM)
14:56:30.106568 run osmo-pcu(pid=7435): Terminating (SIGTERM)
14:56:30.150385 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGTERM)
14:56:30.330809 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGTERM(pid=7491): Launched
14:56:31.430370 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGTERM(pid=7491): Terminated: ok {rc=0}
14:56:31.444735 --- ParallelTerminationStrategy: PID 7379 died...
14:56:31.501462 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Terminated: ok {rc=0}
14:56:31.511665 --- ParallelTerminationStrategy: PID 7383 died...
14:56:31.567778 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Terminated: ok {rc=0}
14:56:31.577923 --- ParallelTerminationStrategy: PID 7384 died...
14:56:31.633998 run osmo-hlr_10.42.42.2(pid=7384): Terminated: ok {rc=0}
14:56:31.644128 --- ParallelTerminationStrategy: PID 7385 died...
14:56:31.700390 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Terminated: ok {rc=0}
14:56:31.710451 --- ParallelTerminationStrategy: PID 7386 died...
14:56:31.766507 run osmo-stp_10.42.42.5(pid=7386): Terminated: ok {rc=0}
14:56:31.776650 --- ParallelTerminationStrategy: PID 7387 died...
14:56:31.832559 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Terminated: ok {rc=0}
14:56:31.842682 --- ParallelTerminationStrategy: PID 7392 died...
14:56:31.898542 run osmo-ggsn_10.42.42.6(pid=7392): Terminated: ok {rc=0}
14:56:31.908707 --- ParallelTerminationStrategy: PID 7409 died...
14:56:31.964857 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Terminated: ok {rc=0}
14:56:31.975086 --- ParallelTerminationStrategy: PID 7410 died...
14:56:32.031173 run osmo-sgsn_10.42.42.7(pid=7410): Terminated: ok {rc=0}
14:56:32.041501 --- ParallelTerminationStrategy: PID 7411 died...
14:56:32.098009 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Terminated: ok {rc=0}
14:56:32.108316 --- ParallelTerminationStrategy: PID 7412 died...
14:56:32.164366 run osmo-msc_10.42.42.8(pid=7412): Terminated: ok {rc=0}
14:56:32.174811 --- ParallelTerminationStrategy: PID 7413 died...
14:56:32.230845 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Terminated: ok {rc=0}
14:56:32.241556 --- ParallelTerminationStrategy: PID 7414 died...
14:56:32.298367 run osmo-mgw_10.42.42.3(pid=7414): Terminated {rc=15}
14:56:32.308818 --- ParallelTerminationStrategy: PID 7415 died...
14:56:32.365212 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Terminated: ok {rc=0}
14:56:32.375524 --- ParallelTerminationStrategy: PID 7416 died...
14:56:32.431502 run osmo-mgw_10.42.42.4(pid=7416): Terminated {rc=15}
14:56:32.441948 --- ParallelTerminationStrategy: PID 7417 died...
14:56:32.497975 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Terminated: ok {rc=0}
14:56:32.508584 --- ParallelTerminationStrategy: PID 7418 died...
14:56:32.565038 run osmo-bsc_10.42.42.9(pid=7418): Terminated: ok {rc=0}
14:56:32.578391 --- ParallelTerminationStrategy: PID 7427 died...
14:56:32.635438 run osmo-bts-sysmo(pid=7427): Terminated: ok {rc=0}
14:56:32.645632 --- ParallelTerminationStrategy: PID 7435 died...
14:56:32.702106 run osmo-pcu(pid=7435): Terminated: ok {rc=0}
14:56:38.569626 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGINT)
14:56:38.719506 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGINT(pid=7496): Launched
14:56:39.814286 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGINT(pid=7496): Terminated: ok {rc=0}
14:56:45.685912 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGKILL)
14:56:45.841834 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGUSR1(pid=7499): Launched
14:56:46.937002 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGUSR1(pid=7499): Terminated: ok {rc=0}
14:56:47.210220 bus /gobi_6: Setting Online False
14:56:50.050377 bus /gobi_6: Setting Powered False
14:57:03.977535 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
setup_run_iperf3_test_parallel(1)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'