Skip to content

Failed

gprs:sysmo.iperf3.py (from gprs_sysmo)

Failing for the past 1 build (Since Failed#2063 )
Took 3 min 30 sec.

Stacktrace

'sum_sent'

Standard Output

----------------------------------------------
trial-2063 gprs:sysmo iperf3.py
----------------------------------------------
14:52:59.295047 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.306116 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
14:52:59.428640 tst                        iperf3.py: using LAC 3939
14:52:59.545284 tst                        iperf3.py: using RAC 114
14:52:59.662409 tst                        iperf3.py: using CellId 3939
14:52:59.782121 tst                        iperf3.py: using BVCI 3940
14:52:59.805138 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.815756 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.826118 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.836712 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.847245 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.857680 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.868371 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.878856 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
14:52:59.889541 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
14:53:00.164579 tst                      iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
14:53:00.289703 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
14:53:00.603715 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Launched
14:53:00.864326 run iperf3-srv_10.42.42.10(pid=7380): Launched
14:53:01.046990 tst                      iperf3.py:8: start network...
14:53:01.171095 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
14:53:01.655377 run          create_hlr_db(pid=7381): Launched
14:53:01.827395 bus                          /gobi_6: Setting Powered False
14:53:02.849823 run          create_hlr_db(pid=7381): Terminated: ok {rc=0}
14:53:03.250893 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Launched
14:53:03.554420 run    osmo-hlr_10.42.42.2(pid=7384): Launched
14:53:03.663603 run              osmo-stp_10.42.42.5: Starting osmo-stp
14:53:04.185074 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Launched
14:53:04.488975 run    osmo-stp_10.42.42.5(pid=7386): Launched
14:53:04.595159 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
14:53:05.118502 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Launched
14:53:05.225061 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
14:53:05.488675 run               patchelf(pid=7388): Launched
14:53:06.667431 run               patchelf(pid=7388): Terminated: ok {rc=0}
14:53:06.777548 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
14:53:07.057040 run osmo-gsm-tester_setcap_net_admin.sh(pid=7389): Launched
14:53:08.247348 run osmo-gsm-tester_setcap_net_admin.sh(pid=7389): Terminated: ok {rc=0}
14:53:08.575910 run   osmo-ggsn_10.42.42.6(pid=7392): Launched
14:53:08.799101 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
14:53:09.379533 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Launched
14:53:09.705996 run   osmo-sgsn_10.42.42.7(pid=7410): Launched
14:53:09.825923 run              osmo-msc_10.42.42.8: Starting osmo-msc
14:53:10.420843 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Launched
14:53:10.749992 run    osmo-msc_10.42.42.8(pid=7412): Launched
14:53:13.923969 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
14:53:14.490463 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Launched
14:53:14.818634 run    osmo-mgw_10.42.42.3(pid=7414): Launched
14:53:14.935436 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
14:53:15.490641 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Launched
14:53:15.817442 run    osmo-mgw_10.42.42.4(pid=7416): Launched
14:53:15.934094 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
14:53:16.616082 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Launched
14:53:16.945354 run    osmo-bsc_10.42.42.9(pid=7418): Launched
14:53:23.223754 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
14:53:23.805496 run          rm-remote-dir(pid=7419): Launched
14:53:26.007601 run          rm-remote-dir(pid=7419): Terminated: ok {rc=0}
14:53:26.354965 run          mk-remote-dir(pid=7420): Launched
14:53:27.553974 run          mk-remote-dir(pid=7420): Terminated: ok {rc=0}
14:53:27.848702 run     scp-inst-to-remote(pid=7421): Launched
14:53:40.054041 run     scp-inst-to-remote(pid=7421): Terminated: ok {rc=0}
14:53:40.375519 run          mk-remote-dir(pid=7423): Launched
14:53:41.559384 run          mk-remote-dir(pid=7423): Terminated: ok {rc=0}
14:53:41.833619 run      scp-cfg-to-remote(pid=7424): Launched
14:53:43.030303 run      scp-cfg-to-remote(pid=7424): Terminated: ok {rc=0}
14:53:43.370482 run    reload-dsp-firmware(pid=7426): Launched
14:53:45.567725 run    reload-dsp-firmware(pid=7426): Terminated: ok {rc=0}
14:53:45.900049 run         osmo-bts-sysmo(pid=7427): Launched
14:53:50.214549 tst                      iperf3.py:8: Waiting for bts to be ready...
14:53:51.857035 run          rm-remote-dir(pid=7428): Launched
14:53:54.040240 run          rm-remote-dir(pid=7428): Terminated: ok {rc=0}
14:53:54.353906 run          mk-remote-dir(pid=7429): Launched
14:53:56.536674 run          mk-remote-dir(pid=7429): Terminated: ok {rc=0}
14:53:56.804438 run        scp-inst-to-bts(pid=7430): Launched
14:54:10.001885 run        scp-inst-to-bts(pid=7430): Terminated: ok {rc=0}
14:54:10.315974 run      mk-remote-run-dir(pid=7432): Launched
14:54:12.498382 run      mk-remote-run-dir(pid=7432): Terminated: ok {rc=0}
14:54:12.765911 run    scp-cfg-to-sysmobts(pid=7433): Launched
14:54:14.947582 run    scp-cfg-to-sysmobts(pid=7433): Terminated: ok {rc=0}
14:54:15.261922 run               osmo-pcu(pid=7435): Launched
14:54:15.385379 tst                        iperf3.py: using MSISDN 3940
14:54:15.564610 bus                          /gobi_6: Setting Powered True
14:54:19.048990 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3940', subscriber_id=1}
14:54:19.322831 bus                          /gobi_6: Setting Online False
14:54:20.517371 bus                          /gobi_6: Setting Powered False
14:54:27.288487 bus                          /gobi_6: Setting Powered True
14:54:28.696759 bus                          /gobi_6: Setting Online True
14:54:31.166646 tst                          /gobi_6: Connect to ('901', '70')
14:54:31.444845 tst                          /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G  1  [Jul  5 2017 10:49:17]', Serial: '868323024273648'
14:54:31.646027 tst                      iperf3.py:8: waiting for modems to attach...
14:54:44.244803 tst                          /gobi_6: Already registered with network ('901', '70')
14:54:50.090265 tst                      iperf3.py:8: waiting for modems to attach to data services...
14:55:14.104375 tst                          /gobi_6: context activated {apn='inet46', path='/gobi_6/context4', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
14:55:14.305778 tst                      iperf3.py:8: Setting up data plan for "'/gobi_6/context4'"
14:55:14.682650 run             move_netns(pid=7442): Launched
14:55:15.879244 run             move_netns(pid=7442): Terminated: ok {rc=0}
14:55:16.225805 run                   ifup(pid=7445): Launched
14:55:17.421131 run                   ifup(pid=7445): Terminated: ok {rc=0}
14:55:17.765629 run                   dhcp(pid=7448): Launched
14:55:18.962892 run                   dhcp(pid=7448): Terminated: ok {rc=0}
14:55:19.172461 tst                      iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context4'"
14:55:19.312473 run           iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
14:55:19.714895 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Launched
14:55:20.032224 run iperf3-cli_10.42.42.10(pid=7467): Launched
14:56:28.353069 run iperf3-cli_10.42.42.10(pid=7467): ERR: Terminated: ERROR {rc=1}  [trial-2063↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=7467)]
14:56:28.490608 run iperf3-cli_10.42.42.10(pid=7467): stdout: 
| 				"bits_per_second":	0,
| 				"retransmits":	0,
| 				"omitted":	false
| 			}
| 		}],
| 	"end":	{
| 	},
| 	"error":	"error - control socket has closed unexpectedly"
| }
| child exited with 1 
14:56:28.626915 run iperf3-cli_10.42.42.10(pid=7467): stderr: 
| (launched: 2022-02-24_14:55:19.919964) 
14:56:28.752451 run iperf3-srv_10.42.42.10(pid=7380): Terminating (SIGINT)
14:56:28.926316 run iperf3-srv_10.42.42.10(pid=7380): Terminated {rc=1}
14:56:29.129427 tst                      iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context4'"
14:56:29.188793 tst                      iperf3.py:8: ERR: KeyError: 'sum_sent'  [trial-2063↪gprs:sysmo↪iperf3.py:8]
14:56:29.196330 tst                      iperf3.py:8: Test FAILED (209.9 sec)
14:56:29.234515 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Terminating (SIGTERM)
14:56:29.278421 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Terminating (SIGTERM)
14:56:29.321844 run    osmo-hlr_10.42.42.2(pid=7384): Terminating (SIGTERM)
14:56:29.365725 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Terminating (SIGTERM)
14:56:29.410343 run    osmo-stp_10.42.42.5(pid=7386): Terminating (SIGTERM)
14:56:29.480269 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Terminating (SIGTERM)
14:56:29.536713 run   osmo-ggsn_10.42.42.6(pid=7392): Terminating (SIGTERM)
14:56:29.605153 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Terminating (SIGTERM)
14:56:29.661429 run   osmo-sgsn_10.42.42.7(pid=7410): Terminating (SIGTERM)
14:56:29.710880 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Terminating (SIGTERM)
14:56:29.759276 run    osmo-msc_10.42.42.8(pid=7412): Terminating (SIGTERM)
14:56:29.805931 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Terminating (SIGTERM)
14:56:29.849720 run    osmo-mgw_10.42.42.3(pid=7414): Terminating (SIGTERM)
14:56:29.892989 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Terminating (SIGTERM)
14:56:29.935638 run    osmo-mgw_10.42.42.4(pid=7416): Terminating (SIGTERM)
14:56:29.979178 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Terminating (SIGTERM)
14:56:30.021566 run    osmo-bsc_10.42.42.9(pid=7418): Terminating (SIGTERM)
14:56:30.064127 run         osmo-bts-sysmo(pid=7427): Terminating (SIGTERM)
14:56:30.106568 run               osmo-pcu(pid=7435): Terminating (SIGTERM)
14:56:30.150385 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGTERM)
14:56:30.330809 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGTERM(pid=7491): Launched
14:56:31.430370 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGTERM(pid=7491): Terminated: ok {rc=0}
14:56:31.444735 ---      ParallelTerminationStrategy: PID 7379 died...
14:56:31.501462 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Terminated: ok {rc=0}
14:56:31.511665 ---      ParallelTerminationStrategy: PID 7383 died...
14:56:31.567778 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Terminated: ok {rc=0}
14:56:31.577923 ---      ParallelTerminationStrategy: PID 7384 died...
14:56:31.633998 run    osmo-hlr_10.42.42.2(pid=7384): Terminated: ok {rc=0}
14:56:31.644128 ---      ParallelTerminationStrategy: PID 7385 died...
14:56:31.700390 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Terminated: ok {rc=0}
14:56:31.710451 ---      ParallelTerminationStrategy: PID 7386 died...
14:56:31.766507 run    osmo-stp_10.42.42.5(pid=7386): Terminated: ok {rc=0}
14:56:31.776650 ---      ParallelTerminationStrategy: PID 7387 died...
14:56:31.832559 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Terminated: ok {rc=0}
14:56:31.842682 ---      ParallelTerminationStrategy: PID 7392 died...
14:56:31.898542 run   osmo-ggsn_10.42.42.6(pid=7392): Terminated: ok {rc=0}
14:56:31.908707 ---      ParallelTerminationStrategy: PID 7409 died...
14:56:31.964857 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Terminated: ok {rc=0}
14:56:31.975086 ---      ParallelTerminationStrategy: PID 7410 died...
14:56:32.031173 run   osmo-sgsn_10.42.42.7(pid=7410): Terminated: ok {rc=0}
14:56:32.041501 ---      ParallelTerminationStrategy: PID 7411 died...
14:56:32.098009 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Terminated: ok {rc=0}
14:56:32.108316 ---      ParallelTerminationStrategy: PID 7412 died...
14:56:32.164366 run    osmo-msc_10.42.42.8(pid=7412): Terminated: ok {rc=0}
14:56:32.174811 ---      ParallelTerminationStrategy: PID 7413 died...
14:56:32.230845 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Terminated: ok {rc=0}
14:56:32.241556 ---      ParallelTerminationStrategy: PID 7414 died...
14:56:32.298367 run    osmo-mgw_10.42.42.3(pid=7414): Terminated {rc=15}
14:56:32.308818 ---      ParallelTerminationStrategy: PID 7415 died...
14:56:32.365212 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Terminated: ok {rc=0}
14:56:32.375524 ---      ParallelTerminationStrategy: PID 7416 died...
14:56:32.431502 run    osmo-mgw_10.42.42.4(pid=7416): Terminated {rc=15}
14:56:32.441948 ---      ParallelTerminationStrategy: PID 7417 died...
14:56:32.497975 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Terminated: ok {rc=0}
14:56:32.508584 ---      ParallelTerminationStrategy: PID 7418 died...
14:56:32.565038 run    osmo-bsc_10.42.42.9(pid=7418): Terminated: ok {rc=0}
14:56:32.578391 ---      ParallelTerminationStrategy: PID 7427 died...
14:56:32.635438 run         osmo-bts-sysmo(pid=7427): Terminated: ok {rc=0}
14:56:32.645632 ---      ParallelTerminationStrategy: PID 7435 died...
14:56:32.702106 run               osmo-pcu(pid=7435): Terminated: ok {rc=0}
14:56:38.569626 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGINT)
14:56:38.719506 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGINT(pid=7496): Launched
14:56:39.814286 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGINT(pid=7496): Terminated: ok {rc=0}
14:56:45.685912 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGKILL)
14:56:45.841834 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGUSR1(pid=7499): Launched
14:56:46.937002 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGUSR1(pid=7499): Terminated: ok {rc=0}
14:56:47.210220 bus                          /gobi_6: Setting Online False
14:56:50.050377 bus                          /gobi_6: Setting Powered False
14:57:03.977535 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
    setup_run_iperf3_test_parallel(1)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
    print_results(clients[i], servers[i])
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
    cli.print_results()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
    print_result_node_tcp(self.get_results(), 'client')
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
    sent = result['end']['sum_sent']
KeyError: 'sum_sent'