Skip to content

Failed

gprs:sysmo.iperf3.py (from gprs_sysmo)

Failing for the past 2 builds (Since Failed#2063 )
Took 3 min 17 sec.

Stacktrace

'sum_sent'

Standard Output

----------------------------------------------
trial-2064 gprs:sysmo iperf3.py
----------------------------------------------
18:40:11.134089 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.145169 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
18:40:11.265542 tst                        iperf3.py: using LAC 3946
18:40:11.384384 tst                        iperf3.py: using RAC 121
18:40:11.505400 tst                        iperf3.py: using CellId 3946
18:40:11.628289 tst                        iperf3.py: using BVCI 3947
18:40:11.652939 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.664234 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.675311 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.686610 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.697842 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.708852 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.719955 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.730864 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:40:11.741928 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
18:40:12.032669 tst                      iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
18:40:12.166594 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
18:40:12.503352 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26843): Launched
18:40:12.785344 run iperf3-srv_10.42.42.10(pid=26844): Launched
18:40:12.984549 tst                      iperf3.py:8: start network...
18:40:13.120299 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
18:40:13.630669 run         create_hlr_db(pid=26845): Launched
18:40:13.818142 bus                          /gobi_6: Setting Powered False
18:40:14.839412 run         create_hlr_db(pid=26845): Terminated: ok {rc=0}
18:40:15.271067 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26847): Launched
18:40:15.596630 run   osmo-hlr_10.42.42.2(pid=26848): Launched
18:40:15.712963 run              osmo-stp_10.42.42.5: Starting osmo-stp
18:40:16.274415 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26849): Launched
18:40:16.599266 run   osmo-stp_10.42.42.5(pid=26850): Launched
18:40:16.714606 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:40:17.280219 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26851): Launched
18:40:17.396390 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:40:17.679758 run              patchelf(pid=26852): Launched
18:40:18.873030 run              patchelf(pid=26852): Terminated: ok {rc=0}
18:40:18.992460 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
18:40:19.284962 run osmo-gsm-tester_setcap_net_admin.sh(pid=26853): Launched
18:40:20.477585 run osmo-gsm-tester_setcap_net_admin.sh(pid=26853): Terminated: ok {rc=0}
18:40:20.809183 run  osmo-ggsn_10.42.42.6(pid=26856): Launched
18:40:21.037983 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
18:40:21.647198 run pcap-recorder_any(filters='host 10.42.42.7')(pid=26879): Launched
18:40:21.970963 run  osmo-sgsn_10.42.42.7(pid=26880): Launched
18:40:22.088996 run              osmo-msc_10.42.42.8: Starting osmo-msc
18:40:22.681198 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=26881): Launched
18:40:23.006474 run   osmo-msc_10.42.42.8(pid=26882): Launched
18:40:26.178450 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
18:40:26.739460 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=26883): Launched
18:40:27.067941 run   osmo-mgw_10.42.42.3(pid=26884): Launched
18:40:27.184851 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
18:40:27.743389 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=26885): Launched
18:40:28.071920 run   osmo-mgw_10.42.42.4(pid=26886): Launched
18:40:28.189068 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
18:40:28.874191 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=26887): Launched
18:40:29.205099 run   osmo-bsc_10.42.42.9(pid=26888): Launched
18:40:35.483562 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
18:40:36.069004 run         rm-remote-dir(pid=26889): Launched
18:40:38.273038 run         rm-remote-dir(pid=26889): Terminated: ok {rc=0}
18:40:38.625761 run         mk-remote-dir(pid=26890): Launched
18:40:39.825418 run         mk-remote-dir(pid=26890): Terminated: ok {rc=0}
18:40:40.122122 run    scp-inst-to-remote(pid=26891): Launched
18:40:52.330599 run    scp-inst-to-remote(pid=26891): Terminated: ok {rc=0}
18:40:52.655173 run         mk-remote-dir(pid=26893): Launched
18:40:53.840404 run         mk-remote-dir(pid=26893): Terminated: ok {rc=0}
18:40:54.118457 run     scp-cfg-to-remote(pid=26894): Launched
18:40:55.304494 run     scp-cfg-to-remote(pid=26894): Terminated: ok {rc=0}
18:40:55.624346 run   reload-dsp-firmware(pid=26896): Launched
18:40:57.809607 run   reload-dsp-firmware(pid=26896): Terminated: ok {rc=0}
18:40:58.121252 run        osmo-bts-sysmo(pid=26897): Launched
18:41:02.436979 tst                      iperf3.py:8: Waiting for bts to be ready...
18:41:04.081416 run         rm-remote-dir(pid=26898): Launched
18:41:06.266407 run         rm-remote-dir(pid=26898): Terminated: ok {rc=0}
18:41:06.579910 run         mk-remote-dir(pid=26899): Launched
18:41:08.760513 run         mk-remote-dir(pid=26899): Terminated: ok {rc=0}
18:41:09.028462 run       scp-inst-to-bts(pid=26900): Launched
18:41:22.225483 run       scp-inst-to-bts(pid=26900): Terminated: ok {rc=0}
18:41:22.539772 run     mk-remote-run-dir(pid=26902): Launched
18:41:24.732505 run     mk-remote-run-dir(pid=26902): Terminated: ok {rc=0}
18:41:25.020777 run   scp-cfg-to-sysmobts(pid=26903): Launched
18:41:27.215568 run   scp-cfg-to-sysmobts(pid=26903): Terminated: ok {rc=0}
18:41:27.551475 run              osmo-pcu(pid=26905): Launched
18:41:27.684940 tst                        iperf3.py: using MSISDN 3951
18:41:27.857140 bus                          /gobi_6: Setting Powered True
18:41:31.369123 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3951', subscriber_id=1}
18:41:31.659124 bus                          /gobi_6: Setting Online False
18:41:32.845922 bus                          /gobi_6: Setting Powered False
18:41:40.232766 bus                          /gobi_6: Setting Powered True
18:41:41.631903 bus                          /gobi_6: Setting Online True
18:41:44.171967 tst                          /gobi_6: Connect to ('901', '70')
18:41:44.433578 tst                          /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G  1  [Jul  5 2017 10:49:17]', Serial: '868323024273648'
18:41:44.619708 tst                      iperf3.py:8: waiting for modems to attach...
18:41:50.060913 tst                          /gobi_6: Already registered with network ('901', '70')
18:42:02.086847 tst                      iperf3.py:8: waiting for modems to attach to data services...
18:42:06.665173 tst                          /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
18:42:06.868413 tst                      iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
18:42:07.246417 run            move_netns(pid=26909): Launched
18:42:08.448341 run            move_netns(pid=26909): Terminated: ok {rc=0}
18:42:08.802178 run                  ifup(pid=26912): Launched
18:42:10.001451 run                  ifup(pid=26912): Terminated: ok {rc=0}
18:42:10.347898 run                  dhcp(pid=26915): Launched
18:42:11.530357 run                  dhcp(pid=26915): Terminated: ok {rc=0}
18:42:11.724176 tst                      iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
18:42:11.854073 run           iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
18:42:12.236255 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931): Launched
18:42:12.551963 run iperf3-cli_10.42.42.10(pid=26934): Launched
18:43:26.886780 run iperf3-cli_10.42.42.10(pid=26934): ERR: Terminated: ERROR {rc=1}  [trial-2064↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=26934)]
18:43:27.028931 run iperf3-cli_10.42.42.10(pid=26934): stdout: 
| 				"bits_per_second":	0,
| 				"retransmits":	0,
| 				"omitted":	false
| 			}
| 		}],
| 	"end":	{
| 	},
| 	"error":	"error - control socket has closed unexpectedly"
| }
| child exited with 1 
18:43:27.168916 run iperf3-cli_10.42.42.10(pid=26934): stderr: 
| (launched: 2022-03-29_18:42:12.439797) 
18:43:27.295853 run iperf3-srv_10.42.42.10(pid=26844): Terminating (SIGINT)
18:43:27.642126 run iperf3-srv_10.42.42.10(pid=26844): Terminated {rc=1}
18:43:27.852097 tst                      iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
18:43:27.913617 tst                      iperf3.py:8: ERR: KeyError: 'sum_sent'  [trial-2064↪gprs:sysmo↪iperf3.py:8]
18:43:27.921782 tst                      iperf3.py:8: Test FAILED (196.8 sec)
18:43:27.959836 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26843): Terminating (SIGTERM)
18:43:28.003684 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26847): Terminating (SIGTERM)
18:43:28.046597 run   osmo-hlr_10.42.42.2(pid=26848): Terminating (SIGTERM)
18:43:28.091841 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26849): Terminating (SIGTERM)
18:43:28.140227 run   osmo-stp_10.42.42.5(pid=26850): Terminating (SIGTERM)
18:43:28.204874 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26851): Terminating (SIGTERM)
18:43:28.250579 run  osmo-ggsn_10.42.42.6(pid=26856): Terminating (SIGTERM)
18:43:28.318615 run pcap-recorder_any(filters='host 10.42.42.7')(pid=26879): Terminating (SIGTERM)
18:43:28.385042 run  osmo-sgsn_10.42.42.7(pid=26880): Terminating (SIGTERM)
18:43:28.438397 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=26881): Terminating (SIGTERM)
18:43:28.487140 run   osmo-msc_10.42.42.8(pid=26882): Terminating (SIGTERM)
18:43:28.532768 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=26883): Terminating (SIGTERM)
18:43:28.579923 run   osmo-mgw_10.42.42.3(pid=26884): Terminating (SIGTERM)
18:43:28.624085 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=26885): Terminating (SIGTERM)
18:43:28.667599 run   osmo-mgw_10.42.42.4(pid=26886): Terminating (SIGTERM)
18:43:28.712326 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=26887): Terminating (SIGTERM)
18:43:28.754974 run   osmo-bsc_10.42.42.9(pid=26888): Terminating (SIGTERM)
18:43:28.799408 run        osmo-bts-sysmo(pid=26897): Terminating (SIGTERM)
18:43:28.843627 run              osmo-pcu(pid=26905): Terminating (SIGTERM)
18:43:28.887432 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931): Terminating (SIGTERM)
18:43:29.049282 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGTERM(pid=26955): Launched
18:43:30.142577 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGTERM(pid=26955): Terminated: ok {rc=0}
18:43:30.157057 ---      ParallelTerminationStrategy: PID 26843 died...
18:43:30.213297 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26843): Terminated: ok {rc=0}
18:43:30.223497 ---      ParallelTerminationStrategy: PID 26847 died...
18:43:30.280120 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26847): Terminated: ok {rc=0}
18:43:30.290396 ---      ParallelTerminationStrategy: PID 26848 died...
18:43:30.346740 run   osmo-hlr_10.42.42.2(pid=26848): Terminated: ok {rc=0}
18:43:30.356865 ---      ParallelTerminationStrategy: PID 26849 died...
18:43:30.413129 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26849): Terminated: ok {rc=0}
18:43:30.423692 ---      ParallelTerminationStrategy: PID 26850 died...
18:43:30.479888 run   osmo-stp_10.42.42.5(pid=26850): Terminated: ok {rc=0}
18:43:30.490018 ---      ParallelTerminationStrategy: PID 26851 died...
18:43:30.546265 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26851): Terminated: ok {rc=0}
18:43:30.556455 ---      ParallelTerminationStrategy: PID 26856 died...
18:43:30.612642 run  osmo-ggsn_10.42.42.6(pid=26856): Terminated: ok {rc=0}
18:43:30.622974 ---      ParallelTerminationStrategy: PID 26879 died...
18:43:30.679318 run pcap-recorder_any(filters='host 10.42.42.7')(pid=26879): Terminated: ok {rc=0}
18:43:30.689498 ---      ParallelTerminationStrategy: PID 26880 died...
18:43:30.745950 run  osmo-sgsn_10.42.42.7(pid=26880): Terminated: ok {rc=0}
18:43:30.756143 ---      ParallelTerminationStrategy: PID 26881 died...
18:43:30.812278 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=26881): Terminated: ok {rc=0}
18:43:30.822500 ---      ParallelTerminationStrategy: PID 26882 died...
18:43:30.879020 run   osmo-msc_10.42.42.8(pid=26882): Terminated: ok {rc=0}
18:43:30.889226 ---      ParallelTerminationStrategy: PID 26883 died...
18:43:30.945578 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=26883): Terminated: ok {rc=0}
18:43:30.955819 ---      ParallelTerminationStrategy: PID 26884 died...
18:43:31.012099 run   osmo-mgw_10.42.42.3(pid=26884): Terminated {rc=15}
18:43:31.022361 ---      ParallelTerminationStrategy: PID 26885 died...
18:43:31.078599 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=26885): Terminated: ok {rc=0}
18:43:31.088888 ---      ParallelTerminationStrategy: PID 26886 died...
18:43:31.145241 run   osmo-mgw_10.42.42.4(pid=26886): Terminated {rc=15}
18:43:31.155553 ---      ParallelTerminationStrategy: PID 26887 died...
18:43:31.211931 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=26887): Terminated: ok {rc=0}
18:43:31.222453 ---      ParallelTerminationStrategy: PID 26888 died...
18:43:31.278578 run   osmo-bsc_10.42.42.9(pid=26888): Terminated: ok {rc=0}
18:43:31.288830 ---      ParallelTerminationStrategy: PID 26897 died...
18:43:31.345338 run        osmo-bts-sysmo(pid=26897): Terminated: ok {rc=0}
18:43:31.355710 ---      ParallelTerminationStrategy: PID 26905 died...
18:43:31.411854 run              osmo-pcu(pid=26905): Terminated: ok {rc=0}
18:43:37.278182 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931): Terminating (SIGINT)
18:43:37.432065 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGINT(pid=26960): Launched
18:43:38.523294 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGINT(pid=26960): Terminated: ok {rc=0}
18:43:44.393943 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931): Terminating (SIGKILL)
18:43:44.547767 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGUSR1(pid=26963): Launched
18:43:45.641991 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26931)-killSignals.SIGUSR1(pid=26963): Terminated: ok {rc=0}
18:43:45.879384 bus                          /gobi_6: Setting Online False
18:43:48.910062 bus                          /gobi_6: Setting Powered False
18:44:03.972907 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
    setup_run_iperf3_test_parallel(1)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
    print_results(clients[i], servers[i])
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
    cli.print_results()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
    print_result_node_tcp(self.get_results(), 'client')
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
    sent = result['end']['sum_sent']
KeyError: 'sum_sent'