Skip to content

Failed

gprs:sysmo.iperf3.py (from gprs_sysmo)

Failing for the past 4 builds (Since Failed#2063 )
Took 3 min 25 sec.

Stacktrace

'sum_sent'

Standard Output

----------------------------------------------
trial-2066 gprs:sysmo iperf3.py
----------------------------------------------
11:09:40.151011 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.162854 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
11:09:40.292881 tst                        iperf3.py: using LAC 3960
11:09:40.414052 tst                        iperf3.py: using RAC 135
11:09:40.540455 tst                        iperf3.py: using CellId 3960
11:09:40.667308 tst                        iperf3.py: using BVCI 3961
11:09:40.690388 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.700999 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.711521 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.722491 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.733377 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.743942 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.754545 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.765076 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
11:09:40.775863 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
11:09:41.058385 tst                      iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
11:09:41.185415 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
11:09:41.506837 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18165): Launched
11:09:41.769631 run iperf3-srv_10.42.42.10(pid=18166): Launched
11:09:41.955061 tst                      iperf3.py:8: start network...
11:09:42.081532 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
11:09:42.564308 run         create_hlr_db(pid=18167): Launched
11:09:42.739729 bus                          /gobi_6: Setting Powered False
11:09:43.762830 run         create_hlr_db(pid=18167): Terminated: ok {rc=0}
11:09:44.199605 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18169): Launched
11:09:44.531938 run   osmo-hlr_10.42.42.2(pid=18170): Launched
11:09:44.649997 run              osmo-stp_10.42.42.5: Starting osmo-stp
11:09:45.216929 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18171): Launched
11:09:45.545247 run   osmo-stp_10.42.42.5(pid=18172): Launched
11:09:45.661780 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
11:09:46.230687 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18173): Launched
11:09:46.347869 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
11:09:46.635775 run              patchelf(pid=18174): Launched
11:09:47.829846 run              patchelf(pid=18174): Terminated: ok {rc=0}
11:09:47.950444 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
11:09:48.242233 run osmo-gsm-tester_setcap_net_admin.sh(pid=18175): Launched
11:09:49.436422 run osmo-gsm-tester_setcap_net_admin.sh(pid=18175): Terminated: ok {rc=0}
11:09:49.769926 run  osmo-ggsn_10.42.42.6(pid=18178): Launched
11:09:50.015955 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
11:09:50.568880 run pcap-recorder_any(filters='host 10.42.42.7')(pid=18199): Launched
11:09:50.877878 run  osmo-sgsn_10.42.42.7(pid=18200): Launched
11:09:50.991756 run              osmo-msc_10.42.42.8: Starting osmo-msc
11:09:51.555760 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=18201): Launched
11:09:51.866583 run   osmo-msc_10.42.42.8(pid=18202): Launched
11:09:55.030595 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
11:09:55.561812 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=18203): Launched
11:09:55.869861 run   osmo-mgw_10.42.42.3(pid=18204): Launched
11:09:55.979390 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
11:09:56.504332 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=18205): Launched
11:09:56.813018 run   osmo-mgw_10.42.42.4(pid=18206): Launched
11:09:56.922801 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
11:09:57.566072 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=18207): Launched
11:09:57.875600 run   osmo-bsc_10.42.42.9(pid=18208): Launched
11:10:04.137256 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
11:10:04.684880 run         rm-remote-dir(pid=18212): Launched
11:10:06.876153 run         rm-remote-dir(pid=18212): Terminated: ok {rc=0}
11:10:07.206339 run         mk-remote-dir(pid=18213): Launched
11:10:08.395439 run         mk-remote-dir(pid=18213): Terminated: ok {rc=0}
11:10:08.673794 run    scp-inst-to-remote(pid=18214): Launched
11:10:20.884496 run    scp-inst-to-remote(pid=18214): Terminated: ok {rc=0}
11:10:21.233752 run         mk-remote-dir(pid=18216): Launched
11:10:22.433115 run         mk-remote-dir(pid=18216): Terminated: ok {rc=0}
11:10:22.729212 run     scp-cfg-to-remote(pid=18217): Launched
11:10:23.927340 run     scp-cfg-to-remote(pid=18217): Terminated: ok {rc=0}
11:10:24.268243 run   reload-dsp-firmware(pid=18219): Launched
11:10:26.454219 run   reload-dsp-firmware(pid=18219): Terminated: ok {rc=0}
11:10:26.765639 run        osmo-bts-sysmo(pid=18220): Launched
11:10:31.083253 tst                      iperf3.py:8: Waiting for bts to be ready...
11:10:32.727433 run         rm-remote-dir(pid=18221): Launched
11:10:34.921105 run         rm-remote-dir(pid=18221): Terminated: ok {rc=0}
11:10:35.258294 run         mk-remote-dir(pid=18222): Launched
11:10:37.451736 run         mk-remote-dir(pid=18222): Terminated: ok {rc=0}
11:10:37.739163 run       scp-inst-to-bts(pid=18223): Launched
11:10:50.942442 run       scp-inst-to-bts(pid=18223): Terminated: ok {rc=0}
11:10:51.258504 run     mk-remote-run-dir(pid=18226): Launched
11:10:53.440178 run     mk-remote-run-dir(pid=18226): Terminated: ok {rc=0}
11:10:53.710222 run   scp-cfg-to-sysmobts(pid=18227): Launched
11:10:55.902001 run   scp-cfg-to-sysmobts(pid=18227): Terminated: ok {rc=0}
11:10:56.240722 run              osmo-pcu(pid=18229): Launched
11:10:56.370817 tst                        iperf3.py: using MSISDN 3973
11:10:56.539235 bus                          /gobi_6: Setting Powered True
11:11:00.041469 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3973', subscriber_id=1}
11:11:00.335661 bus                          /gobi_6: Setting Online False
11:11:01.523505 bus                          /gobi_6: Setting Powered False
11:11:08.291484 bus                          /gobi_6: Setting Powered True
11:11:09.686082 bus                          /gobi_6: Setting Online True
11:11:12.159195 tst                          /gobi_6: Connect to ('901', '70')
11:11:12.443572 tst                          /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G  1  [Jul  5 2017 10:49:17]', Serial: '868323024273648'
11:11:12.646312 tst                      iperf3.py:8: waiting for modems to attach...
11:11:19.740207 tst                          /gobi_6: Already registered with network ('901', '70')
11:11:31.178934 tst                      iperf3.py:8: waiting for modems to attach to data services...
11:11:49.085649 tst                          /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
11:11:49.276496 tst                      iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
11:11:49.641006 run            move_netns(pid=18232): Launched
11:11:50.838801 run            move_netns(pid=18232): Terminated: ok {rc=0}
11:11:51.192394 run                  ifup(pid=18235): Launched
11:11:52.390583 run                  ifup(pid=18235): Terminated: ok {rc=0}
11:11:52.732224 run                  dhcp(pid=18238): Launched
11:11:53.917242 run                  dhcp(pid=18238): Terminated: ok {rc=0}
11:11:54.112190 tst                      iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
11:11:54.243176 run           iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
11:11:54.626346 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254): Launched
11:11:54.941828 run iperf3-cli_10.42.42.10(pid=18257): Launched
11:13:03.261392 run iperf3-cli_10.42.42.10(pid=18257): ERR: Terminated: ERROR {rc=1}  [trial-2066↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=18257)]
11:13:03.394757 run iperf3-cli_10.42.42.10(pid=18257): stdout: 
| 				"bits_per_second":	0,
| 				"retransmits":	0,
| 				"omitted":	false
| 			}
| 		}],
| 	"end":	{
| 	},
| 	"error":	"error - control socket has closed unexpectedly"
| }
| child exited with 1 
11:13:03.527048 run iperf3-cli_10.42.42.10(pid=18257): stderr: 
| (launched: 2022-09-07_11:11:54.831390) 
11:13:03.647871 run iperf3-srv_10.42.42.10(pid=18166): Terminating (SIGINT)
11:13:03.994111 run iperf3-srv_10.42.42.10(pid=18166): Terminated {rc=1}
11:13:04.198689 tst                      iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
11:13:04.260102 tst                      iperf3.py:8: ERR: KeyError: 'sum_sent'  [trial-2066↪gprs:sysmo↪iperf3.py:8]
11:13:04.267803 tst                      iperf3.py:8: Test FAILED (204.2 sec)
11:13:04.305776 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18165): Terminating (SIGTERM)
11:13:04.350639 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18169): Terminating (SIGTERM)
11:13:04.393669 run   osmo-hlr_10.42.42.2(pid=18170): Terminating (SIGTERM)
11:13:04.437420 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18171): Terminating (SIGTERM)
11:13:04.483368 run   osmo-stp_10.42.42.5(pid=18172): Terminating (SIGTERM)
11:13:04.538455 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18173): Terminating (SIGTERM)
11:13:04.586419 run  osmo-ggsn_10.42.42.6(pid=18178): Terminating (SIGTERM)
11:13:04.660198 run pcap-recorder_any(filters='host 10.42.42.7')(pid=18199): Terminating (SIGTERM)
11:13:04.749713 run  osmo-sgsn_10.42.42.7(pid=18200): Terminating (SIGTERM)
11:13:04.796942 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=18201): Terminating (SIGTERM)
11:13:04.848612 run   osmo-msc_10.42.42.8(pid=18202): Terminating (SIGTERM)
11:13:04.896300 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=18203): Terminating (SIGTERM)
11:13:04.940384 run   osmo-mgw_10.42.42.3(pid=18204): Terminating (SIGTERM)
11:13:04.983412 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=18205): Terminating (SIGTERM)
11:13:05.031041 run   osmo-mgw_10.42.42.4(pid=18206): Terminating (SIGTERM)
11:13:05.075266 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=18207): Terminating (SIGTERM)
11:13:05.121905 run   osmo-bsc_10.42.42.9(pid=18208): Terminating (SIGTERM)
11:13:05.166820 run        osmo-bts-sysmo(pid=18220): Terminating (SIGTERM)
11:13:05.212297 run              osmo-pcu(pid=18229): Terminating (SIGTERM)
11:13:05.260373 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254): Terminating (SIGTERM)
11:13:05.415159 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGTERM(pid=18280): Launched
11:13:06.506836 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGTERM(pid=18280): Terminated: ok {rc=0}
11:13:06.521025 ---      ParallelTerminationStrategy: PID 18165 died...
11:13:06.576623 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18165): Terminated: ok {rc=0}
11:13:06.586641 ---      ParallelTerminationStrategy: PID 18169 died...
11:13:06.641141 run pcap-recorder_any(filters='host 10.42.42.2')(pid=18169): Terminated: ok {rc=0}
11:13:06.651634 ---      ParallelTerminationStrategy: PID 18170 died...
11:13:06.706117 run   osmo-hlr_10.42.42.2(pid=18170): Terminated: ok {rc=0}
11:13:06.716243 ---      ParallelTerminationStrategy: PID 18171 died...
11:13:06.770648 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=18171): Terminated: ok {rc=0}
11:13:06.780959 ---      ParallelTerminationStrategy: PID 18172 died...
11:13:06.835508 run   osmo-stp_10.42.42.5(pid=18172): Terminated: ok {rc=0}
11:13:06.845538 ---      ParallelTerminationStrategy: PID 18173 died...
11:13:06.899703 run pcap-recorder_any(filters='host 10.42.42.6')(pid=18173): Terminated: ok {rc=0}
11:13:06.909685 ---      ParallelTerminationStrategy: PID 18178 died...
11:13:06.964652 run  osmo-ggsn_10.42.42.6(pid=18178): Terminated: ok {rc=0}
11:13:06.974964 ---      ParallelTerminationStrategy: PID 18199 died...
11:13:07.029274 run pcap-recorder_any(filters='host 10.42.42.7')(pid=18199): Terminated: ok {rc=0}
11:13:07.039649 ---      ParallelTerminationStrategy: PID 18200 died...
11:13:07.094530 run  osmo-sgsn_10.42.42.7(pid=18200): Terminated: ok {rc=0}
11:13:07.104740 ---      ParallelTerminationStrategy: PID 18201 died...
11:13:07.159233 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=18201): Terminated: ok {rc=0}
11:13:07.169283 ---      ParallelTerminationStrategy: PID 18202 died...
11:13:07.224371 run   osmo-msc_10.42.42.8(pid=18202): Terminated: ok {rc=0}
11:13:07.234258 ---      ParallelTerminationStrategy: PID 18203 died...
11:13:07.289633 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=18203): Terminated: ok {rc=0}
11:13:07.300058 ---      ParallelTerminationStrategy: PID 18204 died...
11:13:07.354817 run   osmo-mgw_10.42.42.3(pid=18204): Terminated {rc=15}
11:13:07.365038 ---      ParallelTerminationStrategy: PID 18205 died...
11:13:07.419596 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=18205): Terminated: ok {rc=0}
11:13:07.429821 ---      ParallelTerminationStrategy: PID 18206 died...
11:13:07.484611 run   osmo-mgw_10.42.42.4(pid=18206): Terminated {rc=15}
11:13:07.494912 ---      ParallelTerminationStrategy: PID 18207 died...
11:13:07.550140 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=18207): Terminated: ok {rc=0}
11:13:07.560547 ---      ParallelTerminationStrategy: PID 18208 died...
11:13:07.615006 run   osmo-bsc_10.42.42.9(pid=18208): Terminated: ok {rc=0}
11:13:07.625344 ---      ParallelTerminationStrategy: PID 18220 died...
11:13:07.680609 run        osmo-bts-sysmo(pid=18220): Terminated: ok {rc=0}
11:13:07.690874 ---      ParallelTerminationStrategy: PID 18229 died...
11:13:07.745495 run              osmo-pcu(pid=18229): Terminated: ok {rc=0}
11:13:13.611982 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254): Terminating (SIGINT)
11:13:13.763409 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGINT(pid=18284): Launched
11:13:14.855010 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGINT(pid=18284): Terminated: ok {rc=0}
11:13:20.724863 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254): Terminating (SIGKILL)
11:13:20.876485 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGUSR1(pid=18287): Launched
11:13:21.968589 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=18254)-killSignals.SIGUSR1(pid=18287): Terminated: ok {rc=0}
11:13:22.205598 bus                          /gobi_6: Setting Online False
11:13:25.253697 bus                          /gobi_6: Setting Powered False
11:13:40.094141 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
    setup_run_iperf3_test_parallel(1)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
    print_results(clients[i], servers[i])
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
    cli.print_results()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
    print_result_node_tcp(self.get_results(), 'client')
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
    sent = result['end']['sum_sent']
KeyError: 'sum_sent'