Skip to content

Failed

gprs:sysmo.iperf3.py (from gprs_sysmo)

Failing for the past 3 builds (Since Failed#2063 )
Took 3 min 10 sec.

Stacktrace

'sum_sent'

Standard Output

----------------------------------------------
trial-2065 gprs:sysmo iperf3.py
----------------------------------------------
12:18:51.153797 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.165368 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
12:18:51.294519 tst                        iperf3.py: using LAC 3953
12:18:51.416572 tst                        iperf3.py: using RAC 128
12:18:51.540823 tst                        iperf3.py: using CellId 3953
12:18:51.658910 tst                        iperf3.py: using BVCI 3954
12:18:51.682124 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.692966 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.703595 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.714878 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.726079 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.737572 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.749109 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.760662 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.772167 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
12:18:52.068298 tst                      iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
12:18:52.204122 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
12:18:52.543601 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31702): Launched
12:18:52.825180 run iperf3-srv_10.42.42.10(pid=31703): Launched
12:18:53.024353 tst                      iperf3.py:8: start network...
12:18:53.160529 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
12:18:53.672351 run         create_hlr_db(pid=31704): Launched
12:18:53.847555 bus                          /gobi_6: Setting Powered False
12:18:54.869246 run         create_hlr_db(pid=31704): Terminated: ok {rc=0}
12:18:55.277499 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31706): Launched
12:18:55.586029 run   osmo-hlr_10.42.42.2(pid=31707): Launched
12:18:55.695860 run              osmo-stp_10.42.42.5: Starting osmo-stp
12:18:56.224755 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31708): Launched
12:18:56.532767 run   osmo-stp_10.42.42.5(pid=31709): Launched
12:18:56.640857 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:18:57.173462 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Launched
12:18:57.283228 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:18:57.552453 run              patchelf(pid=31711): Launched
12:18:58.734617 run              patchelf(pid=31711): Terminated: ok {rc=0}
12:18:58.846809 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:18:59.119294 run osmo-gsm-tester_setcap_net_admin.sh(pid=31712): Launched
12:19:00.301194 run osmo-gsm-tester_setcap_net_admin.sh(pid=31712): Terminated: ok {rc=0}
12:19:00.612263 run  osmo-ggsn_10.42.42.6(pid=31715): Launched
12:19:00.884053 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:19:01.456971 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31732): Launched
12:19:01.787450 run  osmo-sgsn_10.42.42.7(pid=31733): Launched
12:19:01.906356 run              osmo-msc_10.42.42.8: Starting osmo-msc
12:19:02.503036 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31734): Launched
12:19:02.830055 run   osmo-msc_10.42.42.8(pid=31735): Launched
12:19:06.003792 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
12:19:06.567176 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31736): Launched
12:19:06.893418 run   osmo-mgw_10.42.42.3(pid=31737): Launched
12:19:07.009175 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
12:19:07.564633 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31738): Launched
12:19:07.893002 run   osmo-mgw_10.42.42.4(pid=31739): Launched
12:19:08.008856 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
12:19:08.691321 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31740): Launched
12:19:09.019609 run   osmo-bsc_10.42.42.9(pid=31741): Launched
12:19:15.285085 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:19:15.828571 run         rm-remote-dir(pid=31742): Launched
12:19:18.019954 run         rm-remote-dir(pid=31742): Terminated: ok {rc=0}
12:19:18.347624 run         mk-remote-dir(pid=31743): Launched
12:19:19.534458 run         mk-remote-dir(pid=31743): Terminated: ok {rc=0}
12:19:19.809904 run    scp-inst-to-remote(pid=31744): Launched
12:19:32.017616 run    scp-inst-to-remote(pid=31744): Terminated: ok {rc=0}
12:19:32.361792 run         mk-remote-dir(pid=31746): Launched
12:19:33.558084 run         mk-remote-dir(pid=31746): Terminated: ok {rc=0}
12:19:33.852709 run     scp-cfg-to-remote(pid=31747): Launched
12:19:35.048154 run     scp-cfg-to-remote(pid=31747): Terminated: ok {rc=0}
12:19:35.387677 run   reload-dsp-firmware(pid=31749): Launched
12:19:37.583762 run   reload-dsp-firmware(pid=31749): Terminated: ok {rc=0}
12:19:37.913876 run        osmo-bts-sysmo(pid=31750): Launched
12:19:42.227388 tst                      iperf3.py:8: Waiting for bts to be ready...
12:19:43.869414 run         rm-remote-dir(pid=31751): Launched
12:19:46.062837 run         rm-remote-dir(pid=31751): Terminated: ok {rc=0}
12:19:46.396659 run         mk-remote-dir(pid=31752): Launched
12:19:48.578355 run         mk-remote-dir(pid=31752): Terminated: ok {rc=0}
12:19:48.844194 run       scp-inst-to-bts(pid=31753): Launched
12:20:02.049550 run       scp-inst-to-bts(pid=31753): Terminated: ok {rc=0}
12:20:02.382594 run     mk-remote-run-dir(pid=31759): Launched
12:20:04.565993 run     mk-remote-run-dir(pid=31759): Terminated: ok {rc=0}
12:20:04.832096 run   scp-cfg-to-sysmobts(pid=31760): Launched
12:20:07.012406 run   scp-cfg-to-sysmobts(pid=31760): Terminated: ok {rc=0}
12:20:07.326074 run              osmo-pcu(pid=31762): Launched
12:20:07.453699 tst                        iperf3.py: using MSISDN 3962
12:20:07.627424 bus                          /gobi_6: Setting Powered True
12:20:11.113278 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3962', subscriber_id=1}
12:20:11.401124 bus                          /gobi_6: Setting Online False
12:20:12.586456 bus                          /gobi_6: Setting Powered False
12:20:19.313328 bus                          /gobi_6: Setting Powered True
12:20:20.707048 bus                          /gobi_6: Setting Online True
12:20:23.127677 tst                          /gobi_6: Connect to ('901', '70')
12:20:23.398352 tst                          /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G  1  [Jul  5 2017 10:49:17]', Serial: '868323024273648'
12:20:23.582843 tst                      iperf3.py:8: waiting for modems to attach...
12:20:37.605716 tst                          /gobi_6: Already registered with network ('901', '70')
12:20:43.025968 tst                      iperf3.py:8: waiting for modems to attach to data services...
12:20:45.993107 tst                          /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:20:46.177645 tst                      iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
12:20:46.529453 run            move_netns(pid=31766): Launched
12:20:47.710907 run            move_netns(pid=31766): Terminated: ok {rc=0}
12:20:48.036421 run                  ifup(pid=31769): Launched
12:20:49.218521 run                  ifup(pid=31769): Terminated: ok {rc=0}
12:20:49.551562 run                  dhcp(pid=31772): Launched
12:20:50.733333 run                  dhcp(pid=31772): Terminated: ok {rc=0}
12:20:50.926298 tst                      iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
12:20:51.055797 run           iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:20:51.429755 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Launched
12:20:51.741394 run iperf3-cli_10.42.42.10(pid=31792): Launched
12:22:00.065028 run iperf3-cli_10.42.42.10(pid=31792): ERR: Terminated: ERROR {rc=1}  [trial-2065↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=31792)]
12:22:00.203671 run iperf3-cli_10.42.42.10(pid=31792): stdout: 
| 				"bits_per_second":	0,
| 				"retransmits":	0,
| 				"omitted":	false
| 			}
| 		}],
| 	"end":	{
| 	},
| 	"error":	"error - control socket has closed unexpectedly"
| }
| child exited with 1 
12:22:00.343343 run iperf3-cli_10.42.42.10(pid=31792): stderr: 
| (launched: 2022-06-18_12:20:51.632024) 
12:22:00.470363 run iperf3-srv_10.42.42.10(pid=31703): Terminating (SIGINT)
12:22:00.814070 run iperf3-srv_10.42.42.10(pid=31703): Terminated {rc=1}
12:22:01.021317 tst                      iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
12:22:01.082054 tst                      iperf3.py:8: ERR: KeyError: 'sum_sent'  [trial-2065↪gprs:sysmo↪iperf3.py:8]
12:22:01.089648 tst                      iperf3.py:8: Test FAILED (190.0 sec)
12:22:01.127271 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31702): Terminating (SIGTERM)
12:22:01.170450 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31706): Terminating (SIGTERM)
12:22:01.213432 run   osmo-hlr_10.42.42.2(pid=31707): Terminating (SIGTERM)
12:22:01.257764 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31708): Terminating (SIGTERM)
12:22:01.303785 run   osmo-stp_10.42.42.5(pid=31709): Terminating (SIGTERM)
12:22:01.355729 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Terminating (SIGTERM)
12:22:01.409377 run  osmo-ggsn_10.42.42.6(pid=31715): Terminating (SIGTERM)
12:22:01.490349 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31732): Terminating (SIGTERM)
12:22:01.546249 run  osmo-sgsn_10.42.42.7(pid=31733): Terminating (SIGTERM)
12:22:01.592368 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31734): Terminating (SIGTERM)
12:22:01.644488 run   osmo-msc_10.42.42.8(pid=31735): Terminating (SIGTERM)
12:22:01.690141 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31736): Terminating (SIGTERM)
12:22:01.735610 run   osmo-mgw_10.42.42.3(pid=31737): Terminating (SIGTERM)
12:22:01.777393 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31738): Terminating (SIGTERM)
12:22:01.818671 run   osmo-mgw_10.42.42.4(pid=31739): Terminating (SIGTERM)
12:22:01.860046 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31740): Terminating (SIGTERM)
12:22:01.900829 run   osmo-bsc_10.42.42.9(pid=31741): Terminating (SIGTERM)
12:22:01.941667 run        osmo-bts-sysmo(pid=31750): Terminating (SIGTERM)
12:22:01.983059 run              osmo-pcu(pid=31762): Terminating (SIGTERM)
12:22:02.025531 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Terminating (SIGTERM)
12:22:02.173577 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGTERM(pid=31812): Launched
12:22:03.264996 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGTERM(pid=31812): Terminated: ok {rc=0}
12:22:03.278834 ---      ParallelTerminationStrategy: PID 31702 died...
12:22:03.333412 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31702): Terminated: ok {rc=0}
12:22:03.343504 ---      ParallelTerminationStrategy: PID 31706 died...
12:22:03.397860 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31706): Terminated: ok {rc=0}
12:22:03.407953 ---      ParallelTerminationStrategy: PID 31707 died...
12:22:03.461989 run   osmo-hlr_10.42.42.2(pid=31707): Terminated: ok {rc=0}
12:22:03.471530 ---      ParallelTerminationStrategy: PID 31708 died...
12:22:03.525168 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31708): Terminated: ok {rc=0}
12:22:03.535001 ---      ParallelTerminationStrategy: PID 31709 died...
12:22:03.588592 run   osmo-stp_10.42.42.5(pid=31709): Terminated: ok {rc=0}
12:22:03.598221 ---      ParallelTerminationStrategy: PID 31710 died...
12:22:03.652360 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Terminated: ok {rc=0}
12:22:03.661925 ---      ParallelTerminationStrategy: PID 31715 died...
12:22:03.715773 run  osmo-ggsn_10.42.42.6(pid=31715): Terminated: ok {rc=0}
12:22:03.725438 ---      ParallelTerminationStrategy: PID 31732 died...
12:22:03.779965 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31732): Terminated: ok {rc=0}
12:22:03.789671 ---      ParallelTerminationStrategy: PID 31733 died...
12:22:03.844136 run  osmo-sgsn_10.42.42.7(pid=31733): Terminated: ok {rc=0}
12:22:03.854027 ---      ParallelTerminationStrategy: PID 31734 died...
12:22:03.908033 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31734): Terminated: ok {rc=0}
12:22:03.918026 ---      ParallelTerminationStrategy: PID 31735 died...
12:22:03.971891 run   osmo-msc_10.42.42.8(pid=31735): Terminated: ok {rc=0}
12:22:03.981891 ---      ParallelTerminationStrategy: PID 31736 died...
12:22:04.035714 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31736): Terminated: ok {rc=0}
12:22:04.045688 ---      ParallelTerminationStrategy: PID 31737 died...
12:22:04.099929 run   osmo-mgw_10.42.42.3(pid=31737): Terminated {rc=15}
12:22:04.109914 ---      ParallelTerminationStrategy: PID 31738 died...
12:22:04.163849 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31738): Terminated: ok {rc=0}
12:22:04.173707 ---      ParallelTerminationStrategy: PID 31739 died...
12:22:04.227438 run   osmo-mgw_10.42.42.4(pid=31739): Terminated {rc=15}
12:22:04.237395 ---      ParallelTerminationStrategy: PID 31740 died...
12:22:04.291435 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31740): Terminated: ok {rc=0}
12:22:04.301420 ---      ParallelTerminationStrategy: PID 31741 died...
12:22:04.355337 run   osmo-bsc_10.42.42.9(pid=31741): Terminated: ok {rc=0}
12:22:04.365446 ---      ParallelTerminationStrategy: PID 31750 died...
12:22:04.419775 run        osmo-bts-sysmo(pid=31750): Terminated: ok {rc=0}
12:22:04.429834 ---      ParallelTerminationStrategy: PID 31762 died...
12:22:04.484832 run              osmo-pcu(pid=31762): Terminated: ok {rc=0}
12:22:10.351736 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Terminating (SIGINT)
12:22:10.502301 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGINT(pid=31817): Launched
12:22:11.592511 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGINT(pid=31817): Terminated: ok {rc=0}
12:22:17.462515 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Terminating (SIGKILL)
12:22:17.609518 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGUSR1(pid=31820): Launched
12:22:18.700506 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGUSR1(pid=31820): Terminated: ok {rc=0}
12:22:18.932014 bus                          /gobi_6: Setting Online False
12:22:21.972358 bus                          /gobi_6: Setting Powered False
12:22:39.046993 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module>
    setup_run_iperf3_test_parallel(1)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
    print_results(clients[i], servers[i])
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
    cli.print_results()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
    print_result_node_tcp(self.get_results(), 'client')
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
    sent = result['end']['sum_sent']
KeyError: 'sum_sent'