1gprs:sysmo226.0'sum_sent'2063cs_paging_gprs_active.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/cs_paging_gprs_active.py", line 38, in <module> setup_run_iperf3_test_parallel(2, ready_cb=ready_cb_place_voicecall) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel print_results(clients[i], servers[i]) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results cli.print_results() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results print_result_node_tcp(self.get_results(), 'client') File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp raise e File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp sent = result['end']['sum_sent'] KeyError: 'sum_sent' ---------------------------------------------- trial-2063 gprs:sysmo cs_paging_gprs_active.py ---------------------------------------------- 14:48:22.031224 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.041759 tst gprs:sysmo: Using 1 x bts (candidates: 1) 14:48:22.174406 tst cs_paging_gprs_active.py: using LAC 3938 14:48:22.304245 tst cs_paging_gprs_active.py: using RAC 113 14:48:22.434228 tst cs_paging_gprs_active.py: using CellId 3938 14:48:22.561473 tst cs_paging_gprs_active.py: using BVCI 3939 14:48:22.586466 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.597593 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.608745 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.619959 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.631079 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.642141 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.653433 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.664596 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.676004 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:48:22.792729 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:48:23.088461 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003... 14:48:23.226667 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:48:23.563098 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7213): Launched 14:48:23.846829 run iperf3-srv_10.42.42.10(pid=7214): Launched 14:48:24.046214 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004... 14:48:24.182158 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:48:24.518521 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7215): Launched 14:48:24.798925 run iperf3-srv_10.42.42.10(pid=7216): Launched 14:48:24.998399 tst cs_paging_gprs_active.py:38: start network... 14:48:25.134681 run osmo-hlr_10.42.42.2: Starting osmo-hlr 14:48:25.653564 run create_hlr_db(pid=7217): Launched 14:48:25.841578 bus /sierra_2: Setting Powered False 14:48:26.862685 run create_hlr_db(pid=7217): Terminated: ok {rc=0} 14:48:27.271543 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7219): Launched 14:48:27.574888 run osmo-hlr_10.42.42.2(pid=7220): Launched 14:48:27.684899 run osmo-stp_10.42.42.5: Starting osmo-stp 14:48:28.209065 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7221): Launched 14:48:28.513243 run osmo-stp_10.42.42.5(pid=7222): Launched 14:48:28.622479 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 14:48:30.095800 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7223): Launched 14:48:30.204831 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 14:48:30.475844 run patchelf(pid=7224): Launched 14:48:31.657441 run patchelf(pid=7224): Terminated: ok {rc=0} 14:48:31.769758 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 14:48:32.052772 run osmo-gsm-tester_setcap_net_admin.sh(pid=7225): Launched 14:48:33.246544 run osmo-gsm-tester_setcap_net_admin.sh(pid=7225): Terminated: ok {rc=0} 14:48:33.578545 run osmo-ggsn_10.42.42.6(pid=7228): Launched 14:48:33.791147 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 14:48:38.722115 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7245): Launched 14:48:39.048605 run osmo-sgsn_10.42.42.7(pid=7246): Launched 14:48:39.168175 run osmo-msc_10.42.42.8: Starting osmo-msc 14:48:39.779910 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7247): Launched 14:48:40.107203 run osmo-msc_10.42.42.8(pid=7248): Launched 14:48:43.280837 run osmo-mgw_10.42.42.3: Starting osmo-mgw 14:48:43.845358 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7249): Launched 14:48:44.172195 run osmo-mgw_10.42.42.3(pid=7250): Launched 14:48:44.289045 run osmo-mgw_10.42.42.4: Starting osmo-mgw 14:48:44.845673 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7251): Launched 14:48:45.171411 run osmo-mgw_10.42.42.4(pid=7252): Launched 14:48:45.288464 run osmo-bsc_10.42.42.9: Starting osmo-bsc 14:48:45.969274 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7253): Launched 14:48:46.297633 run osmo-bsc_10.42.42.9(pid=7254): Launched 14:48:52.576579 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 14:48:53.158669 run rm-remote-dir(pid=7255): Launched 14:48:55.361149 run rm-remote-dir(pid=7255): Terminated: ok {rc=0} 14:48:55.710310 run mk-remote-dir(pid=7256): Launched 14:48:56.910331 run mk-remote-dir(pid=7256): Terminated: ok {rc=0} 14:48:57.204711 run scp-inst-to-remote(pid=7257): Launched 14:49:09.412339 run scp-inst-to-remote(pid=7257): Terminated: ok {rc=0} 14:49:09.732779 run mk-remote-dir(pid=7259): Launched 14:49:10.917230 run mk-remote-dir(pid=7259): Terminated: ok {rc=0} 14:49:11.191153 run scp-cfg-to-remote(pid=7260): Launched 14:49:12.376738 run scp-cfg-to-remote(pid=7260): Terminated: ok {rc=0} 14:49:12.693163 run reload-dsp-firmware(pid=7262): Launched 14:49:14.888741 run reload-dsp-firmware(pid=7262): Terminated: ok {rc=0} 14:49:15.219135 run osmo-bts-sysmo(pid=7263): Launched 14:49:19.533464 tst cs_paging_gprs_active.py:38: Waiting for bts to be ready... 14:49:21.540163 run rm-remote-dir(pid=7264): Launched 14:49:23.734555 run rm-remote-dir(pid=7264): Terminated: ok {rc=0} 14:49:24.069098 run mk-remote-dir(pid=7265): Launched 14:49:26.261520 run mk-remote-dir(pid=7265): Terminated: ok {rc=0} 14:49:26.545760 run scp-inst-to-bts(pid=7266): Launched 14:49:39.749538 run scp-inst-to-bts(pid=7266): Terminated: ok {rc=0} 14:49:40.062109 run mk-remote-run-dir(pid=7268): Launched 14:49:42.244717 run mk-remote-run-dir(pid=7268): Terminated: ok {rc=0} 14:49:42.512487 run scp-cfg-to-sysmobts(pid=7269): Launched 14:49:44.693647 run scp-cfg-to-sysmobts(pid=7269): Terminated: ok {rc=0} 14:49:45.005517 run osmo-pcu(pid=7271): Launched 14:49:45.134859 tst cs_paging_gprs_active.py: using MSISDN 3938 14:49:45.307336 bus /gobi_6: Setting Powered True 14:49:49.056938 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3938', subscriber_id=1} 14:49:49.357157 bus /gobi_6: Setting Online False 14:49:50.543899 bus /gobi_6: Setting Powered False 14:49:57.274303 bus /gobi_6: Setting Powered True 14:49:58.644187 bus /gobi_6: Setting Online True 14:50:01.100871 tst /gobi_6: Connect to ('901', '70') 14:50:01.376640 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648' 14:50:01.499808 tst cs_paging_gprs_active.py: using MSISDN 3939 14:50:01.731248 bus /gobi_0: Setting Powered True 14:50:05.493586 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3939', subscriber_id=2} 14:50:05.791951 bus /gobi_0: Setting Online False 14:50:06.992399 bus /gobi_0: Setting Powered False 14:50:08.168670 tst /gobi_6: Already registered with network ('901', '70') 14:50:14.219088 bus /gobi_0: Setting Powered True 14:50:15.604844 bus /gobi_0: Setting Online True 14:50:18.048545 tst /gobi_0: Connect to ('901', '70') 14:50:18.319167 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873' 14:50:18.503092 tst cs_paging_gprs_active.py:38: waiting for modems to attach... 14:50:25.116359 tst /gobi_0: Already registered with network ('901', '70') 14:50:37.004753 tst cs_paging_gprs_active.py:38: waiting for modems to attach to data services... 14:50:40.026169 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 14:50:40.224451 tst cs_paging_gprs_active.py:38: Setting up data plan for "'/gobi_6/context3'" 14:50:40.607671 run move_netns(pid=7281): Launched 14:50:41.793971 run move_netns(pid=7281): Terminated: ok {rc=0} 14:50:42.118673 run ifup(pid=7287): Launched 14:50:43.300242 run ifup(pid=7287): Terminated: ok {rc=0} 14:50:43.616952 run dhcp(pid=7290): Launched 14:50:44.800545 run dhcp(pid=7290): Terminated: ok {rc=0} 14:50:49.713394 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 14:50:49.901542 tst cs_paging_gprs_active.py:38: Setting up data plan for "'/gobi_0/context2'" 14:50:50.283026 run move_netns(pid=7307): Launched 14:50:51.476474 run move_netns(pid=7307): Terminated: ok {rc=0} 14:50:51.815520 run ifup(pid=7312): Launched 14:50:53.009046 run ifup(pid=7312): Terminated: ok {rc=0} 14:50:53.349739 run dhcp(pid=7315): Launched 14:50:54.533914 run dhcp(pid=7315): Terminated: ok {rc=0} 14:50:54.726070 tst cs_paging_gprs_active.py:38: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'" 14:50:54.854607 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s) 14:50:55.228108 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331): Launched 14:50:55.473530 tst cs_paging_gprs_active.py:38: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'" 14:50:55.608645 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s) 14:50:55.997771 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334): Launched 14:50:56.333882 run iperf3-cli_10.42.42.10(pid=7337): Launched 14:50:56.610987 run iperf3-cli_10.42.42.10(pid=7340): Launched 14:50:56.820283 tst cs_paging_gprs_active.py:13: waiting a few secs to make sure iperf3 test is running 14:50:59.073888 tst cs_paging_gprs_active.py:16: iperf3 running, let's place a call 14:51:03.820307 tst cs_paging_gprs_active.py:22: dial success 14:51:07.007728 tst cs_paging_gprs_active.py:28: answer success, call established and ongoing 14:51:13.966512 tst cs_paging_gprs_active.py:35: hangup success 14:52:05.298166 run iperf3-cli_10.42.42.10(pid=7337): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=7337)] 14:52:05.429904 run iperf3-cli_10.42.42.10(pid=7337): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 14:52:05.561388 run iperf3-cli_10.42.42.10(pid=7337): stderr: | (launched: 2022-02-24_14:50:56.214926) 14:52:06.750891 run iperf3-cli_10.42.42.10(pid=7340): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=7340)] 14:52:06.881639 run iperf3-cli_10.42.42.10(pid=7340): stdout: | "bits_per_second": 1633929.1177060113, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 14:52:07.013815 run iperf3-cli_10.42.42.10(pid=7340): stderr: | (launched: 2022-02-24_14:50:56.495962) 14:52:07.135924 run iperf3-srv_10.42.42.10(pid=7214): Terminating (SIGINT) 14:52:07.471897 run iperf3-srv_10.42.42.10(pid=7214): Terminated {rc=1} 14:52:07.666378 tst cs_paging_gprs_active.py:38: Results for 10.42.42.10:5003 through "'/gobi_6/context3'" 14:52:07.719662 tst cs_paging_gprs_active.py:38: ERR: KeyError: 'sum_sent' [trial-2063↪gprs:sysmo↪cs_paging_gprs_active.py:38] 14:52:07.726850 tst cs_paging_gprs_active.py:38: Test FAILED (225.7 sec) 14:52:07.762050 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7213): Terminating (SIGTERM) 14:52:07.802770 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7215): Terminating (SIGTERM) 14:52:07.843070 run iperf3-srv_10.42.42.10(pid=7216): Terminating (SIGTERM) 14:52:07.883741 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7219): Terminating (SIGTERM) 14:52:07.924266 run osmo-hlr_10.42.42.2(pid=7220): Terminating (SIGTERM) 14:52:07.966998 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7221): Terminating (SIGTERM) 14:52:08.009572 run osmo-stp_10.42.42.5(pid=7222): Terminating (SIGTERM) 14:52:08.062238 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7223): Terminating (SIGTERM) 14:52:08.108608 run osmo-ggsn_10.42.42.6(pid=7228): Terminating (SIGTERM) 14:52:08.190209 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7245): Terminating (SIGTERM) 14:52:08.257693 run osmo-sgsn_10.42.42.7(pid=7246): Terminating (SIGTERM) 14:52:08.307044 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7247): Terminating (SIGTERM) 14:52:08.359840 run osmo-msc_10.42.42.8(pid=7248): Terminating (SIGTERM) 14:52:08.404650 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7249): Terminating (SIGTERM) 14:52:08.447631 run osmo-mgw_10.42.42.3(pid=7250): Terminating (SIGTERM) 14:52:08.491355 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7251): Terminating (SIGTERM) 14:52:08.534186 run osmo-mgw_10.42.42.4(pid=7252): Terminating (SIGTERM) 14:52:08.581171 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7253): Terminating (SIGTERM) 14:52:08.622659 run osmo-bsc_10.42.42.9(pid=7254): Terminating (SIGTERM) 14:52:08.663309 run osmo-bts-sysmo(pid=7263): Terminating (SIGTERM) 14:52:08.704070 run osmo-pcu(pid=7271): Terminating (SIGTERM) 14:52:08.744307 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331): Terminating (SIGTERM) 14:52:08.887254 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGTERM(pid=7358): Launched 14:52:09.986266 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGTERM(pid=7358): Terminated: ok {rc=0} 14:52:10.033437 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334): Terminating (SIGTERM) 14:52:10.188283 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGTERM(pid=7364): Launched 14:52:11.280739 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGTERM(pid=7364): Terminated: ok {rc=0} 14:52:11.294490 --- ParallelTerminationStrategy: PID 7213 died... 14:52:11.347694 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7213): Terminated: ok {rc=0} 14:52:11.357187 --- ParallelTerminationStrategy: PID 7215 died... 14:52:11.409945 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7215): Terminated: ok {rc=0} 14:52:11.419382 --- ParallelTerminationStrategy: PID 7216 died... 14:52:11.472411 run iperf3-srv_10.42.42.10(pid=7216): Terminated {rc=256} 14:52:11.482343 --- ParallelTerminationStrategy: PID 7219 died... 14:52:11.535315 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7219): Terminated: ok {rc=0} 14:52:11.545177 --- ParallelTerminationStrategy: PID 7220 died... 14:52:11.598082 run osmo-hlr_10.42.42.2(pid=7220): Terminated: ok {rc=0} 14:52:11.607574 --- ParallelTerminationStrategy: PID 7221 died... 14:52:11.660587 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7221): Terminated: ok {rc=0} 14:52:11.670016 --- ParallelTerminationStrategy: PID 7222 died... 14:52:11.723098 run osmo-stp_10.42.42.5(pid=7222): Terminated: ok {rc=0} 14:52:11.732554 --- ParallelTerminationStrategy: PID 7223 died... 14:52:11.785603 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7223): Terminated: ok {rc=0} 14:52:11.795162 --- ParallelTerminationStrategy: PID 7228 died... 14:52:11.847934 run osmo-ggsn_10.42.42.6(pid=7228): Terminated: ok {rc=0} 14:52:11.857394 --- ParallelTerminationStrategy: PID 7245 died... 14:52:11.910940 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7245): Terminated: ok {rc=0} 14:52:11.920505 --- ParallelTerminationStrategy: PID 7246 died... 14:52:11.973966 run osmo-sgsn_10.42.42.7(pid=7246): Terminated: ok {rc=0} 14:52:11.983623 --- ParallelTerminationStrategy: PID 7247 died... 14:52:12.037553 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7247): Terminated: ok {rc=0} 14:52:12.047629 --- ParallelTerminationStrategy: PID 7248 died... 14:52:12.100447 run osmo-msc_10.42.42.8(pid=7248): Terminated: ok {rc=0} 14:52:12.109965 --- ParallelTerminationStrategy: PID 7249 died... 14:52:12.162766 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7249): Terminated: ok {rc=0} 14:52:12.172349 --- ParallelTerminationStrategy: PID 7250 died... 14:52:12.225738 run osmo-mgw_10.42.42.3(pid=7250): Terminated {rc=15} 14:52:12.235382 --- ParallelTerminationStrategy: PID 7251 died... 14:52:12.288141 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7251): Terminated: ok {rc=0} 14:52:12.298151 --- ParallelTerminationStrategy: PID 7252 died... 14:52:12.351253 run osmo-mgw_10.42.42.4(pid=7252): Terminated {rc=15} 14:52:12.360813 --- ParallelTerminationStrategy: PID 7253 died... 14:52:12.413676 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7253): Terminated: ok {rc=0} 14:52:12.423388 --- ParallelTerminationStrategy: PID 7254 died... 14:52:12.476127 run osmo-bsc_10.42.42.9(pid=7254): Terminated: ok {rc=0} 14:52:12.485672 --- ParallelTerminationStrategy: PID 7263 died... 14:52:12.538741 run osmo-bts-sysmo(pid=7263): Terminated: ok {rc=0} 14:52:12.548411 --- ParallelTerminationStrategy: PID 7271 died... 14:52:12.601783 run osmo-pcu(pid=7271): Terminated: ok {rc=0} 14:52:18.466772 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331): Terminating (SIGINT) 14:52:18.612658 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGINT(pid=7367): Launched 14:52:19.701627 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGINT(pid=7367): Terminated: ok {rc=0} 14:52:19.746179 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334): Terminating (SIGINT) 14:52:19.889173 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGINT(pid=7370): Launched 14:52:20.978023 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGINT(pid=7370): Terminated: ok {rc=0} 14:52:26.847385 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331): Terminating (SIGKILL) 14:52:26.987678 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGUSR1(pid=7373): Launched 14:52:28.080982 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGUSR1(pid=7373): Terminated: ok {rc=0} 14:52:28.128023 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334): Terminating (SIGKILL) 14:52:28.284527 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGUSR1(pid=7376): Launched 14:52:29.373611 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGUSR1(pid=7376): Terminated: ok {rc=0} 14:52:29.597794 bus /gobi_0: Setting Online False 14:52:31.711479 bus /gobi_0: Setting Powered False 14:52:46.981796 bus /gobi_0: Setting Powered False 14:52:48.082842 bus /gobi_0: Setting Powered False 14:52:49.229733 bus /gobi_6: Setting Online False 14:52:50.363674 bus /gobi_6: Setting Powered False 14:52:58.130594 bus /gobi_6: Setting Powered False 1gprs:sysmo210.0'sum_sent'2063iperf3.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3.py", line 8, in <module> setup_run_iperf3_test_parallel(1) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel print_results(clients[i], servers[i]) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results cli.print_results() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results print_result_node_tcp(self.get_results(), 'client') File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp raise e File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp sent = result['end']['sum_sent'] KeyError: 'sum_sent' ---------------------------------------------- trial-2063 gprs:sysmo iperf3.py ---------------------------------------------- 14:52:59.295047 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.306116 tst gprs:sysmo: Using 1 x bts (candidates: 1) 14:52:59.428640 tst iperf3.py: using LAC 3939 14:52:59.545284 tst iperf3.py: using RAC 114 14:52:59.662409 tst iperf3.py: using CellId 3939 14:52:59.782121 tst iperf3.py: using BVCI 3940 14:52:59.805138 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.815756 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.826118 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.836712 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.847245 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.857680 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.868371 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.878856 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.889541 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:53:00.164579 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003... 14:53:00.289703 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:53:00.603715 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Launched 14:53:00.864326 run iperf3-srv_10.42.42.10(pid=7380): Launched 14:53:01.046990 tst iperf3.py:8: start network... 14:53:01.171095 run osmo-hlr_10.42.42.2: Starting osmo-hlr 14:53:01.655377 run create_hlr_db(pid=7381): Launched 14:53:01.827395 bus /gobi_6: Setting Powered False 14:53:02.849823 run create_hlr_db(pid=7381): Terminated: ok {rc=0} 14:53:03.250893 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Launched 14:53:03.554420 run osmo-hlr_10.42.42.2(pid=7384): Launched 14:53:03.663603 run osmo-stp_10.42.42.5: Starting osmo-stp 14:53:04.185074 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Launched 14:53:04.488975 run osmo-stp_10.42.42.5(pid=7386): Launched 14:53:04.595159 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 14:53:05.118502 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Launched 14:53:05.225061 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 14:53:05.488675 run patchelf(pid=7388): Launched 14:53:06.667431 run patchelf(pid=7388): Terminated: ok {rc=0} 14:53:06.777548 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 14:53:07.057040 run osmo-gsm-tester_setcap_net_admin.sh(pid=7389): Launched 14:53:08.247348 run osmo-gsm-tester_setcap_net_admin.sh(pid=7389): Terminated: ok {rc=0} 14:53:08.575910 run osmo-ggsn_10.42.42.6(pid=7392): Launched 14:53:08.799101 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 14:53:09.379533 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Launched 14:53:09.705996 run osmo-sgsn_10.42.42.7(pid=7410): Launched 14:53:09.825923 run osmo-msc_10.42.42.8: Starting osmo-msc 14:53:10.420843 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Launched 14:53:10.749992 run osmo-msc_10.42.42.8(pid=7412): Launched 14:53:13.923969 run osmo-mgw_10.42.42.3: Starting osmo-mgw 14:53:14.490463 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Launched 14:53:14.818634 run osmo-mgw_10.42.42.3(pid=7414): Launched 14:53:14.935436 run osmo-mgw_10.42.42.4: Starting osmo-mgw 14:53:15.490641 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Launched 14:53:15.817442 run osmo-mgw_10.42.42.4(pid=7416): Launched 14:53:15.934094 run osmo-bsc_10.42.42.9: Starting osmo-bsc 14:53:16.616082 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Launched 14:53:16.945354 run osmo-bsc_10.42.42.9(pid=7418): Launched 14:53:23.223754 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 14:53:23.805496 run rm-remote-dir(pid=7419): Launched 14:53:26.007601 run rm-remote-dir(pid=7419): Terminated: ok {rc=0} 14:53:26.354965 run mk-remote-dir(pid=7420): Launched 14:53:27.553974 run mk-remote-dir(pid=7420): Terminated: ok {rc=0} 14:53:27.848702 run scp-inst-to-remote(pid=7421): Launched 14:53:40.054041 run scp-inst-to-remote(pid=7421): Terminated: ok {rc=0} 14:53:40.375519 run mk-remote-dir(pid=7423): Launched 14:53:41.559384 run mk-remote-dir(pid=7423): Terminated: ok {rc=0} 14:53:41.833619 run scp-cfg-to-remote(pid=7424): Launched 14:53:43.030303 run scp-cfg-to-remote(pid=7424): Terminated: ok {rc=0} 14:53:43.370482 run reload-dsp-firmware(pid=7426): Launched 14:53:45.567725 run reload-dsp-firmware(pid=7426): Terminated: ok {rc=0} 14:53:45.900049 run osmo-bts-sysmo(pid=7427): Launched 14:53:50.214549 tst iperf3.py:8: Waiting for bts to be ready... 14:53:51.857035 run rm-remote-dir(pid=7428): Launched 14:53:54.040240 run rm-remote-dir(pid=7428): Terminated: ok {rc=0} 14:53:54.353906 run mk-remote-dir(pid=7429): Launched 14:53:56.536674 run mk-remote-dir(pid=7429): Terminated: ok {rc=0} 14:53:56.804438 run scp-inst-to-bts(pid=7430): Launched 14:54:10.001885 run scp-inst-to-bts(pid=7430): Terminated: ok {rc=0} 14:54:10.315974 run mk-remote-run-dir(pid=7432): Launched 14:54:12.498382 run mk-remote-run-dir(pid=7432): Terminated: ok {rc=0} 14:54:12.765911 run scp-cfg-to-sysmobts(pid=7433): Launched 14:54:14.947582 run scp-cfg-to-sysmobts(pid=7433): Terminated: ok {rc=0} 14:54:15.261922 run osmo-pcu(pid=7435): Launched 14:54:15.385379 tst iperf3.py: using MSISDN 3940 14:54:15.564610 bus /gobi_6: Setting Powered True 14:54:19.048990 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3940', subscriber_id=1} 14:54:19.322831 bus /gobi_6: Setting Online False 14:54:20.517371 bus /gobi_6: Setting Powered False 14:54:27.288487 bus /gobi_6: Setting Powered True 14:54:28.696759 bus /gobi_6: Setting Online True 14:54:31.166646 tst /gobi_6: Connect to ('901', '70') 14:54:31.444845 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648' 14:54:31.646027 tst iperf3.py:8: waiting for modems to attach... 14:54:44.244803 tst /gobi_6: Already registered with network ('901', '70') 14:54:50.090265 tst iperf3.py:8: waiting for modems to attach to data services... 14:55:14.104375 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context4', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 14:55:14.305778 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context4'" 14:55:14.682650 run move_netns(pid=7442): Launched 14:55:15.879244 run move_netns(pid=7442): Terminated: ok {rc=0} 14:55:16.225805 run ifup(pid=7445): Launched 14:55:17.421131 run ifup(pid=7445): Terminated: ok {rc=0} 14:55:17.765629 run dhcp(pid=7448): Launched 14:55:18.962892 run dhcp(pid=7448): Terminated: ok {rc=0} 14:55:19.172461 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context4'" 14:55:19.312473 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s) 14:55:19.714895 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Launched 14:55:20.032224 run iperf3-cli_10.42.42.10(pid=7467): Launched 14:56:28.353069 run iperf3-cli_10.42.42.10(pid=7467): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=7467)] 14:56:28.490608 run iperf3-cli_10.42.42.10(pid=7467): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 14:56:28.626915 run iperf3-cli_10.42.42.10(pid=7467): stderr: | (launched: 2022-02-24_14:55:19.919964) 14:56:28.752451 run iperf3-srv_10.42.42.10(pid=7380): Terminating (SIGINT) 14:56:28.926316 run iperf3-srv_10.42.42.10(pid=7380): Terminated {rc=1} 14:56:29.129427 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context4'" 14:56:29.188793 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2063↪gprs:sysmo↪iperf3.py:8] 14:56:29.196330 tst iperf3.py:8: Test FAILED (209.9 sec) 14:56:29.234515 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Terminating (SIGTERM) 14:56:29.278421 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Terminating (SIGTERM) 14:56:29.321844 run osmo-hlr_10.42.42.2(pid=7384): Terminating (SIGTERM) 14:56:29.365725 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Terminating (SIGTERM) 14:56:29.410343 run osmo-stp_10.42.42.5(pid=7386): Terminating (SIGTERM) 14:56:29.480269 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Terminating (SIGTERM) 14:56:29.536713 run osmo-ggsn_10.42.42.6(pid=7392): Terminating (SIGTERM) 14:56:29.605153 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Terminating (SIGTERM) 14:56:29.661429 run osmo-sgsn_10.42.42.7(pid=7410): Terminating (SIGTERM) 14:56:29.710880 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Terminating (SIGTERM) 14:56:29.759276 run osmo-msc_10.42.42.8(pid=7412): Terminating (SIGTERM) 14:56:29.805931 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Terminating (SIGTERM) 14:56:29.849720 run osmo-mgw_10.42.42.3(pid=7414): Terminating (SIGTERM) 14:56:29.892989 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Terminating (SIGTERM) 14:56:29.935638 run osmo-mgw_10.42.42.4(pid=7416): Terminating (SIGTERM) 14:56:29.979178 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Terminating (SIGTERM) 14:56:30.021566 run osmo-bsc_10.42.42.9(pid=7418): Terminating (SIGTERM) 14:56:30.064127 run osmo-bts-sysmo(pid=7427): Terminating (SIGTERM) 14:56:30.106568 run osmo-pcu(pid=7435): Terminating (SIGTERM) 14:56:30.150385 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGTERM) 14:56:30.330809 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGTERM(pid=7491): Launched 14:56:31.430370 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGTERM(pid=7491): Terminated: ok {rc=0} 14:56:31.444735 --- ParallelTerminationStrategy: PID 7379 died... 14:56:31.501462 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Terminated: ok {rc=0} 14:56:31.511665 --- ParallelTerminationStrategy: PID 7383 died... 14:56:31.567778 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Terminated: ok {rc=0} 14:56:31.577923 --- ParallelTerminationStrategy: PID 7384 died... 14:56:31.633998 run osmo-hlr_10.42.42.2(pid=7384): Terminated: ok {rc=0} 14:56:31.644128 --- ParallelTerminationStrategy: PID 7385 died... 14:56:31.700390 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Terminated: ok {rc=0} 14:56:31.710451 --- ParallelTerminationStrategy: PID 7386 died... 14:56:31.766507 run osmo-stp_10.42.42.5(pid=7386): Terminated: ok {rc=0} 14:56:31.776650 --- ParallelTerminationStrategy: PID 7387 died... 14:56:31.832559 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Terminated: ok {rc=0} 14:56:31.842682 --- ParallelTerminationStrategy: PID 7392 died... 14:56:31.898542 run osmo-ggsn_10.42.42.6(pid=7392): Terminated: ok {rc=0} 14:56:31.908707 --- ParallelTerminationStrategy: PID 7409 died... 14:56:31.964857 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Terminated: ok {rc=0} 14:56:31.975086 --- ParallelTerminationStrategy: PID 7410 died... 14:56:32.031173 run osmo-sgsn_10.42.42.7(pid=7410): Terminated: ok {rc=0} 14:56:32.041501 --- ParallelTerminationStrategy: PID 7411 died... 14:56:32.098009 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Terminated: ok {rc=0} 14:56:32.108316 --- ParallelTerminationStrategy: PID 7412 died... 14:56:32.164366 run osmo-msc_10.42.42.8(pid=7412): Terminated: ok {rc=0} 14:56:32.174811 --- ParallelTerminationStrategy: PID 7413 died... 14:56:32.230845 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Terminated: ok {rc=0} 14:56:32.241556 --- ParallelTerminationStrategy: PID 7414 died... 14:56:32.298367 run osmo-mgw_10.42.42.3(pid=7414): Terminated {rc=15} 14:56:32.308818 --- ParallelTerminationStrategy: PID 7415 died... 14:56:32.365212 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Terminated: ok {rc=0} 14:56:32.375524 --- ParallelTerminationStrategy: PID 7416 died... 14:56:32.431502 run osmo-mgw_10.42.42.4(pid=7416): Terminated {rc=15} 14:56:32.441948 --- ParallelTerminationStrategy: PID 7417 died... 14:56:32.497975 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Terminated: ok {rc=0} 14:56:32.508584 --- ParallelTerminationStrategy: PID 7418 died... 14:56:32.565038 run osmo-bsc_10.42.42.9(pid=7418): Terminated: ok {rc=0} 14:56:32.578391 --- ParallelTerminationStrategy: PID 7427 died... 14:56:32.635438 run osmo-bts-sysmo(pid=7427): Terminated: ok {rc=0} 14:56:32.645632 --- ParallelTerminationStrategy: PID 7435 died... 14:56:32.702106 run osmo-pcu(pid=7435): Terminated: ok {rc=0} 14:56:38.569626 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGINT) 14:56:38.719506 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGINT(pid=7496): Launched 14:56:39.814286 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGINT(pid=7496): Terminated: ok {rc=0} 14:56:45.685912 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGKILL) 14:56:45.841834 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGUSR1(pid=7499): Launched 14:56:46.937002 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGUSR1(pid=7499): Terminated: ok {rc=0} 14:56:47.210220 bus /gobi_6: Setting Online False 14:56:50.050377 bus /gobi_6: Setting Powered False 14:57:03.977535 bus /gobi_6: Setting Powered False 1gprs:sysmo289.0'sum_sent'2063iperf3m4.pyfalseFAILEDTraceback (most recent call last): File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run self.path) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file spec.loader.exec_module( importlib.util.module_from_spec(spec) ) File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module> setup_run_iperf3_test_parallel(4) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel print_results(clients[i], servers[i]) File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results cli.print_results() File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results print_result_node_tcp(self.get_results(), 'client') File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp raise e File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp sent = result['end']['sum_sent'] KeyError: 'sum_sent' ---------------------------------------------- trial-2063 gprs:sysmo iperf3m4.py ---------------------------------------------- 14:57:05.142947 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.154084 tst gprs:sysmo: Using 1 x bts (candidates: 1) 14:57:05.285984 tst iperf3m4.py: using LAC 3940 14:57:05.408861 tst iperf3m4.py: using RAC 115 14:57:05.530496 tst iperf3m4.py: using CellId 3940 14:57:05.651203 tst iperf3m4.py: using BVCI 3941 14:57:05.674472 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.685058 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.695476 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.705761 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.715972 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.726283 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.736695 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.747414 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.758050 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:57:05.878210 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:57:06.000595 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:57:06.116631 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:57:06.412882 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003... 14:57:06.552176 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:57:06.893952 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7502): Launched 14:57:07.178491 run iperf3-srv_10.42.42.10(pid=7503): Launched 14:57:07.380256 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004... 14:57:07.518211 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:57:07.858523 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7504): Launched 14:57:08.142518 run iperf3-srv_10.42.42.10(pid=7505): Launched 14:57:08.344907 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005... 14:57:08.483184 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:57:08.825905 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7506): Launched 14:57:09.109532 run iperf3-srv_10.42.42.10(pid=7507): Launched 14:57:09.310775 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006... 14:57:09.447848 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:57:09.789556 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7508): Launched 14:57:10.073547 run iperf3-srv_10.42.42.10(pid=7509): Launched 14:57:10.275173 tst iperf3m4.py:8: start network... 14:57:10.412219 run osmo-hlr_10.42.42.2: Starting osmo-hlr 14:57:10.933006 run create_hlr_db(pid=7510): Launched 14:57:11.123595 bus /gobi_6: Setting Powered False 14:57:12.144779 run create_hlr_db(pid=7510): Terminated: ok {rc=0} 14:57:12.583491 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7512): Launched 14:57:12.915648 run osmo-hlr_10.42.42.2(pid=7513): Launched 14:57:13.034340 run osmo-stp_10.42.42.5: Starting osmo-stp 14:57:13.603600 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7514): Launched 14:57:13.933630 run osmo-stp_10.42.42.5(pid=7515): Launched 14:57:14.050391 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 14:57:14.622318 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7516): Launched 14:57:14.739855 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 14:57:15.025241 run patchelf(pid=7517): Launched 14:57:16.220041 run patchelf(pid=7517): Terminated: ok {rc=0} 14:57:16.340841 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 14:57:16.632866 run osmo-gsm-tester_setcap_net_admin.sh(pid=7518): Launched 14:57:17.827931 run osmo-gsm-tester_setcap_net_admin.sh(pid=7518): Terminated: ok {rc=0} 14:57:18.162582 run osmo-ggsn_10.42.42.6(pid=7521): Launched 14:57:18.400450 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 14:57:18.964444 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7541): Launched 14:57:19.271312 run osmo-sgsn_10.42.42.7(pid=7542): Launched 14:57:19.383735 run osmo-msc_10.42.42.8: Starting osmo-msc 14:57:19.987766 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7543): Launched 14:57:20.328931 run osmo-msc_10.42.42.8(pid=7544): Launched 14:57:23.492761 run osmo-mgw_10.42.42.3: Starting osmo-mgw 14:57:24.024842 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7545): Launched 14:57:24.332342 run osmo-mgw_10.42.42.3(pid=7546): Launched 14:57:24.441405 run osmo-mgw_10.42.42.4: Starting osmo-mgw 14:57:24.964696 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7547): Launched 14:57:25.274198 run osmo-mgw_10.42.42.4(pid=7548): Launched 14:57:25.383238 run osmo-bsc_10.42.42.9: Starting osmo-bsc 14:57:26.028522 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7549): Launched 14:57:26.340009 run osmo-bsc_10.42.42.9(pid=7550): Launched 14:57:32.603372 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 14:57:33.156303 run rm-remote-dir(pid=7551): Launched 14:57:35.348022 run rm-remote-dir(pid=7551): Terminated: ok {rc=0} 14:57:35.679170 run mk-remote-dir(pid=7552): Launched 14:57:36.868021 run mk-remote-dir(pid=7552): Terminated: ok {rc=0} 14:57:37.146814 run scp-inst-to-remote(pid=7553): Launched 14:57:49.350286 run scp-inst-to-remote(pid=7553): Terminated: ok {rc=0} 14:57:49.675327 run mk-remote-dir(pid=7566): Launched 14:57:50.876374 run mk-remote-dir(pid=7566): Terminated: ok {rc=0} 14:57:51.172903 run scp-cfg-to-remote(pid=7567): Launched 14:57:52.371586 run scp-cfg-to-remote(pid=7567): Terminated: ok {rc=0} 14:57:52.713967 run reload-dsp-firmware(pid=7569): Launched 14:57:54.913290 run reload-dsp-firmware(pid=7569): Terminated: ok {rc=0} 14:57:55.247933 run osmo-bts-sysmo(pid=7570): Launched 14:57:59.566608 tst iperf3m4.py:8: Waiting for bts to be ready... 14:58:01.215936 run rm-remote-dir(pid=7571): Launched 14:58:03.412090 run rm-remote-dir(pid=7571): Terminated: ok {rc=0} 14:58:03.750657 run mk-remote-dir(pid=7572): Launched 14:58:05.945494 run mk-remote-dir(pid=7572): Terminated: ok {rc=0} 14:58:06.235187 run scp-inst-to-bts(pid=7573): Launched 14:58:19.452976 run scp-inst-to-bts(pid=7573): Terminated: ok {rc=0} 14:58:19.792263 run mk-remote-run-dir(pid=7575): Launched 14:58:21.987269 run mk-remote-run-dir(pid=7575): Terminated: ok {rc=0} 14:58:22.276928 run scp-cfg-to-sysmobts(pid=7576): Launched 14:58:24.474262 run scp-cfg-to-sysmobts(pid=7576): Terminated: ok {rc=0} 14:58:24.812158 run osmo-pcu(pid=7578): Launched 14:58:24.943116 tst iperf3m4.py: using MSISDN 3941 14:58:25.114779 bus /gobi_6: Setting Powered True 14:58:28.649791 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3941', subscriber_id=1} 14:58:28.942153 bus /gobi_6: Setting Online False 14:58:30.129798 bus /gobi_6: Setting Powered False 14:58:37.233053 bus /gobi_6: Setting Powered True 14:58:38.609321 bus /gobi_6: Setting Online True 14:58:41.052330 tst /gobi_6: Connect to ('901', '70') 14:58:41.339881 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648' 14:58:41.471904 tst iperf3m4.py: using MSISDN 3942 14:58:41.753082 bus /gobi_0: Setting Powered True 14:58:45.254516 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3942', subscriber_id=2} 14:58:45.528846 bus /gobi_0: Setting Online False 14:58:46.724656 bus /gobi_0: Setting Powered False 14:58:48.226616 tst /gobi_6: Already registered with network ('901', '70') 14:58:54.212561 bus /gobi_0: Setting Powered True 14:58:55.584423 bus /gobi_0: Setting Online True 14:58:58.157401 tst /gobi_0: Connect to ('901', '70') 14:58:58.440418 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873' 14:58:58.565660 tst iperf3m4.py: using MSISDN 3943 14:58:58.856740 bus /gobi_1: Setting Powered True 14:59:02.670251 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023848', msisdn='3943', subscriber_id=3} 14:59:02.970137 bus /gobi_1: Setting Online False 14:59:04.156341 bus /gobi_1: Setting Powered False 14:59:11.225700 bus /gobi_1: Setting Powered True 14:59:12.625478 bus /gobi_1: Setting Online True 14:59:14.077529 tst /gobi_0: Already registered with network ('901', '70') 14:59:15.092529 tst /gobi_1: Connect to ('901', '70') 14:59:15.361081 tst /gobi_1: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024272103' 14:59:15.483904 tst iperf3m4.py: using MSISDN 3944 14:59:15.716849 bus /gobi_4: Setting Powered True 14:59:19.487131 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023849', msisdn='3944', subscriber_id=4} 14:59:19.824159 bus /gobi_4: Setting Online False 14:59:21.012633 bus /gobi_4: Setting Powered False 14:59:24.281798 tst /gobi_1: Already registered with network ('901', '70') 14:59:30.347354 bus /gobi_4: Setting Powered True 14:59:32.109426 bus /gobi_4: Setting Online True 14:59:34.516169 tst /gobi_4: Connect to ('901', '70') 14:59:34.779403 tst /gobi_4: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A08E2G 1 [Jul 21 2016 10:00:50]', Serial: '868323025161511' 14:59:34.965871 tst iperf3m4.py:8: waiting for modems to attach... 14:59:43.843043 tst /gobi_4: Already registered with network ('901', '70') 14:59:47.762384 tst iperf3m4.py:8: waiting for modems to attach to data services... 14:59:56.152488 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 14:59:56.342042 tst iperf3m4.py:8: Setting up data plan for "'/gobi_6/context2'" 14:59:56.703181 run move_netns(pid=7582): Launched 14:59:57.887630 run move_netns(pid=7582): Terminated: ok {rc=0} 14:59:58.221551 run ifup(pid=7585): Launched 14:59:59.417665 run ifup(pid=7585): Terminated: ok {rc=0} 14:59:59.761122 run dhcp(pid=7588): Launched 15:00:00.953640 run dhcp(pid=7588): Terminated: ok {rc=0} 15:00:09.275134 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 15:00:09.474446 tst iperf3m4.py:8: Setting up data plan for "'/gobi_0/context2'" 15:00:09.844945 run move_netns(pid=7639): Launched 15:00:11.040307 run move_netns(pid=7639): Terminated: ok {rc=0} 15:00:11.382551 run ifup(pid=7642): Launched 15:00:12.581356 run ifup(pid=7642): Terminated: ok {rc=0} 15:00:12.966618 run dhcp(pid=7758): Launched 15:00:14.171394 run dhcp(pid=7758): Terminated: ok {rc=0} 15:00:17.280235 tst /gobi_1: context activated {apn='inet46', path='/gobi_1/context8', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.3', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='rf303c5eb721dad', Method='static', Netmask='255.255.255.248'}, Type='internet', Username='ogt'}, user='ogt'} 15:00:17.489249 tst iperf3m4.py:8: Setting up data plan for "'/gobi_1/context8'" 15:00:17.901878 run move_netns(pid=8162): Launched 15:00:19.090335 run move_netns(pid=8162): Terminated: ok {rc=0} 15:00:19.475998 run ifup(pid=8167): Launched 15:00:20.661338 run ifup(pid=8167): Terminated: ok {rc=0} 15:00:20.992737 run dhcp(pid=8171): Launched 15:00:22.187445 run dhcp(pid=8171): Terminated: ok {rc=0} 15:00:25.280098 tst /gobi_4: context activated {apn='inet46', path='/gobi_4/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.4', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.5', Interface='r9d788e55295d1b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 15:00:25.480497 tst iperf3m4.py:8: Setting up data plan for "'/gobi_4/context3'" 15:00:25.861082 run move_netns(pid=8187): Launched 15:00:27.055013 run move_netns(pid=8187): Terminated: ok {rc=0} 15:00:27.396959 run ifup(pid=8192): Launched 15:00:28.592469 run ifup(pid=8192): Terminated: ok {rc=0} 15:00:28.935930 run dhcp(pid=8195): Launched 15:00:30.118339 run dhcp(pid=8195): Terminated: ok {rc=0} 15:00:30.311871 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'" 15:00:30.441692 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s) 15:00:30.832538 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211): Launched 15:00:31.087125 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'" 15:00:31.227368 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s) 15:00:31.629854 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214): Launched 15:00:31.867464 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5005 through "'/gobi_1/context8'" 15:00:31.997979 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5005 (proto=tcp,time=60s) 15:00:32.378224 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217): Launched 15:00:32.615251 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5006 through "'/gobi_4/context3'" 15:00:32.744548 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5006 (proto=tcp,time=60s) 15:00:33.121033 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220): Launched 15:00:33.436181 run iperf3-cli_10.42.42.10(pid=8223): Launched 15:00:33.709012 run iperf3-cli_10.42.42.10(pid=8226): Launched 15:00:33.982758 run iperf3-cli_10.42.42.10(pid=8229): Launched 15:00:34.254084 run iperf3-cli_10.42.42.10(pid=8232): Launched 15:01:48.580595 run iperf3-cli_10.42.42.10(pid=8223): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=8223)] 15:01:48.716453 run iperf3-cli_10.42.42.10(pid=8223): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 15:01:48.851243 run iperf3-cli_10.42.42.10(pid=8223): stderr: | (launched: 2022-02-24_15:00:33.325873) 15:01:50.041884 run iperf3-cli_10.42.42.10(pid=8226): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=8226)] 15:01:50.177302 run iperf3-cli_10.42.42.10(pid=8226): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 15:01:50.312217 run iperf3-cli_10.42.42.10(pid=8226): stderr: | (launched: 2022-02-24_15:00:33.595607) 15:01:51.501975 run iperf3-cli_10.42.42.10(pid=8229): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=8229)] 15:01:51.637844 run iperf3-cli_10.42.42.10(pid=8229): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 15:01:51.773410 run iperf3-cli_10.42.42.10(pid=8229): stderr: | (launched: 2022-02-24_15:00:33.871204) 15:01:52.970974 run iperf3-cli_10.42.42.10(pid=8232): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=8232)] 15:01:53.107355 run iperf3-cli_10.42.42.10(pid=8232): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 15:01:53.242850 run iperf3-cli_10.42.42.10(pid=8232): stderr: | (launched: 2022-02-24_15:00:34.144755) 15:01:53.367810 run iperf3-srv_10.42.42.10(pid=7503): Terminating (SIGINT) 15:01:53.544203 run iperf3-srv_10.42.42.10(pid=7503): Terminated {rc=1} 15:01:53.744508 tst iperf3m4.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context2'" 15:01:53.803366 tst iperf3m4.py:8: ERR: KeyError: 'sum_sent' [trial-2063↪gprs:sysmo↪iperf3m4.py:8] 15:01:53.810817 tst iperf3m4.py:8: Test FAILED (288.7 sec) 15:01:53.847724 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7502): Terminating (SIGTERM) 15:01:53.889851 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7504): Terminating (SIGTERM) 15:01:53.931496 run iperf3-srv_10.42.42.10(pid=7505): Terminating (SIGTERM) 15:01:53.973277 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7506): Terminating (SIGTERM) 15:01:54.014533 run iperf3-srv_10.42.42.10(pid=7507): Terminating (SIGTERM) 15:01:54.055655 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7508): Terminating (SIGTERM) 15:01:54.097200 run iperf3-srv_10.42.42.10(pid=7509): Terminating (SIGTERM) 15:01:54.138843 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7512): Terminating (SIGTERM) 15:01:54.180030 run osmo-hlr_10.42.42.2(pid=7513): Terminating (SIGTERM) 15:01:54.222674 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7514): Terminating (SIGTERM) 15:01:54.264952 run osmo-stp_10.42.42.5(pid=7515): Terminating (SIGTERM) 15:01:54.316643 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7516): Terminating (SIGTERM) 15:01:54.359686 run osmo-ggsn_10.42.42.6(pid=7521): Terminating (SIGTERM) 15:01:54.414640 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7541): Terminating (SIGTERM) 15:01:54.475353 run osmo-sgsn_10.42.42.7(pid=7542): Terminating (SIGTERM) 15:01:54.523364 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7543): Terminating (SIGTERM) 15:01:54.571878 run osmo-msc_10.42.42.8(pid=7544): Terminating (SIGTERM) 15:01:54.623399 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7545): Terminating (SIGTERM) 15:01:54.672342 run osmo-mgw_10.42.42.3(pid=7546): Terminating (SIGTERM) 15:01:54.715647 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7547): Terminating (SIGTERM) 15:01:54.761157 run osmo-mgw_10.42.42.4(pid=7548): Terminating (SIGTERM) 15:01:54.806608 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7549): Terminating (SIGTERM) 15:01:54.850424 run osmo-bsc_10.42.42.9(pid=7550): Terminating (SIGTERM) 15:01:54.893129 run osmo-bts-sysmo(pid=7570): Terminating (SIGTERM) 15:01:54.937285 run osmo-pcu(pid=7578): Terminating (SIGTERM) 15:01:54.980994 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211): Terminating (SIGTERM) 15:01:55.160509 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGTERM(pid=8307): Launched 15:01:56.255071 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGTERM(pid=8307): Terminated: ok {rc=0} 15:01:56.302661 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214): Terminating (SIGTERM) 15:01:56.455382 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGTERM(pid=8312): Launched 15:01:57.544049 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGTERM(pid=8312): Terminated: ok {rc=0} 15:01:57.589366 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217): Terminating (SIGTERM) 15:01:57.731844 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGTERM(pid=8315): Launched 15:01:58.822578 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGTERM(pid=8315): Terminated: ok {rc=0} 15:01:58.868077 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220): Terminating (SIGTERM) 15:01:59.018303 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGTERM(pid=8318): Launched 15:02:00.112528 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGTERM(pid=8318): Terminated: ok {rc=0} 15:02:00.126914 --- ParallelTerminationStrategy: PID 7502 died... 15:02:00.183652 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7502): Terminated: ok {rc=0} 15:02:00.193694 --- ParallelTerminationStrategy: PID 7504 died... 15:02:00.250002 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7504): Terminated: ok {rc=0} 15:02:00.260225 --- ParallelTerminationStrategy: PID 7505 died... 15:02:00.316252 run iperf3-srv_10.42.42.10(pid=7505): Terminated {rc=256} 15:02:00.326140 --- ParallelTerminationStrategy: PID 7506 died... 15:02:00.382130 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7506): Terminated: ok {rc=0} 15:02:00.392139 --- ParallelTerminationStrategy: PID 7507 died... 15:02:00.448230 run iperf3-srv_10.42.42.10(pid=7507): Terminated {rc=256} 15:02:00.458378 --- ParallelTerminationStrategy: PID 7508 died... 15:02:00.514493 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7508): Terminated: ok {rc=0} 15:02:00.524454 --- ParallelTerminationStrategy: PID 7509 died... 15:02:00.580233 run iperf3-srv_10.42.42.10(pid=7509): Terminated {rc=256} 15:02:00.590440 --- ParallelTerminationStrategy: PID 7512 died... 15:02:00.647178 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7512): Terminated: ok {rc=0} 15:02:00.657159 --- ParallelTerminationStrategy: PID 7513 died... 15:02:00.713236 run osmo-hlr_10.42.42.2(pid=7513): Terminated: ok {rc=0} 15:02:00.723496 --- ParallelTerminationStrategy: PID 7514 died... 15:02:00.780010 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7514): Terminated: ok {rc=0} 15:02:00.790120 --- ParallelTerminationStrategy: PID 7515 died... 15:02:00.846618 run osmo-stp_10.42.42.5(pid=7515): Terminated: ok {rc=0} 15:02:00.856613 --- ParallelTerminationStrategy: PID 7516 died... 15:02:00.913025 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7516): Terminated: ok {rc=0} 15:02:00.923093 --- ParallelTerminationStrategy: PID 7521 died... 15:02:00.979203 run osmo-ggsn_10.42.42.6(pid=7521): Terminated: ok {rc=0} 15:02:00.989157 --- ParallelTerminationStrategy: PID 7541 died... 15:02:01.045652 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7541): Terminated: ok {rc=0} 15:02:01.055672 --- ParallelTerminationStrategy: PID 7542 died... 15:02:01.111930 run osmo-sgsn_10.42.42.7(pid=7542): Terminated: ok {rc=0} 15:02:01.121903 --- ParallelTerminationStrategy: PID 7543 died... 15:02:01.178233 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7543): Terminated: ok {rc=0} 15:02:01.188318 --- ParallelTerminationStrategy: PID 7544 died... 15:02:01.244125 run osmo-msc_10.42.42.8(pid=7544): Terminated: ok {rc=0} 15:02:01.254258 --- ParallelTerminationStrategy: PID 7545 died... 15:02:01.310757 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7545): Terminated: ok {rc=0} 15:02:01.320720 --- ParallelTerminationStrategy: PID 7546 died... 15:02:01.376806 run osmo-mgw_10.42.42.3(pid=7546): Terminated {rc=15} 15:02:01.386681 --- ParallelTerminationStrategy: PID 7547 died... 15:02:01.443489 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7547): Terminated: ok {rc=0} 15:02:01.453438 --- ParallelTerminationStrategy: PID 7548 died... 15:02:01.509666 run osmo-mgw_10.42.42.4(pid=7548): Terminated {rc=15} 15:02:01.519680 --- ParallelTerminationStrategy: PID 7549 died... 15:02:01.575676 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7549): Terminated: ok {rc=0} 15:02:01.586309 --- ParallelTerminationStrategy: PID 7550 died... 15:02:01.642032 run osmo-bsc_10.42.42.9(pid=7550): Terminated: ok {rc=0} 15:02:01.652119 --- ParallelTerminationStrategy: PID 7570 died... 15:02:01.708425 run osmo-bts-sysmo(pid=7570): Terminated: ok {rc=0} 15:02:01.718858 --- ParallelTerminationStrategy: PID 7578 died... 15:02:01.774874 run osmo-pcu(pid=7578): Terminated: ok {rc=0} 15:02:07.642330 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211): Terminating (SIGINT) 15:02:07.795985 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGINT(pid=8321): Launched 15:02:08.890247 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGINT(pid=8321): Terminated: ok {rc=0} 15:02:08.943398 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214): Terminating (SIGINT) 15:02:09.089401 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGINT(pid=8324): Launched 15:02:10.178380 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGINT(pid=8324): Terminated: ok {rc=0} 15:02:10.223362 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217): Terminating (SIGINT) 15:02:10.377120 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGINT(pid=8327): Launched 15:02:11.470589 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGINT(pid=8327): Terminated: ok {rc=0} 15:02:11.518248 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220): Terminating (SIGINT) 15:02:11.672212 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGINT(pid=8330): Launched 15:02:12.766813 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGINT(pid=8330): Terminated: ok {rc=0} 15:02:18.637862 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211): Terminating (SIGKILL) 15:02:18.790236 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGUSR1(pid=8333): Launched 15:02:19.884430 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGUSR1(pid=8333): Terminated: ok {rc=0} 15:02:19.931754 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214): Terminating (SIGKILL) 15:02:20.083900 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGUSR1(pid=8336): Launched 15:02:21.178096 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGUSR1(pid=8336): Terminated: ok {rc=0} 15:02:21.225458 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217): Terminating (SIGKILL) 15:02:21.380129 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGUSR1(pid=8339): Launched 15:02:22.469727 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGUSR1(pid=8339): Terminated: ok {rc=0} 15:02:22.514633 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220): Terminating (SIGKILL) 15:02:22.661322 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGUSR1(pid=8342): Launched 15:02:23.751563 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGUSR1(pid=8342): Terminated: ok {rc=0} 15:02:23.985332 bus /gobi_4: Setting Online False 15:02:25.127145 bus /gobi_4: Setting Powered False 15:02:32.981372 bus /gobi_4: Setting Powered False 15:02:34.083212 bus /gobi_4: Setting Powered False 15:02:35.232574 bus /gobi_1: Setting Online False 15:02:36.358060 bus /gobi_1: Setting Powered False 15:02:44.068108 bus /gobi_1: Setting Powered False 15:02:45.170793 bus /gobi_1: Setting Powered False 15:02:46.328132 bus /gobi_0: Setting Online False 15:02:47.460572 bus /gobi_0: Setting Powered False 15:02:55.971201 bus /gobi_0: Setting Powered False 15:02:57.076926 bus /gobi_0: Setting Powered False 15:02:58.226847 bus /gobi_6: Setting Online False 15:02:59.366074 bus /gobi_6: Setting Powered False 15:03:07.982410 bus /gobi_6: Setting Powered False 0gprs:sysmo164.00ping.pyfalsePASSED(launched: 2022-02-24_15:05:25.137632) 8428: waiting for 8429 PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data. 64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=439 ms 64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=618 ms 64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=595 ms 64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=434 ms 64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=435 ms 64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=411 ms 64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=393 ms 64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=389 ms 64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=408 ms 64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=390 ms --- 10.42.42.6 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 21ms rtt min/avg/max/mdev = 388.920/451.214/618.001/79.859 ms child exited with 0 0gprs:sysmo205.00ping_idle_ping.pyfalsePASSED[1] Running 10 ping requests for "'/gobi_6/context2'" (launched: 2022-02-24_15:08:15.132810) 8539: waiting for 8540 PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data. 64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=539 ms 64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=499 ms 64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=456 ms 64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=438 ms 64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=457 ms 64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=453 ms 64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=435 ms 64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=431 ms 64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=431 ms 64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=408 ms --- 10.42.42.6 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 20ms rtt min/avg/max/mdev = 408.119/454.703/539.042/36.087 ms child exited with 0 Sleeping for 60 seconds [2] Running 10 ping requests for "'/gobi_6/context2'" (launched: 2022-02-24_15:09:26.289423) 8544: waiting for 8546 PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data. 64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=468 ms 64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=409 ms 64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=367 ms 64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=363 ms 64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=723 ms 64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=425 ms 64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=420 ms 64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=402 ms 64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=420 ms 64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=416 ms --- 10.42.42.6 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 20ms rtt min/avg/max/mdev = 363.394/441.378/723.056/98.020 ms child exited with 0 3gprs:sysmo20