Skipping 59 KB..
Full Logtrial-2065 gprs:sysmo cs_paging_gprs_active.py
----------------------------------------------
12:12:04.055063 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:12:04.065993 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:12:04.101845 tst cs_paging_gprs_active.py: using LAC 3952
12:12:04.229846 tst cs_paging_gprs_active.py: using RAC 127
12:12:04.356776 tst cs_paging_gprs_active.py: using CellId 3952
12:12:04.483554 tst cs_paging_gprs_active.py: using BVCI 3953
12:12:04.601122 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:12:04.612207 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:12:04.623196 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:12:04.634063 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:12:04.644985 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:12:04.655916 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:12:04.666899 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:12:04.677720 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:12:04.688898 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:12:04.805929 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:12:04.920288 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003...
12:12:05.124128 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:12:05.430616 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31521): Launched
12:12:05.697802 run iperf3-srv_10.42.42.10(pid=31522): Launched
12:12:05.813319 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004...
12:12:06.016414 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:12:06.323330 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31523): Launched
12:12:06.587669 run iperf3-srv_10.42.42.10(pid=31524): Launched
12:12:06.700963 tst cs_paging_gprs_active.py:38: start network...
12:12:06.903951 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:12:07.376369 run create_hlr_db(pid=31525): Launched
12:12:07.494874 bus /sierra_2: Setting Powered False
12:12:08.555785 run create_hlr_db(pid=31525): Terminated: ok {rc=0}
12:12:09.001114 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31527): Launched
12:12:09.328523 run osmo-hlr_10.42.42.2(pid=31528): Launched
12:12:09.453584 run osmo-stp_10.42.42.5: Starting osmo-stp
12:12:10.011583 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31529): Launched
12:12:10.339352 run osmo-stp_10.42.42.5(pid=31530): Launched
12:12:10.462199 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:12:12.043726 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31531): Launched
12:12:12.168015 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:12:12.446651 run patchelf(pid=31532): Launched
12:12:13.633430 run patchelf(pid=31532): Terminated: ok {rc=0}
12:12:13.770128 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:12:14.051612 run osmo-gsm-tester_setcap_net_admin.sh(pid=31534): Launched
12:12:15.242243 run osmo-gsm-tester_setcap_net_admin.sh(pid=31534): Terminated: ok {rc=0}
12:12:15.580780 run osmo-ggsn_10.42.42.6(pid=31537): Launched
12:12:15.708925 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:12:20.641670 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31556): Launched
12:12:20.946591 run osmo-sgsn_10.42.42.7(pid=31557): Launched
12:12:21.063582 run osmo-msc_10.42.42.8: Starting osmo-msc
12:12:21.640017 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31558): Launched
12:12:21.969198 run osmo-msc_10.42.42.8(pid=31559): Launched
12:12:25.152632 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:12:25.711070 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31560): Launched
12:12:26.039060 run osmo-mgw_10.42.42.3(pid=31561): Launched
12:12:26.163781 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:12:26.714355 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31562): Launched
12:12:27.044688 run osmo-mgw_10.42.42.4(pid=31563): Launched
12:12:27.170779 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:12:27.861758 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31564): Launched
12:12:28.191483 run osmo-bsc_10.42.42.9(pid=31565): Launched
12:12:34.482613 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:12:35.050895 run rm-remote-dir(pid=31566): Launched
12:12:37.245721 run rm-remote-dir(pid=31566): Terminated: ok {rc=0}
12:12:37.603901 run mk-remote-dir(pid=31567): Launched
12:12:38.796164 run mk-remote-dir(pid=31567): Terminated: ok {rc=0}
12:12:39.101320 run scp-inst-to-remote(pid=31568): Launched
12:12:51.312186 run scp-inst-to-remote(pid=31568): Terminated: ok {rc=0}
12:12:51.664825 run mk-remote-dir(pid=31570): Launched
12:12:52.854138 run mk-remote-dir(pid=31570): Terminated: ok {rc=0}
12:12:53.157185 run scp-cfg-to-remote(pid=31571): Launched
12:12:54.348223 run scp-cfg-to-remote(pid=31571): Terminated: ok {rc=0}
12:12:54.697604 run reload-dsp-firmware(pid=31573): Launched
12:12:56.883648 run reload-dsp-firmware(pid=31573): Terminated: ok {rc=0}
12:12:57.204223 run osmo-bts-sysmo(pid=31574): Launched
12:13:01.427454 tst cs_paging_gprs_active.py:38: Waiting for bts to be ready...
12:13:03.453366 run rm-remote-dir(pid=31575): Launched
12:13:05.627859 run rm-remote-dir(pid=31575): Terminated: ok {rc=0}
12:13:05.949950 run mk-remote-dir(pid=31576): Launched
12:13:08.123697 run mk-remote-dir(pid=31576): Terminated: ok {rc=0}
12:13:08.400050 run scp-inst-to-bts(pid=31577): Launched
12:13:21.590607 run scp-inst-to-bts(pid=31577): Terminated: ok {rc=0}
12:13:21.913012 run mk-remote-run-dir(pid=31579): Launched
12:13:24.086432 run mk-remote-run-dir(pid=31579): Terminated: ok {rc=0}
12:13:24.363695 run scp-cfg-to-sysmobts(pid=31580): Launched
12:13:26.537586 run scp-cfg-to-sysmobts(pid=31580): Terminated: ok {rc=0}
12:13:26.860017 run osmo-pcu(pid=31582): Launched
12:13:26.986455 tst cs_paging_gprs_active.py: using MSISDN 3960
12:13:27.162000 bus /gobi_6: Setting Powered True
12:13:30.662413 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3960', subscriber_id=1}
12:13:30.937908 bus /gobi_6: Setting Online False
12:13:32.133494 bus /gobi_6: Setting Powered False
12:13:39.132076 bus /gobi_6: Setting Powered True
12:13:40.503513 bus /gobi_6: Setting Online True
12:13:42.930055 tst /gobi_6: Connect to ('901', '70')
12:13:43.199681 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:13:43.325389 tst cs_paging_gprs_active.py: using MSISDN 3961
12:13:43.551885 bus /gobi_0: Setting Powered True
12:13:47.044155 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3961', subscriber_id=2}
12:13:47.316928 bus /gobi_0: Setting Online False
12:13:48.508877 bus /gobi_0: Setting Powered False
12:13:55.261436 bus /gobi_0: Setting Powered True
12:13:56.659955 bus /gobi_0: Setting Online True
12:13:58.172673 tst /gobi_6: Already registered with network ('901', '70')
12:13:59.108195 tst /gobi_0: Connect to ('901', '70')
12:13:59.396156 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873'
12:13:59.511703 tst cs_paging_gprs_active.py:38: waiting for modems to attach...
12:14:13.756502 tst /gobi_0: Already registered with network ('901', '70')
12:14:19.175220 tst cs_paging_gprs_active.py:38: waiting for modems to attach to data services...
12:14:22.300460 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:22.417117 tst cs_paging_gprs_active.py:38: Setting up data plan for "'/gobi_6/context2'"
12:14:22.849839 run move_netns(pid=31585): Launched
12:14:24.034899 run move_netns(pid=31585): Terminated: ok {rc=0}
12:14:24.364426 run ifup(pid=31588): Launched
12:14:25.544581 run ifup(pid=31588): Terminated: ok {rc=0}
12:14:25.869485 run dhcp(pid=31591): Launched
12:14:27.061057 run dhcp(pid=31591): Terminated: ok {rc=0}
12:14:31.424323 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:31.533178 tst cs_paging_gprs_active.py:38: Setting up data plan for "'/gobi_0/context3'"
12:14:31.945356 run move_netns(pid=31609): Launched
12:14:33.126832 run move_netns(pid=31609): Terminated: ok {rc=0}
12:14:33.453340 run ifup(pid=31612): Launched
12:14:34.638127 run ifup(pid=31612): Terminated: ok {rc=0}
12:14:34.989309 run dhcp(pid=31615): Launched
12:14:36.180316 run dhcp(pid=31615): Terminated: ok {rc=0}
12:14:36.316918 tst cs_paging_gprs_active.py:38: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'"
12:14:36.541965 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:14:36.930967 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631): Launched
12:14:37.113828 tst cs_paging_gprs_active.py:38: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context3'"
12:14:37.335576 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s)
12:14:37.720524 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634): Launched
12:14:38.059259 run iperf3-cli_10.42.42.10(pid=31637): Launched
12:14:38.344852 run iperf3-cli_10.42.42.10(pid=31640): Launched
12:14:38.476458 tst cs_paging_gprs_active.py:13: waiting a few secs to make sure iperf3 test is running
12:14:40.728970 tst cs_paging_gprs_active.py:16: iperf3 running, let's place a call
12:14:44.477097 tst cs_paging_gprs_active.py:22: dial success
12:14:47.243164 tst cs_paging_gprs_active.py:28: answer success, call established and ongoing
12:14:53.853265 tst cs_paging_gprs_active.py:35: hangup success
12:17:54.451554 run iperf3-cli_10.42.42.10(pid=31637): Terminating (SIGINT)
12:17:54.797128 run iperf3-cli_10.42.42.10(pid=31637)-killSignals.SIGINT(pid=31661): Launched
12:17:56.049282 run iperf3-cli_10.42.42.10(pid=31637)-killSignals.SIGINT(pid=31661): Terminated: ok {rc=0}
12:17:56.255166 run iperf3-cli_10.42.42.10(pid=31637): Terminated {rc=130}
12:17:56.384184 run iperf3-cli_10.42.42.10(pid=31640): Terminating (SIGINT)
12:17:56.748032 run iperf3-cli_10.42.42.10(pid=31640)-killSignals.SIGINT(pid=31664): Launched
12:17:57.956621 run iperf3-cli_10.42.42.10(pid=31640)-killSignals.SIGINT(pid=31664): Terminated: ok {rc=0}
12:17:58.159707 run iperf3-cli_10.42.42.10(pid=31640): Terminated {rc=1}
12:17:58.287404 tst cs_paging_gprs_active.py:38: ERR: Error: iperf3-cli_10.42.42.10(pid=31637): Wait timeout {condition=<bound method Process.terminated of iperf3-cli_10.42.42.10(pid=31637)>, timeout=180, timestep=1} [trial-2065↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=31637)] [trial-2065↪gprs:sysmo↪cs_paging_gprs_active.py:38]
12:17:58.295799 tst cs_paging_gprs_active.py:38: Test FAILED (354.3 sec)
12:17:58.302965 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31521): Terminating (SIGTERM)
12:17:58.345204 run iperf3-srv_10.42.42.10(pid=31522): Terminating (SIGTERM)
12:17:58.388310 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31523): Terminating (SIGTERM)
12:17:58.431321 run iperf3-srv_10.42.42.10(pid=31524): Terminating (SIGTERM)
12:17:58.472984 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31527): Terminating (SIGTERM)
12:17:58.515295 run osmo-hlr_10.42.42.2(pid=31528): Terminating (SIGTERM)
12:17:58.557092 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31529): Terminating (SIGTERM)
12:17:58.600999 run osmo-stp_10.42.42.5(pid=31530): Terminating (SIGTERM)
12:17:58.644198 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31531): Terminating (SIGTERM)
12:17:58.708689 run osmo-ggsn_10.42.42.6(pid=31537): Terminating (SIGTERM)
12:17:58.752568 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31556): Terminating (SIGTERM)
12:17:58.849606 run osmo-sgsn_10.42.42.7(pid=31557): Terminating (SIGTERM)
12:17:58.893318 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31558): Terminating (SIGTERM)
12:17:58.937455 run osmo-msc_10.42.42.8(pid=31559): Terminating (SIGTERM)
12:17:58.987559 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31560): Terminating (SIGTERM)
12:17:59.031635 run osmo-mgw_10.42.42.3(pid=31561): Terminating (SIGTERM)
12:17:59.074408 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31562): Terminating (SIGTERM)
12:17:59.118398 run osmo-mgw_10.42.42.4(pid=31563): Terminating (SIGTERM)
12:17:59.161001 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31564): Terminating (SIGTERM)
12:17:59.204540 run osmo-bsc_10.42.42.9(pid=31565): Terminating (SIGTERM)
12:17:59.247416 run osmo-bts-sysmo(pid=31574): Terminating (SIGTERM)
12:17:59.291415 run osmo-pcu(pid=31582): Terminating (SIGTERM)
12:17:59.336183 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631): Terminating (SIGTERM)
12:17:59.481651 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631)-killSignals.SIGTERM(pid=31678): Launched
12:18:00.568684 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631)-killSignals.SIGTERM(pid=31678): Terminated: ok {rc=0}
12:18:00.636362 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634): Terminating (SIGTERM)
12:18:00.779169 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634)-killSignals.SIGTERM(pid=31685): Launched
12:18:01.869876 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634)-killSignals.SIGTERM(pid=31685): Terminated: ok {rc=0}
12:18:01.937079 --- ParallelTerminationStrategy: PID 31521 died...
12:18:01.957344 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31521): Terminated: ok {rc=0}
12:18:02.004606 --- ParallelTerminationStrategy: PID 31522 died...
12:18:02.024592 run iperf3-srv_10.42.42.10(pid=31522): Terminated {rc=256}
12:18:02.071815 --- ParallelTerminationStrategy: PID 31523 died...
12:18:02.091889 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31523): Terminated: ok {rc=0}
12:18:02.139203 --- ParallelTerminationStrategy: PID 31524 died...
12:18:02.159020 run iperf3-srv_10.42.42.10(pid=31524): Terminated {rc=256}
12:18:02.205862 --- ParallelTerminationStrategy: PID 31527 died...
12:18:02.225889 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31527): Terminated: ok {rc=0}
12:18:02.272640 --- ParallelTerminationStrategy: PID 31528 died...
12:18:02.292576 run osmo-hlr_10.42.42.2(pid=31528): Terminated: ok {rc=0}
12:18:02.339359 --- ParallelTerminationStrategy: PID 31529 died...
12:18:02.359140 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31529): Terminated: ok {rc=0}
12:18:02.406366 --- ParallelTerminationStrategy: PID 31530 died...
12:18:02.426500 run osmo-stp_10.42.42.5(pid=31530): Terminated: ok {rc=0}
12:18:02.473363 --- ParallelTerminationStrategy: PID 31531 died...
12:18:02.493641 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31531): Terminated: ok {rc=0}
12:18:02.540616 --- ParallelTerminationStrategy: PID 31537 died...
12:18:02.560577 run osmo-ggsn_10.42.42.6(pid=31537): Terminated: ok {rc=0}
12:18:02.607347 --- ParallelTerminationStrategy: PID 31556 died...
12:18:02.627355 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31556): Terminated: ok {rc=0}
12:18:02.673838 --- ParallelTerminationStrategy: PID 31557 died...
12:18:02.693779 run osmo-sgsn_10.42.42.7(pid=31557): Terminated: ok {rc=0}
12:18:02.740401 --- ParallelTerminationStrategy: PID 31558 died...
12:18:02.760782 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31558): Terminated: ok {rc=0}
12:18:02.807608 --- ParallelTerminationStrategy: PID 31559 died...
12:18:02.827603 run osmo-msc_10.42.42.8(pid=31559): Terminated: ok {rc=0}
12:18:02.874836 --- ParallelTerminationStrategy: PID 31560 died...
12:18:02.895047 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31560): Terminated: ok {rc=0}
12:18:02.941972 --- ParallelTerminationStrategy: PID 31561 died...
12:18:02.962065 run osmo-mgw_10.42.42.3(pid=31561): Terminated {rc=15}
12:18:03.009137 --- ParallelTerminationStrategy: PID 31562 died...
12:18:03.029456 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31562): Terminated: ok {rc=0}
12:18:03.076538 --- ParallelTerminationStrategy: PID 31563 died...
12:18:03.096473 run osmo-mgw_10.42.42.4(pid=31563): Terminated {rc=15}
12:18:03.143340 --- ParallelTerminationStrategy: PID 31564 died...
12:18:03.163416 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31564): Terminated: ok {rc=0}
12:18:03.210316 --- ParallelTerminationStrategy: PID 31565 died...
12:18:03.230479 run osmo-bsc_10.42.42.9(pid=31565): Terminated: ok {rc=0}
12:18:03.277289 --- ParallelTerminationStrategy: PID 31574 died...
12:18:03.297393 run osmo-bts-sysmo(pid=31574): Terminated: ok {rc=0}
12:18:03.344962 --- ParallelTerminationStrategy: PID 31582 died...
12:18:03.365042 run osmo-pcu(pid=31582): Terminated: ok {rc=0}
12:18:09.235038 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631): Terminating (SIGINT)
12:18:09.378387 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631)-killSignals.SIGINT(pid=31688): Launched
12:18:10.465284 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631)-killSignals.SIGINT(pid=31688): Terminated: ok {rc=0}
12:18:10.532853 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634): Terminating (SIGINT)
12:18:10.675358 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634)-killSignals.SIGINT(pid=31691): Launched
12:18:11.764192 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634)-killSignals.SIGINT(pid=31691): Terminated: ok {rc=0}
12:18:17.652892 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631): Terminating (SIGKILL)
12:18:17.797391 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631)-killSignals.SIGUSR1(pid=31696): Launched
12:18:18.885493 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31631)-killSignals.SIGUSR1(pid=31696): Terminated: ok {rc=0}
12:18:18.951516 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634): Terminating (SIGKILL)
12:18:19.093240 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634)-killSignals.SIGUSR1(pid=31699): Launched
12:18:20.175828 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31634)-killSignals.SIGUSR1(pid=31699): Terminated: ok {rc=0}
12:18:20.442861 bus /gobi_0: Setting Online False
12:18:21.583997 bus /gobi_0: Setting Powered False
12:18:37.940943 bus /gobi_0: Setting Powered False
12:18:39.039426 bus /gobi_0: Setting Powered False
12:18:40.199975 bus /gobi_6: Setting Online False
12:18:41.330262 bus /gobi_6: Setting Powered False
12:18:49.944541 bus /gobi_6: Setting Powered False
----------------------------------------------
trial-2065 gprs:sysmo iperf3.py
----------------------------------------------
12:18:51.152955 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.164748 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:18:51.200817 tst iperf3.py: using LAC 3953
12:18:51.327157 tst iperf3.py: using RAC 128
12:18:51.450447 tst iperf3.py: using CellId 3953
12:18:51.570379 tst iperf3.py: using BVCI 3954
12:18:51.681299 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.692177 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.702792 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.714053 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.725301 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.736788 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.748269 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.759888 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:18:51.771393 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:18:51.891513 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
12:18:52.112176 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:18:52.440252 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31702): Launched
12:18:52.725609 run iperf3-srv_10.42.42.10(pid=31703): Launched
12:18:52.849053 tst iperf3.py:8: start network...
12:18:53.068423 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:18:53.575278 run create_hlr_db(pid=31704): Launched
12:18:53.695513 bus /gobi_6: Setting Powered False
12:18:54.756492 run create_hlr_db(pid=31704): Terminated: ok {rc=0}
12:18:55.181247 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31706): Launched
12:18:55.491881 run osmo-hlr_10.42.42.2(pid=31707): Launched
12:18:55.608559 run osmo-stp_10.42.42.5: Starting osmo-stp
12:18:56.129055 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31708): Launched
12:18:56.437096 run osmo-stp_10.42.42.5(pid=31709): Launched
12:18:56.554889 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:18:57.077889 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Launched
12:18:57.196272 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:18:57.457097 run patchelf(pid=31711): Launched
12:18:58.632290 run patchelf(pid=31711): Terminated: ok {rc=0}
12:18:58.760530 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:18:59.022206 run osmo-gsm-tester_setcap_net_admin.sh(pid=31712): Launched
12:19:00.198013 run osmo-gsm-tester_setcap_net_admin.sh(pid=31712): Terminated: ok {rc=0}
12:19:00.516435 run osmo-ggsn_10.42.42.6(pid=31715): Launched
12:19:00.651989 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:19:01.356792 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31732): Launched
12:19:01.684904 run osmo-sgsn_10.42.42.7(pid=31733): Launched
12:19:01.811560 run osmo-msc_10.42.42.8: Starting osmo-msc
12:19:02.403386 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31734): Launched
12:19:02.729249 run osmo-msc_10.42.42.8(pid=31735): Launched
12:19:05.911498 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:19:06.467631 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31736): Launched
12:19:06.792836 run osmo-mgw_10.42.42.3(pid=31737): Launched
12:19:06.917162 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:19:07.463517 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31738): Launched
12:19:07.791557 run osmo-mgw_10.42.42.4(pid=31739): Launched
12:19:07.916906 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:19:08.589809 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31740): Launched
12:19:08.917572 run osmo-bsc_10.42.42.9(pid=31741): Launched
12:19:15.200269 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:19:15.730142 run rm-remote-dir(pid=31742): Launched
12:19:17.913733 run rm-remote-dir(pid=31742): Terminated: ok {rc=0}
12:19:18.248953 run mk-remote-dir(pid=31743): Launched
12:19:19.428807 run mk-remote-dir(pid=31743): Terminated: ok {rc=0}
12:19:19.714111 run scp-inst-to-remote(pid=31744): Launched
12:19:31.906661 run scp-inst-to-remote(pid=31744): Terminated: ok {rc=0}
12:19:32.258387 run mk-remote-dir(pid=31746): Launched
12:19:33.447751 run mk-remote-dir(pid=31746): Terminated: ok {rc=0}
12:19:33.749160 run scp-cfg-to-remote(pid=31747): Launched
12:19:34.937493 run scp-cfg-to-remote(pid=31747): Terminated: ok {rc=0}
12:19:35.285526 run reload-dsp-firmware(pid=31749): Launched
12:19:37.474514 run reload-dsp-firmware(pid=31749): Terminated: ok {rc=0}
12:19:37.815341 run osmo-bts-sysmo(pid=31750): Launched
12:19:42.052922 tst iperf3.py:8: Waiting for bts to be ready...
12:19:43.769654 run rm-remote-dir(pid=31751): Launched
12:19:45.955980 run rm-remote-dir(pid=31751): Terminated: ok {rc=0}
12:19:46.296784 run mk-remote-dir(pid=31752): Launched
12:19:48.477629 run mk-remote-dir(pid=31752): Terminated: ok {rc=0}
12:19:48.752155 run scp-inst-to-bts(pid=31753): Launched
12:20:01.942188 run scp-inst-to-bts(pid=31753): Terminated: ok {rc=0}
12:20:02.284158 run mk-remote-run-dir(pid=31759): Launched
12:20:04.465634 run mk-remote-run-dir(pid=31759): Terminated: ok {rc=0}
12:20:04.740026 run scp-cfg-to-sysmobts(pid=31760): Launched
12:20:06.911083 run scp-cfg-to-sysmobts(pid=31760): Terminated: ok {rc=0}
12:20:07.232654 run osmo-pcu(pid=31762): Launched
12:20:07.358003 tst iperf3.py: using MSISDN 3962
12:20:07.533074 bus /gobi_6: Setting Powered True
12:20:11.026635 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3962', subscriber_id=1}
12:20:11.300788 bus /gobi_6: Setting Online False
12:20:12.491991 bus /gobi_6: Setting Powered False
12:20:19.221321 bus /gobi_6: Setting Powered True
12:20:20.609517 bus /gobi_6: Setting Online True
12:20:23.033000 tst /gobi_6: Connect to ('901', '70')
12:20:23.311998 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:20:23.419945 tst iperf3.py:8: waiting for modems to attach...
12:20:37.513878 tst /gobi_6: Already registered with network ('901', '70')
12:20:42.863131 tst iperf3.py:8: waiting for modems to attach to data services...
12:20:45.905969 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:20:46.014911 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
12:20:46.428190 run move_netns(pid=31766): Launched
12:20:47.608612 run move_netns(pid=31766): Terminated: ok {rc=0}
12:20:47.935622 run ifup(pid=31769): Launched
12:20:49.116094 run ifup(pid=31769): Terminated: ok {rc=0}
12:20:49.444709 run dhcp(pid=31772): Launched
12:20:50.631386 run dhcp(pid=31772): Terminated: ok {rc=0}
12:20:50.759297 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
12:20:50.968272 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:20:51.331026 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Launched
12:20:51.647721 run iperf3-cli_10.42.42.10(pid=31792): Launched
12:21:59.955739 run iperf3-cli_10.42.42.10(pid=31792): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=31792)]
12:22:00.093727 run iperf3-cli_10.42.42.10(pid=31792): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:22:00.232041 run iperf3-cli_10.42.42.10(pid=31792): stderr:
| (launched: 2022-06-18_12:20:51.632024)
12:22:00.371472 run iperf3-srv_10.42.42.10(pid=31703): Terminating (SIGINT)
12:22:00.715447 run iperf3-srv_10.42.42.10(pid=31703): Terminated {rc=1}
12:22:00.839357 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
Exception while using iperf3 client results: "{'start': {'connected': [{'socket': 5, 'local_host': '176.16.46.1', 'local_port': 41504, 'remote_host': '10.42.42.10', 'remote_port': 5003}], 'version': 'iperf 3.6', 'system_info': 'Linux osmo-gsm-tester-prod 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64', 'timestamp': {'time': 'Sat, 18 Jun 2022 10:20:52 GMT', 'timesecs': 1655547652}, 'connecting_to': {'host': '10.42.42.10', 'port': 5003}, 'cookie': 'r3uzamxv5umnd6v6cb6goadt4sodtkovgrtu', 'tcp_mss_default': 1448, 'sock_bufsize': 0, 'sndbuf_actual': 16384, 'rcvbuf_actual': 131072, 'test_start': {'protocol': 'TCP', 'num_streams': 1, 'blksize': 131072, 'omit': 0, 'duration': 60, 'bytes': 0, 'blocks': 0, 'reverse': 0, 'tos': 0}}, 'intervals': [{'streams': [{'socket': 5, 'start': 0, 'end': 1.000844955444336, 'seconds': 1.000844955444336, 'bytes': 101360, 'bits_per_second': 810195.420968077, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 419469, 'rttvar': 185019, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 0, 'end': 1.000844955444336, 'seconds': 1.000844955444336, 'bytes': 101360, 'bits_per_second': 810195.420968077, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 1.000844955444336, 'end': 2.001121997833252, 'seconds': 1.000277042388916, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 15928, 'rtt': 506650, 'rttvar': 313126, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 1.000844955444336, 'end': 2.001121997833252, 'seconds': 1.000277042388916, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 2.001121997833252, 'end': 3.001163959503174, 'seconds': 1.0000419616699219, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 18824, 'rtt': 889744, 'rttvar': 862934, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 2.001121997833252, 'end': 3.001163959503174, 'seconds': 1.0000419616699219, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 3.001163959503174, 'end': 4.001378059387207, 'seconds': 1.0002140998840332, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 20272, 'rtt': 1170753, 'rttvar': 1209220, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 3.001163959503174, 'end': 4.001378059387207, 'seconds': 1.0002140998840332, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 4.001378059387207, 'end': 5.001038074493408, 'seconds': 0.9996600151062012, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 23168, 'rtt': 1883250, 'rttvar': 1938402, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 4.001378059387207, 'end': 5.001038074493408, 'seconds': 0.9996600151062012, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 5.001038074493408, 'end': 6.001368999481201, 'seconds': 1.000330924987793, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 24616, 'rtt': 2304747, 'rttvar': 2296796, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 5.001038074493408, 'end': 6.001368999481201, 'seconds': 1.000330924987793, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 6.001368999481201, 'end': 7.001110076904297, 'seconds': 0.9997410774230957, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 26064, 'rtt': 2761389, 'rttvar': 2635881, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 6.001368999481201, 'end': 7.001110076904297, 'seconds': 0.9997410774230957, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 7.001110076904297, 'end': 8.001168966293335, 'seconds': 1.000058889389038, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 28960, 'rtt': 3757011, 'rttvar': 3231805, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 7.001110076904297, 'end': 8.001168966293335, 'seconds': 1.000058889389038, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 8.001168966293335, 'end': 9.00111699104309, 'seconds': 0.9999480247497559, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 30408, 'rtt': 4183116, 'rttvar': 3276063, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 8.001168966293335, 'end': 9.00111699104309, 'seconds': 0.9999480247497559, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 9.00111699104309, 'end': 10.001262903213501, 'seconds': 1.0001459121704102, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 33304, 'rtt': 5001402, 'rttvar': 3263145, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 9.00111699104309, 'end': 10.001262903213501, 'seconds': 1.0001459121704102, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 10.001262903213501, 'end': 11.001368045806885, 'seconds': 1.0001051425933838, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 5426227, 'rttvar': 3297007, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 10.001262903213501, 'end': 11.001368045806885, 'seconds': 1.0001051425933838, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 11.001368045806885, 'end': 12.001264095306396, 'seconds': 0.9998960494995117, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 37648, 'rtt': 6205300, 'rttvar': 3228367, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 11.001368045806885, 'end': 12.001264095306396, 'seconds': 0.9998960494995117, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 12.001264095306396, 'end': 13.001070976257324, 'seconds': 0.9998068809509277, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 39096, 'rtt': 6571874, 'rttvar': 3154422, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 12.001264095306396, 'end': 13.001070976257324, 'seconds': 0.9998068809509277, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 13.001070976257324, 'end': 14.001070976257324, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 40544, 'rtt': 6987807, 'rttvar': 3197684, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 13.001070976257324, 'end': 14.001070976257324, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 14.001070976257324, 'end': 15.001136064529419, 'seconds': 1.0000650882720947, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 7361514, 'rttvar': 3145676, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 14.001070976257324, 'end': 15.001136064529419, 'seconds': 1.0000650882720947, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 15.001136064529419, 'end': 16.0012629032135, 'seconds': 1.000126838684082, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 8186360, 'rttvar': 3201938, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 15.001136064529419, 'end': 16.0012629032135, 'seconds': 1.000126838684082, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 16.0012629032135, 'end': 17.001078128814697, 'seconds': 0.9998152256011963, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 8615637, 'rttvar': 3260008, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 16.0012629032135, 'end': 17.001078128814697, 'seconds': 0.9998152256011963, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 17.001078128814697, 'end': 18.001156091690063, 'seconds': 1.0000779628753662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 9404541, 'rttvar': 3222577, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 17.001078128814697, 'end': 18.001156091690063, 'seconds': 1.0000779628753662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 18.001156091690063, 'end': 19.00106906890869, 'seconds': 0.9999129772186279, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 9763942, 'rttvar': 3135735, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 18.001156091690063, 'end': 19.00106906890869, 'seconds': 0.9999129772186279, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 19.00106906890869, 'end': 20.001096963882446, 'seconds': 1.0000278949737549, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 10168499, 'rttvar': 3160916, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 19.00106906890869, 'end': 20.001096963882446, 'seconds': 1.0000278949737549, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 20.001096963882446, 'end': 21.001368045806885, 'seconds': 1.0002710819244385, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 10921947, 'rttvar': 3106768, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 20.001096963882446, 'end': 21.001368045806885, 'seconds': 1.0002710819244385, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 21.001368045806885, 'end': 22.00137996673584, 'seconds': 1.000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 11552191, 'rttvar': 2861986, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 21.001368045806885, 'end': 22.00137996673584, 'seconds': 1.000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 22.00137996673584, 'end': 23.001096963882446, 'seconds': 0.9997169971466064, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 11848095, 'rttvar': 2738299, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 22.00137996673584, 'end': 23.001096963882446, 'seconds': 0.9997169971466064, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 23.001096963882446, 'end': 24.001074075698853, 'seconds': 0.9999771118164062, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 12494382, 'rttvar': 2659912, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 23.001096963882446, 'end': 24.001074075698853, 'seconds': 0.9999771118164062, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 24.001074075698853, 'end': 25.00116205215454, 'seconds': 1.0000879764556885, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 12847233, 'rttvar': 2700636, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 24.001074075698853, 'end': 25.00116205215454, 'seconds': 1.0000879764556885, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 25.00116205215454, 'end': 26.00087594985962, 'seconds': 0.9997138977050781, 'bytes': 244712, 'bits_per_second': 1958256.261610492, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 13521320, 'rttvar': 2712901, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 25.00116205215454, 'end': 26.00087594985962, 'seconds': 0.9997138977050781, 'bytes': 244712, 'bits_per_second': 1958256.261610492, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 26.00087594985962, 'end': 27.001265048980713, 'seconds': 1.0003890991210938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 13851289, 'rttvar': 2694614, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 26.00087594985962, 'end': 27.001265048980713, 'seconds': 1.0003890991210938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 27.001265048980713, 'end': 28.001365900039673, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 14247378, 'rttvar': 2813140, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 27.001265048980713, 'end': 28.001365900039673, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 28.001365900039673, 'end': 29.001163005828857, 'seconds': 0.9997971057891846, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 14621686, 'rttvar': 2858469, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 28.001365900039673, 'end': 29.001163005828857, 'seconds': 0.9997971057891846, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 29.001163005828857, 'end': 30.001078128814697, 'seconds': 0.9999151229858398, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 56472, 'rtt': 15399235, 'rttvar': 2954232, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 29.001163005828857, 'end': 30.001078128814697, 'seconds': 0.9999151229858398, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 30.001078128814697, 'end': 31.00107502937317, 'seconds': 0.9999969005584717, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 57920, 'rtt': 15801326, 'rttvar': 3019857, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 30.001078128814697, 'end': 31.00107502937317, 'seconds': 0.9999969005584717, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 31.00107502937317, 'end': 32.001363039016724, 'seconds': 1.0002880096435547, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 59368, 'rtt': 16542713, 'rttvar': 3007747, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 31.00107502937317, 'end': 32.001363039016724, 'seconds': 1.0002880096435547, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 32.001363039016724, 'end': 33.0010781288147, 'seconds': 0.9997150897979736, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 59368, 'rtt': 16879890, 'rttvar': 2930163, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 32.001363039016724, 'end': 33.0010781288147, 'seconds': 0.9997150897979736, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 33.0010781288147, 'end': 34.001368045806885, 'seconds': 1.0002899169921875, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 60816, 'rtt': 17259742, 'rttvar': 2957327, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 33.0010781288147, 'end': 34.001368045806885, 'seconds': 1.0002899169921875, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 34.001368045806885, 'end': 35.0011670589447, 'seconds': 0.9997990131378174, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 62264, 'rtt': 17951271, 'rttvar': 2884149, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 34.001368045806885, 'end': 35.0011670589447, 'seconds': 0.9997990131378174, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 35.0011670589447, 'end': 36.001121044158936, 'seconds': 0.9999539852142334, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 62264, 'rtt': 18252506, 'rttvar': 2765580, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 35.0011670589447, 'end': 36.001121044158936, 'seconds': 0.9999539852142334, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 36.001121044158936, 'end': 37.00126600265503, 'seconds': 1.0001449584960938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 63712, 'rtt': 18850197, 'rttvar': 2574269, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 36.001121044158936, 'end': 37.00126600265503, 'seconds': 1.0001449584960938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 37.00126600265503, 'end': 38.00107407569885, 'seconds': 0.9998080730438232, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 65160, 'rtt': 19236718, 'rttvar': 2116768, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 37.00126600265503, 'end': 38.00107407569885, 'seconds': 0.9998080730438232, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 38.00107407569885, 'end': 39.00106692314148, 'seconds': 0.999992847442627, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 66608, 'rtt': 19399209, 'rttvar': 1912557, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 38.00107407569885, 'end': 39.00106692314148, 'seconds': 0.999992847442627, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 39.00106692314148, 'end': 40.001370906829834, 'seconds': 1.0003039836883545, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 68056, 'rtt': 19834473, 'rttvar': 1830272, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 39.00106692314148, 'end': 40.001370906829834, 'seconds': 1.0003039836883545, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 40.001370906829834, 'end': 41.001368045806885, 'seconds': 0.9999971389770508, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 68056, 'rtt': 20009834, 'rttvar': 1723425, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 40.001370906829834, 'end': 41.001368045806885, 'seconds': 0.9999971389770508, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 41.001368045806885, 'end': 42.00137209892273, 'seconds': 1.0000040531158447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 69504, 'rtt': 20248142, 'rttvar': 1769184, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 41.001368045806885, 'end': 42.00137209892273, 'seconds': 1.0000040531158447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 42.00137209892273, 'end': 43.001145124435425, 'seconds': 0.9997730255126953, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 70952, 'rtt': 20644072, 'rttvar': 1681254, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 42.00137209892273, 'end': 43.001145124435425, 'seconds': 0.9997730255126953, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 43.001145124435425, 'end': 44.001068115234375, 'seconds': 0.9999229907989502, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 70952, 'rtt': 20893119, 'rttvar': 1759034, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 43.001145124435425, 'end': 44.001068115234375, 'seconds': 0.9999229907989502, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 44.001068115234375, 'end': 45.0012640953064, 'seconds': 1.0001959800720215, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 72400, 'rtt': 21414223, 'rttvar': 1892693, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 44.001068115234375, 'end': 45.0012640953064, 'seconds': 1.0001959800720215, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 45.0012640953064, 'end': 46.00116801261902, 'seconds': 0.9999039173126221, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 73848, 'rtt': 21712374, 'rttvar': 2015823, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 45.0012640953064, 'end': 46.00116801261902, 'seconds': 0.9999039173126221, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 46.00116801261902, 'end': 47.00137209892273, 'seconds': 1.000204086303711, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 75296, 'rtt': 22196700, 'rttvar': 1973612, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 46.00116801261902, 'end': 47.00137209892273, 'seconds': 1.000204086303711, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 47.00137209892273, 'end': 48.001368045806885, 'seconds': 0.9999959468841553, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 75296, 'rtt': 22487189, 'rttvar': 2061187, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 47.00137209892273, 'end': 48.001368045806885, 'seconds': 0.9999959468841553, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 48.001368045806885, 'end': 49.001073122024536, 'seconds': 0.9997050762176514, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 76744, 'rtt': 22953420, 'rttvar': 1966321, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 48.001368045806885, 'end': 49.001073122024536, 'seconds': 0.9997050762176514, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 49.001073122024536, 'end': 50.00114893913269, 'seconds': 1.0000758171081543, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 78192, 'rtt': 23228843, 'rttvar': 2025586, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 49.001073122024536, 'end': 50.00114893913269, 'seconds': 1.0000758171081543, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 50.00114893913269, 'end': 51.00126504898071, 'seconds': 1.0001161098480225, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 78192, 'rtt': 23467479, 'rttvar': 1996461, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 50.00114893913269, 'end': 51.00126504898071, 'seconds': 1.0001161098480225, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 51.00126504898071, 'end': 52.001370906829834, 'seconds': 1.000105857849121, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 79640, 'rtt': 23937327, 'rttvar': 1962052, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 51.00126504898071, 'end': 52.001370906829834, 'seconds': 1.000105857849121, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 52.001370906829834, 'end': 53.00116300582886, 'seconds': 0.9997920989990234, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 81088, 'rtt': 24159536, 'rttvar': 1915957, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 52.001370906829834, 'end': 53.00116300582886, 'seconds': 0.9997920989990234, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 53.00116300582886, 'end': 54.001163959503174, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 82536, 'rtt': 24569859, 'rttvar': 1812172, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 53.00116300582886, 'end': 54.001163959503174, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 54.001163959503174, 'end': 55.00126504898071, 'seconds': 1.000101089477539, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 82536, 'rtt': 24755704, 'rttvar': 1730819, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 54.001163959503174, 'end': 55.00126504898071, 'seconds': 1.000101089477539, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 55.00126504898071, 'end': 56.00140905380249, 'seconds': 1.0001440048217773, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 83984, 'rtt': 25152553, 'rttvar': 1684882, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 55.00126504898071, 'end': 56.00140905380249, 'seconds': 1.0001440048217773, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 56.00140905380249, 'end': 57.001200914382935, 'seconds': 0.9997918605804443, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 85432, 'rtt': 25355536, 'rttvar': 1669628, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 56.00140905380249, 'end': 57.001200914382935, 'seconds': 0.9997918605804443, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 57.001200914382935, 'end': 58.00121212005615, 'seconds': 1.0000112056732178, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 85432, 'rtt': 25533748, 'rttvar': 1608646, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 57.001200914382935, 'end': 58.00121212005615, 'seconds': 1.0000112056732178, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 58.00121212005615, 'end': 59.00122404098511, 'seconds': 1.000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 86880, 'rtt': 26015453, 'rttvar': 1740394, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 58.00121212005615, 'end': 59.00122404098511, 'seconds': 1.000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}], 'end': {}, 'error': 'error - control socket has closed unexpectedly'}"
12:22:01.077222 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2065↪gprs:sysmo↪iperf3.py:8]
12:22:01.085580 tst iperf3.py:8: Test FAILED (190.0 sec)
12:22:01.092676 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31702): Terminating (SIGTERM)
12:22:01.135835 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31706): Terminating (SIGTERM)
12:22:01.179096 run osmo-hlr_10.42.42.2(pid=31707): Terminating (SIGTERM)
12:22:01.222137 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31708): Terminating (SIGTERM)
12:22:01.267016 run osmo-stp_10.42.42.5(pid=31709): Terminating (SIGTERM)
12:22:01.317336 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Terminating (SIGTERM)
12:22:01.365204 run osmo-ggsn_10.42.42.6(pid=31715): Terminating (SIGTERM)
12:22:01.422623 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31732): Terminating (SIGTERM)
12:22:01.511878 run osmo-sgsn_10.42.42.7(pid=31733): Terminating (SIGTERM)
12:22:01.556692 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31734): Terminating (SIGTERM)
12:22:01.603133 run osmo-msc_10.42.42.8(pid=31735): Terminating (SIGTERM)
12:22:01.653381 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31736): Terminating (SIGTERM)
12:22:01.699069 run osmo-mgw_10.42.42.3(pid=31737): Terminating (SIGTERM)
12:22:01.744549 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31738): Terminating (SIGTERM)
12:22:01.786389 run osmo-mgw_10.42.42.4(pid=31739): Terminating (SIGTERM)
12:22:01.827226 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31740): Terminating (SIGTERM)
12:22:01.868506 run osmo-bsc_10.42.42.9(pid=31741): Terminating (SIGTERM)
12:22:01.909334 run osmo-bts-sysmo(pid=31750): Terminating (SIGTERM)
12:22:01.950156 run osmo-pcu(pid=31762): Terminating (SIGTERM)
12:22:01.991824 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Terminating (SIGTERM)
12:22:02.131391 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGTERM(pid=31812): Launched
12:22:03.214302 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGTERM(pid=31812): Terminated: ok {rc=0}
12:22:03.278086 --- ParallelTerminationStrategy: PID 31702 died...
12:22:03.297188 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31702): Terminated: ok {rc=0}
12:22:03.342567 --- ParallelTerminationStrategy: PID 31706 died...
12:22:03.362145 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31706): Terminated: ok {rc=0}
12:22:03.407376 --- ParallelTerminationStrategy: PID 31707 died...
12:22:03.426331 run osmo-hlr_10.42.42.2(pid=31707): Terminated: ok {rc=0}
12:22:03.470978 --- ParallelTerminationStrategy: PID 31708 died...
12:22:03.489674 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31708): Terminated: ok {rc=0}
12:22:03.534338 --- ParallelTerminationStrategy: PID 31709 died...
12:22:03.553136 run osmo-stp_10.42.42.5(pid=31709): Terminated: ok {rc=0}
12:22:03.597667 --- ParallelTerminationStrategy: PID 31710 died...
12:22:03.616431 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31710): Terminated: ok {rc=0}
12:22:03.661371 --- ParallelTerminationStrategy: PID 31715 died...
12:22:03.680197 run osmo-ggsn_10.42.42.6(pid=31715): Terminated: ok {rc=0}
12:22:03.724883 --- ParallelTerminationStrategy: PID 31732 died...
12:22:03.743590 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31732): Terminated: ok {rc=0}
12:22:03.789095 --- ParallelTerminationStrategy: PID 31733 died...
12:22:03.807983 run osmo-sgsn_10.42.42.7(pid=31733): Terminated: ok {rc=0}
12:22:03.853384 --- ParallelTerminationStrategy: PID 31734 died...
12:22:03.872487 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31734): Terminated: ok {rc=0}
12:22:03.917424 --- ParallelTerminationStrategy: PID 31735 died...
12:22:03.936309 run osmo-msc_10.42.42.8(pid=31735): Terminated: ok {rc=0}
12:22:03.981298 --- ParallelTerminationStrategy: PID 31736 died...
12:22:04.000336 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31736): Terminated: ok {rc=0}
12:22:04.045087 --- ParallelTerminationStrategy: PID 31737 died...
12:22:04.064316 run osmo-mgw_10.42.42.3(pid=31737): Terminated {rc=15}
12:22:04.109312 --- ParallelTerminationStrategy: PID 31738 died...
12:22:04.128265 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31738): Terminated: ok {rc=0}
12:22:04.173108 --- ParallelTerminationStrategy: PID 31739 died...
12:22:04.192160 run osmo-mgw_10.42.42.4(pid=31739): Terminated {rc=15}
12:22:04.236794 --- ParallelTerminationStrategy: PID 31740 died...
12:22:04.255827 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31740): Terminated: ok {rc=0}
12:22:04.300813 --- ParallelTerminationStrategy: PID 31741 died...
12:22:04.319750 run osmo-bsc_10.42.42.9(pid=31741): Terminated: ok {rc=0}
12:22:04.364840 --- ParallelTerminationStrategy: PID 31750 died...
12:22:04.383934 run osmo-bts-sysmo(pid=31750): Terminated: ok {rc=0}
12:22:04.429220 --- ParallelTerminationStrategy: PID 31762 died...
12:22:04.448398 run osmo-pcu(pid=31762): Terminated: ok {rc=0}
12:22:10.317061 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Terminating (SIGINT)
12:22:10.455753 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGINT(pid=31817): Launched
12:22:11.542130 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGINT(pid=31817): Terminated: ok {rc=0}
12:22:17.428293 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788): Terminating (SIGKILL)
12:22:17.566433 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGUSR1(pid=31820): Launched
12:22:18.650292 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31788)-killSignals.SIGUSR1(pid=31820): Terminated: ok {rc=0}
12:22:18.895267 bus /gobi_6: Setting Online False
12:22:21.933050 bus /gobi_6: Setting Powered False
12:22:39.011941 bus /gobi_6: Setting Powered False
----------------------------------------------
trial-2065 gprs:sysmo iperf3m4.py
----------------------------------------------
12:22:40.220219 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.232229 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:22:40.267424 tst iperf3m4.py: using LAC 3954
12:22:40.395960 tst iperf3m4.py: using RAC 129
12:22:40.522266 tst iperf3m4.py: using CellId 3954
12:22:40.648271 tst iperf3m4.py: using BVCI 3955
12:22:40.766077 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.777170 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.788121 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.799056 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.809749 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.820537 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.831454 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.842215 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.853307 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:22:40.969451 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:22:41.091726 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:22:41.208555 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:22:41.329555 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
12:22:41.549405 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:22:41.877309 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31823): Launched
12:22:42.164274 run iperf3-srv_10.42.42.10(pid=31824): Launched
12:22:42.288419 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
12:22:42.508340 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:22:42.836204 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31825): Launched
12:22:43.118800 run iperf3-srv_10.42.42.10(pid=31826): Launched
12:22:43.241198 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
12:22:43.460010 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:22:43.786902 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31827): Launched
12:22:44.070471 run iperf3-srv_10.42.42.10(pid=31828): Launched
12:22:44.192619 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
12:22:44.410316 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:22:44.736888 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31829): Launched
12:22:45.020715 run iperf3-srv_10.42.42.10(pid=31830): Launched
12:22:45.142833 tst iperf3m4.py:8: start network...
12:22:45.360094 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:22:45.865617 run create_hlr_db(pid=31831): Launched
12:22:45.989793 bus /gobi_6: Setting Powered False
12:22:47.053610 run create_hlr_db(pid=31831): Terminated: ok {rc=0}
12:22:47.505603 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31833): Launched
12:22:47.832800 run osmo-hlr_10.42.42.2(pid=31834): Launched
12:22:47.957344 run osmo-stp_10.42.42.5: Starting osmo-stp
12:22:48.513304 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31835): Launched
12:22:48.838417 run osmo-stp_10.42.42.5(pid=31836): Launched
12:22:48.962189 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:22:49.518838 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31837): Launched
12:22:49.644028 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:22:49.919526 run patchelf(pid=31838): Launched
12:22:51.104476 run patchelf(pid=31838): Terminated: ok {rc=0}
12:22:51.240535 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:22:51.519142 run osmo-gsm-tester_setcap_net_admin.sh(pid=31839): Launched
12:22:52.702291 run osmo-gsm-tester_setcap_net_admin.sh(pid=31839): Terminated: ok {rc=0}
12:22:53.038261 run osmo-ggsn_10.42.42.6(pid=31842): Launched
12:22:53.163017 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:22:53.867000 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31860): Launched
12:22:54.193273 run osmo-sgsn_10.42.42.7(pid=31861): Launched
12:22:54.317802 run osmo-msc_10.42.42.8: Starting osmo-msc
12:22:54.925134 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31862): Launched
12:22:55.252940 run osmo-msc_10.42.42.8(pid=31863): Launched
12:22:58.435784 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:22:58.990152 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31864): Launched
12:22:59.316335 run osmo-mgw_10.42.42.3(pid=31865): Launched
12:22:59.440612 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:22:59.985837 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31866): Launched
12:23:00.312972 run osmo-mgw_10.42.42.4(pid=31867): Launched
12:23:00.437052 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:23:01.109304 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31868): Launched
12:23:01.421910 run osmo-bsc_10.42.42.9(pid=31869): Launched
12:23:07.695741 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:23:08.225856 run rm-remote-dir(pid=31870): Launched
12:23:10.408860 run rm-remote-dir(pid=31870): Terminated: ok {rc=0}
12:23:10.744169 run mk-remote-dir(pid=31871): Launched
12:23:11.925999 run mk-remote-dir(pid=31871): Terminated: ok {rc=0}
12:23:12.230000 run scp-inst-to-remote(pid=31872): Launched
12:23:24.442685 run scp-inst-to-remote(pid=31872): Terminated: ok {rc=0}
12:23:24.796748 run mk-remote-dir(pid=31874): Launched
12:23:25.983154 run mk-remote-dir(pid=31874): Terminated: ok {rc=0}
12:23:26.286393 run scp-cfg-to-remote(pid=31875): Launched
12:23:27.476879 run scp-cfg-to-remote(pid=31875): Terminated: ok {rc=0}
12:23:27.825913 run reload-dsp-firmware(pid=31877): Launched
12:23:30.013719 run reload-dsp-firmware(pid=31877): Terminated: ok {rc=0}
12:23:30.354487 run osmo-bts-sysmo(pid=31878): Launched
12:23:34.594220 tst iperf3m4.py:8: Waiting for bts to be ready...
12:23:36.314398 run rm-remote-dir(pid=31879): Launched
12:23:38.501386 run rm-remote-dir(pid=31879): Terminated: ok {rc=0}
12:23:38.844822 run mk-remote-dir(pid=31880): Launched
12:23:41.027835 run mk-remote-dir(pid=31880): Terminated: ok {rc=0}
12:23:41.303640 run scp-inst-to-bts(pid=31881): Launched
12:23:54.489856 run scp-inst-to-bts(pid=31881): Terminated: ok {rc=0}
12:23:54.811154 run mk-remote-run-dir(pid=31883): Launched
12:23:56.982830 run mk-remote-run-dir(pid=31883): Terminated: ok {rc=0}
12:23:57.259283 run scp-cfg-to-sysmobts(pid=31884): Launched
12:23:59.433287 run scp-cfg-to-sysmobts(pid=31884): Terminated: ok {rc=0}
12:23:59.755533 run osmo-pcu(pid=31886): Launched
12:23:59.881877 tst iperf3m4.py: using MSISDN 3963
12:24:00.056185 bus /gobi_6: Setting Powered True
12:24:03.576538 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3963', subscriber_id=1}
12:24:03.855034 bus /gobi_6: Setting Online False
12:24:05.047486 bus /gobi_6: Setting Powered False
12:24:12.130087 bus /gobi_6: Setting Powered True
12:24:13.505745 bus /gobi_6: Setting Online True
12:24:15.933193 tst /gobi_6: Connect to ('901', '70')
12:24:16.216105 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:24:16.341409 tst iperf3m4.py: using MSISDN 3964
12:24:16.565946 bus /gobi_0: Setting Powered True
12:24:20.093221 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3964', subscriber_id=2}
12:24:20.367273 bus /gobi_0: Setting Online False
12:24:21.552591 bus /gobi_0: Setting Powered False
12:24:22.855527 tst /gobi_6: Already registered with network ('901', '70')
12:24:28.332913 bus /gobi_0: Setting Powered True
12:24:29.775305 bus /gobi_0: Setting Online True
12:24:32.288411 tst /gobi_0: Connect to ('901', '70')
12:24:32.577154 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873'
12:24:32.702064 tst iperf3m4.py: using MSISDN 3965
12:24:32.985017 bus /gobi_1: Setting Powered True
12:24:36.539089 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023848', msisdn='3965', subscriber_id=3}
12:24:36.809480 bus /gobi_1: Setting Online False
12:24:37.993315 bus /gobi_1: Setting Powered False
12:24:45.133373 bus /gobi_1: Setting Powered True
12:24:46.529629 bus /gobi_1: Setting Online True
12:24:47.606501 tst /gobi_0: Already registered with network ('901', '70')
12:24:49.210550 tst /gobi_1: Connect to ('901', '70')
12:24:49.472362 tst /gobi_1: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024272103'
12:24:49.592866 tst iperf3m4.py: using MSISDN 3966
12:24:49.756870 bus /gobi_4: Setting Powered True
12:24:53.351798 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023849', msisdn='3966', subscriber_id=4}
12:24:53.623533 bus /gobi_4: Setting Online False
12:24:54.817745 bus /gobi_4: Setting Powered False
12:25:02.120840 bus /gobi_4: Setting Powered True
12:25:03.484834 bus /gobi_4: Setting Online True
12:25:05.903015 tst /gobi_4: Connect to ('901', '70')
12:25:06.175416 tst /gobi_4: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A08E2G 1 [Jul 21 2016 10:00:50]', Serial: '868323025161511'
12:25:06.284667 tst iperf3m4.py:8: waiting for modems to attach...
12:25:06.543173 tst /gobi_1: Already registered with network ('901', '70')
12:25:22.728916 tst /gobi_4: Already registered with network ('901', '70')
12:25:28.159939 tst iperf3m4.py:8: waiting for modems to attach to data services...
12:25:36.617105 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:25:36.727240 tst iperf3m4.py:8: Setting up data plan for "'/gobi_6/context2'"
12:25:37.142337 run move_netns(pid=31894): Launched
12:25:38.325327 run move_netns(pid=31894): Terminated: ok {rc=0}
12:25:38.654539 run ifup(pid=31897): Launched
12:25:39.843871 run ifup(pid=31897): Terminated: ok {rc=0}
12:25:40.194481 run dhcp(pid=31900): Launched
12:25:41.383873 run dhcp(pid=31900): Terminated: ok {rc=0}
12:25:44.423433 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:25:44.534099 tst iperf3m4.py:8: Setting up data plan for "'/gobi_0/context2'"
12:25:44.949995 run move_netns(pid=31918): Launched
12:25:46.128671 run move_netns(pid=31918): Terminated: ok {rc=0}
12:25:46.456213 run ifup(pid=31921): Launched
12:25:47.633096 run ifup(pid=31921): Terminated: ok {rc=0}
12:25:47.960067 run dhcp(pid=31924): Launched
12:25:49.149029 run dhcp(pid=31924): Terminated: ok {rc=0}
12:25:52.171040 tst /gobi_1: context activated {apn='inet46', path='/gobi_1/context10', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.3', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='rf303c5eb721dad', Method='static', Netmask='255.255.255.248'}, Type='internet', Username='ogt'}, user='ogt'}
12:25:52.289220 tst iperf3m4.py:8: Setting up data plan for "'/gobi_1/context10'"
12:25:52.742971 run move_netns(pid=31942): Launched
12:25:53.924537 run move_netns(pid=31942): Terminated: ok {rc=0}
12:25:54.253880 run ifup(pid=31945): Launched
12:25:55.430055 run ifup(pid=31945): Terminated: ok {rc=0}
12:25:55.760803 run dhcp(pid=31948): Launched
12:25:56.939504 run dhcp(pid=31948): Terminated: ok {rc=0}
12:25:59.946870 tst /gobi_4: context activated {apn='inet46', path='/gobi_4/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.4', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.5', Interface='r9d788e55295d1b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:26:00.069024 tst iperf3m4.py:8: Setting up data plan for "'/gobi_4/context2'"
12:26:00.519363 run move_netns(pid=31966): Launched
12:26:01.715087 run move_netns(pid=31966): Terminated: ok {rc=0}
12:26:02.065546 run ifup(pid=31969): Launched
12:26:03.249341 run ifup(pid=31969): Terminated: ok {rc=0}
12:26:03.576601 run dhcp(pid=31972): Launched
12:26:04.765843 run dhcp(pid=31972): Terminated: ok {rc=0}
12:26:04.902290 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'"
12:26:05.126607 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:26:05.514985 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992): Launched
12:26:05.695046 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'"
12:26:05.918148 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s)
12:26:06.307337 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995): Launched
12:26:06.487301 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5005 through "'/gobi_1/context10'"
12:26:06.711207 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5005 (proto=tcp,time=60s)
12:26:07.100429 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998): Launched
12:26:07.279914 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5006 through "'/gobi_4/context2'"
12:26:07.503847 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5006 (proto=tcp,time=60s)
12:26:07.891893 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001): Launched
12:26:08.251119 run iperf3-cli_10.42.42.10(pid=32004): Launched
12:26:08.537060 run iperf3-cli_10.42.42.10(pid=32007): Launched
12:26:08.830908 run iperf3-cli_10.42.42.10(pid=32010): Launched
12:26:09.130549 run iperf3-cli_10.42.42.10(pid=32013): Launched
12:27:17.459947 run iperf3-cli_10.42.42.10(pid=32004): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32004)]
12:27:17.594037 run iperf3-cli_10.42.42.10(pid=32004): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:27:17.731315 run iperf3-cli_10.42.42.10(pid=32004): stderr:
| (launched: 2022-06-18_12:26:08.235233)
12:27:18.935670 run iperf3-cli_10.42.42.10(pid=32007): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32007)]
12:27:19.075835 run iperf3-cli_10.42.42.10(pid=32007): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:27:19.217437 run iperf3-cli_10.42.42.10(pid=32007): stderr:
| (launched: 2022-06-18_12:26:08.521394)
12:27:20.420562 run iperf3-cli_10.42.42.10(pid=32010): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32010)]
12:27:20.561563 run iperf3-cli_10.42.42.10(pid=32010): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:27:20.703387 run iperf3-cli_10.42.42.10(pid=32010): stderr:
| (launched: 2022-06-18_12:26:08.815224)
12:27:21.904743 run iperf3-cli_10.42.42.10(pid=32013): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32013)]
12:27:22.046132 run iperf3-cli_10.42.42.10(pid=32013): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:27:22.187387 run iperf3-cli_10.42.42.10(pid=32013): stderr:
| (launched: 2022-06-18_12:26:09.115161)
12:27:22.329759 run iperf3-srv_10.42.42.10(pid=31824): Terminating (SIGINT)
12:27:22.677183 run iperf3-srv_10.42.42.10(pid=31824): Terminated {rc=1}
12:27:22.804633 tst iperf3m4.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context2'"
Exception while using iperf3 client results: "{'start': {'connected': [{'socket': 5, 'local_host': '176.16.46.1', 'local_port': 41546, 'remote_host': '10.42.42.10', 'remote_port': 5003}], 'version': 'iperf 3.6', 'system_info': 'Linux osmo-gsm-tester-prod 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64', 'timestamp': {'time': 'Sat, 18 Jun 2022 10:26:09 GMT', 'timesecs': 1655547969}, 'connecting_to': {'host': '10.42.42.10', 'port': 5003}, 'cookie': 'un7yf77w6umzhh2lorxcdimcafm2s7tzmjut', 'tcp_mss_default': 1448, 'sock_bufsize': 0, 'sndbuf_actual': 16384, 'rcvbuf_actual': 131072, 'test_start': {'protocol': 'TCP', 'num_streams': 1, 'blksize': 131072, 'omit': 0, 'duration': 60, 'bytes': 0, 'blocks': 0, 'reverse': 0, 'tos': 0}}, 'intervals': [{'streams': [{'socket': 5, 'start': 0, 'end': 1.000636100769043, 'seconds': 1.000636100769043, 'bytes': 101360, 'bits_per_second': 810364.5265014872, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 398965, 'rttvar': 182063, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 0, 'end': 1.000636100769043, 'seconds': 1.000636100769043, 'bytes': 101360, 'bits_per_second': 810364.5265014872, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 1.000636100769043, 'end': 2.0012640953063965, 'seconds': 1.0006279945373535, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 15928, 'rtt': 541750, 'rttvar': 422118, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 1.000636100769043, 'end': 2.0012640953063965, 'seconds': 1.0006279945373535, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 2.0012640953063965, 'end': 3.001112937927246, 'seconds': 0.9998488426208496, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'snd_cwnd': 4344, 'rtt': 541750, 'rttvar': 422118, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 2.0012640953063965, 'end': 3.001112937927246, 'seconds': 0.9998488426208496, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 3.001112937927246, 'end': 4.001070022583008, 'seconds': 0.9999570846557617, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 17376, 'rtt': 829406, 'rttvar': 891901, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 3.001112937927246, 'end': 4.001070022583008, 'seconds': 0.9999570846557617, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 4.001070022583008, 'end': 5.001313924789429, 'seconds': 1.000243902206421, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 18824, 'rtt': 1270825, 'rttvar': 1551764, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 4.001070022583008, 'end': 5.001313924789429, 'seconds': 1.000243902206421, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 5.001313924789429, 'end': 6.001070022583008, 'seconds': 0.9997560977935791, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 18824, 'rtt': 1270825, 'rttvar': 1551764, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 5.001313924789429, 'end': 6.001070022583008, 'seconds': 0.9997560977935791, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 6.001070022583008, 'end': 7.001316070556641, 'seconds': 1.0002460479736328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 20272, 'rtt': 1852083, 'rttvar': 2326339, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 6.001070022583008, 'end': 7.001316070556641, 'seconds': 1.0002460479736328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 7.001316070556641, 'end': 8.001423120498657, 'seconds': 1.0001070499420166, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 21720, 'rtt': 2557924, 'rttvar': 3156435, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 7.001316070556641, 'end': 8.001423120498657, 'seconds': 1.0001070499420166, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 8.001423120498657, 'end': 9.001075029373169, 'seconds': 0.9996519088745117, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 21720, 'rtt': 2557924, 'rttvar': 3156435, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 8.001423120498657, 'end': 9.001075029373169, 'seconds': 0.9996519088745117, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 9.001075029373169, 'end': 10.001075983047485, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 23168, 'rtt': 3360709, 'rttvar': 3972896, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 9.001075029373169, 'end': 10.001075983047485, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 10.001075983047485, 'end': 11.00108003616333, 'seconds': 1.0000040531158447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 24616, 'rtt': 4245520, 'rttvar': 4749295, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 10.001075983047485, 'end': 11.00108003616333, 'seconds': 1.0000040531158447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 11.00108003616333, 'end': 12.001266956329346, 'seconds': 1.0001869201660156, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 24616, 'rtt': 4245520, 'rttvar': 4749295, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 11.00108003616333, 'end': 12.001266956329346, 'seconds': 1.0001869201660156, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 12.001266956329346, 'end': 13.001266956329346, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 26064, 'rtt': 5204881, 'rttvar': 5480693, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 12.001266956329346, 'end': 13.001266956329346, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 13.001266956329346, 'end': 14.001313924789429, 'seconds': 1.000046968460083, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 27512, 'rtt': 6234098, 'rttvar': 6168955, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 13.001266956329346, 'end': 14.001313924789429, 'seconds': 1.000046968460083, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 14.001313924789429, 'end': 15.001074075698853, 'seconds': 0.9997601509094238, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 27512, 'rtt': 6234098, 'rttvar': 6168955, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 14.001313924789429, 'end': 15.001074075698853, 'seconds': 0.9997601509094238, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 15.001074075698853, 'end': 16.001152992248535, 'seconds': 1.0000789165496826, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 28960, 'rtt': 7322081, 'rttvar': 6802682, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 15.001074075698853, 'end': 16.001152992248535, 'seconds': 1.0000789165496826, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 16.001152992248535, 'end': 17.001163005828857, 'seconds': 1.0000100135803223, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 30408, 'rtt': 8362591, 'rttvar': 7183031, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 16.001152992248535, 'end': 17.001163005828857, 'seconds': 1.0000100135803223, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 17.001163005828857, 'end': 18.001305103302002, 'seconds': 1.0001420974731445, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 30408, 'rtt': 8362591, 'rttvar': 7183031, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 17.001163005828857, 'end': 18.001305103302002, 'seconds': 1.0001420974731445, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 18.001305103302002, 'end': 19.001158952713013, 'seconds': 0.9998538494110107, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 9372098, 'rttvar': 7406288, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 18.001305103302002, 'end': 19.001158952713013, 'seconds': 0.9998538494110107, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 19.001158952713013, 'end': 20.001305103302002, 'seconds': 1.0001461505889893, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 9372098, 'rttvar': 7406288, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 19.001158952713013, 'end': 20.001305103302002, 'seconds': 1.0001461505889893, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 20.001305103302002, 'end': 21.00120496749878, 'seconds': 0.9998998641967773, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 9372098, 'rttvar': 7406288, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 20.001305103302002, 'end': 21.00120496749878, 'seconds': 0.9998998641967773, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 21.00120496749878, 'end': 22.00025510787964, 'seconds': 0.9990501403808594, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 33304, 'rtt': 10465336, 'rttvar': 7741190, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 21.00120496749878, 'end': 22.00025510787964, 'seconds': 0.9990501403808594, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 22.00025510787964, 'end': 23.001266956329346, 'seconds': 1.001011848449707, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 11611841, 'rttvar': 8098903, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 22.00025510787964, 'end': 23.001266956329346, 'seconds': 1.001011848449707, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 23.001266956329346, 'end': 24.001163959503174, 'seconds': 0.9998970031738281, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 11611841, 'rttvar': 8098903, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 23.001266956329346, 'end': 24.001163959503174, 'seconds': 0.9998970031738281, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 24.001163959503174, 'end': 25.00116991996765, 'seconds': 1.0000059604644775, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 36200, 'rtt': 12632873, 'rttvar': 8116241, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 24.001163959503174, 'end': 25.00116991996765, 'seconds': 1.0000059604644775, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 25.00116991996765, 'end': 26.001163959503174, 'seconds': 0.9999940395355225, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 36200, 'rtt': 12632873, 'rttvar': 8116241, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 25.00116991996765, 'end': 26.001163959503174, 'seconds': 0.9999940395355225, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 26.001163959503174, 'end': 27.00116491317749, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 37648, 'rtt': 13713833, 'rttvar': 8249101, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 26.001163959503174, 'end': 27.00116491317749, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 27.00116491317749, 'end': 28.00116991996765, 'seconds': 1.0000050067901611, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 39096, 'rtt': 14652152, 'rttvar': 8063464, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 27.00116491317749, 'end': 28.00116991996765, 'seconds': 1.0000050067901611, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 28.00116991996765, 'end': 29.001163959503174, 'seconds': 0.9999940395355225, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 39096, 'rtt': 14652152, 'rttvar': 8063464, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 28.00116991996765, 'end': 29.001163959503174, 'seconds': 0.9999940395355225, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 29.001163959503174, 'end': 30.00115990638733, 'seconds': 0.9999959468841553, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 40544, 'rtt': 15655368, 'rttvar': 8054030, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 29.001163959503174, 'end': 30.00115990638733, 'seconds': 0.9999959468841553, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 30.00115990638733, 'end': 31.00126600265503, 'seconds': 1.0001060962677002, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 16525791, 'rttvar': 7781368, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 30.00115990638733, 'end': 31.00126600265503, 'seconds': 1.0001060962677002, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 31.00126600265503, 'end': 32.0010769367218, 'seconds': 0.9998109340667725, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 16525791, 'rttvar': 7781368, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 31.00126600265503, 'end': 32.0010769367218, 'seconds': 0.9998109340667725, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 32.0010769367218, 'end': 33.00126504898071, 'seconds': 1.0001881122589111, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 17477757, 'rttvar': 7739958, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 32.0010769367218, 'end': 33.00126504898071, 'seconds': 1.0001881122589111, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 33.00126504898071, 'end': 34.00117802619934, 'seconds': 0.9999129772186279, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 18317609, 'rttvar': 7484672, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 33.00126504898071, 'end': 34.00117802619934, 'seconds': 0.9999129772186279, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 34.00117802619934, 'end': 35.00115990638733, 'seconds': 0.9999818801879883, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 18317609, 'rttvar': 7484672, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 34.00117802619934, 'end': 35.00115990638733, 'seconds': 0.9999818801879883, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 35.00115990638733, 'end': 36.00116205215454, 'seconds': 1.000002145767212, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 19239971, 'rttvar': 7458229, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 35.00115990638733, 'end': 36.00116205215454, 'seconds': 1.000002145767212, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 36.00116205215454, 'end': 37.00116205215454, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 20052087, 'rttvar': 7217905, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 36.00116205215454, 'end': 37.00116205215454, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 37.00116205215454, 'end': 38.0007050037384, 'seconds': 0.9995429515838623, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 20052087, 'rttvar': 7217905, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 37.00116205215454, 'end': 38.0007050037384, 'seconds': 0.9995429515838623, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 38.0007050037384, 'end': 39.00116300582886, 'seconds': 1.000458002090454, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 20950263, 'rttvar': 7209779, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 38.0007050037384, 'end': 39.00116300582886, 'seconds': 1.000458002090454, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 39.00116300582886, 'end': 40.00114393234253, 'seconds': 0.9999809265136719, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 21743600, 'rttvar': 6994009, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 39.00116300582886, 'end': 40.00114393234253, 'seconds': 0.9999809265136719, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 40.00114393234253, 'end': 41.00147199630737, 'seconds': 1.0003280639648438, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 21743600, 'rttvar': 6994009, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 40.00114393234253, 'end': 41.00147199630737, 'seconds': 1.0003280639648438, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 41.00147199630737, 'end': 42.00111103057861, 'seconds': 0.9996390342712402, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 22632860, 'rttvar': 7024027, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 41.00147199630737, 'end': 42.00111103057861, 'seconds': 0.9996390342712402, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 42.00111103057861, 'end': 43.00106692314148, 'seconds': 0.9999558925628662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 23408541, 'rttvar': 6819381, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 42.00111103057861, 'end': 43.00106692314148, 'seconds': 0.9999558925628662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 43.00106692314148, 'end': 44.00073504447937, 'seconds': 0.9996681213378906, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 23408541, 'rttvar': 6819381, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 43.00106692314148, 'end': 44.00073504447937, 'seconds': 0.9996681213378906, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 44.00073504447937, 'end': 45.00131893157959, 'seconds': 1.0005838871002197, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 24284560, 'rttvar': 6866574, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 44.00073504447937, 'end': 45.00131893157959, 'seconds': 1.0005838871002197, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 45.00131893157959, 'end': 46.001161098480225, 'seconds': 0.9998421669006348, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 25046534, 'rttvar': 6673880, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 45.00131893157959, 'end': 46.001161098480225, 'seconds': 0.9998421669006348, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 46.001161098480225, 'end': 47.00193691253662, 'seconds': 1.0007758140563965, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 25046534, 'rttvar': 6673880, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 46.001161098480225, 'end': 47.00193691253662, 'seconds': 1.0007758140563965, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 47.00193691253662, 'end': 48.00113606452942, 'seconds': 0.9991991519927979, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 25908235, 'rttvar': 6728811, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 47.00193691253662, 'end': 48.00113606452942, 'seconds': 0.9991991519927979, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 48.00113606452942, 'end': 49.00226902961731, 'seconds': 1.0011329650878906, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 26649604, 'rttvar': 6529346, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 48.00113606452942, 'end': 49.00226902961731, 'seconds': 1.0011329650878906, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 49.00226902961731, 'end': 50.001315116882324, 'seconds': 0.9990460872650146, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 26649604, 'rttvar': 6529346, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 49.00226902961731, 'end': 50.001315116882324, 'seconds': 0.9990460872650146, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 50.001315116882324, 'end': 51.0010769367218, 'seconds': 0.9997618198394775, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 27482800, 'rttvar': 6563402, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 50.001315116882324, 'end': 51.0010769367218, 'seconds': 0.9997618198394775, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 51.0010769367218, 'end': 52.00107192993164, 'seconds': 0.9999949932098389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 28224621, 'rttvar': 6406192, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 51.0010769367218, 'end': 52.00107192993164, 'seconds': 0.9999949932098389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 52.00107192993164, 'end': 53.00121307373047, 'seconds': 1.0001411437988281, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 28224621, 'rttvar': 6406192, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 52.00107192993164, 'end': 53.00121307373047, 'seconds': 1.0001411437988281, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 53.00121307373047, 'end': 54.00079298019409, 'seconds': 0.999579906463623, 'bytes': 256296, 'bits_per_second': 2051229.7083420989, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 29056054, 'rttvar': 6467511, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 53.00121307373047, 'end': 54.00079298019409, 'seconds': 0.999579906463623, 'bytes': 256296, 'bits_per_second': 2051229.7083420989, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 54.00079298019409, 'end': 55.00126099586487, 'seconds': 1.0004680156707764, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 29605850, 'rttvar': 5950226, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 54.00079298019409, 'end': 55.00126099586487, 'seconds': 1.0004680156707764, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 55.00126099586487, 'end': 56.001420974731445, 'seconds': 1.0001599788665771, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 29605850, 'rttvar': 5950226, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 55.00126099586487, 'end': 56.001420974731445, 'seconds': 1.0001599788665771, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 56.001420974731445, 'end': 57.00106692314148, 'seconds': 0.9996459484100342, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 30277261, 'rttvar': 5805492, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 56.001420974731445, 'end': 57.00106692314148, 'seconds': 0.9996459484100342, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 57.00106692314148, 'end': 58.001073122024536, 'seconds': 1.0000061988830566, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 56472, 'rtt': 30859461, 'rttvar': 5518519, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 57.00106692314148, 'end': 58.001073122024536, 'seconds': 1.0000061988830566, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 58.001073122024536, 'end': 59.00110912322998, 'seconds': 1.0000360012054443, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 56472, 'rtt': 30859461, 'rttvar': 5518519, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 58.001073122024536, 'end': 59.00110912322998, 'seconds': 1.0000360012054443, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}], 'end': {}, 'error': 'error - control socket has closed unexpectedly'}"
12:27:23.044996 tst iperf3m4.py:8: ERR: KeyError: 'sum_sent' [trial-2065↪gprs:sysmo↪iperf3m4.py:8]
12:27:23.054404 tst iperf3m4.py:8: Test FAILED (282.9 sec)
12:27:23.061773 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31823): Terminating (SIGTERM)
12:27:23.106246 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31825): Terminating (SIGTERM)
12:27:23.151291 run iperf3-srv_10.42.42.10(pid=31826): Terminating (SIGTERM)
12:27:23.195450 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31827): Terminating (SIGTERM)
12:27:23.239759 run iperf3-srv_10.42.42.10(pid=31828): Terminating (SIGTERM)
12:27:23.283672 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31829): Terminating (SIGTERM)
12:27:23.327792 run iperf3-srv_10.42.42.10(pid=31830): Terminating (SIGTERM)
12:27:23.372098 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31833): Terminating (SIGTERM)
12:27:23.416137 run osmo-hlr_10.42.42.2(pid=31834): Terminating (SIGTERM)
12:27:23.460063 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31835): Terminating (SIGTERM)
12:27:23.505678 run osmo-stp_10.42.42.5(pid=31836): Terminating (SIGTERM)
12:27:23.556938 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31837): Terminating (SIGTERM)
12:27:23.602001 run osmo-ggsn_10.42.42.6(pid=31842): Terminating (SIGTERM)
12:27:23.647977 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31860): Terminating (SIGTERM)
12:27:23.749585 run osmo-sgsn_10.42.42.7(pid=31861): Terminating (SIGTERM)
12:27:23.796842 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31862): Terminating (SIGTERM)
12:27:23.851805 run osmo-msc_10.42.42.8(pid=31863): Terminating (SIGTERM)
12:27:23.906317 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31864): Terminating (SIGTERM)
12:27:23.956331 run osmo-mgw_10.42.42.3(pid=31865): Terminating (SIGTERM)
12:27:24.003345 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31866): Terminating (SIGTERM)
12:27:24.048538 run osmo-mgw_10.42.42.4(pid=31867): Terminating (SIGTERM)
12:27:24.093507 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31868): Terminating (SIGTERM)
12:27:24.139377 run osmo-bsc_10.42.42.9(pid=31869): Terminating (SIGTERM)
12:27:24.182865 run osmo-bts-sysmo(pid=31878): Terminating (SIGTERM)
12:27:24.227411 run osmo-pcu(pid=31886): Terminating (SIGTERM)
12:27:24.271358 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992): Terminating (SIGTERM)
12:27:24.417272 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGTERM(pid=32041): Launched
12:27:25.504210 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGTERM(pid=32041): Terminated: ok {rc=0}
12:27:25.570393 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995): Terminating (SIGTERM)
12:27:25.713180 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGTERM(pid=32047): Launched
12:27:26.798918 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGTERM(pid=32047): Terminated: ok {rc=0}
12:27:26.864931 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998): Terminating (SIGTERM)
12:27:27.011098 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGTERM(pid=32050): Launched
12:27:28.096145 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGTERM(pid=32050): Terminated: ok {rc=0}
12:27:28.162240 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001): Terminating (SIGTERM)
12:27:28.305596 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGTERM(pid=32053): Launched
12:27:29.391737 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGTERM(pid=32053): Terminated: ok {rc=0}
12:27:29.457681 --- ParallelTerminationStrategy: PID 31823 died...
12:27:29.477986 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31823): Terminated: ok {rc=0}
12:27:29.524608 --- ParallelTerminationStrategy: PID 31825 died...
12:27:29.544161 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31825): Terminated: ok {rc=0}
12:27:29.591202 --- ParallelTerminationStrategy: PID 31826 died...
12:27:29.610631 run iperf3-srv_10.42.42.10(pid=31826): Terminated {rc=256}
12:27:29.656897 --- ParallelTerminationStrategy: PID 31827 died...
12:27:29.676406 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31827): Terminated: ok {rc=0}
12:27:29.723099 --- ParallelTerminationStrategy: PID 31828 died...
12:27:29.742792 run iperf3-srv_10.42.42.10(pid=31828): Terminated {rc=256}
12:27:29.789144 --- ParallelTerminationStrategy: PID 31829 died...
12:27:29.808695 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31829): Terminated: ok {rc=0}
12:27:29.855054 --- ParallelTerminationStrategy: PID 31830 died...
12:27:29.874543 run iperf3-srv_10.42.42.10(pid=31830): Terminated {rc=256}
12:27:29.921561 --- ParallelTerminationStrategy: PID 31833 died...
12:27:29.941503 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31833): Terminated: ok {rc=0}
12:27:29.988418 --- ParallelTerminationStrategy: PID 31834 died...
12:27:30.008232 run osmo-hlr_10.42.42.2(pid=31834): Terminated: ok {rc=0}
12:27:30.054602 --- ParallelTerminationStrategy: PID 31835 died...
12:27:30.074473 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31835): Terminated: ok {rc=0}
12:27:30.121043 --- ParallelTerminationStrategy: PID 31836 died...
12:27:30.140986 run osmo-stp_10.42.42.5(pid=31836): Terminated: ok {rc=0}
12:27:30.187299 --- ParallelTerminationStrategy: PID 31837 died...
12:27:30.206870 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31837): Terminated: ok {rc=0}
12:27:30.253030 --- ParallelTerminationStrategy: PID 31842 died...
12:27:30.272617 run osmo-ggsn_10.42.42.6(pid=31842): Terminated: ok {rc=0}
12:27:30.319701 --- ParallelTerminationStrategy: PID 31860 died...
12:27:30.339284 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31860): Terminated: ok {rc=0}
12:27:30.385484 --- ParallelTerminationStrategy: PID 31861 died...
12:27:30.405026 run osmo-sgsn_10.42.42.7(pid=31861): Terminated: ok {rc=0}
12:27:30.451476 --- ParallelTerminationStrategy: PID 31862 died...
12:27:30.471154 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31862): Terminated: ok {rc=0}
12:27:30.517380 --- ParallelTerminationStrategy: PID 31863 died...
12:27:30.536983 run osmo-msc_10.42.42.8(pid=31863): Terminated: ok {rc=0}
12:27:30.583546 --- ParallelTerminationStrategy: PID 31864 died...
12:27:30.603189 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31864): Terminated: ok {rc=0}
12:27:30.649398 --- ParallelTerminationStrategy: PID 31865 died...
12:27:30.668928 run osmo-mgw_10.42.42.3(pid=31865): Terminated {rc=15}
12:27:30.715507 --- ParallelTerminationStrategy: PID 31866 died...
12:27:30.735249 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31866): Terminated: ok {rc=0}
12:27:30.781575 --- ParallelTerminationStrategy: PID 31867 died...
12:27:30.801204 run osmo-mgw_10.42.42.4(pid=31867): Terminated {rc=15}
12:27:30.847481 --- ParallelTerminationStrategy: PID 31868 died...
12:27:30.867120 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31868): Terminated: ok {rc=0}
12:27:30.913657 --- ParallelTerminationStrategy: PID 31869 died...
12:27:30.933241 run osmo-bsc_10.42.42.9(pid=31869): Terminated: ok {rc=0}
12:27:30.979910 --- ParallelTerminationStrategy: PID 31878 died...
12:27:31.000018 run osmo-bts-sysmo(pid=31878): Terminated: ok {rc=0}
12:27:31.047092 --- ParallelTerminationStrategy: PID 31886 died...
12:27:31.067149 run osmo-pcu(pid=31886): Terminated: ok {rc=0}
12:27:36.936306 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992): Terminating (SIGINT)
12:27:37.075340 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGINT(pid=32056): Launched
12:27:38.160041 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGINT(pid=32056): Terminated: ok {rc=0}
12:27:38.223642 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995): Terminating (SIGINT)
12:27:38.359997 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGINT(pid=32059): Launched
12:27:39.447677 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGINT(pid=32059): Terminated: ok {rc=0}
12:27:39.514019 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998): Terminating (SIGINT)
12:27:39.656111 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGINT(pid=32062): Launched
12:27:40.747355 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGINT(pid=32062): Terminated: ok {rc=0}
12:27:40.813304 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001): Terminating (SIGINT)
12:27:40.955343 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGINT(pid=32065): Launched
12:27:42.046895 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGINT(pid=32065): Terminated: ok {rc=0}
12:27:47.935507 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992): Terminating (SIGKILL)
12:27:48.075877 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGUSR1(pid=32068): Launched
12:27:49.160975 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGUSR1(pid=32068): Terminated: ok {rc=0}
12:27:49.224596 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995): Terminating (SIGKILL)
12:27:49.360637 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGUSR1(pid=32071): Launched
12:27:50.441267 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGUSR1(pid=32071): Terminated: ok {rc=0}
12:27:50.504609 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998): Terminating (SIGKILL)
12:27:50.641794 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGUSR1(pid=32074): Launched
12:27:51.728718 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGUSR1(pid=32074): Terminated: ok {rc=0}
12:27:51.792264 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001): Terminating (SIGKILL)
12:27:51.929388 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGUSR1(pid=32077): Launched
12:27:53.019208 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGUSR1(pid=32077): Terminated: ok {rc=0}
12:27:53.270542 bus /gobi_4: Setting Online False
12:27:54.438659 bus /gobi_4: Setting Powered False
12:28:03.939482 bus /gobi_4: Setting Powered False
12:28:05.039208 bus /gobi_4: Setting Powered False
12:28:06.198190 bus /gobi_1: Setting Online False
12:28:07.333287 bus /gobi_1: Setting Powered False
12:28:15.039319 bus /gobi_1: Setting Powered False
12:28:16.137820 bus /gobi_1: Setting Powered False
12:28:17.295075 bus /gobi_0: Setting Online False
12:28:18.435192 bus /gobi_0: Setting Powered False
12:28:26.938063 bus /gobi_0: Setting Powered False
12:28:28.036223 bus /gobi_0: Setting Powered False
12:28:29.193743 bus /gobi_6: Setting Online False
12:28:30.307387 bus /gobi_6: Setting Powered False
12:28:38.072879 bus /gobi_6: Setting Powered False
----------------------------------------------
trial-2065 gprs:sysmo ping.py
----------------------------------------------
12:28:39.233060 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:28:39.244972 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:28:39.279652 tst ping.py: using LAC 3955
12:28:39.408021 tst ping.py: using RAC 130
12:28:39.538392 tst ping.py: using CellId 3955
12:28:39.673502 tst ping.py: using BVCI 3956
12:28:39.791672 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:28:39.802796 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:28:39.813673 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:28:39.824542 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:28:39.835319 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:28:39.846040 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:28:39.857037 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:28:39.867877 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:28:39.986593 tst ping.py:19: start network...
12:28:40.202587 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:28:40.698967 run create_hlr_db(pid=32080): Launched
12:28:40.820733 bus /gobi_6: Setting Powered False
12:28:41.884036 run create_hlr_db(pid=32080): Terminated: ok {rc=0}
12:28:42.315034 run pcap-recorder_any(filters='host 10.42.42.2')(pid=32082): Launched
12:28:42.616075 run osmo-hlr_10.42.42.2(pid=32083): Launched
12:28:42.731832 run osmo-stp_10.42.42.5: Starting osmo-stp
12:28:43.247768 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=32084): Launched
12:28:43.550071 run osmo-stp_10.42.42.5(pid=32085): Launched
12:28:43.665967 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:28:44.180112 run pcap-recorder_any(filters='host 10.42.42.6')(pid=32086): Launched
12:28:44.295802 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:28:44.550049 run patchelf(pid=32087): Launched
12:28:45.719945 run patchelf(pid=32087): Terminated: ok {rc=0}
12:28:45.846860 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:28:46.104035 run osmo-gsm-tester_setcap_net_admin.sh(pid=32088): Launched
12:28:47.277192 run osmo-gsm-tester_setcap_net_admin.sh(pid=32088): Terminated: ok {rc=0}
12:28:47.610047 run osmo-ggsn_10.42.42.6(pid=32091): Launched
12:28:47.733001 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:28:48.424571 run pcap-recorder_any(filters='host 10.42.42.7')(pid=32110): Launched
12:28:48.726077 run osmo-sgsn_10.42.42.7(pid=32111): Launched
12:28:48.842602 run osmo-msc_10.42.42.8: Starting osmo-msc
12:28:49.388072 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=32112): Launched
12:28:49.692187 run osmo-msc_10.42.42.8(pid=32113): Launched
12:28:52.867203 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:28:53.415110 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=32114): Launched
12:28:53.736418 run osmo-mgw_10.42.42.3(pid=32115): Launched
12:28:53.856837 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:28:54.396199 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=32116): Launched
12:28:54.720244 run osmo-mgw_10.42.42.4(pid=32117): Launched
12:28:54.842964 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:28:55.522348 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=32118): Launched
12:28:55.845491 run osmo-bsc_10.42.42.9(pid=32119): Launched
12:29:02.132924 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:29:02.694464 run rm-remote-dir(pid=32120): Launched
12:29:04.887381 run rm-remote-dir(pid=32120): Terminated: ok {rc=0}
12:29:05.240759 run mk-remote-dir(pid=32121): Launched
12:29:06.429562 run mk-remote-dir(pid=32121): Terminated: ok {rc=0}
12:29:06.730944 run scp-inst-to-remote(pid=32122): Launched
12:29:18.936952 run scp-inst-to-remote(pid=32122): Terminated: ok {rc=0}
12:29:19.285414 run mk-remote-dir(pid=32124): Launched
12:29:20.472690 run mk-remote-dir(pid=32124): Terminated: ok {rc=0}
12:29:20.772262 run scp-cfg-to-remote(pid=32125): Launched
12:29:21.958838 run scp-cfg-to-remote(pid=32125): Terminated: ok {rc=0}
12:29:22.303141 run reload-dsp-firmware(pid=32127): Launched
12:29:24.486960 run reload-dsp-firmware(pid=32127): Terminated: ok {rc=0}
12:29:24.825864 run osmo-bts-sysmo(pid=32128): Launched
12:29:29.064525 tst ping.py:31: Waiting for bts to be ready...
12:29:30.777124 run rm-remote-dir(pid=32129): Launched
12:29:32.960077 run rm-remote-dir(pid=32129): Terminated: ok {rc=0}
12:29:33.298544 run mk-remote-dir(pid=32130): Launched
12:29:35.483279 run mk-remote-dir(pid=32130): Terminated: ok {rc=0}
12:29:35.774542 run scp-inst-to-bts(pid=32131): Launched
12:29:48.972242 run scp-inst-to-bts(pid=32131): Terminated: ok {rc=0}
12:29:49.312093 run mk-remote-run-dir(pid=32133): Launched
12:29:51.493718 run mk-remote-run-dir(pid=32133): Terminated: ok {rc=0}
12:29:51.785095 run scp-cfg-to-sysmobts(pid=32134): Launched
12:29:53.969410 run scp-cfg-to-sysmobts(pid=32134): Terminated: ok {rc=0}
12:29:54.307990 run osmo-pcu(pid=32136): Launched
12:29:54.439691 tst ping.py: using MSISDN 3967
12:29:54.614365 bus /gobi_6: Setting Powered True
12:29:58.120722 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3967', subscriber_id=1}
12:29:58.393343 bus /gobi_6: Setting Online False
12:29:59.578933 bus /gobi_6: Setting Powered False
12:30:06.323888 bus /gobi_6: Setting Powered True
12:30:07.700412 bus /gobi_6: Setting Online True
12:30:10.141999 tst /gobi_6: Connect to ('901', '70')
12:30:10.416779 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:30:10.531490 tst ping.py:42: waiting for modems to attach...
12:30:24.876649 tst /gobi_6: Already registered with network ('901', '70')
12:30:29.987281 tst ping.py:46: waiting for modems to attach to data services...
12:30:33.151835 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:30:33.270323 tst ping.py:51: Setting up data plan for "'/gobi_6/context3'"
12:30:33.717932 run move_netns(pid=32143): Launched
12:30:34.898839 run move_netns(pid=32143): Terminated: ok {rc=0}
12:30:35.221908 run ifup(pid=32146): Launched
12:30:36.398101 run ifup(pid=32146): Terminated: ok {rc=0}
12:30:36.720280 run dhcp(pid=32149): Launched
12:30:37.893653 run dhcp(pid=32149): Terminated: ok {rc=0}
12:30:38.020002 tst ping.py:53: Running 10 ping requests for "'/gobi_6/context3'"
12:30:38.413138 run ping(pid=32165): Launched
12:30:48.618638 run ping(pid=32165): Terminated: ok {rc=0}
12:30:50.349474 tst /gobi_6: context deactivated {path='/gobi_6/context3'}
12:30:57.500368 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={Interface='r433306bbc2604b'}, Name='Internet', Password='', Protocol='ipv6', Settings={}, Type='internet', Username='ogt'}, user='ogt'}
12:31:14.253374 tst /gobi_6: context deactivated {path='/gobi_6/context3'}
12:31:14.364789 tst ping.py:70: (launched: 2022-06-18_12:30:38.352862)
32166: waiting for 32167
PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data.
64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=524 ms
64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=465 ms
64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=424 ms
64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=419 ms
64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=420 ms
64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=398 ms
64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=378 ms
64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=374 ms
64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=554 ms
64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=533 ms
--- 10.42.42.6 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 20ms
rtt min/avg/max/mdev = 374.410/448.998/554.059/62.821 ms
child exited with 0
12:31:14.565951 tst ping.py: Test passed (155.3 sec)
12:31:14.631669 run pcap-recorder_any(filters='host 10.42.42.2')(pid=32082): Terminating (SIGTERM)
12:31:14.672951 run osmo-hlr_10.42.42.2(pid=32083): Terminating (SIGTERM)
12:31:14.713943 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=32084): Terminating (SIGTERM)
12:31:14.756924 run osmo-stp_10.42.42.5(pid=32085): Terminating (SIGTERM)
12:31:14.800004 run pcap-recorder_any(filters='host 10.42.42.6')(pid=32086): Terminating (SIGTERM)
12:31:14.844634 run osmo-ggsn_10.42.42.6(pid=32091): Terminating (SIGTERM)
12:31:14.888239 run pcap-recorder_any(filters='host 10.42.42.7')(pid=32110): Terminating (SIGTERM)
12:31:14.956281 run osmo-sgsn_10.42.42.7(pid=32111): Terminating (SIGTERM)
12:31:15.005218 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=32112): Terminating (SIGTERM)
12:31:15.050147 run osmo-msc_10.42.42.8(pid=32113): Terminating (SIGTERM)
12:31:15.099804 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=32114): Terminating (SIGTERM)
12:31:15.143388 run osmo-mgw_10.42.42.3(pid=32115): Terminating (SIGTERM)
12:31:15.185511 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=32116): Terminating (SIGTERM)
12:31:15.227160 run osmo-mgw_10.42.42.4(pid=32117): Terminating (SIGTERM)
12:31:15.268382 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=32118): Terminating (SIGTERM)
12:31:15.310182 run osmo-bsc_10.42.42.9(pid=32119): Terminating (SIGTERM)
12:31:15.351481 run osmo-bts-sysmo(pid=32128): Terminating (SIGTERM)
12:31:15.392888 run osmo-pcu(pid=32136): Terminating (SIGTERM)
12:31:15.435200 --- ParallelTerminationStrategy: PID 32082 died...
12:31:15.454823 run pcap-recorder_any(filters='host 10.42.42.2')(pid=32082): Terminated: ok {rc=0}
12:31:15.501007 --- ParallelTerminationStrategy: PID 32083 died...
12:31:15.519974 run osmo-hlr_10.42.42.2(pid=32083): Terminated: ok {rc=0}
12:31:15.566391 --- ParallelTerminationStrategy: PID 32084 died...
12:31:15.586053 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=32084): Terminated: ok {rc=0}
12:31:15.633350 --- ParallelTerminationStrategy: PID 32085 died...
12:31:15.652861 run osmo-stp_10.42.42.5(pid=32085): Terminated: ok {rc=0}
12:31:15.700094 --- ParallelTerminationStrategy: PID 32086 died...
12:31:15.719631 run pcap-recorder_any(filters='host 10.42.42.6')(pid=32086): Terminated: ok {rc=0}
12:31:15.765716 --- ParallelTerminationStrategy: PID 32091 died...
12:31:15.784713 run osmo-ggsn_10.42.42.6(pid=32091): Terminated: ok {rc=0}
12:31:15.831571 --- ParallelTerminationStrategy: PID 32110 died...
12:31:15.851567 run pcap-recorder_any(filters='host 10.42.42.7')(pid=32110): Terminated: ok {rc=0}
12:31:15.898882 --- ParallelTerminationStrategy: PID 32112 died...
12:31:15.918484 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=32112): Terminated: ok {rc=0}
12:31:15.966062 --- ParallelTerminationStrategy: PID 32113 died...
12:31:15.985011 run osmo-msc_10.42.42.8(pid=32113): Terminated: ok {rc=0}
12:31:16.032468 --- ParallelTerminationStrategy: PID 32114 died...
12:31:16.052266 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=32114): Terminated: ok {rc=0}
12:31:16.100224 --- ParallelTerminationStrategy: PID 32115 died...
12:31:16.120505 run osmo-mgw_10.42.42.3(pid=32115): Terminated {rc=15}
12:31:16.168984 --- ParallelTerminationStrategy: PID 32116 died...
12:31:16.189140 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=32116): Terminated: ok {rc=0}
12:31:16.237730 --- ParallelTerminationStrategy: PID 32117 died...
12:31:16.257907 run osmo-mgw_10.42.42.4(pid=32117): Terminated {rc=15}
12:31:16.305555 --- ParallelTerminationStrategy: PID 32118 died...
12:31:16.325188 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=32118): Terminated: ok {rc=0}
12:31:16.371784 --- ParallelTerminationStrategy: PID 32111 died...
12:31:16.390856 run osmo-sgsn_10.42.42.7(pid=32111): Terminated: ok {rc=0}
12:31:16.437278 --- ParallelTerminationStrategy: PID 32119 died...
12:31:16.456248 run osmo-bsc_10.42.42.9(pid=32119): Terminated: ok {rc=0}
12:31:16.502374 --- ParallelTerminationStrategy: PID 32128 died...
12:31:16.521580 run osmo-bts-sysmo(pid=32128): Terminated: ok {rc=0}
12:31:16.567966 --- ParallelTerminationStrategy: PID 32136 died...
12:31:16.587100 run osmo-pcu(pid=32136): Terminated: ok {rc=0}
12:31:16.818541 bus /gobi_6: Setting Online False
12:31:20.967287 bus /gobi_6: Setting Powered False
12:31:38.937754 bus /gobi_6: Setting Powered False
----------------------------------------------
trial-2065 gprs:sysmo ping_idle_ping.py
----------------------------------------------
12:31:40.081570 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:31:40.093467 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:31:40.129511 tst ping_idle_ping.py: using LAC 3956
12:31:40.250295 tst ping_idle_ping.py: using RAC 131
12:31:40.368810 tst ping_idle_ping.py: using CellId 3956
12:31:40.488707 tst ping_idle_ping.py: using BVCI 3957
12:31:40.598042 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:31:40.608812 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:31:40.619629 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:31:40.630581 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:31:40.641553 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:31:40.652487 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:31:40.663208 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:31:40.673771 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:31:40.792126 tst ping_idle_ping.py:24: start network...
12:31:41.008945 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:31:41.508224 run create_hlr_db(pid=32186): Launched
12:31:41.632269 bus /gobi_6: Setting Powered False
12:31:42.696052 run create_hlr_db(pid=32186): Terminated: ok {rc=0}
12:31:43.127031 run pcap-recorder_any(filters='host 10.42.42.2')(pid=32188): Launched
12:31:43.430136 run osmo-hlr_10.42.42.2(pid=32189): Launched
12:31:43.545226 run osmo-stp_10.42.42.5: Starting osmo-stp
12:31:44.067364 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=32190): Launched
12:31:44.369313 run osmo-stp_10.42.42.5(pid=32191): Launched
12:31:44.484476 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:31:45.004894 run pcap-recorder_any(filters='host 10.42.42.6')(pid=32192): Launched
12:31:45.122522 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:31:45.379522 run patchelf(pid=32193): Launched
12:31:46.550864 run patchelf(pid=32193): Terminated: ok {rc=0}
12:31:46.678060 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:31:46.937807 run osmo-gsm-tester_setcap_net_admin.sh(pid=32194): Launched
12:31:48.112501 run osmo-gsm-tester_setcap_net_admin.sh(pid=32194): Terminated: ok {rc=0}
12:31:48.426797 run osmo-ggsn_10.42.42.6(pid=32197): Launched
12:31:48.548957 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:31:49.230289 run pcap-recorder_any(filters='host 10.42.42.7')(pid=32215): Launched
12:31:49.536283 run osmo-sgsn_10.42.42.7(pid=32216): Launched
12:31:49.654056 run osmo-msc_10.42.42.8: Starting osmo-msc
12:31:50.210509 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=32217): Launched
12:31:50.515084 run osmo-msc_10.42.42.8(pid=32218): Launched
12:31:53.685553 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:31:54.204992 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=32219): Launched
12:31:54.509958 run osmo-mgw_10.42.42.3(pid=32220): Launched
12:31:54.626155 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:31:55.136014 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=32221): Launched
12:31:55.439332 run osmo-mgw_10.42.42.4(pid=32222): Launched
12:31:55.555608 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:31:56.184298 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=32223): Launched
12:31:56.489326 run osmo-bsc_10.42.42.9(pid=32224): Launched
12:32:02.758913 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:32:03.286652 run rm-remote-dir(pid=32225): Launched
12:32:05.467396 run rm-remote-dir(pid=32225): Terminated: ok {rc=0}
12:32:05.800775 run mk-remote-dir(pid=32226): Launched
12:32:06.979846 run mk-remote-dir(pid=32226): Terminated: ok {rc=0}
12:32:07.263257 run scp-inst-to-remote(pid=32227): Launched
12:32:19.455670 run scp-inst-to-remote(pid=32227): Terminated: ok {rc=0}
12:32:19.804263 run mk-remote-dir(pid=32229): Launched
12:32:20.995389 run mk-remote-dir(pid=32229): Terminated: ok {rc=0}
12:32:21.295114 run scp-cfg-to-remote(pid=32230): Launched
12:32:22.481681 run scp-cfg-to-remote(pid=32230): Terminated: ok {rc=0}
12:32:22.826940 run reload-dsp-firmware(pid=32232): Launched
12:32:25.014437 run reload-dsp-firmware(pid=32232): Terminated: ok {rc=0}
12:32:25.352343 run osmo-bts-sysmo(pid=32233): Launched
12:32:29.588968 tst ping_idle_ping.py:36: Waiting for bts to be ready...
12:32:31.305030 run rm-remote-dir(pid=32234): Launched
12:32:33.491872 run rm-remote-dir(pid=32234): Terminated: ok {rc=0}
12:32:33.832858 run mk-remote-dir(pid=32236): Launched
12:32:36.017537 run mk-remote-dir(pid=32236): Terminated: ok {rc=0}
12:32:36.310795 run scp-inst-to-bts(pid=32237): Launched
12:32:49.510285 run scp-inst-to-bts(pid=32237): Terminated: ok {rc=0}
12:32:49.851347 run mk-remote-run-dir(pid=32239): Launched
12:32:52.033439 run mk-remote-run-dir(pid=32239): Terminated: ok {rc=0}
12:32:52.326263 run scp-cfg-to-sysmobts(pid=32240): Launched
12:32:54.509611 run scp-cfg-to-sysmobts(pid=32240): Terminated: ok {rc=0}
12:32:54.852165 run osmo-pcu(pid=32242): Launched
12:32:54.986003 tst ping_idle_ping.py: using MSISDN 3968
12:32:55.160432 bus /gobi_6: Setting Powered True
12:32:58.654381 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3968', subscriber_id=1}
12:32:58.917301 bus /gobi_6: Setting Online False
12:33:00.103462 bus /gobi_6: Setting Powered False
12:33:07.107597 bus /gobi_6: Setting Powered True
12:33:08.493012 bus /gobi_6: Setting Online True
12:33:10.979748 tst /gobi_6: Connect to ('901', '70')
12:33:11.272025 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:33:11.388401 tst ping_idle_ping.py:47: waiting for modems to attach...
12:33:17.141365 tst /gobi_6: Already registered with network ('901', '70')
12:33:28.941913 tst ping_idle_ping.py:51: waiting for modems to attach to data services...
12:33:33.914318 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:33:34.031149 tst ping_idle_ping.py:56: Setting up data plane for "'/gobi_6/context2'"
12:33:34.474378 run move_netns(pid=32249): Launched
12:33:35.670793 run move_netns(pid=32249): Terminated: ok {rc=0}
12:33:36.019705 run ifup(pid=32252): Launched
12:33:37.207316 run ifup(pid=32252): Terminated: ok {rc=0}
12:33:37.555875 run dhcp(pid=32255): Launched
12:33:38.745404 run dhcp(pid=32255): Terminated: ok {rc=0}
12:33:38.882566 tst ping_idle_ping.py:60: [1] Running 10 ping requests for "'/gobi_6/context2'"
12:33:39.307910 run ping1(pid=32271): Launched
12:33:49.512806 run ping1(pid=32271): Terminated: ok {rc=0}
12:33:49.648558 tst ping_idle_ping.py:64: (launched: 2022-06-18_12:33:39.243322)
32272: waiting for 32273
PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data.
64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=448 ms
64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=564 ms
64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=523 ms
64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=505 ms
64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=524 ms
64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=520 ms
64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=502 ms
64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=521 ms
64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=517 ms
64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=477 ms
--- 10.42.42.6 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 18ms
rtt min/avg/max/mdev = 448.093/510.030/563.894/29.294 ms
child exited with 0
12:33:49.869351 tst ping_idle_ping.py:68: Sleeping for 60 seconds
12:34:50.090963 tst ping_idle_ping.py:73: [2] Running 10 ping requests for "'/gobi_6/context2'"
12:34:50.491368 run ping2(pid=32276): Launched
12:35:00.698986 run ping2(pid=32276): Terminated: ok {rc=0}
12:35:00.825825 tst ping_idle_ping.py:77: (launched: 2022-06-18_12:34:50.429834)
32277: waiting for 32278
PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data.
64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=550 ms
64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=469 ms
64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=447 ms
64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=429 ms
64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=447 ms
64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=442 ms
64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=424 ms
64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=420 ms
64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=420 ms
64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=398 ms
--- 10.42.42.6 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 19ms
rtt min/avg/max/mdev = 397.900/444.396/549.869/39.722 ms
child exited with 0
12:35:02.661723 tst /gobi_6: context deactivated {path='/gobi_6/context2'}
12:35:02.772216 tst ping_idle_ping.py: Test passed (202.7 sec)
12:35:02.837056 run pcap-recorder_any(filters='host 10.42.42.2')(pid=32188): Terminating (SIGTERM)
12:35:02.877958 run osmo-hlr_10.42.42.2(pid=32189): Terminating (SIGTERM)
12:35:02.918535 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=32190): Terminating (SIGTERM)
12:35:02.961135 run osmo-stp_10.42.42.5(pid=32191): Terminating (SIGTERM)
12:35:03.011041 run pcap-recorder_any(filters='host 10.42.42.6')(pid=32192): Terminating (SIGTERM)
12:35:03.055033 run osmo-ggsn_10.42.42.6(pid=32197): Terminating (SIGTERM)
12:35:03.102080 run pcap-recorder_any(filters='host 10.42.42.7')(pid=32215): Terminating (SIGTERM)
12:35:03.153919 run osmo-sgsn_10.42.42.7(pid=32216): Terminating (SIGTERM)
12:35:03.199311 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=32217): Terminating (SIGTERM)
12:35:03.242913 run osmo-msc_10.42.42.8(pid=32218): Terminating (SIGTERM)
12:35:03.287767 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=32219): Terminating (SIGTERM)
12:35:03.330138 run osmo-mgw_10.42.42.3(pid=32220): Terminating (SIGTERM)
12:35:03.372212 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=32221): Terminating (SIGTERM)
12:35:03.413358 run osmo-mgw_10.42.42.4(pid=32222): Terminating (SIGTERM)
12:35:03.455098 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=32223): Terminating (SIGTERM)
12:35:03.496630 run osmo-bsc_10.42.42.9(pid=32224): Terminating (SIGTERM)
12:35:03.537643 run osmo-bts-sysmo(pid=32233): Terminating (SIGTERM)
12:35:03.578875 run osmo-pcu(pid=32242): Terminating (SIGTERM)
12:35:03.620085 --- ParallelTerminationStrategy: PID 32188 died...
12:35:03.639520 run pcap-recorder_any(filters='host 10.42.42.2')(pid=32188): Terminated: ok {rc=0}
12:35:03.685354 --- ParallelTerminationStrategy: PID 32189 died...
12:35:03.704265 run osmo-hlr_10.42.42.2(pid=32189): Terminated: ok {rc=0}
12:35:03.754004 --- ParallelTerminationStrategy: PID 32190 died...
12:35:03.773973 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=32190): Terminated: ok {rc=0}
12:35:03.863173 --- ParallelTerminationStrategy: PID 32191 died...
12:35:03.895758 run osmo-stp_10.42.42.5(pid=32191): Terminated: ok {rc=0}
12:35:03.942826 --- ParallelTerminationStrategy: PID 32192 died...
12:35:03.962428 run pcap-recorder_any(filters='host 10.42.42.6')(pid=32192): Terminated: ok {rc=0}
12:35:04.008463 --- ParallelTerminationStrategy: PID 32197 died...
12:35:04.027391 run osmo-ggsn_10.42.42.6(pid=32197): Terminated: ok {rc=0}
12:35:04.073266 --- ParallelTerminationStrategy: PID 32215 died...
12:35:04.092271 run pcap-recorder_any(filters='host 10.42.42.7')(pid=32215): Terminated: ok {rc=0}
12:35:04.138268 --- ParallelTerminationStrategy: PID 32217 died...
12:35:04.157541 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=32217): Terminated: ok {rc=0}
12:35:04.204017 --- ParallelTerminationStrategy: PID 32218 died...
12:35:04.223959 run osmo-msc_10.42.42.8(pid=32218): Terminated: ok {rc=0}
12:35:04.274992 --- ParallelTerminationStrategy: PID 32219 died...
12:35:04.295071 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=32219): Terminated: ok {rc=0}
12:35:04.344601 --- ParallelTerminationStrategy: PID 32220 died...
12:35:04.371334 run osmo-mgw_10.42.42.3(pid=32220): Terminated {rc=15}
12:35:04.426940 --- ParallelTerminationStrategy: PID 32221 died...
12:35:04.448486 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=32221): Terminated: ok {rc=0}
12:35:04.496982 --- ParallelTerminationStrategy: PID 32222 died...
12:35:04.516904 run osmo-mgw_10.42.42.4(pid=32222): Terminated {rc=15}
12:35:04.564099 --- ParallelTerminationStrategy: PID 32216 died...
12:35:04.583610 run osmo-sgsn_10.42.42.7(pid=32216): Terminated: ok {rc=0}
12:35:04.630179 --- ParallelTerminationStrategy: PID 32223 died...
12:35:04.649596 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=32223): Terminated: ok {rc=0}
12:35:04.696110 --- ParallelTerminationStrategy: PID 32224 died...
12:35:04.715279 run osmo-bsc_10.42.42.9(pid=32224): Terminated: ok {rc=0}
12:35:04.761933 --- ParallelTerminationStrategy: PID 32233 died...
12:35:04.781253 run osmo-bts-sysmo(pid=32233): Terminated: ok {rc=0}
12:35:04.828270 --- ParallelTerminationStrategy: PID 32242 died...
12:35:04.847698 run osmo-pcu(pid=32242): Terminated: ok {rc=0}
12:35:05.072172 bus /gobi_6: Setting Online False
12:35:08.117646 bus /gobi_6: Setting Powered False
12:35:24.937919 bus /gobi_6: Setting Powered False
---------------------------------------------------------------------
trial-2065 gprs:sysmo FAIL
---------------------------------------------------------------------
12:35:26.277420 tst trial-2065: Storing JUnit report in /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2065/run.2022-06-18_12-09-32/trial-2065.xml
--------------------------------------------------------------------------------------------
trial-2065 FAIL
--------------------------------------------------------------------------------------------
12:35:26.508740 tst trial-2065: trial-2065: FAIL, 2 suites failed, 1 suites passed
FAIL: nitb_sms:sysmo (fail: 1)
FAIL: mo_mt_sms.py (4.8 sec) Error: osmo-nitb_10.42.42.2(pid=31480): Process ended prematurely: osmo-nitb_10.42.42.2(pid=31480) [trial-2065↪nitb_sms:sysmo↪mo_mt_sms.py:11↪mo_mt_sms.py↪osmo-nitb_10.42.42.2↪osmo-nitb_10.42.42.2(pid=31480)]
FAIL: gprs:sysmo (fail: 3, pass: 2)
FAIL: cs_paging_gprs_active.py (354.3 sec) Error: iperf3-cli_10.42.42.10(pid=31637): Wait timeout {condition=<bound method Process.terminated of iperf3-cli_10.42.42.10(pid=31637)>, timeout=180, timestep=1} [trial-2065↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=31637)]
FAIL: iperf3.py (190.0 sec) KeyError: 'sum_sent'
FAIL: iperf3m4.py (282.9 sec) KeyError: 'sum_sent'
pass: ping.py (155.3 sec)
pass: ping_idle_ping.py (202.7 sec)
PASS: sms:sysmo (pass: 1)
pass: mo_mt_sms.py (113.8 sec)
+ rm -rf trial-2065/inst
+ cd trial-2065
+ command -v journalctl
+ [ ! -z /bin/journalctl ]
+ readlink last_run
+ journalctl -u ofono -o short-precise --since 2022-06-18 12:09:30
+ readlink last_run
+ tar czf /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2065-run.tgz run.2022-06-18_12-09-32
+ tar czf /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2065-bin.tgz checksums.md5 open5gs.build-8.tgz osmo-bsc.build-16254.tgz osmo-bts-oc2g.build-4906.tgz osmo-bts-sysmo.build-7410.tgz osmo-bts.build-6880.tgz osmo-ggsn.build-6084.tgz osmo-hlr.build-7890.tgz osmo-hnbgw.build-1182.tgz osmo-mgw.build-8090.tgz osmo-msc.build-27045.tgz osmo-nitb.build-11826.tgz osmo-pcu-oc2g.build-3500.tgz osmo-pcu-sysmo.build-4781.tgz osmo-pcu.build-4742.tgz osmo-sgsn.build-27507.tgz osmo-stp.build-8419.tgz osmo-trx.build-4103.tgz osmocom-bb.build-4224.tgz srslte.build-19.tgz
+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Finished: FAILURE
![[Jenkins]](/jenkins/static/342c126a/images/svgs/logo.svg){kind=logo}
