Skipping 67 KB..
Full Log12:05:10.277329 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:05:10.394811 tst cs_paging_gprs_active.py:38: Setting up data plan for "'/gobi_6/context2'"
12:05:10.842396 run move_netns(pid=20985): Launched
12:05:12.038364 run move_netns(pid=20985): Terminated: ok {rc=0}
12:05:12.397287 run ifup(pid=20988): Launched
12:05:13.595901 run ifup(pid=20988): Terminated: ok {rc=0}
12:05:13.947936 run dhcp(pid=20991): Launched
12:05:15.136330 run dhcp(pid=20991): Terminated: ok {rc=0}
12:05:18.213760 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:05:18.331343 tst cs_paging_gprs_active.py:38: Setting up data plan for "'/gobi_0/context3'"
12:05:18.775720 run move_netns(pid=21009): Launched
12:05:19.954553 run move_netns(pid=21009): Terminated: ok {rc=0}
12:05:20.282109 run ifup(pid=21012): Launched
12:05:21.457171 run ifup(pid=21012): Terminated: ok {rc=0}
12:05:21.785131 run dhcp(pid=21015): Launched
12:05:22.962508 run dhcp(pid=21015): Terminated: ok {rc=0}
12:05:23.091414 tst cs_paging_gprs_active.py:38: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'"
12:05:23.298500 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:05:23.662877 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031): Launched
12:05:23.845811 tst cs_paging_gprs_active.py:38: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context3'"
12:05:24.068188 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s)
12:05:24.455665 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034): Launched
12:05:24.794813 run iperf3-cli_10.42.42.10(pid=21037): Launched
12:05:25.076693 run iperf3-cli_10.42.42.10(pid=21040): Launched
12:05:25.202338 tst cs_paging_gprs_active.py:13: waiting a few secs to make sure iperf3 test is running
12:05:27.454015 tst cs_paging_gprs_active.py:16: iperf3 running, let's place a call
12:05:32.297556 tst cs_paging_gprs_active.py:22: dial success
12:05:35.103549 tst cs_paging_gprs_active.py:28: answer success, call established and ongoing
12:05:41.707626 tst cs_paging_gprs_active.py:35: hangup success
12:06:34.068477 run iperf3-cli_10.42.42.10(pid=21037): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21037)]
12:06:34.209645 run iperf3-cli_10.42.42.10(pid=21037): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:06:34.351512 run iperf3-cli_10.42.42.10(pid=21037): stderr:
| (launched: 2022-09-08_12:05:24.778295)
12:06:35.548359 run iperf3-cli_10.42.42.10(pid=21040): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21040)]
12:06:35.675657 run iperf3-cli_10.42.42.10(pid=21040): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:06:35.804547 run iperf3-cli_10.42.42.10(pid=21040): stderr:
| (launched: 2022-09-08_12:05:25.061346)
12:06:35.932736 run iperf3-srv_10.42.42.10(pid=20908): Terminating (SIGINT)
12:06:36.262802 run iperf3-srv_10.42.42.10(pid=20908): Terminated {rc=1}
12:06:36.376438 tst cs_paging_gprs_active.py:38: Results for 10.42.42.10:5003 through "'/gobi_6/context2'"
Exception while using iperf3 client results: "{'start': {'connected': [{'socket': 5, 'local_host': '176.16.46.1', 'local_port': 47382, 'remote_host': '10.42.42.10', 'remote_port': 5003}], 'version': 'iperf 3.6', 'system_info': 'Linux osmo-gsm-tester-prod 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64', 'timestamp': {'time': 'Thu, 08 Sep 2022 10:05:26 GMT', 'timesecs': 1662631526}, 'connecting_to': {'host': '10.42.42.10', 'port': 5003}, 'cookie': 'dlztxdpb62bd3uuyr5ktbmr5w4rqyjrhpna6', 'tcp_mss_default': 1448, 'sock_bufsize': 0, 'sndbuf_actual': 16384, 'rcvbuf_actual': 131072, 'test_start': {'protocol': 'TCP', 'num_streams': 1, 'blksize': 131072, 'omit': 0, 'duration': 60, 'bytes': 0, 'blocks': 0, 'reverse': 0, 'tos': 0}}, 'intervals': [{'streams': [{'socket': 5, 'start': 0, 'end': 1.0012640953063965, 'seconds': 1.0012640953063965, 'bytes': 101360, 'bits_per_second': 809856.2644971933, 'retransmits': 4, 'snd_cwnd': 15928, 'rtt': 1969880, 'rttvar': 3111235, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 0, 'end': 1.0012640953063965, 'seconds': 1.0012640953063965, 'bytes': 101360, 'bits_per_second': 809856.2644971933, 'retransmits': 4, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 1.0012640953063965, 'end': 2.001099109649658, 'seconds': 0.9998350143432617, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 18824, 'rtt': 1846438, 'rttvar': 1948827, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 1.0012640953063965, 'end': 2.001099109649658, 'seconds': 0.9998350143432617, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 2.001099109649658, 'end': 3.0013179779052734, 'seconds': 1.0002188682556152, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 21720, 'rtt': 1960599, 'rttvar': 1308694, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 2.001099109649658, 'end': 3.0013179779052734, 'seconds': 1.0002188682556152, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 3.0013179779052734, 'end': 4.001054048538208, 'seconds': 0.9997360706329346, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 26064, 'rtt': 2460540, 'rttvar': 1353507, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 3.0013179779052734, 'end': 4.001054048538208, 'seconds': 0.9997360706329346, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 4.001054048538208, 'end': 5.001269102096558, 'seconds': 1.0002150535583496, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 28960, 'rtt': 2952943, 'rttvar': 1630066, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 4.001054048538208, 'end': 5.001269102096558, 'seconds': 1.0002150535583496, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 5.001269102096558, 'end': 6.001161098480225, 'seconds': 0.999891996383667, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 3358158, 'rttvar': 1633296, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 5.001269102096558, 'end': 6.001161098480225, 'seconds': 0.999891996383667, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 6.001161098480225, 'end': 7.001321077346802, 'seconds': 1.0001599788665771, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 36200, 'rtt': 3976328, 'rttvar': 1641618, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 6.001161098480225, 'end': 7.001321077346802, 'seconds': 1.0001599788665771, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 7.001321077346802, 'end': 8.00126600265503, 'seconds': 0.9999449253082275, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 39096, 'rtt': 4408929, 'rttvar': 1675705, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 7.001321077346802, 'end': 8.00126600265503, 'seconds': 0.9999449253082275, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 8.00126600265503, 'end': 9.001164197921753, 'seconds': 0.9998981952667236, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 4866691, 'rttvar': 1738790, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 8.00126600265503, 'end': 9.001164197921753, 'seconds': 0.9998981952667236, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 9.001164197921753, 'end': 10.001183986663818, 'seconds': 1.0000197887420654, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 5143199, 'rttvar': 1857108, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 9.001164197921753, 'end': 10.001183986663818, 'seconds': 1.0000197887420654, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 10.001183986663818, 'end': 11.00134015083313, 'seconds': 1.0001561641693115, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 5395028, 'rttvar': 1896489, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 10.001183986663818, 'end': 11.00134015083313, 'seconds': 1.0001561641693115, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 11.00134015083313, 'end': 12.000159978866577, 'seconds': 0.9988198280334473, 'bytes': 204168, 'bits_per_second': 1635273.9044196312, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 5968752, 'rttvar': 2075250, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 11.00134015083313, 'end': 12.000159978866577, 'seconds': 0.9988198280334473, 'bytes': 204168, 'bits_per_second': 1635273.9044196312, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 12.000159978866577, 'end': 13.001100063323975, 'seconds': 1.0009400844573975, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 6282451, 'rttvar': 2183837, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 12.000159978866577, 'end': 13.001100063323975, 'seconds': 1.0009400844573975, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 13.001100063323975, 'end': 14.001145124435425, 'seconds': 1.0000450611114502, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 6607148, 'rttvar': 2287271, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 13.001100063323975, 'end': 14.001145124435425, 'seconds': 1.0000450611114502, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 14.001145124435425, 'end': 15.001073122024536, 'seconds': 0.9999279975891113, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 7460864, 'rttvar': 2274154, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 14.001145124435425, 'end': 15.001073122024536, 'seconds': 0.9999279975891113, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 15.001073122024536, 'end': 16.001265048980713, 'seconds': 1.0001919269561768, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 8011395, 'rttvar': 2234269, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 15.001073122024536, 'end': 16.001265048980713, 'seconds': 1.0001919269561768, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 16.001265048980713, 'end': 17.001079082489014, 'seconds': 0.9998140335083008, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 8528767, 'rttvar': 2152136, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 16.001265048980713, 'end': 17.001079082489014, 'seconds': 0.9998140335083008, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 17.001079082489014, 'end': 18.001317977905273, 'seconds': 1.0002388954162598, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 9028223, 'rttvar': 2077681, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 17.001079082489014, 'end': 18.001317977905273, 'seconds': 1.0002388954162598, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 18.001317977905273, 'end': 19.001159191131592, 'seconds': 0.9998412132263184, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 56472, 'rtt': 9773117, 'rttvar': 2023080, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 18.001317977905273, 'end': 19.001159191131592, 'seconds': 0.9998412132263184, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 19.001159191131592, 'end': 20.001160144805908, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 57920, 'rtt': 10232558, 'rttvar': 1948957, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 19.001159191131592, 'end': 20.001160144805908, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 20.001160144805908, 'end': 21.001103162765503, 'seconds': 0.9999430179595947, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 59368, 'rtt': 10689966, 'rttvar': 1904520, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 20.001160144805908, 'end': 21.001103162765503, 'seconds': 0.9999430179595947, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 21.001103162765503, 'end': 22.001068115234375, 'seconds': 0.9999649524688721, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 60816, 'rtt': 11121829, 'rttvar': 1834493, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 21.001103162765503, 'end': 22.001068115234375, 'seconds': 0.9999649524688721, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}], 'end': {}, 'error': 'error - control socket has closed unexpectedly'}"
12:06:36.586250 tst cs_paging_gprs_active.py:38: ERR: KeyError: 'sum_sent' [trial-2067↪gprs:sysmo↪cs_paging_gprs_active.py:38]
12:06:36.594049 tst cs_paging_gprs_active.py:38: Test FAILED (224.6 sec)
12:06:36.600336 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20907): Terminating (SIGTERM)
12:06:36.643028 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20909): Terminating (SIGTERM)
12:06:36.683781 run iperf3-srv_10.42.42.10(pid=20910): Terminating (SIGTERM)
12:06:36.724650 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20913): Terminating (SIGTERM)
12:06:36.765500 run osmo-hlr_10.42.42.2(pid=20914): Terminating (SIGTERM)
12:06:36.807047 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20916): Terminating (SIGTERM)
12:06:36.850537 run osmo-stp_10.42.42.5(pid=20917): Terminating (SIGTERM)
12:06:36.895024 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20918): Terminating (SIGTERM)
12:06:36.948975 run osmo-ggsn_10.42.42.6(pid=20923): Terminating (SIGTERM)
12:06:36.990473 run pcap-recorder_any(filters='host 10.42.42.7')(pid=20950): Terminating (SIGTERM)
12:06:37.039160 run osmo-sgsn_10.42.42.7(pid=20951): Terminating (SIGTERM)
12:06:37.085610 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=20952): Terminating (SIGTERM)
12:06:37.129236 run osmo-msc_10.42.42.8(pid=20953): Terminating (SIGTERM)
12:06:37.174475 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=20954): Terminating (SIGTERM)
12:06:37.218327 run osmo-mgw_10.42.42.3(pid=20955): Terminating (SIGTERM)
12:06:37.273043 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=20956): Terminating (SIGTERM)
12:06:37.327722 run osmo-mgw_10.42.42.4(pid=20957): Terminating (SIGTERM)
12:06:37.368605 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=20958): Terminating (SIGTERM)
12:06:37.409974 run osmo-bsc_10.42.42.9(pid=20959): Terminating (SIGTERM)
12:06:37.450628 run osmo-bts-sysmo(pid=20968): Terminating (SIGTERM)
12:06:37.494187 run osmo-pcu(pid=20976): Terminating (SIGTERM)
12:06:37.538406 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031): Terminating (SIGTERM)
12:06:37.678030 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031)-killSignals.SIGTERM(pid=21060): Launched
12:06:38.763542 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031)-killSignals.SIGTERM(pid=21060): Terminated: ok {rc=0}
12:06:38.834655 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034): Terminating (SIGTERM)
12:06:38.987437 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034)-killSignals.SIGTERM(pid=21065): Launched
12:06:40.096851 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034)-killSignals.SIGTERM(pid=21065): Terminated: ok {rc=0}
12:06:40.160225 --- ParallelTerminationStrategy: PID 20907 died...
12:06:40.179443 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20907): Terminated: ok {rc=0}
12:06:40.225134 --- ParallelTerminationStrategy: PID 20909 died...
12:06:40.244023 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=20909): Terminated: ok {rc=0}
12:06:40.290090 --- ParallelTerminationStrategy: PID 20910 died...
12:06:40.308929 run iperf3-srv_10.42.42.10(pid=20910): Terminated {rc=256}
12:06:40.353537 --- ParallelTerminationStrategy: PID 20913 died...
12:06:40.372409 run pcap-recorder_any(filters='host 10.42.42.2')(pid=20913): Terminated: ok {rc=0}
12:06:40.417221 --- ParallelTerminationStrategy: PID 20914 died...
12:06:40.436091 run osmo-hlr_10.42.42.2(pid=20914): Terminated: ok {rc=0}
12:06:40.480982 --- ParallelTerminationStrategy: PID 20916 died...
12:06:40.499858 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=20916): Terminated: ok {rc=0}
12:06:40.544804 --- ParallelTerminationStrategy: PID 20917 died...
12:06:40.563699 run osmo-stp_10.42.42.5(pid=20917): Terminated: ok {rc=0}
12:06:40.608741 --- ParallelTerminationStrategy: PID 20918 died...
12:06:40.628243 run pcap-recorder_any(filters='host 10.42.42.6')(pid=20918): Terminated: ok {rc=0}
12:06:40.673539 --- ParallelTerminationStrategy: PID 20923 died...
12:06:40.693043 run osmo-ggsn_10.42.42.6(pid=20923): Terminated: ok {rc=0}
12:06:40.738166 --- ParallelTerminationStrategy: PID 20950 died...
12:06:40.757787 run pcap-recorder_any(filters='host 10.42.42.7')(pid=20950): Terminated: ok {rc=0}
12:06:40.802645 --- ParallelTerminationStrategy: PID 20951 died...
12:06:40.821950 run osmo-sgsn_10.42.42.7(pid=20951): Terminated: ok {rc=0}
12:06:40.866917 --- ParallelTerminationStrategy: PID 20952 died...
12:06:40.885754 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=20952): Terminated: ok {rc=0}
12:06:40.930883 --- ParallelTerminationStrategy: PID 20953 died...
12:06:40.949876 run osmo-msc_10.42.42.8(pid=20953): Terminated: ok {rc=0}
12:06:40.995053 --- ParallelTerminationStrategy: PID 20954 died...
12:06:41.014281 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=20954): Terminated: ok {rc=0}
12:06:41.059524 --- ParallelTerminationStrategy: PID 20955 died...
12:06:41.078453 run osmo-mgw_10.42.42.3(pid=20955): Terminated {rc=15}
12:06:41.123430 --- ParallelTerminationStrategy: PID 20956 died...
12:06:41.142159 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=20956): Terminated: ok {rc=0}
12:06:41.187076 --- ParallelTerminationStrategy: PID 20957 died...
12:06:41.205726 run osmo-mgw_10.42.42.4(pid=20957): Terminated {rc=15}
12:06:41.250663 --- ParallelTerminationStrategy: PID 20958 died...
12:06:41.269929 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=20958): Terminated: ok {rc=0}
12:06:41.315131 --- ParallelTerminationStrategy: PID 20959 died...
12:06:41.334062 run osmo-bsc_10.42.42.9(pid=20959): Terminated: ok {rc=0}
12:06:41.379098 --- ParallelTerminationStrategy: PID 20968 died...
12:06:41.397944 run osmo-bts-sysmo(pid=20968): Terminated: ok {rc=0}
12:06:41.442969 --- ParallelTerminationStrategy: PID 20976 died...
12:06:41.461888 run osmo-pcu(pid=20976): Terminated: ok {rc=0}
12:06:47.329637 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031): Terminating (SIGINT)
12:06:47.467776 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031)-killSignals.SIGINT(pid=21068): Launched
12:06:48.550348 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031)-killSignals.SIGINT(pid=21068): Terminated: ok {rc=0}
12:06:48.613484 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034): Terminating (SIGINT)
12:06:48.749691 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034)-killSignals.SIGINT(pid=21071): Launched
12:06:49.833647 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034)-killSignals.SIGINT(pid=21071): Terminated: ok {rc=0}
12:06:55.719737 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031): Terminating (SIGKILL)
12:06:55.857615 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031)-killSignals.SIGUSR1(pid=21074): Launched
12:06:56.945193 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21031)-killSignals.SIGUSR1(pid=21074): Terminated: ok {rc=0}
12:06:57.008102 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034): Terminating (SIGKILL)
12:06:57.144337 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034)-killSignals.SIGUSR1(pid=21077): Launched
12:06:58.234046 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21034)-killSignals.SIGUSR1(pid=21077): Terminated: ok {rc=0}
12:06:58.525884 bus /gobi_0: Setting Online False
12:07:01.310117 bus /gobi_0: Setting Powered False
12:07:15.991656 bus /gobi_0: Setting Powered False
12:07:17.090273 bus /gobi_0: Setting Powered False
12:07:18.249186 bus /gobi_6: Setting Online False
12:07:19.389107 bus /gobi_6: Setting Powered False
12:07:27.939223 bus /gobi_6: Setting Powered False
----------------------------------------------
trial-2067 gprs:sysmo iperf3.py
----------------------------------------------
12:07:29.145052 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.156939 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:07:29.200997 tst iperf3.py: using LAC 3967
12:07:29.338183 tst iperf3.py: using RAC 142
12:07:29.466834 tst iperf3.py: using CellId 3967
12:07:29.594001 tst iperf3.py: using BVCI 3968
12:07:29.711709 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.722968 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.733979 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.745158 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.756333 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.767516 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.778630 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.790439 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:07:29.801793 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:07:29.932960 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003...
12:07:30.154590 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:07:30.484787 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21080): Launched
12:07:30.772077 run iperf3-srv_10.42.42.10(pid=21081): Launched
12:07:30.894571 tst iperf3.py:8: start network...
12:07:31.113859 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:07:31.621700 run create_hlr_db(pid=21082): Launched
12:07:31.747457 bus /gobi_6: Setting Powered False
12:07:32.812492 run create_hlr_db(pid=21082): Terminated: ok {rc=0}
12:07:33.251183 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21084): Launched
12:07:33.561120 run osmo-hlr_10.42.42.2(pid=21085): Launched
12:07:33.681147 run osmo-stp_10.42.42.5: Starting osmo-stp
12:07:34.206610 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21086): Launched
12:07:34.514439 run osmo-stp_10.42.42.5(pid=21087): Launched
12:07:34.631784 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:07:35.157327 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21088): Launched
12:07:35.274313 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:07:35.533105 run patchelf(pid=21089): Launched
12:07:36.707624 run patchelf(pid=21089): Terminated: ok {rc=0}
12:07:36.836724 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:07:37.099343 run osmo-gsm-tester_setcap_net_admin.sh(pid=21090): Launched
12:07:38.279604 run osmo-gsm-tester_setcap_net_admin.sh(pid=21090): Terminated: ok {rc=0}
12:07:38.598441 run osmo-ggsn_10.42.42.6(pid=21093): Launched
12:07:38.714516 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:07:39.415359 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21112): Launched
12:07:39.746048 run osmo-sgsn_10.42.42.7(pid=21113): Launched
12:07:39.873079 run osmo-msc_10.42.42.8: Starting osmo-msc
12:07:40.468569 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21114): Launched
12:07:40.797882 run osmo-msc_10.42.42.8(pid=21115): Launched
12:07:43.981365 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:07:44.542415 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21116): Launched
12:07:44.872524 run osmo-mgw_10.42.42.3(pid=21117): Launched
12:07:44.999110 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:07:45.550956 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21118): Launched
12:07:45.880859 run osmo-mgw_10.42.42.4(pid=21119): Launched
12:07:46.007401 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:07:46.683856 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21120): Launched
12:07:47.014503 run osmo-bsc_10.42.42.9(pid=21121): Launched
12:07:53.303768 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:07:53.871738 run rm-remote-dir(pid=21122): Launched
12:07:56.064558 run rm-remote-dir(pid=21122): Terminated: ok {rc=0}
12:07:56.423578 run mk-remote-dir(pid=21123): Launched
12:07:57.613885 run mk-remote-dir(pid=21123): Terminated: ok {rc=0}
12:07:57.921083 run scp-inst-to-remote(pid=21124): Launched
12:08:10.128460 run scp-inst-to-remote(pid=21124): Terminated: ok {rc=0}
12:08:10.482547 run mk-remote-dir(pid=21126): Launched
12:08:11.670890 run mk-remote-dir(pid=21126): Terminated: ok {rc=0}
12:08:11.975778 run scp-cfg-to-remote(pid=21127): Launched
12:08:13.163088 run scp-cfg-to-remote(pid=21127): Terminated: ok {rc=0}
12:08:13.514168 run reload-dsp-firmware(pid=21129): Launched
12:08:15.699208 run reload-dsp-firmware(pid=21129): Terminated: ok {rc=0}
12:08:16.042370 run osmo-bts-sysmo(pid=21130): Launched
12:08:20.288868 tst iperf3.py:8: Waiting for bts to be ready...
12:08:21.965967 run rm-remote-dir(pid=21131): Launched
12:08:24.142230 run rm-remote-dir(pid=21131): Terminated: ok {rc=0}
12:08:24.486529 run mk-remote-dir(pid=21132): Launched
12:08:26.673076 run mk-remote-dir(pid=21132): Terminated: ok {rc=0}
12:08:26.968629 run scp-inst-to-bts(pid=21133): Launched
12:08:40.167166 run scp-inst-to-bts(pid=21133): Terminated: ok {rc=0}
12:08:40.512068 run mk-remote-run-dir(pid=21135): Launched
12:08:42.696171 run mk-remote-run-dir(pid=21135): Terminated: ok {rc=0}
12:08:42.992856 run scp-cfg-to-sysmobts(pid=21136): Launched
12:08:45.176615 run scp-cfg-to-sysmobts(pid=21136): Terminated: ok {rc=0}
12:08:45.521120 run osmo-pcu(pid=21138): Launched
12:08:45.655416 tst iperf3.py: using MSISDN 3984
12:08:45.830390 bus /gobi_6: Setting Powered True
12:08:49.342543 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3984', subscriber_id=1}
12:08:49.617898 bus /gobi_6: Setting Online False
12:08:50.812709 bus /gobi_6: Setting Powered False
12:08:58.111847 bus /gobi_6: Setting Powered True
12:08:59.497357 bus /gobi_6: Setting Online True
12:09:01.986314 tst /gobi_6: Connect to ('901', '70')
12:09:02.252695 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:09:02.361968 tst iperf3.py:8: waiting for modems to attach...
12:09:09.461099 tst /gobi_6: Already registered with network ('901', '70')
12:09:19.823079 tst iperf3.py:8: waiting for modems to attach to data services...
12:09:29.784299 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:09:29.901857 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context3'"
12:09:30.346272 run move_netns(pid=21141): Launched
12:09:31.530182 run move_netns(pid=21141): Terminated: ok {rc=0}
12:09:31.865499 run ifup(pid=21144): Launched
12:09:33.041679 run ifup(pid=21144): Terminated: ok {rc=0}
12:09:33.372032 run dhcp(pid=21147): Launched
12:09:34.548463 run dhcp(pid=21147): Terminated: ok {rc=0}
12:09:34.677155 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'"
12:09:34.887493 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:09:35.253249 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163): Launched
12:09:35.597025 run iperf3-cli_10.42.42.10(pid=21166): Launched
12:10:49.924311 run iperf3-cli_10.42.42.10(pid=21166): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21166)]
12:10:50.055442 run iperf3-cli_10.42.42.10(pid=21166): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:10:50.185818 run iperf3-cli_10.42.42.10(pid=21166): stderr:
| (launched: 2022-09-08_12:09:35.580850)
12:10:50.317839 run iperf3-srv_10.42.42.10(pid=21081): Terminating (SIGINT)
12:10:50.652455 run iperf3-srv_10.42.42.10(pid=21081): Terminated {rc=1}
12:10:50.770250 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context3'"
Exception while using iperf3 client results: "{'start': {'connected': [{'socket': 5, 'local_host': '176.16.46.1', 'local_port': 47420, 'remote_host': '10.42.42.10', 'remote_port': 5003}], 'version': 'iperf 3.6', 'system_info': 'Linux osmo-gsm-tester-prod 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64', 'timestamp': {'time': 'Thu, 08 Sep 2022 10:09:42 GMT', 'timesecs': 1662631782}, 'connecting_to': {'host': '10.42.42.10', 'port': 5003}, 'cookie': '6utatofvupnwo76ieefiao5ouwjplf72isn2', 'tcp_mss_default': 1448, 'sock_bufsize': 0, 'sndbuf_actual': 16384, 'rcvbuf_actual': 131072, 'test_start': {'protocol': 'TCP', 'num_streams': 1, 'blksize': 131072, 'omit': 0, 'duration': 60, 'bytes': 0, 'blocks': 0, 'reverse': 0, 'tos': 0}}, 'intervals': [{'streams': [{'socket': 5, 'start': 0, 'end': 1.0006608963012695, 'seconds': 1.0006608963012695, 'bytes': 101360, 'bits_per_second': 810344.4463526512, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 409720, 'rttvar': 175280, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 0, 'end': 1.0006608963012695, 'seconds': 1.0006608963012695, 'bytes': 101360, 'bits_per_second': 810344.4463526512, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 1.0006608963012695, 'end': 2.0013020038604736, 'seconds': 1.000641107559204, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 15928, 'rtt': 501092, 'rttvar': 314205, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 1.0006608963012695, 'end': 2.0013020038604736, 'seconds': 1.000641107559204, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 2.0013020038604736, 'end': 3.0012640953063965, 'seconds': 0.9999620914459229, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'snd_cwnd': 18824, 'rtt': 888183, 'rttvar': 870927, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 2.0013020038604736, 'end': 3.0012640953063965, 'seconds': 0.9999620914459229, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 3.0012640953063965, 'end': 4.001322031021118, 'seconds': 1.0000579357147217, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 20272, 'rtt': 1174652, 'rttvar': 1226134, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 3.0012640953063965, 'end': 4.001322031021118, 'seconds': 1.0000579357147217, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 4.001322031021118, 'end': 5.001301050186157, 'seconds': 0.9999790191650391, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 21720, 'rtt': 1512946, 'rttvar': 1596188, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 4.001322031021118, 'end': 5.001301050186157, 'seconds': 0.9999790191650391, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 5.001301050186157, 'end': 6.001262903213501, 'seconds': 0.9999618530273438, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 24616, 'rtt': 2326721, 'rttvar': 2332369, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 5.001301050186157, 'end': 6.001262903213501, 'seconds': 0.9999618530273438, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 6.001262903213501, 'end': 7.001219987869263, 'seconds': 0.9999570846557617, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 26064, 'rtt': 2783463, 'rttvar': 2662759, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 6.001262903213501, 'end': 7.001219987869263, 'seconds': 0.9999570846557617, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 7.001219987869263, 'end': 8.00131893157959, 'seconds': 1.0000989437103271, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 28960, 'rtt': 3783796, 'rttvar': 3255077, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 7.001219987869263, 'end': 8.00131893157959, 'seconds': 1.0000989437103271, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 8.00131893157959, 'end': 9.001285076141357, 'seconds': 0.9999661445617676, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 30408, 'rtt': 4216495, 'rttvar': 3306706, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 8.00131893157959, 'end': 9.001285076141357, 'seconds': 0.9999661445617676, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 9.001285076141357, 'end': 10.001281023025513, 'seconds': 0.9999959468841553, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 4646970, 'rttvar': 3340978, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 9.001285076141357, 'end': 10.001281023025513, 'seconds': 0.9999959468841553, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 10.001281023025513, 'end': 11.001210927963257, 'seconds': 0.9999299049377441, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 33304, 'rtt': 5116004, 'rttvar': 3443802, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 10.001281023025513, 'end': 11.001210927963257, 'seconds': 0.9999299049377441, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 11.001210927963257, 'end': 12.001169919967651, 'seconds': 0.9999589920043945, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 5616428, 'rttvar': 3583700, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 11.001210927963257, 'end': 12.001169919967651, 'seconds': 0.9999589920043945, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 12.001169919967651, 'end': 13.001281976699829, 'seconds': 1.0001120567321777, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 37648, 'rtt': 6523208, 'rttvar': 3611564, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 12.001169919967651, 'end': 13.001281976699829, 'seconds': 1.0001120567321777, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 13.001281976699829, 'end': 14.001282930374146, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 39096, 'rtt': 6937662, 'rttvar': 3537581, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 13.001281976699829, 'end': 14.001282930374146, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 14.001282930374146, 'end': 15.001264095306396, 'seconds': 0.999981164932251, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 7772336, 'rttvar': 3436666, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 14.001282930374146, 'end': 15.001264095306396, 'seconds': 0.999981164932251, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 15.001264095306396, 'end': 16.001312017440796, 'seconds': 1.0000479221343994, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 8203166, 'rttvar': 3439159, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 15.001264095306396, 'end': 16.001312017440796, 'seconds': 1.0000479221343994, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 16.001312017440796, 'end': 17.0012629032135, 'seconds': 0.9999508857727051, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 9004020, 'rttvar': 3345450, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 16.001312017440796, 'end': 17.0012629032135, 'seconds': 0.9999508857727051, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 17.0012629032135, 'end': 18.0012629032135, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 9365691, 'rttvar': 3232430, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 17.0012629032135, 'end': 18.0012629032135, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 18.0012629032135, 'end': 19.001161098480225, 'seconds': 0.9998981952667236, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 9769849, 'rttvar': 3232638, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 18.0012629032135, 'end': 19.001161098480225, 'seconds': 0.9998981952667236, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 19.001161098480225, 'end': 20.001161098480225, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 10529202, 'rttvar': 3157956, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 19.001161098480225, 'end': 20.001161098480225, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 20.001161098480225, 'end': 21.001210927963257, 'seconds': 1.0000498294830322, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 10885377, 'rttvar': 3080817, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 20.001161098480225, 'end': 21.001210927963257, 'seconds': 1.0000498294830322, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 21.001210927963257, 'end': 22.001219987869263, 'seconds': 1.0000090599060059, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 11640844, 'rttvar': 3043094, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 21.001210927963257, 'end': 22.001219987869263, 'seconds': 1.0000090599060059, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 22.001219987869263, 'end': 23.0013210773468, 'seconds': 1.000101089477539, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 12047872, 'rttvar': 3096377, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 22.001219987869263, 'end': 23.0013210773468, 'seconds': 1.000101089477539, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 23.0013210773468, 'end': 24.001322984695435, 'seconds': 1.0000019073486328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 12428836, 'rttvar': 3084210, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 23.0013210773468, 'end': 24.001322984695435, 'seconds': 1.0000019073486328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 24.001322984695435, 'end': 25.001312017440796, 'seconds': 0.9999890327453613, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 12864918, 'rttvar': 3185321, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 24.001322984695435, 'end': 25.001312017440796, 'seconds': 0.9999890327453613, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 25.001312017440796, 'end': 26.00046396255493, 'seconds': 0.9991519451141357, 'bytes': 256296, 'bits_per_second': 2052108.3004705366, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 13655891, 'rttvar': 3185545, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 25.001312017440796, 'end': 26.00046396255493, 'seconds': 0.9991519451141357, 'bytes': 256296, 'bits_per_second': 2052108.3004705366, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 26.00046396255493, 'end': 27.001264095306396, 'seconds': 1.0008001327514648, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 13933357, 'rttvar': 2944091, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 26.00046396255493, 'end': 27.001264095306396, 'seconds': 1.0008001327514648, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 27.001264095306396, 'end': 28.001178979873657, 'seconds': 0.9999148845672607, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 56472, 'rtt': 14560238, 'rttvar': 2743411, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 27.001264095306396, 'end': 28.001178979873657, 'seconds': 0.9999148845672607, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 28.001178979873657, 'end': 29.001301050186157, 'seconds': 1.0001220703125, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 56472, 'rtt': 14904689, 'rttvar': 2746462, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 28.001178979873657, 'end': 29.001301050186157, 'seconds': 1.0001220703125, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 29.001301050186157, 'end': 30.00116205215454, 'seconds': 0.9998610019683838, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 57920, 'rtt': 15211148, 'rttvar': 2672764, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 29.001301050186157, 'end': 30.00116205215454, 'seconds': 0.9998610019683838, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 30.00116205215454, 'end': 31.00128197669983, 'seconds': 1.000119924545288, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 59368, 'rtt': 15866390, 'rttvar': 2638554, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 30.00116205215454, 'end': 31.00128197669983, 'seconds': 1.000119924545288, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 31.00128197669983, 'end': 32.00128102302551, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 59368, 'rtt': 16220574, 'rttvar': 2687283, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 31.00128197669983, 'end': 32.00128102302551, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 32.00128102302551, 'end': 33.0012629032135, 'seconds': 0.9999818801879883, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 60816, 'rtt': 16890940, 'rttvar': 2696187, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 32.00128102302551, 'end': 33.0012629032135, 'seconds': 0.9999818801879883, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 33.0012629032135, 'end': 34.001319885253906, 'seconds': 1.0000569820404053, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 62264, 'rtt': 17464856, 'rttvar': 2532612, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 33.0012629032135, 'end': 34.001319885253906, 'seconds': 1.0000569820404053, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 34.001319885253906, 'end': 35.001161098480225, 'seconds': 0.9998412132263184, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 63712, 'rtt': 17734059, 'rttvar': 2437864, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 34.001319885253906, 'end': 35.001161098480225, 'seconds': 0.9998412132263184, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 35.001161098480225, 'end': 36.001261949539185, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 65160, 'rtt': 18334671, 'rttvar': 2412377, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 35.001161098480225, 'end': 36.001261949539185, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 36.001261949539185, 'end': 37.0013210773468, 'seconds': 1.0000591278076172, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 65160, 'rtt': 18667629, 'rttvar': 2475197, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 36.001261949539185, 'end': 37.0013210773468, 'seconds': 1.0000591278076172, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 37.0013210773468, 'end': 38.001161098480225, 'seconds': 0.9998400211334229, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 66608, 'rtt': 19283110, 'rttvar': 2482498, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 37.0013210773468, 'end': 38.001161098480225, 'seconds': 0.9998400211334229, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 38.001161098480225, 'end': 39.0012149810791, 'seconds': 1.000053882598877, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 68056, 'rtt': 19577101, 'rttvar': 2449854, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 38.001161098480225, 'end': 39.0012149810791, 'seconds': 1.000053882598877, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 39.0012149810791, 'end': 40.001312017440796, 'seconds': 1.0000970363616943, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 68056, 'rtt': 19839640, 'rttvar': 2362470, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 39.0012149810791, 'end': 40.001312017440796, 'seconds': 1.0000970363616943, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 40.001312017440796, 'end': 41.001319885253906, 'seconds': 1.0000078678131104, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 69504, 'rtt': 20357868, 'rttvar': 2250542, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 40.001312017440796, 'end': 41.001319885253906, 'seconds': 1.0000078678131104, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 41.001319885253906, 'end': 42.001322984695435, 'seconds': 1.0000030994415283, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 70952, 'rtt': 20608083, 'rttvar': 2188336, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 41.001319885253906, 'end': 42.001322984695435, 'seconds': 1.0000030994415283, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 42.001322984695435, 'end': 43.00132203102112, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 72400, 'rtt': 21187466, 'rttvar': 2235194, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 42.001322984695435, 'end': 43.00132203102112, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 43.00132203102112, 'end': 44.0011510848999, 'seconds': 0.9998290538787842, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 72400, 'rtt': 21418832, 'rttvar': 2139127, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 43.00132203102112, 'end': 44.0011510848999, 'seconds': 0.9998290538787842, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 44.0011510848999, 'end': 45.00106501579285, 'seconds': 0.9999139308929443, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 73848, 'rtt': 21957501, 'rttvar': 2135588, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 44.0011510848999, 'end': 45.00106501579285, 'seconds': 0.9999139308929443, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 45.00106501579285, 'end': 46.0012640953064, 'seconds': 1.0001990795135498, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 75296, 'rtt': 22172680, 'rttvar': 2032049, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 45.00106501579285, 'end': 46.0012640953064, 'seconds': 1.0001990795135498, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 46.0012640953064, 'end': 47.00126099586487, 'seconds': 0.9999969005584717, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 75296, 'rtt': 22448688, 'rttvar': 2076054, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 46.0012640953064, 'end': 47.00126099586487, 'seconds': 0.9999969005584717, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 47.00126099586487, 'end': 48.00113797187805, 'seconds': 0.9998769760131836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 76744, 'rtt': 22862237, 'rttvar': 1877800, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 47.00126099586487, 'end': 48.00113797187805, 'seconds': 0.9998769760131836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 48.00113797187805, 'end': 49.00115394592285, 'seconds': 1.0000159740447998, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 78192, 'rtt': 23106857, 'rttvar': 1897590, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 48.00113797187805, 'end': 49.00115394592285, 'seconds': 1.0000159740447998, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 49.00115394592285, 'end': 50.001261949539185, 'seconds': 1.000108003616333, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 79640, 'rtt': 23480229, 'rttvar': 1714587, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 49.00115394592285, 'end': 50.001261949539185, 'seconds': 1.000108003616333, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 50.001261949539185, 'end': 51.00108003616333, 'seconds': 0.9998180866241455, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 79640, 'rtt': 23720331, 'rttvar': 1766145, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 50.001261949539185, 'end': 51.00108003616333, 'seconds': 0.9998180866241455, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 51.00108003616333, 'end': 52.001302003860474, 'seconds': 1.0002219676971436, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 81088, 'rtt': 24122848, 'rttvar': 1691161, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 51.00108003616333, 'end': 52.001302003860474, 'seconds': 1.0002219676971436, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 52.001302003860474, 'end': 53.001070976257324, 'seconds': 0.9997689723968506, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 82536, 'rtt': 24374904, 'rttvar': 1772482, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 52.001302003860474, 'end': 53.001070976257324, 'seconds': 0.9997689723968506, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 53.001070976257324, 'end': 54.00107288360596, 'seconds': 1.0000019073486328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 82536, 'rtt': 24590238, 'rttvar': 1760029, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 53.001070976257324, 'end': 54.00107288360596, 'seconds': 1.0000019073486328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 54.00107288360596, 'end': 55.00128197669983, 'seconds': 1.000209093093872, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 83984, 'rtt': 25023740, 'rttvar': 1765415, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 54.00107288360596, 'end': 55.00128197669983, 'seconds': 1.000209093093872, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 55.00128197669983, 'end': 56.0013210773468, 'seconds': 1.0000391006469727, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 85432, 'rtt': 25243335, 'rttvar': 1763251, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 55.00128197669983, 'end': 56.0013210773468, 'seconds': 1.0000391006469727, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 56.0013210773468, 'end': 57.00132489204407, 'seconds': 1.0000038146972656, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 86880, 'rtt': 25683687, 'rttvar': 1777874, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 56.0013210773468, 'end': 57.00132489204407, 'seconds': 1.0000038146972656, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 57.00132489204407, 'end': 58.001261949539185, 'seconds': 0.9999370574951172, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 86880, 'rtt': 25918239, 'rttvar': 1802509, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 57.00132489204407, 'end': 58.001261949539185, 'seconds': 0.9999370574951172, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 58.001261949539185, 'end': 59.001261949539185, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 88328, 'rtt': 26120669, 'rttvar': 1756742, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 58.001261949539185, 'end': 59.001261949539185, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}], 'end': {}, 'error': 'error - control socket has closed unexpectedly'}"
12:10:50.993226 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2067↪gprs:sysmo↪iperf3.py:8]
12:10:51.001208 tst iperf3.py:8: Test FAILED (201.9 sec)
12:10:51.007949 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21080): Terminating (SIGTERM)
12:10:51.050050 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21084): Terminating (SIGTERM)
12:10:51.092303 run osmo-hlr_10.42.42.2(pid=21085): Terminating (SIGTERM)
12:10:51.133849 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21086): Terminating (SIGTERM)
12:10:51.176774 run osmo-stp_10.42.42.5(pid=21087): Terminating (SIGTERM)
12:10:51.220406 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21088): Terminating (SIGTERM)
12:10:51.263463 run osmo-ggsn_10.42.42.6(pid=21093): Terminating (SIGTERM)
12:10:51.306208 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21112): Terminating (SIGTERM)
12:10:51.357186 run osmo-sgsn_10.42.42.7(pid=21113): Terminating (SIGTERM)
12:10:51.409497 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21114): Terminating (SIGTERM)
12:10:51.453713 run osmo-msc_10.42.42.8(pid=21115): Terminating (SIGTERM)
12:10:51.510645 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21116): Terminating (SIGTERM)
12:10:51.555035 run osmo-mgw_10.42.42.3(pid=21117): Terminating (SIGTERM)
12:10:51.599617 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21118): Terminating (SIGTERM)
12:10:51.642052 run osmo-mgw_10.42.42.4(pid=21119): Terminating (SIGTERM)
12:10:51.682982 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21120): Terminating (SIGTERM)
12:10:51.724086 run osmo-bsc_10.42.42.9(pid=21121): Terminating (SIGTERM)
12:10:51.765579 run osmo-bts-sysmo(pid=21130): Terminating (SIGTERM)
12:10:51.806125 run osmo-pcu(pid=21138): Terminating (SIGTERM)
12:10:51.847069 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163): Terminating (SIGTERM)
12:10:51.985626 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGTERM(pid=21189): Launched
12:10:53.066557 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGTERM(pid=21189): Terminated: ok {rc=0}
12:10:53.130249 --- ParallelTerminationStrategy: PID 21080 died...
12:10:53.149639 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21080): Terminated: ok {rc=0}
12:10:53.194511 --- ParallelTerminationStrategy: PID 21084 died...
12:10:53.213734 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21084): Terminated: ok {rc=0}
12:10:53.258364 --- ParallelTerminationStrategy: PID 21085 died...
12:10:53.277549 run osmo-hlr_10.42.42.2(pid=21085): Terminated: ok {rc=0}
12:10:53.322549 --- ParallelTerminationStrategy: PID 21086 died...
12:10:53.342035 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21086): Terminated: ok {rc=0}
12:10:53.386923 --- ParallelTerminationStrategy: PID 21087 died...
12:10:53.405754 run osmo-stp_10.42.42.5(pid=21087): Terminated: ok {rc=0}
12:10:53.450128 --- ParallelTerminationStrategy: PID 21088 died...
12:10:53.469159 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21088): Terminated: ok {rc=0}
12:10:53.513632 --- ParallelTerminationStrategy: PID 21093 died...
12:10:53.532584 run osmo-ggsn_10.42.42.6(pid=21093): Terminated: ok {rc=0}
12:10:53.577273 --- ParallelTerminationStrategy: PID 21112 died...
12:10:53.596432 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21112): Terminated: ok {rc=0}
12:10:53.641516 --- ParallelTerminationStrategy: PID 21113 died...
12:10:53.660431 run osmo-sgsn_10.42.42.7(pid=21113): Terminated: ok {rc=0}
12:10:53.705016 --- ParallelTerminationStrategy: PID 21114 died...
12:10:53.724187 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21114): Terminated: ok {rc=0}
12:10:53.768897 --- ParallelTerminationStrategy: PID 21115 died...
12:10:53.787839 run osmo-msc_10.42.42.8(pid=21115): Terminated: ok {rc=0}
12:10:53.832403 --- ParallelTerminationStrategy: PID 21116 died...
12:10:53.851434 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21116): Terminated: ok {rc=0}
12:10:53.896321 --- ParallelTerminationStrategy: PID 21117 died...
12:10:53.915307 run osmo-mgw_10.42.42.3(pid=21117): Terminated {rc=15}
12:10:53.960637 --- ParallelTerminationStrategy: PID 21118 died...
12:10:53.979801 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21118): Terminated: ok {rc=0}
12:10:54.024437 --- ParallelTerminationStrategy: PID 21119 died...
12:10:54.043456 run osmo-mgw_10.42.42.4(pid=21119): Terminated {rc=15}
12:10:54.088072 --- ParallelTerminationStrategy: PID 21120 died...
12:10:54.107141 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21120): Terminated: ok {rc=0}
12:10:54.151787 --- ParallelTerminationStrategy: PID 21121 died...
12:10:54.170940 run osmo-bsc_10.42.42.9(pid=21121): Terminated: ok {rc=0}
12:10:54.215853 --- ParallelTerminationStrategy: PID 21130 died...
12:10:54.235028 run osmo-bts-sysmo(pid=21130): Terminated: ok {rc=0}
12:10:54.279958 --- ParallelTerminationStrategy: PID 21138 died...
12:10:54.299435 run osmo-pcu(pid=21138): Terminated: ok {rc=0}
12:11:00.166869 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163): Terminating (SIGINT)
12:11:00.305838 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGINT(pid=21195): Launched
12:11:01.393543 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGINT(pid=21195): Terminated: ok {rc=0}
12:11:07.279070 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163): Terminating (SIGKILL)
12:11:07.417628 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGUSR1(pid=21199): Launched
12:11:08.508161 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21163)-killSignals.SIGUSR1(pid=21199): Terminated: ok {rc=0}
12:11:08.764530 bus /gobi_6: Setting Online False
12:11:11.787353 bus /gobi_6: Setting Powered False
12:11:27.183988 bus /gobi_6: Setting Powered False
----------------------------------------------
trial-2067 gprs:sysmo iperf3m4.py
----------------------------------------------
12:11:28.384003 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.395446 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:11:28.431056 tst iperf3m4.py: using LAC 3968
12:11:28.550276 tst iperf3m4.py: using RAC 143
12:11:28.668751 tst iperf3m4.py: using CellId 3968
12:11:28.787890 tst iperf3m4.py: using BVCI 3969
12:11:28.900390 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.911149 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.921645 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.932208 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.942826 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.953241 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.963905 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.974434 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.985081 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.103531 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.224807 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.342410 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.459596 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
12:11:29.680058 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:30.009646 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21202): Launched
12:11:30.296529 run iperf3-srv_10.42.42.10(pid=21203): Launched
12:11:30.419486 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
12:11:30.639186 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:30.968116 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21204): Launched
12:11:31.253544 run iperf3-srv_10.42.42.10(pid=21205): Launched
12:11:31.377229 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
12:11:31.595585 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:31.923683 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21206): Launched
12:11:32.208897 run iperf3-srv_10.42.42.10(pid=21207): Launched
12:11:32.332208 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
12:11:32.550589 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:32.878653 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21208): Launched
12:11:33.164342 run iperf3-srv_10.42.42.10(pid=21209): Launched
12:11:33.285889 tst iperf3m4.py:8: start network...
12:11:33.503672 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:11:34.012087 run create_hlr_db(pid=21210): Launched
12:11:34.144639 bus /gobi_6: Setting Powered False
12:11:35.209213 run create_hlr_db(pid=21210): Terminated: ok {rc=0}
12:11:35.647541 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21212): Launched
12:11:35.955109 run osmo-hlr_10.42.42.2(pid=21213): Launched
12:11:36.072243 run osmo-stp_10.42.42.5: Starting osmo-stp
12:11:36.596938 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21214): Launched
12:11:36.905571 run osmo-stp_10.42.42.5(pid=21215): Launched
12:11:37.023602 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:11:37.546954 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21216): Launched
12:11:37.664764 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:11:37.923089 run patchelf(pid=21217): Launched
12:11:39.096651 run patchelf(pid=21217): Terminated: ok {rc=0}
12:11:39.224613 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:11:39.486000 run osmo-gsm-tester_setcap_net_admin.sh(pid=21218): Launched
12:11:40.666497 run osmo-gsm-tester_setcap_net_admin.sh(pid=21218): Terminated: ok {rc=0}
12:11:40.984569 run osmo-ggsn_10.42.42.6(pid=21221): Launched
12:11:41.106036 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:11:41.779554 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21238): Launched
12:11:42.087089 run osmo-sgsn_10.42.42.7(pid=21239): Launched
12:11:42.203454 run osmo-msc_10.42.42.8: Starting osmo-msc
12:11:42.761266 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21240): Launched
12:11:43.071582 run osmo-msc_10.42.42.8(pid=21241): Launched
12:11:46.242287 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:11:46.780984 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21242): Launched
12:11:47.089717 run osmo-mgw_10.42.42.3(pid=21243): Launched
12:11:47.205741 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:11:47.717901 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21244): Launched
12:11:48.024814 run osmo-mgw_10.42.42.4(pid=21245): Launched
12:11:48.142544 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:11:48.773949 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21246): Launched
12:11:49.082553 run osmo-bsc_10.42.42.9(pid=21247): Launched
12:11:55.353567 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:11:55.884557 run rm-remote-dir(pid=21248): Launched
12:11:58.068952 run rm-remote-dir(pid=21248): Terminated: ok {rc=0}
12:11:58.406187 run mk-remote-dir(pid=21249): Launched
12:11:59.584378 run mk-remote-dir(pid=21249): Terminated: ok {rc=0}
12:11:59.872330 run scp-inst-to-remote(pid=21250): Launched
12:12:12.069405 run scp-inst-to-remote(pid=21250): Terminated: ok {rc=0}
12:12:12.424698 run mk-remote-dir(pid=21252): Launched
12:12:13.613724 run mk-remote-dir(pid=21252): Terminated: ok {rc=0}
12:12:13.917714 run scp-cfg-to-remote(pid=21253): Launched
12:12:15.105122 run scp-cfg-to-remote(pid=21253): Terminated: ok {rc=0}
12:12:15.456355 run reload-dsp-firmware(pid=21255): Launched
12:12:17.645098 run reload-dsp-firmware(pid=21255): Terminated: ok {rc=0}
12:12:17.988437 run osmo-bts-sysmo(pid=21256): Launched
12:12:22.224269 tst iperf3m4.py:8: Waiting for bts to be ready...
12:12:23.945400 run rm-remote-dir(pid=21257): Launched
12:12:26.133495 run rm-remote-dir(pid=21257): Terminated: ok {rc=0}
12:12:26.478836 run mk-remote-dir(pid=21258): Launched
12:12:28.661419 run mk-remote-dir(pid=21258): Terminated: ok {rc=0}
12:12:28.957983 run scp-inst-to-bts(pid=21259): Launched
12:12:42.163024 run scp-inst-to-bts(pid=21259): Terminated: ok {rc=0}
12:12:42.486836 run mk-remote-run-dir(pid=21261): Launched
12:12:44.660848 run mk-remote-run-dir(pid=21261): Terminated: ok {rc=0}
12:12:44.939910 run scp-cfg-to-sysmobts(pid=21262): Launched
12:12:47.114560 run scp-cfg-to-sysmobts(pid=21262): Terminated: ok {rc=0}
12:12:47.438562 run osmo-pcu(pid=21264): Launched
12:12:47.563668 tst iperf3m4.py: using MSISDN 3985
12:12:47.739112 bus /gobi_6: Setting Powered True
12:12:51.248761 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3985', subscriber_id=1}
12:12:51.515085 bus /gobi_6: Setting Online False
12:12:52.701763 bus /gobi_6: Setting Powered False
12:13:00.111592 bus /gobi_6: Setting Powered True
12:13:01.468638 bus /gobi_6: Setting Online True
12:13:03.911468 tst /gobi_6: Connect to ('901', '70')
12:13:04.188571 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:13:04.310566 tst iperf3m4.py: using MSISDN 3986
12:13:04.586416 bus /gobi_0: Setting Powered True
12:13:08.046551 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3986', subscriber_id=2}
12:13:08.319244 bus /gobi_0: Setting Online False
12:13:09.513095 bus /gobi_0: Setting Powered False
12:13:16.291138 bus /gobi_0: Setting Powered True
12:13:17.664118 bus /gobi_0: Setting Online True
12:13:19.490609 tst /gobi_6: Already registered with network ('901', '70')
12:13:20.227886 tst /gobi_0: Connect to ('901', '70')
12:13:20.510456 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873'
12:13:20.634269 tst iperf3m4.py: using MSISDN 3987
12:13:20.865790 bus /gobi_1: Setting Powered True
12:13:24.374020 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023848', msisdn='3987', subscriber_id=3}
12:13:24.641941 bus /gobi_1: Setting Online False
12:13:25.828852 bus /gobi_1: Setting Powered False
12:13:33.125594 bus /gobi_1: Setting Powered True
12:13:34.494953 bus /gobi_1: Setting Online True
12:13:35.553379 tst /gobi_0: Already registered with network ('901', '70')
12:13:37.188906 tst /gobi_1: Connect to ('901', '70')
12:13:37.455900 tst /gobi_1: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024272103'
12:13:37.574543 tst iperf3m4.py: using MSISDN 3988
12:13:37.743031 bus /gobi_4: Setting Powered True
12:13:41.397692 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023849', msisdn='3988', subscriber_id=4}
12:13:41.670192 bus /gobi_4: Setting Online False
12:13:42.859108 bus /gobi_4: Setting Powered False
12:13:50.128454 bus /gobi_4: Setting Powered True
12:13:51.520078 bus /gobi_4: Setting Online True
12:13:54.001494 tst /gobi_4: Connect to ('901', '70')
12:13:54.288455 tst /gobi_4: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A08E2G 1 [Jul 21 2016 10:00:50]', Serial: '868323025161511'
12:13:54.405304 tst iperf3m4.py:8: waiting for modems to attach...
12:13:55.046687 tst /gobi_1: Already registered with network ('901', '70')
12:14:09.960730 tst /gobi_4: Already registered with network ('901', '70')
12:14:15.035163 tst iperf3m4.py:8: waiting for modems to attach to data services...
12:14:18.146259 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:18.264777 tst iperf3m4.py:8: Setting up data plan for "'/gobi_6/context2'"
12:14:18.711233 run move_netns(pid=21268): Launched
12:14:19.903227 run move_netns(pid=21268): Terminated: ok {rc=0}
12:14:20.255338 run ifup(pid=21271): Launched
12:14:21.447544 run ifup(pid=21271): Terminated: ok {rc=0}
12:14:21.777096 run dhcp(pid=21274): Launched
12:14:22.970070 run dhcp(pid=21274): Terminated: ok {rc=0}
12:14:26.121945 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:26.233462 tst iperf3m4.py:8: Setting up data plan for "'/gobi_0/context2'"
12:14:26.655066 run move_netns(pid=21292): Launched
12:14:27.848406 run move_netns(pid=21292): Terminated: ok {rc=0}
12:14:28.200391 run ifup(pid=21295): Launched
12:14:29.389588 run ifup(pid=21295): Terminated: ok {rc=0}
12:14:29.741867 run dhcp(pid=21298): Launched
12:14:30.932556 run dhcp(pid=21298): Terminated: ok {rc=0}
12:14:39.335976 tst /gobi_1: context activated {apn='inet46', path='/gobi_1/context12', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.3', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='rf303c5eb721dad', Method='static', Netmask='255.255.255.248'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:39.454178 tst iperf3m4.py:8: Setting up data plan for "'/gobi_1/context12'"
12:14:39.897590 run move_netns(pid=21316): Launched
12:14:41.093505 run move_netns(pid=21316): Terminated: ok {rc=0}
12:14:41.444972 run ifup(pid=21319): Launched
12:14:42.640326 run ifup(pid=21319): Terminated: ok {rc=0}
12:14:42.992193 run dhcp(pid=21322): Launched
12:14:44.177569 run dhcp(pid=21322): Terminated: ok {rc=0}
12:14:47.303839 tst /gobi_4: context activated {apn='inet46', path='/gobi_4/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.4', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.5', Interface='r9d788e55295d1b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:47.421760 tst iperf3m4.py:8: Setting up data plan for "'/gobi_4/context2'"
12:14:47.866923 run move_netns(pid=21344): Launched
12:14:49.057395 run move_netns(pid=21344): Terminated: ok {rc=0}
12:14:49.409027 run ifup(pid=21347): Launched
12:14:50.591685 run ifup(pid=21347): Terminated: ok {rc=0}
12:14:50.921944 run dhcp(pid=21350): Launched
12:14:52.101072 run dhcp(pid=21350): Terminated: ok {rc=0}
12:14:52.230967 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'"
12:14:52.440242 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:14:52.804921 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Launched
12:14:52.988920 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'"
12:14:53.213137 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s)
12:14:53.602018 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Launched
12:14:53.783997 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5005 through "'/gobi_1/context12'"
12:14:54.007789 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5005 (proto=tcp,time=60s)
12:14:54.397364 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Launched
12:14:54.579788 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5006 through "'/gobi_4/context2'"
12:14:54.803333 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5006 (proto=tcp,time=60s)
12:14:55.191443 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Launched
12:14:55.532646 run iperf3-cli_10.42.42.10(pid=21378): Launched
12:14:55.817872 run iperf3-cli_10.42.42.10(pid=21381): Launched
12:14:56.104486 run iperf3-cli_10.42.42.10(pid=21384): Launched
12:14:56.394604 run iperf3-cli_10.42.42.10(pid=21387): Launched
12:16:19.737364 run iperf3-cli_10.42.42.10(pid=21378): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21378)]
12:16:19.872153 run iperf3-cli_10.42.42.10(pid=21378): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:16:20.007173 run iperf3-cli_10.42.42.10(pid=21378): stderr:
| (launched: 2022-09-08_12:14:55.515951)
12:16:21.197603 run iperf3-cli_10.42.42.10(pid=21381): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21381)]
12:16:21.330423 run iperf3-cli_10.42.42.10(pid=21381): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:16:21.463686 run iperf3-cli_10.42.42.10(pid=21381): stderr:
| (launched: 2022-09-08_12:14:55.801521)
12:16:22.657322 run iperf3-cli_10.42.42.10(pid=21384): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21384)]
12:16:22.795647 run iperf3-cli_10.42.42.10(pid=21384): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:16:22.932723 run iperf3-cli_10.42.42.10(pid=21384): stderr:
| (launched: 2022-09-08_12:14:56.087095)
12:16:24.130997 run iperf3-cli_10.42.42.10(pid=21387): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21387)]
12:16:24.265284 run iperf3-cli_10.42.42.10(pid=21387): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:16:24.399499 run iperf3-cli_10.42.42.10(pid=21387): stderr:
| (launched: 2022-09-08_12:14:56.377632)
12:16:24.533153 run iperf3-srv_10.42.42.10(pid=21203): Terminating (SIGINT)
12:16:24.870598 run iperf3-srv_10.42.42.10(pid=21203): Terminated {rc=1}
12:16:24.991261 tst iperf3m4.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context2'"
Exception while using iperf3 client results: "{'start': {'connected': [{'socket': 5, 'local_host': '176.16.46.1', 'local_port': 47468, 'remote_host': '10.42.42.10', 'remote_port': 5003}], 'version': 'iperf 3.6', 'system_info': 'Linux osmo-gsm-tester-prod 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64', 'timestamp': {'time': 'Thu, 08 Sep 2022 10:15:11 GMT', 'timesecs': 1662632111}, 'connecting_to': {'host': '10.42.42.10', 'port': 5003}, 'cookie': 'nv347cizjtmgswwq3u3gartd2nu7dlx7uv27', 'tcp_mss_default': 1448, 'sock_bufsize': 0, 'sndbuf_actual': 16384, 'rcvbuf_actual': 131072, 'test_start': {'protocol': 'TCP', 'num_streams': 1, 'blksize': 131072, 'omit': 0, 'duration': 60, 'bytes': 0, 'blocks': 0, 'reverse': 0, 'tos': 0}}, 'intervals': [{'streams': [{'socket': 5, 'start': 0, 'end': 1.0007679462432861, 'seconds': 1.0007679462432861, 'bytes': 101360, 'bits_per_second': 810257.7655928195, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 507991, 'rttvar': 267499, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 0, 'end': 1.0007679462432861, 'seconds': 1.0007679462432861, 'bytes': 101360, 'bits_per_second': 810257.7655928195, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 1.0007679462432861, 'end': 2.000662088394165, 'seconds': 0.9998941421508789, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 507991, 'rttvar': 267499, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 1.0007679462432861, 'end': 2.000662088394165, 'seconds': 0.9998941421508789, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 2.000662088394165, 'end': 3.0012669563293457, 'seconds': 1.0006048679351807, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 15928, 'rtt': 692039, 'rttvar': 568720, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 2.000662088394165, 'end': 3.0012669563293457, 'seconds': 1.0006048679351807, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 3.0012669563293457, 'end': 4.00131893157959, 'seconds': 1.0000519752502441, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'snd_cwnd': 17376, 'rtt': 1035284, 'rttvar': 1113030, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 3.0012669563293457, 'end': 4.00131893157959, 'seconds': 1.0000519752502441, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 4.00131893157959, 'end': 5.001167058944702, 'seconds': 0.9998481273651123, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 17376, 'rtt': 1035284, 'rttvar': 1113030, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 4.00131893157959, 'end': 5.001167058944702, 'seconds': 0.9998481273651123, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 5.001167058944702, 'end': 6.001317977905273, 'seconds': 1.0001509189605713, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 18824, 'rtt': 1530575, 'rttvar': 1825353, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 5.001167058944702, 'end': 6.001317977905273, 'seconds': 1.0001509189605713, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 6.001317977905273, 'end': 7.001166105270386, 'seconds': 0.9998481273651123, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 20272, 'rtt': 2151465, 'rttvar': 2610796, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 6.001317977905273, 'end': 7.001166105270386, 'seconds': 0.9998481273651123, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 7.001166105270386, 'end': 8.001166105270386, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 20272, 'rtt': 2151465, 'rttvar': 2610796, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 7.001166105270386, 'end': 8.001166105270386, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 8.001166105270386, 'end': 9.00116515159607, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 21720, 'rtt': 2885095, 'rttvar': 3425357, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 8.001166105270386, 'end': 9.00116515159607, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 9.00116515159607, 'end': 10.001085042953491, 'seconds': 0.9999198913574219, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 21720, 'rtt': 2885095, 'rttvar': 3425357, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 9.00116515159607, 'end': 10.001085042953491, 'seconds': 0.9999198913574219, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 10.001085042953491, 'end': 11.001328945159912, 'seconds': 1.000243902206421, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 23168, 'rtt': 3716796, 'rttvar': 4232419, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 10.001085042953491, 'end': 11.001328945159912, 'seconds': 1.000243902206421, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 11.001328945159912, 'end': 12.001320123672485, 'seconds': 0.9999911785125732, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 24616, 'rtt': 4629685, 'rttvar': 5000092, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 11.001328945159912, 'end': 12.001320123672485, 'seconds': 0.9999911785125732, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 12.001320123672485, 'end': 13.001320123672485, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 24616, 'rtt': 4629685, 'rttvar': 5000092, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 12.001320123672485, 'end': 13.001320123672485, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 13.001320123672485, 'end': 14.001325130462646, 'seconds': 1.0000050067901611, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 26064, 'rtt': 5628060, 'rttvar': 5746819, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 13.001320123672485, 'end': 14.001325130462646, 'seconds': 1.0000050067901611, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 14.001325130462646, 'end': 15.001151084899902, 'seconds': 0.9998259544372559, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 27512, 'rtt': 6691993, 'rttvar': 6437980, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 14.001325130462646, 'end': 15.001151084899902, 'seconds': 0.9998259544372559, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 15.001151084899902, 'end': 16.001330137252808, 'seconds': 1.0001790523529053, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 27512, 'rtt': 6691993, 'rttvar': 6437980, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 15.001151084899902, 'end': 16.001330137252808, 'seconds': 1.0001790523529053, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 16.001330137252808, 'end': 17.00131893157959, 'seconds': 0.9999887943267822, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 28960, 'rtt': 7814996, 'rttvar': 7074492, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 16.001330137252808, 'end': 17.00131893157959, 'seconds': 0.9999887943267822, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 17.00131893157959, 'end': 18.00114607810974, 'seconds': 0.9998271465301514, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 30408, 'rtt': 8854569, 'rttvar': 7385014, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 17.00131893157959, 'end': 18.00114607810974, 'seconds': 0.9998271465301514, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 18.00114607810974, 'end': 19.001327991485596, 'seconds': 1.0001819133758545, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 30408, 'rtt': 8854569, 'rttvar': 7385014, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 18.00114607810974, 'end': 19.001327991485596, 'seconds': 1.0001819133758545, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 19.001327991485596, 'end': 20.00126600265503, 'seconds': 0.9999380111694336, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 9837713, 'rttvar': 7505050, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 19.001327991485596, 'end': 20.00126600265503, 'seconds': 0.9999380111694336, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 20.00126600265503, 'end': 21.001070976257324, 'seconds': 0.9998049736022949, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 9837713, 'rttvar': 7505050, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 20.00126600265503, 'end': 21.001070976257324, 'seconds': 0.9998049736022949, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 21.001070976257324, 'end': 22.001305103302002, 'seconds': 1.0002341270446777, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 9837713, 'rttvar': 7505050, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 21.001070976257324, 'end': 22.001305103302002, 'seconds': 1.0002341270446777, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 22.001305103302002, 'end': 23.001161098480225, 'seconds': 0.9998559951782227, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 33304, 'rtt': 10895330, 'rttvar': 7744021, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 22.001305103302002, 'end': 23.001161098480225, 'seconds': 0.9998559951782227, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 23.001161098480225, 'end': 24.00130605697632, 'seconds': 1.0001449584960938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 12013358, 'rttvar': 8044070, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 23.001161098480225, 'end': 24.00130605697632, 'seconds': 1.0001449584960938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 24.00130605697632, 'end': 25.001070976257324, 'seconds': 0.9997649192810059, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 12013358, 'rttvar': 8044070, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 24.00130605697632, 'end': 25.001070976257324, 'seconds': 0.9997649192810059, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 25.001070976257324, 'end': 26.00126600265503, 'seconds': 1.000195026397705, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 36200, 'rtt': 12981903, 'rttvar': 7970143, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 25.001070976257324, 'end': 26.00126600265503, 'seconds': 1.000195026397705, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 26.00126600265503, 'end': 27.00131893157959, 'seconds': 1.0000529289245605, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 37648, 'rtt': 14019142, 'rttvar': 8052085, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 26.00126600265503, 'end': 27.00131893157959, 'seconds': 1.0000529289245605, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 27.00131893157959, 'end': 28.001266956329346, 'seconds': 0.9999480247497559, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 37648, 'rtt': 14019142, 'rttvar': 8052085, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 27.00131893157959, 'end': 28.001266956329346, 'seconds': 0.9999480247497559, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 28.001266956329346, 'end': 29.001308917999268, 'seconds': 1.0000419616699219, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 39096, 'rtt': 14934241, 'rttvar': 7869263, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 28.001266956329346, 'end': 29.001308917999268, 'seconds': 1.0000419616699219, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 29.001308917999268, 'end': 30.001138925552368, 'seconds': 0.9998300075531006, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 40544, 'rtt': 15917279, 'rttvar': 7868024, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 29.001308917999268, 'end': 30.001138925552368, 'seconds': 0.9998300075531006, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 30.001138925552368, 'end': 31.001156091690063, 'seconds': 1.0000171661376953, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 40544, 'rtt': 15917279, 'rttvar': 7868024, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 30.001138925552368, 'end': 31.001156091690063, 'seconds': 1.0000171661376953, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 31.001156091690063, 'end': 32.001073122024536, 'seconds': 0.9999170303344727, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 16772137, 'rttvar': 7610734, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 31.001156091690063, 'end': 32.001073122024536, 'seconds': 0.9999170303344727, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 32.001073122024536, 'end': 33.00126600265503, 'seconds': 1.0001928806304932, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 17712816, 'rttvar': 7589407, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 32.001073122024536, 'end': 33.00126600265503, 'seconds': 1.0001928806304932, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 33.00126600265503, 'end': 34.0011420249939, 'seconds': 0.9998760223388672, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 17712816, 'rttvar': 7589407, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 33.00126600265503, 'end': 34.0011420249939, 'seconds': 0.9998760223388672, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 34.0011420249939, 'end': 35.001166105270386, 'seconds': 1.0000240802764893, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 18538249, 'rttvar': 7342922, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 34.0011420249939, 'end': 35.001166105270386, 'seconds': 1.0000240802764893, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 35.001166105270386, 'end': 36.00111794471741, 'seconds': 0.9999518394470215, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 19453154, 'rttvar': 7337001, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 35.001166105270386, 'end': 36.00111794471741, 'seconds': 0.9999518394470215, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 36.00111794471741, 'end': 37.00128698348999, 'seconds': 1.000169038772583, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 19453154, 'rttvar': 7337001, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 36.00111794471741, 'end': 37.00128698348999, 'seconds': 1.000169038772583, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 37.00128698348999, 'end': 38.00131702423096, 'seconds': 1.0000300407409668, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 20250812, 'rttvar': 7098067, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 37.00128698348999, 'end': 38.00131702423096, 'seconds': 1.0000300407409668, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 38.00131702423096, 'end': 39.00122094154358, 'seconds': 0.9999039173126221, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 21136253, 'rttvar': 7094432, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 38.00131702423096, 'end': 39.00122094154358, 'seconds': 0.9999039173126221, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 39.00122094154358, 'end': 40.00131893157959, 'seconds': 1.0000979900360107, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 21136253, 'rttvar': 7094432, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 39.00122094154358, 'end': 40.00131893157959, 'seconds': 1.0000979900360107, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 40.00131893157959, 'end': 41.00126910209656, 'seconds': 0.9999501705169678, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 21904109, 'rttvar': 6856536, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 40.00131893157959, 'end': 41.00126910209656, 'seconds': 0.9999501705169678, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 41.00126910209656, 'end': 42.001307010650635, 'seconds': 1.0000379085540771, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 22763479, 'rttvar': 6861143, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 41.00126910209656, 'end': 42.001307010650635, 'seconds': 1.0000379085540771, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 42.001307010650635, 'end': 43.001307010650635, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 22763479, 'rttvar': 6861143, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 42.001307010650635, 'end': 43.001307010650635, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 43.001307010650635, 'end': 44.00131607055664, 'seconds': 1.0000090599060059, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 23525223, 'rttvar': 6669344, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 43.001307010650635, 'end': 44.00131607055664, 'seconds': 1.0000090599060059, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 44.00131607055664, 'end': 45.00116801261902, 'seconds': 0.9998519420623779, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 23525223, 'rttvar': 6669344, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 44.00131607055664, 'end': 45.00116801261902, 'seconds': 0.9998519420623779, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 45.00116801261902, 'end': 46.00107002258301, 'seconds': 0.9999020099639893, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 24386695, 'rttvar': 6724952, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 45.00116801261902, 'end': 46.00107002258301, 'seconds': 0.9999020099639893, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 46.00107002258301, 'end': 47.00105810165405, 'seconds': 0.9999880790710449, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 25133094, 'rttvar': 6536512, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 46.00107002258301, 'end': 47.00105810165405, 'seconds': 0.9999880790710449, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 47.00105810165405, 'end': 48.001057147979736, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 25133094, 'rttvar': 6536512, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 47.00105810165405, 'end': 48.001057147979736, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 48.001057147979736, 'end': 49.001057147979736, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 25988586, 'rttvar': 6613368, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 48.001057147979736, 'end': 49.001057147979736, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 49.001057147979736, 'end': 50.001304149627686, 'seconds': 1.0002470016479492, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 26764938, 'rttvar': 6512731, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 49.001057147979736, 'end': 50.001304149627686, 'seconds': 1.0002470016479492, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 50.001304149627686, 'end': 51.00120306015015, 'seconds': 0.9998989105224609, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 26764938, 'rttvar': 6512731, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 50.001304149627686, 'end': 51.00120306015015, 'seconds': 0.9998989105224609, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 51.00120306015015, 'end': 52.00122809410095, 'seconds': 1.0000250339508057, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 27661740, 'rttvar': 6678153, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 51.00120306015015, 'end': 52.00122809410095, 'seconds': 1.0000250339508057, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 52.00122809410095, 'end': 53.00126814842224, 'seconds': 1.000040054321289, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 27661740, 'rttvar': 6678153, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 52.00122809410095, 'end': 53.00126814842224, 'seconds': 1.000040054321289, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 53.00126814842224, 'end': 54.00116801261902, 'seconds': 0.9998998641967773, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 28448716, 'rttvar': 6582565, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 53.00126814842224, 'end': 54.00116801261902, 'seconds': 0.9998998641967773, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 54.00116801261902, 'end': 55.000683069229126, 'seconds': 0.9995150566101074, 'bytes': 256296, 'bits_per_second': 2051362.7948276233, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 29321833, 'rttvar': 6683158, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 54.00116801261902, 'end': 55.000683069229126, 'seconds': 0.9995150566101074, 'bytes': 256296, 'bits_per_second': 2051362.7948276233, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 55.000683069229126, 'end': 56.00126814842224, 'seconds': 1.0005850791931152, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 29321833, 'rttvar': 6683158, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 55.000683069229126, 'end': 56.00126814842224, 'seconds': 1.0005850791931152, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 56.00126814842224, 'end': 57.0011670589447, 'seconds': 0.9998989105224609, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 29896515, 'rttvar': 6161733, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 56.00126814842224, 'end': 57.0011670589447, 'seconds': 0.9998989105224609, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 57.0011670589447, 'end': 58.00122404098511, 'seconds': 1.0000569820404053, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 30594352, 'rttvar': 6016973, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 57.0011670589447, 'end': 58.00122404098511, 'seconds': 1.0000569820404053, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 58.00122404098511, 'end': 59.00116515159607, 'seconds': 0.9999411106109619, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 30594352, 'rttvar': 6016973, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 58.00122404098511, 'end': 59.00116515159607, 'seconds': 0.9999411106109619, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}], 'end': {}, 'error': 'error - control socket has closed unexpectedly'}"
12:16:25.220774 tst iperf3m4.py:8: ERR: KeyError: 'sum_sent' [trial-2067↪gprs:sysmo↪iperf3m4.py:8]
12:16:25.228332 tst iperf3m4.py:8: Test FAILED (296.9 sec)
12:16:25.235379 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21202): Terminating (SIGTERM)
12:16:25.278066 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21204): Terminating (SIGTERM)
12:16:25.320404 run iperf3-srv_10.42.42.10(pid=21205): Terminating (SIGTERM)
12:16:25.362500 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21206): Terminating (SIGTERM)
12:16:25.405066 run iperf3-srv_10.42.42.10(pid=21207): Terminating (SIGTERM)
12:16:25.447263 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21208): Terminating (SIGTERM)
12:16:25.489519 run iperf3-srv_10.42.42.10(pid=21209): Terminating (SIGTERM)
12:16:25.531838 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21212): Terminating (SIGTERM)
12:16:25.574175 run osmo-hlr_10.42.42.2(pid=21213): Terminating (SIGTERM)
12:16:25.616500 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21214): Terminating (SIGTERM)
12:16:25.660002 run osmo-stp_10.42.42.5(pid=21215): Terminating (SIGTERM)
12:16:25.703559 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21216): Terminating (SIGTERM)
12:16:25.766511 run osmo-ggsn_10.42.42.6(pid=21221): Terminating (SIGTERM)
12:16:25.809609 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21238): Terminating (SIGTERM)
12:16:25.874115 run osmo-sgsn_10.42.42.7(pid=21239): Terminating (SIGTERM)
12:16:25.937925 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21240): Terminating (SIGTERM)
12:16:26.010230 run osmo-msc_10.42.42.8(pid=21241): Terminating (SIGTERM)
12:16:26.061108 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21242): Terminating (SIGTERM)
12:16:26.108416 run osmo-mgw_10.42.42.3(pid=21243): Terminating (SIGTERM)
12:16:26.152172 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21244): Terminating (SIGTERM)
12:16:26.196699 run osmo-mgw_10.42.42.4(pid=21245): Terminating (SIGTERM)
12:16:26.240199 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21246): Terminating (SIGTERM)
12:16:26.284120 run osmo-bsc_10.42.42.9(pid=21247): Terminating (SIGTERM)
12:16:26.330611 run osmo-bts-sysmo(pid=21256): Terminating (SIGTERM)
12:16:26.377397 run osmo-pcu(pid=21264): Terminating (SIGTERM)
12:16:26.421520 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Terminating (SIGTERM)
12:16:26.561361 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGTERM(pid=21422): Launched
12:16:27.654258 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGTERM(pid=21422): Terminated: ok {rc=0}
12:16:27.717981 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Terminating (SIGTERM)
12:16:27.855473 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGTERM(pid=21427): Launched
12:16:28.946489 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGTERM(pid=21427): Terminated: ok {rc=0}
12:16:29.013419 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Terminating (SIGTERM)
12:16:29.158021 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGTERM(pid=21430): Launched
12:16:30.249996 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGTERM(pid=21430): Terminated: ok {rc=0}
12:16:30.317052 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Terminating (SIGTERM)
12:16:30.462136 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGTERM(pid=21433): Launched
12:16:31.555542 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGTERM(pid=21433): Terminated: ok {rc=0}
12:16:31.622585 --- ParallelTerminationStrategy: PID 21202 died...
12:16:31.642845 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21202): Terminated: ok {rc=0}
12:16:31.690104 --- ParallelTerminationStrategy: PID 21204 died...
12:16:31.709853 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21204): Terminated: ok {rc=0}
12:16:31.756907 --- ParallelTerminationStrategy: PID 21205 died...
12:16:31.776632 run iperf3-srv_10.42.42.10(pid=21205): Terminated {rc=256}
12:16:31.823671 --- ParallelTerminationStrategy: PID 21206 died...
12:16:31.843334 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21206): Terminated: ok {rc=0}
12:16:31.890167 --- ParallelTerminationStrategy: PID 21207 died...
12:16:31.909939 run iperf3-srv_10.42.42.10(pid=21207): Terminated {rc=256}
12:16:31.957008 --- ParallelTerminationStrategy: PID 21208 died...
12:16:31.976776 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21208): Terminated: ok {rc=0}
12:16:32.023716 --- ParallelTerminationStrategy: PID 21209 died...
12:16:32.043412 run iperf3-srv_10.42.42.10(pid=21209): Terminated {rc=256}
12:16:32.090085 --- ParallelTerminationStrategy: PID 21212 died...
12:16:32.109920 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21212): Terminated: ok {rc=0}
12:16:32.156748 --- ParallelTerminationStrategy: PID 21213 died...
12:16:32.176656 run osmo-hlr_10.42.42.2(pid=21213): Terminated: ok {rc=0}
12:16:32.223775 --- ParallelTerminationStrategy: PID 21214 died...
12:16:32.243521 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21214): Terminated: ok {rc=0}
12:16:32.290343 --- ParallelTerminationStrategy: PID 21215 died...
12:16:32.310073 run osmo-stp_10.42.42.5(pid=21215): Terminated: ok {rc=0}
12:16:32.356992 --- ParallelTerminationStrategy: PID 21216 died...
12:16:32.376806 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21216): Terminated: ok {rc=0}
12:16:32.423956 --- ParallelTerminationStrategy: PID 21221 died...
12:16:32.443632 run osmo-ggsn_10.42.42.6(pid=21221): Terminated: ok {rc=0}
12:16:32.490672 --- ParallelTerminationStrategy: PID 21238 died...
12:16:32.510682 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21238): Terminated: ok {rc=0}
12:16:32.557675 --- ParallelTerminationStrategy: PID 21239 died...
12:16:32.577412 run osmo-sgsn_10.42.42.7(pid=21239): Terminated: ok {rc=0}
12:16:32.624607 --- ParallelTerminationStrategy: PID 21240 died...
12:16:32.644464 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21240): Terminated: ok {rc=0}
12:16:32.691656 --- ParallelTerminationStrategy: PID 21241 died...
12:16:32.711344 run osmo-msc_10.42.42.8(pid=21241): Terminated: ok {rc=0}
12:16:32.758271 --- ParallelTerminationStrategy: PID 21242 died...
12:16:32.778055 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21242): Terminated: ok {rc=0}
12:16:32.825129 --- ParallelTerminationStrategy: PID 21243 died...
12:16:32.845043 run osmo-mgw_10.42.42.3(pid=21243): Terminated {rc=15}
12:16:32.892114 --- ParallelTerminationStrategy: PID 21244 died...
12:16:32.911927 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21244): Terminated: ok {rc=0}
12:16:32.959034 --- ParallelTerminationStrategy: PID 21245 died...
12:16:32.979047 run osmo-mgw_10.42.42.4(pid=21245): Terminated {rc=15}
12:16:33.026348 --- ParallelTerminationStrategy: PID 21246 died...
12:16:33.046116 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21246): Terminated: ok {rc=0}
12:16:33.093579 --- ParallelTerminationStrategy: PID 21247 died...
12:16:33.113446 run osmo-bsc_10.42.42.9(pid=21247): Terminated: ok {rc=0}
12:16:33.160501 --- ParallelTerminationStrategy: PID 21256 died...
12:16:33.180373 run osmo-bts-sysmo(pid=21256): Terminated: ok {rc=0}
12:16:33.227416 --- ParallelTerminationStrategy: PID 21264 died...
12:16:33.247424 run osmo-pcu(pid=21264): Terminated: ok {rc=0}
12:16:39.117951 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Terminating (SIGINT)
12:16:39.264422 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGINT(pid=21436): Launched
12:16:40.351273 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGINT(pid=21436): Terminated: ok {rc=0}
12:16:40.418583 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Terminating (SIGINT)
12:16:40.563765 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGINT(pid=21439): Launched
12:16:41.655299 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGINT(pid=21439): Terminated: ok {rc=0}
12:16:41.722445 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Terminating (SIGINT)
12:16:41.867252 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGINT(pid=21442): Launched
12:16:42.957759 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGINT(pid=21442): Terminated: ok {rc=0}
12:16:43.024837 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Terminating (SIGINT)
12:16:43.169622 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGINT(pid=21445): Launched
12:16:44.255153 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGINT(pid=21445): Terminated: ok {rc=0}
12:16:50.141528 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Terminating (SIGKILL)
12:16:50.287454 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGUSR1(pid=21448): Launched
12:16:51.383032 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGUSR1(pid=21448): Terminated: ok {rc=0}
12:16:51.447184 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Terminating (SIGKILL)
12:16:51.585330 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGUSR1(pid=21451): Launched
12:16:52.672745 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGUSR1(pid=21451): Terminated: ok {rc=0}
12:16:52.737068 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Terminating (SIGKILL)
12:16:52.875360 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGUSR1(pid=21454): Launched
12:16:53.962014 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGUSR1(pid=21454): Terminated: ok {rc=0}
12:16:54.025489 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Terminating (SIGKILL)
12:16:54.164474 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGUSR1(pid=21457): Launched
12:16:55.249989 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGUSR1(pid=21457): Terminated: ok {rc=0}
12:16:55.499633 bus /gobi_4: Setting Online False
12:16:56.640050 bus /gobi_4: Setting Powered False
12:17:06.942104 bus /gobi_4: Setting Powered False
12:17:08.041874 bus /gobi_4: Setting Powered False
12:17:09.200900 bus /gobi_1: Setting Online False
12:17:10.336516 bus /gobi_1: Setting Powered False
12:17:18.047015 bus /gobi_1: Setting Powered False
12:17:19.147116 bus /gobi_1: Setting Powered False
12:17:20.305538 bus /gobi_0: Setting Online False
12:17:21.438623 bus /gobi_0: Setting Powered False
12:17:29.941755 bus /gobi_0: Setting Powered False
12:17:31.040233 bus /gobi_0: Setting Powered False
12:17:32.198462 bus /gobi_6: Setting Online False
12:17:33.309492 bus /gobi_6: Setting Powered False
12:17:41.015174 bus /gobi_6: Setting Powered False
----------------------------------------------
trial-2067 gprs:sysmo ping.py
----------------------------------------------
12:17:42.175169 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:17:42.188562 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:17:42.224575 tst ping.py: using LAC 3969
12:17:42.355536 tst ping.py: using RAC 144
12:17:42.479640 tst ping.py: using CellId 3969
12:17:42.606062 tst ping.py: using BVCI 3970
12:17:42.718632 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:17:42.729645 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:17:42.740554 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:17:42.752146 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:17:42.763633 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:17:42.774962 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:17:42.786423 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:17:42.797861 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:17:42.912641 tst ping.py:19: start network...
12:17:43.112735 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:17:43.577843 run create_hlr_db(pid=21463): Launched
12:17:43.701812 bus /gobi_6: Setting Powered False
12:17:44.762501 run create_hlr_db(pid=21463): Terminated: ok {rc=0}
12:17:45.204091 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21465): Launched
12:17:45.527140 run osmo-hlr_10.42.42.2(pid=21466): Launched
12:17:45.649907 run osmo-stp_10.42.42.5: Starting osmo-stp
12:17:46.200905 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21467): Launched
12:17:46.527772 run osmo-stp_10.42.42.5(pid=21468): Launched
12:17:46.651691 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:17:47.204898 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21469): Launched
12:17:47.328558 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:17:47.602118 run patchelf(pid=21470): Launched
12:17:48.784841 run patchelf(pid=21470): Terminated: ok {rc=0}
12:17:48.920129 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:17:49.196573 run osmo-gsm-tester_setcap_net_admin.sh(pid=21471): Launched
12:17:50.383840 run osmo-gsm-tester_setcap_net_admin.sh(pid=21471): Terminated: ok {rc=0}
12:17:50.719505 run osmo-ggsn_10.42.42.6(pid=21474): Launched
12:17:50.842016 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:17:51.523828 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21491): Launched
12:17:51.825238 run osmo-sgsn_10.42.42.7(pid=21492): Launched
12:17:51.942294 run osmo-msc_10.42.42.8: Starting osmo-msc
12:17:52.489744 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21493): Launched
12:17:52.790805 run osmo-msc_10.42.42.8(pid=21494): Launched
12:17:55.957393 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:17:56.472686 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21495): Launched
12:17:56.775521 run osmo-mgw_10.42.42.3(pid=21496): Launched
12:17:56.891001 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:17:57.396935 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21497): Launched
12:17:57.699785 run osmo-mgw_10.42.42.4(pid=21498): Launched
12:17:57.813684 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:17:58.438063 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21499): Launched
12:17:58.740624 run osmo-bsc_10.42.42.9(pid=21500): Launched
12:18:05.006140 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:18:05.529573 run rm-remote-dir(pid=21501): Launched
12:18:07.709326 run rm-remote-dir(pid=21501): Terminated: ok {rc=0}
12:18:08.040283 run mk-remote-dir(pid=21502): Launched
12:18:09.216145 run mk-remote-dir(pid=21502): Terminated: ok {rc=0}
12:18:09.497953 run scp-inst-to-remote(pid=21503): Launched
12:18:21.687106 run scp-inst-to-remote(pid=21503): Terminated: ok {rc=0}
12:18:22.015355 run mk-remote-dir(pid=21505): Launched
12:18:23.189353 run mk-remote-dir(pid=21505): Terminated: ok {rc=0}
12:18:23.469864 run scp-cfg-to-remote(pid=21506): Launched
12:18:24.645108 run scp-cfg-to-remote(pid=21506): Terminated: ok {rc=0}
12:18:24.967837 run reload-dsp-firmware(pid=21508): Launched
12:18:27.140748 run reload-dsp-firmware(pid=21508): Terminated: ok {rc=0}
12:18:27.457181 run osmo-bts-sysmo(pid=21513): Launched
12:18:31.679327 tst ping.py:31: Waiting for bts to be ready...
12:18:33.343106 run rm-remote-dir(pid=21514): Launched
12:18:35.513467 run rm-remote-dir(pid=21514): Terminated: ok {rc=0}
12:18:35.831138 run mk-remote-dir(pid=21515): Launched
12:18:38.005047 run mk-remote-dir(pid=21515): Terminated: ok {rc=0}
12:18:38.279568 run scp-inst-to-bts(pid=21516): Launched
12:18:51.464951 run scp-inst-to-bts(pid=21516): Terminated: ok {rc=0}
12:18:51.782334 run mk-remote-run-dir(pid=21518): Launched
12:18:53.952829 run mk-remote-run-dir(pid=21518): Terminated: ok {rc=0}
12:18:54.227777 run scp-cfg-to-sysmobts(pid=21519): Launched
12:18:56.401757 run scp-cfg-to-sysmobts(pid=21519): Terminated: ok {rc=0}
12:18:56.741898 run osmo-pcu(pid=21521): Launched
12:18:56.874309 tst ping.py: using MSISDN 3989
12:18:57.048643 bus /gobi_6: Setting Powered True
12:19:00.558056 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3989', subscriber_id=1}
12:19:00.838243 bus /gobi_6: Setting Online False
12:19:02.025788 bus /gobi_6: Setting Powered False
12:19:09.127061 bus /gobi_6: Setting Powered True
12:19:10.509718 bus /gobi_6: Setting Online True
12:19:12.944654 tst /gobi_6: Connect to ('901', '70')
12:19:13.204235 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:19:13.311976 tst ping.py:42: waiting for modems to attach...
12:19:28.055880 tst /gobi_6: Already registered with network ('901', '70')
12:19:33.760765 tst ping.py:46: waiting for modems to attach to data services...
12:19:36.860190 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:19:36.969779 tst ping.py:51: Setting up data plan for "'/gobi_6/context3'"
12:19:37.397437 run move_netns(pid=21525): Launched
12:19:38.577893 run move_netns(pid=21525): Terminated: ok {rc=0}
12:19:38.906380 run ifup(pid=21528): Launched
12:19:40.083722 run ifup(pid=21528): Terminated: ok {rc=0}
12:19:40.411116 run dhcp(pid=21531): Launched
12:19:41.584443 run dhcp(pid=21531): Terminated: ok {rc=0}
12:19:41.712943 tst ping.py:53: Running 10 ping requests for "'/gobi_6/context3'"
12:19:42.140861 run ping(pid=21547): Launched
12:19:52.344683 run ping(pid=21547): Terminated: ok {rc=0}
12:19:53.992670 tst /gobi_6: context deactivated {path='/gobi_6/context3'}
12:20:01.107624 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={Interface='r433306bbc2604b'}, Name='Internet', Password='', Protocol='ipv6', Settings={}, Type='internet', Username='ogt'}, user='ogt'}
12:20:08.648434 tst /gobi_6: context deactivated {path='/gobi_6/context3'}
12:20:08.756995 tst ping.py:70: (launched: 2022-09-08_12:19:42.075900)
21548: waiting for 21549
PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data.
64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=614 ms
64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=532 ms
64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=509 ms
64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=492 ms
64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=511 ms
64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=507 ms
64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=488 ms
64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=484 ms
64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=485 ms
64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=462 ms
--- 10.42.42.6 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 18ms
rtt min/avg/max/mdev = 461.661/508.363/613.770/39.585 ms
child exited with 0
12:20:08.960428 tst ping.py: Test passed (146.8 sec)
12:20:09.025434 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21465): Terminating (SIGTERM)
12:20:09.066241 run osmo-hlr_10.42.42.2(pid=21466): Terminating (SIGTERM)
12:20:09.106928 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21467): Terminating (SIGTERM)
12:20:09.150149 run osmo-stp_10.42.42.5(pid=21468): Terminating (SIGTERM)
12:20:09.194808 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21469): Terminating (SIGTERM)
12:20:09.238422 run osmo-ggsn_10.42.42.6(pid=21474): Terminating (SIGTERM)
12:20:09.281864 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21491): Terminating (SIGTERM)
12:20:09.329969 run osmo-sgsn_10.42.42.7(pid=21492): Terminating (SIGTERM)
12:20:09.393358 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21493): Terminating (SIGTERM)
12:20:09.439174 run osmo-msc_10.42.42.8(pid=21494): Terminating (SIGTERM)
12:20:09.483613 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21495): Terminating (SIGTERM)
12:20:09.528905 run osmo-mgw_10.42.42.3(pid=21496): Terminating (SIGTERM)
12:20:09.573021 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21497): Terminating (SIGTERM)
12:20:09.616872 run osmo-mgw_10.42.42.4(pid=21498): Terminating (SIGTERM)
12:20:09.659882 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21499): Terminating (SIGTERM)
12:20:09.703535 run osmo-bsc_10.42.42.9(pid=21500): Terminating (SIGTERM)
12:20:09.746415 run osmo-bts-sysmo(pid=21513): Terminating (SIGTERM)
12:20:09.789823 run osmo-pcu(pid=21521): Terminating (SIGTERM)
12:20:09.832742 --- ParallelTerminationStrategy: PID 21465 died...
12:20:09.852867 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21465): Terminated: ok {rc=0}
12:20:09.899802 --- ParallelTerminationStrategy: PID 21466 died...
12:20:09.919482 run osmo-hlr_10.42.42.2(pid=21466): Terminated: ok {rc=0}
12:20:09.966353 --- ParallelTerminationStrategy: PID 21467 died...
12:20:09.986952 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21467): Terminated: ok {rc=0}
12:20:10.065492 --- ParallelTerminationStrategy: PID 21468 died...
12:20:10.090348 run osmo-stp_10.42.42.5(pid=21468): Terminated: ok {rc=0}
12:20:10.138451 --- ParallelTerminationStrategy: PID 21469 died...
12:20:10.176019 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21469): Terminated: ok {rc=0}
12:20:10.263991 --- ParallelTerminationStrategy: PID 21474 died...
12:20:10.284387 run osmo-ggsn_10.42.42.6(pid=21474): Terminated: ok {rc=0}
12:20:10.331494 --- ParallelTerminationStrategy: PID 21491 died...
12:20:10.351215 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21491): Terminated: ok {rc=0}
12:20:10.398076 --- ParallelTerminationStrategy: PID 21493 died...
12:20:10.418798 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21493): Terminated: ok {rc=0}
12:20:10.466452 --- ParallelTerminationStrategy: PID 21494 died...
12:20:10.487058 run osmo-msc_10.42.42.8(pid=21494): Terminated: ok {rc=0}
12:20:10.534989 --- ParallelTerminationStrategy: PID 21495 died...
12:20:10.555441 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21495): Terminated: ok {rc=0}
12:20:10.603525 --- ParallelTerminationStrategy: PID 21496 died...
12:20:10.624093 run osmo-mgw_10.42.42.3(pid=21496): Terminated {rc=15}
12:20:10.671989 --- ParallelTerminationStrategy: PID 21497 died...
12:20:10.692551 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21497): Terminated: ok {rc=0}
12:20:10.742365 --- ParallelTerminationStrategy: PID 21498 died...
12:20:10.763287 run osmo-mgw_10.42.42.4(pid=21498): Terminated {rc=15}
12:20:10.812256 --- ParallelTerminationStrategy: PID 21499 died...
12:20:10.833178 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21499): Terminated: ok {rc=0}
12:20:10.882417 --- ParallelTerminationStrategy: PID 21500 died...
12:20:10.903488 run osmo-bsc_10.42.42.9(pid=21500): Terminated: ok {rc=0}
12:20:10.952975 --- ParallelTerminationStrategy: PID 21513 died...
12:20:10.973894 run osmo-bts-sysmo(pid=21513): Terminated: ok {rc=0}
12:20:11.023390 --- ParallelTerminationStrategy: PID 21521 died...
12:20:11.044427 run osmo-pcu(pid=21521): Terminated: ok {rc=0}
12:20:11.261940 --- ParallelTerminationStrategy: PID 21492 died...
12:20:11.283597 run osmo-sgsn_10.42.42.7(pid=21492): Terminated: ok {rc=0}
12:20:11.512050 bus /gobi_6: Setting Online False
12:20:14.548554 bus /gobi_6: Setting Powered False
12:20:28.938313 bus /gobi_6: Setting Powered False
----------------------------------------------
trial-2067 gprs:sysmo ping_idle_ping.py
----------------------------------------------
12:20:30.081086 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.093109 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:20:30.129548 tst ping_idle_ping.py: using LAC 3970
12:20:30.257386 tst ping_idle_ping.py: using RAC 145
12:20:30.386056 tst ping_idle_ping.py: using CellId 3970
12:20:30.510484 tst ping_idle_ping.py: using BVCI 3971
12:20:30.627375 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.638433 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.649582 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.660686 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.671661 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.682861 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.694624 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.705681 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:20:30.823984 tst ping_idle_ping.py:24: start network...
12:20:31.041894 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:20:31.542239 run create_hlr_db(pid=21574): Launched
12:20:31.672453 bus /gobi_6: Setting Powered False
12:20:32.735954 run create_hlr_db(pid=21574): Terminated: ok {rc=0}
12:20:33.171991 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21576): Launched
12:20:33.476490 run osmo-hlr_10.42.42.2(pid=21577): Launched
12:20:33.592100 run osmo-stp_10.42.42.5: Starting osmo-stp
12:20:34.109534 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21578): Launched
12:20:34.414143 run osmo-stp_10.42.42.5(pid=21579): Launched
12:20:34.530054 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:20:35.048264 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21580): Launched
12:20:35.164846 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:20:35.421766 run patchelf(pid=21581): Launched
12:20:36.594015 run patchelf(pid=21581): Terminated: ok {rc=0}
12:20:36.720707 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:20:36.979622 run osmo-gsm-tester_setcap_net_admin.sh(pid=21582): Launched
12:20:38.158111 run osmo-gsm-tester_setcap_net_admin.sh(pid=21582): Terminated: ok {rc=0}
12:20:38.493213 run osmo-ggsn_10.42.42.6(pid=21585): Launched
12:20:38.617266 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:20:39.307287 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21607): Launched
12:20:39.626679 run osmo-sgsn_10.42.42.7(pid=21608): Launched
12:20:39.744538 run osmo-msc_10.42.42.8: Starting osmo-msc
12:20:40.300267 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21609): Launched
12:20:40.606792 run osmo-msc_10.42.42.8(pid=21610): Launched
12:20:43.779103 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:20:44.299546 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21611): Launched
12:20:44.603942 run osmo-mgw_10.42.42.3(pid=21612): Launched
12:20:44.721684 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:20:45.232688 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21613): Launched
12:20:45.537798 run osmo-mgw_10.42.42.4(pid=21614): Launched
12:20:45.655535 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:20:46.285500 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21615): Launched
12:20:46.592565 run osmo-bsc_10.42.42.9(pid=21616): Launched
12:20:52.865053 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:20:53.396222 run rm-remote-dir(pid=21617): Launched
12:20:55.579766 run rm-remote-dir(pid=21617): Terminated: ok {rc=0}
12:20:55.914142 run mk-remote-dir(pid=21618): Launched
12:20:57.094605 run mk-remote-dir(pid=21618): Terminated: ok {rc=0}
12:20:57.380853 run scp-inst-to-remote(pid=21619): Launched
12:21:09.574321 run scp-inst-to-remote(pid=21619): Terminated: ok {rc=0}
12:21:09.925369 run mk-remote-dir(pid=21621): Launched
12:21:11.113545 run mk-remote-dir(pid=21621): Terminated: ok {rc=0}
12:21:11.416498 run scp-cfg-to-remote(pid=21622): Launched
12:21:12.602596 run scp-cfg-to-remote(pid=21622): Terminated: ok {rc=0}
12:21:12.950388 run reload-dsp-firmware(pid=21624): Launched
12:21:15.138015 run reload-dsp-firmware(pid=21624): Terminated: ok {rc=0}
12:21:15.479266 run osmo-bts-sysmo(pid=21625): Launched
12:21:19.715134 tst ping_idle_ping.py:36: Waiting for bts to be ready...
12:21:21.431717 run rm-remote-dir(pid=21626): Launched
12:21:23.615812 run rm-remote-dir(pid=21626): Terminated: ok {rc=0}
12:21:23.957073 run mk-remote-dir(pid=21627): Launched
12:21:26.135534 run mk-remote-dir(pid=21627): Terminated: ok {rc=0}
12:21:26.412522 run scp-inst-to-bts(pid=21628): Launched
12:21:39.597815 run scp-inst-to-bts(pid=21628): Terminated: ok {rc=0}
12:21:39.918514 run mk-remote-run-dir(pid=21631): Launched
12:21:42.092276 run mk-remote-run-dir(pid=21631): Terminated: ok {rc=0}
12:21:42.368179 run scp-cfg-to-sysmobts(pid=21632): Launched
12:21:44.539311 run scp-cfg-to-sysmobts(pid=21632): Terminated: ok {rc=0}
12:21:44.859440 run osmo-pcu(pid=21634): Launched
12:21:44.980166 tst ping_idle_ping.py: using MSISDN 3990
12:21:45.148432 bus /gobi_6: Setting Powered True
12:21:48.669780 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3990', subscriber_id=1}
12:21:48.942361 bus /gobi_6: Setting Online False
12:21:50.128245 bus /gobi_6: Setting Powered False
12:21:57.110469 bus /gobi_6: Setting Powered True
12:21:58.507009 bus /gobi_6: Setting Online True
12:22:00.925174 tst /gobi_6: Connect to ('901', '70')
12:22:01.197183 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:22:01.310065 tst ping_idle_ping.py:47: waiting for modems to attach...
12:22:07.951012 tst /gobi_6: Already registered with network ('901', '70')
12:22:19.752410 tst ping_idle_ping.py:51: waiting for modems to attach to data services...
12:22:43.825873 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:22:43.942033 tst ping_idle_ping.py:56: Setting up data plane for "'/gobi_6/context2'"
12:22:44.387206 run move_netns(pid=21637): Launched
12:22:45.582421 run move_netns(pid=21637): Terminated: ok {rc=0}
12:22:45.933190 run ifup(pid=21640): Launched
12:22:47.128234 run ifup(pid=21640): Terminated: ok {rc=0}
12:22:47.479606 run dhcp(pid=21643): Launched
12:22:48.667342 run dhcp(pid=21643): Terminated: ok {rc=0}
12:22:48.796885 tst ping_idle_ping.py:60: [1] Running 10 ping requests for "'/gobi_6/context2'"
12:22:49.203355 run ping1(pid=21659): Launched
12:23:09.422140 run ping1(pid=21659): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪ping_idle_ping.py:61↪ping_idle_ping.py↪/gobi_6↪ping1(pid=21659)]
12:23:09.558426 run ping1(pid=21659): stdout:
| (launched: 2022-09-08_12:22:49.140188)
| 21660: waiting for 21661
| PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data.
|
| --- 10.42.42.6 ping statistics ---
| 10 packets transmitted, 0 received, 100% packet loss, time 204ms
|
| child exited with 1
12:23:09.696573 run ping1(pid=21659): stderr:
| (launched: 2022-09-08_12:22:49.187567)
12:23:09.857273 tst ping_idle_ping.py:61: ERR: Error: ping1(pid=21659): launch_sync(): local process exited with status 1 [trial-2067↪gprs:sysmo↪ping_idle_ping.py:61↪ping_idle_ping.py↪/gobi_6↪ping1(pid=21659)] [trial-2067↪gprs:sysmo↪ping_idle_ping.py:61]
12:23:09.864608 tst ping_idle_ping.py:61: Test FAILED (159.8 sec)
12:23:09.871267 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21576): Terminating (SIGTERM)
12:23:09.913750 run osmo-hlr_10.42.42.2(pid=21577): Terminating (SIGTERM)
12:23:09.956274 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21578): Terminating (SIGTERM)
12:23:10.000115 run osmo-stp_10.42.42.5(pid=21579): Terminating (SIGTERM)
12:23:10.045318 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21580): Terminating (SIGTERM)
12:23:10.110501 run osmo-ggsn_10.42.42.6(pid=21585): Terminating (SIGTERM)
12:23:10.154432 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21607): Terminating (SIGTERM)
12:23:10.253769 run osmo-sgsn_10.42.42.7(pid=21608): Terminating (SIGTERM)
12:23:10.299032 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21609): Terminating (SIGTERM)
12:23:10.342627 run osmo-msc_10.42.42.8(pid=21610): Terminating (SIGTERM)
12:23:10.386385 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21611): Terminating (SIGTERM)
12:23:10.428230 run osmo-mgw_10.42.42.3(pid=21612): Terminating (SIGTERM)
12:23:10.468910 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21613): Terminating (SIGTERM)
12:23:10.510022 run osmo-mgw_10.42.42.4(pid=21614): Terminating (SIGTERM)
12:23:10.550498 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21615): Terminating (SIGTERM)
12:23:10.591930 run osmo-bsc_10.42.42.9(pid=21616): Terminating (SIGTERM)
12:23:10.632538 run osmo-bts-sysmo(pid=21625): Terminating (SIGTERM)
12:23:10.673239 run osmo-pcu(pid=21634): Terminating (SIGTERM)
12:23:10.714777 --- ParallelTerminationStrategy: PID 21576 died...
12:23:10.734379 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21576): Terminated: ok {rc=0}
12:23:10.780653 --- ParallelTerminationStrategy: PID 21577 died...
12:23:10.800847 run osmo-hlr_10.42.42.2(pid=21577): Terminated: ok {rc=0}
12:23:10.848725 --- ParallelTerminationStrategy: PID 21578 died...
12:23:10.868390 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21578): Terminated: ok {rc=0}
12:23:10.953170 --- ParallelTerminationStrategy: PID 21579 died...
12:23:11.007010 run osmo-stp_10.42.42.5(pid=21579): Terminated: ok {rc=0}
12:23:11.056155 --- ParallelTerminationStrategy: PID 21580 died...
12:23:11.076202 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21580): Terminated: ok {rc=0}
12:23:11.124568 --- ParallelTerminationStrategy: PID 21585 died...
12:23:11.144531 run osmo-ggsn_10.42.42.6(pid=21585): Terminated: ok {rc=0}
12:23:11.192785 --- ParallelTerminationStrategy: PID 21607 died...
12:23:11.212812 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21607): Terminated: ok {rc=0}
12:23:11.261160 --- ParallelTerminationStrategy: PID 21609 died...
12:23:11.281434 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21609): Terminated: ok {rc=0}
12:23:11.330462 --- ParallelTerminationStrategy: PID 21610 died...
12:23:11.351111 run osmo-msc_10.42.42.8(pid=21610): Terminated: ok {rc=0}
12:23:11.400254 --- ParallelTerminationStrategy: PID 21611 died...
12:23:11.420982 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21611): Terminated: ok {rc=0}
12:23:11.470480 --- ParallelTerminationStrategy: PID 21612 died...
12:23:11.491797 run osmo-mgw_10.42.42.3(pid=21612): Terminated {rc=15}
12:23:11.541150 --- ParallelTerminationStrategy: PID 21613 died...
12:23:11.562143 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21613): Terminated: ok {rc=0}
12:23:11.611798 --- ParallelTerminationStrategy: PID 21614 died...
12:23:11.632611 run osmo-mgw_10.42.42.4(pid=21614): Terminated {rc=15}
12:23:11.681993 --- ParallelTerminationStrategy: PID 21615 died...
12:23:11.703206 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21615): Terminated: ok {rc=0}
12:23:11.752869 --- ParallelTerminationStrategy: PID 21616 died...
12:23:11.773691 run osmo-bsc_10.42.42.9(pid=21616): Terminated: ok {rc=0}
12:23:11.823365 --- ParallelTerminationStrategy: PID 21625 died...
12:23:11.844207 run osmo-bts-sysmo(pid=21625): Terminated: ok {rc=0}
12:23:11.894032 --- ParallelTerminationStrategy: PID 21634 died...
12:23:11.915048 run osmo-pcu(pid=21634): Terminated: ok {rc=0}
12:23:12.132197 --- ParallelTerminationStrategy: PID 21608 died...
12:23:12.154472 run osmo-sgsn_10.42.42.7(pid=21608): Terminated: ok {rc=0}
12:23:12.379481 bus /gobi_6: Setting Online False
12:23:16.305442 bus /gobi_6: Setting Powered False
12:23:33.098188 bus /gobi_6: Setting Powered False
---------------------------------------------------------------------
trial-2067 gprs:sysmo FAIL
---------------------------------------------------------------------
12:23:34.439390 tst trial-2067: Storing JUnit report in /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2067/run.2022-09-08_12-00-20/trial-2067.xml
--------------------------------------------------------------------------------------------
trial-2067 FAIL
--------------------------------------------------------------------------------------------
12:23:34.658849 tst trial-2067: trial-2067: FAIL, 2 suites failed, 1 suites passed
FAIL: nitb_sms:sysmo (fail: 1)
FAIL: mo_mt_sms.py (4.8 sec) Error: osmo-nitb_10.42.42.2(pid=20869): Process ended prematurely: osmo-nitb_10.42.42.2(pid=20869) [trial-2067↪nitb_sms:sysmo↪mo_mt_sms.py:11↪mo_mt_sms.py↪osmo-nitb_10.42.42.2↪osmo-nitb_10.42.42.2(pid=20869)]
FAIL: gprs:sysmo (fail: 4, pass: 1)
FAIL: cs_paging_gprs_active.py (224.6 sec) KeyError: 'sum_sent'
FAIL: iperf3.py (201.9 sec) KeyError: 'sum_sent'
FAIL: iperf3m4.py (296.9 sec) KeyError: 'sum_sent'
pass: ping.py (146.8 sec)
FAIL: ping_idle_ping.py (159.8 sec) Error: ping1(pid=21659): launch_sync(): local process exited with status 1 [trial-2067↪gprs:sysmo↪ping_idle_ping.py:61↪ping_idle_ping.py↪/gobi_6↪ping1(pid=21659)]
PASS: sms:sysmo (pass: 1)
pass: mo_mt_sms.py (113.7 sec)
+ rm -rf trial-2067/inst
+ cd trial-2067
+ command -v journalctl
+ [ ! -z /bin/journalctl ]
+ readlink last_run
+ journalctl -u ofono -o short-precise --since 2022-09-08 12:00:17
+ readlink last_run
+ tar czf /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2067-run.tgz run.2022-09-08_12-00-20
+ tar czf /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2067-bin.tgz checksums.md5 open5gs.build-8.tgz osmo-bsc.build-16895.tgz osmo-bts-oc2g.build-5192.tgz osmo-bts-sysmo.build-7696.tgz osmo-bts.build-7170.tgz osmo-ggsn.build-6368.tgz osmo-hlr.build-8266.tgz osmo-hnbgw.build-2029.tgz osmo-mgw.build-8474.tgz osmo-msc.build-28089.tgz osmo-nitb.build-12352.tgz osmo-pcu-oc2g.build-3692.tgz osmo-pcu-sysmo.build-4973.tgz osmo-pcu.build-4935.tgz osmo-sgsn.build-28698.tgz osmo-stp.build-8803.tgz osmo-trx.build-4363.tgz osmocom-bb.build-4444.tgz srslte.build-19.tgz
+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
ERROR Gerrit response: ConflictFinished: FAILURE
![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
