Skip to content

Failed

gprs:sysmo.iperf3m4.py (from gprs_sysmo)

Failing for the past 5 builds (Since Failed#2063 )
Took 4 min 57 sec.

Stacktrace

'sum_sent'

Standard Output

----------------------------------------------
trial-2067 gprs:sysmo iperf3m4.py
----------------------------------------------
12:11:28.384621 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.396039 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
12:11:28.519635 tst                      iperf3m4.py: using LAC 3968
12:11:28.638268 tst                      iperf3m4.py: using RAC 143
12:11:28.757337 tst                      iperf3m4.py: using CellId 3968
12:11:28.877363 tst                      iperf3m4.py: using BVCI 3969
12:11:28.901006 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.911734 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.922233 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.932780 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.943428 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.953824 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.964558 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.975079 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.985667 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.104475 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.225655 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.343283 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.636006 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
12:11:29.772812 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:30.112555 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21202): Launched
12:11:30.395136 run iperf3-srv_10.42.42.10(pid=21203): Launched
12:11:30.594876 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
12:11:30.731313 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:31.070373 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21204): Launched
12:11:31.353300 run iperf3-srv_10.42.42.10(pid=21205): Launched
12:11:31.551804 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
12:11:31.687552 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:32.026211 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21206): Launched
12:11:32.308308 run iperf3-srv_10.42.42.10(pid=21207): Launched
12:11:32.507044 tst                    iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
12:11:32.642921 run           iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:32.981362 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21208): Launched
12:11:33.262119 run iperf3-srv_10.42.42.10(pid=21209): Launched
12:11:33.460123 tst                    iperf3m4.py:8: start network...
12:11:33.595416 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
12:11:34.119829 run         create_hlr_db(pid=21210): Launched
12:11:34.307885 bus                          /gobi_6: Setting Powered False
12:11:35.330104 run         create_hlr_db(pid=21210): Terminated: ok {rc=0}
12:11:35.741645 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21212): Launched
12:11:36.050013 run   osmo-hlr_10.42.42.2(pid=21213): Launched
12:11:36.159546 run              osmo-stp_10.42.42.5: Starting osmo-stp
12:11:36.692023 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21214): Launched
12:11:37.001427 run   osmo-stp_10.42.42.5(pid=21215): Launched
12:11:37.109491 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:11:37.642130 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21216): Launched
12:11:37.750184 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:11:38.016156 run              patchelf(pid=21217): Launched
12:11:39.198578 run              patchelf(pid=21217): Terminated: ok {rc=0}
12:11:39.310350 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:11:39.587049 run osmo-gsm-tester_setcap_net_admin.sh(pid=21218): Launched
12:11:40.768594 run osmo-gsm-tester_setcap_net_admin.sh(pid=21218): Terminated: ok {rc=0}
12:11:41.079368 run  osmo-ggsn_10.42.42.6(pid=21221): Launched
12:11:41.312334 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:11:41.873770 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21238): Launched
12:11:42.180746 run  osmo-sgsn_10.42.42.7(pid=21239): Launched
12:11:42.293902 run              osmo-msc_10.42.42.8: Starting osmo-msc
12:11:42.856918 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21240): Launched
12:11:43.166389 run   osmo-msc_10.42.42.8(pid=21241): Launched
12:11:46.328458 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
12:11:46.876278 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21242): Launched
12:11:47.183575 run   osmo-mgw_10.42.42.3(pid=21243): Launched
12:11:47.291398 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
12:11:47.812179 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21244): Launched
12:11:48.120509 run   osmo-mgw_10.42.42.4(pid=21245): Launched
12:11:48.228230 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
12:11:48.869239 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21246): Launched
12:11:49.178072 run   osmo-bsc_10.42.42.9(pid=21247): Launched
12:11:55.437998 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:11:55.984198 run         rm-remote-dir(pid=21248): Launched
12:11:58.175327 run         rm-remote-dir(pid=21248): Terminated: ok {rc=0}
12:11:58.502877 run         mk-remote-dir(pid=21249): Launched
12:11:59.690314 run         mk-remote-dir(pid=21249): Terminated: ok {rc=0}
12:11:59.968047 run    scp-inst-to-remote(pid=21250): Launched
12:12:12.181031 run    scp-inst-to-remote(pid=21250): Terminated: ok {rc=0}
12:12:12.527308 run         mk-remote-dir(pid=21252): Launched
12:12:13.725142 run         mk-remote-dir(pid=21252): Terminated: ok {rc=0}
12:12:14.019130 run     scp-cfg-to-remote(pid=21253): Launched
12:12:15.216796 run     scp-cfg-to-remote(pid=21253): Terminated: ok {rc=0}
12:12:15.557714 run   reload-dsp-firmware(pid=21255): Launched
12:12:17.754583 run   reload-dsp-firmware(pid=21255): Terminated: ok {rc=0}
12:12:18.084912 run        osmo-bts-sysmo(pid=21256): Launched
12:12:22.399990 tst                    iperf3m4.py:8: Waiting for bts to be ready...
12:12:24.045193 run         rm-remote-dir(pid=21257): Launched
12:12:26.242062 run         rm-remote-dir(pid=21257): Terminated: ok {rc=0}
12:12:26.577295 run         mk-remote-dir(pid=21258): Launched
12:12:28.769683 run         mk-remote-dir(pid=21258): Terminated: ok {rc=0}
12:12:29.058192 run       scp-inst-to-bts(pid=21259): Launched
12:12:42.263798 run       scp-inst-to-bts(pid=21259): Terminated: ok {rc=0}
12:12:42.579493 run     mk-remote-run-dir(pid=21261): Launched
12:12:44.762330 run     mk-remote-run-dir(pid=21261): Terminated: ok {rc=0}
12:12:45.034081 run   scp-cfg-to-sysmobts(pid=21262): Launched
12:12:47.216206 run   scp-cfg-to-sysmobts(pid=21262): Terminated: ok {rc=0}
12:12:47.531375 run              osmo-pcu(pid=21264): Launched
12:12:47.659298 tst                      iperf3m4.py: using MSISDN 3985
12:12:47.831240 bus                          /gobi_6: Setting Powered True
12:12:51.335436 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3985', subscriber_id=1}
12:12:51.607981 bus                          /gobi_6: Setting Online False
12:12:52.802368 bus                          /gobi_6: Setting Powered False
12:13:00.203440 bus                          /gobi_6: Setting Powered True
12:13:01.565566 bus                          /gobi_6: Setting Online True
12:13:03.999762 tst                          /gobi_6: Connect to ('901', '70')
12:13:04.276040 tst                          /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G  1  [Jul  5 2017 10:49:17]', Serial: '868323024273648'
12:13:04.408913 tst                      iperf3m4.py: using MSISDN 3986
12:13:04.679627 bus                          /gobi_0: Setting Powered True
12:13:08.134264 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3986', subscriber_id=2}
12:13:08.419310 bus                          /gobi_0: Setting Online False
12:13:09.613159 bus                          /gobi_0: Setting Powered False
12:13:16.389001 bus                          /gobi_0: Setting Powered True
12:13:17.764557 bus                          /gobi_0: Setting Online True
12:13:19.586365 tst                          /gobi_6: Already registered with network ('901', '70')
12:13:20.320389 tst                          /gobi_0: Connect to ('901', '70')
12:13:20.602968 tst                          /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G  1  [Mar 14 2016 19:33:01]', Serial: '868323024271873'
12:13:20.734532 tst                      iperf3m4.py: using MSISDN 3987
12:13:20.964104 bus                          /gobi_1: Setting Powered True
12:13:24.464786 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023848', msisdn='3987', subscriber_id=3}
12:13:24.738010 bus                          /gobi_1: Setting Online False
12:13:25.923743 bus                          /gobi_1: Setting Powered False
12:13:33.224246 bus                          /gobi_1: Setting Powered True
12:13:34.597021 bus                          /gobi_1: Setting Online True
12:13:35.662311 tst                          /gobi_0: Already registered with network ('901', '70')
12:13:37.277102 tst                          /gobi_1: Connect to ('901', '70')
12:13:37.544259 tst                          /gobi_1: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G  1  [Mar 14 2016 19:33:01]', Serial: '868323024272103'
12:13:37.667139 tst                      iperf3m4.py: using MSISDN 3988
12:13:37.835635 bus                          /gobi_4: Setting Powered True
12:13:41.490774 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023849', msisdn='3988', subscriber_id=4}
12:13:41.765016 bus                          /gobi_4: Setting Online False
12:13:42.959797 bus                          /gobi_4: Setting Powered False
12:13:50.226870 bus                          /gobi_4: Setting Powered True
12:13:51.612541 bus                          /gobi_4: Setting Online True
12:13:54.094963 tst                          /gobi_4: Connect to ('901', '70')
12:13:54.382104 tst                          /gobi_4: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A08E2G  1  [Jul 21 2016 10:00:50]', Serial: '868323025161511'
12:13:54.581523 tst                    iperf3m4.py:8: waiting for modems to attach...
12:13:55.139563 tst                          /gobi_1: Already registered with network ('901', '70')
12:14:10.059830 tst                          /gobi_4: Already registered with network ('901', '70')
12:14:15.201781 tst                    iperf3m4.py:8: waiting for modems to attach to data services...
12:14:18.241199 tst                          /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:18.441883 tst                    iperf3m4.py:8: Setting up data plan for "'/gobi_6/context2'"
12:14:18.818467 run            move_netns(pid=21268): Launched
12:14:20.013709 run            move_netns(pid=21268): Terminated: ok {rc=0}
12:14:20.366648 run                  ifup(pid=21271): Launched
12:14:21.550188 run                  ifup(pid=21271): Terminated: ok {rc=0}
12:14:21.885075 run                  dhcp(pid=21274): Launched
12:14:23.080306 run                  dhcp(pid=21274): Terminated: ok {rc=0}
12:14:26.211202 tst                          /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:26.399843 tst                    iperf3m4.py:8: Setting up data plan for "'/gobi_0/context2'"
12:14:26.763701 run            move_netns(pid=21292): Launched
12:14:27.958439 run            move_netns(pid=21292): Terminated: ok {rc=0}
12:14:28.303717 run                  ifup(pid=21295): Launched
12:14:29.499821 run                  ifup(pid=21295): Terminated: ok {rc=0}
12:14:29.847564 run                  dhcp(pid=21298): Launched
12:14:31.043002 run                  dhcp(pid=21298): Terminated: ok {rc=0}
12:14:39.430561 tst                          /gobi_1: context activated {apn='inet46', path='/gobi_1/context12', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.3', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='rf303c5eb721dad', Method='static', Netmask='255.255.255.248'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:39.630190 tst                    iperf3m4.py:8: Setting up data plan for "'/gobi_1/context12'"
12:14:40.008769 run            move_netns(pid=21316): Launched
12:14:41.203594 run            move_netns(pid=21316): Terminated: ok {rc=0}
12:14:41.555073 run                  ifup(pid=21319): Launched
12:14:42.750593 run                  ifup(pid=21319): Terminated: ok {rc=0}
12:14:43.094092 run                  dhcp(pid=21322): Launched
12:14:44.288607 run                  dhcp(pid=21322): Terminated: ok {rc=0}
12:14:47.398290 tst                          /gobi_4: context activated {apn='inet46', path='/gobi_4/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.4', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.5', Interface='r9d788e55295d1b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:47.598515 tst                    iperf3m4.py:8: Setting up data plan for "'/gobi_4/context2'"
12:14:47.972558 run            move_netns(pid=21344): Launched
12:14:49.167644 run            move_netns(pid=21344): Terminated: ok {rc=0}
12:14:49.511099 run                  ifup(pid=21347): Launched
12:14:50.695102 run                  ifup(pid=21347): Terminated: ok {rc=0}
12:14:51.022092 run                  dhcp(pid=21350): Launched
12:14:52.204665 run                  dhcp(pid=21350): Terminated: ok {rc=0}
12:14:52.398498 tst                    iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'"
12:14:52.528723 run           iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:14:52.914061 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Launched
12:14:53.168421 tst                    iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'"
12:14:53.307840 run           iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s)
12:14:53.709624 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Launched
12:14:53.962766 tst                    iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5005 through "'/gobi_1/context12'"
12:14:54.102251 run           iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5005 (proto=tcp,time=60s)
12:14:54.505434 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Launched
12:14:54.758418 tst                    iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5006 through "'/gobi_4/context2'"
12:14:54.897262 run           iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5006 (proto=tcp,time=60s)
12:14:55.297856 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Launched
12:14:55.633982 run iperf3-cli_10.42.42.10(pid=21378): Launched
12:14:55.918859 run iperf3-cli_10.42.42.10(pid=21381): Launched
12:14:56.208202 run iperf3-cli_10.42.42.10(pid=21384): Launched
12:14:56.498429 run iperf3-cli_10.42.42.10(pid=21387): Launched
12:16:19.844408 run iperf3-cli_10.42.42.10(pid=21378): ERR: Terminated: ERROR {rc=1}  [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21378)]
12:16:19.980002 run iperf3-cli_10.42.42.10(pid=21378): stdout: 
| 				"bits_per_second":	0,
| 				"retransmits":	0,
| 				"omitted":	false
| 			}
| 		}],
| 	"end":	{
| 	},
| 	"error":	"error - control socket has closed unexpectedly"
| }
| child exited with 1 
12:16:20.113772 run iperf3-cli_10.42.42.10(pid=21378): stderr: 
| (launched: 2022-09-08_12:14:55.515951) 
12:16:21.302189 run iperf3-cli_10.42.42.10(pid=21381): ERR: Terminated: ERROR {rc=1}  [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21381)]
12:16:21.436576 run iperf3-cli_10.42.42.10(pid=21381): stdout: 
| 				"bits_per_second":	0,
| 				"retransmits":	0,
| 				"omitted":	false
| 			}
| 		}],
| 	"end":	{
| 	},
| 	"error":	"error - control socket has closed unexpectedly"
| }
| child exited with 1 
12:16:21.570684 run iperf3-cli_10.42.42.10(pid=21381): stderr: 
| (launched: 2022-09-08_12:14:55.801521) 
12:16:22.767473 run iperf3-cli_10.42.42.10(pid=21384): ERR: Terminated: ERROR {rc=1}  [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21384)]
12:16:22.904946 run iperf3-cli_10.42.42.10(pid=21384): stdout: 
| 				"bits_per_second":	0,
| 				"retransmits":	0,
| 				"omitted":	false
| 			}
| 		}],
| 	"end":	{
| 	},
| 	"error":	"error - control socket has closed unexpectedly"
| }
| child exited with 1 
12:16:23.041323 run iperf3-cli_10.42.42.10(pid=21384): stderr: 
| (launched: 2022-09-08_12:14:56.087095) 
12:16:24.237555 run iperf3-cli_10.42.42.10(pid=21387): ERR: Terminated: ERROR {rc=1}  [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21387)]
12:16:24.372865 run iperf3-cli_10.42.42.10(pid=21387): stdout: 
| 				"bits_per_second":	0,
| 				"retransmits":	0,
| 				"omitted":	false
| 			}
| 		}],
| 	"end":	{
| 	},
| 	"error":	"error - control socket has closed unexpectedly"
| }
| child exited with 1 
12:16:24.506316 run iperf3-cli_10.42.42.10(pid=21387): stderr: 
| (launched: 2022-09-08_12:14:56.377632) 
12:16:24.628427 run iperf3-srv_10.42.42.10(pid=21203): Terminating (SIGINT)
12:16:24.966664 run iperf3-srv_10.42.42.10(pid=21203): Terminated {rc=1}
12:16:25.166682 tst                    iperf3m4.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context2'"
12:16:25.225067 tst                    iperf3m4.py:8: ERR: KeyError: 'sum_sent'  [trial-2067↪gprs:sysmo↪iperf3m4.py:8]
12:16:25.232006 tst                    iperf3m4.py:8: Test FAILED (296.9 sec)
12:16:25.269516 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21202): Terminating (SIGTERM)
12:16:25.311985 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21204): Terminating (SIGTERM)
12:16:25.353839 run iperf3-srv_10.42.42.10(pid=21205): Terminating (SIGTERM)
12:16:25.396412 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21206): Terminating (SIGTERM)
12:16:25.438433 run iperf3-srv_10.42.42.10(pid=21207): Terminating (SIGTERM)
12:16:25.480936 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21208): Terminating (SIGTERM)
12:16:25.523604 run iperf3-srv_10.42.42.10(pid=21209): Terminating (SIGTERM)
12:16:25.565657 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21212): Terminating (SIGTERM)
12:16:25.608160 run   osmo-hlr_10.42.42.2(pid=21213): Terminating (SIGTERM)
12:16:25.651464 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21214): Terminating (SIGTERM)
12:16:25.694859 run   osmo-stp_10.42.42.5(pid=21215): Terminating (SIGTERM)
12:16:25.757834 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21216): Terminating (SIGTERM)
12:16:25.801100 run  osmo-ggsn_10.42.42.6(pid=21221): Terminating (SIGTERM)
12:16:25.864673 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21238): Terminating (SIGTERM)
12:16:25.928722 run  osmo-sgsn_10.42.42.7(pid=21239): Terminating (SIGTERM)
12:16:25.991089 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21240): Terminating (SIGTERM)
12:16:26.052432 run   osmo-msc_10.42.42.8(pid=21241): Terminating (SIGTERM)
12:16:26.099766 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21242): Terminating (SIGTERM)
12:16:26.143596 run   osmo-mgw_10.42.42.3(pid=21243): Terminating (SIGTERM)
12:16:26.187521 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21244): Terminating (SIGTERM)
12:16:26.231474 run   osmo-mgw_10.42.42.4(pid=21245): Terminating (SIGTERM)
12:16:26.274626 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21246): Terminating (SIGTERM)
12:16:26.322089 run   osmo-bsc_10.42.42.9(pid=21247): Terminating (SIGTERM)
12:16:26.367138 run        osmo-bts-sysmo(pid=21256): Terminating (SIGTERM)
12:16:26.412659 run              osmo-pcu(pid=21264): Terminating (SIGTERM)
12:16:26.455494 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Terminating (SIGTERM)
12:16:26.612839 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGTERM(pid=21422): Launched
12:16:27.704877 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGTERM(pid=21422): Terminated: ok {rc=0}
12:16:27.751075 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Terminating (SIGTERM)
12:16:27.904689 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGTERM(pid=21427): Launched
12:16:28.999863 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGTERM(pid=21427): Terminated: ok {rc=0}
12:16:29.047928 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Terminating (SIGTERM)
12:16:29.208359 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGTERM(pid=21430): Launched
12:16:30.303402 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGTERM(pid=21430): Terminated: ok {rc=0}
12:16:30.351861 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Terminating (SIGTERM)
12:16:30.512537 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGTERM(pid=21433): Launched
12:16:31.608861 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGTERM(pid=21433): Terminated: ok {rc=0}
12:16:31.623397 ---      ParallelTerminationStrategy: PID 21202 died...
12:16:31.680652 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21202): Terminated: ok {rc=0}
12:16:31.690827 ---      ParallelTerminationStrategy: PID 21204 died...
12:16:31.747549 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21204): Terminated: ok {rc=0}
12:16:31.757507 ---      ParallelTerminationStrategy: PID 21205 died...
12:16:31.814036 run iperf3-srv_10.42.42.10(pid=21205): Terminated {rc=256}
12:16:31.824276 ---      ParallelTerminationStrategy: PID 21206 died...
12:16:31.880538 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21206): Terminated: ok {rc=0}
12:16:31.890929 ---      ParallelTerminationStrategy: PID 21207 died...
12:16:31.947411 run iperf3-srv_10.42.42.10(pid=21207): Terminated {rc=256}
12:16:31.957615 ---      ParallelTerminationStrategy: PID 21208 died...
12:16:32.014299 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21208): Terminated: ok {rc=0}
12:16:32.024316 ---      ParallelTerminationStrategy: PID 21209 died...
12:16:32.080791 run iperf3-srv_10.42.42.10(pid=21209): Terminated {rc=256}
12:16:32.090808 ---      ParallelTerminationStrategy: PID 21212 died...
12:16:32.147348 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21212): Terminated: ok {rc=0}
12:16:32.157352 ---      ParallelTerminationStrategy: PID 21213 died...
12:16:32.214347 run   osmo-hlr_10.42.42.2(pid=21213): Terminated: ok {rc=0}
12:16:32.224388 ---      ParallelTerminationStrategy: PID 21214 died...
12:16:32.280732 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21214): Terminated: ok {rc=0}
12:16:32.291036 ---      ParallelTerminationStrategy: PID 21215 died...
12:16:32.347530 run   osmo-stp_10.42.42.5(pid=21215): Terminated: ok {rc=0}
12:16:32.357654 ---      ParallelTerminationStrategy: PID 21216 died...
12:16:32.414318 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21216): Terminated: ok {rc=0}
12:16:32.424561 ---      ParallelTerminationStrategy: PID 21221 died...
12:16:32.481054 run  osmo-ggsn_10.42.42.6(pid=21221): Terminated: ok {rc=0}
12:16:32.491414 ---      ParallelTerminationStrategy: PID 21238 died...
12:16:32.548260 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21238): Terminated: ok {rc=0}
12:16:32.558281 ---      ParallelTerminationStrategy: PID 21239 died...
12:16:32.615021 run  osmo-sgsn_10.42.42.7(pid=21239): Terminated: ok {rc=0}
12:16:32.625215 ---      ParallelTerminationStrategy: PID 21240 died...
12:16:32.682156 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21240): Terminated: ok {rc=0}
12:16:32.692263 ---      ParallelTerminationStrategy: PID 21241 died...
12:16:32.748864 run   osmo-msc_10.42.42.8(pid=21241): Terminated: ok {rc=0}
12:16:32.759004 ---      ParallelTerminationStrategy: PID 21242 died...
12:16:32.815666 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21242): Terminated: ok {rc=0}
12:16:32.825733 ---      ParallelTerminationStrategy: PID 21243 died...
12:16:32.882639 run   osmo-mgw_10.42.42.3(pid=21243): Terminated {rc=15}
12:16:32.892716 ---      ParallelTerminationStrategy: PID 21244 died...
12:16:32.949488 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21244): Terminated: ok {rc=0}
12:16:32.959638 ---      ParallelTerminationStrategy: PID 21245 died...
12:16:33.016987 run   osmo-mgw_10.42.42.4(pid=21245): Terminated {rc=15}
12:16:33.027078 ---      ParallelTerminationStrategy: PID 21246 died...
12:16:33.083914 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21246): Terminated: ok {rc=0}
12:16:33.094256 ---      ParallelTerminationStrategy: PID 21247 died...
12:16:33.151037 run   osmo-bsc_10.42.42.9(pid=21247): Terminated: ok {rc=0}
12:16:33.161108 ---      ParallelTerminationStrategy: PID 21256 died...
12:16:33.217850 run        osmo-bts-sysmo(pid=21256): Terminated: ok {rc=0}
12:16:33.228025 ---      ParallelTerminationStrategy: PID 21264 died...
12:16:33.285572 run              osmo-pcu(pid=21264): Terminated: ok {rc=0}
12:16:39.154267 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Terminating (SIGINT)
12:16:39.309667 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGINT(pid=21436): Launched
12:16:40.404915 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGINT(pid=21436): Terminated: ok {rc=0}
12:16:40.453447 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Terminating (SIGINT)
12:16:40.613845 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGINT(pid=21439): Launched
12:16:41.708654 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGINT(pid=21439): Terminated: ok {rc=0}
12:16:41.757246 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Terminating (SIGINT)
12:16:41.916333 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGINT(pid=21442): Launched
12:16:43.011078 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGINT(pid=21442): Terminated: ok {rc=0}
12:16:43.059868 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Terminating (SIGINT)
12:16:43.215292 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGINT(pid=21445): Launched
12:16:44.305955 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGINT(pid=21445): Terminated: ok {rc=0}
12:16:50.176527 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Terminating (SIGKILL)
12:16:50.342896 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGUSR1(pid=21448): Launched
12:16:51.434046 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGUSR1(pid=21448): Terminated: ok {rc=0}
12:16:51.480013 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Terminating (SIGKILL)
12:16:51.632717 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGUSR1(pid=21451): Launched
12:16:52.724058 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGUSR1(pid=21451): Terminated: ok {rc=0}
12:16:52.769859 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Terminating (SIGKILL)
12:16:52.922247 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGUSR1(pid=21454): Launched
12:16:54.012502 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGUSR1(pid=21454): Terminated: ok {rc=0}
12:16:54.058627 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Terminating (SIGKILL)
12:16:54.209302 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGUSR1(pid=21457): Launched
12:16:55.300985 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGUSR1(pid=21457): Terminated: ok {rc=0}
12:16:55.539496 bus                          /gobi_4: Setting Online False
12:16:56.677312 bus                          /gobi_4: Setting Powered False
12:17:06.977899 bus                          /gobi_4: Setting Powered False
12:17:08.085825 bus                          /gobi_4: Setting Powered False
12:17:09.238187 bus                          /gobi_1: Setting Online False
12:17:10.373666 bus                          /gobi_1: Setting Powered False
12:17:18.084487 bus                          /gobi_1: Setting Powered False
12:17:19.190688 bus                          /gobi_1: Setting Powered False
12:17:20.342609 bus                          /gobi_0: Setting Online False
12:17:21.478233 bus                          /gobi_0: Setting Powered False
12:17:29.976993 bus                          /gobi_0: Setting Powered False
12:17:31.083720 bus                          /gobi_0: Setting Powered False
12:17:32.235462 bus                          /gobi_6: Setting Online False
12:17:33.348859 bus                          /gobi_6: Setting Powered False
12:17:41.052366 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
    setup_run_iperf3_test_parallel(4)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
    print_results(clients[i], servers[i])
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
    cli.print_results()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
    print_result_node_tcp(self.get_results(), 'client')
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
    sent = result['end']['sum_sent']
KeyError: 'sum_sent'