![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3m4.py (from gprs_sysmo)
Failing for the past 5 builds
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2067 gprs:sysmo iperf3m4.py
----------------------------------------------
12:11:28.384621 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.396039 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:11:28.519635 tst iperf3m4.py: using LAC 3968
12:11:28.638268 tst iperf3m4.py: using RAC 143
12:11:28.757337 tst iperf3m4.py: using CellId 3968
12:11:28.877363 tst iperf3m4.py: using BVCI 3969
12:11:28.901006 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.911734 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.922233 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.932780 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.943428 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.953824 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.964558 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.975079 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:11:28.985667 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.104475 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.225655 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.343283 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:11:29.636006 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
12:11:29.772812 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:30.112555 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21202): Launched
12:11:30.395136 run iperf3-srv_10.42.42.10(pid=21203): Launched
12:11:30.594876 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
12:11:30.731313 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:31.070373 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21204): Launched
12:11:31.353300 run iperf3-srv_10.42.42.10(pid=21205): Launched
12:11:31.551804 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
12:11:31.687552 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:32.026211 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21206): Launched
12:11:32.308308 run iperf3-srv_10.42.42.10(pid=21207): Launched
12:11:32.507044 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
12:11:32.642921 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:11:32.981362 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21208): Launched
12:11:33.262119 run iperf3-srv_10.42.42.10(pid=21209): Launched
12:11:33.460123 tst iperf3m4.py:8: start network...
12:11:33.595416 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:11:34.119829 run create_hlr_db(pid=21210): Launched
12:11:34.307885 bus /gobi_6: Setting Powered False
12:11:35.330104 run create_hlr_db(pid=21210): Terminated: ok {rc=0}
12:11:35.741645 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21212): Launched
12:11:36.050013 run osmo-hlr_10.42.42.2(pid=21213): Launched
12:11:36.159546 run osmo-stp_10.42.42.5: Starting osmo-stp
12:11:36.692023 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21214): Launched
12:11:37.001427 run osmo-stp_10.42.42.5(pid=21215): Launched
12:11:37.109491 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:11:37.642130 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21216): Launched
12:11:37.750184 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:11:38.016156 run patchelf(pid=21217): Launched
12:11:39.198578 run patchelf(pid=21217): Terminated: ok {rc=0}
12:11:39.310350 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:11:39.587049 run osmo-gsm-tester_setcap_net_admin.sh(pid=21218): Launched
12:11:40.768594 run osmo-gsm-tester_setcap_net_admin.sh(pid=21218): Terminated: ok {rc=0}
12:11:41.079368 run osmo-ggsn_10.42.42.6(pid=21221): Launched
12:11:41.312334 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:11:41.873770 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21238): Launched
12:11:42.180746 run osmo-sgsn_10.42.42.7(pid=21239): Launched
12:11:42.293902 run osmo-msc_10.42.42.8: Starting osmo-msc
12:11:42.856918 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21240): Launched
12:11:43.166389 run osmo-msc_10.42.42.8(pid=21241): Launched
12:11:46.328458 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:11:46.876278 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21242): Launched
12:11:47.183575 run osmo-mgw_10.42.42.3(pid=21243): Launched
12:11:47.291398 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:11:47.812179 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21244): Launched
12:11:48.120509 run osmo-mgw_10.42.42.4(pid=21245): Launched
12:11:48.228230 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:11:48.869239 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21246): Launched
12:11:49.178072 run osmo-bsc_10.42.42.9(pid=21247): Launched
12:11:55.437998 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:11:55.984198 run rm-remote-dir(pid=21248): Launched
12:11:58.175327 run rm-remote-dir(pid=21248): Terminated: ok {rc=0}
12:11:58.502877 run mk-remote-dir(pid=21249): Launched
12:11:59.690314 run mk-remote-dir(pid=21249): Terminated: ok {rc=0}
12:11:59.968047 run scp-inst-to-remote(pid=21250): Launched
12:12:12.181031 run scp-inst-to-remote(pid=21250): Terminated: ok {rc=0}
12:12:12.527308 run mk-remote-dir(pid=21252): Launched
12:12:13.725142 run mk-remote-dir(pid=21252): Terminated: ok {rc=0}
12:12:14.019130 run scp-cfg-to-remote(pid=21253): Launched
12:12:15.216796 run scp-cfg-to-remote(pid=21253): Terminated: ok {rc=0}
12:12:15.557714 run reload-dsp-firmware(pid=21255): Launched
12:12:17.754583 run reload-dsp-firmware(pid=21255): Terminated: ok {rc=0}
12:12:18.084912 run osmo-bts-sysmo(pid=21256): Launched
12:12:22.399990 tst iperf3m4.py:8: Waiting for bts to be ready...
12:12:24.045193 run rm-remote-dir(pid=21257): Launched
12:12:26.242062 run rm-remote-dir(pid=21257): Terminated: ok {rc=0}
12:12:26.577295 run mk-remote-dir(pid=21258): Launched
12:12:28.769683 run mk-remote-dir(pid=21258): Terminated: ok {rc=0}
12:12:29.058192 run scp-inst-to-bts(pid=21259): Launched
12:12:42.263798 run scp-inst-to-bts(pid=21259): Terminated: ok {rc=0}
12:12:42.579493 run mk-remote-run-dir(pid=21261): Launched
12:12:44.762330 run mk-remote-run-dir(pid=21261): Terminated: ok {rc=0}
12:12:45.034081 run scp-cfg-to-sysmobts(pid=21262): Launched
12:12:47.216206 run scp-cfg-to-sysmobts(pid=21262): Terminated: ok {rc=0}
12:12:47.531375 run osmo-pcu(pid=21264): Launched
12:12:47.659298 tst iperf3m4.py: using MSISDN 3985
12:12:47.831240 bus /gobi_6: Setting Powered True
12:12:51.335436 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3985', subscriber_id=1}
12:12:51.607981 bus /gobi_6: Setting Online False
12:12:52.802368 bus /gobi_6: Setting Powered False
12:13:00.203440 bus /gobi_6: Setting Powered True
12:13:01.565566 bus /gobi_6: Setting Online True
12:13:03.999762 tst /gobi_6: Connect to ('901', '70')
12:13:04.276040 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:13:04.408913 tst iperf3m4.py: using MSISDN 3986
12:13:04.679627 bus /gobi_0: Setting Powered True
12:13:08.134264 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3986', subscriber_id=2}
12:13:08.419310 bus /gobi_0: Setting Online False
12:13:09.613159 bus /gobi_0: Setting Powered False
12:13:16.389001 bus /gobi_0: Setting Powered True
12:13:17.764557 bus /gobi_0: Setting Online True
12:13:19.586365 tst /gobi_6: Already registered with network ('901', '70')
12:13:20.320389 tst /gobi_0: Connect to ('901', '70')
12:13:20.602968 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873'
12:13:20.734532 tst iperf3m4.py: using MSISDN 3987
12:13:20.964104 bus /gobi_1: Setting Powered True
12:13:24.464786 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023848', msisdn='3987', subscriber_id=3}
12:13:24.738010 bus /gobi_1: Setting Online False
12:13:25.923743 bus /gobi_1: Setting Powered False
12:13:33.224246 bus /gobi_1: Setting Powered True
12:13:34.597021 bus /gobi_1: Setting Online True
12:13:35.662311 tst /gobi_0: Already registered with network ('901', '70')
12:13:37.277102 tst /gobi_1: Connect to ('901', '70')
12:13:37.544259 tst /gobi_1: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024272103'
12:13:37.667139 tst iperf3m4.py: using MSISDN 3988
12:13:37.835635 bus /gobi_4: Setting Powered True
12:13:41.490774 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023849', msisdn='3988', subscriber_id=4}
12:13:41.765016 bus /gobi_4: Setting Online False
12:13:42.959797 bus /gobi_4: Setting Powered False
12:13:50.226870 bus /gobi_4: Setting Powered True
12:13:51.612541 bus /gobi_4: Setting Online True
12:13:54.094963 tst /gobi_4: Connect to ('901', '70')
12:13:54.382104 tst /gobi_4: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A08E2G 1 [Jul 21 2016 10:00:50]', Serial: '868323025161511'
12:13:54.581523 tst iperf3m4.py:8: waiting for modems to attach...
12:13:55.139563 tst /gobi_1: Already registered with network ('901', '70')
12:14:10.059830 tst /gobi_4: Already registered with network ('901', '70')
12:14:15.201781 tst iperf3m4.py:8: waiting for modems to attach to data services...
12:14:18.241199 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:18.441883 tst iperf3m4.py:8: Setting up data plan for "'/gobi_6/context2'"
12:14:18.818467 run move_netns(pid=21268): Launched
12:14:20.013709 run move_netns(pid=21268): Terminated: ok {rc=0}
12:14:20.366648 run ifup(pid=21271): Launched
12:14:21.550188 run ifup(pid=21271): Terminated: ok {rc=0}
12:14:21.885075 run dhcp(pid=21274): Launched
12:14:23.080306 run dhcp(pid=21274): Terminated: ok {rc=0}
12:14:26.211202 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:26.399843 tst iperf3m4.py:8: Setting up data plan for "'/gobi_0/context2'"
12:14:26.763701 run move_netns(pid=21292): Launched
12:14:27.958439 run move_netns(pid=21292): Terminated: ok {rc=0}
12:14:28.303717 run ifup(pid=21295): Launched
12:14:29.499821 run ifup(pid=21295): Terminated: ok {rc=0}
12:14:29.847564 run dhcp(pid=21298): Launched
12:14:31.043002 run dhcp(pid=21298): Terminated: ok {rc=0}
12:14:39.430561 tst /gobi_1: context activated {apn='inet46', path='/gobi_1/context12', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.3', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='rf303c5eb721dad', Method='static', Netmask='255.255.255.248'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:39.630190 tst iperf3m4.py:8: Setting up data plan for "'/gobi_1/context12'"
12:14:40.008769 run move_netns(pid=21316): Launched
12:14:41.203594 run move_netns(pid=21316): Terminated: ok {rc=0}
12:14:41.555073 run ifup(pid=21319): Launched
12:14:42.750593 run ifup(pid=21319): Terminated: ok {rc=0}
12:14:43.094092 run dhcp(pid=21322): Launched
12:14:44.288607 run dhcp(pid=21322): Terminated: ok {rc=0}
12:14:47.398290 tst /gobi_4: context activated {apn='inet46', path='/gobi_4/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.4', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.5', Interface='r9d788e55295d1b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:14:47.598515 tst iperf3m4.py:8: Setting up data plan for "'/gobi_4/context2'"
12:14:47.972558 run move_netns(pid=21344): Launched
12:14:49.167644 run move_netns(pid=21344): Terminated: ok {rc=0}
12:14:49.511099 run ifup(pid=21347): Launched
12:14:50.695102 run ifup(pid=21347): Terminated: ok {rc=0}
12:14:51.022092 run dhcp(pid=21350): Launched
12:14:52.204665 run dhcp(pid=21350): Terminated: ok {rc=0}
12:14:52.398498 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'"
12:14:52.528723 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:14:52.914061 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Launched
12:14:53.168421 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'"
12:14:53.307840 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s)
12:14:53.709624 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Launched
12:14:53.962766 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5005 through "'/gobi_1/context12'"
12:14:54.102251 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5005 (proto=tcp,time=60s)
12:14:54.505434 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Launched
12:14:54.758418 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5006 through "'/gobi_4/context2'"
12:14:54.897262 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5006 (proto=tcp,time=60s)
12:14:55.297856 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Launched
12:14:55.633982 run iperf3-cli_10.42.42.10(pid=21378): Launched
12:14:55.918859 run iperf3-cli_10.42.42.10(pid=21381): Launched
12:14:56.208202 run iperf3-cli_10.42.42.10(pid=21384): Launched
12:14:56.498429 run iperf3-cli_10.42.42.10(pid=21387): Launched
12:16:19.844408 run iperf3-cli_10.42.42.10(pid=21378): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21378)]
12:16:19.980002 run iperf3-cli_10.42.42.10(pid=21378): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:16:20.113772 run iperf3-cli_10.42.42.10(pid=21378): stderr:
| (launched: 2022-09-08_12:14:55.515951)
12:16:21.302189 run iperf3-cli_10.42.42.10(pid=21381): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21381)]
12:16:21.436576 run iperf3-cli_10.42.42.10(pid=21381): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:16:21.570684 run iperf3-cli_10.42.42.10(pid=21381): stderr:
| (launched: 2022-09-08_12:14:55.801521)
12:16:22.767473 run iperf3-cli_10.42.42.10(pid=21384): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21384)]
12:16:22.904946 run iperf3-cli_10.42.42.10(pid=21384): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:16:23.041323 run iperf3-cli_10.42.42.10(pid=21384): stderr:
| (launched: 2022-09-08_12:14:56.087095)
12:16:24.237555 run iperf3-cli_10.42.42.10(pid=21387): ERR: Terminated: ERROR {rc=1} [trial-2067↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=21387)]
12:16:24.372865 run iperf3-cli_10.42.42.10(pid=21387): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:16:24.506316 run iperf3-cli_10.42.42.10(pid=21387): stderr:
| (launched: 2022-09-08_12:14:56.377632)
12:16:24.628427 run iperf3-srv_10.42.42.10(pid=21203): Terminating (SIGINT)
12:16:24.966664 run iperf3-srv_10.42.42.10(pid=21203): Terminated {rc=1}
12:16:25.166682 tst iperf3m4.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context2'"
12:16:25.225067 tst iperf3m4.py:8: ERR: KeyError: 'sum_sent' [trial-2067↪gprs:sysmo↪iperf3m4.py:8]
12:16:25.232006 tst iperf3m4.py:8: Test FAILED (296.9 sec)
12:16:25.269516 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21202): Terminating (SIGTERM)
12:16:25.311985 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21204): Terminating (SIGTERM)
12:16:25.353839 run iperf3-srv_10.42.42.10(pid=21205): Terminating (SIGTERM)
12:16:25.396412 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21206): Terminating (SIGTERM)
12:16:25.438433 run iperf3-srv_10.42.42.10(pid=21207): Terminating (SIGTERM)
12:16:25.480936 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21208): Terminating (SIGTERM)
12:16:25.523604 run iperf3-srv_10.42.42.10(pid=21209): Terminating (SIGTERM)
12:16:25.565657 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21212): Terminating (SIGTERM)
12:16:25.608160 run osmo-hlr_10.42.42.2(pid=21213): Terminating (SIGTERM)
12:16:25.651464 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21214): Terminating (SIGTERM)
12:16:25.694859 run osmo-stp_10.42.42.5(pid=21215): Terminating (SIGTERM)
12:16:25.757834 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21216): Terminating (SIGTERM)
12:16:25.801100 run osmo-ggsn_10.42.42.6(pid=21221): Terminating (SIGTERM)
12:16:25.864673 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21238): Terminating (SIGTERM)
12:16:25.928722 run osmo-sgsn_10.42.42.7(pid=21239): Terminating (SIGTERM)
12:16:25.991089 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21240): Terminating (SIGTERM)
12:16:26.052432 run osmo-msc_10.42.42.8(pid=21241): Terminating (SIGTERM)
12:16:26.099766 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21242): Terminating (SIGTERM)
12:16:26.143596 run osmo-mgw_10.42.42.3(pid=21243): Terminating (SIGTERM)
12:16:26.187521 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21244): Terminating (SIGTERM)
12:16:26.231474 run osmo-mgw_10.42.42.4(pid=21245): Terminating (SIGTERM)
12:16:26.274626 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21246): Terminating (SIGTERM)
12:16:26.322089 run osmo-bsc_10.42.42.9(pid=21247): Terminating (SIGTERM)
12:16:26.367138 run osmo-bts-sysmo(pid=21256): Terminating (SIGTERM)
12:16:26.412659 run osmo-pcu(pid=21264): Terminating (SIGTERM)
12:16:26.455494 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Terminating (SIGTERM)
12:16:26.612839 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGTERM(pid=21422): Launched
12:16:27.704877 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGTERM(pid=21422): Terminated: ok {rc=0}
12:16:27.751075 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Terminating (SIGTERM)
12:16:27.904689 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGTERM(pid=21427): Launched
12:16:28.999863 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGTERM(pid=21427): Terminated: ok {rc=0}
12:16:29.047928 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Terminating (SIGTERM)
12:16:29.208359 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGTERM(pid=21430): Launched
12:16:30.303402 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGTERM(pid=21430): Terminated: ok {rc=0}
12:16:30.351861 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Terminating (SIGTERM)
12:16:30.512537 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGTERM(pid=21433): Launched
12:16:31.608861 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGTERM(pid=21433): Terminated: ok {rc=0}
12:16:31.623397 --- ParallelTerminationStrategy: PID 21202 died...
12:16:31.680652 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21202): Terminated: ok {rc=0}
12:16:31.690827 --- ParallelTerminationStrategy: PID 21204 died...
12:16:31.747549 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21204): Terminated: ok {rc=0}
12:16:31.757507 --- ParallelTerminationStrategy: PID 21205 died...
12:16:31.814036 run iperf3-srv_10.42.42.10(pid=21205): Terminated {rc=256}
12:16:31.824276 --- ParallelTerminationStrategy: PID 21206 died...
12:16:31.880538 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21206): Terminated: ok {rc=0}
12:16:31.890929 --- ParallelTerminationStrategy: PID 21207 died...
12:16:31.947411 run iperf3-srv_10.42.42.10(pid=21207): Terminated {rc=256}
12:16:31.957615 --- ParallelTerminationStrategy: PID 21208 died...
12:16:32.014299 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21208): Terminated: ok {rc=0}
12:16:32.024316 --- ParallelTerminationStrategy: PID 21209 died...
12:16:32.080791 run iperf3-srv_10.42.42.10(pid=21209): Terminated {rc=256}
12:16:32.090808 --- ParallelTerminationStrategy: PID 21212 died...
12:16:32.147348 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21212): Terminated: ok {rc=0}
12:16:32.157352 --- ParallelTerminationStrategy: PID 21213 died...
12:16:32.214347 run osmo-hlr_10.42.42.2(pid=21213): Terminated: ok {rc=0}
12:16:32.224388 --- ParallelTerminationStrategy: PID 21214 died...
12:16:32.280732 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21214): Terminated: ok {rc=0}
12:16:32.291036 --- ParallelTerminationStrategy: PID 21215 died...
12:16:32.347530 run osmo-stp_10.42.42.5(pid=21215): Terminated: ok {rc=0}
12:16:32.357654 --- ParallelTerminationStrategy: PID 21216 died...
12:16:32.414318 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21216): Terminated: ok {rc=0}
12:16:32.424561 --- ParallelTerminationStrategy: PID 21221 died...
12:16:32.481054 run osmo-ggsn_10.42.42.6(pid=21221): Terminated: ok {rc=0}
12:16:32.491414 --- ParallelTerminationStrategy: PID 21238 died...
12:16:32.548260 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21238): Terminated: ok {rc=0}
12:16:32.558281 --- ParallelTerminationStrategy: PID 21239 died...
12:16:32.615021 run osmo-sgsn_10.42.42.7(pid=21239): Terminated: ok {rc=0}
12:16:32.625215 --- ParallelTerminationStrategy: PID 21240 died...
12:16:32.682156 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21240): Terminated: ok {rc=0}
12:16:32.692263 --- ParallelTerminationStrategy: PID 21241 died...
12:16:32.748864 run osmo-msc_10.42.42.8(pid=21241): Terminated: ok {rc=0}
12:16:32.759004 --- ParallelTerminationStrategy: PID 21242 died...
12:16:32.815666 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21242): Terminated: ok {rc=0}
12:16:32.825733 --- ParallelTerminationStrategy: PID 21243 died...
12:16:32.882639 run osmo-mgw_10.42.42.3(pid=21243): Terminated {rc=15}
12:16:32.892716 --- ParallelTerminationStrategy: PID 21244 died...
12:16:32.949488 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21244): Terminated: ok {rc=0}
12:16:32.959638 --- ParallelTerminationStrategy: PID 21245 died...
12:16:33.016987 run osmo-mgw_10.42.42.4(pid=21245): Terminated {rc=15}
12:16:33.027078 --- ParallelTerminationStrategy: PID 21246 died...
12:16:33.083914 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21246): Terminated: ok {rc=0}
12:16:33.094256 --- ParallelTerminationStrategy: PID 21247 died...
12:16:33.151037 run osmo-bsc_10.42.42.9(pid=21247): Terminated: ok {rc=0}
12:16:33.161108 --- ParallelTerminationStrategy: PID 21256 died...
12:16:33.217850 run osmo-bts-sysmo(pid=21256): Terminated: ok {rc=0}
12:16:33.228025 --- ParallelTerminationStrategy: PID 21264 died...
12:16:33.285572 run osmo-pcu(pid=21264): Terminated: ok {rc=0}
12:16:39.154267 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Terminating (SIGINT)
12:16:39.309667 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGINT(pid=21436): Launched
12:16:40.404915 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGINT(pid=21436): Terminated: ok {rc=0}
12:16:40.453447 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Terminating (SIGINT)
12:16:40.613845 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGINT(pid=21439): Launched
12:16:41.708654 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGINT(pid=21439): Terminated: ok {rc=0}
12:16:41.757246 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Terminating (SIGINT)
12:16:41.916333 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGINT(pid=21442): Launched
12:16:43.011078 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGINT(pid=21442): Terminated: ok {rc=0}
12:16:43.059868 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Terminating (SIGINT)
12:16:43.215292 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGINT(pid=21445): Launched
12:16:44.305955 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGINT(pid=21445): Terminated: ok {rc=0}
12:16:50.176527 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366): Terminating (SIGKILL)
12:16:50.342896 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGUSR1(pid=21448): Launched
12:16:51.434046 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21366)-killSignals.SIGUSR1(pid=21448): Terminated: ok {rc=0}
12:16:51.480013 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369): Terminating (SIGKILL)
12:16:51.632717 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGUSR1(pid=21451): Launched
12:16:52.724058 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21369)-killSignals.SIGUSR1(pid=21451): Terminated: ok {rc=0}
12:16:52.769859 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372): Terminating (SIGKILL)
12:16:52.922247 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGUSR1(pid=21454): Launched
12:16:54.012502 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21372)-killSignals.SIGUSR1(pid=21454): Terminated: ok {rc=0}
12:16:54.058627 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375): Terminating (SIGKILL)
12:16:54.209302 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGUSR1(pid=21457): Launched
12:16:55.300985 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=21375)-killSignals.SIGUSR1(pid=21457): Terminated: ok {rc=0}
12:16:55.539496 bus /gobi_4: Setting Online False
12:16:56.677312 bus /gobi_4: Setting Powered False
12:17:06.977899 bus /gobi_4: Setting Powered False
12:17:08.085825 bus /gobi_4: Setting Powered False
12:17:09.238187 bus /gobi_1: Setting Online False
12:17:10.373666 bus /gobi_1: Setting Powered False
12:17:18.084487 bus /gobi_1: Setting Powered False
12:17:19.190688 bus /gobi_1: Setting Powered False
12:17:20.342609 bus /gobi_0: Setting Online False
12:17:21.478233 bus /gobi_0: Setting Powered False
12:17:29.976993 bus /gobi_0: Setting Powered False
12:17:31.083720 bus /gobi_0: Setting Powered False
12:17:32.235462 bus /gobi_6: Setting Online False
12:17:33.348859 bus /gobi_6: Setting Powered False
12:17:41.052366 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'