![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3m4.py (from gprs_sysmo)
Failing for the past 3 builds
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2065 gprs:sysmo iperf3m4.py
----------------------------------------------
12:22:40.220896 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.232862 tst gprs:sysmo: Using 1 x bts (candidates: 1)
12:22:40.363396 tst iperf3m4.py: using LAC 3954
12:22:40.489677 tst iperf3m4.py: using RAC 129
12:22:40.615872 tst iperf3m4.py: using CellId 3954
12:22:40.741940 tst iperf3m4.py: using BVCI 3955
12:22:40.766870 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.777801 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.788740 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.799673 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.810360 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.821157 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.832070 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.842902 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:22:40.853923 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:22:40.970383 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:22:41.092690 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:22:41.209313 tst gprs:sysmo: Using 1 x modem (candidates: 4)
12:22:41.505726 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
12:22:41.641587 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:22:41.980963 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31823): Launched
12:22:42.264521 run iperf3-srv_10.42.42.10(pid=31824): Launched
12:22:42.464477 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
12:22:42.600732 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:22:42.937464 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31825): Launched
12:22:43.217562 run iperf3-srv_10.42.42.10(pid=31826): Launched
12:22:43.416082 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
12:22:43.551534 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:22:43.888955 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31827): Launched
12:22:44.168815 run iperf3-srv_10.42.42.10(pid=31828): Launched
12:22:44.366604 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
12:22:44.501410 run iperf3-srv_10.42.42.10: Starting iperf3-srv
12:22:44.838975 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31829): Launched
12:22:45.119182 run iperf3-srv_10.42.42.10(pid=31830): Launched
12:22:45.316548 tst iperf3m4.py:8: start network...
12:22:45.451451 run osmo-hlr_10.42.42.2: Starting osmo-hlr
12:22:45.965129 run create_hlr_db(pid=31831): Launched
12:22:46.152496 bus /gobi_6: Setting Powered False
12:22:47.173254 run create_hlr_db(pid=31831): Terminated: ok {rc=0}
12:22:47.606426 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31833): Launched
12:22:47.933495 run osmo-hlr_10.42.42.2(pid=31834): Launched
12:22:48.049831 run osmo-stp_10.42.42.5: Starting osmo-stp
12:22:48.613538 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31835): Launched
12:22:48.938518 run osmo-stp_10.42.42.5(pid=31836): Launched
12:22:49.053561 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:22:49.619900 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31837): Launched
12:22:49.735511 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:22:50.020133 run patchelf(pid=31838): Launched
12:22:51.212953 run patchelf(pid=31838): Terminated: ok {rc=0}
12:22:51.331889 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:22:51.619110 run osmo-gsm-tester_setcap_net_admin.sh(pid=31839): Launched
12:22:52.810319 run osmo-gsm-tester_setcap_net_admin.sh(pid=31839): Terminated: ok {rc=0}
12:22:53.139146 run osmo-ggsn_10.42.42.6(pid=31842): Launched
12:22:53.354676 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:22:53.967857 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31860): Launched
12:22:54.294100 run osmo-sgsn_10.42.42.7(pid=31861): Launched
12:22:54.411749 run osmo-msc_10.42.42.8: Starting osmo-msc
12:22:55.026312 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31862): Launched
12:22:55.354310 run osmo-msc_10.42.42.8(pid=31863): Launched
12:22:58.528023 run osmo-mgw_10.42.42.3: Starting osmo-mgw
12:22:59.090514 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31864): Launched
12:22:59.416757 run osmo-mgw_10.42.42.3(pid=31865): Launched
12:22:59.532777 run osmo-mgw_10.42.42.4: Starting osmo-mgw
12:23:00.087265 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31866): Launched
12:23:00.413357 run osmo-mgw_10.42.42.4(pid=31867): Launched
12:23:00.528858 run osmo-bsc_10.42.42.9: Starting osmo-bsc
12:23:01.208379 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31868): Launched
12:23:01.520136 run osmo-bsc_10.42.42.9(pid=31869): Launched
12:23:07.780735 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:23:08.324654 run rm-remote-dir(pid=31870): Launched
12:23:10.514324 run rm-remote-dir(pid=31870): Terminated: ok {rc=0}
12:23:10.840516 run mk-remote-dir(pid=31871): Launched
12:23:12.037897 run mk-remote-dir(pid=31871): Terminated: ok {rc=0}
12:23:12.333831 run scp-inst-to-remote(pid=31872): Launched
12:23:24.555021 run scp-inst-to-remote(pid=31872): Terminated: ok {rc=0}
12:23:24.898798 run mk-remote-dir(pid=31874): Launched
12:23:26.094261 run mk-remote-dir(pid=31874): Terminated: ok {rc=0}
12:23:26.389791 run scp-cfg-to-remote(pid=31875): Launched
12:23:27.587562 run scp-cfg-to-remote(pid=31875): Terminated: ok {rc=0}
12:23:27.928315 run reload-dsp-firmware(pid=31877): Launched
12:23:30.122510 run reload-dsp-firmware(pid=31877): Terminated: ok {rc=0}
12:23:30.453010 run osmo-bts-sysmo(pid=31878): Launched
12:23:34.769559 tst iperf3m4.py:8: Waiting for bts to be ready...
12:23:36.414927 run rm-remote-dir(pid=31879): Launched
12:23:38.609519 run rm-remote-dir(pid=31879): Terminated: ok {rc=0}
12:23:38.945021 run mk-remote-dir(pid=31880): Launched
12:23:41.129006 run mk-remote-dir(pid=31880): Terminated: ok {rc=0}
12:23:41.395680 run scp-inst-to-bts(pid=31881): Launched
12:23:54.591007 run scp-inst-to-bts(pid=31881): Terminated: ok {rc=0}
12:23:54.902997 run mk-remote-run-dir(pid=31883): Launched
12:23:57.084375 run mk-remote-run-dir(pid=31883): Terminated: ok {rc=0}
12:23:57.351295 run scp-cfg-to-sysmobts(pid=31884): Launched
12:23:59.534357 run scp-cfg-to-sysmobts(pid=31884): Terminated: ok {rc=0}
12:23:59.849470 run osmo-pcu(pid=31886): Launched
12:23:59.977329 tst iperf3m4.py: using MSISDN 3963
12:24:00.148367 bus /gobi_6: Setting Powered True
12:24:03.663135 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3963', subscriber_id=1}
12:24:03.956142 bus /gobi_6: Setting Online False
12:24:05.141665 bus /gobi_6: Setting Powered False
12:24:12.229279 bus /gobi_6: Setting Powered True
12:24:13.605736 bus /gobi_6: Setting Online True
12:24:16.025902 tst /gobi_6: Connect to ('901', '70')
12:24:16.308531 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
12:24:16.437771 tst iperf3m4.py: using MSISDN 3964
12:24:16.659085 bus /gobi_0: Setting Powered True
12:24:20.188327 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3964', subscriber_id=2}
12:24:20.462344 bus /gobi_0: Setting Online False
12:24:21.647291 bus /gobi_0: Setting Powered False
12:24:22.957257 tst /gobi_6: Already registered with network ('901', '70')
12:24:28.425279 bus /gobi_0: Setting Powered True
12:24:29.871935 bus /gobi_0: Setting Online True
12:24:32.383534 tst /gobi_0: Connect to ('901', '70')
12:24:32.668930 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873'
12:24:32.799977 tst iperf3m4.py: using MSISDN 3965
12:24:33.087271 bus /gobi_1: Setting Powered True
12:24:36.633597 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023848', msisdn='3965', subscriber_id=3}
12:24:36.903511 bus /gobi_1: Setting Online False
12:24:38.087147 bus /gobi_1: Setting Powered False
12:24:45.231395 bus /gobi_1: Setting Powered True
12:24:46.628946 bus /gobi_1: Setting Online True
12:24:47.715741 tst /gobi_0: Already registered with network ('901', '70')
12:24:49.297305 tst /gobi_1: Connect to ('901', '70')
12:24:49.563405 tst /gobi_1: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024272103'
12:24:49.682412 tst iperf3m4.py: using MSISDN 3966
12:24:49.849494 bus /gobi_4: Setting Powered True
12:24:53.438819 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023849', msisdn='3966', subscriber_id=4}
12:24:53.725239 bus /gobi_4: Setting Online False
12:24:54.911725 bus /gobi_4: Setting Powered False
12:25:02.212618 bus /gobi_4: Setting Powered True
12:25:03.580946 bus /gobi_4: Setting Online True
12:25:05.989696 tst /gobi_4: Connect to ('901', '70')
12:25:06.263176 tst /gobi_4: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A08E2G 1 [Jul 21 2016 10:00:50]', Serial: '868323025161511'
12:25:06.449304 tst iperf3m4.py:8: waiting for modems to attach...
12:25:06.635206 tst /gobi_1: Already registered with network ('901', '70')
12:25:22.828045 tst /gobi_4: Already registered with network ('901', '70')
12:25:28.326844 tst iperf3m4.py:8: waiting for modems to attach to data services...
12:25:36.704858 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:25:36.891262 tst iperf3m4.py:8: Setting up data plan for "'/gobi_6/context2'"
12:25:37.246985 run move_netns(pid=31894): Launched
12:25:38.428592 run move_netns(pid=31894): Terminated: ok {rc=0}
12:25:38.760118 run ifup(pid=31897): Launched
12:25:39.953203 run ifup(pid=31897): Terminated: ok {rc=0}
12:25:40.299655 run dhcp(pid=31900): Launched
12:25:41.493379 run dhcp(pid=31900): Terminated: ok {rc=0}
12:25:44.512260 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:25:44.698305 tst iperf3m4.py:8: Setting up data plan for "'/gobi_0/context2'"
12:25:45.048608 run move_netns(pid=31918): Launched
12:25:46.231035 run move_netns(pid=31918): Terminated: ok {rc=0}
12:25:46.552899 run ifup(pid=31921): Launched
12:25:47.735836 run ifup(pid=31921): Terminated: ok {rc=0}
12:25:48.064163 run dhcp(pid=31924): Launched
12:25:49.258905 run dhcp(pid=31924): Terminated: ok {rc=0}
12:25:52.265355 tst /gobi_1: context activated {apn='inet46', path='/gobi_1/context10', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.3', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='rf303c5eb721dad', Method='static', Netmask='255.255.255.248'}, Type='internet', Username='ogt'}, user='ogt'}
12:25:52.469991 tst iperf3m4.py:8: Setting up data plan for "'/gobi_1/context10'"
12:25:52.844323 run move_netns(pid=31942): Launched
12:25:54.027824 run move_netns(pid=31942): Terminated: ok {rc=0}
12:25:54.349500 run ifup(pid=31945): Launched
12:25:55.534260 run ifup(pid=31945): Terminated: ok {rc=0}
12:25:55.858834 run dhcp(pid=31948): Launched
12:25:57.042354 run dhcp(pid=31948): Terminated: ok {rc=0}
12:26:00.044820 tst /gobi_4: context activated {apn='inet46', path='/gobi_4/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.4', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.5', Interface='r9d788e55295d1b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:26:00.246970 tst iperf3m4.py:8: Setting up data plan for "'/gobi_4/context2'"
12:26:00.630255 run move_netns(pid=31966): Launched
12:26:01.824032 run move_netns(pid=31966): Terminated: ok {rc=0}
12:26:02.168890 run ifup(pid=31969): Launched
12:26:03.351792 run ifup(pid=31969): Terminated: ok {rc=0}
12:26:03.681675 run dhcp(pid=31972): Launched
12:26:04.874619 run dhcp(pid=31972): Terminated: ok {rc=0}
12:26:05.081137 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'"
12:26:05.220299 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
12:26:05.620347 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992): Launched
12:26:05.873288 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'"
12:26:06.012294 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s)
12:26:06.411997 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995): Launched
12:26:06.666165 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5005 through "'/gobi_1/context10'"
12:26:06.804820 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5005 (proto=tcp,time=60s)
12:26:07.205533 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998): Launched
12:26:07.458831 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5006 through "'/gobi_4/context2'"
12:26:07.597397 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5006 (proto=tcp,time=60s)
12:26:07.998429 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001): Launched
12:26:08.352359 run iperf3-cli_10.42.42.10(pid=32004): Launched
12:26:08.641603 run iperf3-cli_10.42.42.10(pid=32007): Launched
12:26:08.939792 run iperf3-cli_10.42.42.10(pid=32010): Launched
12:26:09.235162 run iperf3-cli_10.42.42.10(pid=32013): Launched
12:27:17.567164 run iperf3-cli_10.42.42.10(pid=32004): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32004)]
12:27:17.704271 run iperf3-cli_10.42.42.10(pid=32004): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:27:17.846788 run iperf3-cli_10.42.42.10(pid=32004): stderr:
| (launched: 2022-06-18_12:26:08.235233)
12:27:19.046448 run iperf3-cli_10.42.42.10(pid=32007): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32007)]
12:27:19.188815 run iperf3-cli_10.42.42.10(pid=32007): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:27:19.329717 run iperf3-cli_10.42.42.10(pid=32007): stderr:
| (launched: 2022-06-18_12:26:08.521394)
12:27:20.532726 run iperf3-cli_10.42.42.10(pid=32010): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32010)]
12:27:20.675144 run iperf3-cli_10.42.42.10(pid=32010): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:27:20.815139 run iperf3-cli_10.42.42.10(pid=32010): stderr:
| (launched: 2022-06-18_12:26:08.815224)
12:27:22.016458 run iperf3-cli_10.42.42.10(pid=32013): ERR: Terminated: ERROR {rc=1} [trial-2065↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32013)]
12:27:22.158941 run iperf3-cli_10.42.42.10(pid=32013): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
12:27:22.301096 run iperf3-cli_10.42.42.10(pid=32013): stderr:
| (launched: 2022-06-18_12:26:09.115161)
12:27:22.430441 run iperf3-srv_10.42.42.10(pid=31824): Terminating (SIGINT)
12:27:22.779391 run iperf3-srv_10.42.42.10(pid=31824): Terminated {rc=1}
12:27:22.987676 tst iperf3m4.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context2'"
12:27:23.050124 tst iperf3m4.py:8: ERR: KeyError: 'sum_sent' [trial-2065↪gprs:sysmo↪iperf3m4.py:8]
12:27:23.058482 tst iperf3m4.py:8: Test FAILED (282.9 sec)
12:27:23.097541 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31823): Terminating (SIGTERM)
12:27:23.142161 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31825): Terminating (SIGTERM)
12:27:23.186159 run iperf3-srv_10.42.42.10(pid=31826): Terminating (SIGTERM)
12:27:23.230864 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31827): Terminating (SIGTERM)
12:27:23.275051 run iperf3-srv_10.42.42.10(pid=31828): Terminating (SIGTERM)
12:27:23.318956 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31829): Terminating (SIGTERM)
12:27:23.362833 run iperf3-srv_10.42.42.10(pid=31830): Terminating (SIGTERM)
12:27:23.406986 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31833): Terminating (SIGTERM)
12:27:23.451128 run osmo-hlr_10.42.42.2(pid=31834): Terminating (SIGTERM)
12:27:23.496216 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31835): Terminating (SIGTERM)
12:27:23.543101 run osmo-stp_10.42.42.5(pid=31836): Terminating (SIGTERM)
12:27:23.593449 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31837): Terminating (SIGTERM)
12:27:23.639073 run osmo-ggsn_10.42.42.6(pid=31842): Terminating (SIGTERM)
12:27:23.740505 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31860): Terminating (SIGTERM)
12:27:23.786742 run osmo-sgsn_10.42.42.7(pid=31861): Terminating (SIGTERM)
12:27:23.842861 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31862): Terminating (SIGTERM)
12:27:23.897421 run osmo-msc_10.42.42.8(pid=31863): Terminating (SIGTERM)
12:27:23.945892 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31864): Terminating (SIGTERM)
12:27:23.994892 run osmo-mgw_10.42.42.3(pid=31865): Terminating (SIGTERM)
12:27:24.038936 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31866): Terminating (SIGTERM)
12:27:24.082086 run osmo-mgw_10.42.42.4(pid=31867): Terminating (SIGTERM)
12:27:24.130384 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31868): Terminating (SIGTERM)
12:27:24.173754 run osmo-bsc_10.42.42.9(pid=31869): Terminating (SIGTERM)
12:27:24.218464 run osmo-bts-sysmo(pid=31878): Terminating (SIGTERM)
12:27:24.262381 run osmo-pcu(pid=31886): Terminating (SIGTERM)
12:27:24.306969 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992): Terminating (SIGTERM)
12:27:24.462465 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGTERM(pid=32041): Launched
12:27:25.556907 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGTERM(pid=32041): Terminated: ok {rc=0}
12:27:25.604675 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995): Terminating (SIGTERM)
12:27:25.757822 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGTERM(pid=32047): Launched
12:27:26.851347 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGTERM(pid=32047): Terminated: ok {rc=0}
12:27:26.902175 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998): Terminating (SIGTERM)
12:27:27.055076 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGTERM(pid=32050): Launched
12:27:28.148726 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGTERM(pid=32050): Terminated: ok {rc=0}
12:27:28.196894 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001): Terminating (SIGTERM)
12:27:28.349543 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGTERM(pid=32053): Launched
12:27:29.444138 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGTERM(pid=32053): Terminated: ok {rc=0}
12:27:29.458344 --- ParallelTerminationStrategy: PID 31823 died...
12:27:29.515138 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31823): Terminated: ok {rc=0}
12:27:29.525199 --- ParallelTerminationStrategy: PID 31825 died...
12:27:29.581691 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31825): Terminated: ok {rc=0}
12:27:29.591795 --- ParallelTerminationStrategy: PID 31826 died...
12:27:29.647582 run iperf3-srv_10.42.42.10(pid=31826): Terminated {rc=256}
12:27:29.657468 --- ParallelTerminationStrategy: PID 31827 died...
12:27:29.713637 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31827): Terminated: ok {rc=0}
12:27:29.723695 --- ParallelTerminationStrategy: PID 31828 died...
12:27:29.779805 run iperf3-srv_10.42.42.10(pid=31828): Terminated {rc=256}
12:27:29.789730 --- ParallelTerminationStrategy: PID 31829 died...
12:27:29.845697 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31829): Terminated: ok {rc=0}
12:27:29.855648 --- ParallelTerminationStrategy: PID 31830 died...
12:27:29.912160 run iperf3-srv_10.42.42.10(pid=31830): Terminated {rc=256}
12:27:29.922158 --- ParallelTerminationStrategy: PID 31833 died...
12:27:29.978967 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31833): Terminated: ok {rc=0}
12:27:29.989024 --- ParallelTerminationStrategy: PID 31834 died...
12:27:30.045268 run osmo-hlr_10.42.42.2(pid=31834): Terminated: ok {rc=0}
12:27:30.055338 --- ParallelTerminationStrategy: PID 31835 died...
12:27:30.111581 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31835): Terminated: ok {rc=0}
12:27:30.121642 --- ParallelTerminationStrategy: PID 31836 died...
12:27:30.177914 run osmo-stp_10.42.42.5(pid=31836): Terminated: ok {rc=0}
12:27:30.187895 --- ParallelTerminationStrategy: PID 31837 died...
12:27:30.243674 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31837): Terminated: ok {rc=0}
12:27:30.253621 --- ParallelTerminationStrategy: PID 31842 died...
12:27:30.310269 run osmo-ggsn_10.42.42.6(pid=31842): Terminated: ok {rc=0}
12:27:30.320301 --- ParallelTerminationStrategy: PID 31860 died...
12:27:30.376101 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31860): Terminated: ok {rc=0}
12:27:30.386081 --- ParallelTerminationStrategy: PID 31861 died...
12:27:30.442124 run osmo-sgsn_10.42.42.7(pid=31861): Terminated: ok {rc=0}
12:27:30.452063 --- ParallelTerminationStrategy: PID 31862 died...
12:27:30.507937 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31862): Terminated: ok {rc=0}
12:27:30.517971 --- ParallelTerminationStrategy: PID 31863 died...
12:27:30.574217 run osmo-msc_10.42.42.8(pid=31863): Terminated: ok {rc=0}
12:27:30.584142 --- ParallelTerminationStrategy: PID 31864 died...
12:27:30.639983 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31864): Terminated: ok {rc=0}
12:27:30.649994 --- ParallelTerminationStrategy: PID 31865 died...
12:27:30.706150 run osmo-mgw_10.42.42.3(pid=31865): Terminated {rc=15}
12:27:30.716102 --- ParallelTerminationStrategy: PID 31866 died...
12:27:30.772174 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31866): Terminated: ok {rc=0}
12:27:30.782174 --- ParallelTerminationStrategy: PID 31867 died...
12:27:30.838095 run osmo-mgw_10.42.42.4(pid=31867): Terminated {rc=15}
12:27:30.848070 --- ParallelTerminationStrategy: PID 31868 died...
12:27:30.904266 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31868): Terminated: ok {rc=0}
12:27:30.914257 --- ParallelTerminationStrategy: PID 31869 died...
12:27:30.970167 run osmo-bsc_10.42.42.9(pid=31869): Terminated: ok {rc=0}
12:27:30.980577 --- ParallelTerminationStrategy: PID 31878 died...
12:27:31.037230 run osmo-bts-sysmo(pid=31878): Terminated: ok {rc=0}
12:27:31.047752 --- ParallelTerminationStrategy: PID 31886 died...
12:27:31.104208 run osmo-pcu(pid=31886): Terminated: ok {rc=0}
12:27:36.971734 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992): Terminating (SIGINT)
12:27:37.119802 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGINT(pid=32056): Launched
12:27:38.210629 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGINT(pid=32056): Terminated: ok {rc=0}
12:27:38.256232 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995): Terminating (SIGINT)
12:27:38.406835 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGINT(pid=32059): Launched
12:27:39.500438 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGINT(pid=32059): Terminated: ok {rc=0}
12:27:39.548102 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998): Terminating (SIGINT)
12:27:39.706220 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGINT(pid=32062): Launched
12:27:40.799891 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGINT(pid=32062): Terminated: ok {rc=0}
12:27:40.847134 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001): Terminating (SIGINT)
12:27:41.004198 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGINT(pid=32065): Launched
12:27:42.100704 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGINT(pid=32065): Terminated: ok {rc=0}
12:27:47.970009 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992): Terminating (SIGKILL)
12:27:48.119832 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGUSR1(pid=32068): Launched
12:27:49.211696 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31992)-killSignals.SIGUSR1(pid=32068): Terminated: ok {rc=0}
12:27:49.256772 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995): Terminating (SIGKILL)
12:27:49.401856 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGUSR1(pid=32071): Launched
12:27:50.491726 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31995)-killSignals.SIGUSR1(pid=32071): Terminated: ok {rc=0}
12:27:50.538039 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998): Terminating (SIGKILL)
12:27:50.689001 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGUSR1(pid=32074): Launched
12:27:51.779239 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31998)-killSignals.SIGUSR1(pid=32074): Terminated: ok {rc=0}
12:27:51.825090 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001): Terminating (SIGKILL)
12:27:51.976847 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGUSR1(pid=32077): Launched
12:27:53.071653 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32001)-killSignals.SIGUSR1(pid=32077): Terminated: ok {rc=0}
12:27:53.309547 bus /gobi_4: Setting Online False
12:27:54.476002 bus /gobi_4: Setting Powered False
12:28:03.975166 bus /gobi_4: Setting Powered False
12:28:05.082734 bus /gobi_4: Setting Powered False
12:28:06.235383 bus /gobi_1: Setting Online False
12:28:07.372336 bus /gobi_1: Setting Powered False
12:28:15.074747 bus /gobi_1: Setting Powered False
12:28:16.181031 bus /gobi_1: Setting Powered False
12:28:17.332359 bus /gobi_0: Setting Online False
12:28:18.472652 bus /gobi_0: Setting Powered False
12:28:26.973085 bus /gobi_0: Setting Powered False
12:28:28.080101 bus /gobi_0: Setting Powered False
12:28:29.230845 bus /gobi_6: Setting Online False
12:28:30.344763 bus /gobi_6: Setting Powered False
12:28:38.107979 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'