![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3m4.py (from gprs_sysmo)
Failing for the past 2 builds
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2064 gprs:sysmo iperf3m4.py
----------------------------------------------
18:44:05.137592 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:44:05.148771 tst gprs:sysmo: Using 1 x bts (candidates: 1)
18:44:05.278636 tst iperf3m4.py: using LAC 3947
18:44:05.405534 tst iperf3m4.py: using RAC 122
18:44:05.532426 tst iperf3m4.py: using CellId 3947
18:44:05.661002 tst iperf3m4.py: using BVCI 3948
18:44:05.686137 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:44:05.697201 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:44:05.708141 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:44:05.719101 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:44:05.729748 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:44:05.740513 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:44:05.751487 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:44:05.762254 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
18:44:05.773309 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:44:05.890839 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:44:06.008335 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:44:06.127802 tst gprs:sysmo: Using 1 x modem (candidates: 4)
18:44:06.408265 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
18:44:06.535543 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:44:06.855611 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26966): Launched
18:44:07.123220 run iperf3-srv_10.42.42.10(pid=26967): Launched
18:44:07.310900 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
18:44:07.439290 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:44:07.774366 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26969): Launched
18:44:08.056879 run iperf3-srv_10.42.42.10(pid=26971): Launched
18:44:08.246307 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
18:44:08.373201 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:44:08.693131 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26972): Launched
18:44:08.955159 run iperf3-srv_10.42.42.10(pid=26973): Launched
18:44:09.140343 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
18:44:09.266910 run iperf3-srv_10.42.42.10: Starting iperf3-srv
18:44:09.587463 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26974): Launched
18:44:09.850295 run iperf3-srv_10.42.42.10(pid=26975): Launched
18:44:10.035147 tst iperf3m4.py:8: start network...
18:44:10.161194 run osmo-hlr_10.42.42.2: Starting osmo-hlr
18:44:10.642529 run create_hlr_db(pid=26976): Launched
18:44:10.816968 bus /gobi_6: Setting Powered False
18:44:11.838948 run create_hlr_db(pid=26976): Terminated: ok {rc=0}
18:44:12.275131 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26978): Launched
18:44:12.602971 run osmo-hlr_10.42.42.2(pid=26979): Launched
18:44:12.722012 run osmo-stp_10.42.42.5: Starting osmo-stp
18:44:13.288570 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26980): Launched
18:44:13.616117 run osmo-stp_10.42.42.5(pid=26981): Launched
18:44:13.731768 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
18:44:14.299711 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26982): Launched
18:44:14.415877 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
18:44:14.701127 run patchelf(pid=26983): Launched
18:44:15.888523 run patchelf(pid=26983): Terminated: ok {rc=0}
18:44:16.000202 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
18:44:16.280088 run osmo-gsm-tester_setcap_net_admin.sh(pid=26984): Launched
18:44:17.474112 run osmo-gsm-tester_setcap_net_admin.sh(pid=26984): Terminated: ok {rc=0}
18:44:17.806952 run osmo-ggsn_10.42.42.6(pid=26987): Launched
18:44:18.047507 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
18:44:18.632087 run pcap-recorder_any(filters='host 10.42.42.7')(pid=27008): Launched
18:44:18.960679 run osmo-sgsn_10.42.42.7(pid=27009): Launched
18:44:19.080186 run osmo-msc_10.42.42.8: Starting osmo-msc
18:44:19.680991 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=27010): Launched
18:44:20.010044 run osmo-msc_10.42.42.8(pid=27011): Launched
18:44:23.184256 run osmo-mgw_10.42.42.3: Starting osmo-mgw
18:44:23.766435 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=27012): Launched
18:44:24.095233 run osmo-mgw_10.42.42.3(pid=27013): Launched
18:44:24.212492 run osmo-mgw_10.42.42.4: Starting osmo-mgw
18:44:24.768905 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=27014): Launched
18:44:25.097561 run osmo-mgw_10.42.42.4(pid=27015): Launched
18:44:25.213490 run osmo-bsc_10.42.42.9: Starting osmo-bsc
18:44:25.897052 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=27016): Launched
18:44:26.227442 run osmo-bsc_10.42.42.9(pid=27017): Launched
18:44:32.506217 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
18:44:33.093500 run rm-remote-dir(pid=27018): Launched
18:44:35.298429 run rm-remote-dir(pid=27018): Terminated: ok {rc=0}
18:44:35.650268 run mk-remote-dir(pid=27019): Launched
18:44:36.850313 run mk-remote-dir(pid=27019): Terminated: ok {rc=0}
18:44:37.148430 run scp-inst-to-remote(pid=27020): Launched
18:44:49.369233 run scp-inst-to-remote(pid=27020): Terminated: ok {rc=0}
18:44:49.715558 run mk-remote-dir(pid=27022): Launched
18:44:50.913827 run mk-remote-dir(pid=27022): Terminated: ok {rc=0}
18:44:51.212131 run scp-cfg-to-remote(pid=27023): Launched
18:44:52.410776 run scp-cfg-to-remote(pid=27023): Terminated: ok {rc=0}
18:44:52.756104 run reload-dsp-firmware(pid=27025): Launched
18:44:54.953914 run reload-dsp-firmware(pid=27025): Terminated: ok {rc=0}
18:44:55.288694 run osmo-bts-sysmo(pid=27026): Launched
18:44:59.606783 tst iperf3m4.py:8: Waiting for bts to be ready...
18:45:01.255613 run rm-remote-dir(pid=27027): Launched
18:45:03.450908 run rm-remote-dir(pid=27027): Terminated: ok {rc=0}
18:45:03.792328 run mk-remote-dir(pid=27031): Launched
18:45:05.989180 run mk-remote-dir(pid=27031): Terminated: ok {rc=0}
18:45:06.278270 run scp-inst-to-bts(pid=27032): Launched
18:45:19.486671 run scp-inst-to-bts(pid=27032): Terminated: ok {rc=0}
18:45:19.824965 run mk-remote-run-dir(pid=27034): Launched
18:45:22.020956 run mk-remote-run-dir(pid=27034): Terminated: ok {rc=0}
18:45:22.308075 run scp-cfg-to-sysmobts(pid=27035): Launched
18:45:24.502960 run scp-cfg-to-sysmobts(pid=27035): Terminated: ok {rc=0}
18:45:24.839704 run osmo-pcu(pid=27037): Launched
18:45:24.964043 tst iperf3m4.py: using MSISDN 3952
18:45:25.145101 bus /gobi_6: Setting Powered True
18:45:28.656895 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3952', subscriber_id=1}
18:45:28.931670 bus /gobi_6: Setting Online False
18:45:30.127585 bus /gobi_6: Setting Powered False
18:45:37.202113 bus /gobi_6: Setting Powered True
18:45:38.599992 bus /gobi_6: Setting Online True
18:45:41.050326 tst /gobi_6: Connect to ('901', '70')
18:45:41.322623 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
18:45:41.444238 tst iperf3m4.py: using MSISDN 3953
18:45:41.728702 bus /gobi_0: Setting Powered True
18:45:45.233093 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3953', subscriber_id=2}
18:45:45.517243 bus /gobi_0: Setting Online False
18:45:46.707955 bus /gobi_0: Setting Powered False
18:45:54.202169 bus /gobi_0: Setting Powered True
18:45:55.482799 tst /gobi_6: Already registered with network ('901', '70')
18:45:55.705225 bus /gobi_0: Setting Online True
18:45:58.167488 tst /gobi_0: Connect to ('901', '70')
18:45:58.438959 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873'
18:45:58.562181 tst iperf3m4.py: using MSISDN 3954
18:45:59.122338 bus /gobi_1: Setting Powered True
18:46:02.608423 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023848', msisdn='3954', subscriber_id=3}
18:46:02.893761 bus /gobi_1: Setting Online False
18:46:04.081155 bus /gobi_1: Setting Powered False
18:46:11.233717 bus /gobi_1: Setting Powered True
18:46:12.619846 bus /gobi_1: Setting Online True
18:46:13.544115 tst /gobi_0: Already registered with network ('901', '70')
18:46:15.332485 tst /gobi_1: Connect to ('901', '70')
18:46:15.610039 tst /gobi_1: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024272103'
18:46:15.740159 tst iperf3m4.py: using MSISDN 3955
18:46:15.920247 bus /gobi_4: Setting Powered True
18:46:19.433551 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023849', msisdn='3955', subscriber_id=4}
18:46:19.706668 bus /gobi_4: Setting Online False
18:46:20.891089 bus /gobi_4: Setting Powered False
18:46:28.215273 bus /gobi_4: Setting Powered True
18:46:29.605953 bus /gobi_4: Setting Online True
18:46:31.041191 tst /gobi_1: Already registered with network ('901', '70')
18:46:32.271498 tst /gobi_4: Connect to ('901', '70')
18:46:32.552167 tst /gobi_4: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A08E2G 1 [Jul 21 2016 10:00:50]', Serial: '868323025161511'
18:46:32.739019 tst iperf3m4.py:8: waiting for modems to attach...
18:46:48.744405 tst /gobi_4: Already registered with network ('901', '70')
18:46:54.384630 tst iperf3m4.py:8: waiting for modems to attach to data services...
18:47:02.732807 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
18:47:02.936494 tst iperf3m4.py:8: Setting up data plan for "'/gobi_6/context2'"
18:47:03.315762 run move_netns(pid=27042): Launched
18:47:04.512928 run move_netns(pid=27042): Terminated: ok {rc=0}
18:47:04.860441 run ifup(pid=27045): Launched
18:47:06.057649 run ifup(pid=27045): Terminated: ok {rc=0}
18:47:06.402997 run dhcp(pid=27048): Launched
18:47:07.598513 run dhcp(pid=27048): Terminated: ok {rc=0}
18:47:15.910220 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
18:47:16.096898 tst iperf3m4.py:8: Setting up data plan for "'/gobi_0/context2'"
18:47:16.456441 run move_netns(pid=27066): Launched
18:47:17.639908 run move_netns(pid=27066): Terminated: ok {rc=0}
18:47:17.970025 run ifup(pid=27069): Launched
18:47:19.155092 run ifup(pid=27069): Terminated: ok {rc=0}
18:47:19.485572 run dhcp(pid=27072): Launched
18:47:20.681677 run dhcp(pid=27072): Terminated: ok {rc=0}
18:47:23.684988 tst /gobi_1: context activated {apn='inet46', path='/gobi_1/context9', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.3', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='rf303c5eb721dad', Method='static', Netmask='255.255.255.248'}, Type='internet', Username='ogt'}, user='ogt'}
18:47:23.871755 tst iperf3m4.py:8: Setting up data plan for "'/gobi_1/context9'"
18:47:24.230206 run move_netns(pid=27090): Launched
18:47:25.414374 run move_netns(pid=27090): Terminated: ok {rc=0}
18:47:25.747258 run ifup(pid=27093): Launched
18:47:26.944844 run ifup(pid=27093): Terminated: ok {rc=0}
18:47:27.293317 run dhcp(pid=27096): Launched
18:47:28.489135 run dhcp(pid=27096): Terminated: ok {rc=0}
18:47:36.872190 tst /gobi_4: context activated {apn='inet46', path='/gobi_4/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.4', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.5', Interface='r9d788e55295d1b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
18:47:37.059051 tst iperf3m4.py:8: Setting up data plan for "'/gobi_4/context2'"
18:47:37.420915 run move_netns(pid=27114): Launched
18:47:38.617966 run move_netns(pid=27114): Terminated: ok {rc=0}
18:47:38.968435 run ifup(pid=27117): Launched
18:47:40.165241 run ifup(pid=27117): Terminated: ok {rc=0}
18:47:40.512800 run dhcp(pid=27120): Launched
18:47:41.709021 run dhcp(pid=27120): Terminated: ok {rc=0}
18:47:41.918244 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'"
18:47:42.058488 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
18:47:42.463373 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136): Launched
18:47:42.719635 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'"
18:47:42.859682 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s)
18:47:43.263720 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139): Launched
18:47:43.518027 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5005 through "'/gobi_1/context9'"
18:47:43.656821 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5005 (proto=tcp,time=60s)
18:47:44.057464 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142): Launched
18:47:44.311522 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5006 through "'/gobi_4/context2'"
18:47:44.450430 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5006 (proto=tcp,time=60s)
18:47:44.884665 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145): Launched
18:47:45.209769 run iperf3-cli_10.42.42.10(pid=27148): Launched
18:47:45.484308 run iperf3-cli_10.42.42.10(pid=27151): Launched
18:47:45.754659 run iperf3-cli_10.42.42.10(pid=27154): Launched
18:47:46.026203 run iperf3-cli_10.42.42.10(pid=27157): Launched
18:48:54.469178 run iperf3-cli_10.42.42.10(pid=27148): ERR: Terminated: ERROR {rc=1} [trial-2064↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=27148)]
18:48:54.627124 run iperf3-cli_10.42.42.10(pid=27148): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
18:48:54.762003 run iperf3-cli_10.42.42.10(pid=27148): stderr:
| (launched: 2022-03-29_18:47:45.089716)
18:48:55.951606 run iperf3-cli_10.42.42.10(pid=27151): ERR: Terminated: ERROR {rc=1} [trial-2064↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=27151)]
18:48:56.095301 run iperf3-cli_10.42.42.10(pid=27151): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
18:48:56.238047 run iperf3-cli_10.42.42.10(pid=27151): stderr:
| (launched: 2022-03-29_18:47:45.365519)
18:48:57.441464 run iperf3-cli_10.42.42.10(pid=27154): ERR: Terminated: ERROR {rc=1} [trial-2064↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=27154)]
18:48:57.585447 run iperf3-cli_10.42.42.10(pid=27154): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
18:48:57.727525 run iperf3-cli_10.42.42.10(pid=27154): stderr:
| (launched: 2022-03-29_18:47:45.642079)
18:48:58.912307 run iperf3-cli_10.42.42.10(pid=27157): ERR: Terminated: ERROR {rc=1} [trial-2064↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=27157)]
18:48:59.041297 run iperf3-cli_10.42.42.10(pid=27157): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
18:48:59.168709 run iperf3-cli_10.42.42.10(pid=27157): stderr:
| (launched: 2022-03-29_18:47:45.912502)
18:48:59.285650 run iperf3-srv_10.42.42.10(pid=26967): Terminating (SIGINT)
18:48:59.617276 run iperf3-srv_10.42.42.10(pid=26967): Terminated {rc=1}
18:48:59.807600 tst iperf3m4.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context2'"
18:48:59.863462 tst iperf3m4.py:8: ERR: KeyError: 'sum_sent' [trial-2064↪gprs:sysmo↪iperf3m4.py:8]
18:48:59.870281 tst iperf3m4.py:8: Test FAILED (294.8 sec)
18:48:59.907576 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26966): Terminating (SIGTERM)
18:48:59.949820 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26969): Terminating (SIGTERM)
18:48:59.990945 run iperf3-srv_10.42.42.10(pid=26971): Terminating (SIGTERM)
18:49:00.032065 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26972): Terminating (SIGTERM)
18:49:00.072958 run iperf3-srv_10.42.42.10(pid=26973): Terminating (SIGTERM)
18:49:00.113642 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26974): Terminating (SIGTERM)
18:49:00.154497 run iperf3-srv_10.42.42.10(pid=26975): Terminating (SIGTERM)
18:49:00.195714 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26978): Terminating (SIGTERM)
18:49:00.236537 run osmo-hlr_10.42.42.2(pid=26979): Terminating (SIGTERM)
18:49:00.278668 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26980): Terminating (SIGTERM)
18:49:00.320413 run osmo-stp_10.42.42.5(pid=26981): Terminating (SIGTERM)
18:49:00.374252 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26982): Terminating (SIGTERM)
18:49:00.415987 run osmo-ggsn_10.42.42.6(pid=26987): Terminating (SIGTERM)
18:49:00.463651 run pcap-recorder_any(filters='host 10.42.42.7')(pid=27008): Terminating (SIGTERM)
18:49:00.518975 run osmo-sgsn_10.42.42.7(pid=27009): Terminating (SIGTERM)
18:49:00.569651 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=27010): Terminating (SIGTERM)
18:49:00.622377 run osmo-msc_10.42.42.8(pid=27011): Terminating (SIGTERM)
18:49:00.676750 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=27012): Terminating (SIGTERM)
18:49:00.728862 run osmo-mgw_10.42.42.3(pid=27013): Terminating (SIGTERM)
18:49:00.789825 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=27014): Terminating (SIGTERM)
18:49:00.835908 run osmo-mgw_10.42.42.4(pid=27015): Terminating (SIGTERM)
18:49:00.881113 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=27016): Terminating (SIGTERM)
18:49:00.925531 run osmo-bsc_10.42.42.9(pid=27017): Terminating (SIGTERM)
18:49:00.972073 run osmo-bts-sysmo(pid=27026): Terminating (SIGTERM)
18:49:01.019565 run osmo-pcu(pid=27037): Terminating (SIGTERM)
18:49:01.065938 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136): Terminating (SIGTERM)
18:49:01.235041 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136)-killSignals.SIGTERM(pid=27181): Launched
18:49:02.330774 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136)-killSignals.SIGTERM(pid=27181): Terminated: ok {rc=0}
18:49:02.378634 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139): Terminating (SIGTERM)
18:49:02.535809 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139)-killSignals.SIGTERM(pid=27186): Launched
18:49:03.630145 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139)-killSignals.SIGTERM(pid=27186): Terminated: ok {rc=0}
18:49:03.678048 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142): Terminating (SIGTERM)
18:49:03.836292 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142)-killSignals.SIGTERM(pid=27189): Launched
18:49:04.932924 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142)-killSignals.SIGTERM(pid=27189): Terminated: ok {rc=0}
18:49:04.981063 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145): Terminating (SIGTERM)
18:49:05.135246 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145)-killSignals.SIGTERM(pid=27192): Launched
18:49:06.230408 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145)-killSignals.SIGTERM(pid=27192): Terminated: ok {rc=0}
18:49:06.244851 --- ParallelTerminationStrategy: PID 26966 died...
18:49:06.301943 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26966): Terminated: ok {rc=0}
18:49:06.312031 --- ParallelTerminationStrategy: PID 26969 died...
18:49:06.368133 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26969): Terminated: ok {rc=0}
18:49:06.378013 --- ParallelTerminationStrategy: PID 26971 died...
18:49:06.434019 run iperf3-srv_10.42.42.10(pid=26971): Terminated {rc=256}
18:49:06.444011 --- ParallelTerminationStrategy: PID 26972 died...
18:49:06.499901 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26972): Terminated: ok {rc=0}
18:49:06.509925 --- ParallelTerminationStrategy: PID 26973 died...
18:49:06.567695 run iperf3-srv_10.42.42.10(pid=26973): Terminated {rc=256}
18:49:06.578065 --- ParallelTerminationStrategy: PID 26974 died...
18:49:06.633678 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=26974): Terminated: ok {rc=0}
18:49:06.643538 --- ParallelTerminationStrategy: PID 26975 died...
18:49:06.698658 run iperf3-srv_10.42.42.10(pid=26975): Terminated {rc=256}
18:49:06.708741 --- ParallelTerminationStrategy: PID 26978 died...
18:49:06.763577 run pcap-recorder_any(filters='host 10.42.42.2')(pid=26978): Terminated: ok {rc=0}
18:49:06.774133 --- ParallelTerminationStrategy: PID 26979 died...
18:49:06.829543 run osmo-hlr_10.42.42.2(pid=26979): Terminated: ok {rc=0}
18:49:06.839409 --- ParallelTerminationStrategy: PID 26980 died...
18:49:06.894144 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=26980): Terminated: ok {rc=0}
18:49:06.904676 --- ParallelTerminationStrategy: PID 26981 died...
18:49:06.959973 run osmo-stp_10.42.42.5(pid=26981): Terminated: ok {rc=0}
18:49:06.969725 --- ParallelTerminationStrategy: PID 26982 died...
18:49:07.024763 run pcap-recorder_any(filters='host 10.42.42.6')(pid=26982): Terminated: ok {rc=0}
18:49:07.035152 --- ParallelTerminationStrategy: PID 26987 died...
18:49:07.090308 run osmo-ggsn_10.42.42.6(pid=26987): Terminated: ok {rc=0}
18:49:07.100170 --- ParallelTerminationStrategy: PID 27008 died...
18:49:07.155555 run pcap-recorder_any(filters='host 10.42.42.7')(pid=27008): Terminated: ok {rc=0}
18:49:07.165421 --- ParallelTerminationStrategy: PID 27009 died...
18:49:07.220355 run osmo-sgsn_10.42.42.7(pid=27009): Terminated: ok {rc=0}
18:49:07.230285 --- ParallelTerminationStrategy: PID 27010 died...
18:49:07.284993 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=27010): Terminated: ok {rc=0}
18:49:07.295594 --- ParallelTerminationStrategy: PID 27011 died...
18:49:07.350059 run osmo-msc_10.42.42.8(pid=27011): Terminated: ok {rc=0}
18:49:07.360544 --- ParallelTerminationStrategy: PID 27012 died...
18:49:07.415946 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=27012): Terminated: ok {rc=0}
18:49:07.426431 --- ParallelTerminationStrategy: PID 27013 died...
18:49:07.481897 run osmo-mgw_10.42.42.3(pid=27013): Terminated {rc=15}
18:49:07.492163 --- ParallelTerminationStrategy: PID 27014 died...
18:49:07.547887 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=27014): Terminated: ok {rc=0}
18:49:07.558064 --- ParallelTerminationStrategy: PID 27015 died...
18:49:07.617727 run osmo-mgw_10.42.42.4(pid=27015): Terminated {rc=15}
18:49:07.628416 --- ParallelTerminationStrategy: PID 27016 died...
18:49:07.685117 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=27016): Terminated: ok {rc=0}
18:49:07.695548 --- ParallelTerminationStrategy: PID 27017 died...
18:49:07.752413 run osmo-bsc_10.42.42.9(pid=27017): Terminated: ok {rc=0}
18:49:07.762885 --- ParallelTerminationStrategy: PID 27026 died...
18:49:07.819330 run osmo-bts-sysmo(pid=27026): Terminated: ok {rc=0}
18:49:07.829728 --- ParallelTerminationStrategy: PID 27037 died...
18:49:07.886471 run osmo-pcu(pid=27037): Terminated: ok {rc=0}
18:49:13.754422 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136): Terminating (SIGINT)
18:49:13.910607 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136)-killSignals.SIGINT(pid=27199): Launched
18:49:15.004329 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136)-killSignals.SIGINT(pid=27199): Terminated: ok {rc=0}
18:49:15.052756 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139): Terminating (SIGINT)
18:49:15.205594 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139)-killSignals.SIGINT(pid=27202): Launched
18:49:16.299606 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139)-killSignals.SIGINT(pid=27202): Terminated: ok {rc=0}
18:49:16.347359 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142): Terminating (SIGINT)
18:49:16.499120 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142)-killSignals.SIGINT(pid=27205): Launched
18:49:17.592566 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142)-killSignals.SIGINT(pid=27205): Terminated: ok {rc=0}
18:49:17.640904 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145): Terminating (SIGINT)
18:49:17.801929 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145)-killSignals.SIGINT(pid=27208): Launched
18:49:18.895014 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145)-killSignals.SIGINT(pid=27208): Terminated: ok {rc=0}
18:49:24.766681 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136): Terminating (SIGKILL)
18:49:24.924357 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136)-killSignals.SIGUSR1(pid=27211): Launched
18:49:26.018401 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27136)-killSignals.SIGUSR1(pid=27211): Terminated: ok {rc=0}
18:49:26.066160 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139): Terminating (SIGKILL)
18:49:26.223400 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139)-killSignals.SIGUSR1(pid=27214): Launched
18:49:27.318090 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27139)-killSignals.SIGUSR1(pid=27214): Terminated: ok {rc=0}
18:49:27.365812 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142): Terminating (SIGKILL)
18:49:27.524070 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142)-killSignals.SIGUSR1(pid=27217): Launched
18:49:28.617986 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27142)-killSignals.SIGUSR1(pid=27217): Terminated: ok {rc=0}
18:49:28.666018 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145): Terminating (SIGKILL)
18:49:28.818018 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145)-killSignals.SIGUSR1(pid=27220): Launched
18:49:29.912139 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=27145)-killSignals.SIGUSR1(pid=27220): Terminated: ok {rc=0}
18:49:30.144815 bus /gobi_4: Setting Online False
18:49:31.284004 bus /gobi_4: Setting Powered False
18:49:45.981582 bus /gobi_4: Setting Powered False
18:49:47.087576 bus /gobi_4: Setting Powered False
18:49:48.239710 bus /gobi_1: Setting Online False
18:49:49.363326 bus /gobi_1: Setting Powered False
18:49:57.133678 bus /gobi_1: Setting Powered False
18:49:58.240323 bus /gobi_1: Setting Powered False
18:49:59.392129 bus /gobi_0: Setting Online False
18:50:00.530743 bus /gobi_0: Setting Powered False
18:50:08.973464 bus /gobi_0: Setting Powered False
18:50:10.081358 bus /gobi_0: Setting Powered False
18:50:11.233130 bus /gobi_6: Setting Online False
18:50:12.367834 bus /gobi_6: Setting Powered False
18:50:20.982103 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'