![[Jenkins]](/jenkins/static/b68c8964/images/svgs/logo.svg){kind=logo}
Failed
gprs:sysmo.iperf3m4.py (from gprs_sysmo)
Failing for the past 6 builds
(Since Failed )
Stacktrace
'sum_sent'
Standard Output
----------------------------------------------
trial-2068 gprs:sysmo iperf3m4.py
----------------------------------------------
17:14:37.375226 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:14:37.386284 tst gprs:sysmo: Using 1 x bts (candidates: 1)
17:14:37.515099 tst iperf3m4.py: using LAC 4436
17:14:37.641436 tst iperf3m4.py: using RAC 101
17:14:37.767640 tst iperf3m4.py: using CellId 4436
17:14:37.894388 tst iperf3m4.py: using BVCI 4437
17:14:37.919798 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:14:37.931202 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:14:37.942302 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:14:37.953859 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:14:37.965274 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:14:37.976649 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:14:37.988337 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:14:37.999807 tst gprs:sysmo: Using 1 x ip_address (candidates: 9)
17:14:38.011485 tst gprs:sysmo: Using 1 x modem (candidates: 4)
17:14:38.133629 tst gprs:sysmo: Using 1 x modem (candidates: 4)
17:14:38.256946 tst gprs:sysmo: Using 1 x modem (candidates: 4)
17:14:38.379291 tst gprs:sysmo: Using 1 x modem (candidates: 4)
17:14:38.674580 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003...
17:14:38.812300 run iperf3-srv_10.42.42.10: Starting iperf3-srv
17:14:39.158461 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31920): Launched
17:14:39.444560 run iperf3-srv_10.42.42.10(pid=31921): Launched
17:14:39.645741 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004...
17:14:39.782930 run iperf3-srv_10.42.42.10: Starting iperf3-srv
17:14:40.123623 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31922): Launched
17:14:40.406873 run iperf3-srv_10.42.42.10(pid=31923): Launched
17:14:40.606889 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005...
17:14:40.742952 run iperf3-srv_10.42.42.10: Starting iperf3-srv
17:14:41.083237 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31924): Launched
17:14:41.369070 run iperf3-srv_10.42.42.10(pid=31925): Launched
17:14:41.569769 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006...
17:14:41.706310 run iperf3-srv_10.42.42.10: Starting iperf3-srv
17:14:42.046930 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31926): Launched
17:14:42.331392 run iperf3-srv_10.42.42.10(pid=31927): Launched
17:14:42.531910 tst iperf3m4.py:8: start network...
17:14:42.668666 run osmo-hlr_10.42.42.2: Starting osmo-hlr
17:14:43.187068 run create_hlr_db(pid=31928): Launched
17:14:43.375618 bus /gobi_6: Setting Powered False
17:14:44.398536 run create_hlr_db(pid=31928): Terminated: ok {rc=0}
17:14:44.837562 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31930): Launched
17:14:45.167796 run osmo-hlr_10.42.42.2(pid=31931): Launched
17:14:45.284671 run osmo-stp_10.42.42.5: Starting osmo-stp
17:14:45.855637 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31932): Launched
17:14:46.183931 run osmo-stp_10.42.42.5(pid=31933): Launched
17:14:46.299813 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn
17:14:46.869662 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31934): Launched
17:14:46.985772 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
17:14:47.272927 run patchelf(pid=31935): Launched
17:14:48.468440 run patchelf(pid=31935): Terminated: ok {rc=0}
17:14:48.588350 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
17:14:48.878843 run osmo-gsm-tester_setcap_net_admin.sh(pid=31936): Launched
17:14:50.074331 run osmo-gsm-tester_setcap_net_admin.sh(pid=31936): Terminated: ok {rc=0}
17:14:50.405663 run osmo-ggsn_10.42.42.6(pid=31939): Launched
17:14:50.664550 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn
17:14:51.258541 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31965): Launched
17:14:51.589747 run osmo-sgsn_10.42.42.7(pid=31966): Launched
17:14:51.710270 run osmo-msc_10.42.42.8: Starting osmo-msc
17:14:52.319333 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31967): Launched
17:14:52.650246 run osmo-msc_10.42.42.8(pid=31970): Launched
17:14:55.825263 run osmo-mgw_10.42.42.3: Starting osmo-mgw
17:14:56.401712 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31971): Launched
17:14:56.711143 run osmo-mgw_10.42.42.3(pid=31972): Launched
17:14:56.819498 run osmo-mgw_10.42.42.4: Starting osmo-mgw
17:14:57.367514 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31973): Launched
17:14:57.698858 run osmo-mgw_10.42.42.4(pid=31974): Launched
17:14:57.816418 run osmo-bsc_10.42.42.9: Starting osmo-bsc
17:14:58.507602 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31979): Launched
17:14:58.838948 run osmo-bsc_10.42.42.9(pid=31980): Launched
17:15:05.119992 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
17:15:05.706309 run rm-remote-dir(pid=31989): Launched
17:15:07.898210 run rm-remote-dir(pid=31989): Terminated: ok {rc=0}
17:15:08.229481 run mk-remote-dir(pid=31990): Launched
17:15:09.416421 run mk-remote-dir(pid=31990): Terminated: ok {rc=0}
17:15:09.694659 run scp-inst-to-remote(pid=31991): Launched
17:15:21.905155 run scp-inst-to-remote(pid=31991): Terminated: ok {rc=0}
17:15:22.251618 run mk-remote-dir(pid=31993): Launched
17:15:23.449475 run mk-remote-dir(pid=31993): Terminated: ok {rc=0}
17:15:23.747126 run scp-cfg-to-remote(pid=31994): Launched
17:15:24.944935 run scp-cfg-to-remote(pid=31994): Terminated: ok {rc=0}
17:15:25.288558 run reload-dsp-firmware(pid=31996): Launched
17:15:27.486814 run reload-dsp-firmware(pid=31996): Terminated: ok {rc=0}
17:15:27.820564 run osmo-bts-sysmo(pid=31997): Launched
17:15:32.137902 tst iperf3m4.py:8: Waiting for bts to be ready...
17:15:33.790094 run rm-remote-dir(pid=31998): Launched
17:15:35.987803 run rm-remote-dir(pid=31998): Terminated: ok {rc=0}
17:15:36.326520 run mk-remote-dir(pid=31999): Launched
17:15:38.523256 run mk-remote-dir(pid=31999): Terminated: ok {rc=0}
17:15:38.812242 run scp-inst-to-bts(pid=32000): Launched
17:15:52.029747 run scp-inst-to-bts(pid=32000): Terminated: ok {rc=0}
17:15:52.371591 run mk-remote-run-dir(pid=32002): Launched
17:15:54.557577 run mk-remote-run-dir(pid=32002): Terminated: ok {rc=0}
17:15:54.829622 run scp-cfg-to-sysmobts(pid=32003): Launched
17:15:57.012493 run scp-cfg-to-sysmobts(pid=32003): Terminated: ok {rc=0}
17:15:57.328935 run osmo-pcu(pid=32005): Launched
17:15:57.450324 tst iperf3m4.py: using MSISDN 4031
17:15:57.632025 bus /gobi_6: Setting Powered True
17:16:01.127131 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='4031', subscriber_id=1}
17:16:01.402032 bus /gobi_6: Setting Online False
17:16:02.596852 bus /gobi_6: Setting Powered False
17:16:09.328878 bus /gobi_6: Setting Powered True
17:16:10.719171 bus /gobi_6: Setting Online True
17:16:13.197753 tst /gobi_6: Connect to ('901', '70')
17:16:13.481021 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648'
17:16:13.603681 tst iperf3m4.py: using MSISDN 4032
17:16:13.837497 bus /gobi_0: Setting Powered True
17:16:17.335813 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='4032', subscriber_id=2}
17:16:17.605643 bus /gobi_0: Setting Online False
17:16:18.800941 bus /gobi_0: Setting Powered False
17:16:26.232283 bus /gobi_0: Setting Powered True
17:16:27.616039 bus /gobi_0: Setting Online True
17:16:30.202542 tst /gobi_0: Connect to ('901', '70')
17:16:30.496218 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873'
17:16:30.626622 tst iperf3m4.py: using MSISDN 4033
17:16:30.867503 tst /gobi_6: Already registered with network ('901', '70')
17:16:31.043549 bus /gobi_1: Setting Powered True
17:16:34.632994 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023848', msisdn='4033', subscriber_id=3}
17:16:34.960004 bus /gobi_1: Setting Online False
17:16:36.143982 bus /gobi_1: Setting Powered False
17:16:43.217708 bus /gobi_1: Setting Powered True
17:16:44.620353 bus /gobi_1: Setting Online True
17:16:45.644132 tst /gobi_0: Already registered with network ('901', '70')
17:16:47.259909 tst /gobi_1: Connect to ('901', '70')
17:16:47.532655 tst /gobi_1: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024272103'
17:16:47.671770 tst iperf3m4.py: using MSISDN 4034
17:16:47.844074 bus /gobi_4: Setting Powered True
17:16:51.343284 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023849', msisdn='4034', subscriber_id=4}
17:16:51.641751 bus /gobi_4: Setting Online False
17:16:52.829315 bus /gobi_4: Setting Powered False
17:17:00.233993 bus /gobi_4: Setting Powered True
17:17:01.607831 bus /gobi_4: Setting Online True
17:17:04.079583 tst /gobi_4: Connect to ('901', '70')
17:17:04.368683 tst /gobi_4: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A08E2G 1 [Jul 21 2016 10:00:50]', Serial: '868323025161511'
17:17:04.569620 tst iperf3m4.py:8: waiting for modems to attach...
17:17:05.223565 tst /gobi_1: Already registered with network ('901', '70')
17:17:22.516685 tst /gobi_4: Already registered with network ('901', '70')
17:17:27.452259 tst iperf3m4.py:8: waiting for modems to attach to data services...
17:17:30.569450 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
17:17:30.756584 tst iperf3m4.py:8: Setting up data plan for "'/gobi_6/context2'"
17:17:31.118816 run move_netns(pid=32014): Launched
17:17:32.303559 run move_netns(pid=32014): Terminated: ok {rc=0}
17:17:32.630373 run ifup(pid=32017): Launched
17:17:33.814641 run ifup(pid=32017): Terminated: ok {rc=0}
17:17:34.138562 run dhcp(pid=32020): Launched
17:17:35.321703 run dhcp(pid=32020): Terminated: ok {rc=0}
17:17:38.316781 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
17:17:38.518945 tst iperf3m4.py:8: Setting up data plan for "'/gobi_0/context2'"
17:17:38.899491 run move_netns(pid=32038): Launched
17:17:40.095493 run move_netns(pid=32038): Terminated: ok {rc=0}
17:17:40.444409 run ifup(pid=32041): Launched
17:17:41.641397 run ifup(pid=32041): Terminated: ok {rc=0}
17:17:41.989606 run dhcp(pid=32044): Launched
17:17:43.175133 run dhcp(pid=32044): Terminated: ok {rc=0}
17:17:51.495222 tst /gobi_1: context activated {apn='inet46', path='/gobi_1/context13', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.3', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='rf303c5eb721dad', Method='static', Netmask='255.255.255.248'}, Type='internet', Username='ogt'}, user='ogt'}
17:17:51.681921 tst iperf3m4.py:8: Setting up data plan for "'/gobi_1/context13'"
17:17:52.041626 run move_netns(pid=32062): Launched
17:17:53.226317 run move_netns(pid=32062): Terminated: ok {rc=0}
17:17:53.560797 run ifup(pid=32065): Launched
17:17:54.758219 run ifup(pid=32065): Terminated: ok {rc=0}
17:17:55.108444 run dhcp(pid=32068): Launched
17:17:56.304482 run dhcp(pid=32068): Terminated: ok {rc=0}
17:17:59.302633 tst /gobi_4: context activated {apn='inet46', path='/gobi_4/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.4', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.5', Interface='r9d788e55295d1b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
17:17:59.489518 tst iperf3m4.py:8: Setting up data plan for "'/gobi_4/context2'"
17:17:59.845418 run move_netns(pid=32088): Launched
17:18:01.040571 run move_netns(pid=32088): Terminated: ok {rc=0}
17:18:01.390233 run ifup(pid=32092): Launched
17:18:02.587752 run ifup(pid=32092): Terminated: ok {rc=0}
17:18:02.935265 run dhcp(pid=32095): Launched
17:18:04.129706 run dhcp(pid=32095): Terminated: ok {rc=0}
17:18:04.338437 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'"
17:18:04.478390 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s)
17:18:04.883484 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111): Launched
17:18:05.170637 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'"
17:18:05.310540 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s)
17:18:05.738351 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114): Launched
17:18:06.008729 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5005 through "'/gobi_1/context13'"
17:18:06.148474 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5005 (proto=tcp,time=60s)
17:18:06.573060 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117): Launched
17:18:06.841521 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5006 through "'/gobi_4/context2'"
17:18:06.981165 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5006 (proto=tcp,time=60s)
17:18:07.385205 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120): Launched
17:18:07.724300 run iperf3-cli_10.42.42.10(pid=32123): Launched
17:18:08.016172 run iperf3-cli_10.42.42.10(pid=32126): Launched
17:18:08.313218 run iperf3-cli_10.42.42.10(pid=32129): Launched
17:18:08.603967 run iperf3-cli_10.42.42.10(pid=32132): Launched
17:19:22.948132 run iperf3-cli_10.42.42.10(pid=32123): ERR: Terminated: ERROR {rc=1} [trial-2068↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32123)]
17:19:23.093385 run iperf3-cli_10.42.42.10(pid=32123): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
17:19:23.234881 run iperf3-cli_10.42.42.10(pid=32123): stderr:
| (launched: 2022-12-01_17:18:07.606904)
17:19:24.438392 run iperf3-cli_10.42.42.10(pid=32126): ERR: Terminated: ERROR {rc=1} [trial-2068↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32126)]
17:19:24.575796 run iperf3-cli_10.42.42.10(pid=32126): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
17:19:24.705001 run iperf3-cli_10.42.42.10(pid=32126): stderr:
| (launched: 2022-12-01_17:18:07.896232)
17:19:25.888642 run iperf3-cli_10.42.42.10(pid=32129): ERR: Terminated: ERROR {rc=1} [trial-2068↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32129)]
17:19:26.019223 run iperf3-cli_10.42.42.10(pid=32129): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
17:19:26.149901 run iperf3-cli_10.42.42.10(pid=32129): stderr:
| (launched: 2022-12-01_17:18:08.190002)
17:19:27.333179 run iperf3-cli_10.42.42.10(pid=32132): ERR: Terminated: ERROR {rc=1} [trial-2068↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=32132)]
17:19:27.462566 run iperf3-cli_10.42.42.10(pid=32132): stdout:
| "bits_per_second": 0,
| "retransmits": 0,
| "omitted": false
| }
| }],
| "end": {
| },
| "error": "error - control socket has closed unexpectedly"
| }
| child exited with 1
17:19:27.590839 run iperf3-cli_10.42.42.10(pid=32132): stderr:
| (launched: 2022-12-01_17:18:08.474417)
17:19:27.708008 run iperf3-srv_10.42.42.10(pid=31921): Terminating (SIGINT)
17:19:28.040310 run iperf3-srv_10.42.42.10(pid=31921): Terminated {rc=1}
17:19:28.228023 tst iperf3m4.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context2'"
17:19:28.284101 tst iperf3m4.py:8: ERR: KeyError: 'sum_sent' [trial-2068↪gprs:sysmo↪iperf3m4.py:8]
17:19:28.291204 tst iperf3m4.py:8: Test FAILED (291.0 sec)
17:19:28.328837 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31920): Terminating (SIGTERM)
17:19:28.371582 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31922): Terminating (SIGTERM)
17:19:28.413463 run iperf3-srv_10.42.42.10(pid=31923): Terminating (SIGTERM)
17:19:28.454747 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31924): Terminating (SIGTERM)
17:19:28.495625 run iperf3-srv_10.42.42.10(pid=31925): Terminating (SIGTERM)
17:19:28.536459 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31926): Terminating (SIGTERM)
17:19:28.577475 run iperf3-srv_10.42.42.10(pid=31927): Terminating (SIGTERM)
17:19:28.618935 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31930): Terminating (SIGTERM)
17:19:28.659656 run osmo-hlr_10.42.42.2(pid=31931): Terminating (SIGTERM)
17:19:28.702306 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31932): Terminating (SIGTERM)
17:19:28.745253 run osmo-stp_10.42.42.5(pid=31933): Terminating (SIGTERM)
17:19:28.801427 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31934): Terminating (SIGTERM)
17:19:28.847004 run osmo-ggsn_10.42.42.6(pid=31939): Terminating (SIGTERM)
17:19:28.897484 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31965): Terminating (SIGTERM)
17:19:28.956047 run osmo-sgsn_10.42.42.7(pid=31966): Terminating (SIGTERM)
17:19:29.003929 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31967): Terminating (SIGTERM)
17:19:29.078739 run osmo-msc_10.42.42.8(pid=31970): Terminating (SIGTERM)
17:19:29.140575 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31971): Terminating (SIGTERM)
17:19:29.188756 run osmo-mgw_10.42.42.3(pid=31972): Terminating (SIGTERM)
17:19:29.234774 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31973): Terminating (SIGTERM)
17:19:29.275617 run osmo-mgw_10.42.42.4(pid=31974): Terminating (SIGTERM)
17:19:29.316671 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31979): Terminating (SIGTERM)
17:19:29.359776 run osmo-bsc_10.42.42.9(pid=31980): Terminating (SIGTERM)
17:19:29.402135 run osmo-bts-sysmo(pid=31997): Terminating (SIGTERM)
17:19:29.444496 run osmo-pcu(pid=32005): Terminating (SIGTERM)
17:19:29.485365 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111): Terminating (SIGTERM)
17:19:29.668078 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111)-killSignals.SIGTERM(pid=32159): Launched
17:19:30.765640 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111)-killSignals.SIGTERM(pid=32159): Terminated: ok {rc=0}
17:19:30.814215 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114): Terminating (SIGTERM)
17:19:30.975151 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114)-killSignals.SIGTERM(pid=32164): Launched
17:19:32.071154 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114)-killSignals.SIGTERM(pid=32164): Terminated: ok {rc=0}
17:19:32.119324 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117): Terminating (SIGTERM)
17:19:32.276107 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117)-killSignals.SIGTERM(pid=32167): Launched
17:19:33.371824 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117)-killSignals.SIGTERM(pid=32167): Terminated: ok {rc=0}
17:19:33.420195 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120): Terminating (SIGTERM)
17:19:33.576010 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120)-killSignals.SIGTERM(pid=32170): Launched
17:19:34.672226 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120)-killSignals.SIGTERM(pid=32170): Terminated: ok {rc=0}
17:19:34.686674 --- ParallelTerminationStrategy: PID 31920 died...
17:19:34.744028 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31920): Terminated: ok {rc=0}
17:19:34.754454 --- ParallelTerminationStrategy: PID 31922 died...
17:19:34.811387 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31922): Terminated: ok {rc=0}
17:19:34.821484 --- ParallelTerminationStrategy: PID 31923 died...
17:19:34.878272 run iperf3-srv_10.42.42.10(pid=31923): Terminated {rc=256}
17:19:34.888358 --- ParallelTerminationStrategy: PID 31924 died...
17:19:34.945249 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31924): Terminated: ok {rc=0}
17:19:34.955495 --- ParallelTerminationStrategy: PID 31925 died...
17:19:35.012360 run iperf3-srv_10.42.42.10(pid=31925): Terminated {rc=256}
17:19:35.022437 --- ParallelTerminationStrategy: PID 31926 died...
17:19:35.079176 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=31926): Terminated: ok {rc=0}
17:19:35.089245 --- ParallelTerminationStrategy: PID 31927 died...
17:19:35.145683 run iperf3-srv_10.42.42.10(pid=31927): Terminated {rc=256}
17:19:35.155734 --- ParallelTerminationStrategy: PID 31930 died...
17:19:35.212872 run pcap-recorder_any(filters='host 10.42.42.2')(pid=31930): Terminated: ok {rc=0}
17:19:35.223061 --- ParallelTerminationStrategy: PID 31931 died...
17:19:35.279521 run osmo-hlr_10.42.42.2(pid=31931): Terminated: ok {rc=0}
17:19:35.289553 --- ParallelTerminationStrategy: PID 31932 died...
17:19:35.346194 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=31932): Terminated: ok {rc=0}
17:19:35.356271 --- ParallelTerminationStrategy: PID 31933 died...
17:19:35.412972 run osmo-stp_10.42.42.5(pid=31933): Terminated: ok {rc=0}
17:19:35.423182 --- ParallelTerminationStrategy: PID 31934 died...
17:19:35.479689 run pcap-recorder_any(filters='host 10.42.42.6')(pid=31934): Terminated: ok {rc=0}
17:19:35.489936 --- ParallelTerminationStrategy: PID 31939 died...
17:19:35.546370 run osmo-ggsn_10.42.42.6(pid=31939): Terminated: ok {rc=0}
17:19:35.556434 --- ParallelTerminationStrategy: PID 31965 died...
17:19:35.613098 run pcap-recorder_any(filters='host 10.42.42.7')(pid=31965): Terminated: ok {rc=0}
17:19:35.623412 --- ParallelTerminationStrategy: PID 31966 died...
17:19:35.680270 run osmo-sgsn_10.42.42.7(pid=31966): Terminated: ok {rc=0}
17:19:35.690380 --- ParallelTerminationStrategy: PID 31967 died...
17:19:35.746898 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=31967): Terminated: ok {rc=0}
17:19:35.756975 --- ParallelTerminationStrategy: PID 31970 died...
17:19:35.813457 run osmo-msc_10.42.42.8(pid=31970): Terminated: ok {rc=0}
17:19:35.823532 --- ParallelTerminationStrategy: PID 31971 died...
17:19:35.880623 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=31971): Terminated: ok {rc=0}
17:19:35.891059 --- ParallelTerminationStrategy: PID 31972 died...
17:19:35.947771 run osmo-mgw_10.42.42.3(pid=31972): Terminated {rc=15}
17:19:35.958179 --- ParallelTerminationStrategy: PID 31973 died...
17:19:36.015154 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=31973): Terminated: ok {rc=0}
17:19:36.025633 --- ParallelTerminationStrategy: PID 31974 died...
17:19:36.082441 run osmo-mgw_10.42.42.4(pid=31974): Terminated {rc=15}
17:19:36.092889 --- ParallelTerminationStrategy: PID 31979 died...
17:19:36.149481 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=31979): Terminated: ok {rc=0}
17:19:36.160047 --- ParallelTerminationStrategy: PID 31980 died...
17:19:36.216530 run osmo-bsc_10.42.42.9(pid=31980): Terminated: ok {rc=0}
17:19:36.227135 --- ParallelTerminationStrategy: PID 31997 died...
17:19:36.284311 run osmo-bts-sysmo(pid=31997): Terminated: ok {rc=0}
17:19:36.295277 --- ParallelTerminationStrategy: PID 32005 died...
17:19:36.352339 run osmo-pcu(pid=32005): Terminated: ok {rc=0}
17:19:42.220100 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111): Terminating (SIGINT)
17:19:42.372276 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111)-killSignals.SIGINT(pid=32173): Launched
17:19:43.468682 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111)-killSignals.SIGINT(pid=32173): Terminated: ok {rc=0}
17:19:43.517009 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114): Terminating (SIGINT)
17:19:43.672889 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114)-killSignals.SIGINT(pid=32176): Launched
17:19:44.763899 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114)-killSignals.SIGINT(pid=32176): Terminated: ok {rc=0}
17:19:44.810141 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117): Terminating (SIGINT)
17:19:44.958850 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117)-killSignals.SIGINT(pid=32179): Launched
17:19:46.049472 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117)-killSignals.SIGINT(pid=32179): Terminated: ok {rc=0}
17:19:46.096043 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120): Terminating (SIGINT)
17:19:46.248702 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120)-killSignals.SIGINT(pid=32182): Launched
17:19:47.341350 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120)-killSignals.SIGINT(pid=32182): Terminated: ok {rc=0}
17:19:53.211245 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111): Terminating (SIGKILL)
17:19:53.361896 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111)-killSignals.SIGUSR1(pid=32185): Launched
17:19:54.452897 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32111)-killSignals.SIGUSR1(pid=32185): Terminated: ok {rc=0}
17:19:54.498750 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114): Terminating (SIGKILL)
17:19:54.647461 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114)-killSignals.SIGUSR1(pid=32188): Launched
17:19:55.737783 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32114)-killSignals.SIGUSR1(pid=32188): Terminated: ok {rc=0}
17:19:55.783910 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117): Terminating (SIGKILL)
17:19:55.936463 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117)-killSignals.SIGUSR1(pid=32191): Launched
17:19:57.027475 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32117)-killSignals.SIGUSR1(pid=32191): Terminated: ok {rc=0}
17:19:57.073124 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120): Terminating (SIGKILL)
17:19:57.226571 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120)-killSignals.SIGUSR1(pid=32194): Launched
17:19:58.318300 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=32120)-killSignals.SIGUSR1(pid=32194): Terminated: ok {rc=0}
17:19:58.564931 bus /gobi_4: Setting Online False
17:19:59.741166 bus /gobi_4: Setting Powered False
17:20:07.971683 bus /gobi_4: Setting Powered False
17:20:09.073761 bus /gobi_4: Setting Powered False
17:20:10.230180 bus /gobi_1: Setting Online False
17:20:11.345804 bus /gobi_1: Setting Powered False
17:20:19.087951 bus /gobi_1: Setting Powered False
17:20:20.191901 bus /gobi_1: Setting Powered False
17:20:21.348940 bus /gobi_0: Setting Online False
17:20:22.479410 bus /gobi_0: Setting Powered False
17:20:30.975399 bus /gobi_0: Setting Powered False
17:20:32.083300 bus /gobi_0: Setting Powered False
17:20:33.235312 bus /gobi_6: Setting Online False
17:20:34.353313 bus /gobi_6: Setting Powered False
17:20:42.151863 bus /gobi_6: Setting Powered False
Standard Error
Traceback (most recent call last):
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
self.path)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
spec.loader.exec_module( importlib.util.module_from_spec(spec) )
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/iperf3m4.py", line 8, in <module>
setup_run_iperf3_test_parallel(4)
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 105, in setup_run_iperf3_test_parallel
print_results(clients[i], servers[i])
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/lib/testlib.py", line 5, in print_results
cli.print_results()
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 342, in print_results
print_result_node_tcp(self.get_results(), 'client')
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 72, in print_result_node_tcp
raise e
File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/iperf3.py", line 65, in print_result_node_tcp
sent = result['end']['sum_sent']
KeyError: 'sum_sent'