![[Jenkins]](/jenkins/static/342c126a/images/svgs/logo.svg){kind=logo}
Triggered by Gerrit: https://gerrit.osmocom.org/c/osmo-gsm-tester/+/27319 Running as SYSTEM Building remotely on osmo-gsm-tester-prod (osmo-gsm-tester) in workspace /home/jenkins/workspace/osmo-gsm-tester_gerrit The recommended git tool is: NONE using credential d5eda5e9-b59d-44ba-88d2-43473cb6e42d > git rev-parse --resolve-git-dir /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/.git # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url ssh://jenkins@gerrit.osmocom.org:29418/osmo-gsm-tester # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Fetching upstream changes from ssh://jenkins@gerrit.osmocom.org:29418/osmo-gsm-tester > git --version # timeout=10 > git --version # 'git version 2.20.1' using GIT_SSH to set credentials > git fetch --tags --force --progress -- ssh://jenkins@gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/19/27319/1 # timeout=10 > git rev-parse ae8aa17301137f70e586478f9db5e19ab5380ea9^{commit} # timeout=10 JENKINS-19022: warning: possible memory leak due to Git plugin usage; see: https://plugins.jenkins.io/git/#remove-git-plugin-buildsbybranch-builddata-script Checking out Revision ae8aa17301137f70e586478f9db5e19ab5380ea9 (refs/changes/19/27319/1) > git config core.sparsecheckout # timeout=10 > git checkout -f ae8aa17301137f70e586478f9db5e19ab5380ea9 # timeout=10 Commit message: "jenkins-build-osmo-hnbgw: Build osmo-mgw as dependency" > git rev-parse FETCH_HEAD^{commit} # timeout=10 > git rev-list --no-walk b4cacca4537e830a048528d197d97b2e0ad2d06b # timeout=10 [osmo-gsm-tester_gerrit] $ /bin/sh -xe /tmp/jenkins15543178365045826164.sh + rm -f osmo-bsc.build-15008.tgz osmo-bts.build-6308.tgz osmo-ggsn.build-5545.tgz osmo-hlr.build-7196.tgz osmo-mgw.build-7386.tgz osmo-bsc.build-15008.md5 osmo-bts.build-6308.md5 osmo-ggsn.build-5545.md5 osmo-hlr.build-7196.md5 osmo-mgw.build-7386.md5 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-bsc" build number 15317 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-bts" build number 6453 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-ggsn" build number 5693 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-hlr" build number 7389 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-mgw" build number 7582 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-hnbgw" build number 340 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-msc" build number 25551 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-pcu" build number 4468 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-sgsn" build number 25820 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-trx" build number 3723 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-nitb" build number 11093 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-stp" build number 7910 Copied 2 artifacts from "osmo-gsm-tester_build-osmocom-bb" build number 3951 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-bts-sysmo" build number 6983 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-pcu-sysmo" build number 4507 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-bts-oc2g" build number 4479 Copied 2 artifacts from "osmo-gsm-tester_build-osmo-pcu-oc2g" build number 3226 Copied 2 artifacts from "osmo-gsm-tester_build-open5gs" build number 8 Copied 2 artifacts from "osmo-gsm-tester_build-srslte" build number 19 [osmo-gsm-tester_gerrit] $ /bin/sh /tmp/jenkins3240704784289049152.sh + unlink osmo-gsm-tester/sysmocom/resources.conf unlink: cannot unlink 'osmo-gsm-tester/sysmocom/resources.conf': No such file or directory + true + ln -s resources.conf.prod osmo-gsm-tester/sysmocom/resources.conf + export OSMO_GSM_TESTER_OPTS=-s nitb_sms:sysmo -s sms:sysmo -s gprs:sysmo + ./osmo-gsm-tester/contrib/jenkins-make-check-and-run.sh + base=/home/jenkins/workspace/osmo-gsm-tester_gerrit + run_script=osmo-gsm-tester/contrib/jenkins-run.sh + test -x osmo-gsm-tester/contrib/jenkins-run.sh + cd osmo-gsm-tester + make deps ./contrib/check_dependencies.py Skip checking modules: [] Changing workdir dir to /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester importing files in directory /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_ms_driver importing src.osmo_ms_driver.location_update_test importing src.osmo_ms_driver.event_server importing src.osmo_ms_driver.cdf importing src.osmo_ms_driver.__main__ importing src.osmo_ms_driver.__init__ importing src.osmo_ms_driver.test_support importing src.osmo_ms_driver.starter importing src.osmo_ms_driver.simple_loop importing files in directory /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core importing src.osmo_gsm_tester.core.util importing src.osmo_gsm_tester.core.schema importing src.osmo_gsm_tester.core.trial importing src.osmo_gsm_tester.core.template importing src.osmo_gsm_tester.core.config importing src.osmo_gsm_tester.core.event_loop importing src.osmo_gsm_tester.core.log importing src.osmo_gsm_tester.core.scenario importing src.osmo_gsm_tester.core.remote importing src.osmo_gsm_tester.core.__init__ importing src.osmo_gsm_tester.core.process importing src.osmo_gsm_tester.core.report importing src.osmo_gsm_tester.core.test importing src.osmo_gsm_tester.core.resource importing src.osmo_gsm_tester.core.suite importing files in directory /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj importing src.osmo_gsm_tester.obj.android_apn importing src.osmo_gsm_tester.obj.bts importing src.osmo_gsm_tester.obj.mgcpgw_osmo importing src.osmo_gsm_tester.obj.sgsn_osmo importing src.osmo_gsm_tester.obj.epc_srs importing src.osmo_gsm_tester.obj.bts_nanobts importing src.osmo_gsm_tester.obj.enb_srs importing src.osmo_gsm_tester.obj.esme importing src.osmo_gsm_tester.obj.qc_diag importing src.osmo_gsm_tester.obj.stp_osmo importing src.osmo_gsm_tester.obj.ms_ofono importing src.osmo_gsm_tester.obj.ms_android importing src.osmo_gsm_tester.obj.osmo_vty importing src.osmo_gsm_tester.obj.hlr_osmo importing src.osmo_gsm_tester.obj.enb_amarisoft importing src.osmo_gsm_tester.obj.android_host importing src.osmo_gsm_tester.obj.sgwu_open5gs importing src.osmo_gsm_tester.obj.powersupply importing src.osmo_gsm_tester.obj.iperf3 importing src.osmo_gsm_tester.obj.epc importing src.osmo_gsm_tester.obj.rfemu_minicircuits importing src.osmo_gsm_tester.obj.bts_osmo importing src.osmo_gsm_tester.obj.upf_open5gs importing src.osmo_gsm_tester.obj.nitb_osmo importing src.osmo_gsm_tester.obj.android_bitrate_monitor importing src.osmo_gsm_tester.obj.pcu_oc2g importing src.osmo_gsm_tester.obj.ms_srs importing src.osmo_gsm_tester.obj.ms_driver importing src.osmo_gsm_tester.obj.bts_sysmo importing src.osmo_gsm_tester.obj.rfemu_srsenb_stdin importing src.osmo_gsm_tester.obj.pcu importing src.osmo_gsm_tester.obj.osmocon importing src.osmo_gsm_tester.obj.epc_amarisoft importing src.osmo_gsm_tester.obj.ggsn_osmo importing src.osmo_gsm_tester.obj.enb importing src.osmo_gsm_tester.obj.rfemu importing src.osmo_gsm_tester.obj.bts_octphy importing src.osmo_gsm_tester.obj.pcap_recorder importing src.osmo_gsm_tester.obj.sms importing src.osmo_gsm_tester.obj.bts_oc2g importing src.osmo_gsm_tester.obj.stress importing src.osmo_gsm_tester.obj.smsc importing src.osmo_gsm_tester.obj.msc_osmo importing src.osmo_gsm_tester.obj.sgwc_open5gs importing src.osmo_gsm_tester.obj.osmo_ctrl importing src.osmo_gsm_tester.obj.bts_osmotrx importing src.osmo_gsm_tester.obj.rfemu_amarisoftctrl importing src.osmo_gsm_tester.obj.ms importing src.osmo_gsm_tester.obj.powersupply_intellinet importing src.osmo_gsm_tester.obj.ms_amarisoft importing src.osmo_gsm_tester.obj.srslte_common importing src.osmo_gsm_tester.obj.powersupply_sispm importing src.osmo_gsm_tester.obj.gnuradio_zmq_broker importing src.osmo_gsm_tester.obj.pcu_sysmo importing src.osmo_gsm_tester.obj.bts_osmovirtual importing src.osmo_gsm_tester.obj.hss_open5gs importing src.osmo_gsm_tester.obj.mgw_osmo importing src.osmo_gsm_tester.obj.smf_open5gs importing src.osmo_gsm_tester.obj.pcrf_open5gs importing src.osmo_gsm_tester.obj.mme_open5gs importing src.osmo_gsm_tester.obj.bsc_osmo importing src.osmo_gsm_tester.obj.epc_open5gs importing src.osmo_gsm_tester.obj.run_node importing src.osmo_gsm_tester.obj.pcu_osmo importing src.osmo_gsm_tester.obj.rfemu_gnuradio_zmq importing src.osmo_gsm_tester.obj.ms_osmo_mobile importing files in directory /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester importing src.osmo_gsm_tester.testenv importing src.osmo_gsm_tester.__init__ Importing dependencies ok, all installed Retreiving list of imported modules... + make check make -C selftest check make[1]: Entering directory '/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest' echo "export PYTHONPATH=\"/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/../src\"" > set_pythonpath ./all_tests.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/cdf_test/cdf_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/config_test/config_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/lock_test/lock_test.sh /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/log_test/log_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/process_test/process_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/rate_ctrs_test/rate_ctrs_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/report_test/report_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/resource_test/resource_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/scenario_test/scenario_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/schema_test/schema_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/sms_test/sms_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/suite_test/suite_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/template_test/template_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/trial_test/trial_test.py /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest/util_test/util_test.py 15 tests ok make[1]: Leaving directory '/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/selftest' make check: success + ./contrib/jenkins-build-manuals.sh + base=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester + export OSMO_GSM_MANUALS_DIR=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals + [ -d /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/doc/manuals ] + [ -d /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals ] + git -C /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals pull From https://git.osmocom.org/osmo-gsm-manuals 738c084..b1318e3 master -> origin/master Updating 738c084..b1318e3 Fast-forward common/chapters/counters-overview.adoc | 74 +++++++++++++++++++++++++++++++ common/chapters/gb-ns2-configuration.adoc | 15 ++----- common/chapters/gb-ns2.adoc | 5 --- 3 files changed, 77 insertions(+), 17 deletions(-) + git -C /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals checkout -f HEAD Your branch is up to date with 'origin/master'. + temp=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp + [ -d /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp ] + cp -r /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/doc/manuals /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp + cd /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp + make OSMO_GSM_MANUALS_DIR: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals (from env) ln -s /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build build ln -s /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/common common # a2x can't use a different output file. To support out-of-tree builds, # we create a symlink at /srcfile.adoc pointing at # /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/srcfile.adoc. osmo-gsm-tester-manual.adoc is the /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/srcfile.adoc, # is like basename from coreutils, and is WD. if ! [ -f osmo-gsm-tester-manual.adoc ]; then \ ln -s osmo-gsm-tester-manual.adoc osmo-gsm-tester-manual.adoc; \ fi # TEXINPUTS: find LaTeX includes like \includegraphics{./common/images/sysmocom.pdf} # in /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals. TEXINPUTS="/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals" \ a2x -vv -L --asciidoc-opts="-f /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/mscgen-filter.conf -f /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/diag-filter.conf -f /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/docinfo-releaseinfo.conf -a srcdir='/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp' -a commondir='/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/common'" --dblatex-opts="-s /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/custom-dblatex.sty -P draft.mode=yes -P draft.watermark=0" -a docinfo -a revnumber="DRAFT 0.1-1213-gae8a" -a revdate="2022-Feb-24" osmo-gsm-tester-manual.adoc a2x: args: ['-vv', '-L', "--asciidoc-opts=-f /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/mscgen-filter.conf -f /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/diag-filter.conf -f /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/docinfo-releaseinfo.conf -a srcdir='/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp' -a commondir='/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/common'", '--dblatex-opts=-s /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/custom-dblatex.sty -P draft.mode=yes -P draft.watermark=0', '-a', 'docinfo', '-a', 'revnumber=DRAFT 0.1-1213-gae8a', '-a', 'revdate=2022-Feb-24', 'osmo-gsm-tester-manual.adoc'] a2x: resource files: [] a2x: resource directories: ['/etc/asciidoc/images', '/etc/asciidoc/stylesheets'] a2x: executing: "/usr/bin/asciidoc" --backend docbook -a "a2x-format=pdf" -f /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/mscgen-filter.conf -f /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/diag-filter.conf -f /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/docinfo-releaseinfo.conf -a srcdir='/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp' -a commondir='/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/common' --attribute "docinfo" --attribute "revnumber=DRAFT 0.1-1213-gae8a" --attribute "revdate=2022-Feb-24" --verbose --out-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual.xml" "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual.adoc" asciidoc: reading: /etc/asciidoc/asciidoc.conf asciidoc: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/mscgen-filter.conf asciidoc: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/diag-filter.conf asciidoc: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/docinfo-releaseinfo.conf asciidoc: reading: /etc/asciidoc/asciidoc.conf asciidoc: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual.adoc asciidoc: reading: /etc/asciidoc/docbook45.conf asciidoc: reading: /etc/asciidoc/filters/code/code-filter.conf asciidoc: reading: /etc/asciidoc/filters/graphviz/graphviz-filter.conf asciidoc: reading: /etc/asciidoc/filters/music/music-filter.conf asciidoc: reading: /etc/asciidoc/filters/source/source-highlight-filter.conf asciidoc: reading: /etc/asciidoc/filters/latex/latex-filter.conf asciidoc: reading: /etc/asciidoc/lang-en.conf asciidoc: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/mscgen-filter.conf asciidoc: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/diag-filter.conf asciidoc: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/docinfo-releaseinfo.conf asciidoc: writing: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual.xml asciidoc: osmo-gsm-tester-manual.adoc: line 5: evaluating: {include:/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual-docinfo.xml} asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/intro.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 13: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/intro.adoc asciidoc: intro.adoc: line 55: evaluating: {counter2:target-number} asciidoc: intro.adoc: line 55: evaluating: {set2:target:osmo-gsm-tester-manual__1.png} asciidoc: intro.adoc: line 103: filtering: "/usr/bin/python2" "/etc/asciidoc/filters/graphviz/graphviz2png.py" -v -o "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual__1.png" -L dot -F png - Execute: dot -Tpng "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual__1.txt" > "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual__1.png" asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/trial.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 15: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/trial.adoc asciidoc: trial.adoc: line 14: evaluating: {counter2:target-number} asciidoc: trial.adoc: line 14: evaluating: {set2:target:osmo-gsm-tester-manual__2.png} asciidoc: trial.adoc: line 25: filtering: "/usr/bin/python2" "/etc/asciidoc/filters/graphviz/graphviz2png.py" -v -o "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual__2.png" -L dot -F png - Execute: dot -Tpng "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual__2.txt" > "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual__2.png" asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/config.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 17: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/config.adoc asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/resource_pool.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 19: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/resource_pool.adoc asciidoc: resource_pool.adoc: line 14: evaluating: {counter2:target-number} asciidoc: resource_pool.adoc: line 14: evaluating: {set2:target:osmo-gsm-tester-manual__3.png} asciidoc: resource_pool.adoc: line 79: filtering: "/usr/bin/python2" "/etc/asciidoc/filters/graphviz/graphviz2png.py" -v -o "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual__3.png" -L dot -F png - Execute: dot -Tpng "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual__3.txt" > "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual__3.png" asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/config_mangling.adoc asciidoc: resource_pool.adoc: line 111: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/config_mangling.adoc asciidoc: config_mangling.adoc: line 165: filtering: "/usr/bin/python2" "/usr/bin/asciidoc" -b docbook45 --conf-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/mscgen-filter.conf" --conf-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/diag-filter.conf" --conf-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/docinfo-releaseinfo.conf" --attribute "revnumber=DRAFT 0.1-1213-gae8a" --attribute "commondir=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/common" --attribute "srcdir=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp" --attribute "app-name=Osmo-GSM-Tester" --attribute "revdate=2022-Feb-24" --attribute "a2x-format=pdf" --attribute "docinfo" -a "indir=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters" -a "blockname=table" -s - asciidoc: config_mangling.adoc: line 255: filtering: "/usr/bin/python2" "/usr/bin/asciidoc" -b docbook45 --conf-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/mscgen-filter.conf" --conf-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/diag-filter.conf" --conf-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/docinfo-releaseinfo.conf" --attribute "revnumber=DRAFT 0.1-1213-gae8a" --attribute "commondir=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/common" --attribute "srcdir=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp" --attribute "app-name=Osmo-GSM-Tester" --attribute "revdate=2022-Feb-24" --attribute "a2x-format=pdf" --attribute "docinfo" -a "indir=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters" -a "blockname=table" -s - asciidoc: config_mangling.adoc: line 330: filtering: "/usr/bin/python2" "/usr/bin/asciidoc" -b docbook45 --conf-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/mscgen-filter.conf" --conf-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/diag-filter.conf" --conf-file "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/docinfo-releaseinfo.conf" --attribute "revnumber=DRAFT 0.1-1213-gae8a" --attribute "commondir=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/common" --attribute "srcdir=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp" --attribute "app-name=Osmo-GSM-Tester" --attribute "revdate=2022-Feb-24" --attribute "a2x-format=pdf" --attribute "docinfo" -a "indir=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters" -a "blockname=table" -s - asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/test_api.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 21: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/test_api.adoc asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/install.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 23: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/install.adoc asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/install_device.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 25: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/install_device.adoc asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/ansible.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 27: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/ansible.adoc asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/docker.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 29: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/docker.adoc asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/debugging.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 31: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/debugging.adoc asciidoc: include: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/troubleshooting.adoc asciidoc: osmo-gsm-tester-manual.adoc: line 33: reading: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/chapters/troubleshooting.adoc a2x: executing: "dblatex" -t pdf -p "/etc/asciidoc/dblatex/asciidoc-dblatex.xsl" -s "/etc/asciidoc/dblatex/asciidoc-dblatex.sty" -s /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals/build/custom-dblatex.sty -P draft.mode=yes -P draft.watermark=0 -V "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual.xml" Build the book set list... xsltproc -o doclist.txt --xinclude --xincludestyle doclist.xsl /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual.xml Build the listings... xsltproc -o listings.xml --xinclude --xincludestyle --param current.dir '/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp' /usr/share/dblatex/xsl/common/mklistings.xsl /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual.xml xsltproc -o osmo-gsm-tester-manual.rtex --xinclude --xincludestyle --param current.dir '/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp' --param listings.xml '/tmp/tmp2B4Zj8/listings.xml' /tmp/tmp2B4Zj8/custom.xsl /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual.xml XSLT stylesheets DocBook - LaTeX 2e (0.3.10) =================================================== WARNING: xref to <simpara id="scenario_suite_params"> has no generated text. Trying its ancestor elements. Image 'dblatex' not found Cannot convert 'osmo-gsm-tester-manual__1.png' to pdf Cannot convert 'osmo-gsm-tester-manual__2.png' to pdf Cannot convert 'osmo-gsm-tester-manual__3.png' to pdf Build osmo-gsm-tester-manual.pdf built-in module pdftex registered no support found for ifthen no support found for ifxetex no support found for fontspec no support found for xltxtra no support found for fontenc no support found for inputenc no support found for fancybox built-in module makeidx registered no support found for custom-dblatex building additional files... checking if compiling is necessary... the output file doesn't exist pdflatex -interaction=batchmode osmo-gsm-tester-manual.tex running post-compilation scripts... [index] the index file /tmp/tmp2B4Zj8/osmo-gsm-tester-manual.idx is empty osmo-gsm-tester-manual.toc MD5 checksum changed osmo-gsm-tester-manual.aux MD5 checksum changed the /tmp/tmp2B4Zj8/osmo-gsm-tester-manual.toc file has changed pdflatex -interaction=batchmode osmo-gsm-tester-manual.tex running post-compilation scripts... [index] the index file /tmp/tmp2B4Zj8/osmo-gsm-tester-manual.idx is empty no new compilation is needed running last-compilation scripts... 'osmo-gsm-tester-manual.pdf' successfully built a2x: deleting /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp/osmo-gsm-tester-manual.xml + make check OSMO_GSM_MANUALS_DIR: /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/osmo-gsm-manuals (from env) if [ -n "$ASCIIDOC_WARNINGS_CHECK" ]; then \ make osmo-gsm-tester-manual.check; \ fi + [ = 1 ] + rm -r /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/_manuals_temp + cd /home/jenkins/workspace/osmo-gsm-tester_gerrit + PATH=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src:/usr/local/bin:/usr/bin:/bin:/usr/games osmo-gsm-tester/contrib/jenkins-run.sh + base=/home/jenkins/workspace/osmo-gsm-tester_gerrit + dirname -- osmo-gsm-tester/contrib/jenkins-run.sh + CDPATH= cd -- osmo-gsm-tester/contrib + pwd -P + SCRIPT_DIR=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/contrib + OSMO_GSM_TESTER_CONF=/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/contrib/../sysmocom/main.conf + date +%F %T + time_start=2022-02-24 14:45:46 + trial_dir_prefix=trial- + rm -rf trial-2061 + trial_dir=trial-2063 + mkdir -p trial-2063 + mv open5gs.build-8.tgz osmo-bsc.build-15317.tgz osmo-bts-oc2g.build-4479.tgz osmo-bts-sysmo.build-6983.tgz osmo-bts.build-6453.tgz osmo-ggsn.build-5693.tgz osmo-hlr.build-7389.tgz osmo-hnbgw.build-340.tgz osmo-mgw.build-7582.tgz osmo-msc.build-25551.tgz osmo-nitb.build-11093.tgz osmo-pcu-oc2g.build-3226.tgz osmo-pcu-sysmo.build-4507.tgz osmo-pcu.build-4468.tgz osmo-sgsn.build-25820.tgz osmo-stp.build-7910.tgz osmo-trx.build-3723.tgz osmocom-bb.build-3951.tgz srslte.build-19.tgz trial-2063 + cat open5gs.build-8.md5 osmo-bsc.build-15317.md5 osmo-bts-oc2g.build-4479.md5 osmo-bts-sysmo.build-6983.md5 osmo-bts.build-6453.md5 osmo-ggsn.build-5693.md5 osmo-hlr.build-7389.md5 osmo-hnbgw.build-340.md5 osmo-mgw.build-7582.md5 osmo-msc.build-25551.md5 osmo-nitb.build-11093.md5 osmo-pcu-oc2g.build-3226.md5 osmo-pcu-sysmo.build-4507.md5 osmo-pcu.build-4468.md5 osmo-sgsn.build-25820.md5 osmo-stp.build-7910.md5 osmo-trx.build-3723.md5 osmocom-bb.build-3951.md5 srslte.build-19.md5 + rm open5gs.build-8.md5 osmo-bsc.build-15317.md5 osmo-bts-oc2g.build-4479.md5 osmo-bts-sysmo.build-6983.md5 osmo-bts.build-6453.md5 osmo-ggsn.build-5693.md5 osmo-hlr.build-7389.md5 osmo-hnbgw.build-340.md5 osmo-mgw.build-7582.md5 osmo-msc.build-25551.md5 osmo-nitb.build-11093.md5 osmo-pcu-oc2g.build-3226.md5 osmo-pcu-sysmo.build-4507.md5 osmo-pcu.build-4468.md5 osmo-sgsn.build-25820.md5 osmo-stp.build-7910.md5 osmo-trx.build-3723.md5 osmocom-bb.build-3951.md5 srslte.build-19.md5 + exit_code=1 + which osmo-gsm-tester.py + python3 -u /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo-gsm-tester.py -c /home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/contrib/../sysmocom/main.conf trial-2063 -s nitb_sms:sysmo -s sms:sysmo -s gprs:sysmo combinations: ['nitb_sms:sysmo', 'sms:sysmo', 'gprs:sysmo'] trial: 'trial-2063' tests: None -------------------------------------------------------------------------------------------- trial-2063 -------------------------------------------------------------------------------------------- 14:45:48.896819 tst trial-2063: Detailed log at /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2063/run.2022-02-24_14-45-48/log --------------------------------------------------------------------- trial-2063 nitb_sms:sysmo --------------------------------------------------------------------- 14:45:51.807963 tst nitb_sms:sysmo: reserving resources in /var/tmp/osmo-gsm-tester/state ... 14:45:51.811568 tst nitb_sms:sysmo: Verifying 1 x bts (candidates: 11) 14:45:51.832204 tst nitb_sms:sysmo: Verifying 1 x ip_address (candidates: 9) 14:45:51.839371 tst nitb_sms:sysmo: Verifying 2 x modem (candidates: 12) 14:45:51.892865 tst nitb_sms:sysmo: Reserving 1 x bts (candidates: 11) 14:45:51.913830 tst nitb_sms:sysmo: Reserving 1 x ip_address (candidates: 9) 14:45:51.920563 tst nitb_sms:sysmo: Reserving 2 x modem (candidates: 12) 14:45:52.460854 tst nitb_sms:sysmo: RESERVED RESOURCES for nitb_sms: bts sysmoBTS 1002 ip_address 10.42.42.2 modem sierra_1st sierra_2nd ---------------------------------------------- trial-2063 nitb_sms:sysmo mo_mt_sms.py ---------------------------------------------- 14:45:52.468886 tst nitb_sms:sysmo: Using 1 x ip_address (candidates: 1) 14:45:52.480163 tst nitb_sms:sysmo: Using 1 x bts (candidates: 1) 14:45:52.868518 tst mo_mt_sms.py: using LAC 3936 14:45:53.110457 tst mo_mt_sms.py: using RAC 111 14:45:53.229594 tst mo_mt_sms.py: using CellId 3936 14:45:53.349298 tst mo_mt_sms.py: using BVCI 3937 14:45:53.460412 tst nitb_sms:sysmo: Using 1 x modem (candidates: 2) 14:45:53.664944 tst nitb_sms:sysmo: Using 1 x modem (candidates: 2) 14:45:53.779130 tst mo_mt_sms.py:9: start nitb and bts... 14:45:53.983638 run osmo-nitb_10.42.42.2: Starting osmo-nitb 14:45:56.221474 run pcap-recorder_any(filters='host 10.42.42.2 and port not 22')(pid=7175): Launched 14:45:56.518984 run osmo-nitb_10.42.42.2(pid=7176): Launched 14:45:56.680871 run osmo-nitb_10.42.42.2(pid=7176): ERR: Terminated: ERROR {rc=2} [trial-2063↪nitb_sms:sysmo↪mo_mt_sms.py:11↪mo_mt_sms.py↪osmo-nitb_10.42.42.2↪osmo-nitb_10.42.42.2(pid=7176)] 14:45:56.803452 run osmo-nitb_10.42.42.2(pid=7176): stdout: | (launched: 2022-02-24_14:45:56.464884) 14:45:56.927518 run osmo-nitb_10.42.42.2(pid=7176): stderr: | (launched: 2022-02-24_14:45:56.509116) | ====================================================================== | ERROR: You should not run osmo-nitb. It is ancient, unmaintained and | obsolete for several years. If you really want to ignore years of hard work on | the new post-NITB stack (osmo-bsc, osmo-mgw, osmo-msc, osmo-hlr, osmo-stp), | you may ues the '--yes-i-really-want-to-run-prehistoric-software' command line | argument. DON'T EVER BOTHER THE DEVELOPERS IF YOU ENCOUNTER PROBLEMS WITH IT!!! | Use https://osmocom.org/projects/cellular-infrastructure/wiki/Osmocom_Network_In_The_Box | ====================================================================== 14:45:57.051549 run osmo-nitb_10.42.42.2(pid=7176): stdout: | (launched: 2022-02-24_14:45:56.464884) 14:45:57.170889 run osmo-nitb_10.42.42.2(pid=7176): stderr: | (launched: 2022-02-24_14:45:56.509116) | ====================================================================== | ERROR: You should not run osmo-nitb. It is ancient, unmaintained and | obsolete for several years. If you really want to ignore years of hard work on | the new post-NITB stack (osmo-bsc, osmo-mgw, osmo-msc, osmo-hlr, osmo-stp), | you may ues the '--yes-i-really-want-to-run-prehistoric-software' command line | argument. DON'T EVER BOTHER THE DEVELOPERS IF YOU ENCOUNTER PROBLEMS WITH IT!!! | Use https://osmocom.org/projects/cellular-infrastructure/wiki/Osmocom_Network_In_The_Box | ====================================================================== 14:45:57.313456 tst mo_mt_sms.py:11: ERR: Error: osmo-nitb_10.42.42.2(pid=7176): Process ended prematurely: osmo-nitb_10.42.42.2(pid=7176) [trial-2063↪nitb_sms:sysmo↪mo_mt_sms.py:11↪mo_mt_sms.py↪osmo-nitb_10.42.42.2↪osmo-nitb_10.42.42.2(pid=7176)] [trial-2063↪nitb_sms:sysmo↪mo_mt_sms.py:11] 14:45:57.321555 tst mo_mt_sms.py:11: Test FAILED (4.8 sec) 14:45:57.328046 run pcap-recorder_any(filters='host 10.42.42.2 and port not 22')(pid=7175): Terminating (SIGTERM) 14:45:57.536133 --- ParallelTerminationStrategy: PID 7175 died... 14:45:57.556844 run pcap-recorder_any(filters='host 10.42.42.2 and port not 22')(pid=7175): Terminated: ok {rc=0} 14:45:57.643346 bus /sierra_3: Setting Powered False 14:45:58.745309 bus /sierra_3: Setting Powered False 14:45:59.867329 bus /sierra_2: Setting Powered False --------------------------------------------------------------------- trial-2063 nitb_sms:sysmo FAIL --------------------------------------------------------------------- --------------------------------------------------------------------- trial-2063 sms:sysmo --------------------------------------------------------------------- 14:46:01.047711 tst sms:sysmo: reserving resources in /var/tmp/osmo-gsm-tester/state ... 14:46:01.051731 tst sms:sysmo: Verifying 1 x bts (candidates: 11) 14:46:01.072591 tst sms:sysmo: Verifying 6 x ip_address (candidates: 9) 14:46:01.100149 tst sms:sysmo: Verifying 2 x modem (candidates: 12) 14:46:01.153276 tst sms:sysmo: Reserving 1 x bts (candidates: 11) 14:46:01.174341 tst sms:sysmo: Reserving 6 x ip_address (candidates: 9) 14:46:01.200890 tst sms:sysmo: Reserving 2 x modem (candidates: 12) 14:46:01.748657 tst sms:sysmo: RESERVED RESOURCES for sms: bts sysmoBTS 1002 ip_address 10.42.42.2 10.42.42.3 10.42.42.4 10.42.42.5 10.42.42.6 10.42.42.7 modem sierra_1st sierra_2nd ---------------------------------------------- trial-2063 sms:sysmo mo_mt_sms.py ---------------------------------------------- 14:46:01.756776 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 14:46:01.767524 tst sms:sysmo: Using 1 x bts (candidates: 1) 14:46:01.804679 tst mo_mt_sms.py: using LAC 3937 14:46:01.920275 tst mo_mt_sms.py: using RAC 112 14:46:02.036409 tst mo_mt_sms.py: using CellId 3937 14:46:02.153455 tst mo_mt_sms.py: using BVCI 3938 14:46:02.260045 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 14:46:02.270539 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 14:46:02.281115 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 14:46:02.291498 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 14:46:02.301853 tst sms:sysmo: Using 1 x ip_address (candidates: 6) 14:46:02.312265 tst sms:sysmo: Using 1 x modem (candidates: 2) 14:46:02.424950 tst sms:sysmo: Using 1 x modem (candidates: 2) 14:46:02.537779 run osmo-hlr_10.42.42.2: Starting osmo-hlr 14:46:04.113148 run create_hlr_db(pid=7180): Launched 14:46:04.235099 bus /sierra_2: Setting Powered False 14:46:05.294895 run create_hlr_db(pid=7180): Terminated: ok {rc=0} 14:46:05.709796 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7182): Launched 14:46:06.006866 run osmo-hlr_10.42.42.2(pid=7183): Launched 14:46:06.122597 run osmo-stp_10.42.42.5: Starting osmo-stp 14:46:08.575258 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7184): Launched 14:46:08.871944 run osmo-stp_10.42.42.5(pid=7185): Launched 14:46:08.986550 run osmo-msc_10.42.42.6: Starting osmo-msc 14:46:14.155375 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=7186): Launched 14:46:14.452033 run osmo-msc_10.42.42.6(pid=7187): Launched 14:46:17.617267 run osmo-mgw_10.42.42.3: Starting osmo-mgw 14:46:19.284922 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7188): Launched 14:46:19.581710 run osmo-mgw_10.42.42.3(pid=7189): Launched 14:46:19.695555 run osmo-mgw_10.42.42.4: Starting osmo-mgw 14:46:20.198107 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7190): Launched 14:46:20.494205 run osmo-mgw_10.42.42.4(pid=7191): Launched 14:46:20.607348 run osmo-bsc_10.42.42.7: Starting osmo-bsc 14:46:24.906813 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=7192): Launched 14:46:25.203885 run osmo-bsc_10.42.42.7(pid=7193): Launched 14:46:31.471814 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.7 14:46:32.350479 run rm-remote-dir(pid=7194): Launched 14:46:34.527789 run rm-remote-dir(pid=7194): Terminated: ok {rc=0} 14:46:34.853791 run mk-remote-dir(pid=7195): Launched 14:46:36.028008 run mk-remote-dir(pid=7195): Terminated: ok {rc=0} 14:46:36.310046 run scp-inst-to-remote(pid=7196): Launched 14:46:48.501782 run scp-inst-to-remote(pid=7196): Terminated: ok {rc=0} 14:46:48.844618 run mk-remote-dir(pid=7198): Launched 14:46:50.028130 run mk-remote-dir(pid=7198): Terminated: ok {rc=0} 14:46:50.322647 run scp-cfg-to-remote(pid=7199): Launched 14:46:51.503963 run scp-cfg-to-remote(pid=7199): Terminated: ok {rc=0} 14:46:51.844768 run reload-dsp-firmware(pid=7201): Launched 14:46:54.024819 run reload-dsp-firmware(pid=7201): Terminated: ok {rc=0} 14:46:54.357441 run osmo-bts-sysmo(pid=7202): Launched 14:46:58.598443 tst mo_mt_sms.py: using MSISDN 3936 14:46:58.763447 bus /sierra_2: Setting Powered True 14:47:02.235842 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015252', msisdn='3936', subscriber_id=1} 14:47:02.360375 tst mo_mt_sms.py: using MSISDN 3937 14:47:02.526441 bus /sierra_3: Setting Powered True 14:47:06.011712 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000015253', msisdn='3937', subscriber_id=2} 14:47:06.267243 bus /sierra_2: Setting Online False 14:47:07.450765 bus /sierra_2: Setting Powered False 14:47:13.112974 bus /sierra_2: Setting Powered True 14:47:14.475335 bus /sierra_2: Setting Online True 14:47:17.054505 tst /sierra_2: Connect to ('901', '70') 14:47:17.366922 bus /sierra_3: Setting Online False 14:47:18.550780 bus /sierra_3: Setting Powered False 14:47:24.235947 bus /sierra_3: Setting Powered True 14:47:25.615995 bus /sierra_3: Setting Online True 14:47:28.117284 tst /sierra_3: Connect to ('901', '70') 14:47:28.274578 tst /sierra_2: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054228053' 14:47:28.388371 tst /sierra_3: Manufacturer: 'Sierra Wireless, Incorporated', Model: 'MC7304', Revision: 'SWI9X15C_06.03.32.04 r28324 CNSHZ-AR-BUILD 2015/05/25 01:09:15', Serial: '356853054229028' 14:47:28.493472 tst mo_mt_sms.py:35: waiting for modems to attach... 14:47:30.680042 tst /sierra_2: Already registered with network ('901', '70') 14:47:43.738846 tst /sierra_3: Already registered with network ('901', '70') 14:47:49.070604 tst /sierra_2: sending sms to MSISDN 3937 {sms='message nr. 1, from /sierra_2, to /sierra_3, from 3936, to 3937'} 14:47:55.456897 tst /sierra_3: Incoming SMS: 'message nr. 1, from /sierra_2, to /sierra_3, from 3936, to 3937' 14:47:56.197398 tst /sierra_3: SMS received as expected: 'message nr. 1, from /sierra_2, to /sierra_3, from 3936, to 3937' 14:47:56.359217 tst mo_mt_sms.py: Test passed (114.6 sec) 14:47:56.423570 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7182): Terminating (SIGTERM) 14:47:56.464021 run osmo-hlr_10.42.42.2(pid=7183): Terminating (SIGTERM) 14:47:56.504240 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7184): Terminating (SIGTERM) 14:47:56.548093 run osmo-stp_10.42.42.5(pid=7185): Terminating (SIGTERM) 14:47:56.595819 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=7186): Terminating (SIGTERM) 14:47:56.641050 run osmo-msc_10.42.42.6(pid=7187): Terminating (SIGTERM) 14:47:56.687016 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7188): Terminating (SIGTERM) 14:47:56.731268 run osmo-mgw_10.42.42.3(pid=7189): Terminating (SIGTERM) 14:47:56.782284 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7190): Terminating (SIGTERM) 14:47:56.826993 run osmo-mgw_10.42.42.4(pid=7191): Terminating (SIGTERM) 14:47:56.872193 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=7192): Terminating (SIGTERM) 14:47:56.916564 run osmo-bsc_10.42.42.7(pid=7193): Terminating (SIGTERM) 14:47:56.961829 run osmo-bts-sysmo(pid=7202): Terminating (SIGTERM) 14:47:57.006123 --- ParallelTerminationStrategy: PID 7182 died... 14:47:57.026479 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7182): Terminated: ok {rc=0} 14:47:57.074542 --- ParallelTerminationStrategy: PID 7183 died... 14:47:57.094409 run osmo-hlr_10.42.42.2(pid=7183): Terminated: ok {rc=0} 14:47:57.144501 --- ParallelTerminationStrategy: PID 7184 died... 14:47:57.165046 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7184): Terminated: ok {rc=0} 14:47:57.213930 --- ParallelTerminationStrategy: PID 7185 died... 14:47:57.234285 run osmo-stp_10.42.42.5(pid=7185): Terminated: ok {rc=0} 14:47:57.282518 --- ParallelTerminationStrategy: PID 7186 died... 14:47:57.303011 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=7186): Terminated: ok {rc=0} 14:47:57.351874 --- ParallelTerminationStrategy: PID 7187 died... 14:47:57.371421 run osmo-msc_10.42.42.6(pid=7187): Terminated: ok {rc=0} 14:47:57.418985 --- ParallelTerminationStrategy: PID 7188 died... 14:47:57.438775 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7188): Terminated: ok {rc=0} 14:47:57.486399 --- ParallelTerminationStrategy: PID 7189 died... 14:47:57.506129 run osmo-mgw_10.42.42.3(pid=7189): Terminated {rc=15} 14:47:57.553613 --- ParallelTerminationStrategy: PID 7190 died... 14:47:57.573265 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7190): Terminated: ok {rc=0} 14:47:57.621166 --- ParallelTerminationStrategy: PID 7191 died... 14:47:57.640794 run osmo-mgw_10.42.42.4(pid=7191): Terminated {rc=15} 14:47:57.688517 --- ParallelTerminationStrategy: PID 7192 died... 14:47:57.708296 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=7192): Terminated: ok {rc=0} 14:47:57.755890 --- ParallelTerminationStrategy: PID 7193 died... 14:47:57.775719 run osmo-bsc_10.42.42.7(pid=7193): Terminated: ok {rc=0} 14:47:57.823195 --- ParallelTerminationStrategy: PID 7202 died... 14:47:57.842968 run osmo-bts-sysmo(pid=7202): Terminated: ok {rc=0} 14:47:57.973027 bus /sierra_3: Setting Online False 14:47:59.106844 bus /sierra_3: Setting Powered False 14:48:09.918656 bus /sierra_3: Setting Powered False 14:48:11.017266 bus /sierra_3: Setting Powered False 14:48:12.174841 bus /sierra_2: Setting Online False 14:48:13.303590 bus /sierra_2: Setting Powered False 14:48:19.913153 bus /sierra_2: Setting Powered False --------------------------------------------------------------------- trial-2063 sms:sysmo PASS --------------------------------------------------------------------- --------------------------------------------------------------------- trial-2063 gprs:sysmo --------------------------------------------------------------------- 14:48:21.171864 tst gprs:sysmo: reserving resources in /var/tmp/osmo-gsm-tester/state ... 14:48:21.176158 tst gprs:sysmo: Verifying 1 x bts (candidates: 11) 14:48:21.197093 tst gprs:sysmo: Verifying 9 x ip_address (candidates: 9) 14:48:21.235541 tst gprs:sysmo: Verifying 4 x modem (candidates: 12) 14:48:21.323261 tst gprs:sysmo: Reserving 1 x bts (candidates: 11) 14:48:21.344571 tst gprs:sysmo: Reserving 9 x ip_address (candidates: 9) 14:48:21.382484 tst gprs:sysmo: Reserving 4 x modem (candidates: 12) 14:48:21.970354 tst gprs:sysmo: RESERVED RESOURCES for gprs: bts sysmoBTS 1002 ip_address 10.42.42.2 10.42.42.3 10.42.42.4 10.42.42.5 10.42.42.6 10.42.42.7 10.42.42.8 10.42.42.9 10.42.42.10 modem ec20_1 ec20_2 ec20_3 ec20_4 ---------------------------------------------- trial-2063 gprs:sysmo cs_paging_gprs_active.py ---------------------------------------------- 14:48:22.030466 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.041158 tst gprs:sysmo: Using 1 x bts (candidates: 1) 14:48:22.077251 tst cs_paging_gprs_active.py: using LAC 3938 14:48:22.208258 tst cs_paging_gprs_active.py: using RAC 113 14:48:22.337999 tst cs_paging_gprs_active.py: using CellId 3938 14:48:22.467746 tst cs_paging_gprs_active.py: using BVCI 3939 14:48:22.585673 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.596972 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.608073 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.619329 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.630388 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.641519 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.652809 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.663970 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:48:22.675378 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:48:22.791935 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:48:22.911791 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5003... 14:48:23.132679 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:48:23.461724 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7213): Launched 14:48:23.747124 run iperf3-srv_10.42.42.10(pid=7214): Launched 14:48:23.870603 tst cs_paging_gprs_active.py:38: start iperfv3 server 10.42.42.10:5004... 14:48:24.090102 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:48:24.417192 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7215): Launched 14:48:24.699385 run iperf3-srv_10.42.42.10(pid=7216): Launched 14:48:24.822783 tst cs_paging_gprs_active.py:38: start network... 14:48:25.042277 run osmo-hlr_10.42.42.2: Starting osmo-hlr 14:48:25.546180 run create_hlr_db(pid=7217): Launched 14:48:25.678133 bus /sierra_2: Setting Powered False 14:48:26.742538 run create_hlr_db(pid=7217): Terminated: ok {rc=0} 14:48:27.175752 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7219): Launched 14:48:27.481235 run osmo-hlr_10.42.42.2(pid=7220): Launched 14:48:27.597069 run osmo-stp_10.42.42.5: Starting osmo-stp 14:48:28.115919 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7221): Launched 14:48:28.419057 run osmo-stp_10.42.42.5(pid=7222): Launched 14:48:28.535500 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 14:48:30.002241 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7223): Launched 14:48:30.118283 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 14:48:30.379852 run patchelf(pid=7224): Launched 14:48:31.555243 run patchelf(pid=7224): Terminated: ok {rc=0} 14:48:31.683266 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 14:48:31.947484 run osmo-gsm-tester_setcap_net_admin.sh(pid=7225): Launched 14:48:33.137126 run osmo-gsm-tester_setcap_net_admin.sh(pid=7225): Terminated: ok {rc=0} 14:48:33.476000 run osmo-ggsn_10.42.42.6(pid=7228): Launched 14:48:33.602493 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 14:48:38.621455 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7245): Launched 14:48:38.947173 run osmo-sgsn_10.42.42.7(pid=7246): Launched 14:48:39.072941 run osmo-msc_10.42.42.8: Starting osmo-msc 14:48:39.678817 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7247): Launched 14:48:40.005231 run osmo-msc_10.42.42.8(pid=7248): Launched 14:48:43.188331 run osmo-mgw_10.42.42.3: Starting osmo-mgw 14:48:43.744295 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7249): Launched 14:48:44.070660 run osmo-mgw_10.42.42.3(pid=7250): Launched 14:48:44.196183 run osmo-mgw_10.42.42.4: Starting osmo-mgw 14:48:44.744702 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7251): Launched 14:48:45.070831 run osmo-mgw_10.42.42.4(pid=7252): Launched 14:48:45.195801 run osmo-bsc_10.42.42.9: Starting osmo-bsc 14:48:45.868487 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7253): Launched 14:48:46.195312 run osmo-bsc_10.42.42.9(pid=7254): Launched 14:48:52.485274 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 14:48:53.056217 run rm-remote-dir(pid=7255): Launched 14:48:55.248289 run rm-remote-dir(pid=7255): Terminated: ok {rc=0} 14:48:55.606118 run mk-remote-dir(pid=7256): Launched 14:48:56.797596 run mk-remote-dir(pid=7256): Terminated: ok {rc=0} 14:48:57.101887 run scp-inst-to-remote(pid=7257): Launched 14:49:09.307573 run scp-inst-to-remote(pid=7257): Terminated: ok {rc=0} 14:49:09.637551 run mk-remote-dir(pid=7259): Launched 14:49:10.813519 run mk-remote-dir(pid=7259): Terminated: ok {rc=0} 14:49:11.096229 run scp-cfg-to-remote(pid=7260): Launched 14:49:12.273117 run scp-cfg-to-remote(pid=7260): Terminated: ok {rc=0} 14:49:12.599068 run reload-dsp-firmware(pid=7262): Launched 14:49:14.779177 run reload-dsp-firmware(pid=7262): Terminated: ok {rc=0} 14:49:15.120428 run osmo-bts-sysmo(pid=7263): Launched 14:49:19.357453 tst cs_paging_gprs_active.py:38: Waiting for bts to be ready... 14:49:21.440027 run rm-remote-dir(pid=7264): Launched 14:49:23.626130 run rm-remote-dir(pid=7264): Terminated: ok {rc=0} 14:49:23.969152 run mk-remote-dir(pid=7265): Launched 14:49:26.153565 run mk-remote-dir(pid=7265): Terminated: ok {rc=0} 14:49:26.447027 run scp-inst-to-bts(pid=7266): Launched 14:49:39.648722 run scp-inst-to-bts(pid=7266): Terminated: ok {rc=0} 14:49:39.969781 run mk-remote-run-dir(pid=7268): Launched 14:49:42.143164 run mk-remote-run-dir(pid=7268): Terminated: ok {rc=0} 14:49:42.419691 run scp-cfg-to-sysmobts(pid=7269): Launched 14:49:44.592764 run scp-cfg-to-sysmobts(pid=7269): Terminated: ok {rc=0} 14:49:44.913443 run osmo-pcu(pid=7271): Launched 14:49:45.038456 tst cs_paging_gprs_active.py: using MSISDN 3938 14:49:45.214090 bus /gobi_6: Setting Powered True 14:49:48.962891 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3938', subscriber_id=1} 14:49:49.257368 bus /gobi_6: Setting Online False 14:49:50.450431 bus /gobi_6: Setting Powered False 14:49:57.181227 bus /gobi_6: Setting Powered True 14:49:58.544537 bus /gobi_6: Setting Online True 14:50:01.011597 tst /gobi_6: Connect to ('901', '70') 14:50:01.279533 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648' 14:50:01.407741 tst cs_paging_gprs_active.py: using MSISDN 3939 14:50:01.632045 bus /gobi_0: Setting Powered True 14:50:05.400405 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3939', subscriber_id=2} 14:50:05.691415 bus /gobi_0: Setting Online False 14:50:06.890155 bus /gobi_0: Setting Powered False 14:50:08.060975 tst /gobi_6: Already registered with network ('901', '70') 14:50:14.118932 bus /gobi_0: Setting Powered True 14:50:15.504651 bus /gobi_0: Setting Online True 14:50:17.956828 tst /gobi_0: Connect to ('901', '70') 14:50:18.232786 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873' 14:50:18.340469 tst cs_paging_gprs_active.py:38: waiting for modems to attach... 14:50:25.017331 tst /gobi_0: Already registered with network ('901', '70') 14:50:36.839626 tst cs_paging_gprs_active.py:38: waiting for modems to attach to data services... 14:50:39.931543 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 14:50:40.049382 tst cs_paging_gprs_active.py:38: Setting up data plan for "'/gobi_6/context3'" 14:50:40.489585 run move_netns(pid=7281): Launched 14:50:41.688376 run move_netns(pid=7281): Terminated: ok {rc=0} 14:50:42.023652 run ifup(pid=7287): Launched 14:50:43.198240 run ifup(pid=7287): Terminated: ok {rc=0} 14:50:43.523531 run dhcp(pid=7290): Launched 14:50:44.696306 run dhcp(pid=7290): Terminated: ok {rc=0} 14:50:49.626798 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 14:50:49.736111 tst cs_paging_gprs_active.py:38: Setting up data plan for "'/gobi_0/context2'" 14:50:50.161613 run move_netns(pid=7307): Launched 14:50:51.367035 run move_netns(pid=7307): Terminated: ok {rc=0} 14:50:51.713777 run ifup(pid=7312): Launched 14:50:52.899980 run ifup(pid=7312): Terminated: ok {rc=0} 14:50:53.245647 run dhcp(pid=7315): Launched 14:50:54.429462 run dhcp(pid=7315): Terminated: ok {rc=0} 14:50:54.560331 tst cs_paging_gprs_active.py:38: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context3'" 14:50:54.767828 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s) 14:50:55.128959 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331): Launched 14:50:55.300681 tst cs_paging_gprs_active.py:38: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'" 14:50:55.517132 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s) 14:50:55.888397 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334): Launched 14:50:56.229397 run iperf3-cli_10.42.42.10(pid=7337): Launched 14:50:56.510135 run iperf3-cli_10.42.42.10(pid=7340): Launched 14:50:56.634923 tst cs_paging_gprs_active.py:13: waiting a few secs to make sure iperf3 test is running 14:50:58.871297 tst cs_paging_gprs_active.py:16: iperf3 running, let's place a call 14:51:03.568894 tst cs_paging_gprs_active.py:22: dial success 14:51:06.725251 tst cs_paging_gprs_active.py:28: answer success, call established and ongoing 14:51:13.574581 tst cs_paging_gprs_active.py:35: hangup success 14:52:05.194855 run iperf3-cli_10.42.42.10(pid=7337): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=7337)] 14:52:05.324826 run iperf3-cli_10.42.42.10(pid=7337): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 14:52:05.456644 run iperf3-cli_10.42.42.10(pid=7337): stderr: | (launched: 2022-02-24_14:50:56.214926) 14:52:06.646360 run iperf3-cli_10.42.42.10(pid=7340): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪cs_paging_gprs_active.py:38↪cs_paging_gprs_active.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=7340)] 14:52:06.777244 run iperf3-cli_10.42.42.10(pid=7340): stdout: | "bits_per_second": 1633929.1177060113, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 14:52:06.908640 run iperf3-cli_10.42.42.10(pid=7340): stderr: | (launched: 2022-02-24_14:50:56.495962) 14:52:07.040900 run iperf3-srv_10.42.42.10(pid=7214): Terminating (SIGINT) 14:52:07.377530 run iperf3-srv_10.42.42.10(pid=7214): Terminated {rc=1} 14:52:07.495252 tst cs_paging_gprs_active.py:38: Results for 10.42.42.10:5003 through "'/gobi_6/context3'" Exception while using iperf3 client results: "{'start': {'connected': [{'socket': 5, 'local_host': '176.16.46.1', 'local_port': 33794, 'remote_host': '10.42.42.10', 'remote_port': 5003}], 'version': 'iperf 3.6', 'system_info': 'Linux osmo-gsm-tester-prod 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64', 'timestamp': {'time': 'Thu, 24 Feb 2022 13:50:57 GMT', 'timesecs': 1645710657}, 'connecting_to': {'host': '10.42.42.10', 'port': 5003}, 'cookie': 'vupkr6v7ozrhc76qqrpgbkp5rlm7pms6raon', 'tcp_mss_default': 1448, 'sock_bufsize': 0, 'sndbuf_actual': 16384, 'rcvbuf_actual': 131072, 'test_start': {'protocol': 'TCP', 'num_streams': 1, 'blksize': 131072, 'omit': 0, 'duration': 60, 'bytes': 0, 'blocks': 0, 'reverse': 0, 'tos': 0}}, 'intervals': [{'streams': [{'socket': 5, 'start': 0, 'end': 27.022989988327026, 'seconds': 27.02298927307129, 'bytes': 101360, 'bits_per_second': 30007.0429590871, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 415817, 'rttvar': 179406, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 0, 'end': 27.022989988327026, 'seconds': 27.02298927307129, 'bytes': 101360, 'bits_per_second': 30007.0429590871, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 27.023369073867798, 'end': 28.00113606452942, 'seconds': 0.9777669906616211, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 415817, 'rttvar': 179406, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 27.023369073867798, 'end': 28.00113606452942, 'seconds': 0.9777669906616211, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 28.00113606452942, 'end': 29.001147985458374, 'seconds': 1.000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'snd_cwnd': 17376, 'rtt': 639757, 'rttvar': 512486, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 28.00113606452942, 'end': 29.001147985458374, 'seconds': 1.000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 29.001147985458374, 'end': 30.001158952713013, 'seconds': 1.0000109672546387, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 18824, 'rtt': 859751, 'rttvar': 824353, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 29.001147985458374, 'end': 30.001158952713013, 'seconds': 1.0000109672546387, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 30.001158952713013, 'end': 31.00109601020813, 'seconds': 0.9999370574951172, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 21720, 'rtt': 1471619, 'rttvar': 1547647, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 30.001158952713013, 'end': 31.00109601020813, 'seconds': 0.9999370574951172, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 31.00109601020813, 'end': 32.00113606452942, 'seconds': 1.000040054321289, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 23168, 'rtt': 1850165, 'rttvar': 1917827, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 31.00109601020813, 'end': 32.00113606452942, 'seconds': 1.000040054321289, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 32.00113606452942, 'end': 33.00115895271301, 'seconds': 1.0000228881835938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 26064, 'rtt': 2720030, 'rttvar': 2610519, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 32.00113606452942, 'end': 33.00115895271301, 'seconds': 1.0000228881835938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 33.00115895271301, 'end': 34.00116014480591, 'seconds': 1.0000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 27512, 'rtt': 3204874, 'rttvar': 2927578, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 33.00115895271301, 'end': 34.00116014480591, 'seconds': 1.0000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 34.00116014480591, 'end': 35.00113916397095, 'seconds': 0.9999790191650391, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 28960, 'rtt': 3716203, 'rttvar': 3218341, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 34.00116014480591, 'end': 35.00113916397095, 'seconds': 0.9999790191650391, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 35.00113916397095, 'end': 36.001198053359985, 'seconds': 1.000058889389038, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 4592887, 'rttvar': 3347451, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 35.00113916397095, 'end': 36.001198053359985, 'seconds': 1.000058889389038, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 36.001198053359985, 'end': 37.00116205215454, 'seconds': 0.9999639987945557, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 4592887, 'rttvar': 3347451, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 36.001198053359985, 'end': 37.00116205215454, 'seconds': 0.9999639987945557, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 37.00116205215454, 'end': 38.00119209289551, 'seconds': 1.0000300407409668, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 5577253, 'rttvar': 3613784, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 37.00116205215454, 'end': 38.00119209289551, 'seconds': 1.0000300407409668, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 38.00119209289551, 'end': 39.00115895271301, 'seconds': 0.9999668598175049, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 36200, 'rtt': 6027425, 'rttvar': 3610682, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 38.00119209289551, 'end': 39.00115895271301, 'seconds': 0.9999668598175049, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 39.00115895271301, 'end': 40.00115895271301, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 39096, 'rtt': 6925661, 'rttvar': 3588103, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 39.00115895271301, 'end': 40.00115895271301, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 40.00115895271301, 'end': 41.00117206573486, 'seconds': 1.0000131130218506, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 40544, 'rtt': 7377556, 'rttvar': 3594867, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 40.00115895271301, 'end': 41.00117206573486, 'seconds': 1.0000131130218506, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 41.00117206573486, 'end': 42.001089096069336, 'seconds': 0.9999170303344727, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 7772933, 'rttvar': 3486905, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 41.00117206573486, 'end': 42.001089096069336, 'seconds': 0.9999170303344727, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 42.001089096069336, 'end': 43.001157999038696, 'seconds': 1.0000689029693604, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 8587109, 'rttvar': 3373216, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 42.001089096069336, 'end': 43.001157999038696, 'seconds': 1.0000689029693604, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 43.001157999038696, 'end': 44.001198053359985, 'seconds': 1.000040054321289, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 9005966, 'rttvar': 3367626, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 43.001157999038696, 'end': 44.001198053359985, 'seconds': 1.000040054321289, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 44.001198053359985, 'end': 45.00115895271301, 'seconds': 0.9999608993530273, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 9785175, 'rttvar': 3268298, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 44.001198053359985, 'end': 45.00115895271301, 'seconds': 0.9999608993530273, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 45.00115895271301, 'end': 46.001161098480225, 'seconds': 1.000002145767212, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 10142033, 'rttvar': 3164940, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 45.00115895271301, 'end': 46.001161098480225, 'seconds': 1.000002145767212, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 46.001161098480225, 'end': 47.000959157943726, 'seconds': 0.999798059463501, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 10541412, 'rttvar': 3172464, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 46.001161098480225, 'end': 47.000959157943726, 'seconds': 0.999798059463501, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 47.000959157943726, 'end': 48.00110101699829, 'seconds': 1.0001418590545654, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 10928416, 'rttvar': 3153355, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 47.000959157943726, 'end': 48.00110101699829, 'seconds': 1.0001418590545654, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 48.00110101699829, 'end': 49.001177072525024, 'seconds': 1.0000760555267334, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 11444679, 'rttvar': 3397543, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 48.00110101699829, 'end': 49.001177072525024, 'seconds': 1.0000760555267334, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 49.001177072525024, 'end': 50.00109910964966, 'seconds': 0.9999220371246338, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 11444679, 'rttvar': 3397543, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 49.001177072525024, 'end': 50.00109910964966, 'seconds': 0.9999220371246338, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 50.00109910964966, 'end': 51.00120210647583, 'seconds': 1.0001029968261719, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 11984189, 'rttvar': 3627177, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 50.00109910964966, 'end': 51.00120210647583, 'seconds': 1.0001029968261719, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 51.00120210647583, 'end': 52.00143814086914, 'seconds': 1.0002360343933105, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 12630980, 'rttvar': 4013965, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 51.00120210647583, 'end': 52.00143814086914, 'seconds': 1.0002360343933105, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 52.00143814086914, 'end': 53.0012731552124, 'seconds': 0.9998350143432617, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 13279392, 'rttvar': 4307297, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 52.00143814086914, 'end': 53.0012731552124, 'seconds': 0.9998350143432617, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 53.0012731552124, 'end': 54.00126504898071, 'seconds': 0.9999918937683105, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 13279392, 'rttvar': 4307297, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 53.0012731552124, 'end': 54.00126504898071, 'seconds': 0.9999918937683105, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 54.00126504898071, 'end': 55.001161098480225, 'seconds': 0.9998960494995117, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 14024329, 'rttvar': 4720348, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 54.00126504898071, 'end': 55.001161098480225, 'seconds': 0.9998960494995117, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 55.001161098480225, 'end': 56.001211166381836, 'seconds': 1.0000500679016113, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 14758142, 'rttvar': 5007886, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 55.001161098480225, 'end': 56.001211166381836, 'seconds': 1.0000500679016113, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 56.001211166381836, 'end': 57.00017213821411, 'seconds': 0.9989609718322754, 'bytes': 270776, 'bits_per_second': 2168461.0921553643, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 15565212, 'rttvar': 5370054, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 56.001211166381836, 'end': 57.00017213821411, 'seconds': 0.9989609718322754, 'bytes': 270776, 'bits_per_second': 2168461.0921553643, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 57.00017213821411, 'end': 58.000957012176514, 'seconds': 1.0007848739624023, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 15565212, 'rttvar': 5370054, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 57.00017213821411, 'end': 58.000957012176514, 'seconds': 1.0007848739624023, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 58.000957012176514, 'end': 59.00126504898071, 'seconds': 1.0003080368041992, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 16266811, 'rttvar': 5430739, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 58.000957012176514, 'end': 59.00126504898071, 'seconds': 1.0003080368041992, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}], 'end': {}, 'error': 'error - control socket has closed unexpectedly'}" 14:52:07.715784 tst cs_paging_gprs_active.py:38: ERR: KeyError: 'sum_sent' [trial-2063↪gprs:sysmo↪cs_paging_gprs_active.py:38] 14:52:07.722862 tst cs_paging_gprs_active.py:38: Test FAILED (225.7 sec) 14:52:07.729431 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7213): Terminating (SIGTERM) 14:52:07.770565 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7215): Terminating (SIGTERM) 14:52:07.811000 run iperf3-srv_10.42.42.10(pid=7216): Terminating (SIGTERM) 14:52:07.851155 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7219): Terminating (SIGTERM) 14:52:07.892218 run osmo-hlr_10.42.42.2(pid=7220): Terminating (SIGTERM) 14:52:07.932713 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7221): Terminating (SIGTERM) 14:52:07.975744 run osmo-stp_10.42.42.5(pid=7222): Terminating (SIGTERM) 14:52:08.018332 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7223): Terminating (SIGTERM) 14:52:08.071473 run osmo-ggsn_10.42.42.6(pid=7228): Terminating (SIGTERM) 14:52:08.120733 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7245): Terminating (SIGTERM) 14:52:08.211335 run osmo-sgsn_10.42.42.7(pid=7246): Terminating (SIGTERM) 14:52:08.266333 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7247): Terminating (SIGTERM) 14:52:08.316267 run osmo-msc_10.42.42.8(pid=7248): Terminating (SIGTERM) 14:52:08.368978 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7249): Terminating (SIGTERM) 14:52:08.413316 run osmo-mgw_10.42.42.3(pid=7250): Terminating (SIGTERM) 14:52:08.456985 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7251): Terminating (SIGTERM) 14:52:08.500552 run osmo-mgw_10.42.42.4(pid=7252): Terminating (SIGTERM) 14:52:08.544751 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7253): Terminating (SIGTERM) 14:52:08.590093 run osmo-bsc_10.42.42.9(pid=7254): Terminating (SIGTERM) 14:52:08.630781 run osmo-bts-sysmo(pid=7263): Terminating (SIGTERM) 14:52:08.671668 run osmo-pcu(pid=7271): Terminating (SIGTERM) 14:52:08.712253 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331): Terminating (SIGTERM) 14:52:08.845028 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGTERM(pid=7358): Launched 14:52:09.930805 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGTERM(pid=7358): Terminated: ok {rc=0} 14:52:09.999741 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334): Terminating (SIGTERM) 14:52:10.140596 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGTERM(pid=7364): Launched 14:52:11.230502 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGTERM(pid=7364): Terminated: ok {rc=0} 14:52:11.293848 --- ParallelTerminationStrategy: PID 7213 died... 14:52:11.312568 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7213): Terminated: ok {rc=0} 14:52:11.356629 --- ParallelTerminationStrategy: PID 7215 died... 14:52:11.375382 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7215): Terminated: ok {rc=0} 14:52:11.418823 --- ParallelTerminationStrategy: PID 7216 died... 14:52:11.437206 run iperf3-srv_10.42.42.10(pid=7216): Terminated {rc=256} 14:52:11.481762 --- ParallelTerminationStrategy: PID 7219 died... 14:52:11.500624 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7219): Terminated: ok {rc=0} 14:52:11.544225 --- ParallelTerminationStrategy: PID 7220 died... 14:52:11.563306 run osmo-hlr_10.42.42.2(pid=7220): Terminated: ok {rc=0} 14:52:11.607006 --- ParallelTerminationStrategy: PID 7221 died... 14:52:11.625690 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7221): Terminated: ok {rc=0} 14:52:11.669455 --- ParallelTerminationStrategy: PID 7222 died... 14:52:11.687939 run osmo-stp_10.42.42.5(pid=7222): Terminated: ok {rc=0} 14:52:11.731988 --- ParallelTerminationStrategy: PID 7223 died... 14:52:11.750979 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7223): Terminated: ok {rc=0} 14:52:11.794534 --- ParallelTerminationStrategy: PID 7228 died... 14:52:11.813044 run osmo-ggsn_10.42.42.6(pid=7228): Terminated: ok {rc=0} 14:52:11.856832 --- ParallelTerminationStrategy: PID 7245 died... 14:52:11.875379 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7245): Terminated: ok {rc=0} 14:52:11.919932 --- ParallelTerminationStrategy: PID 7246 died... 14:52:11.938325 run osmo-sgsn_10.42.42.7(pid=7246): Terminated: ok {rc=0} 14:52:11.983043 --- ParallelTerminationStrategy: PID 7247 died... 14:52:12.001507 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7247): Terminated: ok {rc=0} 14:52:12.046571 --- ParallelTerminationStrategy: PID 7248 died... 14:52:12.065644 run osmo-msc_10.42.42.8(pid=7248): Terminated: ok {rc=0} 14:52:12.109396 --- ParallelTerminationStrategy: PID 7249 died... 14:52:12.128004 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7249): Terminated: ok {rc=0} 14:52:12.171780 --- ParallelTerminationStrategy: PID 7250 died... 14:52:12.190439 run osmo-mgw_10.42.42.3(pid=7250): Terminated {rc=15} 14:52:12.234801 --- ParallelTerminationStrategy: PID 7251 died... 14:52:12.253293 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7251): Terminated: ok {rc=0} 14:52:12.297559 --- ParallelTerminationStrategy: PID 7252 died... 14:52:12.316467 run osmo-mgw_10.42.42.4(pid=7252): Terminated {rc=15} 14:52:12.360236 --- ParallelTerminationStrategy: PID 7253 died... 14:52:12.378871 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7253): Terminated: ok {rc=0} 14:52:12.422688 --- ParallelTerminationStrategy: PID 7254 died... 14:52:12.441439 run osmo-bsc_10.42.42.9(pid=7254): Terminated: ok {rc=0} 14:52:12.485098 --- ParallelTerminationStrategy: PID 7263 died... 14:52:12.503778 run osmo-bts-sysmo(pid=7263): Terminated: ok {rc=0} 14:52:12.547824 --- ParallelTerminationStrategy: PID 7271 died... 14:52:12.566422 run osmo-pcu(pid=7271): Terminated: ok {rc=0} 14:52:18.433253 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331): Terminating (SIGINT) 14:52:18.567733 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGINT(pid=7367): Launched 14:52:19.652391 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGINT(pid=7367): Terminated: ok {rc=0} 14:52:19.714325 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334): Terminating (SIGINT) 14:52:19.847346 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGINT(pid=7370): Launched 14:52:20.928402 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGINT(pid=7370): Terminated: ok {rc=0} 14:52:26.813676 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331): Terminating (SIGKILL) 14:52:26.947865 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGUSR1(pid=7373): Launched 14:52:28.028697 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7331)-killSignals.SIGUSR1(pid=7373): Terminated: ok {rc=0} 14:52:28.094463 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334): Terminating (SIGKILL) 14:52:28.234303 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGUSR1(pid=7376): Launched 14:52:29.324091 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7334)-killSignals.SIGUSR1(pid=7376): Terminated: ok {rc=0} 14:52:29.561743 bus /gobi_0: Setting Online False 14:52:31.672955 bus /gobi_0: Setting Powered False 14:52:46.944898 bus /gobi_0: Setting Powered False 14:52:48.042097 bus /gobi_0: Setting Powered False 14:52:49.193790 bus /gobi_6: Setting Online False 14:52:50.327063 bus /gobi_6: Setting Powered False 14:52:58.094057 bus /gobi_6: Setting Powered False ---------------------------------------------- trial-2063 gprs:sysmo iperf3.py ---------------------------------------------- 14:52:59.294303 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.305522 tst gprs:sysmo: Using 1 x bts (candidates: 1) 14:52:59.341174 tst iperf3.py: using LAC 3939 14:52:59.459073 tst iperf3.py: using RAC 114 14:52:59.576193 tst iperf3.py: using CellId 3939 14:52:59.693097 tst iperf3.py: using BVCI 3940 14:52:59.804529 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.815176 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.825540 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.836136 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.846541 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.857112 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.867801 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.878224 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:52:59.888966 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:53:00.002310 tst iperf3.py:8: start iperfv3 server 10.42.42.10:5003... 14:53:00.204732 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:53:00.510360 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Launched 14:53:00.774402 run iperf3-srv_10.42.42.10(pid=7380): Launched 14:53:00.886033 tst iperf3.py:8: start network... 14:53:01.086929 run osmo-hlr_10.42.42.2: Starting osmo-hlr 14:53:01.556001 run create_hlr_db(pid=7381): Launched 14:53:01.677777 bus /gobi_6: Setting Powered False 14:53:02.738019 run create_hlr_db(pid=7381): Terminated: ok {rc=0} 14:53:03.158892 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Launched 14:53:03.462673 run osmo-hlr_10.42.42.2(pid=7384): Launched 14:53:03.576417 run osmo-stp_10.42.42.5: Starting osmo-stp 14:53:04.093221 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Launched 14:53:04.397057 run osmo-stp_10.42.42.5(pid=7386): Launched 14:53:04.510575 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 14:53:05.026643 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Launched 14:53:05.140651 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 14:53:05.397037 run patchelf(pid=7388): Launched 14:53:06.566387 run patchelf(pid=7388): Terminated: ok {rc=0} 14:53:06.692915 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 14:53:06.953470 run osmo-gsm-tester_setcap_net_admin.sh(pid=7389): Launched 14:53:08.139277 run osmo-gsm-tester_setcap_net_admin.sh(pid=7389): Terminated: ok {rc=0} 14:53:08.476569 run osmo-ggsn_10.42.42.6(pid=7392): Launched 14:53:08.599469 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 14:53:09.281140 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Launched 14:53:09.606678 run osmo-sgsn_10.42.42.7(pid=7410): Launched 14:53:09.730168 run osmo-msc_10.42.42.8: Starting osmo-msc 14:53:10.320760 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Launched 14:53:10.647737 run osmo-msc_10.42.42.8(pid=7412): Launched 14:53:13.831389 run osmo-mgw_10.42.42.3: Starting osmo-mgw 14:53:14.388528 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Launched 14:53:14.717153 run osmo-mgw_10.42.42.3(pid=7414): Launched 14:53:14.842565 run osmo-mgw_10.42.42.4: Starting osmo-mgw 14:53:15.390874 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Launched 14:53:15.716815 run osmo-mgw_10.42.42.4(pid=7416): Launched 14:53:15.841586 run osmo-bsc_10.42.42.9: Starting osmo-bsc 14:53:16.515740 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Launched 14:53:16.843164 run osmo-bsc_10.42.42.9(pid=7418): Launched 14:53:23.132279 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 14:53:23.701102 run rm-remote-dir(pid=7419): Launched 14:53:25.895258 run rm-remote-dir(pid=7419): Terminated: ok {rc=0} 14:53:26.252074 run mk-remote-dir(pid=7420): Launched 14:53:27.441419 run mk-remote-dir(pid=7420): Terminated: ok {rc=0} 14:53:27.745703 run scp-inst-to-remote(pid=7421): Launched 14:53:39.950179 run scp-inst-to-remote(pid=7421): Terminated: ok {rc=0} 14:53:40.279828 run mk-remote-dir(pid=7423): Launched 14:53:41.455249 run mk-remote-dir(pid=7423): Terminated: ok {rc=0} 14:53:41.738413 run scp-cfg-to-remote(pid=7424): Launched 14:53:42.918521 run scp-cfg-to-remote(pid=7424): Terminated: ok {rc=0} 14:53:43.268837 run reload-dsp-firmware(pid=7426): Launched 14:53:45.457855 run reload-dsp-firmware(pid=7426): Terminated: ok {rc=0} 14:53:45.800606 run osmo-bts-sysmo(pid=7427): Launched 14:53:50.038097 tst iperf3.py:8: Waiting for bts to be ready... 14:53:51.758678 run rm-remote-dir(pid=7428): Launched 14:53:53.938676 run rm-remote-dir(pid=7428): Terminated: ok {rc=0} 14:53:54.260312 run mk-remote-dir(pid=7429): Launched 14:53:56.434607 run mk-remote-dir(pid=7429): Terminated: ok {rc=0} 14:53:56.711370 run scp-inst-to-bts(pid=7430): Launched 14:54:09.900598 run scp-inst-to-bts(pid=7430): Terminated: ok {rc=0} 14:54:10.222863 run mk-remote-run-dir(pid=7432): Launched 14:54:12.397395 run mk-remote-run-dir(pid=7432): Terminated: ok {rc=0} 14:54:12.673236 run scp-cfg-to-sysmobts(pid=7433): Launched 14:54:14.846278 run scp-cfg-to-sysmobts(pid=7433): Terminated: ok {rc=0} 14:54:15.167704 run osmo-pcu(pid=7435): Launched 14:54:15.294414 tst iperf3.py: using MSISDN 3940 14:54:15.465623 bus /gobi_6: Setting Powered True 14:54:18.962209 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3940', subscriber_id=1} 14:54:19.228473 bus /gobi_6: Setting Online False 14:54:20.415856 bus /gobi_6: Setting Powered False 14:54:27.188235 bus /gobi_6: Setting Powered True 14:54:28.593390 bus /gobi_6: Setting Online True 14:54:31.073021 tst /gobi_6: Connect to ('901', '70') 14:54:31.350739 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648' 14:54:31.468187 tst iperf3.py:8: waiting for modems to attach... 14:54:44.151068 tst /gobi_6: Already registered with network ('901', '70') 14:54:49.925303 tst iperf3.py:8: waiting for modems to attach to data services... 14:55:14.009518 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context4', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 14:55:14.127791 tst iperf3.py:8: Setting up data plan for "'/gobi_6/context4'" 14:55:14.573898 run move_netns(pid=7442): Launched 14:55:15.768589 run move_netns(pid=7442): Terminated: ok {rc=0} 14:55:16.120394 run ifup(pid=7445): Launched 14:55:17.310567 run ifup(pid=7445): Terminated: ok {rc=0} 14:55:17.663023 run dhcp(pid=7448): Launched 14:55:18.852640 run dhcp(pid=7448): Terminated: ok {rc=0} 14:55:18.990860 tst iperf3.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context4'" 14:55:19.217853 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s) 14:55:19.608607 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Launched 14:55:19.934045 run iperf3-cli_10.42.42.10(pid=7467): Launched 14:56:28.246014 run iperf3-cli_10.42.42.10(pid=7467): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3.py:8↪iperf3.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=7467)] 14:56:28.381567 run iperf3-cli_10.42.42.10(pid=7467): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 14:56:28.518207 run iperf3-cli_10.42.42.10(pid=7467): stderr: | (launched: 2022-02-24_14:55:19.919964) 14:56:28.654256 run iperf3-srv_10.42.42.10(pid=7380): Terminating (SIGINT) 14:56:28.827908 run iperf3-srv_10.42.42.10(pid=7380): Terminated {rc=1} 14:56:28.950869 tst iperf3.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context4'" Exception while using iperf3 client results: "{'start': {'connected': [{'socket': 5, 'local_host': '176.16.46.1', 'local_port': 33832, 'remote_host': '10.42.42.10', 'remote_port': 5003}], 'version': 'iperf 3.6', 'system_info': 'Linux osmo-gsm-tester-prod 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64', 'timestamp': {'time': 'Thu, 24 Feb 2022 13:55:21 GMT', 'timesecs': 1645710921}, 'connecting_to': {'host': '10.42.42.10', 'port': 5003}, 'cookie': 'gwyafh7xce3kcaj3jl75sd6oab6l3lqoqhx3', 'tcp_mss_default': 1448, 'sock_bufsize': 0, 'sndbuf_actual': 16384, 'rcvbuf_actual': 131072, 'test_start': {'protocol': 'TCP', 'num_streams': 1, 'blksize': 131072, 'omit': 0, 'duration': 60, 'bytes': 0, 'blocks': 0, 'reverse': 0, 'tos': 0}}, 'intervals': [{'streams': [{'socket': 5, 'start': 0, 'end': 1.0009050369262695, 'seconds': 1.0009050369262695, 'bytes': 101360, 'bits_per_second': 810146.7872418475, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 415515, 'rttvar': 179116, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 0, 'end': 1.0009050369262695, 'seconds': 1.0009050369262695, 'bytes': 101360, 'bits_per_second': 810146.7872418475, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 1.0009050369262695, 'end': 2.000504970550537, 'seconds': 0.9995999336242676, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 15928, 'rtt': 503089, 'rttvar': 309485, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 1.0009050369262695, 'end': 2.000504970550537, 'seconds': 0.9995999336242676, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 2.000504970550537, 'end': 3.0011818408966064, 'seconds': 1.0006768703460693, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 18824, 'rtt': 886766, 'rttvar': 861895, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 2.000504970550537, 'end': 3.0011818408966064, 'seconds': 1.0006768703460693, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 3.0011818408966064, 'end': 4.001266956329346, 'seconds': 1.0000851154327393, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 20272, 'rtt': 1168053, 'rttvar': 1208995, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 3.0011818408966064, 'end': 4.001266956329346, 'seconds': 1.0000851154327393, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 4.001266956329346, 'end': 5.001267910003662, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 23168, 'rtt': 1881054, 'rttvar': 1939150, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 4.001266956329346, 'end': 5.001267910003662, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 5.001267910003662, 'end': 6.001267910003662, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 24616, 'rtt': 2300534, 'rttvar': 2293324, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 5.001267910003662, 'end': 6.001267910003662, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 6.001267910003662, 'end': 7.001167058944702, 'seconds': 0.99989914894104, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 26064, 'rtt': 2755212, 'rttvar': 2629347, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 6.001267910003662, 'end': 7.001167058944702, 'seconds': 0.99989914894104, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 7.001167058944702, 'end': 8.001173973083496, 'seconds': 1.000006914138794, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 28960, 'rtt': 3752063, 'rttvar': 3230264, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 7.001167058944702, 'end': 8.001173973083496, 'seconds': 1.000006914138794, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 8.001173973083496, 'end': 9.001308917999268, 'seconds': 1.0001349449157715, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 30408, 'rtt': 4182268, 'rttvar': 3283108, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 8.001173973083496, 'end': 9.001308917999268, 'seconds': 1.0001349449157715, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 9.001308917999268, 'end': 10.00123405456543, 'seconds': 0.9999251365661621, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 33304, 'rtt': 4995598, 'rttvar': 3257141, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 9.001308917999268, 'end': 10.00123405456543, 'seconds': 0.9999251365661621, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 10.00123405456543, 'end': 11.001266956329346, 'seconds': 1.000032901763916, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 5416487, 'rttvar': 3284634, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 10.00123405456543, 'end': 11.001266956329346, 'seconds': 1.000032901763916, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 11.001266956329346, 'end': 12.001320838928223, 'seconds': 1.000053882598877, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 37648, 'rtt': 6197823, 'rttvar': 3225306, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 11.001266956329346, 'end': 12.001320838928223, 'seconds': 1.000053882598877, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 12.001320838928223, 'end': 13.00131893157959, 'seconds': 0.9999980926513672, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 39096, 'rtt': 6565319, 'rttvar': 3153971, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 12.001320838928223, 'end': 13.00131893157959, 'seconds': 0.9999980926513672, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 13.00131893157959, 'end': 14.001267910003662, 'seconds': 0.9999489784240723, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 40544, 'rtt': 6982137, 'rttvar': 3199116, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 13.00131893157959, 'end': 14.001267910003662, 'seconds': 0.9999489784240723, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 14.001267910003662, 'end': 15.001270055770874, 'seconds': 1.000002145767212, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 7354257, 'rttvar': 3143577, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 14.001267910003662, 'end': 15.001270055770874, 'seconds': 1.000002145767212, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 15.001270055770874, 'end': 16.001266956329346, 'seconds': 0.9999969005584717, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 8183945, 'rttvar': 3211145, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 15.001270055770874, 'end': 16.001266956329346, 'seconds': 0.9999969005584717, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 16.001266956329346, 'end': 17.001270055770874, 'seconds': 1.0000030994415283, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 8618738, 'rttvar': 3277945, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 16.001266956329346, 'end': 17.001270055770874, 'seconds': 1.0000030994415283, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 17.001270055770874, 'end': 18.001250982284546, 'seconds': 0.9999809265136719, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 9416163, 'rttvar': 3248826, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 17.001270055770874, 'end': 18.001250982284546, 'seconds': 0.9999809265136719, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 18.001250982284546, 'end': 19.001167058944702, 'seconds': 0.9999160766601562, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 9781108, 'rttvar': 3166509, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 18.001250982284546, 'end': 19.001167058944702, 'seconds': 0.9999160766601562, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 19.001167058944702, 'end': 20.001267910003662, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 10188178, 'rttvar': 3189022, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 19.001167058944702, 'end': 20.001267910003662, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 20.001267910003662, 'end': 21.001178979873657, 'seconds': 0.9999110698699951, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 10943953, 'rttvar': 3125854, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 20.001267910003662, 'end': 21.001178979873657, 'seconds': 0.9999110698699951, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 21.001178979873657, 'end': 22.001267910003662, 'seconds': 1.0000889301300049, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 11578861, 'rttvar': 2880785, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 21.001178979873657, 'end': 22.001267910003662, 'seconds': 1.0000889301300049, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 22.001267910003662, 'end': 23.001267910003662, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 11875977, 'rttvar': 2754820, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 22.001267910003662, 'end': 23.001267910003662, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 23.001267910003662, 'end': 24.00117802619934, 'seconds': 0.9999101161956787, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 12548019, 'rttvar': 2719843, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 23.001267910003662, 'end': 24.00117802619934, 'seconds': 0.9999101161956787, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 24.00117802619934, 'end': 25.001319885253906, 'seconds': 1.0001418590545654, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 12951860, 'rttvar': 2847565, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 24.00117802619934, 'end': 25.001319885253906, 'seconds': 1.0001418590545654, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 25.001319885253906, 'end': 26.000436067581177, 'seconds': 0.9991161823272705, 'bytes': 244712, 'bits_per_second': 1959427.7768976593, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 13314990, 'rttvar': 2861933, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 25.001319885253906, 'end': 26.000436067581177, 'seconds': 0.9991161823272705, 'bytes': 244712, 'bits_per_second': 1959427.7768976593, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 26.000436067581177, 'end': 27.001322984695435, 'seconds': 1.0008869171142578, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 13735497, 'rttvar': 2987465, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 26.000436067581177, 'end': 27.001322984695435, 'seconds': 1.0008869171142578, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 27.001322984695435, 'end': 28.00132703781128, 'seconds': 1.0000040531158447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 14128186, 'rttvar': 3025976, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 27.001322984695435, 'end': 28.00132703781128, 'seconds': 1.0000040531158447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 28.00132703781128, 'end': 29.001144886016846, 'seconds': 0.9998178482055664, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 14579063, 'rttvar': 3171236, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 28.00132703781128, 'end': 29.001144886016846, 'seconds': 0.9998178482055664, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 29.001144886016846, 'end': 30.001266956329346, 'seconds': 1.0001220703125, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 56472, 'rtt': 15442650, 'rttvar': 3303312, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 29.001144886016846, 'end': 30.001266956329346, 'seconds': 1.0001220703125, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 30.001266956329346, 'end': 31.001167058944702, 'seconds': 0.9999001026153564, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 56472, 'rtt': 15834844, 'rttvar': 3261871, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 30.001266956329346, 'end': 31.001167058944702, 'seconds': 0.9999001026153564, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 31.001167058944702, 'end': 32.00111389160156, 'seconds': 0.9999468326568604, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 57920, 'rtt': 16634707, 'rttvar': 3219379, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 31.001167058944702, 'end': 32.00111389160156, 'seconds': 0.9999468326568604, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 32.00111389160156, 'end': 33.001112937927246, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 59368, 'rtt': 17052678, 'rttvar': 3250478, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 32.00111389160156, 'end': 33.001112937927246, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 33.001112937927246, 'end': 34.001163959503174, 'seconds': 1.0000510215759277, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 59368, 'rtt': 17408538, 'rttvar': 3149578, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 33.001112937927246, 'end': 34.001163959503174, 'seconds': 1.0000510215759277, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 34.001163959503174, 'end': 35.001319885253906, 'seconds': 1.0001559257507324, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 60816, 'rtt': 18140832, 'rttvar': 3036969, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 34.001163959503174, 'end': 35.001319885253906, 'seconds': 1.0001559257507324, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 35.001319885253906, 'end': 36.001163959503174, 'seconds': 0.9998440742492676, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 62264, 'rtt': 18520504, 'rttvar': 3037070, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 35.001319885253906, 'end': 36.001163959503174, 'seconds': 0.9998440742492676, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 36.001163959503174, 'end': 37.001116037368774, 'seconds': 0.9999520778656006, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 63712, 'rtt': 19200890, 'rttvar': 2911658, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 36.001163959503174, 'end': 37.001116037368774, 'seconds': 0.9999520778656006, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 37.001116037368774, 'end': 38.00107502937317, 'seconds': 0.9999589920043945, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 65160, 'rtt': 19652200, 'rttvar': 2419471, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 37.001116037368774, 'end': 38.00107502937317, 'seconds': 0.9999589920043945, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 38.00107502937317, 'end': 39.001319885253906, 'seconds': 1.0002448558807373, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 65160, 'rtt': 19835586, 'rttvar': 2181377, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 38.00107502937317, 'end': 39.001319885253906, 'seconds': 1.0002448558807373, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 39.001319885253906, 'end': 40.00126791000366, 'seconds': 0.9999480247497559, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 66608, 'rtt': 20230557, 'rttvar': 1935629, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 39.001319885253906, 'end': 40.00126791000366, 'seconds': 0.9999480247497559, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 40.00126791000366, 'end': 41.00110197067261, 'seconds': 0.9998340606689453, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 68056, 'rtt': 20434412, 'rttvar': 1859432, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 40.00126791000366, 'end': 41.00110197067261, 'seconds': 0.9998340606689453, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 41.00110197067261, 'end': 42.0011260509491, 'seconds': 1.0000240802764893, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 69504, 'rtt': 20853175, 'rttvar': 1794559, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 41.00110197067261, 'end': 42.0011260509491, 'seconds': 1.0000240802764893, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 42.0011260509491, 'end': 43.001219034194946, 'seconds': 1.0000929832458496, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 69504, 'rtt': 21064024, 'rttvar': 1767616, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 42.0011260509491, 'end': 43.001219034194946, 'seconds': 1.0000929832458496, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 43.001219034194946, 'end': 44.001168966293335, 'seconds': 0.9999499320983887, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 70952, 'rtt': 21250700, 'rttvar': 1699065, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 43.001219034194946, 'end': 44.001168966293335, 'seconds': 0.9999499320983887, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 44.001168966293335, 'end': 45.00132083892822, 'seconds': 1.0001518726348877, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 72400, 'rtt': 21778240, 'rttvar': 1884129, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 44.001168966293335, 'end': 45.00132083892822, 'seconds': 1.0001518726348877, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 45.00132083892822, 'end': 46.00132489204407, 'seconds': 1.0000040531158447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 72400, 'rtt': 22020486, 'rttvar': 1897588, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 45.00132083892822, 'end': 46.00132489204407, 'seconds': 1.0000040531158447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 46.00132489204407, 'end': 47.0011670589447, 'seconds': 0.9998421669006348, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 73848, 'rtt': 22582553, 'rttvar': 2041635, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 46.00132489204407, 'end': 47.0011670589447, 'seconds': 0.9998421669006348, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 47.0011670589447, 'end': 48.00126504898071, 'seconds': 1.0000979900360107, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 75296, 'rtt': 22814292, 'rttvar': 1994704, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 47.0011670589447, 'end': 48.00126504898071, 'seconds': 1.0000979900360107, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 48.00126504898071, 'end': 49.00116586685181, 'seconds': 0.9999008178710938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 76744, 'rtt': 23336429, 'rttvar': 2021044, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 48.00126504898071, 'end': 49.00116586685181, 'seconds': 0.9999008178710938, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 49.00116586685181, 'end': 50.00116491317749, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 76744, 'rtt': 23503890, 'rttvar': 1850703, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 49.00116586685181, 'end': 50.00116491317749, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 50.00116491317749, 'end': 51.00116586685181, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 78192, 'rtt': 23738128, 'rttvar': 1856505, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 50.00116491317749, 'end': 51.00116586685181, 'seconds': 1.0000009536743164, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 51.00116586685181, 'end': 52.0011670589447, 'seconds': 1.0000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 79640, 'rtt': 24095630, 'rttvar': 1660556, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 51.00116586685181, 'end': 52.0011670589447, 'seconds': 1.0000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 52.0011670589447, 'end': 53.0011670589447, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 79640, 'rtt': 24323328, 'rttvar': 1700814, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 52.0011670589447, 'end': 53.0011670589447, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 53.0011670589447, 'end': 54.00111103057861, 'seconds': 0.9999439716339111, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 81088, 'rtt': 24633318, 'rttvar': 1489403, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 53.0011670589447, 'end': 54.00111103057861, 'seconds': 0.9999439716339111, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 54.00111103057861, 'end': 55.000274896621704, 'seconds': 0.9991638660430908, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 82536, 'rtt': 24846472, 'rttvar': 1543360, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 54.00111103057861, 'end': 55.000274896621704, 'seconds': 0.9991638660430908, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 55.000274896621704, 'end': 56.00131106376648, 'seconds': 1.0010361671447754, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 82536, 'rtt': 25012711, 'rttvar': 1489998, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 55.000274896621704, 'end': 56.00131106376648, 'seconds': 1.0010361671447754, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 56.00131106376648, 'end': 57.00131893157959, 'seconds': 1.0000078678131104, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 83984, 'rtt': 25377968, 'rttvar': 1494467, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 56.00131106376648, 'end': 57.00131893157959, 'seconds': 1.0000078678131104, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 57.00131893157959, 'end': 58.001078844070435, 'seconds': 0.9997599124908447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 85432, 'rtt': 25565444, 'rttvar': 1495802, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 57.00131893157959, 'end': 58.001078844070435, 'seconds': 0.9997599124908447, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 58.001078844070435, 'end': 59.00131797790527, 'seconds': 1.0002391338348389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 86880, 'rtt': 25960673, 'rttvar': 1549853, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 58.001078844070435, 'end': 59.00131797790527, 'seconds': 1.0002391338348389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}], 'end': {}, 'error': 'error - control socket has closed unexpectedly'}" 14:56:29.183995 tst iperf3.py:8: ERR: KeyError: 'sum_sent' [trial-2063↪gprs:sysmo↪iperf3.py:8] 14:56:29.192310 tst iperf3.py:8: Test FAILED (209.9 sec) 14:56:29.199424 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Terminating (SIGTERM) 14:56:29.243730 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Terminating (SIGTERM) 14:56:29.287772 run osmo-hlr_10.42.42.2(pid=7384): Terminating (SIGTERM) 14:56:29.330635 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Terminating (SIGTERM) 14:56:29.375078 run osmo-stp_10.42.42.5(pid=7386): Terminating (SIGTERM) 14:56:29.420083 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Terminating (SIGTERM) 14:56:29.502325 run osmo-ggsn_10.42.42.6(pid=7392): Terminating (SIGTERM) 14:56:29.545810 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Terminating (SIGTERM) 14:56:29.618426 run osmo-sgsn_10.42.42.7(pid=7410): Terminating (SIGTERM) 14:56:29.670286 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Terminating (SIGTERM) 14:56:29.720082 run osmo-msc_10.42.42.8(pid=7412): Terminating (SIGTERM) 14:56:29.768943 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Terminating (SIGTERM) 14:56:29.814635 run osmo-mgw_10.42.42.3(pid=7414): Terminating (SIGTERM) 14:56:29.858555 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Terminating (SIGTERM) 14:56:29.901860 run osmo-mgw_10.42.42.4(pid=7416): Terminating (SIGTERM) 14:56:29.944461 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Terminating (SIGTERM) 14:56:29.987892 run osmo-bsc_10.42.42.9(pid=7418): Terminating (SIGTERM) 14:56:30.030366 run osmo-bts-sysmo(pid=7427): Terminating (SIGTERM) 14:56:30.072926 run osmo-pcu(pid=7435): Terminating (SIGTERM) 14:56:30.115431 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGTERM) 14:56:30.259611 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGTERM(pid=7491): Launched 14:56:31.377651 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGTERM(pid=7491): Terminated: ok {rc=0} 14:56:31.444062 --- ParallelTerminationStrategy: PID 7379 died... 14:56:31.464210 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7379): Terminated: ok {rc=0} 14:56:31.511058 --- ParallelTerminationStrategy: PID 7383 died... 14:56:31.530601 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7383): Terminated: ok {rc=0} 14:56:31.577309 --- ParallelTerminationStrategy: PID 7384 died... 14:56:31.596989 run osmo-hlr_10.42.42.2(pid=7384): Terminated: ok {rc=0} 14:56:31.643536 --- ParallelTerminationStrategy: PID 7385 died... 14:56:31.663099 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7385): Terminated: ok {rc=0} 14:56:31.709850 --- ParallelTerminationStrategy: PID 7386 died... 14:56:31.729490 run osmo-stp_10.42.42.5(pid=7386): Terminated: ok {rc=0} 14:56:31.776060 --- ParallelTerminationStrategy: PID 7387 died... 14:56:31.795572 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7387): Terminated: ok {rc=0} 14:56:31.842088 --- ParallelTerminationStrategy: PID 7392 died... 14:56:31.861595 run osmo-ggsn_10.42.42.6(pid=7392): Terminated: ok {rc=0} 14:56:31.908114 --- ParallelTerminationStrategy: PID 7409 died... 14:56:31.927669 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7409): Terminated: ok {rc=0} 14:56:31.974279 --- ParallelTerminationStrategy: PID 7410 died... 14:56:31.993993 run osmo-sgsn_10.42.42.7(pid=7410): Terminated: ok {rc=0} 14:56:32.040870 --- ParallelTerminationStrategy: PID 7411 died... 14:56:32.060710 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7411): Terminated: ok {rc=0} 14:56:32.107677 --- ParallelTerminationStrategy: PID 7412 died... 14:56:32.127421 run osmo-msc_10.42.42.8(pid=7412): Terminated: ok {rc=0} 14:56:32.174057 --- ParallelTerminationStrategy: PID 7413 died... 14:56:32.193839 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7413): Terminated: ok {rc=0} 14:56:32.240917 --- ParallelTerminationStrategy: PID 7414 died... 14:56:32.260723 run osmo-mgw_10.42.42.3(pid=7414): Terminated {rc=15} 14:56:32.308179 --- ParallelTerminationStrategy: PID 7415 died... 14:56:32.327945 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7415): Terminated: ok {rc=0} 14:56:32.374877 --- ParallelTerminationStrategy: PID 7416 died... 14:56:32.394478 run osmo-mgw_10.42.42.4(pid=7416): Terminated {rc=15} 14:56:32.441299 --- ParallelTerminationStrategy: PID 7417 died... 14:56:32.461059 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7417): Terminated: ok {rc=0} 14:56:32.507877 --- ParallelTerminationStrategy: PID 7418 died... 14:56:32.527751 run osmo-bsc_10.42.42.9(pid=7418): Terminated: ok {rc=0} 14:56:32.574571 --- ParallelTerminationStrategy: PID 7427 died... 14:56:32.597963 run osmo-bts-sysmo(pid=7427): Terminated: ok {rc=0} 14:56:32.644989 --- ParallelTerminationStrategy: PID 7435 died... 14:56:32.664900 run osmo-pcu(pid=7435): Terminated: ok {rc=0} 14:56:38.534354 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGINT) 14:56:38.677251 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGINT(pid=7496): Launched 14:56:39.761459 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGINT(pid=7496): Terminated: ok {rc=0} 14:56:45.650393 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464): Terminating (SIGKILL) 14:56:45.793044 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGUSR1(pid=7499): Launched 14:56:46.883819 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7464)-killSignals.SIGUSR1(pid=7499): Terminated: ok {rc=0} 14:56:47.174050 bus /gobi_6: Setting Online False 14:56:50.011371 bus /gobi_6: Setting Powered False 14:57:03.937250 bus /gobi_6: Setting Powered False ---------------------------------------------- trial-2063 gprs:sysmo iperf3m4.py ---------------------------------------------- 14:57:05.142184 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.153495 tst gprs:sysmo: Using 1 x bts (candidates: 1) 14:57:05.190241 tst iperf3m4.py: using LAC 3940 14:57:05.319314 tst iperf3m4.py: using RAC 115 14:57:05.439922 tst iperf3m4.py: using CellId 3940 14:57:05.561469 tst iperf3m4.py: using BVCI 3941 14:57:05.673864 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.684478 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.694887 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.705188 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.715400 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.725703 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.736119 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.746357 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 14:57:05.757397 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:57:05.877351 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:57:05.999605 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:57:06.115524 tst gprs:sysmo: Using 1 x modem (candidates: 4) 14:57:06.233958 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5003... 14:57:06.457548 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:57:06.789541 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7502): Launched 14:57:07.077316 run iperf3-srv_10.42.42.10(pid=7503): Launched 14:57:07.202650 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5004... 14:57:07.424751 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:57:07.755732 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7504): Launched 14:57:08.042484 run iperf3-srv_10.42.42.10(pid=7505): Launched 14:57:08.166673 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5005... 14:57:08.389145 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:57:08.721607 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7506): Launched 14:57:09.009326 run iperf3-srv_10.42.42.10(pid=7507): Launched 14:57:09.133544 tst iperf3m4.py:8: start iperfv3 server 10.42.42.10:5006... 14:57:09.354912 run iperf3-srv_10.42.42.10: Starting iperf3-srv 14:57:09.685161 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7508): Launched 14:57:09.972925 run iperf3-srv_10.42.42.10(pid=7509): Launched 14:57:10.097578 tst iperf3m4.py:8: start network... 14:57:10.319294 run osmo-hlr_10.42.42.2: Starting osmo-hlr 14:57:10.831103 run create_hlr_db(pid=7510): Launched 14:57:10.958031 bus /gobi_6: Setting Powered False 14:57:12.023875 run create_hlr_db(pid=7510): Terminated: ok {rc=0} 14:57:12.480826 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7512): Launched 14:57:12.812284 run osmo-hlr_10.42.42.2(pid=7513): Launched 14:57:12.939767 run osmo-stp_10.42.42.5: Starting osmo-stp 14:57:13.502915 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7514): Launched 14:57:13.831143 run osmo-stp_10.42.42.5(pid=7515): Launched 14:57:13.957397 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 14:57:14.519939 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7516): Launched 14:57:14.646816 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 14:57:14.924499 run patchelf(pid=7517): Launched 14:57:16.110137 run patchelf(pid=7517): Terminated: ok {rc=0} 14:57:16.247815 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 14:57:16.528464 run osmo-gsm-tester_setcap_net_admin.sh(pid=7518): Launched 14:57:17.717638 run osmo-gsm-tester_setcap_net_admin.sh(pid=7518): Terminated: ok {rc=0} 14:57:18.059130 run osmo-ggsn_10.42.42.6(pid=7521): Launched 14:57:18.186600 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 14:57:18.868883 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7541): Launched 14:57:19.176692 run osmo-sgsn_10.42.42.7(pid=7542): Launched 14:57:19.294269 run osmo-msc_10.42.42.8: Starting osmo-msc 14:57:19.884880 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7543): Launched 14:57:20.232107 run osmo-msc_10.42.42.8(pid=7544): Launched 14:57:23.405825 run osmo-mgw_10.42.42.3: Starting osmo-mgw 14:57:23.931019 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7545): Launched 14:57:24.237832 run osmo-mgw_10.42.42.3(pid=7546): Launched 14:57:24.354557 run osmo-mgw_10.42.42.4: Starting osmo-mgw 14:57:24.870827 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7547): Launched 14:57:25.178062 run osmo-mgw_10.42.42.4(pid=7548): Launched 14:57:25.296504 run osmo-bsc_10.42.42.9: Starting osmo-bsc 14:57:25.932728 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7549): Launched 14:57:26.243210 run osmo-bsc_10.42.42.9(pid=7550): Launched 14:57:32.518147 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 14:57:33.054370 run rm-remote-dir(pid=7551): Launched 14:57:35.241980 run rm-remote-dir(pid=7551): Terminated: ok {rc=0} 14:57:35.579710 run mk-remote-dir(pid=7552): Launched 14:57:36.761788 run mk-remote-dir(pid=7552): Terminated: ok {rc=0} 14:57:37.049342 run scp-inst-to-remote(pid=7553): Launched 14:57:49.245002 run scp-inst-to-remote(pid=7553): Terminated: ok {rc=0} 14:57:49.579299 run mk-remote-dir(pid=7566): Launched 14:57:50.762980 run mk-remote-dir(pid=7566): Terminated: ok {rc=0} 14:57:51.070381 run scp-cfg-to-remote(pid=7567): Launched 14:57:52.258950 run scp-cfg-to-remote(pid=7567): Terminated: ok {rc=0} 14:57:52.612254 run reload-dsp-firmware(pid=7569): Launched 14:57:54.802262 run reload-dsp-firmware(pid=7569): Terminated: ok {rc=0} 14:57:55.148238 run osmo-bts-sysmo(pid=7570): Launched 14:57:59.388385 tst iperf3m4.py:8: Waiting for bts to be ready... 14:58:01.116348 run rm-remote-dir(pid=7571): Launched 14:58:03.302285 run rm-remote-dir(pid=7571): Terminated: ok {rc=0} 14:58:03.649925 run mk-remote-dir(pid=7572): Launched 14:58:05.836500 run mk-remote-dir(pid=7572): Terminated: ok {rc=0} 14:58:06.133919 run scp-inst-to-bts(pid=7573): Launched 14:58:19.343300 run scp-inst-to-bts(pid=7573): Terminated: ok {rc=0} 14:58:19.691490 run mk-remote-run-dir(pid=7575): Launched 14:58:21.877845 run mk-remote-run-dir(pid=7575): Terminated: ok {rc=0} 14:58:22.175969 run scp-cfg-to-sysmobts(pid=7576): Launched 14:58:24.364571 run scp-cfg-to-sysmobts(pid=7576): Terminated: ok {rc=0} 14:58:24.712561 run osmo-pcu(pid=7578): Launched 14:58:24.846645 tst iperf3m4.py: using MSISDN 3941 14:58:25.021760 bus /gobi_6: Setting Powered True 14:58:28.555765 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3941', subscriber_id=1} 14:58:28.840192 bus /gobi_6: Setting Online False 14:58:30.034097 bus /gobi_6: Setting Powered False 14:58:37.132924 bus /gobi_6: Setting Powered True 14:58:38.514938 bus /gobi_6: Setting Online True 14:58:40.955995 tst /gobi_6: Connect to ('901', '70') 14:58:41.244451 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648' 14:58:41.374974 tst iperf3m4.py: using MSISDN 3942 14:58:41.655786 bus /gobi_0: Setting Powered True 14:58:45.160357 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023847', msisdn='3942', subscriber_id=2} 14:58:45.433223 bus /gobi_0: Setting Online False 14:58:46.622313 bus /gobi_0: Setting Powered False 14:58:48.124529 tst /gobi_6: Already registered with network ('901', '70') 14:58:54.117975 bus /gobi_0: Setting Powered True 14:58:55.474023 bus /gobi_0: Setting Online True 14:58:58.064889 tst /gobi_0: Connect to ('901', '70') 14:58:58.349760 tst /gobi_0: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024271873' 14:58:58.472454 tst iperf3m4.py: using MSISDN 3943 14:58:58.749967 bus /gobi_1: Setting Powered True 14:59:02.578168 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023848', msisdn='3943', subscriber_id=3} 14:59:02.869657 bus /gobi_1: Setting Online False 14:59:04.062139 bus /gobi_1: Setting Powered False 14:59:11.127590 bus /gobi_1: Setting Powered True 14:59:12.532078 bus /gobi_1: Setting Online True 14:59:13.980083 tst /gobi_0: Already registered with network ('901', '70') 14:59:15.002627 tst /gobi_1: Connect to ('901', '70') 14:59:15.272761 tst /gobi_1: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A05E2G 1 [Mar 14 2016 19:33:01]', Serial: '868323024272103' 14:59:15.392802 tst iperf3m4.py: using MSISDN 3944 14:59:15.617253 bus /gobi_4: Setting Powered True 14:59:19.389873 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000023849', msisdn='3944', subscriber_id=4} 14:59:19.722850 bus /gobi_4: Setting Online False 14:59:20.917144 bus /gobi_4: Setting Powered False 14:59:24.180687 tst /gobi_1: Already registered with network ('901', '70') 14:59:30.252230 bus /gobi_4: Setting Powered True 14:59:32.008353 bus /gobi_4: Setting Online True 14:59:34.429637 tst /gobi_4: Connect to ('901', '70') 14:59:34.691846 tst /gobi_4: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A08E2G 1 [Jul 21 2016 10:00:50]', Serial: '868323025161511' 14:59:34.801098 tst iperf3m4.py:8: waiting for modems to attach... 14:59:43.743323 tst /gobi_4: Already registered with network ('901', '70') 14:59:47.599021 tst iperf3m4.py:8: waiting for modems to attach to data services... 14:59:56.062415 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 14:59:56.175075 tst iperf3m4.py:8: Setting up data plan for "'/gobi_6/context2'" 14:59:56.596039 run move_netns(pid=7582): Launched 14:59:57.784313 run move_netns(pid=7582): Terminated: ok {rc=0} 14:59:58.115967 run ifup(pid=7585): Launched 14:59:59.307282 run ifup(pid=7585): Terminated: ok {rc=0} 14:59:59.657878 run dhcp(pid=7588): Launched 15:00:00.844384 run dhcp(pid=7588): Terminated: ok {rc=0} 15:00:09.181187 tst /gobi_0: context activated {apn='inet46', path='/gobi_0/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.2', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='r82dcf628584c1c', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 15:00:09.298333 tst iperf3m4.py:8: Setting up data plan for "'/gobi_0/context2'" 15:00:09.740953 run move_netns(pid=7639): Launched 15:00:10.930432 run move_netns(pid=7639): Terminated: ok {rc=0} 15:00:11.279546 run ifup(pid=7642): Launched 15:00:12.469000 run ifup(pid=7642): Terminated: ok {rc=0} 15:00:12.824362 run dhcp(pid=7758): Launched 15:00:14.059118 run dhcp(pid=7758): Terminated: ok {rc=0} 15:00:17.180989 tst /gobi_1: context activated {apn='inet46', path='/gobi_1/context8', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.3', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.1', Interface='rf303c5eb721dad', Method='static', Netmask='255.255.255.248'}, Type='internet', Username='ogt'}, user='ogt'} 15:00:17.304661 tst iperf3m4.py:8: Setting up data plan for "'/gobi_1/context8'" 15:00:17.760032 run move_netns(pid=8162): Launched 15:00:18.983668 run move_netns(pid=8162): Terminated: ok {rc=0} 15:00:19.325815 run ifup(pid=8167): Launched 15:00:20.558947 run ifup(pid=8167): Terminated: ok {rc=0} 15:00:20.886447 run dhcp(pid=8171): Launched 15:00:22.077957 run dhcp(pid=8171): Terminated: ok {rc=0} 15:00:25.185892 tst /gobi_4: context activated {apn='inet46', path='/gobi_4/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.4', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.5', Interface='r9d788e55295d1b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 15:00:25.303270 tst iperf3m4.py:8: Setting up data plan for "'/gobi_4/context3'" 15:00:25.748046 run move_netns(pid=8187): Launched 15:00:26.945681 run move_netns(pid=8187): Terminated: ok {rc=0} 15:00:27.294435 run ifup(pid=8192): Launched 15:00:28.482652 run ifup(pid=8192): Terminated: ok {rc=0} 15:00:28.831497 run dhcp(pid=8195): Launched 15:00:30.015690 run dhcp(pid=8195): Terminated: ok {rc=0} 15:00:30.144465 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5003 through "'/gobi_6/context2'" 15:00:30.353810 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5003 (proto=tcp,time=60s) 15:00:30.719071 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211): Launched 15:00:30.906955 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5004 through "'/gobi_0/context2'" 15:00:31.132326 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5004 (proto=tcp,time=60s) 15:00:31.520777 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214): Launched 15:00:31.699434 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5005 through "'/gobi_1/context8'" 15:00:31.909769 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5005 (proto=tcp,time=60s) 15:00:32.275366 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217): Launched 15:00:32.447993 tst iperf3m4.py:8: Running iperf3 client to 10.42.42.10:5006 through "'/gobi_4/context3'" 15:00:32.657118 run iperf3-cli_10.42.42.10: Preparing iperf3-client connecting to 10.42.42.10:5006 (proto=tcp,time=60s) 15:00:33.020661 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220): Launched 15:00:33.340773 run iperf3-cli_10.42.42.10(pid=8223): Launched 15:00:33.610927 run iperf3-cli_10.42.42.10(pid=8226): Launched 15:00:33.886265 run iperf3-cli_10.42.42.10(pid=8229): Launched 15:00:34.160035 run iperf3-cli_10.42.42.10(pid=8232): Launched 15:01:48.473767 run iperf3-cli_10.42.42.10(pid=8223): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=8223)] 15:01:48.608822 run iperf3-cli_10.42.42.10(pid=8223): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 15:01:48.744289 run iperf3-cli_10.42.42.10(pid=8223): stderr: | (launched: 2022-02-24_15:00:33.325873) 15:01:49.936354 run iperf3-cli_10.42.42.10(pid=8226): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=8226)] 15:01:50.070299 run iperf3-cli_10.42.42.10(pid=8226): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 15:01:50.205210 run iperf3-cli_10.42.42.10(pid=8226): stderr: | (launched: 2022-02-24_15:00:33.595607) 15:01:51.397050 run iperf3-cli_10.42.42.10(pid=8229): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=8229)] 15:01:51.530493 run iperf3-cli_10.42.42.10(pid=8229): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 15:01:51.664901 run iperf3-cli_10.42.42.10(pid=8229): stderr: | (launched: 2022-02-24_15:00:33.871204) 15:01:52.863568 run iperf3-cli_10.42.42.10(pid=8232): ERR: Terminated: ERROR {rc=1} [trial-2063↪gprs:sysmo↪iperf3m4.py:8↪iperf3m4.py↪iperf3-srv_10.42.42.10↪iperf3-cli_10.42.42.10↪iperf3-cli_10.42.42.10(pid=8232)] 15:01:52.999113 run iperf3-cli_10.42.42.10(pid=8232): stdout: | "bits_per_second": 0, | "retransmits": 0, | "omitted": false | } | }], | "end": { | }, | "error": "error - control socket has closed unexpectedly" | } | child exited with 1 15:01:53.134640 run iperf3-cli_10.42.42.10(pid=8232): stderr: | (launched: 2022-02-24_15:00:34.144755) 15:01:53.270163 run iperf3-srv_10.42.42.10(pid=7503): Terminating (SIGINT) 15:01:53.447188 run iperf3-srv_10.42.42.10(pid=7503): Terminated {rc=1} 15:01:53.568328 tst iperf3m4.py:8: Results for 10.42.42.10:5003 through "'/gobi_6/context2'" Exception while using iperf3 client results: "{'start': {'connected': [{'socket': 5, 'local_host': '176.16.46.1', 'local_port': 33886, 'remote_host': '10.42.42.10', 'remote_port': 5003}], 'version': 'iperf 3.6', 'system_info': 'Linux osmo-gsm-tester-prod 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64', 'timestamp': {'time': 'Thu, 24 Feb 2022 14:00:40 GMT', 'timesecs': 1645711240}, 'connecting_to': {'host': '10.42.42.10', 'port': 5003}, 'cookie': 'zt2pdik2yclewdxetdwazq5axvloi5e6zzrv', 'tcp_mss_default': 1448, 'sock_bufsize': 0, 'sndbuf_actual': 16384, 'rcvbuf_actual': 131072, 'test_start': {'protocol': 'TCP', 'num_streams': 1, 'blksize': 131072, 'omit': 0, 'duration': 60, 'bytes': 0, 'blocks': 0, 'reverse': 0, 'tos': 0}}, 'intervals': [{'streams': [{'socket': 5, 'start': 0, 'end': 1.0008411407470703, 'seconds': 1.0008411407470703, 'bytes': 101360, 'bits_per_second': 810198.5090208469, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 507826, 'rttvar': 295354, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 0, 'end': 1.0008411407470703, 'seconds': 1.0008411407470703, 'bytes': 101360, 'bits_per_second': 810198.5090208469, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 1.0008411407470703, 'end': 2.0013129711151123, 'seconds': 1.000471830368042, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 14480, 'rtt': 507826, 'rttvar': 295354, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 1.0008411407470703, 'end': 2.0013129711151123, 'seconds': 1.000471830368042, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 2.0013129711151123, 'end': 3.001160144805908, 'seconds': 0.9998471736907959, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 15928, 'rtt': 688929, 'rttvar': 583722, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 2.0013129711151123, 'end': 3.001160144805908, 'seconds': 0.9998471736907959, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 3.001160144805908, 'end': 4.001162052154541, 'seconds': 1.0000019073486328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'snd_cwnd': 17376, 'rtt': 1034938, 'rttvar': 1129810, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 3.001160144805908, 'end': 4.001162052154541, 'seconds': 1.0000019073486328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 1, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 4.001162052154541, 'end': 5.001213073730469, 'seconds': 1.0000510215759277, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 17376, 'rtt': 1034938, 'rttvar': 1129810, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 4.001162052154541, 'end': 5.001213073730469, 'seconds': 1.0000510215759277, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 5.001213073730469, 'end': 6.001160144805908, 'seconds': 0.9999470710754395, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 18824, 'rtt': 1528058, 'rttvar': 1833597, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 5.001213073730469, 'end': 6.001160144805908, 'seconds': 0.9999470710754395, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 6.001160144805908, 'end': 7.001261949539185, 'seconds': 1.0001018047332764, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 20272, 'rtt': 2151663, 'rttvar': 2622408, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 6.001160144805908, 'end': 7.001261949539185, 'seconds': 1.0001018047332764, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 7.001261949539185, 'end': 8.001152038574219, 'seconds': 0.9998900890350342, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 20272, 'rtt': 2151663, 'rttvar': 2622408, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 7.001261949539185, 'end': 8.001152038574219, 'seconds': 0.9998900890350342, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 8.001152038574219, 'end': 9.001158952713013, 'seconds': 1.000006914138794, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 21720, 'rtt': 2887537, 'rttvar': 3438553, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 8.001152038574219, 'end': 9.001158952713013, 'seconds': 1.000006914138794, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 9.001158952713013, 'end': 10.001160144805908, 'seconds': 1.0000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 23168, 'rtt': 3723545, 'rttvar': 4250933, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 9.001158952713013, 'end': 10.001160144805908, 'seconds': 1.0000011920928955, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 10.001160144805908, 'end': 11.000309944152832, 'seconds': 0.9991497993469238, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 23168, 'rtt': 3723545, 'rttvar': 4250933, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 10.001160144805908, 'end': 11.000309944152832, 'seconds': 0.9991497993469238, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 11.000309944152832, 'end': 12.001065015792847, 'seconds': 1.0007550716400146, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 24616, 'rtt': 4637973, 'rttvar': 5017054, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 11.000309944152832, 'end': 12.001065015792847, 'seconds': 1.0007550716400146, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 12.001065015792847, 'end': 13.001322031021118, 'seconds': 1.0002570152282715, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 26064, 'rtt': 5622602, 'rttvar': 5732049, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 12.001065015792847, 'end': 13.001322031021118, 'seconds': 1.0002570152282715, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 13.001322031021118, 'end': 14.001060962677002, 'seconds': 0.9997389316558838, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 26064, 'rtt': 5622602, 'rttvar': 5732049, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 13.001322031021118, 'end': 14.001060962677002, 'seconds': 0.9997389316558838, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 14.001060962677002, 'end': 15.001315116882324, 'seconds': 1.0002541542053223, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 27512, 'rtt': 6671640, 'rttvar': 6397113, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 14.001060962677002, 'end': 15.001315116882324, 'seconds': 1.0002541542053223, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 15.001315116882324, 'end': 16.00114893913269, 'seconds': 0.9998338222503662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 28960, 'rtt': 7782242, 'rttvar': 7019039, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 15.001315116882324, 'end': 16.00114893913269, 'seconds': 0.9998338222503662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 16.00114893913269, 'end': 17.001261949539185, 'seconds': 1.0001130104064941, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 28960, 'rtt': 7782242, 'rttvar': 7019039, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 16.00114893913269, 'end': 17.001261949539185, 'seconds': 1.0001130104064941, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 17.001261949539185, 'end': 18.001073122024536, 'seconds': 0.9998111724853516, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 30408, 'rtt': 8812917, 'rttvar': 7325629, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 17.001261949539185, 'end': 18.001073122024536, 'seconds': 0.9998111724853516, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 18.001073122024536, 'end': 19.00130009651184, 'seconds': 1.0002269744873047, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 9788700, 'rttvar': 7445788, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 18.001073122024536, 'end': 19.00130009651184, 'seconds': 1.0002269744873047, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 19.00130009651184, 'end': 20.00106906890869, 'seconds': 0.9997689723968506, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 9788700, 'rttvar': 7445788, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 19.00130009651184, 'end': 20.00106906890869, 'seconds': 0.9997689723968506, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 20.00106906890869, 'end': 21.001117944717407, 'seconds': 1.0000488758087158, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 31856, 'rtt': 9788700, 'rttvar': 7445788, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 20.00106906890869, 'end': 21.001117944717407, 'seconds': 1.0000488758087158, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 21.001117944717407, 'end': 22.00130009651184, 'seconds': 1.0001821517944336, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 33304, 'rtt': 10827621, 'rttvar': 7662183, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 21.001117944717407, 'end': 22.00130009651184, 'seconds': 1.0001821517944336, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 22.00130009651184, 'end': 23.00131392478943, 'seconds': 1.000013828277588, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 33304, 'rtt': 10827621, 'rttvar': 7662183, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 22.00130009651184, 'end': 23.00131392478943, 'seconds': 1.000013828277588, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 23.00131392478943, 'end': 24.001081943511963, 'seconds': 0.9997680187225342, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 11931626, 'rttvar': 7954646, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 23.00131392478943, 'end': 24.001081943511963, 'seconds': 0.9997680187225342, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 24.001081943511963, 'end': 25.001097917556763, 'seconds': 1.0000159740447998, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 34752, 'rtt': 11931626, 'rttvar': 7954646, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 24.001081943511963, 'end': 25.001097917556763, 'seconds': 1.0000159740447998, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 25.001097917556763, 'end': 26.001063108444214, 'seconds': 0.9999651908874512, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 36200, 'rtt': 12894720, 'rttvar': 7892174, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 25.001097917556763, 'end': 26.001063108444214, 'seconds': 0.9999651908874512, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 26.001063108444214, 'end': 27.001151084899902, 'seconds': 1.0000879764556885, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 37648, 'rtt': 13922595, 'rttvar': 7974879, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 26.001063108444214, 'end': 27.001151084899902, 'seconds': 1.0000879764556885, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 27.001151084899902, 'end': 28.001160144805908, 'seconds': 1.0000090599060059, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 37648, 'rtt': 13922595, 'rttvar': 7974879, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 27.001151084899902, 'end': 28.001160144805908, 'seconds': 1.0000090599060059, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 28.001160144805908, 'end': 29.001160144805908, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 39096, 'rtt': 14819696, 'rttvar': 7775362, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 28.001160144805908, 'end': 29.001160144805908, 'seconds': 1, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 29.001160144805908, 'end': 30.001260995864868, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 40544, 'rtt': 15792185, 'rttvar': 7776501, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 29.001160144805908, 'end': 30.001260995864868, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 30.001260995864868, 'end': 31.001161098480225, 'seconds': 0.9999001026153564, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 40544, 'rtt': 15792185, 'rttvar': 7776501, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 30.001260995864868, 'end': 31.001161098480225, 'seconds': 0.9999001026153564, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 31.001161098480225, 'end': 32.00115609169006, 'seconds': 0.9999949932098389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 41992, 'rtt': 16655268, 'rttvar': 7558540, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 31.001161098480225, 'end': 32.00115609169006, 'seconds': 0.9999949932098389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 32.00115609169006, 'end': 33.001302003860474, 'seconds': 1.0001459121704102, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 17597902, 'rttvar': 7554173, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 32.00115609169006, 'end': 33.001302003860474, 'seconds': 1.0001459121704102, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 33.001302003860474, 'end': 34.001261949539185, 'seconds': 0.9999599456787109, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 43440, 'rtt': 17597902, 'rttvar': 7554173, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 33.001302003860474, 'end': 34.001261949539185, 'seconds': 0.9999599456787109, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 34.001261949539185, 'end': 35.001315116882324, 'seconds': 1.0000531673431396, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 18423310, 'rttvar': 7316446, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 34.001261949539185, 'end': 35.001315116882324, 'seconds': 1.0000531673431396, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 35.001315116882324, 'end': 36.001161098480225, 'seconds': 0.9998459815979004, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 19335354, 'rttvar': 7311422, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 35.001315116882324, 'end': 36.001161098480225, 'seconds': 0.9998459815979004, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 36.001161098480225, 'end': 37.001222133636475, 'seconds': 1.00006103515625, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 44888, 'rtt': 19335354, 'rttvar': 7311422, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 36.001161098480225, 'end': 37.001222133636475, 'seconds': 1.00006103515625, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 37.001222133636475, 'end': 38.00117802619934, 'seconds': 0.9999558925628662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 20132761, 'rttvar': 7078382, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 37.001222133636475, 'end': 38.00117802619934, 'seconds': 0.9999558925628662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 38.00117802619934, 'end': 39.0012149810791, 'seconds': 1.0000369548797607, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 21023103, 'rttvar': 7089469, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 38.00117802619934, 'end': 39.0012149810791, 'seconds': 1.0000369548797607, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 39.0012149810791, 'end': 40.00115609169006, 'seconds': 0.9999411106109619, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 46336, 'rtt': 21023103, 'rttvar': 7089469, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 39.0012149810791, 'end': 40.00115609169006, 'seconds': 0.9999411106109619, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 40.00115609169006, 'end': 41.00115895271301, 'seconds': 1.0000028610229492, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 21807438, 'rttvar': 6885772, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 40.00115609169006, 'end': 41.00115895271301, 'seconds': 1.0000028610229492, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 41.00115895271301, 'end': 42.00122809410095, 'seconds': 1.0000691413879395, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 22684092, 'rttvar': 6917637, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 41.00115895271301, 'end': 42.00122809410095, 'seconds': 1.0000691413879395, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 42.00122809410095, 'end': 43.00122308731079, 'seconds': 0.9999949932098389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 47784, 'rtt': 22684092, 'rttvar': 6917637, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 42.00122809410095, 'end': 43.00122308731079, 'seconds': 0.9999949932098389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 43.00122308731079, 'end': 44.001161098480225, 'seconds': 0.9999380111694336, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 23461052, 'rttvar': 6742149, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 43.00122308731079, 'end': 44.001161098480225, 'seconds': 0.9999380111694336, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 44.001161098480225, 'end': 45.001157999038696, 'seconds': 0.9999969005584717, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 24328350, 'rttvar': 6791206, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 44.001161098480225, 'end': 45.001157999038696, 'seconds': 0.9999969005584717, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 45.001157999038696, 'end': 46.00123596191406, 'seconds': 1.0000779628753662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 49232, 'rtt': 24328350, 'rttvar': 6791206, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 45.001157999038696, 'end': 46.00123596191406, 'seconds': 1.0000779628753662, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 46.00123596191406, 'end': 47.001161098480225, 'seconds': 0.9999251365661621, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 25091763, 'rttvar': 6620231, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 46.00123596191406, 'end': 47.001161098480225, 'seconds': 0.9999251365661621, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 47.001161098480225, 'end': 48.001261949539185, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 25952438, 'rttvar': 6686525, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 47.001161098480225, 'end': 48.001261949539185, 'seconds': 1.00010085105896, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 48.001261949539185, 'end': 49.00122094154358, 'seconds': 0.9999589920043945, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 50680, 'rtt': 25952438, 'rttvar': 6686525, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 48.001261949539185, 'end': 49.00122094154358, 'seconds': 0.9999589920043945, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 49.00122094154358, 'end': 50.001161098480225, 'seconds': 0.9999401569366455, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 26712976, 'rttvar': 6535968, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 49.00122094154358, 'end': 50.001161098480225, 'seconds': 0.9999401569366455, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 50.001161098480225, 'end': 51.00119996070862, 'seconds': 1.0000388622283936, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 26712976, 'rttvar': 6535968, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 50.001161098480225, 'end': 51.00119996070862, 'seconds': 1.0000388622283936, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 51.00119996070862, 'end': 52.00132393836975, 'seconds': 1.0001239776611328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 52128, 'rtt': 27598806, 'rttvar': 6673637, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 51.00119996070862, 'end': 52.00132393836975, 'seconds': 1.0001239776611328, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 52.00132393836975, 'end': 53.001261949539185, 'seconds': 0.9999380111694336, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 28393499, 'rttvar': 6594612, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 52.00132393836975, 'end': 53.001261949539185, 'seconds': 0.9999380111694336, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 53.001261949539185, 'end': 54.00121998786926, 'seconds': 0.9999580383300781, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 28393499, 'rttvar': 6594612, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 53.001261949539185, 'end': 54.00121998786926, 'seconds': 0.9999580383300781, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 54.00121998786926, 'end': 55.000673055648804, 'seconds': 0.999453067779541, 'bytes': 256296, 'bits_per_second': 2051490.025995167, 'retransmits': 0, 'snd_cwnd': 53576, 'rtt': 29279261, 'rttvar': 6717485, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 54.00121998786926, 'end': 55.000673055648804, 'seconds': 0.999453067779541, 'bytes': 256296, 'bits_per_second': 2051490.025995167, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 55.000673055648804, 'end': 56.00105404853821, 'seconds': 1.0003809928894043, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 29866263, 'rttvar': 6212118, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 55.000673055648804, 'end': 56.00105404853821, 'seconds': 1.0003809928894043, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 56.00105404853821, 'end': 57.00104904174805, 'seconds': 0.9999949932098389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 29866263, 'rttvar': 6212118, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 56.00105404853821, 'end': 57.00104904174805, 'seconds': 0.9999949932098389, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 57.00104904174805, 'end': 58.00104808807373, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 55024, 'rtt': 30570254, 'rttvar': 6067069, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 57.00104904174805, 'end': 58.00104808807373, 'seconds': 0.9999990463256836, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}, {'streams': [{'socket': 5, 'start': 58.00104808807373, 'end': 59.001152992248535, 'seconds': 1.0001049041748047, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'snd_cwnd': 56472, 'rtt': 31178785, 'rttvar': 5767365, 'pmtu': 1500, 'omitted': False}], 'sum': {'start': 58.00104808807373, 'end': 59.001152992248535, 'seconds': 1.0001049041748047, 'bytes': 0, 'bits_per_second': 0, 'retransmits': 0, 'omitted': False}}], 'end': {}, 'error': 'error - control socket has closed unexpectedly'}" 15:01:53.798834 tst iperf3m4.py:8: ERR: KeyError: 'sum_sent' [trial-2063↪gprs:sysmo↪iperf3m4.py:8] 15:01:53.806891 tst iperf3m4.py:8: Test FAILED (288.7 sec) 15:01:53.813778 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7502): Terminating (SIGTERM) 15:01:53.856197 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7504): Terminating (SIGTERM) 15:01:53.898337 run iperf3-srv_10.42.42.10(pid=7505): Terminating (SIGTERM) 15:01:53.939636 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7506): Terminating (SIGTERM) 15:01:53.981357 run iperf3-srv_10.42.42.10(pid=7507): Terminating (SIGTERM) 15:01:54.022658 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7508): Terminating (SIGTERM) 15:01:54.064185 run iperf3-srv_10.42.42.10(pid=7509): Terminating (SIGTERM) 15:01:54.105490 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7512): Terminating (SIGTERM) 15:01:54.147197 run osmo-hlr_10.42.42.2(pid=7513): Terminating (SIGTERM) 15:01:54.188344 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7514): Terminating (SIGTERM) 15:01:54.231466 run osmo-stp_10.42.42.5(pid=7515): Terminating (SIGTERM) 15:01:54.273481 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7516): Terminating (SIGTERM) 15:01:54.325120 run osmo-ggsn_10.42.42.6(pid=7521): Terminating (SIGTERM) 15:01:54.368183 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7541): Terminating (SIGTERM) 15:01:54.428629 run osmo-sgsn_10.42.42.7(pid=7542): Terminating (SIGTERM) 15:01:54.484538 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7543): Terminating (SIGTERM) 15:01:54.533377 run osmo-msc_10.42.42.8(pid=7544): Terminating (SIGTERM) 15:01:54.583106 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7545): Terminating (SIGTERM) 15:01:54.633177 run osmo-mgw_10.42.42.3(pid=7546): Terminating (SIGTERM) 15:01:54.681234 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7547): Terminating (SIGTERM) 15:01:54.726053 run osmo-mgw_10.42.42.4(pid=7548): Terminating (SIGTERM) 15:01:54.770359 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7549): Terminating (SIGTERM) 15:01:54.815522 run osmo-bsc_10.42.42.9(pid=7550): Terminating (SIGTERM) 15:01:54.858963 run osmo-bts-sysmo(pid=7570): Terminating (SIGTERM) 15:01:54.901674 run osmo-pcu(pid=7578): Terminating (SIGTERM) 15:01:54.946107 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211): Terminating (SIGTERM) 15:01:55.090172 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGTERM(pid=8307): Launched 15:01:56.201601 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGTERM(pid=8307): Terminated: ok {rc=0} 15:01:56.268556 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214): Terminating (SIGTERM) 15:01:56.410461 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGTERM(pid=8312): Launched 15:01:57.494485 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGTERM(pid=8312): Terminated: ok {rc=0} 15:01:57.556789 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217): Terminating (SIGTERM) 15:01:57.691881 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGTERM(pid=8315): Launched 15:01:58.772310 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGTERM(pid=8315): Terminated: ok {rc=0} 15:01:58.835561 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220): Terminating (SIGTERM) 15:01:58.971003 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGTERM(pid=8318): Launched 15:02:00.059997 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGTERM(pid=8318): Terminated: ok {rc=0} 15:02:00.126082 --- ParallelTerminationStrategy: PID 7502 died... 15:02:00.146028 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7502): Terminated: ok {rc=0} 15:02:00.193090 --- ParallelTerminationStrategy: PID 7504 died... 15:02:00.212808 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7504): Terminated: ok {rc=0} 15:02:00.259625 --- ParallelTerminationStrategy: PID 7505 died... 15:02:00.279130 run iperf3-srv_10.42.42.10(pid=7505): Terminated {rc=256} 15:02:00.325543 --- ParallelTerminationStrategy: PID 7506 died... 15:02:00.344963 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7506): Terminated: ok {rc=0} 15:02:00.391538 --- ParallelTerminationStrategy: PID 7507 died... 15:02:00.411117 run iperf3-srv_10.42.42.10(pid=7507): Terminated {rc=256} 15:02:00.457776 --- ParallelTerminationStrategy: PID 7508 died... 15:02:00.477217 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=7508): Terminated: ok {rc=0} 15:02:00.523850 --- ParallelTerminationStrategy: PID 7509 died... 15:02:00.543301 run iperf3-srv_10.42.42.10(pid=7509): Terminated {rc=256} 15:02:00.589833 --- ParallelTerminationStrategy: PID 7512 died... 15:02:00.609420 run pcap-recorder_any(filters='host 10.42.42.2')(pid=7512): Terminated: ok {rc=0} 15:02:00.656546 --- ParallelTerminationStrategy: PID 7513 died... 15:02:00.676157 run osmo-hlr_10.42.42.2(pid=7513): Terminated: ok {rc=0} 15:02:00.722876 --- ParallelTerminationStrategy: PID 7514 died... 15:02:00.742627 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=7514): Terminated: ok {rc=0} 15:02:00.789505 --- ParallelTerminationStrategy: PID 7515 died... 15:02:00.809261 run osmo-stp_10.42.42.5(pid=7515): Terminated: ok {rc=0} 15:02:00.856013 --- ParallelTerminationStrategy: PID 7516 died... 15:02:00.875664 run pcap-recorder_any(filters='host 10.42.42.6')(pid=7516): Terminated: ok {rc=0} 15:02:00.922409 --- ParallelTerminationStrategy: PID 7521 died... 15:02:00.941959 run osmo-ggsn_10.42.42.6(pid=7521): Terminated: ok {rc=0} 15:02:00.988554 --- ParallelTerminationStrategy: PID 7541 died... 15:02:01.008494 run pcap-recorder_any(filters='host 10.42.42.7')(pid=7541): Terminated: ok {rc=0} 15:02:01.055056 --- ParallelTerminationStrategy: PID 7542 died... 15:02:01.074361 run osmo-sgsn_10.42.42.7(pid=7542): Terminated: ok {rc=0} 15:02:01.121298 --- ParallelTerminationStrategy: PID 7543 died... 15:02:01.140891 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=7543): Terminated: ok {rc=0} 15:02:01.187708 --- ParallelTerminationStrategy: PID 7544 died... 15:02:01.207262 run osmo-msc_10.42.42.8(pid=7544): Terminated: ok {rc=0} 15:02:01.253654 --- ParallelTerminationStrategy: PID 7545 died... 15:02:01.273263 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=7545): Terminated: ok {rc=0} 15:02:01.320113 --- ParallelTerminationStrategy: PID 7546 died... 15:02:01.339604 run osmo-mgw_10.42.42.3(pid=7546): Terminated {rc=15} 15:02:01.386089 --- ParallelTerminationStrategy: PID 7547 died... 15:02:01.405744 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=7547): Terminated: ok {rc=0} 15:02:01.452830 --- ParallelTerminationStrategy: PID 7548 died... 15:02:01.472446 run osmo-mgw_10.42.42.4(pid=7548): Terminated {rc=15} 15:02:01.519081 --- ParallelTerminationStrategy: PID 7549 died... 15:02:01.538624 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=7549): Terminated: ok {rc=0} 15:02:01.585230 --- ParallelTerminationStrategy: PID 7550 died... 15:02:01.605212 run osmo-bsc_10.42.42.9(pid=7550): Terminated: ok {rc=0} 15:02:01.651515 --- ParallelTerminationStrategy: PID 7570 died... 15:02:01.671129 run osmo-bts-sysmo(pid=7570): Terminated: ok {rc=0} 15:02:01.718059 --- ParallelTerminationStrategy: PID 7578 died... 15:02:01.737823 run osmo-pcu(pid=7578): Terminated: ok {rc=0} 15:02:07.607359 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211): Terminating (SIGINT) 15:02:07.749707 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGINT(pid=8321): Launched 15:02:08.837606 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGINT(pid=8321): Terminated: ok {rc=0} 15:02:08.903882 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214): Terminating (SIGINT) 15:02:09.047151 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGINT(pid=8324): Launched 15:02:10.128645 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGINT(pid=8324): Terminated: ok {rc=0} 15:02:10.191231 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217): Terminating (SIGINT) 15:02:10.329961 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGINT(pid=8327): Launched 15:02:11.418129 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGINT(pid=8327): Terminated: ok {rc=0} 15:02:11.484073 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220): Terminating (SIGINT) 15:02:11.625961 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGINT(pid=8330): Launched 15:02:12.714340 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGINT(pid=8330): Terminated: ok {rc=0} 15:02:18.602450 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211): Terminating (SIGKILL) 15:02:18.745482 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGUSR1(pid=8333): Launched 15:02:19.831393 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8211)-killSignals.SIGUSR1(pid=8333): Terminated: ok {rc=0} 15:02:19.897871 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214): Terminating (SIGKILL) 15:02:20.039617 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGUSR1(pid=8336): Launched 15:02:21.125275 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8214)-killSignals.SIGUSR1(pid=8336): Terminated: ok {rc=0} 15:02:21.191616 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217): Terminating (SIGKILL) 15:02:21.332671 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGUSR1(pid=8339): Launched 15:02:22.420389 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8217)-killSignals.SIGUSR1(pid=8339): Terminated: ok {rc=0} 15:02:22.482447 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220): Terminating (SIGKILL) 15:02:22.617056 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGUSR1(pid=8342): Launched 15:02:23.701635 run pcap-recorder_any(filters='host 10.42.42.10 and port not 22')(pid=8220)-killSignals.SIGUSR1(pid=8342): Terminated: ok {rc=0} 15:02:23.945583 bus /gobi_4: Setting Online False 15:02:25.089369 bus /gobi_4: Setting Powered False 15:02:32.944806 bus /gobi_4: Setting Powered False 15:02:34.042642 bus /gobi_4: Setting Powered False 15:02:35.196085 bus /gobi_1: Setting Online False 15:02:36.320609 bus /gobi_1: Setting Powered False 15:02:44.030828 bus /gobi_1: Setting Powered False 15:02:45.129680 bus /gobi_1: Setting Powered False 15:02:46.289040 bus /gobi_0: Setting Online False 15:02:47.423400 bus /gobi_0: Setting Powered False 15:02:55.936577 bus /gobi_0: Setting Powered False 15:02:57.034144 bus /gobi_0: Setting Powered False 15:02:58.190405 bus /gobi_6: Setting Online False 15:02:59.327777 bus /gobi_6: Setting Powered False 15:03:07.945849 bus /gobi_6: Setting Powered False ---------------------------------------------- trial-2063 gprs:sysmo ping.py ---------------------------------------------- 15:03:09.105316 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:03:09.118296 tst gprs:sysmo: Using 1 x bts (candidates: 1) 15:03:09.153268 tst ping.py: using LAC 3941 15:03:09.274477 tst ping.py: using RAC 116 15:03:09.390296 tst ping.py: using CellId 3941 15:03:09.506495 tst ping.py: using BVCI 3942 15:03:09.613673 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:03:09.624528 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:03:09.635123 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:03:09.645880 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:03:09.656675 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:03:09.667589 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:03:09.677960 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:03:09.688297 tst gprs:sysmo: Using 1 x modem (candidates: 4) 15:03:09.803093 tst ping.py:19: start network... 15:03:10.016019 run osmo-hlr_10.42.42.2: Starting osmo-hlr 15:03:10.479552 run create_hlr_db(pid=8345): Launched 15:03:10.600290 bus /gobi_6: Setting Powered False 15:03:11.659805 run create_hlr_db(pid=8345): Terminated: ok {rc=0} 15:03:12.071845 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8347): Launched 15:03:12.366428 run osmo-hlr_10.42.42.2(pid=8348): Launched 15:03:12.477699 run osmo-stp_10.42.42.5: Starting osmo-stp 15:03:12.984087 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8349): Launched 15:03:13.280014 run osmo-stp_10.42.42.5(pid=8350): Launched 15:03:13.392764 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 15:03:13.899973 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8351): Launched 15:03:14.012114 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 15:03:14.260704 run patchelf(pid=8352): Launched 15:03:15.426561 run patchelf(pid=8352): Terminated: ok {rc=0} 15:03:15.550400 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 15:03:15.808103 run osmo-gsm-tester_setcap_net_admin.sh(pid=8353): Launched 15:03:16.974674 run osmo-gsm-tester_setcap_net_admin.sh(pid=8353): Terminated: ok {rc=0} 15:03:17.279329 run osmo-ggsn_10.42.42.6(pid=8356): Launched 15:03:17.390689 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 15:03:18.032094 run pcap-recorder_any(filters='host 10.42.42.7')(pid=8373): Launched 15:03:18.346933 run osmo-sgsn_10.42.42.7(pid=8374): Launched 15:03:18.468879 run osmo-msc_10.42.42.8: Starting osmo-msc 15:03:19.040568 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=8375): Launched 15:03:19.355772 run osmo-msc_10.42.42.8(pid=8376): Launched 15:03:22.534096 run osmo-mgw_10.42.42.3: Starting osmo-mgw 15:03:23.072697 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8377): Launched 15:03:23.390999 run osmo-mgw_10.42.42.3(pid=8378): Launched 15:03:23.512523 run osmo-mgw_10.42.42.4: Starting osmo-mgw 15:03:24.047861 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8379): Launched 15:03:24.365519 run osmo-mgw_10.42.42.4(pid=8380): Launched 15:03:24.488623 run osmo-bsc_10.42.42.9: Starting osmo-bsc 15:03:25.148876 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=8381): Launched 15:03:25.468571 run osmo-bsc_10.42.42.9(pid=8382): Launched 15:03:31.751161 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 15:03:32.303789 run rm-remote-dir(pid=8383): Launched 15:03:34.491724 run rm-remote-dir(pid=8383): Terminated: ok {rc=0} 15:03:34.840258 run mk-remote-dir(pid=8384): Launched 15:03:36.025985 run mk-remote-dir(pid=8384): Terminated: ok {rc=0} 15:03:36.322995 run scp-inst-to-remote(pid=8385): Launched 15:03:48.523091 run scp-inst-to-remote(pid=8385): Terminated: ok {rc=0} 15:03:48.847408 run mk-remote-dir(pid=8387): Launched 15:03:50.029865 run mk-remote-dir(pid=8387): Terminated: ok {rc=0} 15:03:50.324877 run scp-cfg-to-remote(pid=8388): Launched 15:03:51.507829 run scp-cfg-to-remote(pid=8388): Terminated: ok {rc=0} 15:03:51.849196 run reload-dsp-firmware(pid=8390): Launched 15:03:54.029448 run reload-dsp-firmware(pid=8390): Terminated: ok {rc=0} 15:03:54.341205 run osmo-bts-sysmo(pid=8391): Launched 15:03:58.566630 tst ping.py:31: Waiting for bts to be ready... 15:04:00.279107 run rm-remote-dir(pid=8392): Launched 15:04:02.457692 run rm-remote-dir(pid=8392): Terminated: ok {rc=0} 15:04:02.771355 run mk-remote-dir(pid=8393): Launched 15:04:04.940813 run mk-remote-dir(pid=8393): Terminated: ok {rc=0} 15:04:05.210375 run scp-inst-to-bts(pid=8394): Launched 15:04:18.394172 run scp-inst-to-bts(pid=8394): Terminated: ok {rc=0} 15:04:18.710489 run mk-remote-run-dir(pid=8396): Launched 15:04:20.880767 run mk-remote-run-dir(pid=8396): Terminated: ok {rc=0} 15:04:21.149920 run scp-cfg-to-sysmobts(pid=8397): Launched 15:04:23.319031 run scp-cfg-to-sysmobts(pid=8397): Terminated: ok {rc=0} 15:04:23.636243 run osmo-pcu(pid=8399): Launched 15:04:23.761347 tst ping.py: using MSISDN 3945 15:04:23.933532 bus /gobi_6: Setting Powered True 15:04:27.436219 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3945', subscriber_id=1} 15:04:27.706681 bus /gobi_6: Setting Online False 15:04:28.893956 bus /gobi_6: Setting Powered False 15:04:36.109740 bus /gobi_6: Setting Powered True 15:04:37.494440 bus /gobi_6: Setting Online True 15:04:39.929274 tst /gobi_6: Connect to ('901', '70') 15:04:40.188561 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648' 15:04:40.295859 tst ping.py:42: waiting for modems to attach... 15:04:53.535500 tst /gobi_6: Already registered with network ('901', '70') 15:04:58.873053 tst ping.py:46: waiting for modems to attach to data services... 15:05:19.937640 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 15:05:20.046650 tst ping.py:51: Setting up data plan for "'/gobi_6/context3'" 15:05:20.465692 run move_netns(pid=8405): Launched 15:05:21.645938 run move_netns(pid=8405): Terminated: ok {rc=0} 15:05:21.980661 run ifup(pid=8408): Launched 15:05:23.159517 run ifup(pid=8408): Terminated: ok {rc=0} 15:05:23.487922 run dhcp(pid=8411): Launched 15:05:24.671516 run dhcp(pid=8411): Terminated: ok {rc=0} 15:05:24.799486 tst ping.py:53: Running 10 ping requests for "'/gobi_6/context3'" 15:05:25.199790 run ping(pid=8427): Launched 15:05:35.394677 run ping(pid=8427): Terminated: ok {rc=0} 15:05:37.263612 tst /gobi_6: context deactivated {path='/gobi_6/context3'} 15:05:44.480587 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context3', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={Interface='r433306bbc2604b'}, Name='Internet', Password='', Protocol='ipv6', Settings={}, Type='internet', Username='ogt'}, user='ogt'} 15:05:51.986447 tst /gobi_6: context deactivated {path='/gobi_6/context3'} 15:05:52.101861 tst ping.py:70: (launched: 2022-02-24_15:05:25.137632) 8428: waiting for 8429 PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data. 64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=439 ms 64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=618 ms 64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=595 ms 64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=434 ms 64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=435 ms 64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=411 ms 64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=393 ms 64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=389 ms 64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=408 ms 64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=390 ms --- 10.42.42.6 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 21ms rtt min/avg/max/mdev = 388.920/451.214/618.001/79.859 ms child exited with 0 15:05:52.320702 tst ping.py: Test passed (163.2 sec) 15:05:52.388566 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8347): Terminating (SIGTERM) 15:05:52.431460 run osmo-hlr_10.42.42.2(pid=8348): Terminating (SIGTERM) 15:05:52.474394 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8349): Terminating (SIGTERM) 15:05:52.519629 run osmo-stp_10.42.42.5(pid=8350): Terminating (SIGTERM) 15:05:52.563115 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8351): Terminating (SIGTERM) 15:05:52.635640 run osmo-ggsn_10.42.42.6(pid=8356): Terminating (SIGTERM) 15:05:52.701667 run pcap-recorder_any(filters='host 10.42.42.7')(pid=8373): Terminating (SIGTERM) 15:05:52.760733 run osmo-sgsn_10.42.42.7(pid=8374): Terminating (SIGTERM) 15:05:52.825028 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=8375): Terminating (SIGTERM) 15:05:52.868919 run osmo-msc_10.42.42.8(pid=8376): Terminating (SIGTERM) 15:05:52.913474 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8377): Terminating (SIGTERM) 15:05:52.958819 run osmo-mgw_10.42.42.3(pid=8378): Terminating (SIGTERM) 15:05:53.000764 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8379): Terminating (SIGTERM) 15:05:53.044385 run osmo-mgw_10.42.42.4(pid=8380): Terminating (SIGTERM) 15:05:53.086648 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=8381): Terminating (SIGTERM) 15:05:53.130898 run osmo-bsc_10.42.42.9(pid=8382): Terminating (SIGTERM) 15:05:53.173249 run osmo-bts-sysmo(pid=8391): Terminating (SIGTERM) 15:05:53.215431 run osmo-pcu(pid=8399): Terminating (SIGTERM) 15:05:53.258463 --- ParallelTerminationStrategy: PID 8347 died... 15:05:53.279413 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8347): Terminated: ok {rc=0} 15:05:53.326527 --- ParallelTerminationStrategy: PID 8348 died... 15:05:53.346541 run osmo-hlr_10.42.42.2(pid=8348): Terminated: ok {rc=0} 15:05:53.393727 --- ParallelTerminationStrategy: PID 8349 died... 15:05:53.413600 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8349): Terminated: ok {rc=0} 15:05:53.460798 --- ParallelTerminationStrategy: PID 8350 died... 15:05:53.480550 run osmo-stp_10.42.42.5(pid=8350): Terminated: ok {rc=0} 15:05:53.527344 --- ParallelTerminationStrategy: PID 8351 died... 15:05:53.547368 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8351): Terminated: ok {rc=0} 15:05:53.594633 --- ParallelTerminationStrategy: PID 8356 died... 15:05:53.614580 run osmo-ggsn_10.42.42.6(pid=8356): Terminated: ok {rc=0} 15:05:53.661491 --- ParallelTerminationStrategy: PID 8373 died... 15:05:53.681303 run pcap-recorder_any(filters='host 10.42.42.7')(pid=8373): Terminated: ok {rc=0} 15:05:53.731058 --- ParallelTerminationStrategy: PID 8375 died... 15:05:53.767862 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=8375): Terminated: ok {rc=0} 15:05:53.868808 --- ParallelTerminationStrategy: PID 8377 died... 15:05:53.913674 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8377): Terminated: ok {rc=0} 15:05:54.020279 --- ParallelTerminationStrategy: PID 8376 died... 15:05:54.046803 run osmo-msc_10.42.42.8(pid=8376): Terminated: ok {rc=0} 15:05:54.095281 --- ParallelTerminationStrategy: PID 8378 died... 15:05:54.115211 run osmo-mgw_10.42.42.3(pid=8378): Terminated {rc=15} 15:05:54.161145 --- ParallelTerminationStrategy: PID 8374 died... 15:05:54.180189 run osmo-sgsn_10.42.42.7(pid=8374): Terminated: ok {rc=0} 15:05:54.226676 --- ParallelTerminationStrategy: PID 8379 died... 15:05:54.245611 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8379): Terminated: ok {rc=0} 15:05:54.291325 --- ParallelTerminationStrategy: PID 8380 died... 15:05:54.310311 run osmo-mgw_10.42.42.4(pid=8380): Terminated {rc=15} 15:05:54.355875 --- ParallelTerminationStrategy: PID 8381 died... 15:05:54.374770 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=8381): Terminated: ok {rc=0} 15:05:54.420060 --- ParallelTerminationStrategy: PID 8382 died... 15:05:54.438660 run osmo-bsc_10.42.42.9(pid=8382): Terminated: ok {rc=0} 15:05:54.484485 --- ParallelTerminationStrategy: PID 8391 died... 15:05:54.503858 run osmo-bts-sysmo(pid=8391): Terminated: ok {rc=0} 15:05:54.548822 --- ParallelTerminationStrategy: PID 8399 died... 15:05:54.568255 run osmo-pcu(pid=8399): Terminated: ok {rc=0} 15:05:54.795028 bus /gobi_6: Setting Online False 15:05:57.856591 bus /gobi_6: Setting Powered False 15:06:12.942149 bus /gobi_6: Setting Powered False ---------------------------------------------- trial-2063 gprs:sysmo ping_idle_ping.py ---------------------------------------------- 15:06:14.083492 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:06:14.095046 tst gprs:sysmo: Using 1 x bts (candidates: 1) 15:06:14.131417 tst ping_idle_ping.py: using LAC 3942 15:06:14.253821 tst ping_idle_ping.py: using RAC 117 15:06:14.371888 tst ping_idle_ping.py: using CellId 3942 15:06:14.489803 tst ping_idle_ping.py: using BVCI 3943 15:06:14.599473 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:06:14.610003 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:06:14.620458 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:06:14.630840 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:06:14.641395 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:06:14.651751 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:06:14.662114 tst gprs:sysmo: Using 1 x ip_address (candidates: 9) 15:06:14.672457 tst gprs:sysmo: Using 1 x modem (candidates: 4) 15:06:14.785928 tst ping_idle_ping.py:24: start network... 15:06:15.019588 run osmo-hlr_10.42.42.2: Starting osmo-hlr 15:06:15.521270 run create_hlr_db(pid=8450): Launched 15:06:15.653016 bus /gobi_6: Setting Powered False 15:06:16.716996 run create_hlr_db(pid=8450): Terminated: ok {rc=0} 15:06:17.150986 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8452): Launched 15:06:17.454383 run osmo-hlr_10.42.42.2(pid=8453): Launched 15:06:17.569835 run osmo-stp_10.42.42.5: Starting osmo-stp 15:06:18.088043 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8454): Launched 15:06:18.392345 run osmo-stp_10.42.42.5(pid=8455): Launched 15:06:18.509302 run osmo-ggsn_10.42.42.6: Starting osmo-ggsn 15:06:19.029546 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8456): Launched 15:06:19.146253 run osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn 15:06:19.402187 run patchelf(pid=8457): Launched 15:06:20.572107 run patchelf(pid=8457): Terminated: ok {rc=0} 15:06:20.699730 run osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn 15:06:20.961083 run osmo-gsm-tester_setcap_net_admin.sh(pid=8459): Launched 15:06:22.151765 run osmo-gsm-tester_setcap_net_admin.sh(pid=8459): Terminated: ok {rc=0} 15:06:22.489519 run osmo-ggsn_10.42.42.6(pid=8462): Launched 15:06:22.616893 run osmo-sgsn_10.42.42.7: Starting osmo-sgsn 15:06:23.300780 run pcap-recorder_any(filters='host 10.42.42.7')(pid=8486): Launched 15:06:23.627152 run osmo-sgsn_10.42.42.7(pid=8487): Launched 15:06:23.752122 run osmo-msc_10.42.42.8: Starting osmo-msc 15:06:24.341795 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=8488): Launched 15:06:24.668082 run osmo-msc_10.42.42.8(pid=8489): Launched 15:06:27.850649 run osmo-mgw_10.42.42.3: Starting osmo-mgw 15:06:28.405462 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8490): Launched 15:06:28.731128 run osmo-mgw_10.42.42.3(pid=8491): Launched 15:06:28.854386 run osmo-mgw_10.42.42.4: Starting osmo-mgw 15:06:29.399207 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8492): Launched 15:06:29.723324 run osmo-mgw_10.42.42.4(pid=8493): Launched 15:06:29.846185 run osmo-bsc_10.42.42.9: Starting osmo-bsc 15:06:30.516775 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=8494): Launched 15:06:30.842387 run osmo-bsc_10.42.42.9(pid=8495): Launched 15:06:37.132695 run osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9 15:06:37.700102 run rm-remote-dir(pid=8496): Launched 15:06:39.892663 run rm-remote-dir(pid=8496): Terminated: ok {rc=0} 15:06:40.250167 run mk-remote-dir(pid=8497): Launched 15:06:41.442322 run mk-remote-dir(pid=8497): Terminated: ok {rc=0} 15:06:41.747032 run scp-inst-to-remote(pid=8498): Launched 15:06:53.952674 run scp-inst-to-remote(pid=8498): Terminated: ok {rc=0} 15:06:54.282457 run mk-remote-dir(pid=8500): Launched 15:06:55.458273 run mk-remote-dir(pid=8500): Terminated: ok {rc=0} 15:06:55.743139 run scp-cfg-to-remote(pid=8501): Launched 15:06:56.919739 run scp-cfg-to-remote(pid=8501): Terminated: ok {rc=0} 15:06:57.246782 run reload-dsp-firmware(pid=8503): Launched 15:06:59.423760 run reload-dsp-firmware(pid=8503): Terminated: ok {rc=0} 15:06:59.744027 run osmo-bts-sysmo(pid=8504): Launched 15:07:03.970234 tst ping_idle_ping.py:36: Waiting for bts to be ready... 15:07:05.638962 run rm-remote-dir(pid=8505): Launched 15:07:07.815782 run rm-remote-dir(pid=8505): Terminated: ok {rc=0} 15:07:08.157091 run mk-remote-dir(pid=8506): Launched 15:07:10.341248 run mk-remote-dir(pid=8506): Terminated: ok {rc=0} 15:07:10.635065 run scp-inst-to-bts(pid=8507): Launched 15:07:23.839719 run scp-inst-to-bts(pid=8507): Terminated: ok {rc=0} 15:07:24.182864 run mk-remote-run-dir(pid=8509): Launched 15:07:26.367515 run mk-remote-run-dir(pid=8509): Terminated: ok {rc=0} 15:07:26.647349 run scp-cfg-to-sysmobts(pid=8510): Launched 15:07:28.819262 run scp-cfg-to-sysmobts(pid=8510): Terminated: ok {rc=0} 15:07:29.140744 run osmo-pcu(pid=8512): Launched 15:07:29.269211 tst ping_idle_ping.py: using MSISDN 3946 15:07:29.443172 bus /gobi_6: Setting Powered True 15:07:32.931102 run osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3946', subscriber_id=1} 15:07:33.217415 bus /gobi_6: Setting Online False 15:07:34.415691 bus /gobi_6: Setting Powered False 15:07:41.174281 bus /gobi_6: Setting Powered True 15:07:42.549681 bus /gobi_6: Setting Online True 15:07:45.006034 tst /gobi_6: Connect to ('901', '70') 15:07:45.278559 tst /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G 1 [Jul 5 2017 10:49:17]', Serial: '868323024273648' 15:07:45.396281 tst ping_idle_ping.py:47: waiting for modems to attach... 15:07:52.190182 tst /gobi_6: Already registered with network ('901', '70') 15:08:03.867537 tst ping_idle_ping.py:51: waiting for modems to attach to data services... 15:08:09.892815 tst /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'} 15:08:10.003328 tst ping_idle_ping.py:56: Setting up data plane for "'/gobi_6/context2'" 15:08:10.418926 run move_netns(pid=8516): Launched 15:08:11.604593 run move_netns(pid=8516): Terminated: ok {rc=0} 15:08:11.934478 run ifup(pid=8519): Launched 15:08:13.125651 run ifup(pid=8519): Terminated: ok {rc=0} 15:08:13.479309 run dhcp(pid=8522): Launched 15:08:14.663806 run dhcp(pid=8522): Terminated: ok {rc=0} 15:08:14.792851 tst ping_idle_ping.py:60: [1] Running 10 ping requests for "'/gobi_6/context2'" 15:08:15.193811 run ping1(pid=8538): Launched 15:08:25.382192 run ping1(pid=8538): Terminated: ok {rc=0} 15:08:25.509881 tst ping_idle_ping.py:64: (launched: 2022-02-24_15:08:15.132810) 8539: waiting for 8540 PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data. 64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=539 ms 64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=499 ms 64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=456 ms 64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=438 ms 64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=457 ms 64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=453 ms 64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=435 ms 64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=431 ms 64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=431 ms 64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=408 ms --- 10.42.42.6 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 20ms rtt min/avg/max/mdev = 408.119/454.703/539.042/36.087 ms child exited with 0 15:08:25.714098 tst ping_idle_ping.py:68: Sleeping for 60 seconds 15:09:25.921000 tst ping_idle_ping.py:73: [2] Running 10 ping requests for "'/gobi_6/context2'" 15:09:26.354477 run ping2(pid=8543): Launched 15:09:36.550631 run ping2(pid=8543): Terminated: ok {rc=0} 15:09:36.678212 tst ping_idle_ping.py:77: (launched: 2022-02-24_15:09:26.289423) 8544: waiting for 8546 PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data. 64 bytes from 10.42.42.6: icmp_seq=1 ttl=64 time=468 ms 64 bytes from 10.42.42.6: icmp_seq=2 ttl=64 time=409 ms 64 bytes from 10.42.42.6: icmp_seq=3 ttl=64 time=367 ms 64 bytes from 10.42.42.6: icmp_seq=4 ttl=64 time=363 ms 64 bytes from 10.42.42.6: icmp_seq=5 ttl=64 time=723 ms 64 bytes from 10.42.42.6: icmp_seq=6 ttl=64 time=425 ms 64 bytes from 10.42.42.6: icmp_seq=7 ttl=64 time=420 ms 64 bytes from 10.42.42.6: icmp_seq=8 ttl=64 time=402 ms 64 bytes from 10.42.42.6: icmp_seq=9 ttl=64 time=420 ms 64 bytes from 10.42.42.6: icmp_seq=10 ttl=64 time=416 ms --- 10.42.42.6 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 20ms rtt min/avg/max/mdev = 363.394/441.378/723.056/98.020 ms child exited with 0 15:09:38.419093 tst /gobi_6: context deactivated {path='/gobi_6/context2'} 15:09:38.539213 tst ping_idle_ping.py: Test passed (204.5 sec) 15:09:38.608102 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8452): Terminating (SIGTERM) 15:09:38.651073 run osmo-hlr_10.42.42.2(pid=8453): Terminating (SIGTERM) 15:09:38.694308 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8454): Terminating (SIGTERM) 15:09:38.739104 run osmo-stp_10.42.42.5(pid=8455): Terminating (SIGTERM) 15:09:38.785007 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8456): Terminating (SIGTERM) 15:09:38.858937 run osmo-ggsn_10.42.42.6(pid=8462): Terminating (SIGTERM) 15:09:38.914677 run pcap-recorder_any(filters='host 10.42.42.7')(pid=8486): Terminating (SIGTERM) 15:09:38.973298 run osmo-sgsn_10.42.42.7(pid=8487): Terminating (SIGTERM) 15:09:39.025986 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=8488): Terminating (SIGTERM) 15:09:39.069337 run osmo-msc_10.42.42.8(pid=8489): Terminating (SIGTERM) 15:09:39.112378 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8490): Terminating (SIGTERM) 15:09:39.156053 run osmo-mgw_10.42.42.3(pid=8491): Terminating (SIGTERM) 15:09:39.199626 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8492): Terminating (SIGTERM) 15:09:39.243499 run osmo-mgw_10.42.42.4(pid=8493): Terminating (SIGTERM) 15:09:39.286129 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=8494): Terminating (SIGTERM) 15:09:39.329702 run osmo-bsc_10.42.42.9(pid=8495): Terminating (SIGTERM) 15:09:39.371814 run osmo-bts-sysmo(pid=8504): Terminating (SIGTERM) 15:09:39.413677 run osmo-pcu(pid=8512): Terminating (SIGTERM) 15:09:39.456699 --- ParallelTerminationStrategy: PID 8452 died... 15:09:39.477369 run pcap-recorder_any(filters='host 10.42.42.2')(pid=8452): Terminated: ok {rc=0} 15:09:39.525332 --- ParallelTerminationStrategy: PID 8453 died... 15:09:39.545787 run osmo-hlr_10.42.42.2(pid=8453): Terminated: ok {rc=0} 15:09:39.593388 --- ParallelTerminationStrategy: PID 8454 died... 15:09:39.613857 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=8454): Terminated: ok {rc=0} 15:09:39.661524 --- ParallelTerminationStrategy: PID 8455 died... 15:09:39.681800 run osmo-stp_10.42.42.5(pid=8455): Terminated: ok {rc=0} 15:09:39.729836 --- ParallelTerminationStrategy: PID 8456 died... 15:09:39.749800 run pcap-recorder_any(filters='host 10.42.42.6')(pid=8456): Terminated: ok {rc=0} 15:09:39.796531 --- ParallelTerminationStrategy: PID 8462 died... 15:09:39.816301 run osmo-ggsn_10.42.42.6(pid=8462): Terminated: ok {rc=0} 15:09:39.862629 --- ParallelTerminationStrategy: PID 8486 died... 15:09:39.883394 run pcap-recorder_any(filters='host 10.42.42.7')(pid=8486): Terminated: ok {rc=0} 15:09:39.931259 --- ParallelTerminationStrategy: PID 8488 died... 15:09:39.951935 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=8488): Terminated: ok {rc=0} 15:09:39.999277 --- ParallelTerminationStrategy: PID 8490 died... 15:09:40.019757 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=8490): Terminated: ok {rc=0} 15:09:40.075114 --- ParallelTerminationStrategy: PID 8491 died... 15:09:40.095342 run osmo-mgw_10.42.42.3(pid=8491): Terminated {rc=15} 15:09:40.143753 --- ParallelTerminationStrategy: PID 8489 died... 15:09:40.164160 run osmo-msc_10.42.42.8(pid=8489): Terminated: ok {rc=0} 15:09:40.211804 --- ParallelTerminationStrategy: PID 8492 died... 15:09:40.232795 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=8492): Terminated: ok {rc=0} 15:09:40.280467 --- ParallelTerminationStrategy: PID 8493 died... 15:09:40.300923 run osmo-mgw_10.42.42.4(pid=8493): Terminated {rc=15} 15:09:40.348375 --- ParallelTerminationStrategy: PID 8494 died... 15:09:40.369306 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=8494): Terminated: ok {rc=0} 15:09:40.417113 --- ParallelTerminationStrategy: PID 8495 died... 15:09:40.437629 run osmo-bsc_10.42.42.9(pid=8495): Terminated: ok {rc=0} 15:09:40.485420 --- ParallelTerminationStrategy: PID 8504 died... 15:09:40.506213 run osmo-bts-sysmo(pid=8504): Terminated: ok {rc=0} 15:09:40.555821 --- ParallelTerminationStrategy: PID 8487 died... 15:09:40.574939 run osmo-sgsn_10.42.42.7(pid=8487): Terminated: ok {rc=0} 15:09:40.619578 --- ParallelTerminationStrategy: PID 8512 died... 15:09:40.638332 run osmo-pcu(pid=8512): Terminated: ok {rc=0} 15:09:40.858251 bus /gobi_6: Setting Online False 15:09:43.908790 bus /gobi_6: Setting Powered False 15:09:59.938864 bus /gobi_6: Setting Powered False --------------------------------------------------------------------- trial-2063 gprs:sysmo FAIL --------------------------------------------------------------------- 15:10:01.280392 tst trial-2063: Storing JUnit report in /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2063/run.2022-02-24_14-45-48/trial-2063.xml -------------------------------------------------------------------------------------------- trial-2063 FAIL -------------------------------------------------------------------------------------------- 15:10:01.563066 tst trial-2063: trial-2063: FAIL, 2 suites failed, 1 suites passed FAIL: nitb_sms:sysmo (fail: 1) FAIL: mo_mt_sms.py (4.8 sec) Error: osmo-nitb_10.42.42.2(pid=7176): Process ended prematurely: osmo-nitb_10.42.42.2(pid=7176) [trial-2063↪nitb_sms:sysmo↪mo_mt_sms.py:11↪mo_mt_sms.py↪osmo-nitb_10.42.42.2↪osmo-nitb_10.42.42.2(pid=7176)] FAIL: gprs:sysmo (fail: 3, pass: 2) FAIL: cs_paging_gprs_active.py (225.7 sec) KeyError: 'sum_sent' FAIL: iperf3.py (209.9 sec) KeyError: 'sum_sent' FAIL: iperf3m4.py (288.7 sec) KeyError: 'sum_sent' pass: ping.py (163.2 sec) pass: ping_idle_ping.py (204.5 sec) PASS: sms:sysmo (pass: 1) pass: mo_mt_sms.py (114.6 sec) + rm -rf trial-2063/inst + cd trial-2063 + command -v journalctl + [ ! -z /bin/journalctl ] + readlink last_run + journalctl -u ofono -o short-precise --since 2022-02-24 14:45:46 + readlink last_run + tar czf /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2063-run.tgz run.2022-02-24_14-45-48 + tar czf /home/jenkins/workspace/osmo-gsm-tester_gerrit/trial-2063-bin.tgz checksums.md5 open5gs.build-8.tgz osmo-bsc.build-15317.tgz osmo-bts-oc2g.build-4479.tgz osmo-bts-sysmo.build-6983.tgz osmo-bts.build-6453.tgz osmo-ggsn.build-5693.tgz osmo-hlr.build-7389.tgz osmo-hnbgw.build-340.tgz osmo-mgw.build-7582.tgz osmo-msc.build-25551.tgz osmo-nitb.build-11093.tgz osmo-pcu-oc2g.build-3226.tgz osmo-pcu-sysmo.build-4507.tgz osmo-pcu.build-4468.tgz osmo-sgsn.build-25820.tgz osmo-stp.build-7910.tgz osmo-trx.build-3723.tgz osmocom-bb.build-3951.tgz srslte.build-19.tgz + exit 1 Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results [Checks API] No suitable checks publisher found. Finished: FAILURE