Changes

osmo-sgsn: Fix template cfg to work with NS2 implementation

See following HOW-TO to update existing configs:
https://osmocom.org/projects/libosmocore/wiki/Network_service_(NS)

Change-Id: I391c977ca20ef185db6b1f17912608acaa13e87f

Fix bts.ready_for_pcu() implementation for sysmo and oc2g

osmo-bts.git ae09c8acb4aa93284cdb44f8bbdc14533dc4fa52 modified code to
avoid calling pcu_tx_info_ind() if pcu socket was not connected. Since
osmo-gsm-tester relied on log message sent from within that funtion to
find out whether BTS was able to handle PCUIF connections, that log line
is not printed anymore and hence the function ready_for_pcu() fails to
ever return true.
As a result, gprs tests for sysmo and oc2g bts types always fail with a
timeout.

Other BTS types are not affected (such as osmo-bts-trx) because in there
we simply check whether the osmo-bts process is running.

Change-Id: Ibde4b8b27b8300b4b9197c0f8d82a6ccbf3b0522

contrib: Add build script for open5gs

Change-Id: I0ed43dadbfe08811dd72064ee7332d0872882dfd

enb,epc,ms: refactor KPI API

we previously mixed component specific and component agnostic APIs
(stdout vs. log file for example) for setting and retrieving KPI.

This patch propose to use a single abstract get_kpis() method for
all components that can be enriched with component-specific
stuff as desired.

In the case of srsLTE blocks, the main implementation will
remain in srslte_common() and is shared among srsENB/srsUE/srsEPC.

The KPI analyzer in srslte_common() extract and also manages
all three KPI sources (log, csv and stdout) independently.

In addition to the get_kpis() method that always returns a flat
dictionary, it also exposes get_kpi_tree() that return
a dict of KPI dicts that will be used for the Junit.xml generation.

Change-Id: I4bacc6b8a0cb92a581edfb947100b57022265265

enb.py: increase zmq trx timeout to 4000ms

with many parallel jobs running on one machine we should
relax the timeout a bit to avoid false positives.

Change-Id: Ib32a07559bd5a4b0165f6d3431c5ac57cc050f44

{enb,ms}-label.conf: raneme and add modifier to select eNB/MS based on label

Change-Id: I26ea160b65b9dd5ef3e5ca1100d4541a0bbcbb27

srsenb_drb.conf.tmpl: set PDCP discard timer to 150ms for QCI9

srsENB now supports PDCP discard so we set it to a sane value.
Also enable status reporting for UL.

Change-Id: Ib6ad0ff57b95a1f220f87c1ede0f6eacc5fb21ff

amarisoft_drb.cfg.tmpl: add additional QCI10 with infinit PDCP discard timer

also set QCI9 to 150ms to match default srsENB config

Change-Id: I60cc1390480142bd5bb516934c9f30b6122496d5

srsenb_drb.conf.tmpl: add QCI10 with infinit PDCP discard timer

copy of QCI9 params but with infinit PDCP discard to avoid SDU drops
at PDCP

Change-Id: I64a74d09cc1a55ba21a7be7b5cc203e56bba3b10

ms_srs: add config scheme to select NAS log level

we've provided only all_log_level so far but sometimes it's needed
to select the level per layer. This patch adds the ability
to do so for the NAS layer in the UE.

Change-Id: Iab2bce65e8af81f6d344849c97952e6441cb2846

srsue.conf.tmpl: Adjust ue.config template for new pcap path

Change-Id: I4af6abaa3973652a12f571d88bca59cd443704f3

srsenb_sib.conf.tmpl: reduce the default size of the PUCCH

optimized PUCCH config

Change-Id: I45ae7c33f9d1df20df37c0fe6c3d73aa85ae1b31

jenkins-build-amarisoft: copy ltesim_server binary into epc trial tgz

the ltesim_server is part of the UE package but needs to be executed
on the EPC run node. Therefore its handy to have the binary in the epc
trial tgz.

Change-Id: I21ac57d1c1547c1cf4d2394c8593f58ff2c78697

osmo-gsm-tester_androidue_conn_chk.sh: improve error handling

fix check of $serial and check passed arguments.
exit when adb isn't installed on host

Change-Id: I6190a840dbc86838457da96d188afc17f83ab15f

ms_android: set num_carriers to 1 by default

throughput tests require the UE object to have the num_carriers
member. For Android UEs, set it to 1 by default.

Change-Id: I297d24bb578a7db0eb7629be4f2cc0d54624467d

epc-label@.conf: add scenario to select runnode based on label

this allows to select the resource/machine the EPC runs on based on the label

Change-Id: I710c67ad5a11bd33071acd779e19835ae49ce4a3

run_node: add label to RunNode class

this allows specifying a label on each run node that can be used
to filter/select specific nodes.

Change-Id: Ib6a977007f56d6ff9ff029aaf049a75fb5d5e1dd

mod-enb-ncells@.conf: add scenario file to select number of cells

Change-Id: Ib599de56b0b51a125555895014d78d3d649c01a0

ms_srs: turn tx/rx_gain into INT and make them optional

when they are greater than 0 they are written as config paramter.
if they are -1 they are disbaled and automatic gain calibration is
used.

Change-Id: I473ff3ae679784178574d2f76b612dbf77180490

AndroidUE: Fix connection timeout

Change-Id: Iaacf348e2e018da5b67ada33079018c1859d3b63

iperf3_bidir: add extra test for bi-directional iperf test

Change-Id: Ibb62f53dbb15bc4788c92ebb2dee70e31dd851de

ms_srs: add force_ul_amplitude config

this allows to overwrite the PHY export option phy.force_ul_amplitude

Change-Id: I255f027433944c1739c89757609f0dcc3b8aa7cb

enb.py: increase ZMQ trx timeout to 8s

increase value further to avoid failed tests due to load on system

Change-Id: I77b7864e7ef60faf0262dc24325915d73dd2d5a3

srslte_common: fix condition to check if KPI have already extracted

Change-Id: I6fdcbfadfdcfe131096d0345d2c21c87f8f7b1b6

sysmo/defaults.conf: Fix attribute name

According to schema, the name is cipher_list.
Same goes for "integrity".

Fixes: a81801558e7bb6ef0f0a32ab078d302c91e940a8
Change-Id: Ib7b48ce207b96ea5f5b59db08320563be40edba1

sysmocom/default.conf: Add missing default value for force_ul_amplitude

Change-Id: I04218dca21f27e32d7c85a74a3e9bbf63b499357

contrib: open5gs: Install more binaries to have an EPC

Change-Id: I1ff143b4f876e28ad9a1bc6a4a4cad5dacef8e18

Initial Open5GS support

Change-Id: Iff8b88dc22d8f156572839abb48b0c1377c55e33

open5gs: Avoid use of non-standard ports

Despite open5gs allowing to change the GTP ports in the config file, in
reality changing those values to something else than the standard prot
will fail. Hence, we must use the standard port. As a result, we must
use different IP addresses in each process to avoid ip+port collisions.
Let's use some loopback addresses which shouldn't require extra
configuration on the host, and still only requiring 1 run_node as per
existing EPCs, with the limitation that only 1 open5gs EPC instance can
be run at one in a given run_node.

Related: https://github.com/open5gs/open5gs/issues/897
Change-Id: Id3062c6ad9d6de4c6066547e1e46edad5da285c1

amarisoft_mme: use/expose count paramter from MS config for MME

the UEs count param, if specified, is mapped into the config
values of the EPC so it can be used for template rendering.

Since we only have one count parameter, its added to the first
subscriber. 1 is the default value and its used when no count
param is specified.

this allows to run the Amarisoft UE simulator with the Amarisoft EPC
without having to create subscriber entries for each simulated UE.

Change-Id: I29885791f716c204d0b6f18ba134885bae853b6f

epc_amarisoft: launch ltesim server when starting

this allows to use the lteue to generate DL traffic. the
server listens on the default spgw address.

Change-Id: Iad081e57e2fd9d3cbc3e59e4f729a4721ad778c9

ms_srs: add resource scheme to configure fixed DL and UL freq

this config option overwrites the EARFCN config that is typically
used to tell the UE the bands to scan for cells. When custom
frequencies are used, this option allows to set them explicitly.

Change-Id: Ice070ea6755e273d916db2dc941068d33bbe206a

enb_srs: add support for eNB traces

this patch adds support for srsENB event tracing. it provides
a cfg scenario to enable it.

Change-Id: Icb2342acac5db87a0889cbf1b6c7bb1ad26c16fa

ms_amarisoft: extend AmariUE to support custom frequency bands

this patch adds generic support for custom frequency bands to the ms class.
it then extends the config template for AmariUE to use them.
we set the default values for EARFCNs for DL/UL to the old values.

Change-Id: I7b2250e13fc1161be1a1d73a4348f07b31c0f724

ping.py: add stricter pass/fail check

the ping test should only pass if not a single ping is lost

Change-Id: If089f77df1ffe991f26f1546558d51aac242df83

schema: add 5G as UE feature

Change-Id: I851e95fb0b763f2a838b161be4938715770eaa83

jenkins-build-amarisoft: adjusting lib name to srsran and update versions

fix rename of srslte to srsran libs and also update Amarisoft
versions.

Change-Id: I2174aec3c7ded8c966877a87f4cc6a39ce1325c2

amarisoft_ltemme.cfg.tmpl: add gbr to the template

newer version of the amarisoft mme requires the GBR to be set

Change-Id: Ib0b046ea9fcc26b6bbe03461ca6308acc35175d1

rfemu_srsenb_stdin: catch exception in rfemu and log error

not handling the exception causes all following tests to
fail.

Change-Id: I496313ef8412c8cb18a3c2cb32c52a3b5a672853

srsenb_drb.conf.tmpl: decrease pollPDU for QCI9

due to internal PDU buffering between PDCP/RLC the poll frequency
needs to be increase to get quicker feedback.

Change-Id: I4114fb36e675b3f91c471558f716276ab24f1db5

enb: add rx_ant param

Change-Id: Iada98fce3608808914562d9f7e2af2c230e03cae

4g: add overlay template path for all suites

this allows, without changing the test case, to use a custom
template, which is useful for development. By default, nothing changes
and the OGT template is used

Change-Id: Ifc43ac41b16813116f2559da5223a6fecc186125

ping: add a short delay after UE is attached before starting ping

this is mainly to avoid ping losses in 5G NSA during NR attach.
No impact on 4G though.

Change-Id: Ic507dc4a24e8576b0ec3b301ee8e340ed553b300

ms_srs: extend prach_sent counter with NR version

make LTE version more explicit and add another counter for NR

Change-Id: I8192dc3b4dae27417c93b68d33a686bca4783f4d

srsue.conf.tmpl: update args for 21.04 changes to include NSA

this also removes the entire help text around arguments (that
were very outdated already) and only leaves the actual values with changes.

Change-Id: Icb9e8e7b1c68cf024db3a7273af791f017c32003

enb_amarisoft: add "sdr" as rf_dev_type and rx_ant parameter

this allows to use the Amarisoft PCIe cards with OGT

Change-Id: Ia373a78f59538d6dfca390431f85a7af26c9d914

ms_srs: add short_sn as config option for srsUE

this allows to change the announced support for PDCP
short sequence number lengths

Change-Id: Ibcf72dbb424ae3b6a87b60febb4dadf716407612

amarisoft_rf_driver.cfg.tmpl: only set trx offset for B210

all other RF devices do not required any offset

Change-Id: Id4c5f0bede5b09d59237cdd6a959a3663f1a1b37

srsue.conf.tmpl: set srate param with the X310

Change-Id: Icb04ed51ace97d4467558639aeb3f556dbe8678a

srsenb.conf.tmpl: use LTE sample rates when Lime is found in rf args

the lime doesn't work well with the shorter sample rates the eNB
uses by default.

Change-Id: I2d3dbdf5ff62980854ef67b20469f072ce31612e

amarisoft_enb: add NR support

this patch adds the ability to configure NR cells with
Amarisoft eNB. It adds the new DRB-NR template and updates
the normal enb.cfg template to allow using it as LTE only
or with NR cells (5G NSA).

Change-Id: Ia27bbc6db5920ce14bacabe8043601aa2adaa5fe

ms_srs: add 5G NSA support

extend the MS to allow for NR carriers

Change-Id: I7de9f867e7e32d0ab4f74eb7661928175f6c1a16

amarisoft_ltemme.cfg.tmpl: enable ENDC support in config template

patch also remove the UE forwarding option which isn't used anymore in the MME.

Change-Id: Ia1628e6674b124806e3cd59c5b190d27f6ba4403

amarisoftue: solve conflict of sample rate param in rf_driver

move sample_rate setting into rf_driver template that is shared
between enb and ue

Change-Id: Icc8042c3eec2a1e8bfc6823c06430c1cb18355b7

srsue.conf.tmpl: tidy up config file

no functional changes, just remove whitespaces and newlines

Change-Id: Ib58d3591cc51c11c9e496f6510ebffb91ee71345

epc_open5gs: remove _epc from name

there is no open5gs_enb so having the epc in the name is redundent, also
it safes typing and allows longer scenario configs without exceeding
filename limit.

Change-Id: Ib10bc6031ecc8de88a9c718b136d2641ce46e11d

open5gs: fix stopping of open5gs components after test finishes

use RemoteProcessSafeExit() for launching all open5gs components
to make sure they are stopped at the end as well.

Change-Id: Ia1439577d60f9694da70c4e42c621100c8ef295e

androidue: fix execute permissions for conn_chk and diag parser scripts

Change-Id: I755b65188a485a3c2f30488ab598f7a50cf80dc5

open5gs: fix name

Change-Id: I7b0325718f9572b03e282d17ddf111a9493e2ee7

open5gs: lower default log level to info

Change-Id: I15aaac5838bdb9d271a850b959a3a8fb4a56a313

srsenb: enable S1AP pcaps as well when PCAPs are configured

this stores the S1AP pcaps in an own file but uses
the same switch to enable them.

Change-Id: If24048d61d8dff0b9b31aff9ed6995342830ebf8

open5gs_mme: set short network name

Change-Id: Ifd7a78e35fb47cc5a96452fe93d8ab305359d7bd

sysmocom/defaults.conf: Fix recently changed node name

This node was changed recently but this file was not updated.

Change-Id: Id5df83a356579340259d9d02ebc621f298151599

open5gs: Pre-configure tun device for UPF

open5gs requires the tun device to be pre-created and pre-configured
before using it. A ticket was opened in order to allow upfd itself
configuring it in [1]

[1] https://github.com/open5gs/open5gs/issues/899

Change-Id: I3b1857f09017cacee1c86f41dae1463c11b50490

open5gs: Fix GTP-U addr binding

Change-Id: I8c91a9c4a07f4229cb4ae9af61d94e720d81a961

iperf: adds the option to define the packet length for udp.

Added the feature to configure the packet length in udp iperf. Added a modifier to select the length value.
The default value is configured to 1460.

Change-Id: Ic886faa82b7cb25f2bce685eb80910623e4ff686

4g-iperf3-ul: modified the test to parameterize the number of ue in the test.

Parameterize the iperf3 uplink test for the 4g test suite to execute a
given number of UEs. By default, the number of ue is 1.

Change-Id: I4b006df04bd1af6c117bcb25e6a6b1609ac732fb

4g-iperf3-dl: modified the test to parameterize the number of ue in the test.

Parameterize the iperf3 downlink test for the 4g test suite to execute a
given number of UEs. By default, the number of ue is 1.

Change-Id: Ifafea196289d1ce0d35ae66c6564266ba123b27a

4g-iperf3-bidir: modified the test to parameterize the number of ue in the test.

Parameterize the iperf3 bidirectional test for the 4g test suite to execute a
given number of UEs. By default, the number of ue is 1.

Change-Id: I34fb436d581eda12388819b5dee9a3bafd419a7f

srs-ms: allows to run given scripts before the execution of the UE.

Executes the scripts given by the 'prerun_scripts' variable before launching the UE.

Change-Id: I718e686b5844f2f07eda820914678052f1008182

srs-enb: runs the given list of scripts before and after launching the eNodeB.

Allows to run given scripts before and after the execution of the SRS
eNodeB. The scripts are retrieved as a list from the resource.conf file.

Change-Id: I5017caae933809b89b32ea132be8552d6554e20e

srs-enb: adds fapi to the rf supported devices.

Added support to 'fapi' rf device for the SRS eNodeB.

Change-Id: I0ac2a7db7c0533dfbc90f2526a7b823110d355c0

srs-enb: sets the remote directory to the given one.

Sets the remote directory for fapi to the given in the variable 'remote_dir' in the reource schema.

Change-Id: I77a56835a4054b86d1d3f7da823a013b3370f767

srs-enb: adds support to the malloc interceptor.

Adds support to the malloc interceptor for the SRS eNodeB. This interceptor will generate a log file that lists the mallocs, reallocs and frees produced by the srsenb.

Change-Id: I0078020468f58bdd70b0b5de879eb58192f947a6

jenkins-build-common: adds support to the new variables for patchelf.

2 variables are added in order to increase the flexibility of the add_rpath function.
patchelf_rpath_extra_args: adds more arguments to the rpath patch. For example "--force-rpath"
patchelf_rpath_dir: path to the libraries. This path is appended to the default '$ORIGIN/../lib/'

This commit add the jenkins-build-srslte-bbu.sh script, that builds the given repository filling the information for the 2 newly added variables.

Change-Id: I9f510112c63e7598add8c00f8573e34cac1b6064

util: fix the functions "create_netns" and "setcap_netsys_admin".

This commit fixes 2 functions from util.py:
- setcap_netsys_admin: removed "self" from the arguments.
- create_netns: pass "run_dir" to Process and convert the arguments to
a list as the other methods in the file.

Change-Id: Ida4c6d6500be92b8d1f96e4e7a870e63bd329be9

ttcn3: Use osmocom registry to pull ttcn3-bts-test

Since recently, the ttcn3-bts-test image is available in the osmocom
registry. Let's use that one instead of the sysmocom one.

The imaqge is built and pushed to the registry regularly by osmocom
jenkins job registry-rebuild-upload-ttcn3-bts-tests (see
osmo-ci.git/jobs/registry-rebuild-upload-titan.yml).

Related: SYS#5765
Change-Id: I6de9719830290aa7f3ca80a6080ed37f107bdc16

contrib: Convert trial osmo-iuh->osmo-hnbgw

The real target was not osmo-iuh itself but osmo-hnbgw. Since osmo-hnbgw
is inits own repository now, let's rename it.

Change-Id: I03c0988db5c5d72bcac71b4515e40a099600ce7c

enb: add scenario file for inter-eNB HO with two cells

Change-Id: I356a81814a2d7714a1a5252aa357257901fa6227

cell_modifiers: refactor HO cell configs

* rename intrafreq vs intraenb
* add missing mandatory params

Change-Id: Icc1fe5c1711ecbed2f1b8b1e6a0b51c2d28ec651

enb: add QAM64 support for UL

* add new UE feature
* enable in srsue.conf.templ
* add new table for maximum rates
* add config scenario to enable SIB option for QAM64

Change-Id: I6ac2c9989a761e91b93d76c2507f55f0140b202d

add scenario to modify bandwith of 2 eNB objects

required for Inter-eNB HO configs, for example.

Change-Id: Iaf053941ff31c4ba94edf67fc54c0f1c1441577d

ms_amarisoft: fix various regressions

this patch fixes some regressions in the Amarisoft UE class,
the config template, etc. that have been undetected bc we never
executed tests with it.

Change-Id: I397e675a4018acf3372a3b7e29fd864703b2b919

jenkins-build-amarisoft.sh: set symlink to srslte_rf lib

Change-Id: Ifc8e5ba0dd59fb92cdf22f2a18acea904e2153f7

gnuradio_zmq_broker_remote: fix eNB port mapping for carrier aggregation

the mapping of ports didn't consider when a eNB had two ports for
different EARFCNs

Change-Id: I5548f67a3b37b03482e4b6172828fcd891e704a5

Process: Add remote_port as new parameter

remote_port defines a custom/additional port for
connections over ssh. It may be used in case several
ssh instances share one IP address.

Change-Id: I2c93fd2ea1c10c333d00eafd3c1066c35796e398

prod: after hardware move, adjust modem pci for sierra_1st and sierra_2nd (ONLY TWO)

Doing only the first two modems so far because I need them for handover
runs. The other modems are still todo!

Change-Id: Ibd71acfc76c01ffd105abe5effc1d246b1e65f85

contrib: Disable libmnl dependency cross-compiling osmo-pcu

Only newest OE SDKs have libmnl added, but we don't need this feature
anyway, so let's disable it in order to get builds running fine again.

Change-Id: Id3b30d0b183cd21fd6df89fdda5581ab58c7a9a5

contrib: Disable libmnl dependency cross-compiling osmo-bts

Only newest OE SDKs have libmnl added, but we don't need this feature
anyway, so let's disable it in order to get builds running fine again.

Change-Id: Ief9cbca6e10b1ed6acf6e079f660fc3f7c863c7b

cfg templates: log level and source basename last, log no category-hex

Makes for saner log output in stderr files.

Change-Id: I2755beffd18d6779367c04e4aaaed119d66eb3be

add osmo_vty.py

To trigger manual handovers, I need a VTY interface. The non-trivial
parts of this are copied from osmo-python-tests osmo_interact_vty.py.

Will be used in the upcoming handover_2G test suite in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: I7c17b143b7c690b8c4105ee7c6272670046fa91d

add bsc.vty: always keep a BSC VTY connection ready

The BSC's VTY port will be used to trigger manual handover, and to
retrieve a list of active lchans from the BSC, in the upcoming
handover_2G test suite, I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: I06652db04fc9e48748f3c2196334f5352e9cc48a

bsc: add active lchan matching via vty

Will be used in upcoming handover_2G test suite in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8, which needs to verify that a
handover actually ended up on the expected lchan.

Change-Id: I03df8f3ae2ee47930eee311c7ce104c36dbb3154

bts: allow configuring ARFCN and BSIC: add to schema

resources.conf is already allowed to set these, but in order to let
scenarios and suites modify these values, they need to be in the schema.

Change-Id: I8e0583e9208d563c66b0bdc9f25b1b74f92403c2

log passed tests

In the end of a test suite, do not omit the passed tests. For example,
running handover against N BTS combinations, it was hard to summarize
which BTS models actually succeeded, with only the failures listed.

Besides the "FAIL" listings, now print something like this in the end:

PASS: handover:sysmo+secondbts-trx-b200 (pass: 1)
     pass: handover.py (198.8 sec)
PASS: handover:sysmo+secondbts-trx-umtrx (pass: 1)
     pass: handover.py (192.7 sec)
PASS: handover:trx-b200+secondbts-trx-umtrx (pass: 1)
     pass: handover.py (193.1 sec)

Change-Id: Ib85a5b90e267c2ed2f844691187ecadc8939b1bb

log reserved resources

On non-debug log level, show something like this at the beginning of
each suite:

03:45:49.439720 tst handover:sysmo+secondbts-trx-b200: RESERVED RESOURCES for handover:
bts
   sysmoBTS 1002
   Ettus B200
ip_address
   10.42.42.2
   10.42.42.3
   10.42.42.4
   10.42.42.5
   10.42.42.6
   10.42.42.7
modem
   sierra_1st
   sierra_2nd

Change-Id: Ic23556eafee654c93d13c5ef405028da09bd51d7

configure fixed ARFCN per bts

Remove ARFCNs as a concept from resource pool, assign a fixed ARFCN to
each BTS and TRX in the resource pools.

Using ARFCNs on specific bands as resources was an idea that is hard to
implement, because specific BTS dictate selection of bands which
influences which ARFCNs can be picked. That means reserving ARFCN
resources is only possible after reserving specific BTS resources, but
the tester is currently not capable of such two-stage resolution.

Writing handover tests, I got the problem that both BTS in a scenario
attempt to use the same ARFCN.

The by far easiest solution is to assign one fixed ARFCN to each BTS and
TRX. If ever needed, a scenario modifier can still configure different
ARFCNs.

(Due to uncertainty about OC2G operation stability, I prefer to leave
OC2G on ARFCN 50, as it happened to end up being configured before this
patch.)

Change-Id: I0a6c60544226f4261f9106013478d6a27fc39f38

process: add get_output_mark() and grep_output()

Allow showing log lines matching specific regexes, from a specific start
point of a log.

My use case is to echo the handover related logging after an expected
handover failed, so that the reason is visible already in the console
output of a jenkins run. So far I would need to open the endless bsc log
and look up the matching place in it to get a conclusion about why a
handover failed.

Change-Id: Ib6569f7486e9d961bd79a5f24232e58d053667a1

fix: line nr in test name in wrong places

test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.

Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.

Override src() in Test, not name().

Use src() in backtraces.

The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
   I am 'test_suite' / 'hello_world.py:23'
becomes
   I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)

Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7

typo / newline in selftest log output

...because it otherwise causes confusing selftest log changes in
upcoming I0972c66b9165bd7f2b0b387e0335172849199193

Change-Id: I1eec31c5baba5ef33d0c8441b1e94a00d79af9bf

do not redirect_stdout in test scripts

In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.

Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().

The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.

As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.

I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.

Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193

log: process.py: fix weird space in stdout/stderr log output

Log output before:

  stdout:
   | line1
  | line2
  | line3

After:

  stdout:
  | line1
  | line2
  | line3

Change-Id: I4202e7dfbb550acaf55a6c6487fc07819ab18b94

implement test.get_log_output()

Retrieve a test's own logging. The aim is to provide logging belonging
to a given report fragment in the junit XML output, will be used by
upcoming test.report_fragment() feature.

Change-Id: Idfa0a45f3e6a18dd4fe692e81d732c70b5cffb76

add test.report_fragment()

Allow enriching the junit output with arbitrary subtasks within a test.

The current aim is, for handover tests, to not just show that a test
failed, but to show exactly which steps worked and which didn't, e.g.:

handover.py/01_bts0_started PASSED
handover.py/02.1_ms0_attach PASSED
handover.py/02.2_ms1_attach PASSED
handover.py/02.3_subscribed_in_msc PASSED
handover.py/03_call_established PASSED
handover.py/04.1_bts1_started FAILED

In this case it is immediately obvious from looking at the jenkins
results analyzer that bts1 is the cause of the test failure, and it is
visible which parts of the test are flaky, over time.

First user Will be the upcoming handover_2G suite, in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59

report fragment: strip ansi colors from junit XML

Jenkins does support showing ANSI colors on the web, but apparently not
in the junit results output. Strip ansi colors from report fragment
<system-out> text, to make it less annoying to read those on jenkins.

Change-Id: I656ecc23bbfd3f25bdf012c890e0c998168844d3

ctrl: set socket timeout

The blocking CTRL socket must have a timeout.

So far we have always been reading from the CTRL socket in a blocking
way. So far we are always lucky that we are indeed getting a response to
receive. Should the CTRL interface ever fail to answer, the tester would
be stuck infinitely.

Change-Id: I20091daf13981dc3d0d894af5a65c23f4db0471d

ctrl: increment message IDs for CTRL commands

Implicitly use distinct IDs for CTRL commands. This is a prerequisite
for matching GET_REPLY IDs to GET IDs, to correctly handle CTRL
responses without counting on being lucky.

Rather use 'use_id' as var name instead of overloading the 'id' keyword.

Change-Id: I698faa013eb14f88759685d50d925f5d82d80aa0

OsmoCtrl cleanup: get_var(), set_var(), get_trap()

CTRL interface interaction was mostly inherited from the first legacy
implementation of osmo-gsm-tester, and it was a pain to look at from the
start. Now, while I'm close to the topic, I want this to improve:

Properly match a GET_REPLY/SET_REPLY to a sent GET/SET by the message
ID.

Completely drop the do_get() and do_set(), which were not useful for
correct handling of the CTRL request and response messaging. The API to
use by callers is set_var(), get_var()/get_int_var() and get_trap().
These call the internal _sendrecv() (or for TRAP only _recv())
functions. Make it so that tese work both on an already connected
OsmoCtrl, as well as one that needs to establish a (short) connection,
so that both are trivially possible:

    # one CTRL connection stays open
    with OsmoCtrl(...) as ctrl:
  ctrl.get_var('var1')
  ctrl.get_var('var2')
  ctrl.get_var('var3')

and

  # get_var() opens a connection, does the GET and closes again
  OsmoCtrl(...).get_var('var1')

Do away with doubling the instances OsmoCtrl and e.g. OsmoBscCtrl.
Rather make OsmoBscCtrl a child class of OsmoCtrl, which means that we
no longer have bsc.ctrl().ctrl(), just bsc.ctrl().

Have VERB_* constants instead of dup'd strings.

Apply to / simplify all callers of OsmoCtrl.

Some of these changes are similar to recently added OsmoVty.

Change-Id: Id561e5a55d8057a997a8ec9e7fa6f94840194df1

keep Ctrl connections open for bsc, msc and nitb objects

The pattern to use 'with' to keep a CTRL connection open adds indents to
every test script that wants to avoid multiple reconnections to the
CTRL. Instead, keeping a single open connection that is cleaned up on
{bsc,msc,nitb} object cleanup ensures that a) the program started up
successfully and opened a CTRL port, b) always has a CTRL open without
having to worry about it and c) keeps test scripts less
complex/indented/crufted.

(These are all current users of the OsmoCtrl API.)

Change-Id: I53fedbe569c5ccbc4b1a17dafe1f8d1bb8200b24

osmo_ctrl.py: add RateCounters

First user will be the upcoming handover_2G/handover.py test in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: Id799b3bb81eb9c04d13c26ff611e40363920300e

add handover_2G suite, with handover.py test

Via VTY, handover two lchans of a voice call from bts0 to bts1 and back.

New scenarios/bts1-* allow selecting various types for bts1,
complementing the already existing files for selecting bts0.

Change-Id: I0b2671304165a1aaae2b386af46fbd8b098e3bd8

run handover_2G in default-suites.conf

Change-Id: Ic6bb013cc2c395211eeb86fe7a3d3f42671793c7

process output: do not leak ANSI colors past logged stdout/stderr

In jenkins, I still saw incidents of the entire log becoming colored
after a colored stderr snippet was printed to the log. Make absolutely
sure that no unterminated ANSI coloring is leaked.

Change-Id: Ib9ac1eea4a12d6d43ac8614491f016bbe9ca17b1

add option osmo-gsm-tester.py --suites-file my-suites-listing.conf

For example, run with '-S handover-suites.conf' to just do the handover
tests.

Change-Id: If896ac84d02a275deda2f20a93c29312df128ebb

OsmoCtrl: use one global common counter for CTRL IDs

It is easier to traverse debugging logs if the CTRL request and response
IDs are globally unique across all programs and tests. Before this, we
were using 0 almost everywhere.

(This is not strictly needed for correctness, since each CTRL client has
its own request ID scope; just we open fairly many separate CTRL clients
all the time in our tests.)

Change-Id: I44c51f4fb5beb6cedf98ea0d6684a24c6aa418c7

sysmocom resources.conf.prod: Update modem paths + add new modems after HW upgrade

Change-Id: I346a5bb38270a32695ec73994179bd75d873b67a

resource: Fix next_zmq_port_range using bvci file name

Change-Id: I293b7de1066ea69ccb0e495a69602ed56810018d

sysmocom/resources.conf.virtual: remove no longer existing arfcn resources

ARFCN are not resources anymore since a recent commit.

Fixes: ecce402f43b708bf178d603e2bd8587e72fdb3b1
Change-Id: Icc0971b285e8db085670808b7d7bb98c38db2753

sysmocom: Port nitb_netreg_mass suite to osmo split network

Change-Id: I221df84d76f73e15eed98036b31adc8089c8dc99

jenkins-build-srslte.sh: allow custom binaries

this allows to specify the environmental var
wanted_binaries_bin that will be included in the
trial package. if the variable isn't set, all three
main srsLTE binaries are included

Change-Id: I63f5799328435ce32b5bfd6dae6fd0b37e1e5397

srsenb_drb.conf.tmpl: add QCI1 DRB config

use default QCI1 config for real-time voice traffic

Change-Id: I43f408b1a3c172a5eb8327f03df49801e063e5cd

enb: include name label and run_addr in enb remote path

this allows to quickly identify the enb host in the logs

Change-Id: I9c979de852a23041e43692e97ebcbfb389b52fdb

srsue.conf.tmpl: reorder conditions for setting RRC release

the order of checks needs to go from high to low, i.e. the higher
release feature (e.g. qam256) needs to be checked and set first.

in theory it should also be possible to have a CA-capable UE
that does not support QAM256, but for srsUE we announce both anyway.

Change-Id: I2fa49f0cb5d80db412a811ceeb380359c8ad67a7

intraenb-intrafreq-ho.conf: add eNB to cell config

make sure to have the right eNB that matches the
cells neighbor ECIDs, otherwise the eNB may trigger
a S1 HO if the eNB ID is set elsewhere

Change-Id: Id494ac1984846e78a6456bb6a17e72d63c9dacf5

srsue.conf.tmpl: disable TTI stats for ZMQ

since ZMQ runs are not using wall clock anyway, measuring
TTI execution isn't useful, disable it therefore to avoid
misleading warnings.

Change-Id: I5c2cb0abcfce0ee67806f6611356f4d5d180541d

enb_srs: only use QAM64 tables if both UE and eNB are configured for QAM64

this was causing failed tests because to achieve QAM64 rates both
eNB and UE need to support it and have it activated.

Change-Id: I599df92d69eeb56a5d44327de08f004222cff073

enb_srs: adopt throughput thresholds for QAM256

it turned out that we have to reduce the MCS when using QAM256,
especially for 6 PRB as subframe 0 and 5 contains PBCH and PSS
signals, so the available REs are reduced.

The eNB scheduler now has this limitation in mind and lowers the MCS.

Change-Id: I0e38fe28002fd68c768cc8dcffcf74f4f190df02

srsepc.conf.tmpl: increase default value of T3413 (paging timer)

we've seen that the default value of 2s might be a bit tight.
the value is network dependent so no real default value exists.
however, 6s seems to be a good compromise.

Change-Id: Icc75ee048c2cc605ab26c4c87a039519998f7e57

srsenb.conf.tmpl: set nof_of_prach_threds to 0 for ZMQ

this executed the PRACH search in the main thread so
PHY workers and PRACH processing don't diverge in loaded
scenarios

Change-Id: I97eddb5f9ca682251ded1f73093c2d0a49462bd6

Introduce Android UEs as new modems

To expand the test capacities we would like to introduce
Android UEs as new modems. Currently the following tests
are supported:
- Ping
- iPerf3 DL/UL
- RRC Mobile MT Ping

In the following is a small description.

Prerequisites:
    - Android UE
        - Rooted (Ping, iPerf, RRC Idle MT Ping)
        - Qualcomm baseband with working diag_mdlog (RRC Idle MT Ping)
        - iPerf3
        - Dropbear
    - OGT Slave Unit
        - Android SDK Platform-Tools
  (https://developer.android.com/studio/releases/platform-tools#downloads)
        - Pycrate (https://github.com/P1sec/pycrate)
        - SCAT
            clone https://github.com/bedrankara/scat/ & install dependencies
            checkout branch ogt
            symlink scat (ln -s ~/scat/scat.py /usr/local/bin/scat)

Infrastructure explaination:
The Android UEs are connected to the OGT Units via USB. We
activate tethering and set up a SSH server (with Dropbear).
We chose tethering over WiFi to have a more stable route
for the ssh connection. We forward incoming connections to
the OGT unit hosting the Android UE(s) on specific ports
to the UEs via iptables. This enables OGT to issue commands
directly to the UEs. In case of local execution we use ADB
to issue commands to the AndroidUE. The set up was tested
with 5 Android UEs connected in parallel but it should be
scalable to the number of available IPs in the respective
subnet. Furthermore, we need to cross compile Dropbear
and iPerf3 to use them on the UEs. These tools have to be
added to the $PATH variable of the UEs.

Examplary set up:
In this example we have two separate OGT units (master
and slave) and two Android UEs that are connected to the
slave unit. An illustration may be found here: https://ibb.co/6BXSP2C

On UE 1:
ip address add 192.168.42.130/24 dev rndis0
ip route add 192.168.42.0/24 dev rndis0 table local_network
dropbearmulti dropbear -F -E -p 130 -R -T /data/local/tmp/authorized_keys  -U 0 -G 0 -N root -A

On UE 2:
ip address add 192.168.42.131/24 dev rndis0
ip route add 192.168.42.0/24 dev rndis0 table local_network
dropbearmulti dropbear -F -E -p 131 -R -T /data/local/tmp/authorized_keys  -U 0 -G 0 -N root -A

On OGT slave unit:
sudo ip link add name ogt type bridge
sudo ip l set eth0 master ogt
sudo ip l set enp0s20f0u1 master ogt
sudo ip l set enp0s20f0u2 master ogt
sudo ip a a 192.168.42.1/24 dev ogt
sudo ip link set ogt up

Now we have to manually connect to every UE from OGT Master
to set up SSH keys and verify that the setup works.
Therefore, use:
ssh -p [UE-PORT] root@[OGT SLAVE UNIT's IP]

Finally, to finish the setup procedure create the
remote_run_dir for Android UEs on the slave unit like
following:
mkdir /osmo-gsm-tester-androidue
chown jenkins /osmo-gsm-tester-androidue

Example for modem in resource.conf:
- label: mi5g
  type: androidue
  imsi: '901700000034757'
  ki: '85E9E9A947B9ACBB966ED7113C7E1B8A'
  opc: '3E1C73A29B9C293DC5A763E42C061F15'
  apn:
    apn: 'srsapn'
    mcc: '901'
    mnc: '70'
    select: 'True'
    auth_algo: 'milenage'
  features: ['4g', 'dl_qam256', 'qc_diag']
  run_node:
    run_type: ssh
    run_addr: 100.113.1.170
    ssh_user: jenkins
    ssh_addr: 100.113.1.170
    ue_ssh_port: 130
    adb_serial_id: '8d3c79a7'
  scat_parser:
    run_type: local
    run_addr: 127.0.0.1
    adb_serial_id: '8d3c79a7'

Example for default-suites.conf:
- 4g:ms-label@mi5g+srsenb-rftype@uhd+mod-enb-nprb@25+mod-enb-txmode@1

Change-Id: I79a5d803e869a868d4dac5e0d4c2feb38038dc5c

enb,ue,process: move stdout counter helper to process objects

this patch adds the stdout counter to count events happening
on the stdout (known from the UE already) to the common
process class so they can also be used from the eNB (and other objects)

In addition, we add a PRACH counter to be used for tests.

Change-Id: I434f072b8aa6f4dce9f90889c6b40832f6798ff8

4g: add checks for received PRACHs for ping tests

the tests already check for sent PRACHs, this will
also check for received PRACHs on the eNB to make
sure we don't detect any fake PRACHs during the tests.

Change-Id: Ib23056b1ad4b18a7dde6cd0c997f5114ac2acb61

enb: add abstract method stop()

implement as noop for Amarisoft eNB, srsENB will send q+Enter to stdin,
which is implemented in class srslte_common()

Change-Id: Ide606e1a6b523997215aa2fa39d4d56ae1f49181

enb: Set cipher and integrity via scenario file

This patch enables setting cipher and integrity algorithms
in Amarisoft eNB and srsENB via scenario files. If no
settings are defined following defaults are applied:
- Cipher algorithm: EEA0, EEA2, EEA1
- Integrity algorithm: EIA2, EIA1, EIA0

Example of setting cipher algorithms:
- 4g:srsue-rftype@uhd+srsenb-rftype@uhd+mod-enb-cipher@eea1+mod-enb-cipher@eea0+mod-enb-nprb@6

Change-Id: I595206b7d49016fb6d0aec175c828d9537c53886

scenarios: remove ncells@ scenario and define number of cells explicitly

the specific cell scenarios files already contain cell-related
information so having a extra ncells scenario seems too much overhead.
furthermore, in more complex tests its helps to reduce the filename/path
lenght to avoid hitting the 255 character limit

Change-Id: If022d9e8d239805ee1280fb52660e68dda7484a0

srsepc.conf.tmpl: double value of T3413 (paging timer)

we've already incrased the t3413 value in 9c7fd618d226dc532b9408c329cfc2380de4850a
but realized it may still be too low in some cases.

we set it to 12s now.

Change-Id: I9c3c59c0ff8f539a44b420f8ba31eea016c211ce

srsue.conf.tmpl: reduce nof PHY threads to 2 for ZMQ

we've detected a possible race condition during the Msg3
transmission that caused the thread that sets the Msg3 grant
to be delayed. The PHY worker that executed TTI+2 finished
by that time already and didn't even see the pending UL grant.

This is issue is more likely to happen on loaded system,
for example when running parallel ZMQ jobs. We therefore decided
to reduce the number of parallel PHY workers to 2 so the issue
is circumvented.

Change-Id: Ibdb42a1705d87b6d343201458c8fe397398802bc

enb: Support cells from different ENBs in ncell_list

Change-Id: I1242c56ff42caea06e7f96317def82064fd11325

sysmocom: Introduce suite 4g2enb

Change-Id: I73b1a5e07f81fa1e3d3db1db243a3d0b137bda8e

resource: Support waiting for reserved resources until available

Before this patch, almost everything was in place to support concurrent
osmo-gsm-tester instances sharing a common state dir. However, during
resource reservation, if the reservation couldn't be done due to too
many resources being in use, osmo-gsm-tester would fail and skip the
test suite.
With this patch, OGT will wait until some reserved resources are
released and then try requesting the reservation again.

Change-Id: I938602ee890712fda82fd3f812d8edb1bcd05e08

iperf3_{dl,ul}: adapt pass/fail threshold

change pass threshold to 80% of the max rate for
half of the testduration (rolling average).

the overall average might be lowered because of a slower
TCP start or a late UE attach.

Change-Id: I8a545b8175784e9d6b49d6bf80f637ef7aa731f7

{enb,ms}_srs: Add DL QAM-256 support

Due to the integration of DL-QAM256 another table for DL max rates is needed.
Therefore, I added the parameter 'qam256' to the feature list in the resource.cfg.

The patch also enables the correct UE settings in the config file.

Change-Id: I2d34395449cdcfb31db66ea887d9adbee551e757

epc_srs.py: Fixed reference in local execution

fixes local execution of srsEPC

Change-Id: I36db8f2f65be6c92ccea050f550e21b67bb9e50e

enb-cells_2ca: add missing cell params

the default params are set for a single cell in defaults.conf
but this 2 cell config requires them to be set explicitly.

Change-Id: I8b3c486eb3e42aeb04b6a7548d3f0de2aa40ee0c

schema: allow empty lists in resource schema

Change-Id: I9249e29deef00d4dd68f89c704ef1d3093c799fd

srsenb_rr.conf: add PCI, TAC and root_seq_idx as cell param

make sure to insert cell specific TAC, PCI and root seq ind
into cell config and do not depend on cell index for a particular
enb. This causes issues in multi-eNB setups.

Change-Id: I6642128a449a0562dd23f7fa393ff48ae2503006

enb: Fix non-1st ENB gnubreaker remote address

ENB was using its own address as a remote address when the broker is
used, which actually only holds true for the first ENB (since the broker
is currently run on the same host/address of the first ENB).

Change-Id: I5c83075d140632846c6d357152963754d16b4633

Copy gnuradio zmq broker at runtime

The remotely run script is moved into a new subdir called "external",
where external utils to be used by osmo-gsm-tester (exernal to its own
process) are placed.
It needs to be in another directory because python files in obj/ are
loaded at startup of osmo-gsm-tester to dynamically load schemas.

Change-Id: I633a85294694f2c6efd58535729e9b8af166b3ff

enb: Name preferrably by label, use it in remote run dir

This way several ENB will place its config and logs into different dirs.

Change-Id: I9b0c74cd63bc3888f380c66d94271beafc0db915

sysmocom: emergency_preemptive_call.py: Avoid test failure if 2nd call not fully established

It may happen that the non-emergency call MT leg is still not properly
released when the emergency call MT leg is to be assigned a TCH, meaning
BSC will fail with an Assignment Failure upon Assignment request from
MSC.
The test sometimes passed and sometimes didn't, due to above mentioned
race condition. Let's relax a bit the test expectancies to have it
always passing, while still verifying preemption happens, and MT phone
is reached.

Related: OS#4549
Change-Id: I3697227cac56a1327f9ea08c5c2f52568e8d2a8a

gnuradio zmq broker refactoring

First step towards support everal ENBs and installing the remote script
at runtime:

* The gnuradio broker class is moved to its own file, to decouple it
  from RFemu.
* The ENBs are registered earlier in the test so that the GrBroker knows
  when to start (delay start + setup until all ENBs have been configured).
* Handle ENBs internally as a list.

Change-Id: I4f1095bbc7ed0a816fe47caef44f7becadd9d737