#2762 (Feb 28, 2021, 11:19:00 PM)

enb: add scenario file for inter-eNB HO with two cells

Change-Id: I356a81814a2d7714a1a5252aa357257901fa6227

cell_modifiers: refactor HO cell configs

* rename intrafreq vs intraenb
* add missing mandatory params

Change-Id: Icc1fe5c1711ecbed2f1b8b1e6a0b51c2d28ec651

enb: add QAM64 support for UL

* add new UE feature
* enable in srsue.conf.templ
* add new table for maximum rates
* add config scenario to enable SIB option for QAM64

Change-Id: I6ac2c9989a761e91b93d76c2507f55f0140b202d

add scenario to modify bandwith of 2 eNB objects

required for Inter-eNB HO configs, for example.

Change-Id: Iaf053941ff31c4ba94edf67fc54c0f1c1441577d

ms_amarisoft: fix various regressions

this patch fixes some regressions in the Amarisoft UE class,
the config template, etc. that have been undetected bc we never
executed tests with it.

Change-Id: I397e675a4018acf3372a3b7e29fd864703b2b919

jenkins-build-amarisoft.sh: set symlink to srslte_rf lib

Change-Id: Ifc8e5ba0dd59fb92cdf22f2a18acea904e2153f7

gnuradio_zmq_broker_remote: fix eNB port mapping for carrier aggregation

the mapping of ports didn't consider when a eNB had two ports for
different EARFCNs

Change-Id: I5548f67a3b37b03482e4b6172828fcd891e704a5

Process: Add remote_port as new parameter

remote_port defines a custom/additional port for
connections over ssh. It may be used in case several
ssh instances share one IP address.

Change-Id: I2c93fd2ea1c10c333d00eafd3c1066c35796e398

prod: after hardware move, adjust modem pci for sierra_1st and sierra_2nd (ONLY TWO)

Doing only the first two modems so far because I need them for handover
runs. The other modems are still todo!

Change-Id: Ibd71acfc76c01ffd105abe5effc1d246b1e65f85

contrib: Disable libmnl dependency cross-compiling osmo-pcu

Only newest OE SDKs have libmnl added, but we don't need this feature
anyway, so let's disable it in order to get builds running fine again.

Change-Id: Id3b30d0b183cd21fd6df89fdda5581ab58c7a9a5

contrib: Disable libmnl dependency cross-compiling osmo-bts

Only newest OE SDKs have libmnl added, but we don't need this feature
anyway, so let's disable it in order to get builds running fine again.

Change-Id: Ief9cbca6e10b1ed6acf6e079f660fc3f7c863c7b

cfg templates: log level and source basename last, log no category-hex

Makes for saner log output in stderr files.

Change-Id: I2755beffd18d6779367c04e4aaaed119d66eb3be

add osmo_vty.py

To trigger manual handovers, I need a VTY interface. The non-trivial
parts of this are copied from osmo-python-tests osmo_interact_vty.py.

Will be used in the upcoming handover_2G test suite in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: I7c17b143b7c690b8c4105ee7c6272670046fa91d

add bsc.vty: always keep a BSC VTY connection ready

The BSC's VTY port will be used to trigger manual handover, and to
retrieve a list of active lchans from the BSC, in the upcoming
handover_2G test suite, I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: I06652db04fc9e48748f3c2196334f5352e9cc48a

bsc: add active lchan matching via vty

Will be used in upcoming handover_2G test suite in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8, which needs to verify that a
handover actually ended up on the expected lchan.

Change-Id: I03df8f3ae2ee47930eee311c7ce104c36dbb3154

bts: allow configuring ARFCN and BSIC: add to schema

resources.conf is already allowed to set these, but in order to let
scenarios and suites modify these values, they need to be in the schema.

Change-Id: I8e0583e9208d563c66b0bdc9f25b1b74f92403c2

log passed tests

In the end of a test suite, do not omit the passed tests. For example,
running handover against N BTS combinations, it was hard to summarize
which BTS models actually succeeded, with only the failures listed.

Besides the "FAIL" listings, now print something like this in the end:

PASS: handover:sysmo+secondbts-trx-b200 (pass: 1)
     pass: handover.py (198.8 sec)
PASS: handover:sysmo+secondbts-trx-umtrx (pass: 1)
     pass: handover.py (192.7 sec)
PASS: handover:trx-b200+secondbts-trx-umtrx (pass: 1)
     pass: handover.py (193.1 sec)

Change-Id: Ib85a5b90e267c2ed2f844691187ecadc8939b1bb

log reserved resources

On non-debug log level, show something like this at the beginning of
each suite:

03:45:49.439720 tst handover:sysmo+secondbts-trx-b200: RESERVED RESOURCES for handover:
bts
   sysmoBTS 1002
   Ettus B200
ip_address
   10.42.42.2
   10.42.42.3
   10.42.42.4
   10.42.42.5
   10.42.42.6
   10.42.42.7
modem
   sierra_1st
   sierra_2nd

Change-Id: Ic23556eafee654c93d13c5ef405028da09bd51d7

configure fixed ARFCN per bts

Remove ARFCNs as a concept from resource pool, assign a fixed ARFCN to
each BTS and TRX in the resource pools.

Using ARFCNs on specific bands as resources was an idea that is hard to
implement, because specific BTS dictate selection of bands which
influences which ARFCNs can be picked. That means reserving ARFCN
resources is only possible after reserving specific BTS resources, but
the tester is currently not capable of such two-stage resolution.

Writing handover tests, I got the problem that both BTS in a scenario
attempt to use the same ARFCN.

The by far easiest solution is to assign one fixed ARFCN to each BTS and
TRX. If ever needed, a scenario modifier can still configure different
ARFCNs.

(Due to uncertainty about OC2G operation stability, I prefer to leave
OC2G on ARFCN 50, as it happened to end up being configured before this
patch.)

Change-Id: I0a6c60544226f4261f9106013478d6a27fc39f38

process: add get_output_mark() and grep_output()

Allow showing log lines matching specific regexes, from a specific start
point of a log.

My use case is to echo the handover related logging after an expected
handover failed, so that the reason is visible already in the console
output of a jenkins run. So far I would need to open the endless bsc log
and look up the matching place in it to get a conclusion about why a
handover failed.

Change-Id: Ib6569f7486e9d961bd79a5f24232e58d053667a1

fix: line nr in test name in wrong places

test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.

Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.

Override src() in Test, not name().

Use src() in backtraces.

The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
   I am 'test_suite' / 'hello_world.py:23'
becomes
   I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)

Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7

typo / newline in selftest log output

...because it otherwise causes confusing selftest log changes in
upcoming I0972c66b9165bd7f2b0b387e0335172849199193

Change-Id: I1eec31c5baba5ef33d0c8441b1e94a00d79af9bf

do not redirect_stdout in test scripts

In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.

Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().

The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.

As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.

I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.

Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193

log: process.py: fix weird space in stdout/stderr log output

Log output before:

  stdout:
   | line1
  | line2
  | line3

After:

  stdout:
  | line1
  | line2
  | line3

Change-Id: I4202e7dfbb550acaf55a6c6487fc07819ab18b94

implement test.get_log_output()

Retrieve a test's own logging. The aim is to provide logging belonging
to a given report fragment in the junit XML output, will be used by
upcoming test.report_fragment() feature.

Change-Id: Idfa0a45f3e6a18dd4fe692e81d732c70b5cffb76

add test.report_fragment()

Allow enriching the junit output with arbitrary subtasks within a test.

The current aim is, for handover tests, to not just show that a test
failed, but to show exactly which steps worked and which didn't, e.g.:

handover.py/01_bts0_started PASSED
handover.py/02.1_ms0_attach PASSED
handover.py/02.2_ms1_attach PASSED
handover.py/02.3_subscribed_in_msc PASSED
handover.py/03_call_established PASSED
handover.py/04.1_bts1_started FAILED

In this case it is immediately obvious from looking at the jenkins
results analyzer that bts1 is the cause of the test failure, and it is
visible which parts of the test are flaky, over time.

First user Will be the upcoming handover_2G suite, in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59

report fragment: strip ansi colors from junit XML

Jenkins does support showing ANSI colors on the web, but apparently not
in the junit results output. Strip ansi colors from report fragment
<system-out> text, to make it less annoying to read those on jenkins.

Change-Id: I656ecc23bbfd3f25bdf012c890e0c998168844d3

ctrl: set socket timeout

The blocking CTRL socket must have a timeout.

So far we have always been reading from the CTRL socket in a blocking
way. So far we are always lucky that we are indeed getting a response to
receive. Should the CTRL interface ever fail to answer, the tester would
be stuck infinitely.

Change-Id: I20091daf13981dc3d0d894af5a65c23f4db0471d

ctrl: increment message IDs for CTRL commands

Implicitly use distinct IDs for CTRL commands. This is a prerequisite
for matching GET_REPLY IDs to GET IDs, to correctly handle CTRL
responses without counting on being lucky.

Rather use 'use_id' as var name instead of overloading the 'id' keyword.

Change-Id: I698faa013eb14f88759685d50d925f5d82d80aa0

OsmoCtrl cleanup: get_var(), set_var(), get_trap()

CTRL interface interaction was mostly inherited from the first legacy
implementation of osmo-gsm-tester, and it was a pain to look at from the
start. Now, while I'm close to the topic, I want this to improve:

Properly match a GET_REPLY/SET_REPLY to a sent GET/SET by the message
ID.

Completely drop the do_get() and do_set(), which were not useful for
correct handling of the CTRL request and response messaging. The API to
use by callers is set_var(), get_var()/get_int_var() and get_trap().
These call the internal _sendrecv() (or for TRAP only _recv())
functions. Make it so that tese work both on an already connected
OsmoCtrl, as well as one that needs to establish a (short) connection,
so that both are trivially possible:

    # one CTRL connection stays open
    with OsmoCtrl(...) as ctrl:
  ctrl.get_var('var1')
  ctrl.get_var('var2')
  ctrl.get_var('var3')

and

  # get_var() opens a connection, does the GET and closes again
  OsmoCtrl(...).get_var('var1')

Do away with doubling the instances OsmoCtrl and e.g. OsmoBscCtrl.
Rather make OsmoBscCtrl a child class of OsmoCtrl, which means that we
no longer have bsc.ctrl().ctrl(), just bsc.ctrl().

Have VERB_* constants instead of dup'd strings.

Apply to / simplify all callers of OsmoCtrl.

Some of these changes are similar to recently added OsmoVty.

Change-Id: Id561e5a55d8057a997a8ec9e7fa6f94840194df1

keep Ctrl connections open for bsc, msc and nitb objects

The pattern to use 'with' to keep a CTRL connection open adds indents to
every test script that wants to avoid multiple reconnections to the
CTRL. Instead, keeping a single open connection that is cleaned up on
{bsc,msc,nitb} object cleanup ensures that a) the program started up
successfully and opened a CTRL port, b) always has a CTRL open without
having to worry about it and c) keeps test scripts less
complex/indented/crufted.

(These are all current users of the OsmoCtrl API.)

Change-Id: I53fedbe569c5ccbc4b1a17dafe1f8d1bb8200b24

osmo_ctrl.py: add RateCounters

First user will be the upcoming handover_2G/handover.py test in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: Id799b3bb81eb9c04d13c26ff611e40363920300e

add handover_2G suite, with handover.py test

Via VTY, handover two lchans of a voice call from bts0 to bts1 and back.

New scenarios/bts1-* allow selecting various types for bts1,
complementing the already existing files for selecting bts0.

Change-Id: I0b2671304165a1aaae2b386af46fbd8b098e3bd8

run handover_2G in default-suites.conf

Change-Id: Ic6bb013cc2c395211eeb86fe7a3d3f42671793c7

process output: do not leak ANSI colors past logged stdout/stderr

In jenkins, I still saw incidents of the entire log becoming colored
after a colored stderr snippet was printed to the log. Make absolutely
sure that no unterminated ANSI coloring is leaked.

Change-Id: Ib9ac1eea4a12d6d43ac8614491f016bbe9ca17b1

add option osmo-gsm-tester.py --suites-file my-suites-listing.conf

For example, run with '-S handover-suites.conf' to just do the handover
tests.

Change-Id: If896ac84d02a275deda2f20a93c29312df128ebb

OsmoCtrl: use one global common counter for CTRL IDs

It is easier to traverse debugging logs if the CTRL request and response
IDs are globally unique across all programs and tests. Before this, we
were using 0 almost everywhere.

(This is not strictly needed for correctness, since each CTRL client has
its own request ID scope; just we open fairly many separate CTRL clients
all the time in our tests.)

Change-Id: I44c51f4fb5beb6cedf98ea0d6684a24c6aa418c7

sysmocom resources.conf.prod: Update modem paths + add new modems after HW upgrade

Change-Id: I346a5bb38270a32695ec73994179bd75d873b67a

resource: Fix next_zmq_port_range using bvci file name

Change-Id: I293b7de1066ea69ccb0e495a69602ed56810018d

sysmocom/resources.conf.virtual: remove no longer existing arfcn resources

ARFCN are not resources anymore since a recent commit.

Fixes: ecce402f43b708bf178d603e2bd8587e72fdb3b1
Change-Id: Icc0971b285e8db085670808b7d7bb98c38db2753

sysmocom: Port nitb_netreg_mass suite to osmo split network

Change-Id: I221df84d76f73e15eed98036b31adc8089c8dc99

jenkins-build-srslte.sh: allow custom binaries

this allows to specify the environmental var
wanted_binaries_bin that will be included in the
trial package. if the variable isn't set, all three
main srsLTE binaries are included

Change-Id: I63f5799328435ce32b5bfd6dae6fd0b37e1e5397

srsenb_drb.conf.tmpl: add QCI1 DRB config

use default QCI1 config for real-time voice traffic

Change-Id: I43f408b1a3c172a5eb8327f03df49801e063e5cd

enb: include name label and run_addr in enb remote path

this allows to quickly identify the enb host in the logs

Change-Id: I9c979de852a23041e43692e97ebcbfb389b52fdb

srsue.conf.tmpl: reorder conditions for setting RRC release

the order of checks needs to go from high to low, i.e. the higher
release feature (e.g. qam256) needs to be checked and set first.

in theory it should also be possible to have a CA-capable UE
that does not support QAM256, but for srsUE we announce both anyway.

Change-Id: I2fa49f0cb5d80db412a811ceeb380359c8ad67a7

intraenb-intrafreq-ho.conf: add eNB to cell config

make sure to have the right eNB that matches the
cells neighbor ECIDs, otherwise the eNB may trigger
a S1 HO if the eNB ID is set elsewhere

Change-Id: Id494ac1984846e78a6456bb6a17e72d63c9dacf5

srsue.conf.tmpl: disable TTI stats for ZMQ

since ZMQ runs are not using wall clock anyway, measuring
TTI execution isn't useful, disable it therefore to avoid
misleading warnings.

Change-Id: I5c2cb0abcfce0ee67806f6611356f4d5d180541d

enb_srs: only use QAM64 tables if both UE and eNB are configured for QAM64

this was causing failed tests because to achieve QAM64 rates both
eNB and UE need to support it and have it activated.

Change-Id: I599df92d69eeb56a5d44327de08f004222cff073

enb_srs: adopt throughput thresholds for QAM256

it turned out that we have to reduce the MCS when using QAM256,
especially for 6 PRB as subframe 0 and 5 contains PBCH and PSS
signals, so the available REs are reduced.

The eNB scheduler now has this limitation in mind and lowers the MCS.

Change-Id: I0e38fe28002fd68c768cc8dcffcf74f4f190df02

srsepc.conf.tmpl: increase default value of T3413 (paging timer)

we've seen that the default value of 2s might be a bit tight.
the value is network dependent so no real default value exists.
however, 6s seems to be a good compromise.

Change-Id: Icc75ee048c2cc605ab26c4c87a039519998f7e57

srsenb.conf.tmpl: set nof_of_prach_threds to 0 for ZMQ

this executed the PRACH search in the main thread so
PHY workers and PRACH processing don't diverge in loaded
scenarios

Change-Id: I97eddb5f9ca682251ded1f73093c2d0a49462bd6

Introduce Android UEs as new modems

To expand the test capacities we would like to introduce
Android UEs as new modems. Currently the following tests
are supported:
- Ping
- iPerf3 DL/UL
- RRC Mobile MT Ping

In the following is a small description.

Prerequisites:
    - Android UE
        - Rooted (Ping, iPerf, RRC Idle MT Ping)
        - Qualcomm baseband with working diag_mdlog (RRC Idle MT Ping)
        - iPerf3
        - Dropbear
    - OGT Slave Unit
        - Android SDK Platform-Tools
  (https://developer.android.com/studio/releases/platform-tools#downloads)
        - Pycrate (https://github.com/P1sec/pycrate)
        - SCAT
            clone https://github.com/bedrankara/scat/ & install dependencies
            checkout branch ogt
            symlink scat (ln -s ~/scat/scat.py /usr/local/bin/scat)

Infrastructure explaination:
The Android UEs are connected to the OGT Units via USB. We
activate tethering and set up a SSH server (with Dropbear).
We chose tethering over WiFi to have a more stable route
for the ssh connection. We forward incoming connections to
the OGT unit hosting the Android UE(s) on specific ports
to the UEs via iptables. This enables OGT to issue commands
directly to the UEs. In case of local execution we use ADB
to issue commands to the AndroidUE. The set up was tested
with 5 Android UEs connected in parallel but it should be
scalable to the number of available IPs in the respective
subnet. Furthermore, we need to cross compile Dropbear
and iPerf3 to use them on the UEs. These tools have to be
added to the $PATH variable of the UEs.

Examplary set up:
In this example we have two separate OGT units (master
and slave) and two Android UEs that are connected to the
slave unit. An illustration may be found here: https://ibb.co/6BXSP2C

On UE 1:
ip address add 192.168.42.130/24 dev rndis0
ip route add 192.168.42.0/24 dev rndis0 table local_network
dropbearmulti dropbear -F -E -p 130 -R -T /data/local/tmp/authorized_keys  -U 0 -G 0 -N root -A

On UE 2:
ip address add 192.168.42.131/24 dev rndis0
ip route add 192.168.42.0/24 dev rndis0 table local_network
dropbearmulti dropbear -F -E -p 131 -R -T /data/local/tmp/authorized_keys  -U 0 -G 0 -N root -A

On OGT slave unit:
sudo ip link add name ogt type bridge
sudo ip l set eth0 master ogt
sudo ip l set enp0s20f0u1 master ogt
sudo ip l set enp0s20f0u2 master ogt
sudo ip a a 192.168.42.1/24 dev ogt
sudo ip link set ogt up

Now we have to manually connect to every UE from OGT Master
to set up SSH keys and verify that the setup works.
Therefore, use:
ssh -p [UE-PORT] root@[OGT SLAVE UNIT's IP]

Finally, to finish the setup procedure create the
remote_run_dir for Android UEs on the slave unit like
following:
mkdir /osmo-gsm-tester-androidue
chown jenkins /osmo-gsm-tester-androidue

Example for modem in resource.conf:
- label: mi5g
  type: androidue
  imsi: '901700000034757'
  ki: '85E9E9A947B9ACBB966ED7113C7E1B8A'
  opc: '3E1C73A29B9C293DC5A763E42C061F15'
  apn:
    apn: 'srsapn'
    mcc: '901'
    mnc: '70'
    select: 'True'
    auth_algo: 'milenage'
  features: ['4g', 'dl_qam256', 'qc_diag']
  run_node:
    run_type: ssh
    run_addr: 100.113.1.170
    ssh_user: jenkins
    ssh_addr: 100.113.1.170
    ue_ssh_port: 130
    adb_serial_id: '8d3c79a7'
  scat_parser:
    run_type: local
    run_addr: 127.0.0.1
    adb_serial_id: '8d3c79a7'

Example for default-suites.conf:
- 4g:ms-label@mi5g+srsenb-rftype@uhd+mod-enb-nprb@25+mod-enb-txmode@1

Change-Id: I79a5d803e869a868d4dac5e0d4c2feb38038dc5c

enb,ue,process: move stdout counter helper to process objects

this patch adds the stdout counter to count events happening
on the stdout (known from the UE already) to the common
process class so they can also be used from the eNB (and other objects)

In addition, we add a PRACH counter to be used for tests.

Change-Id: I434f072b8aa6f4dce9f90889c6b40832f6798ff8

4g: add checks for received PRACHs for ping tests

the tests already check for sent PRACHs, this will
also check for received PRACHs on the eNB to make
sure we don't detect any fake PRACHs during the tests.

Change-Id: Ib23056b1ad4b18a7dde6cd0c997f5114ac2acb61

enb: add abstract method stop()

implement as noop for Amarisoft eNB, srsENB will send q+Enter to stdin,
which is implemented in class srslte_common()

Change-Id: Ide606e1a6b523997215aa2fa39d4d56ae1f49181

enb: Set cipher and integrity via scenario file

This patch enables setting cipher and integrity algorithms
in Amarisoft eNB and srsENB via scenario files. If no
settings are defined following defaults are applied:
- Cipher algorithm: EEA0, EEA2, EEA1
- Integrity algorithm: EIA2, EIA1, EIA0

Example of setting cipher algorithms:
- 4g:srsue-rftype@uhd+srsenb-rftype@uhd+mod-enb-cipher@eea1+mod-enb-cipher@eea0+mod-enb-nprb@6

Change-Id: I595206b7d49016fb6d0aec175c828d9537c53886

scenarios: remove ncells@ scenario and define number of cells explicitly

the specific cell scenarios files already contain cell-related
information so having a extra ncells scenario seems too much overhead.
furthermore, in more complex tests its helps to reduce the filename/path
lenght to avoid hitting the 255 character limit

Change-Id: If022d9e8d239805ee1280fb52660e68dda7484a0

srsepc.conf.tmpl: double value of T3413 (paging timer)

we've already incrased the t3413 value in 9c7fd618d226dc532b9408c329cfc2380de4850a
but realized it may still be too low in some cases.

we set it to 12s now.

Change-Id: I9c3c59c0ff8f539a44b420f8ba31eea016c211ce

srsue.conf.tmpl: reduce nof PHY threads to 2 for ZMQ

we've detected a possible race condition during the Msg3
transmission that caused the thread that sets the Msg3 grant
to be delayed. The PHY worker that executed TTI+2 finished
by that time already and didn't even see the pending UL grant.

This is issue is more likely to happen on loaded system,
for example when running parallel ZMQ jobs. We therefore decided
to reduce the number of parallel PHY workers to 2 so the issue
is circumvented.

Change-Id: Ibdb42a1705d87b6d343201458c8fe397398802bc