Skip to content
Failed

Changes

Summary

  1. add osmo_vty.py (details)
  2. add bsc.vty: always keep a BSC VTY connection ready (details)
  3. bsc: add active lchan matching via vty (details)
  4. bts: allow configuring ARFCN and BSIC: add to schema (details)
  5. log passed tests (details)
  6. log reserved resources (details)
  7. configure fixed ARFCN per bts (details)
  8. process: add get_output_mark() and grep_output() (details)
  9. fix: line nr in test name in wrong places (details)
  10. typo / newline in selftest log output (details)
  11. do not redirect_stdout in test scripts (details)
  12. log: process.py: fix weird space in stdout/stderr log output (details)
  13. implement test.get_log_output() (details)
  14. add test.report_fragment() (details)
  15. report fragment: strip ansi colors from junit XML (details)
  16. ctrl: set socket timeout (details)
  17. ctrl: increment message IDs for CTRL commands (details)
  18. OsmoCtrl cleanup: get_var(), set_var(), get_trap() (details)
  19. keep Ctrl connections open for bsc, msc and nitb objects (details)
  20. osmo_ctrl.py: add RateCounters (details)
  21. add handover_2G suite, with handover.py test (details)
  22. run handover_2G in default-suites.conf (details)
  23. process output: do not leak ANSI colors past logged stdout/stderr (details)
  24. add option osmo-gsm-tester.py --suites-file my-suites-listing.conf (details)
  25. OsmoCtrl: use one global common counter for CTRL IDs (details)
Commit eaccba18e4e88ca8510aa0f9b0ac41c05e02e698 by Neels Hofmeyr
add osmo_vty.py

To trigger manual handovers, I need a VTY interface. The non-trivial
parts of this are copied from osmo-python-tests osmo_interact_vty.py.

Will be used in the upcoming handover_2G test suite in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: I7c17b143b7c690b8c4105ee7c6272670046fa91d
The file was addedsrc/osmo_gsm_tester/obj/osmo_vty.py
Commit f6f38151ba83c736210d793dbbb4679802e5cf74 by Neels Hofmeyr
add bsc.vty: always keep a BSC VTY connection ready

The BSC's VTY port will be used to trigger manual handover, and to
retrieve a list of active lchans from the BSC, in the upcoming
handover_2G test suite, I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: I06652db04fc9e48748f3c2196334f5352e9cc48a
The file was modifiedsrc/osmo_gsm_tester/obj/osmo_vty.py
The file was modifiedsrc/osmo_gsm_tester/testenv.py
The file was modifiedsrc/osmo_gsm_tester/obj/bsc_osmo.py
Commit 983bf9b289a60b553ac0ccb4e45c281762b5100b by Neels Hofmeyr
bsc: add active lchan matching via vty

Will be used in upcoming handover_2G test suite in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8, which needs to verify that a
handover actually ended up on the expected lchan.

Change-Id: I03df8f3ae2ee47930eee311c7ce104c36dbb3154
The file was modifiedsrc/osmo_gsm_tester/obj/bsc_osmo.py
Commit 347fc0b9650673f925fab702ba62ea1af2a79e16 by Neels Hofmeyr
bts: allow configuring ARFCN and BSIC: add to schema

resources.conf is already allowed to set these, but in order to let
scenarios and suites modify these values, they need to be in the schema.

Change-Id: I8e0583e9208d563c66b0bdc9f25b1b74f92403c2
The file was modifiedsrc/osmo_gsm_tester/obj/bts.py
Commit 49ef9dc1762d54312b5d439b490079a3f0d0f240 by Neels Hofmeyr
log passed tests

In the end of a test suite, do not omit the passed tests. For example,
running handover against N BTS combinations, it was hard to summarize
which BTS models actually succeeded, with only the failures listed.

Besides the "FAIL" listings, now print something like this in the end:

PASS: handover:sysmo+secondbts-trx-b200 (pass: 1)
     pass: handover.py (198.8 sec)
PASS: handover:sysmo+secondbts-trx-umtrx (pass: 1)
     pass: handover.py (192.7 sec)
PASS: handover:trx-b200+secondbts-trx-umtrx (pass: 1)
     pass: handover.py (193.1 sec)

Change-Id: Ib85a5b90e267c2ed2f844691187ecadc8939b1bb
The file was modifiedsrc/osmo_gsm_tester/core/report.py
Commit cdfd54e2686fcc200a8bc67d1ba3aaff0f16c6f1 by Neels Hofmeyr
log reserved resources

On non-debug log level, show something like this at the beginning of
each suite:

03:45:49.439720 tst handover:sysmo+secondbts-trx-b200: RESERVED RESOURCES for handover:
bts
   sysmoBTS 1002
   Ettus B200
ip_address
   10.42.42.2
   10.42.42.3
   10.42.42.4
   10.42.42.5
   10.42.42.6
   10.42.42.7
modem
   sierra_1st
   sierra_2nd

Change-Id: Ic23556eafee654c93d13c5ef405028da09bd51d7
The file was modifiedselftest/suite_test/suite_test.ok
The file was modifiedsrc/osmo_gsm_tester/core/resource.py
The file was modifiedsrc/osmo_gsm_tester/core/suite.py
Commit a41b1616c238e55f7377e50a8b11628c636a6a82 by Neels Hofmeyr
configure fixed ARFCN per bts

Remove ARFCNs as a concept from resource pool, assign a fixed ARFCN to
each BTS and TRX in the resource pools.

Using ARFCNs on specific bands as resources was an idea that is hard to
implement, because specific BTS dictate selection of bands which
influences which ARFCNs can be picked. That means reserving ARFCN
resources is only possible after reserving specific BTS resources, but
the tester is currently not capable of such two-stage resolution.

Writing handover tests, I got the problem that both BTS in a scenario
attempt to use the same ARFCN.

The by far easiest solution is to assign one fixed ARFCN to each BTS and
TRX. If ever needed, a scenario modifier can still configure different
ARFCNs.

(Due to uncertainty about OC2G operation stability, I prefer to leave
OC2G on ARFCN 50, as it happened to end up being configured before this
patch.)

Change-Id: I0a6c60544226f4261f9106013478d6a27fc39f38
The file was modifiedselftest/resource_test/resource_test.ok
The file was modifiedsysmocom/defaults.conf
The file was modifiedsysmocom/resources.conf.rnd
The file was modifiedselftest/resource_test/conf/resources.conf
The file was modifiedsrc/osmo_gsm_tester/obj/bts_nanobts.py
The file was modifiedsrc/osmo_gsm_tester/core/schema.py
The file was modifiedselftest/resource_test/resource_test.py
The file was modifiedsrc/osmo_gsm_tester/obj/bts_oc2g.py
The file was modifiedsysmocom/resources.conf.prod
The file was modifiedselftest/suite_test/resources.conf
Commit bc5a11532f1e7d60260f80e02c0208cff83de95f by Neels Hofmeyr
process: add get_output_mark() and grep_output()

Allow showing log lines matching specific regexes, from a specific start
point of a log.

My use case is to echo the handover related logging after an expected
handover failed, so that the reason is visible already in the console
output of a jenkins run. So far I would need to open the endless bsc log
and look up the matching place in it to get a conclusion about why a
handover failed.

Change-Id: Ib6569f7486e9d961bd79a5f24232e58d053667a1
The file was modifiedsrc/osmo_gsm_tester/core/process.py
Commit e09d0bf72d5724e92094440d3edffef7e75f0e0d by Neels Hofmeyr
fix: line nr in test name in wrong places

test.Test() overrides name() in order to provide source line number
information. However, overriding name() is the wrong place for that, as
name() is also often used for identifying an object - when listing the
tests of a suite, the line number should not appear in the test name.
For example, the line number sometimes ends up in the test results in
jenkins, making 'foo.py' and 'foo.py:23' two distinct report items.

Instead, add a separate function Origin.src() that defaults to name(),
but specific classes can override src() if they wish to provide more
detailed information with the object name.

Override src() in Test, not name().

Use src() in backtraces.

The suite_test.ok shows that the backtracing in the log remains
unchanged, but the place where the test name is printed is corrected:
   I am 'test_suite' / 'hello_world.py:23'
becomes
   I am 'test_suite' / 'hello_world.py'
(Notice that "[LINENR]" in suite_test.ok is a masking of an actual
number, done within the selftest suite)

Change-Id: I0c4698fa2b3db3de777d8b6dcdcee84e433c62b7
The file was modifiedselftest/suite_test/suite_test.ok
The file was modifiedsrc/osmo_gsm_tester/core/test.py
The file was modifiedsrc/osmo_gsm_tester/core/log.py
Commit b57ebdca98b9af6051a8879028462c98e43b6467 by Neels Hofmeyr
typo / newline in selftest log output

...because it otherwise causes confusing selftest log changes in
upcoming I0972c66b9165bd7f2b0b387e0335172849199193

Change-Id: I1eec31c5baba5ef33d0c8441b1e94a00d79af9bf
The file was modifiedselftest/suite_test/suite_test.ok
The file was modifiedselftest/suite_test/suitedirB/suiteC/test_template_overlay.py
The file was modifiedselftest/template_test/template_test.py
The file was modifiedselftest/template_test/template_test.ok
Commit 4ec6203a2fc922c8548c0908f29d79dfeb117bdc by Neels Hofmeyr
do not redirect_stdout in test scripts

In a test, I called print() on a multi-line string and saw the log
showing each line 0.2 seconds apart. redirect.stdout seems to be pretty
inefficient.

Instead, put a print() function into the testenv, to directly call log()
on the strings passed to print().

The initial idea for redirect_stdout was that we could print() in any
deeper functions called from a test script. But we have no such nested
print() anywhere, only in test scripts themselves.

As a result of this, a multi-line print() in test scripts now no longer
puts the log prefix (timestamp, test name...) and suffix (backtrace /
source position) to each single line, but prints the multiline block
between a single log prefix and suffix -- exactly like the log()
function does everywhere else.

I actually briefly implemented adding the log prefix to each separate
line everywhere, but decided that it is not a good idea: in some places
we log config file snippets and other lists, and prepending the log
prefix to each line makes pasting such a snippet from (say) a jenkins
log super cumbersome. And the log prefix (backtrace) attached on each
separate line makes multiline blocks very noisy, unreadable.

Change-Id: I0972c66b9165bd7f2b0b387e0335172849199193
The file was modifiedsrc/osmo_gsm_tester/testenv.py
The file was modifiedselftest/suite_test/suite_test.ok
The file was modifiedsrc/osmo_gsm_tester/core/test.py
The file was modifiedsrc/osmo_gsm_tester/core/log.py
Commit 0fbfb2e6eaf3df7cc4f56eb5efa847aea51f171e by Neels Hofmeyr
log: process.py: fix weird space in stdout/stderr log output

Log output before:

  stdout:
   | line1
  | line2
  | line3

After:

  stdout:
  | line1
  | line2
  | line3

Change-Id: I4202e7dfbb550acaf55a6c6487fc07819ab18b94
The file was modifiedsrc/osmo_gsm_tester/core/process.py
Commit eb01700682672b24f7edd7079bda73979d90d1ff by Neels Hofmeyr
implement test.get_log_output()

Retrieve a test's own logging. The aim is to provide logging belonging
to a given report fragment in the junit XML output, will be used by
upcoming test.report_fragment() feature.

Change-Id: Idfa0a45f3e6a18dd4fe692e81d732c70b5cffb76
The file was modifiedsrc/osmo_gsm_tester/core/log.py
The file was modifiedsrc/osmo_gsm_tester/core/test.py
Commit 42463a6b022a4e004ce6264c04d4661d76239084 by Neels Hofmeyr
add test.report_fragment()

Allow enriching the junit output with arbitrary subtasks within a test.

The current aim is, for handover tests, to not just show that a test
failed, but to show exactly which steps worked and which didn't, e.g.:

handover.py/01_bts0_started PASSED
handover.py/02.1_ms0_attach PASSED
handover.py/02.2_ms1_attach PASSED
handover.py/02.3_subscribed_in_msc PASSED
handover.py/03_call_established PASSED
handover.py/04.1_bts1_started FAILED

In this case it is immediately obvious from looking at the jenkins
results analyzer that bts1 is the cause of the test failure, and it is
visible which parts of the test are flaky, over time.

First user Will be the upcoming handover_2G suite, in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: I4ca9100b6f8db24d1f7e0a09b3b7ba88b8ae3b59
The file was modifiedselftest/suite_test/suite_test.ok.ign
The file was modifiedselftest/suite_test/suite_test.ok
The file was addedselftest/suite_test/suitedirA/test_suite/test_report_fragment.py
The file was modifiedselftest/suite_test/suite_test.py
The file was modifiedsrc/osmo_gsm_tester/core/report.py
The file was modifiedsrc/osmo_gsm_tester/core/test.py
Commit 7f9c124d83bf15b5563fb888df7fcbdc8d18df35 by Neels Hofmeyr
report fragment: strip ansi colors from junit XML

Jenkins does support showing ANSI colors on the web, but apparently not
in the junit results output. Strip ansi colors from report fragment
<system-out> text, to make it less annoying to read those on jenkins.

Change-Id: I656ecc23bbfd3f25bdf012c890e0c998168844d3
The file was modifiedsrc/osmo_gsm_tester/core/report.py
The file was modifiedsrc/osmo_gsm_tester/core/log.py
Commit 5390c29aecfe65e18970efed4252484676b1163f by Neels Hofmeyr
ctrl: set socket timeout

The blocking CTRL socket must have a timeout.

So far we have always been reading from the CTRL socket in a blocking
way. So far we are always lucky that we are indeed getting a response to
receive. Should the CTRL interface ever fail to answer, the tester would
be stuck infinitely.

Change-Id: I20091daf13981dc3d0d894af5a65c23f4db0471d
The file was modifiedsrc/osmo_gsm_tester/obj/osmo_ctrl.py
Commit 69b1f8a64d2e85f62f33c4582ce23e93708c85df by Neels Hofmeyr
ctrl: increment message IDs for CTRL commands

Implicitly use distinct IDs for CTRL commands. This is a prerequisite
for matching GET_REPLY IDs to GET IDs, to correctly handle CTRL
responses without counting on being lucky.

Rather use 'use_id' as var name instead of overloading the 'id' keyword.

Change-Id: I698faa013eb14f88759685d50d925f5d82d80aa0
The file was modifiedsrc/osmo_gsm_tester/obj/osmo_ctrl.py
Commit 2270d05b7f323cf80ea43445d74249ec2d49afbe by Neels Hofmeyr
OsmoCtrl cleanup: get_var(), set_var(), get_trap()

CTRL interface interaction was mostly inherited from the first legacy
implementation of osmo-gsm-tester, and it was a pain to look at from the
start. Now, while I'm close to the topic, I want this to improve:

Properly match a GET_REPLY/SET_REPLY to a sent GET/SET by the message
ID.

Completely drop the do_get() and do_set(), which were not useful for
correct handling of the CTRL request and response messaging. The API to
use by callers is set_var(), get_var()/get_int_var() and get_trap().
These call the internal _sendrecv() (or for TRAP only _recv())
functions. Make it so that tese work both on an already connected
OsmoCtrl, as well as one that needs to establish a (short) connection,
so that both are trivially possible:

    # one CTRL connection stays open
    with OsmoCtrl(...) as ctrl:
  ctrl.get_var('var1')
  ctrl.get_var('var2')
  ctrl.get_var('var3')

and

  # get_var() opens a connection, does the GET and closes again
  OsmoCtrl(...).get_var('var1')

Do away with doubling the instances OsmoCtrl and e.g. OsmoBscCtrl.
Rather make OsmoBscCtrl a child class of OsmoCtrl, which means that we
no longer have bsc.ctrl().ctrl(), just bsc.ctrl().

Have VERB_* constants instead of dup'd strings.

Apply to / simplify all callers of OsmoCtrl.

Some of these changes are similar to recently added OsmoVty.

Change-Id: Id561e5a55d8057a997a8ec9e7fa6f94840194df1
The file was modifiedsrc/osmo_gsm_tester/obj/msc_osmo.py
The file was modifiedsrc/osmo_gsm_tester/obj/nitb_osmo.py
The file was modifiedsrc/osmo_gsm_tester/obj/bsc_osmo.py
The file was modifiedsrc/osmo_gsm_tester/obj/osmo_ctrl.py
Commit f2c26510c95baa7294ea3b2e9d453a8eea76416f by Neels Hofmeyr
keep Ctrl connections open for bsc, msc and nitb objects

The pattern to use 'with' to keep a CTRL connection open adds indents to
every test script that wants to avoid multiple reconnections to the
CTRL. Instead, keeping a single open connection that is cleaned up on
{bsc,msc,nitb} object cleanup ensures that a) the program started up
successfully and opened a CTRL port, b) always has a CTRL open without
having to worry about it and c) keeps test scripts less
complex/indented/crufted.

(These are all current users of the OsmoCtrl API.)

Change-Id: I53fedbe569c5ccbc4b1a17dafe1f8d1bb8200b24
The file was modifiedsrc/osmo_gsm_tester/testenv.py
The file was modifiedsrc/osmo_gsm_tester/obj/bsc_osmo.py
The file was modifiedsrc/osmo_gsm_tester/obj/nitb_osmo.py
The file was modifiedsrc/osmo_gsm_tester/obj/msc_osmo.py
Commit 9b7116a91117b7fcb7237fc78cc7773ba170937f by Neels Hofmeyr
osmo_ctrl.py: add RateCounters

First user will be the upcoming handover_2G/handover.py test in
I0b2671304165a1aaae2b386af46fbd8b098e3bd8.

Change-Id: Id799b3bb81eb9c04d13c26ff611e40363920300e
The file was modifiedsrc/osmo_gsm_tester/obj/osmo_ctrl.py
The file was addedselftest/rate_ctrs_test/rate_ctrs_test.py
The file was addedselftest/rate_ctrs_test/rate_ctrs_test.err
The file was addedselftest/rate_ctrs_test/_prep.py
The file was addedselftest/rate_ctrs_test/rate_ctrs_test.ok
Commit 7d8fdec62e5fec9b0f0593028eb85e9640492799 by Neels Hofmeyr
add handover_2G suite, with handover.py test

Via VTY, handover two lchans of a voice call from bts0 to bts1 and back.

New scenarios/bts1-* allow selecting various types for bts1,
complementing the already existing files for selecting bts0.

Change-Id: I0b2671304165a1aaae2b386af46fbd8b098e3bd8
The file was addedsysmocom/scenarios/bts1-nanobts.conf
The file was addedsysmocom/scenarios/bts1-trx-lms-limenet-micro.conf
The file was modifiedsrc/osmo_gsm_tester/testenv.py
The file was addedsysmocom/suites/handover_2G/handover.py
The file was addedsysmocom/scenarios/bts1-trx-lms.conf
The file was addedsysmocom/scenarios/bts1-sysmo.conf
The file was addedsysmocom/handover-suites.conf
The file was addedsysmocom/scenarios/bts1-octphy.conf
The file was addedsysmocom/scenarios/bts1-trx-sysmocell5000.conf
The file was addedsysmocom/scenarios/bts1-oc2g.conf
The file was addedsysmocom/suites/handover_2G/suite.conf
The file was addedsysmocom/scenarios/bts1-trx-umtrx.conf
The file was addedsysmocom/scenarios/bts1-trx-b200.conf
Commit ade6da87eb73a6b20910aa4592138bd2e3505429 by Neels Hofmeyr
run handover_2G in default-suites.conf

Change-Id: Ic6bb013cc2c395211eeb86fe7a3d3f42671793c7
The file was modifiedsysmocom/default-suites.conf
Commit 6d564748944379c28b9326a9c2058b10f43b618e by Neels Hofmeyr
process output: do not leak ANSI colors past logged stdout/stderr

In jenkins, I still saw incidents of the entire log becoming colored
after a colored stderr snippet was printed to the log. Make absolutely
sure that no unterminated ANSI coloring is leaked.

Change-Id: Ib9ac1eea4a12d6d43ac8614491f016bbe9ca17b1
The file was modifiedsrc/osmo_gsm_tester/core/process.py
Commit a7516db5f3800702ab9859f91b61ac40ae86c10b by Neels Hofmeyr
add option osmo-gsm-tester.py --suites-file my-suites-listing.conf

For example, run with '-S handover-suites.conf' to just do the handover
tests.

Change-Id: If896ac84d02a275deda2f20a93c29312df128ebb
The file was modifiedsrc/osmo-gsm-tester.py
Commit a143686448794ddc58d3932334f04ef723697f42 by Neels Hofmeyr
OsmoCtrl: use one global common counter for CTRL IDs

It is easier to traverse debugging logs if the CTRL request and response
IDs are globally unique across all programs and tests. Before this, we
were using 0 almost everywhere.

(This is not strictly needed for correctness, since each CTRL client has
its own request ID scope; just we open fairly many separate CTRL clients
all the time in our tests.)

Change-Id: I44c51f4fb5beb6cedf98ea0d6684a24c6aa418c7
The file was modifiedsrc/osmo_gsm_tester/obj/osmo_ctrl.py