Osmocom TTCN-3 Test Suites ========================== :author: Harald Welte :copyright: 2018 by Harald Welte (License: CC-BY-SA) :backend: slidy :max-width: 45em == Osmocom TTCN-3 Test Suites * developed in 2017+2018 * compiled using Eclipse TITAN ** uses just a command-line compiler + Makefiles ** no IDE needed at all, don't let _Eclipse_ fool you * containerized in Docker * executed by Jenkins CI == Terminology ATS:: Abstract Test Suite MTC:: Main Test Component PTC:: Parallel Test Component IUT:: Implementation Under Test == Test Suite Philosophy * test one network element (our IUT) * test external behavior (3GPP and non-3GPP) * emulate entire environment from TTCN-3 * don't reuse Osmocom C-code protocol implementations in the tests * test against independent TTCN-3 implementations! == What to test? * successful cases * erroneous cases (no answer, NACK, ...) ** many difficult to reproduce with real phones/devices * load / resource exhaustion * spec compliance * focus on functionality actually relevant to IUT == Why TTCN-3 + TITAN * TTCN-3 specifically designed for telecom protocol testing * TITAN team released many telecom protocols in TTCN-3, such as ** BSSAP, L3 (RR/MM/CC), SMS (CP/RP/TP), SS, M3UA, SCCP, GTP, NS, BSSGP, ... ** shortens our test development cycle ** permits us to test against known working industry implementations == Test suites for Osmocom CNI components * `osmo-bts` * `osmo-bsc` * `osmo-msc` * `osmo-mgw` * `osmo-hlr` * `osmo-sip-connector` * `osmo-sgsn` * `osmo-ggsn` == Test suites in progress * `osmo-pcu` * `osmo-bsc_nat` == BTS_Tests.ttcn * external interfaces ** A-bis side: RSL (emulates BSC-side server) ** Um side: L1CTL to control MS ** PCU side: pcu_socket [graphviz] ---- digraph G { rankdir=LR; { rank=same; BTS, BSC}; BTS [label="IUT\nosmo-bts-trx",shape="box"]; ATS [label="ATS\nBTS_Tests.ttcn"]; BSC [label="osmo-bsc\nOML only"]; BTS -> fake_trx [label="bursts"]; fake_trx -> trxcon [label="bursts"]; trxcon -> ATS [label="bursts"]; BTS -> BSC [label="A-bis OML"]; BTS -> ATS [label="A-bis RSL"]; ATS -> BTS [label="pcu_sock"]; ATS -> BSC [label="VTY"]; ATS -> BTS [label="CTRL"]; } ---- == BSC_Tests.ttcn * external interfaces ** A-bis side: RSL (emulates BTS-side client) ** A-side: BSSAP/SCCP/M3UA (emulates MSC-side) ** MGW side: MGCP (emulates MGW side) [graphviz] ---- digraph G { rankdir=LR; { rank=same; BTS; STP; }; BSC [label="IUT\nosmo-bsc",shape="box"]; ATS [label="ATS\nBSC_Tests.ttcn"]; BTS [label="osmo-bts-omldummy\nOML only"]; BTS -> BSC [label="A-bis OML"]; ATS -> BSC [label="A-bis RSL"]; ATS -> BSC [label="CTRL"]; ATS -> BSC [label="VTY"]; ATS -> STP [label="A BSSAP\nSCCP/M3UA"]; BSC -> STP [label="A BSSAP\nSCCP/M3UA"]; } ---- == MSC_Tests.ttcn * external interfaces ** A: BSSAP/SCCP/M3UA (emulates BSC-side) ** MNCC: MNCC/unix-domain (emulates ext. MNCC side) ** MGW: MGCP (emulates MGW side) ** GSUP (emulates HLR side) [graphviz] ---- digraph G { rankdir=LR; MSC [label="IUT\nosmo-msc",shape="box"]; ATS [label="ATS\nMSC_Tests.ttcn"]; ATS -> MSC [label="MNCC"]; ATS -> MSC [label="SMPP",style="dashed"]; ATS -> MSC [label="CTRL"]; ATS -> MSC [label="VTY"]; MSC -> ATS [label="GSUP"]; ATS -> STP [label="A BSSAP\nSCCP/M3UA"]; MSC -> STP [label="A BSSAP\nSCCP/M3UA"]; } ---- == MGCP_Test.ttcn * external interfaces ** MGCP (emulates call agent) ** RTP (stream source/sink) [graphviz] ---- digraph G { rankdir=LR; MGW [label="IUT\nosmo-mgw",shape="box"]; ATS [label="ATS\nMGCP_Test.ttcn"]; ATS -> MGW [label="RTP"]; ATS -> MGW [label="MGCP"]; MGW -> ATS [label="RTP"]; } ---- == HLR_Tests.ttcn * external interfaces ** GSUP (emulates VLR/SGSN side) ** VTY [graphviz] ---- digraph G { rankdir=LR; HLR [label="IUT\nosmo-hlr",shape="box"]; ATS [label="ATS\nHLR_Tests.ttcn"]; ATS -> HLR [label="GSUP"]; ATS -> HLR [label="VTY"]; } ---- == SIP_Tests.ttcn * external interfaces ** MNCC (emulates MSC side) ** SIP (emulates SIP switch) ** VTY [graphviz] ---- digraph G { rankdir=LR; SIP [label="SIP\nosmo-sip-connector",shape="box"]; ATS [label="ATS\nSIP_Tests.ttcn"]; ATS -> SIP [label="MNCC"]; ATS -> SIP [label="SIP"]; ATS -> SIP [label="VTY"]; } ---- == SGSN_Tests.ttcn * external interfaces ** Gb (emulates PCU side NS/BSSGP + MS) ** GSUP (emulates HLR) ** VTY [graphviz] ---- digraph G { rankdir=LR; SGSN [label="SGSN\nosmo-sgsn",shape="box"]; ATS [label="ATS\nSGSN_Tests.ttcn"]; ATS -> SGSN [label="Gb"]; SGSN-> ATS [label="Gp (GTP)"]; ATS -> SGSN [label="VTY"]; } ---- == GGSN_Tests.ttcn * external interfaces ** Gp: GTP (emulates SGSN) ** Gi: IP (emulates Internet) [graphviz] ---- digraph G { rankdir=LR; GGSN [label="GGSN\nosmo-ggsn",shape="box"]; ATS [label="ATS\nGGSN_Tests.ttcn"]; ATS -> GGSN [label="Gp (GTP)"]; GGSN -> ATS [label="Gi (IP)"]; ATS -> GGSN [label="VTY"]; } ---- == Dockerized Setup * one process per container * packages either ** IUT (e.g. `osmo-bsc`) ** ATS (compiled docker test suite) ** other utility (e.g. `trxcon` or `osmo-bts-omldummy`) * why? ** no need for local ip/network configuration ** standardized / packaged setup on every machine ** run older/newer versions of ATS against older/newer IUT == Jenkins CI Execution . update `docker-playground.git` .. contains `Dockerfile` for ATS + IUT . rebuild IUT container[s] (e.g. `osmo-bts-master`) .. git magic ensures re-build only if `osmo-bts.git` master changed . rebuild ATS container (e.g. `ttcn3-bts-test`) .. git magic ensures re-build only if `osmo-ttcn3-hacks.git` master changed . run `docker-playground/ttcn3-bts-test/jenkins.sh` .. creates docker network .. starts IUT + ATS docker containers .. collects test results == Jenkins CI Reporting * junit-xml generation * store artefacts ** pcap file of every test case ** ATS log file (TTCN-3 testsuite) ** IUT log file[s] (`osmo-*.log`) ** IUT config file[s] (`osmo-*.cfg`) * see https://jenkins.osmocom.org/jenkins/view/TTCN3/ == Further Reading * http://git.osmocom.org/osmo-ttcn3-hacks/ * http://git.osmocom.org/docker-playground/ * http://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Notes == EOF End of File