Changes

jenkins.sh: pin libosmocore version to current version v1.14

Change-Id: I678957099348e067d62a4abd04cbdac822463cf4

personalization: fix SdKey.apply_val() implementation

'securityDomain' elements are decoded to ProfileElementSD instances,
which keep higher level representations of the key data apart from the
decoded[] lists.

So far, apply_val() was dropping binary values in decoded[], which does
not work, because ProfileElementSD._pre_encode() overwrites
self.decoded[] from the higher level representation.

Implement using
- ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify
  an exsiting entry, or
- ProfileElementSD.add_key() to create a new entry.

Before this patch, SdKey parameters seemed to patch PES successfully,
but their modifications did not end up in the encoded DER.

(BTW, this does not fix any other errors that may still be present in
the various SdKey subclasses, patches coming up.)

Related: SYS#6768
Change-Id: I07dfc378705eba1318e9e8652796cbde106c6a52
Jenkins: skip-card-test

saip SmspTpScAddr: safeguard against decoding error

Reading the TS48 V6.0 eSIM_GTP_SAIP2.1A_NoBERTLV profile results in an
exception [1] in SmspTpScAddr. I have a caller that needs to skip
erratic values instead of raising.

The underlying issue, I presume, is that either the data needs
validation before decode_record_bin(), or decode_record_bin() needs
well-defined error handling.

So far I know only of this IndexError, so, as a workaround, catch that.

[1]
  File "/pysim/pySim/esim/saip/personalization.py", line 617, in get_values_from_pes
    ef_smsp_dec = ef_smsp.decode_record_bin(f_smsp.body, 1)
  File "/pysim/pySim/filesystem.py", line 1047, in decode_record_bin
    return parse_construct(self._construct, raw_bin_data)
  File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 550, in parse_construct
    parsed = c.parse(raw_bin_data, total_len=length, **context)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 404, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 416, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 820, in _parse
    return self._decode(obj, context, path)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 268, in _decode
    if r[-1] == 'f':
       ~^^^^
  File "/application/venv/lib/python3.13/site-packages/osmocom/utils.py", line 50, in __getitem__
    return hexstr(super().__getitem__(val))
                  ~~~~~~~~~~~~~~~~~~~^^^^^
IndexError: string index out of range

Change-Id: Ic436e206776b81f24de126e8ee0ae8bf5f3e8d7a
Jenkins: skip-card-test

transitional name mapping

To help existing applications transition to a common naming scheme for
the SdKey classes, offer this intermediate result, where the SdKey
classes' .name are still unchanged as before generating them.

Change-Id: I974cb6c393a2ed2248a6240c2722d157e9235c33
Jenkins: skip-card-test

remove transitional name mapping

This reverts commit I974cb6c393a2ed2248a6240c2722d157e9235c33

Now, finally, all SdKey classes have a unified logical naming scheme.

Change-Id: Ic185af4a903c2211a5361d023af9e7c6fc57ae78
Jenkins: skip-card-test

add test_configurable_parameters.py

Change-Id: Ia55f0d11f8197ca15a948a83a34b3488acf1a0b4
Jenkins: skip-card-test

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463
Jenkins: skip-card-test

use random.SystemRandom as random nr source (/dev/urandom)

/dev/urandom is somewhat better than python's PRNG

Change-Id: I6de38c14ac6dd55bc84d53974192509c18d02bfa
Jenkins: skip-card-test

personalization: add get_typical_input_len() to ConfigurableParameter

The aim is to tell a user interface how wide an input text field should
be chosen to be convenient -- ideally showing the entire value in all
cases, but not too huge for fields that have no sane size limit.

Change-Id: I2568a032167a10517d4d75d8076a747be6e21890
Jenkins: skip-card-test

personalization.ConfigurableParameter: fix BytesIO() input

Change-Id: I0ad160eef9015e76eef10baee7c6b606fe249123
Jenkins: skip-card-test

saip: add numeric_base indicator to ConfigurableParameter

By default, numeric_base = None, to indicate that there are no explicit
limitations on the number space.

For parameters that are definitely decimal, set numeric_base = 10.
For definitely hexadecimal, set numeric_base = 16.

Do the same for ConfigurableParameter as well as ParamSource, so callers
can match them up: if a parameter is numeric_base = 10, then omit
sources that are numeric_base = 16, and vice versa.

Change-Id: Ib0977bbdd9a85167be7eb46dd331fedd529dae01
Jenkins: skip-card-test

esim.saip.personalization: fix TLSPSK keys

Add AES variant of TLSPSK DEK (SCP81 KVN40 key_id=0x02).

Change-Id: I713a008fd26bbfcf437e0f29717b753f058ce76a
Jenkins: skip-card-test

param_source: allow plugging a random implementation (for testing)

Change-Id: Idce2b18af70c17844d6f09f7704efc869456ac39
Jenkins: skip-card-test

ConfigurableParameter: safer val length check

Change-Id: Ibe91722ed1477b00d20ef5e4e7abd9068ff2f3e4
Jenkins: skip-card-test

MncLen

Change-Id: I6c600faeab00ffb072acbe94c9a8b2d1397c07d3
Jenkins: skip-card-test

personalization: make AlgorithmID a new EnumParam

The AlgorithmID has a few preset values, and hardly anyone knows which
is which. So instead of entering '1', '2' or '3', make it work with
prededined values 'Milenage', 'TUAK' and 'usim-test'.

Implement the enum value part abstractly in new EnumParam.

Make AlgorithmID a subclass of EnumParam and define the values as from
pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn

Related: SYS#6768
Change-Id: I71c2ec1b753c66cb577436944634f32792353240
Jenkins: skip-card-test

generate sdkey classes from a list

Change-Id: Ic92ddea6e1fad8167ea75baf78ffc3eb419838c4
Jenkins: skip-card-test

saip SmspTpScAddr.get_values_from_pes: allow empty values

Change-Id: Ibbdd08f96160579238b50699091826883f2e9f5a
Jenkins: skip-card-test

saip: SdKey.__doc__: update SdKey listing

Change-Id: Ib5011b0c7d76b082231744cf09077628dc4e69b7
Jenkins: skip-card-test

test_configurable_parameters.py: add tests for new parameters

For:
SmspTpScAddr
MilenageRotation
MilenageXoringConstants
TuakNrOfKeccak

Change-Id: Iecbea14fe31a9ee08d871dcde7f295d26d7bd001
Jenkins: skip-card-test

personalization: fix EF_SMSP length, alpha_id padding

The efFileSize needs to be updated and the alpha_id needs to be != None.

Change-Id: Ief6e02517f3e96158a2509d763b88aec4bd5a296
Jenkins: skip-card-test

personalization: add int as input type for BinaryParameter

Change-Id: I31d8142cb0847a8b291f8dc614d57cb4734f0190
Jenkins: skip-card-test

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a
Jenkins: skip-card-test

add comment about not updating existing key_usage_qualifier

Change-Id: Ie23ae5fde17be6b37746784bf1601b4d0874397a
Jenkins: skip-card-test

saip/param_source: try to not repeat random values

Change-Id: I4fa743ef5677580f94b9df16a5051d1d178edeb0
Jenkins: skip-card-test

sdkeys kv40 aes

Change-Id: If5b53c840ebd1f224f9bb4706a602b415194f47b
Jenkins: skip-card-test

SdKey KVN4X ID02: set key_usage_qual=0x48

Related: SYS#7865
Change-Id: Idc5d33a4a003801f60c95fff6931706a9aeb6692
Jenkins: skip-card-test

use secrets.SystemRandom as secure random nr source

secrets.SystemRandom is defined as the most secure random source
available on the given operating system.

Change-Id: I8049cd1292674b3ced82b0926569128535af6efe
Jenkins: skip-card-test

add test_param_src.py

Change-Id: I03087b84030fddae98b965e0075d44e04ec6ba5c
Jenkins: skip-card-test

personalization audit: optionally audit all (unknown) SD keys

By a flag, allow to audit also all Security Domain KVN that we have
*not* created ConfigurableParameter subclasses for.

For example, SCP80 has reserved kvn 0x01..0x0f, but we offer only
Scp80Kvn01, Scp80Kvn02, Scp80Kvn03. So we would not show kvn
0x03..0x0f in an audit.

This patch includes audits of all SD key kvn there may be in the UPP.
This will help to spot SD keys that may already be present in a UPP
template, with unexpected / unusual kvn.

Change-Id: Icaf6f7b589f117868633c0968a99f2f0252cf612
Jenkins: skip-card-test

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32
Jenkins: skip-card-test

UppAudit: better indicate exception cause

Change-Id: I4d986b89a473a5b12ed56b4710263b034876a33e
Jenkins: skip-card-test

comment in uicc.py on Security Domain Keys: add SCP81

Change-Id: Ib0205880f58e78c07688b4637abd5f67ea0570d1
Jenkins: skip-card-test

ConfigurableParameter: do not magically overwrite the 'name' attribute

Change-Id: I6f631444c6addeb7ccc5f6c55b9be3dc83409169
Jenkins: skip-card-test

personalization: implement UppAudit and BatchAudit

Change-Id: Iaab336ca91b483ecdddd5c6c8e08dc475dc6bd0a
Jenkins: skip-card-test

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a
Jenkins: skip-card-test

jobs/gerrit-verifications.yml: pysim: allow skipping card tests

I've already implemented this logic in 6a3b8a0, however by mistake
I added it to master-builds.yml instead of gerrit-verifications.yml.
This is wrong because we specifically need this feature for Jenkins
verification, not for the master builds.

For daily master builds, on the other hand, we specifically want to
execute all existing job types, so revert my recent changes from
master-builds.yml, but keep the new job type.

Change-Id: I986d88545f64e13cd571ba9ff56bc924822e39a0
Fixes: 6a3b8a0 ("jobs/master-builds.yml: pysim: allow skipping card tests")

debian-trixie-build: add tshark

This is needed for running pysim unit tests on generic nodes.
As of now we're executing them on the 'simtester' node, which cannot
run more than one job at once, making build verification for large
pysim patchsets extremely slow.

Change-Id: Idbe934e5cc29b06fd1b13b85893de5856eca836e
Related: osmo-ci.git I986d88545f64e13cd571ba9ff56bc924822e39a0

saip SmspTpScAddr: safeguard against decoding error

Reading the TS48 V6.0 eSIM_GTP_SAIP2.1A_NoBERTLV profile results in an
exception [1] in SmspTpScAddr. I have a caller that needs to skip
erratic values instead of raising.

The underlying issue, I presume, is that either the data needs
validation before decode_record_bin(), or decode_record_bin() needs
well-defined error handling.

So far I know only of this IndexError, so, as a workaround, catch that.

[1]
  File "/pysim/pySim/esim/saip/personalization.py", line 617, in get_values_from_pes
    ef_smsp_dec = ef_smsp.decode_record_bin(f_smsp.body, 1)
  File "/pysim/pySim/filesystem.py", line 1047, in decode_record_bin
    return parse_construct(self._construct, raw_bin_data)
  File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 550, in parse_construct
    parsed = c.parse(raw_bin_data, total_len=length, **context)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 404, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 416, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 820, in _parse
    return self._decode(obj, context, path)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 268, in _decode
    if r[-1] == 'f':
       ~^^^^
  File "/application/venv/lib/python3.13/site-packages/osmocom/utils.py", line 50, in __getitem__
    return hexstr(super().__getitem__(val))
                  ~~~~~~~~~~~~~~~~~~~^^^^^
IndexError: string index out of range

Change-Id: Ic436e206776b81f24de126e8ee0ae8bf5f3e8d7a
Jenkins: skip-card-test

personalization.ConfigurableParameter: fix BytesIO() input

Change-Id: I0ad160eef9015e76eef10baee7c6b606fe249123
Jenkins: skip-card-test

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a
Jenkins: skip-card-test

saip: SdKey.__doc__: update SdKey listing

Change-Id: Ib5011b0c7d76b082231744cf09077628dc4e69b7
Jenkins: skip-card-test

esim.saip.personalization: fix TLSPSK keys

Add AES variant of TLSPSK DEK (SCP81 KVN40 key_id=0x02).

Change-Id: I713a008fd26bbfcf437e0f29717b753f058ce76a
Jenkins: skip-card-test

MncLen

Change-Id: I6c600faeab00ffb072acbe94c9a8b2d1397c07d3
Jenkins: skip-card-test

saip/param_source: try to not repeat random values

Change-Id: I4fa743ef5677580f94b9df16a5051d1d178edeb0
Jenkins: skip-card-test

add comment about not updating existing key_usage_qualifier

Change-Id: Ie23ae5fde17be6b37746784bf1601b4d0874397a
Jenkins: skip-card-test

test_configurable_parameters.py: add tests for new parameters

For:
SmspTpScAddr
MilenageRotation
MilenageXoringConstants
TuakNrOfKeccak

Change-Id: Iecbea14fe31a9ee08d871dcde7f295d26d7bd001
Jenkins: skip-card-test

saip SmspTpScAddr.get_values_from_pes: allow empty values

Change-Id: Ibbdd08f96160579238b50699091826883f2e9f5a
Jenkins: skip-card-test

use secrets.SystemRandom as secure random nr source

secrets.SystemRandom is defined as the most secure random source
available on the given operating system.

Change-Id: I8049cd1292674b3ced82b0926569128535af6efe
Jenkins: skip-card-test

UppAudit: better indicate exception cause

Change-Id: I4d986b89a473a5b12ed56b4710263b034876a33e
Jenkins: skip-card-test

personalization: add get_typical_input_len() to ConfigurableParameter

The aim is to tell a user interface how wide an input text field should
be chosen to be convenient -- ideally showing the entire value in all
cases, but not too huge for fields that have no sane size limit.

Change-Id: I2568a032167a10517d4d75d8076a747be6e21890
Jenkins: skip-card-test

use random.SystemRandom as random nr source (/dev/urandom)

/dev/urandom is somewhat better than python's PRNG

Change-Id: I6de38c14ac6dd55bc84d53974192509c18d02bfa
Jenkins: skip-card-test

personalization: fix SdKey.apply_val() implementation

'securityDomain' elements are decoded to ProfileElementSD instances,
which keep higher level representations of the key data apart from the
decoded[] lists.

So far, apply_val() was dropping binary values in decoded[], which does
not work, because ProfileElementSD._pre_encode() overwrites
self.decoded[] from the higher level representation.

Implement using
- ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify
  an exsiting entry, or
- ProfileElementSD.add_key() to create a new entry.

Before this patch, SdKey parameters seemed to patch PES successfully,
but their modifications did not end up in the encoded DER.

(BTW, this does not fix any other errors that may still be present in
the various SdKey subclasses, patches coming up.)

Related: SYS#6768
Change-Id: I07dfc378705eba1318e9e8652796cbde106c6a52
Jenkins: skip-card-test

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32
Jenkins: skip-card-test

sdkeys kv40 aes

Change-Id: If5b53c840ebd1f224f9bb4706a602b415194f47b
Jenkins: skip-card-test

SdKey KVN4X ID02: set key_usage_qual=0x48

Related: SYS#7865
Change-Id: Idc5d33a4a003801f60c95fff6931706a9aeb6692
Jenkins: skip-card-test

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463
Jenkins: skip-card-test

personalization: implement UppAudit and BatchAudit

Change-Id: Iaab336ca91b483ecdddd5c6c8e08dc475dc6bd0a
Jenkins: skip-card-test

transitional name mapping

To help existing applications transition to a common naming scheme for
the SdKey classes, offer this intermediate result, where the SdKey
classes' .name are still unchanged as before generating them.

Change-Id: I974cb6c393a2ed2248a6240c2722d157e9235c33
Jenkins: skip-card-test

comment in uicc.py on Security Domain Keys: add SCP81

Change-Id: Ib0205880f58e78c07688b4637abd5f67ea0570d1
Jenkins: skip-card-test

add test_configurable_parameters.py

Change-Id: Ia55f0d11f8197ca15a948a83a34b3488acf1a0b4
Jenkins: skip-card-test

remove transitional name mapping

This reverts commit I974cb6c393a2ed2248a6240c2722d157e9235c33

Now, finally, all SdKey classes have a unified logical naming scheme.

Change-Id: Ic185af4a903c2211a5361d023af9e7c6fc57ae78
Jenkins: skip-card-test

personalization: fix EF_SMSP length, alpha_id padding

The efFileSize needs to be updated and the alpha_id needs to be != None.

Change-Id: Ief6e02517f3e96158a2509d763b88aec4bd5a296
Jenkins: skip-card-test

personalization: add int as input type for BinaryParameter

Change-Id: I31d8142cb0847a8b291f8dc614d57cb4734f0190
Jenkins: skip-card-test

generate sdkey classes from a list

Change-Id: Ic92ddea6e1fad8167ea75baf78ffc3eb419838c4
Jenkins: skip-card-test

saip: SmspTpScAddr: fix get_values_from_pes

Change-Id: I2010305340499c907bb7618c04c61e194db34814
Jenkins: skip-card-test

personalization audit: optionally audit all (unknown) SD keys

By a flag, allow to audit also all Security Domain KVN that we have
*not* created ConfigurableParameter subclasses for.

For example, SCP80 has reserved kvn 0x01..0x0f, but we offer only
Scp80Kvn01, Scp80Kvn02, Scp80Kvn03. So we would not show kvn
0x03..0x0f in an audit.

This patch includes audits of all SD key kvn there may be in the UPP.
This will help to spot SD keys that may already be present in a UPP
template, with unexpected / unusual kvn.

Change-Id: Icaf6f7b589f117868633c0968a99f2f0252cf612
Jenkins: skip-card-test

add test_param_src.py

Change-Id: I03087b84030fddae98b965e0075d44e04ec6ba5c
Jenkins: skip-card-test

personalization: make AlgorithmID a new EnumParam

The AlgorithmID has a few preset values, and hardly anyone knows which
is which. So instead of entering '1', '2' or '3', make it work with
prededined values 'Milenage', 'TUAK' and 'usim-test'.

Implement the enum value part abstractly in new EnumParam.

Make AlgorithmID a subclass of EnumParam and define the values as from
pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn

Related: SYS#6768
Change-Id: I71c2ec1b753c66cb577436944634f32792353240
Jenkins: skip-card-test

personalization: allow reading back multiple values from PES

Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2
Jenkins: skip-card-test

ConfigurableParameter: do not magically overwrite the 'name' attribute

Change-Id: I6f631444c6addeb7ccc5f6c55b9be3dc83409169
Jenkins: skip-card-test

ConfigurableParameter: safer val length check

Change-Id: Ibe91722ed1477b00d20ef5e4e7abd9068ff2f3e4
Jenkins: skip-card-test

saip: add numeric_base indicator to ConfigurableParameter

By default, numeric_base = None, to indicate that there are no explicit
limitations on the number space.

For parameters that are definitely decimal, set numeric_base = 10.
For definitely hexadecimal, set numeric_base = 16.

Do the same for ConfigurableParameter as well as ParamSource, so callers
can match them up: if a parameter is numeric_base = 10, then omit
sources that are numeric_base = 16, and vice versa.

Change-Id: Ib0977bbdd9a85167be7eb46dd331fedd529dae01
Jenkins: skip-card-test

param_source: allow plugging a random implementation (for testing)

Change-Id: Idce2b18af70c17844d6f09f7704efc869456ac39
Jenkins: skip-card-test

jobs/gerrit-verifications.yml: pysim: allow skipping card tests

I've already implemented this logic in 6a3b8a0, however by mistake
I added it to master-builds.yml instead of gerrit-verifications.yml.
This is wrong because we specifically need this feature for Jenkins
verification, not for the master builds.

For daily master builds, on the other hand, we specifically want to
execute all existing job types, so revert my recent changes from
master-builds.yml, but keep the new job type.

Change-Id: I986d88545f64e13cd571ba9ff56bc924822e39a0
Fixes: 6a3b8a0 ("jobs/master-builds.yml: pysim: allow skipping card tests")

jobs/master-builds.yml: add missing job type 'distcheck'

We do have this job type in gerrit-verifications.yml, so it's executed
for each patch submitted to Gerrit.  However, we are not executing
'distcheck' during the daily master builds - fix this.

Change-Id: I4898ba712a29a6afc34bc22adec6a3131ce8c794
Fixes: 31fd5cf0 ("gerrit-verifications: execute 'distcheck' job for pysim")

jobs/master-builds.yml: fix typo: crad-test -> card-test

Change-Id: I0686af6c026e298c86d3a64cc161e5c84ec7dc57
Fixes: ee1ef65 ("jobs/master-builds.yml: pysim: separate JOB_TYPE for card tests")

contrib/jenkins.sh: separate JOB_TYPE for card tests

A separate job gives us a possibility to skip tests requiring physical
cards for specific commits that do not touch the core logic.  See the
related commits in osmo-ci.git.

Change-Id: If76d812ee43b7eb3b57fdc660c60bf31fbff5b16
Related: osmo-ci.git Ia48d1b468f65d7c2e6b4128eeac36d0f3d03c45e
Related: osmo-ci.git I986d88545f64e13cd571ba9ff56bc924822e39a0

contrib/jenkins.sh: add setup_venv()

Reduece code duplication by factoring out virtualenv setup and
activation into a shell function.

Change-Id: Ibb193d12d5502c78104ef53badc6037f08e92df1

contrib/jenkins.sh: separate JOB_TYPE for card tests

A separate job gives us a possibility to skip tests requiring physical
cards for specific commits that do not touch the core logic.  See the
related commits in osmo-ci.git.

Change-Id: If76d812ee43b7eb3b57fdc660c60bf31fbff5b16
Related: osmo-ci.git Ia48d1b468f65d7c2e6b4128eeac36d0f3d03c45e
Related: osmo-ci.git I986d88545f64e13cd571ba9ff56bc924822e39a0

contrib/jenkins.sh: add setup_venv()

Reduece code duplication by factoring out virtualenv setup and
activation into a shell function.

Change-Id: Ibb193d12d5502c78104ef53badc6037f08e92df1

contrib/jenkins.sh: separate JOB_TYPE for card tests

A separate job gives us a possibility to skip tests requiring physical
cards for specific commits that do not touch the core logic.  See the
related commits in osmo-ci.git.

Change-Id: If76d812ee43b7eb3b57fdc660c60bf31fbff5b16
Related: osmo-ci.git Ia48d1b468f65d7c2e6b4128eeac36d0f3d03c45e
Related: osmo-ci.git I986d88545f64e13cd571ba9ff56bc924822e39a0

contrib/jenkins.sh: add setup_venv()

Reduece code duplication by factoring out virtualenv setup and
activation into a shell function.

Change-Id: Ibb193d12d5502c78104ef53badc6037f08e92df1

contrib/jenkins.sh: separate JOB_TYPE for card tests

A separate job gives us a possibility to skip tests requiring physical
cards for specific commits that do not touch the core logic.  See the
related commits in osmo-ci.git.

Change-Id: If76d812ee43b7eb3b57fdc660c60bf31fbff5b16
Related: osmo-ci.git Ia48d1b468f65d7c2e6b4128eeac36d0f3d03c45e
Related: osmo-ci.git I986d88545f64e13cd571ba9ff56bc924822e39a0

testenv: Allow adding extra parameters to podman

Allows to add additional command line parameters to podman, if defined
int the testsuite section of testenv.cfg:

[testsuite]
...
podman_extra=--volume /dev/bus/usb:/dev/bus/usb

Change-Id: I6a2be0d60efcd70ea2081e0314927885085bb5e0

OBS: build_srcpkg.Dockerfile: add dh-runit

Add dh-runit, so we can build source packages of newer versions of
pcsc-lite.

Related: https://salsa.debian.org/debian/pcsc-lite/-/commit/c7c40df72dc3c754341501e34aa096fd72a61cef
Change-Id: I8d852cebb9decb3c881fd3ff9b530213bade0216

pySim-prog/cards: fix programming of EF.SMSP

The legacy code found in legacy/cards.py does not use the modern
construct based encoder (pySim-read uses it). The card classes either
use their own implementation of update_smsp or use the generic method
provided by the SimCard class. The latter one is true for FairwavesSIM
and WavemobileSim.

Unfortunately the implementation found in the SimCard is wrong. It
adds padding at the end of the file instead of the beginning. This
completely messes up the contents of EF.SMSP for the cards using this
method. To fix this, let's use the leftpad feature provided by
the update_record. This will ensure a correct alignment of the file
contents.

Related: SYS#7765
Change-Id: Ie112418f1f1461762d61365d3863181ca6be7245

ts_51_011/EF.SMSP: fix handling of 'alpha_id' field

The field 'alpha_id' is technically not an optional field, even though
the specification describes it as optional. Once the card manufacturer
decides that the field should be present, it must be always present and
vice versa.

(see code comment for a more detailed description)

Related: SYS#7765
Change-Id: I0ec99b2648b22c56f9145345e4cd8776f9217701

Fix: Prevent bankd from exiting upon SIGPIPE

IF a client disconnects from a worker thread while the worker is
responding to a to a request from that client, the socket is closed and
SIGPIPE is sent to the worker, causing the application to exit.

To prevent this, all workers have a dummy handler to prevent this. Now
the worker can handle the closing of the socket.

Change-Id: I13d6e9da48d12f93c00bd2d021789bde71aca7cf

Fix: Prevent race conditions when accessing slotmap in bankd

All threads (main and workers) can use slotmap_by_bank() and
slotmap_by_client() to get a mapping entry from the slot map. While
working with this entry (reading values), it could be deleted by a
different thread.

For example, if a client connects to a worker, the worker will check if
there is an existing mapping between a reader and this client. If it
exists, slotmap_by_client() will return a pointer. If the mapping is
deleted at this time by the server, the worker uses a pointer to a
mapping entry that has just been freed.

Change-Id: I3464726f37beb7c47b4e1f00c018ffa4f3948906

pySim-prog/cards: fix programming of EF.SMSP

The legacy code found in legacy/cards.py does not use the modern
construct based encoder (pySim-read uses it). The card classes either
use their own implementation of update_smsp or use the generic method
provided by the SimCard class. The latter one is true for FairwavesSIM
and WavemobileSim.

Unfortunately the implementation found in the SimCard is wrong. It
adds padding at the end of the file instead of the beginning. This
completely messes up the contents of EF.SMSP for the cards using this
method. To fix this, let's use the leftpad feature provided by
the update_record. This will ensure a correct alignment of the file
contents.

Related: SYS#7765
Change-Id: Ie112418f1f1461762d61365d3863181ca6be7245

ts_51_011/EF.SMSP: fix handling of 'alpha_id' field

The field 'alpha_id' is technically not an optional field, even though
the specification describes it as optional. Once the card manufacturer
decides that the field should be present, it must be always present and
vice versa.

(see code comment for a more detailed description)

Related: SYS#7765
Change-Id: I0ec99b2648b22c56f9145345e4cd8776f9217701

ts_51_011/EF.SMSP: fix handling of 'alpha_id' field

The field 'alpha_id' is technically not an optional field, even though
the specification describes it as optional. Once the card manufacturer
decides that the field should be present, it must be always present and
vice versa.

(see code comment for a more detailed description)

Related: SYS#7765
Change-Id: I0ec99b2648b22c56f9145345e4cd8776f9217701

pySim-prog/cards: fix programming of EF.SMSP

The legacy code found in legacy/cards.py does not use the modern
construct based encoder (pySim-read uses it). The card classes either
use their own implementation of update_smsp or use the generic method
provided by the SimCard class. The latter one is true for FairwavesSIM
and WavemobileSim.

Unfortunately the implementation found in the SimCard is wrong. It
adds padding at the end of the file instead of the beginning. This
completely messes up the contents of EF.SMSP for the cards using this
method. To fix this, let's use the leftpad feature provided by
the update_record. This will ensure a correct alignment of the file
contents.

Related: SYS#7765
Change-Id: Ie112418f1f1461762d61365d3863181ca6be7245

testenv: Allow adding extra parameters to podman

Allows to add additional command line parameters to podman, if defined
int the testsuite section of testenv.cfg:

[testsuite]
...
podman_extra=--volume /dev/bus/usb:/dev/bus/usb

Change-Id: I6a2be0d60efcd70ea2081e0314927885085bb5e0

testenv: Allow adding extra parameters to podman

Allows to add additional command line parameters to podman, if defined
int the testsuite section of testenv.cfg:

[testsuite]
...
podman_extra=--volume /dev/bus/usb:/dev/bus/usb

Change-Id: I6a2be0d60efcd70ea2081e0314927885085bb5e0

Fix: Prevent bankd from exiting upon SIGPIPE

IF a client disconnects from a worker thread while the worker is
responding to a to a request from that client, the socket is closed and
SIGPIPE is sent to the worker, causing the application to exit.

To prevent this, all workers have a dummy handler to prevent this. Now
the worker can handle the closing of the socket.

Change-Id: I13d6e9da48d12f93c00bd2d021789bde71aca7cf

Fix: Prevent race conditions when accessing slotmap in bankd

All threads (main and workers) can use slotmap_by_bank() and
slotmap_by_client() to get a mapping entry from the slot map. While
working with this entry (reading values), it could be deleted by a
different thread.

For example, if a client connects to a worker, the worker will check if
there is an existing mapping between a reader and this client. If it
exists, slotmap_by_client() will return a pointer. If the mapping is
deleted at this time by the server, the worker uses a pointer to a
mapping entry that has just been freed.

Change-Id: I3464726f37beb7c47b4e1f00c018ffa4f3948906

jobs/master-builds.yml: pysim: allow skipping card tests

Skip tests requiring physical cards ("card-test") for commits
having a special marker in the COMMIT_MSG:

  Jenkins: skip-card-tests

This speeds up the build verification significantly, and may be
useful for patches that do not modify the core logic (such as
documentation improvements and misc fixes).

Change-Id: I591717a2e2659c02447087dec23aa279cfd55551

tests/sysmo-octsim/run-tests: tempfile -> mktemp

Fix for:
  WARNING: tempfile is deprecated; consider using mktemp instead.

Change-Id: I785a81173aae7202e4c01ead0ced11142d2e1794

tests/sysmo-octsim/run-tests: tempfile -> mktemp

Fix for:
  WARNING: tempfile is deprecated; consider using mktemp instead.

Change-Id: I785a81173aae7202e4c01ead0ced11142d2e1794

ts_51_011/EF.SMSP: fix handling of 'alpha_id' field

The field 'alpha_id' is technically not an optional field, even though
the specification describes it as optional. Once the card manufacturer
decides that the field should be present, it must be always present and
vice versa.

(see code comment for a more detailed description)

Related: SYS#7765
Change-Id: I0ec99b2648b22c56f9145345e4cd8776f9217701

ts_51_011/EF.SMSP: fix handling of 'alpha_id' field

The field 'alpha_id' is technically not an optional field, even though
the specification describes it as optional. Once the card manufacturer
decides that the field should be present, it must be always present and
vice versa.

(see code comment for a more detailed description)

Related: SYS#7765
Change-Id: I0ec99b2648b22c56f9145345e4cd8776f9217701

ts_51_011/EF.SMSP: fix handling of 'alpha_id' field

The field 'alpha_id' is technically not an optional field, even though
the specification describes it as optional. Once the card manufacturer
decides that the field should be present, it must be always present and
vice versa.

(see code comment for a more detailed description)

Related: SYS#7765
Change-Id: I0ec99b2648b22c56f9145345e4cd8776f9217701

jobs/master-builds.yml: pysim: allow skipping card tests

Skip tests requiring physical cards ("card-test") for commits
having a special marker in the COMMIT_MSG:

  Jenkins: skip-card-tests

This speeds up the build verification significantly, and may be
useful for patches that do not modify the core logic (such as
documentation improvements and misc fixes).

Change-Id: I591717a2e2659c02447087dec23aa279cfd55551

jobs/master-builds.yml: pysim: separate JOB_TYPE for card tests

Change-Id: Ia48d1b468f65d7c2e6b4128eeac36d0f3d03c45e
Depends: pysim.git If76d812ee43b7eb3b57fdc660c60bf31fbff5b16

jobs/master-builds.yml: pysim: allow skipping card tests

Skip tests requiring physical cards ("card-test") for commits
having a special marker in the COMMIT_MSG:

  Jenkins: skip-card-tests

This speeds up the build verification significantly, and may be
useful for patches that do not modify the core logic (such as
documentation improvements and misc fixes).

Change-Id: I591717a2e2659c02447087dec23aa279cfd55551

contrib/jenkins.sh: separate JOB_TYPE for card tests

A separate job gives us a possibility to skip tests requiring physical
cards for specific commits that do not touch the core logic.  See the
related commits in osmo-ci.git.

Change-Id: If76d812ee43b7eb3b57fdc660c60bf31fbff5b16
Related: osmo-ci.git Ia48d1b468f65d7c2e6b4128eeac36d0f3d03c45e
Related: osmo-ci.git I591717a2e2659c02447087dec23aa279cfd55551

tests: pySim-smpp2sim_test.sh: fix copy-pasted comment

Change-Id: I8167c6a3251bb6755810c96075010e920ceee8ac
Jenkins: skip-card-tests

tests: pySim-smpp2sim_test.sh: fix copy-pasted comment

Change-Id: I8167c6a3251bb6755810c96075010e920ceee8ac
Jenkins: skip-card-tests

contrib/jenkins.sh: add support for "Jenkins: skip-card-tests" marker

Add a have_jenkins_marker() helper that checks the commit message
for a given value on the "Jenkins:" footer line, for instance:

  Jenkins: skip-card-tests

When this marker is present, the real-card integration tests
(pySim-prog_test and pySim-shell_test) are skipped, allowing commits
that don't affect card I/O to bypass long card-dependent test runs.

Change-Id: If76d812ee43b7eb3b57fdc660c60bf31fbff5b16
Jenkins: skip-card-tests

tests: pySim-smpp2sim_test.sh: fix copy-pasted comment

Change-Id: I8167c6a3251bb6755810c96075010e920ceee8ac
Jenkins: skip-card-tests

contrib/jenkins.sh: add setup_venv()

Change-Id: Ibb193d12d5502c78104ef53badc6037f08e92df1

contrib/jenkins.sh: add support for "Jenkins: skip-card-tests" marker

Add a have_jenkins_marker() helper that checks the commit message
for a given value on the "Jenkins:" footer line, for instance:

  Jenkins: skip-card-tests

When this marker is present, the real-card integration tests
(pySim-prog_test and pySim-shell_test) are skipped, allowing commits
that don't affect card I/O to bypass long card-dependent test runs.

Change-Id: If76d812ee43b7eb3b57fdc660c60bf31fbff5b16
Jenkins: skip-card-tests

build: detect pthread_setname_np and sched_* APIs, guard usage

Add configure checks for pthread_setname_np and sched_* functions.
Wrap their usage with HAVE_* guards to avoid build failures on
platforms where they are unavailable.

Return -ENOTSUP for unsupported scheduler operations.

Improves portability across non-Linux and restricted environments
(e.g. WebAssembly).

Change-Id: Ic3f1492544439d88c5e6683330c6696de1c2f85b

convolve: avoid function pointer casts by adding wrapper functions

Introduce _base_convolve_*_void() wrappers matching the expected
function pointer signatures and use them instead of casting
_base_convolve_* functions to (void *).

This removes unsafe function pointer casts and improves
type safety, which is required for stricter toolchains and
non-native targets (e.g. WebAssembly)

Change-Id: Idecb118be285eb3e4691d1761d0d8fa24fd80c75

device: add WebSDR radio backend

Introduce a new optional WebSDR device backend, enabled via
--with-websdr.

Add WebSDRDevice implementation and build integration, providing
a RadioDevice interface backed by callback hooks for control and
sample I/O.

Intended for Web-based deployments where osmo-trx interacts with
SDR hardware via a WebSDR/WebUSB frontend.

Change-Id: Ie459cbd70388dd8ff5b89221d30770bab0bd9014

transceiver: add optional WebSDR device support

- Add support for WebSDR devices enabled via the --with-websdr
configure flag.
- add libosmo-trx-websdr library and pkg-config file
- update .gitignore for wasm and pkg-config artifacts

Change-Id: Ia0d340c323c2eea28fbe82601ba0af7cfbd68f6d

abis: add optional WebSDR E1 input driver

Add a new optional E1 input backend for WebSDR, enabled via
--with-websdr.

Introduce input/websdr.c and public callback API to replace
socket-based IPA transport with a callback-driven interface
for OML/RSL and OSMUX.

Intended for WebAssembly builds where Osmocom components run
in a browser and communicate with SDR via WebUSB.

Change-Id: Ib804dc8eb67a91678603f54ea0beccd5c61e4a80

abis: add optional WebSDR E1 input driver

Add a new optional E1 input backend for WebSDR, enabled via
--with-websdr.

Introduce input/websdr.c and public callback API to replace
socket-based IPA transport with a callback-driven interface
for OML/RSL and OSMUX.

Intended for WebAssembly builds where Osmocom components run
in a browser and communicate with SDR via WebUSB.

Change-Id: Ib804dc8eb67a91678603f54ea0beccd5c61e4a80

vty: make CPU scheduling optional based on platform support

Add configure checks for sched_* APIs and guard their usage
in cpu_sched_vty with HAVE_SCHED_* macros.

If scheduler functions are not available, return -ENOTSUP
instead of failing at build time.

Also change osmo_cpu_sched_vty_init() to return int.

This improves portability on platforms lacking full sched
support (e.g. non-Linux or restricted environments).

Change-Id: Ic5b7e39fac16531d370cb81f769ba87fef18cb81

abis: add optional WebSDR E1 input driver

Add a new optional E1 input backend for WebSDR, enabled via
--with-websdr.

Introduce input/websdr.c and public callback API to replace
socket-based IPA transport with a callback-driven interface
for OML/RSL and OSMUX.

Intended for WebAssembly builds where Osmocom components run
in a browser and communicate with SDR via WebUSB.

Change-Id: Ib804dc8eb67a91678603f54ea0beccd5c61e4a80

OBS: build_srcpkg.Dockerfile: add dh-cmake

Add dh-cmake, so we can build source packages of programs using cmake,
such as kamailio.

Change-Id: I8b5debce4f2fc0c41c69f861822394e139ac5dfc

timers: fix ATR timeout

Default WWT to 1.5s.

Both atr and card uart timeouts were combined in
7ae9a65c7fadf3d9d89370761c7352cbcac83a00 but the timeout
was accidentally left as 0 which was the original ATR fsm
state change timeout. The current code uses 0 as
"no timeout", so that killed the timeouts during the ATR.

Related: SYS#7991
Change-Id: I94d6641a10bf6d1df8640727e753da8171267339

jobs/gerrit: fix artifacts logic

The previous logic failed when there are not artifacts for archiving,
for multiple reasons. Fix all of them:

* "allow-empty" needs to be set, so jenkins doesn't fail the job when it
  can't find any artifacts to archive.
* The "artifacts" string must not be empty even if "allow-empty" is set,
  as otherwise it assumes a configuration error.
* The jenkins jobs fails in the archiving step if the workspace dir
  doesn't exist.

Fixes: 4c392b9e ("jobs/gerrit: archive osmo-ccid-firmware artifacts")
Change-Id: If7a051d7be2beb617c2eda4fd1a669565c2c5b55

tests/fetch_image: add SKIP_FETCH_IMAGE env var

Allow skipping the image fetch logic to let the gerrit job copy the
image from another job instead.

Related: osmo-ci I736b2b194813e6edff0197145fe08168c3e66903
Change-Id: I9445d6d5e4723bf6a87d4bffcdc638560471328b

Fix: Remove slot mapping at bankd when client disconnects

If a client disconnects before removing the slot mapping, the slotmap
entry will not be removed automatically. A subsequent client connection
for the same mapping will then trigger the the error: “ignoring
identical slotmap.”

In a setup with multiple readers, there are multiple bankd worker
threads. If a client re-connects, but if its connection is accepted by
a different thread, subsequent reader access or unmapping will fail.
There will be no mapping at the new thread.

The approach of this fix is to remove the slot mapping, if the client
connection to the worker thread terminates.

Change-Id: I2fd03490e2506c55104309a0ef952389119023b8

Fix access pcscd in various RemsimBankd_Tests

Access to pcscd was denied, so that bankd never conneced to the reader.
This caused TC_createMapping_exchangeTPDU to fail, because it requires
connection to virtual smart card for this test

Change-Id: Ibae1c5167daa0157058bc40ce1857c1e6bb9da31

Fix: Remove slot mapping at bankd when client disconnects

If a client disconnects before removing the slot mapping, the slotmap
entry will not be removed automatically. A subsequent client connection
for the same mapping will then trigger the the error: “ignoring
identical slotmap.”

In a setup with multiple readers, there are multiple bankd worker
threads. If a client re-connects, but if its connection is accepted by
a different thread, subsequent reader access or unmapping will fail.
There will be no mapping at the new thread.

The approach of this fix is to remove the slot mapping, if the client
connection to the worker thread terminates.

Change-Id: I2fd03490e2506c55104309a0ef952389119023b8

Fix access pcscd in various RemsimBankd_Tests

Access to pcscd was denied, so that bankd never conneced to the reader.
This caused TC_createMapping_exchangeTPDU to fail, because it requires
connection to virtual smart card for this test

Change-Id: Ibae1c5167daa0157058bc40ce1857c1e6bb9da31

Fix bankd connection at RemsimClient_Tests.TC_bank_disconnect_reconnect

After each connection (initial connection and reconnect), the test must
wait for the bankd connection to be fully established. Previously the
IPA/RSPRO emulation failed, because the socket was closed before sending
the acknowledgemen, leading to a 'Broken pipe' error.

Change-Id: I0a8b339bcc4136442994b8825904df8aaf9aa95d

Fix bankd connection at RemsimClient_Tests.TC_bank_disconnect_reconnect

After each connection (initial connection and reconnect), the test must
wait for the bankd connection to be fully established. Previously the
IPA/RSPRO emulation failed, because the socket was closed before sending
the acknowledgemen, leading to a 'Broken pipe' error.

Change-Id: I0a8b339bcc4136442994b8825904df8aaf9aa95d

Fix: Remove slot mapping at bankd when client disconnects

If a client disconnects before removing the slot mapping, the slotmap
entry will not be removed automatically. A subsequent client connection
for the same mapping will then trigger the the error: “ignoring
identical slotmap.”

In a setup with multiple readers, there are multiple bankd worker
threads. If a client re-connects, but if its connection is accepted by
a different thread, subsequent reader access or unmapping will fail.
There will be no mapping at the new thread.

The approach of this fix is to remove the slot mapping, if the client
connection to the worker thread terminates.

Change-Id: I2fd03490e2506c55104309a0ef952389119023b8

Fix bankd connection at RemsimClient_Tests.TC_bank_disconnect_reconnect

After each connection (initial connection and reconnect), the test must
wait for the bankd connection to be fully established. Previously the
IPA/RSPRO emulation failed, because the socket was closed before sending
the acknowledgemen, leading to a 'Broken pipe' error.

Change-Id: I0a8b339bcc4136442994b8825904df8aaf9aa95d

Fix access pcscd in various RemsimBankd_Tests

Access to pcscd was denied, so that bankd never conneced to the reader.
This caused TC_createMapping_exchangeTPDU to fail, because it requires
connection to virtual smart card for this test

Change-Id: Ibae1c5167daa0157058bc40ce1857c1e6bb9da31

jobs/gerrit-verifications: fix hwtest param

This must be a bool instead of a string. Without this patch, the
condition checking for the bool is always true.

Fixes: 986c2648 ("jobs/gerrit: run osmo-ccid-firmware-hwtest")
Change-Id: I3117ddab888dd226ed57b62b9b9e672f1cb9127d

Fix: Remove slot mapping at bankd when client disconnects

If a client disconnects before removing the slot mapping, the slotmap
entry will not be removed automatically. A subsequent client connection
for the same mapping will then trigger the the error: “ignoring
identical slotmap.”

In a setup with multiple readers, there are multiple bankd worker
threads. If a client re-connects, but if its connection is accepted by
a different thread, subsequent reader access or unmapping will fail.
There will be no mapping at the new thread.

The approach of this fix is to remove the slot mapping, if the client
connection to the worker thread terminates.

Change-Id: I2fd03490e2506c55104309a0ef952389119023b8

Fix bankd connection at RemsimClient_Tests.TC_bank_disconnect_reconnect

After each connection (initial connection and reconnect), the test must
wait for the bankd connection to be fully established. Previously the
IPA/RSPRO emulation failed, because the socket was closed before sending
the acknowledgemen, leading to a 'Broken pipe' error.

Change-Id: I0a8b339bcc4136442994b8825904df8aaf9aa95d

Fix access pcscd in various RemsimBankd_Tests

Access to pcscd was denied, so that bankd never conneced to the reader.
This caused TC_createMapping_exchangeTPDU to fail, because it requires
connection to virtual smart card for this test

Change-Id: Ibae1c5167daa0157058bc40ce1857c1e6bb9da31

Fix access pcscd in various RemsimBankd_Tests

Access to pcscd was denied, so that bankd never conneced to the reader.
This caused TC_createMapping_exchangeTPDU to fail, because it requires
connection to virtual smart card for this test

Change-Id: Ibae1c5167daa0157058bc40ce1857c1e6bb9da31

Fix bankd connection at RemsimClient_Tests.TC_bank_disconnect_reconnect

After each connection (initial connection and reconnect), the test must
wait for the bankd connection to be fully established. Previously the
IPA/RSPRO emulation failed, because the socket was closed before sending
the acknowledgemen, leading to a 'Broken pipe' error.

Change-Id: I0a8b339bcc4136442994b8825904df8aaf9aa95d

Fix: Remove slot mapping at bankd when client disconnects

If a client disconnects before removing the slot mapping, the slotmap
entry will not be removed automatically. A subsequent client connection
for the same mapping will then trigger the the error: “ignoring
identical slotmap.”

In a setup with multiple readers, there are multiple bankd worker
threads. If a client re-connects, but if its connection is accepted by
a different thread, subsequent reader access or unmapping will fail.
There will be no mapping at the new thread.

The approach of this fix is to remove the slot mapping, if the client
connection to the worker thread terminates.

Change-Id: I2fd03490e2506c55104309a0ef952389119023b8

ansible/setup-simtest: sort packages

Change-Id: I9fac58407b7c5a38e2ed56e6e5718ce13ed23973

ansible/hosts: update simtest

Change-Id: Ica3c229ddf617b61e88774d3b330263307f84e97

ansible: openjdk: add mkdir ~/jenkins

Ensure the JDK can always be installed to /home/{{ jenkins_user
}}/jenkins/jdk/bin/java, even if ~/jenkins does not exit yet at this
point. This could be because jenkins didn't connect to the node yet, or
because the jenkins dir isn't set to ~/jenkins.

Related: OS#6998#note-4
Change-Id: I384398e2fe45988eaac32d61095c989914699245

ansible/setup-simtest: add sudo and wget

Both are needed for the new openjdk install logic.

Related: OS#6998#note-3
Change-Id: I4d5c400daec9c74d069c60a6221ae57e8c6f7953

ansible/setup-simtest: merge apt tasks

Do apt update and installing packages in one task, similar to how we do
it in other ansible yml files.

Change-Id: I269b368fdca682814976b2cda1c14c33529650b7

ansible/simtest: remove pip install logic

The pip install logic fails with:
  :stderr: error: externally-managed-environment

Remove this. Dependencies are installed venv/virtualenv:
https://gitea.osmocom.org/sim-card/pysim/src/commit/c50f4b4a0222a964710ce3124a66fe13c804be65/contrib/jenkins.sh#L25-L30

Change-Id: I460e22d78966d14e02b639efc4f74d7356e22f28

OBS: pyhss_download_deps: fix dependency loop

Fix that the Osmocom_OBS_nightly job is failing for the pyhss package
with a dependency loop in hatchling:

  LookupError: https://files.pythonhosted.org/…/hatchling-1.29.0.tar.gz (from https://pypi.org/simple/hatchling/) (requires-python:>=3.10) is already being built: hatchling>=1.21.0 from https://files.pythonhosted.org/…/hatchling-1.29.0.tar.gz

This happens as it recursively collects source packages and their
(build) dependencies:

  pymongo
  -> dnspython
  -> hatchling
  -> pluggy
  -> setuptools-scm
  -> vcs-versioning
  -> hatchling

When only ignoring the hatchling source package, we get a similar
dependency loop with pluggy, so ignore both source packages. Hatchling
gets added later as binary package and pluggy is only a build-time
dependency for hatchling. The resulting PyHSS package builds fine with
this change.

Change-Id: I0484627fd105627fabe9435379e704db54e72ae3

osmo_io: iofd_txqueue_dequeue(): fix double next->io_len decrement

The fill-up logic in iofd_txqueue_dequeue() was incorrectly
decrementing next->io_len twice: once when removing a buffer that
didn't fit, and again when assigning to msg_iovlen.  This may cause
next->io_len to reach 0 and trigger an assert at osmo_io.c:274:

  Assert failed next->io_len > 0

Add separate osmo_io_backpressure_test to verify queue management
under network backpressure conditions.  Use separate .ok files for
poll vs io_uring backends since they produce different I/O patterns.
This unit test reproduces the assert when the fix is not applied.

Change-Id: Ia96e25d11d8d683fa9dd0c56bdac257992edd782
Fixes: 3c2a02db4 ("osmo-io: Put together message buffers when dequeued from tx queue")
Related: OS#6997

osmo_io: iofd_txqueue_dequeue(): fix double next->io_len decrement

The fill-up logic in iofd_txqueue_dequeue() was incorrectly
decrementing next->io_len twice: once when removing a buffer that
didn't fit, and again when assigning to msg_iovlen.  This may cause
next->io_len to reach 0 and trigger an assert at osmo_io.c:274:

  Assert failed next->io_len > 0

Add separate osmo_io_backpressure_test to verify queue management
under network backpressure conditions.  Use separate .ok files for
poll vs io_uring backends since they produce different I/O patterns.
This unit test reproduces the assert when the fix is not applied.

Change-Id: Ia96e25d11d8d683fa9dd0c56bdac257992edd782
Fixes: 3c2a02db4 ("osmo-io: Put together message buffers when dequeued from tx queue")
Related: OS#6997

jobs/gerrit: archive osmo-ccid-firmware artifacts

Prepare to use the built firmware in a new hwtest job.

Related: SYS#7963
Change-Id: Ib220f708725e502724ef0375130e7db77c6e91fa

jobs/gerrit-osmo-ccid-firmware-hwtest: new job

Add a new test that flashes the firmware from the previous build job and
runs tests against it. Tests are intentionally from the master branch
for security reasons.

Related: SYS#7963
Depends: osmo-ccid-firmware I9445d6d5e4723bf6a87d4bffcdc638560471328b
Change-Id: I736b2b194813e6edff0197145fe08168c3e66903

jobs/gerrit: run osmo-ccid-firmware-hwtest

Related: SYS#7963
Change-Id: I438239f65f4b52fb5c77654b366b3c3254fcb44f

jenkins-gerrit/artifacts_download: new script

Add a script for downloading artifacts from the build job in a gerrit
pipeline, so it can be used in the hwtest job job.

pipeline: <--- URL to the pipeline is passed to all jobs
  parallel:
    sequential:
      build  <--- artifacts are here
      hwtest <--- will call the new script with the pipeline URL
    lint
    binpkgs
    ...

Related: SYS#7963
Change-Id: Ia96951d62b464b2e0b80f8e555cb77e5b9e52f78

tests/fetch_image: add SKIP_FETCH_IMAGE env var

Allow skipping the image fetch logic to let the gerrit job copy the
image from another job instead.

Related: osmo-ci I736b2b194813e6edff0197145fe08168c3e66903
Change-Id: I9445d6d5e4723bf6a87d4bffcdc638560471328b

tests/fetch_image: support copying from artifact

Allow copying the image from another job in
octsim_osmo-ccid-firmware-artifacts job.

Related: osmo-ci I736b2b194813e6edff0197145fe08168c3e66903
Change-Id: I9445d6d5e4723bf6a87d4bffcdc638560471328b

ansible: ntpd: update for debian 13

Use the openntpd package, as there isn't a ntp package anymore. The
service gets enabled and started automatically after installing it, so
remove the block that did it manually.

Change-Id: I7dad2f0cc034f487d4c97ec37325dab667f24ce3

ansible: add Debian-13.yml

Related: OS#6998
Change-Id: I34db89aba34e7c49843db0aa39fd593e35f4700a

ansible: docker: update apt key install method

Fix that the deprecated method for installing the apt key for the docker
repository was still used.

Fix for (with debian 13):
  Failed to find required executable "apt-key" in paths: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Related: OS#6998
Change-Id: I8184b6f9369886b79a3cf015ceee6e46e38b089c

ansible: lx2-raspbian{12 => 13}build-ansible

Related: OS#6998
Change-Id: I66afdc0ba86cfde6613226abba900c31e486f846

ansible: add Debian-13.yml

Related: OS#6998
Change-Id: I34db89aba34e7c49843db0aa39fd593e35f4700a

ansible: lx2-raspbian{12 => 13}build-ansible

Change-Id: I66afdc0ba86cfde6613226abba900c31e486f846

ansible: jenkins: update install openjdk logic

New jenkins versions require newer java versions to be installed on the
jenkins nodes. Install openjdk from download.java.net instead of using
distribution packages, so we are not limited by what the distribution
offers (which is oftentimes not recent enough).

Change-Id: I1de9a67aafe71e8c8b77c57baddb09611cb2e1fd

ansible: docker: update apt key install method

Fix that the deprecated method for installing the apt key for the docker
repository was still used.

Fix for:
  Failed to find required executable "apt-key" in paths: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Change-Id: I8184b6f9369886b79a3cf015ceee6e46e38b089c

ansible: debian: remove logic for jessie

This is a really old release we don't use anymore, so clean up related
commands from the ansible tasks.

Change-Id: Ifad5e9088e026a61b2e3463936c1ba602f4a99df

tests/fetch_image: support copying from artifact

Allow copying the image from another job in
octsim_osmo-ccid-firmware-artifacts job.

Related: osmo-ci I736b2b194813e6edff0197145fe08168c3e66903
Change-Id: I9445d6d5e4723bf6a87d4bffcdc638560471328b

tests/fetch_image: support copying from artifact

Allow copying the image from another job in
octsim_osmo-ccid-firmware-artifacts job.

Related: osmo-ci I736b2b194813e6edff0197145fe08168c3e66903
Change-Id: I9445d6d5e4723bf6a87d4bffcdc638560471328b

tests/fetch_image: support copying from artifact

Allow copying the image from another job in
octsim_osmo-ccid-firmware-artifacts job.

Related: osmo-ci I736b2b194813e6edff0197145fe08168c3e66903
Change-Id: I9445d6d5e4723bf6a87d4bffcdc638560471328b

tests/fetch_image: support copying from artifact

Allow copying the image from another job in
octsim_osmo-ccid-firmware-artifacts job.

Related: osmo-ci I736b2b194813e6edff0197145fe08168c3e66903
Change-Id: I9445d6d5e4723bf6a87d4bffcdc638560471328b

tests/fetch_image: support copying from artifact

Allow copying the image from another job in
octsim_osmo-ccid-firmware-artifacts job.

Related: osmo-ci I736b2b194813e6edff0197145fe08168c3e66903
Change-Id: I9445d6d5e4723bf6a87d4bffcdc638560471328b

tests/fetch_image: support copying from artifact

Allow copying the image from another job in
octsim_osmo-ccid-firmware-artifacts job.

Related: osmo-ci I736b2b194813e6edff0197145fe08168c3e66903
Change-Id: I9445d6d5e4723bf6a87d4bffcdc638560471328b

requirements: ensure safe version of PyYAML >= 5.4 (CVE-2020-1747)

PyYAML versions 5.1–5.3.1 are vulnerable to CVE-2020-1747, which allows
arbitrary code execution through yaml.FullLoader. While PyYAML 5.4+
patches this, the dependency specification (pyyaml >= 5.1) doesn't
guarantee a safe version. Let's increase the requirement to version
5.4 to ensure a safe version of is used.

This patch is based on suggestions from:
"YanTong C <chyeyantong03@gmail.com>"

Change-Id: I901c76c59e9c1bab030eab81038e04a475b32510

osmo-smdpp.py: fix path Traversal Bypass in SM-DP+ (CWE-22)

Root Cause:
os.path.commonprefix() compares strings character-by-character, NOT by path
components. This is a well-known Python antipattern (Python docs explicitly
warn: "this function may return invalid paths because it works a character
at a time").

Attack Context:
The matchingId parameter is received from a network client via the GSMA
ES9+ authenticateClient API endpoint (POST to
/gsma/rsp2/es9plus/authenticateClient). The SM-DP+ server is a Twisted web
application listening on port 443. An unauthenticated eUICC client sends
the matchingId in the ctxParamsForCommonAuthentication ASN.1 structure.

Fix:
Replace os.path.commonprefix() with proper path component checking:

Change-Id: I7a42b40aa2bbcd5f0ec99f172503354c6eaa9828

requirements: ensure safe version of PyYAML >= 5.4 (CVE-2020-1747)

PyYAML versions 5.1–5.3.1 are vulnerable to CVE-2020-1747, which allows
arbitrary code execution through yaml.FullLoader. While PyYAML 5.4+
patches this, the dependency specification (pyyaml >= 5.1) doesn't
guarantee a safe version. Let's increase the requirement to version
5.4 to ensure a safe version of is used.

This patch is based on suggestions from:
"YanTong C <chyeyantong03@gmail.com>"

Change-Id: I901c76c59e9c1bab030eab81038e04a475b32510

pySim-prog: fix Insecure PRNG for SIM Authentication Keys (CWE-338)

Root Cause:
pySim-prog.py uses Python's random module (Mersenne Twister MT19937) to
generate Ki and OPC — the root authentication keys for SIM cards. MT19937
is a deterministic PRNG that is not cryptographically secure. Its internal
state (624 × 32-bit words, 19,937 bits) can be fully recovered after
observing 624 consecutive outputs.

Impact:
1. SIM Card Cloning: An attacker who determines the PRNG state can predict
all Ki/OPC values generated before and after. With these keys, SIM cards
can be cloned.
2. Network Authentication Bypass: Ki/OPC are used in the Milenage algorithm
for 3G/4G/5G authentication. Predictable keys mean an attacker can
authenticate as any subscriber whose SIM was provisioned with the weak RNG.
3. Batch Compromise: In bulk provisioning scenarios (pySim-prog's primary
use case), hundreds or thousands of SIMs may be programmed sequentially.
Compromising one batch means recovering the PRNG state to predict all keys.

Fix:
Replace random.randrange() with os.urandom()

Change-Id: Id3e00d3ec5386f17c1525cacfc7d3f5bba43381f

formatting: update .clang-format

Updates the formatting rules derived from linux kernel
098b6e44cbaa2d526d06af90c862d13fb414a0ec

At the same time tell clang-format to keep the bespoke formatting
used for the fsm and usb descs.

Change-Id: I1fd4d38897946693b5495fa4d0b61925eb9d07f7

octsim: usb_descriptors: use LE16/LE32 for consistancy

Even this is ran on ARM and most likely on x86, add byte ordering
function (LE16/LE32) to 16 bit and 32 bit fields as the remaining
usb descriptor does.

Change-Id: Ia77d65faf56c5d3c57cc76c91af33fce4fbbff5c

octsim: main: document the fixed allocation sizes for msgb

Use the same allocation for all msgb to prevent
heap allocation fragmentation.
Add a comment to document it.

Change-Id: I33809d2c1973231627f6da5d9b01e9525f2b38c1

octsim: usb_descriptors: use LE16/LE32 for consistancy

Even this is ran on ARM and most likely on x86, add byte ordering
function (LE16/LE32) to 16 bit and 32 bit fields as the remaining
usb descriptor does.

Change-Id: Ia77d65faf56c5d3c57cc76c91af33fce4fbbff5c

Reset 'release_99' flag in test cases at comment_start()

Rather than resetting release_99 flag in every test case that did set
it, do it in comment_start().

Change-Id: I55a49dd24a5df6ee8e28fa6b410988e82d10c554

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

Bump version: 1.13.1.12-8e6ea → 1.14.0

Change-Id: Ifa079c76148fd25f543b7875d3d89b35c19ffe6b

contrib/jenkins.sh: use Gitea mirrors for libnftnl and nftables

Change-Id: I28950d823dfd91e8e2ac23a19e5a86d1027096b3
Depends: osmo-ci.git I830a16c8bf7f595b16113bf2ba5e81bae56533dd

octsim: main: reduce msgb_alloc for irqs to 8 bytes

The message itself is only 3 bytes, but the usb stack might benefit
from 8 byte buffers.

Change-Id: I33809d2c1973231627f6da5d9b01e9525f2b38c1

octsim: usb_descriptor: reduce IRQ endpoint max packet size to 4 bytes

The largest message on the endpoint is the 4 byte hardware error.
The slot status is 3 bytes for 8 slots.

Change-Id: I615ff16d9bc0e8d39156015e41b4abe4502aaca1

octsim: usb_descriptors: use LE16/LE32 for consistancy

Even this is ran on ARM and most likely on x86, add byte ordering
function (LE16/LE32) to 16 bit and 32 bit fields as the remaining
usb descriptor does.

Change-Id: Ia77d65faf56c5d3c57cc76c91af33fce4fbbff5c

cosmetic: fix missing spaces in string_req_h

Change-Id: Ia92ac605d83718399f19db67057e740ceaa8c37f

Implement vendor requests to set debug/break_on_panic

Use vendor request in to read and set the break_on_panic.
In the future this vendor request could be also used to set other
debug related options.

Change-Id: Ia2bffa05b96e3857610cb3ad2f9122fa75a90780

Implement vendor requests to set debug/break_on_panic

Use vendor request in to read and set the break_on_panic.
In the future this vendor request could be also used to set other
debug related options.

Change-Id: Ia2bffa05b96e3857610cb3ad2f9122fa75a90780

Implement vendor requests to set debug/break_on_panic

Use vendor request in to read and set the break_on_panic.
In the future this vendor request could be also used to set other
debug related options.

Change-Id: Ia2bffa05b96e3857610cb3ad2f9122fa75a90780

Rework default IRQ handler and osmo panic handler

Don't break on panic by default. Using a variable allows
to set it by SWD or later by a usb message.

Use the same handler for osmo panics as well as unknown IRQ.
Including faults (bus, hard).

Change-Id: Ibcabe0c58ff14ebe76159470dd90fdef064febea

osmo_git_clone_url(): add Gitea mirrors for libnftnl and nftables

Change-Id: I830a16c8bf7f595b16113bf2ba5e81bae56533dd
Related: osmo-upf.git I28950d823dfd91e8e2ac23a19e5a86d1027096b3

scripts: fix deprecation warnings

scripts/verify_value_string_arrays_are_terminated.py:22: SyntaxWarning:
    "\s" is an invalid escape sequence. Such sequences will not work in the future.
    Did you mean "\\s"? A raw string is also an option.
scripts/verify_value_string_arrays_are_terminated.py:22: SyntaxWarning:
    "\s" is an invalid escape sequence. Such sequences will not work in the future.
    Did you mean "\\s"? A raw string is also an option.
scripts/verify_value_string_arrays_are_terminated.py:23: SyntaxWarning:
    "\s" is an invalid escape sequence. Such sequences will not work in the future.
    Did you mean "\\s"? A raw string is also an option.
scripts/verify_value_string_arrays_are_terminated.py:23: SyntaxWarning:
    "\s" is an invalid escape sequence. Such sequences will not work in the future.
    Did you mean "\\s"? A raw string is also an option.
scripts/verify_value_string_arrays_are_terminated.py:30: DeprecationWarning:
    codecs.open() is deprecated. Use open() instead.

Change-Id: Ie78b84dd556266f96780a4232f95b58e0e3eabc0

contrib/jenkins.sh: use Gitea mirrors for libnftnl and nftables

Change-Id: I28950d823dfd91e8e2ac23a19e5a86d1027096b3
Depends: osmo-ci.git I830a16c8bf7f595b16113bf2ba5e81bae56533dd

jobs/octsim_osmo-ccid-firmware: add branch param

Change-Id: I05321cd2c174060381e099aa49d5932f9cafdd28

jobs/octsim_osmo-ccid-firmware: PYTHONUNBUFFERED=1

Ensure that the python output is getting displayed directly when running
in jenkins.

Change-Id: Iffe74b270f33286f394db1be03e6ce1953aff4bf

jobs/octsim_osmo-ccid-firmware: remove timer

This job already gets triggered once a day from
master-osmo-ccid-firmware. Running it once a day like all other master
jobs should be enough, remove the timed trigger that caused it to run
twice a day.

Change-Id: Ia4fdb6d504c7a08cf258e82436627894602f7301

jobs/octsim_osmo-ccid-firmware: don't use axis

There is only one node, so we can specify it directly instead of using a
build axis.

Change-Id: I552c8f1b244c8e8b0f902720698fbb6ba660fb72

jobs/octsim_osmo-ccid-firmware: various tweaks

* move name and desc up (in preparation to adding another job in the same file)
* remove "concurrent: false": the octsimtest node is already configured
  to only accept one job. if we had multiple octsimtest nodes, we
  wouldn't need this limitation of not running concurrently, so this
  isn't useful.
* remove options without effect (block-downstream, block-upstream,
  disabled, build-blocker, etc.)
* add "scm checkout" comment to retry-count as we have it in other files

Change-Id: If64787efbe4a1305cd60941b69e9416960d056a2

jobs/octsim_osmo-ccid-firmware: add email param

Allow configuring the email notifications before starting a job, so no
notifications can be set while testing changes to the jenkins job. Add
jenkins-notifications@lists.osmocom.org while at it, as we have it with
almost all other jobs.

Change-Id: Ifcd580873479fc259119139d47447ba0e21b21f4

Check for protocol extension bit in message type at a central location

Bits 8 and 7 of message type can be used as sequence number or have
specific default values. The added check will verify the bits according
to the prococol descriptor and Release version.

The Rules for these bits are described in GSM TS 124.007, clause
11.2.3.2.1 for Release <= 98 and 11.2.3.2.2 for Release >= 99.

There is also no need to clear bit 7 in message type for these
protocols, as it is done already in gsm48_hdr_msg_type().

Some unit tests fail without fixes that are included in this patch.

Change-Id: Iae41bc6a2e9fd85583509b6c6154dd5a935fb5df

.gitignore: add missing osmux_{input,output}_test

Take a chance to group test binaries and specify their full paths.

Change-Id: Icb9014fe3c501940cef2d599403aa102135dd384

firmware/se_k2xx: add board specific AFC params

Set afc_slope to 405 and afc_initial_dac_value to -370, derived from
AFC parameter averages computed across TIFFS readings of 7 unique
SE K2xx units (5 x K200i + 1 x K205i + 1 x K220i).

Change-Id: Iee2569c124dd0160f16457e0171cfd48d961045e
Related: OS#5822

firmware/se_k2xx: add board specific RF tables

SE K2xx phones use a different PA than GTA0x/FCDEV3B, so the
compiled-in Tx ramp templates must match the actual hardware.
Replace the previously copied GTA0x ramps with values extracted
from the SE K2xx official firmware using the calextract tool
from the freecalypso-reveng repository:

https://www.freecalypso.org/hg/freecalypso-reveng/

The Tx levels tables are left unchanged, as they are believed to
be the same TI internal defaults across these designs.

Change-Id: If90973ae6bf9af1cd15e2d80e0b99fe098b648bd
Related: OS#5822

firmware/se_k2xx: add board specific AFC params

Set afc_slope to 405 and afc_initial_dac_value to -2957, derived from
AFC parameter averages computed across TIFFS readings of 7 unique
SE K2xx units (5 x K200i + 1 x K205i + 1 x K220i).

Change-Id: Iee2569c124dd0160f16457e0171cfd48d961045e
Related: OS#5822

firmware/se_k2xx: add board specific RF tables

SE K2xx phones use a different PA than GTA0x/FCDEV3B, so the
compiled-in Tx ramp templates must match the actual hardware.
Replace the previously copied GTA0x ramps with values extracted
from the SE K2xx official firmware using the calextract tool
from the freecalypso-reveng repository:

https://www.freecalypso.org/hg/freecalypso-reveng/

The Tx levels tables are left unchanged, as they are believed to
be the same TI internal defaults across these designs.

Change-Id: If90973ae6bf9af1cd15e2d80e0b99fe098b648bd
Related: OS#5822

firmware/se_k2xx: add board specific RF tables

SE K2xx phones use a different PA than GTA0x/FCDEV3B, so the
compiled-in Tx ramp templates must match the actual hardware.
Replace the previously copied GTA0x ramps with values extracted
from the SE K2xx official firmware using the calextract tool
from the freecalypso-reveng repository:

https://www.freecalypso.org/hg/freecalypso-reveng/

The Tx levels tables are left unchanged, as they are believed to
be the same TI internal defaults across these designs.

Change-Id: If90973ae6bf9af1cd15e2d80e0b99fe098b648bd
Related: OS#5822

firmware/se_k2xx: add board specific AFC params

Set afc_slope to 405 and afc_initial_dac_value to -2957, derived from
AFC parameter averages computed across TIFFS readings of 7 unique
SE K2xx units (5 x K200i + 1 x K205i + 1 x K220i).

Change-Id: Iee2569c124dd0160f16457e0171cfd48d961045e
Related: OS#5822

csn1: ProcessError_impl(): add a missing newline

The comment about caller's responsibility is wrong and misleading.
The caller provides format string parameters, not the format string!

Change-Id: I0f7c0a6d25db03c1fe8503b57053130ed29baf23

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

sgsn_libgtp: fix NULL dereference in cb_delete_context()

sgsn_pdp_ctx_free() contains an abnormal path (with a backtrace log)
for the case where a PDP context is freed while its libgtp handle is
still attached.  In that path, lib->priv is set to NULL to prevent
other callbacks (e.g. cb_data_ind()) from dereferencing the already-
freed sgsn_pdp_ctx.

When libgtp subsequently calls cb_delete_context() as part of its own
teardown, pdp->priv is NULL, causing an immediate NULL dereference at
the pctx->lib = NULL assignment.

Guard against this by returning early when priv is NULL: the SGSN-side
context is already gone, so there is nothing left for the callback to do.

Change-Id: I7d800766b08d87b6f2b23a0cd45435925f9998ff

rspro_server: don't access the msg on error cases

The msg is NULL on errors.

Related: SYS#7930
Change-Id: I4e95c2473acd72cd2e01df329691c8f31882b3d9

rspro_server: don't access the msg on error cases

The msg is NULL on errors.

Related: SYS#7930
Change-Id: I4e95c2473acd72cd2e01df329691c8f31882b3d9

handover_fsm: send_handover_performed(): check lchan != NULL

The target lchan is expected to be valid on HO_RESULT_OK, but it's
still safer to check it against NULL before dereferencing.

Change-Id: Iee734c98dc79681808e7d75360c208a941cc6821
Related: 107391d8c ("handover_fsm: fix send_handover_performed() using wrong lchan")

assignment_fsm: ipacc_chan_mode_supported(): log unsupported chan_mode

Change-Id: I1718994e25ff1349618bdb4660afbe735c5c3977
Related: aa959d74f ("assignment_fsm: check ipaccess channel mode support")

filesystem: JsonEditor: offer interactive retry on error

When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'.  If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.

Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899

tests: fix TransRecEF _test_de_encode to operate at file level

Previously, _test_de_encode vectors for TransRecEF subclasses were tested
via decode_record_hex()/encode_record_hex(), i.e. one record at a time,
with the decoded value being a scalar.

Switch test_de_encode_record() in TransRecEF_Test to use decode_hex() /
encode_hex() instead, so that vectors represent whole-file content
(decoded value is a list of records) -- consistent with how LinFixedEF
handles _test_de_encode.  Update all existing vectors accordingly.

Change-Id: I4a9610f9ee39833cd0c90f64f89f5fbdd6f0846d

pySim/ts_51_011.py: add multi-record test vector for EF_PL

Change-Id: I9f7a444b18056b1683cbd52a25af950125531746

filesystem: JsonEditor: use NamedTemporaryFile

A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it.  Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.

Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e

filesystem: JsonEditor: use NamedTemporaryFile

A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it.  Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.

Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e

filesystem: JsonEditor: offer interactive retry on error

When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'.  If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.

Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899

filesystem: edit_{binary,record}_decoded: add encode/decode examples

When invoking `edit_binary_decoded` or `edit_record_decoded`, the
temp file opened in the editor now contains the EF's encode/decode
test vectors as //-comment lines below the JSON content, similar to
how 'git commit' appends comments to the commit message template.
The comment block is stripped before JSON parsing on save,
so it has no effect on the written data.

The feature is implemented via a new module-level JsonEditor context
manager class that encapsulates the full edit cycle:

* write JSON + examples to a TemporaryDirectory
* invoke the editor
* read back, strip //-comments, parse and return the result

Change-Id: I5a046a9c7ba7e08a98cf643d5a26bc669539b38f
Related: OS#6900

gprs_gmm: gsm48_rx_gmm_att_req(): drop stale PDP contexts on re-Attach

When an MS sends an Attach Request with a known IMSI or P-TMSI, it
implies the MS has restarted and lost all its PDP context state.  The
SGSN must clean up any PDP contexts it still holds for that subscriber;
otherwise it will not create new ones upon subsequent PDP Context
Activation, causing data connectivity failure after a device reboot.

Change-Id: I20c1f5f741275115635188b2f4b1c5fe7c6e40f1
Related: OS#6922

tests/osmo-sgsn_test-nodes.vty: add missing list entries

Change-Id: I6b4cbbae30c602d0fd4ef9b2f91c97f20217eb53

gprs_sm: QoS: make max-sdu-size configurable

Some phones, such as the Sony Ericsson Z520, reject or ignore the
Activate PDP Context Accept if they dislike the max-sdu-size value
in the QoS profile.  Make this value configurable via the VTY so
it can be adjusted to accommodate such devices.

Change-Id: I0819da18dafecf322f8902643dae8dbcaa5dc98b
Related: OS#6735

osmo-bts-omldummy: properly report NM_ATT_IPACC_SUPP_FEATURES

osmo-bts-omldummy is used as the OML backend in the ttcn3-bsc-test.
The common OML code encodes bts->support.* and trx->support.* fields
verbatim into the IPA Supported Features IE (NM_ATT_IPACC_SUPP_FEATURES)
in Get Attributes Responses.  Since bts_model_init() and
bts_model_trx_init() never initialized these fields, they were reported
as all-zeros to osmo-bsc.

osmo-bsc now inspects the supported channel modes before activating a
channel.  An all-zero chan_modes value (present but no bits set) causes
osmo-bsc to reject every speech mode, breaking all assignment-related
TTCN-3 test cases.

Initialize ciphers, gprs_codings, freq_bands, chan_types and chan_modes
to sensible values, matching what osmo-bts-virtual does, so that
osmo-bts-omldummy reports its capabilities correctly.

Change-Id: I7fe83f78c829c300e70a59509847b815a77974d7
Fixes: 0978d1df ("oml: implement handling of NM_ATT_IPACC_SUPP_FEATURES")
Related: OS#6324

gprs_sm: QoS: make max-sdu-size configurable

Some phones, such as the Sony Ericsson Z520, reject or ignore the
Activate PDP Context Accept if they dislike the max-sdu-size value
in the QoS profile.  Make this value configurable via the VTY so
it can be adjusted to accommodate such devices.

Change-Id: I0819da18dafecf322f8902643dae8dbcaa5dc98b
Related: OS#6735

tests/osmo-sgsn_test-nodes.vty: add missing list entries

Change-Id: I6b4cbbae30c602d0fd4ef9b2f91c97f20217eb53

gprs_sm: gsm48_tx_gsm_act_pdp_acc(): fix QoS profile length

The Activate PDP Context Accept was always sending sizeof(default_qos)
(14 bytes) as the QoS profile length, regardless of what the UE
requested.  Older modules such as the Air20X may crash and restart
during PDP attachment because of that.

In GTP, qos_req.l encodes 1 ARP byte followed by the QoS profile
octets, so (qos_req.l - 1) is the actual profile length.  Mirror
back the same QoS profile length the UE sent in its request,
capped at sizeof(default_qos) to avoid overrunning the default_qos
buffer.  This matters in particular for R97/R98 UEs that send a
3-byte QoS profile and should not receive a 14-byte response.

Change-Id: I11c24b64f0e49cf80c825969dbf018b2948d855c
Related: OS#6922

gprs_gmm: gsm48_rx_gmm_att_req(): drop stale PDP contexts on re-Attach

When an MS sends an Attach Request with a known IMSI or P-TMSI, it
implies the MS has restarted and lost all its PDP context state.  The
SGSN must clean up any PDP contexts it still holds for that subscriber;
otherwise it will not create new ones upon subsequent PDP Context
Activation, causing data connectivity failure after a device reboot.

Change-Id: I20c1f5f741275115635188b2f4b1c5fe7c6e40f1
Related: OS#6922

bssap: mark some pointers as const

Change-Id: Iebb28d6ae0b468545994669248a442dd17f36d4c

assignment_fsm: pass lchan to lchan_type_compat_with_mode()

The lchan pointer is needed by the next patch, which will check
the BTS ipaccess supported-features flags to reject channel modes
that the BTS has reported as unsupported before attempting to
activate the lchan.

Change-Id: I51d5d1a1cf3c51f85f738196967d5c69870681bb
Related: OS#6324

assignment_fsm: check ipaccess channel mode support

Add ipacc_chan_mode_supported(), which checks the NM_IPAC_F_CHANM_*
flags stored in the Baseband Transceiver's MO state (populated during
OML bring-up) to decide whether the BTS can handle a requested channel
mode.  If the IE containing supported channel modes was never received,
the function returns true as if the given mode was supported.

Without this change, requesting a channel mode not supported by the
BTS results in a NACK to the RSL Channel Activation, which causes
the lchan to be marked as BROKEN and thus unavailable.

Change-Id: I680ba7993786f5486d671f931e75df4543670a37
Related: OS#6324

bssap: use LOGPFSML in bssmap_handle_ass_req_tp_codec_list()

Change-Id: Ie51ada174388f4c3e9ffe8a5bec4297bda38584b

bssap: use LOGPFSML in bssmap_handle_assignm_req()

Change-Id: If99661c0ff1177214bacdb2d1bcb9da1d2d2957c

ipaccess: fix buffer overread in ipacc_parse_supp_flags()

The loop used OSMO_MAX(e->len, 4), which iterates at least 4 times
even when the IE is shorter than 4 bytes, causing a buffer overread.
Replace with OSMO_MIN(e->len, sizeof(u32)) to cap the iteration both
at the actual IE length and at the uint32_t accumulator size.

Change-Id: I97c69a71eb650cbef1cc3652d0a2a966cfd6cf60

lchan_fsm: don't mark lchan as BORKEN on unsupported mode NACK

When the BTS NACKs a Channel Activation with RSL_ERR_SERV_OPT_UNAVAIL
or RSL_ERR_SERV_OPT_UNIMPL, it means the requested service or channel
mode is not supported - not that the hardware is broken.  In this
case the lchan should transition to LCHAN_ST_WAIT_AFTER_ERROR rather
than LCHAN_ST_BORKEN, which is reserved for genuine hardware failures.

Also add LCHAN_EV_RTP_ERROR to LCHAN_ST_WAIT_AFTER_ERROR's in_event_mask
so that a late LCHAN_EV_RTP_ERROR (e.g. MGCP CRCX timeout arriving after
the NACK was already handled) is silently absorbed rather than triggering
an unexpected-event warning.

Change-Id: Ide3830a697501a0c245a41451302f1e68d553b3c
Related: osmo-ttcn3-hacks.git I000b7b44749380c5af4da42abe37b4ada6e06ee4
Related: OS#6324

lchan_fsm: ignore late lchan_rtp_fsm events

The lchan_rtp_fsm is a child FSM that manages the MGW/RTP endpoint.
When the parent lchan_fsm transitions away before the child has
finished its work (e.g. a CRCX timeout or a DLCX completing after
the lchan is already idle), the child can still deliver
LCHAN_EV_RTP_RELEASED or LCHAN_EV_RTP_ERROR to the parent.

Currently these late events are not in the in_event_mask of every
state that can be reached with the child still running:

* LCHAN_ST_UNUSED: entered from WAIT_AFTER_ERROR after the error timer
  fires, while a DLCX triggered at error time may still be in flight.
* LCHAN_ST_WAIT_AFTER_ERROR: already handles LCHAN_EV_RTP_RELEASED but
  misses LCHAN_EV_RTP_ERROR (e.g. CRCX timeout arriving after the
  NACK was handled).

Add the missing events to both states' in_event_mask and provide a
no-op handler in lchan_fsm_unused() so that the assert is not hit.

Change-Id: Ie6333bd941e4e5a6ddf0e3f113b8764e8bc2bbc0

ipaccess: store Supported Features IE to the MO state

Change-Id: Ic89d42c478677ffda4d544f461e7850dd3157040
Related: OS#6324

handover_fsm: fix send_handover_performed() using wrong lchan

send_handover_performed() is called from handover_end() before
gscon_change_primary_lchan() updates conn->lchan to the new lchan.
As a result, the Cell Identifier, Chosen Channel, Chosen Encryption
Algorithm, and Speech Version/Codec IEs were all populated from the
source (old) lchan rather than the target (new) lchan.

As per 3GPP TS 48.008 §3.2.1.25, HANDOVER PERFORMED must report the
target cell and channel.  For intra-BSC handover to a different BTS
this caused the MSC to receive the wrong Cell Identifier.

Fix by initialising lchan from ho->new_lchan instead of conn->lchan,
and deriving the BTS via lchan->ts->trx->bts.

Change-Id: I4111351dc38fc2dbe844c2bd07b3ecfaaadd864e
Related: osmo-ttcn3-hacks.git I82aadcc3a80c183cb93522b829071294b156a218
Found-By: Claude Sonnet 4.6

bsc: verify Cell ID in BSSMAP HandoverPerformed

As per 3GPP TS 48.008 §3.2.1.25, HANDOVER PERFORMED must report the
target cell.  Add an optional cell_id parameter to the template
(defaulting to '?' for backward compatibility) and update the two
intra-BSC handover test helpers to assert that the Cell Identifier
IE carries the target BTS (BTS 1: MCC=001 MNC=01 LAC=1 CI=1).

Also remove the unnecessary interleave wrapper in
f_tc_ho_during_lcs_loc_req(): the two receives arrive on separate
ports (RAN_CONN vs BSSAP_LE) so sequential receive is sufficient.

Change-Id: I82aadcc3a80c183cb93522b829071294b156a218
Related: osmo-bsc.git I4111351dc38fc2dbe844c2bd07b3ecfaaadd864e

bsc: add TC_chan_act_nack_not_borken

Test that when the BTS NACKs a CHANnel ACTIVation with
RSL_ERR_SERV_OPT_UNAVAIL or RSL_ERR_SERV_OPT_UNIMPL, the BSC:
does NOT mark the lchan as BORKEN.

Change-Id: I000b7b44749380c5af4da42abe37b4ada6e06ee4
Related: osmo-bsc.git Ide3830a697501a0c245a41451302f1e68d553b3c
Related: OS#6324

library/RSL_Types: improve CHAN_ACTIV templates

Change-Id: I80af30fe9b24b93908876fd8059f45b6e3105406

msc: ASCI: Fix race condition in TC_assign_fail

The f_sleep(0.5) to delay the registration of the PC for the 2nd SCCP
connection sometimes fired too late and we received the VGCS/VBS
Assignment Request before the channel ConnHdlr was registered in
RAN_Emulation.
Then in turn we need to delay call establishment until the channel is
ready.
Implement proper synchronization points to make sure the order of
registers is accomplished while making sure the 2 ConnHdlrs are
registered before they need it.

Change-Id: Id80982b6b968bbe462aeac2b9be646ab50eace83

pysim/pcsc: do not use getProtocol for protocol selection

The documentation of the getProtocol provided by pyscard says:

"Return bit mask for the protocol of connection, or None if no
protocol set. The return value is a bit mask of
CardConnection.T0_protocol, CardConnection.T1_protocol,
CardConnection.RAW_protocol, CardConnection.T15_protocol"

This suggests that the purpose of getProtocol is not to determine
which protocols are supported. Its purpose is to determine which
protocol is currently selected (either through auto selection or
through the explicit selection made by the API user). This means
we are using getProtocol wrong.

So far this was no problem, since the auto-selected protocol
should be a supported protocol anyway. However, the automatic
protocol selection may not always return a correct result (see
bug report from THD-siegfried [1]).

Let's not trust the automatic protocol selection. Instead let's
parse the ATR and make the decision based on the TD1/TD2 bytes).

[1] https://osmocom.org/issues/6952

Related: OS#6952
Change-Id: Ib119948aa68c430e42ac84daec8b9bd542db7963

contrib/jenkins: add SKIP_FIRMWARE_CLEAN argument

If this is set, then don't clean the firmware at the end. This is needed
to store the built firmware as artifact in jenkins, for passing it along
to another job.

Related: SYS#7963
Change-Id: I9e2342b8111645ef5d3b7bf4ac364ca592aa600b

jobs/octsim_osmo-ccid-firmware: add 30 min timeout

I just stopped a job run that was going for 2 days 16 h. When
successful, the job finishes in 10 min, so let it abort automatically
after 30 min.

Change-Id: Iaccc96b8b29880c9e115e52561fcd43f7b6a8f77

msc: Fix race condition in TC_reset_two(_1iu)

The test failed sporadically with:
"RAN_Emulation.ttcnpp:600 Dynamic test case error: Sending data on the
connection of port BSSAP to 659:SCCP_SP_PORT failed. (Broken pipe)"

That's because it was simply waiting for a fixed amount of time while
letting the underlaying RAN_Emulation do the RESET procedure.
Sometimes it started tearing down the test before the procedure was
fully completed, and RAN_Emulation failed passing the Reset (ACK) up the
stack because the component was already unmapped.

Fix the issue by making sure the RAN_Emulation finished the RESET
procedure before stopping the testThe test.

Change-Id: I27afd7a47e6c6cf0f70e1ecc9e26dd2c220e72ab

msc: Fix race condition in TC_reset_two(_1iu)

The test failed sporadically with:
"RAN_Emulation.ttcnpp:600 Dynamic test case error: Sending data on the
connection of port BSSAP to 659:SCCP_SP_PORT failed. (Broken pipe)"

That's because it was simply waiting for a fixed amount of time while
letting the underlaying RAN_Emulation do the RESET procedure.
Sometimes it started tearing down the test before the procedure was
fully completed, and RAN_Emulation failed passing the Reset (ACK) up the
stack because the component was already unmapped.

Fix the issue by making sure the RAN_Emulation finished the RESET
procedure before stopping the testThe test.

Change-Id: I27afd7a47e6c6cf0f70e1ecc9e26dd2c220e72ab

msc: ASCI: Fix race condition in TC_assign_fail

The f_sleep(0.5) to delay the registration of the PC for the 2nd SCCP
connection sometimes fired too late and we received the VGCS/VBS
Assignment Request before the channel ConnHdlr was registered in
RAN_Emulation.
Implement proper synchronization points to make sure the order of
registers is accomplished while making sure the 2 ConnHdlrs are
registered before they need it.

Change-Id: Id80982b6b968bbe462aeac2b9be646ab50eace83

Support administrative block in role ASP

* Avoid activating later on if inactive when becoming blocked.
* Move to inactive state (and notify peer with ASPIA) if active when becoming blocked.
* Attempt re-activation if needed when becoming non-blocked.

Related: OS#6752
Change-Id: If32902575e874d2acc9a5fc12509af42ed151739

asp: Fix wrong variable logged in ss7_asp_xua_srv_conn_rx_cb()

Change-Id: I24206a19fe57772caf8036464a2e1b6b8630fd7a

Create missing LayerManager fsm in dynamic srv xua asps

Dynamic xUA ASPs in transport_role=server where missing initialization
+start of its LayerManager, meaning they had no xua_default_lm_fsm
instance attached.
Previous the xua_default_lm_fsm didn't do anything internally, but since
recently ir properly transitions through states in SG role too, so
we want to handle blocking/unblocking, etc.

Change-Id: Id01e7a24e92aecc61aa2850632b3712bb338e9f2

stp: m3ua: Expect unsolicited ASPIA ACK after unblocking ASP

Since libosmo-sigtran.git Change-Id
If32902575e874d2acc9a5fc12509af42ed151739, osmo-stp will send an
unsolicited ASPIA ACK to the peer whenever the ASP becomes unblock, in
order to somehow give a chance to the peer to notice something changed
and for instance possibly attempt re-activation of the ASP.

Related: OS#6752
Change-Id: I4b70e576beae3c386d68f17a60d6f23d1a54293a

filesystem: edit_{binary,record}_decoded: add encode/decode examples

When invoking `edit_binary_decoded` or `edit_record_decoded`, the
temp file opened in the editor now contains the EF's encode/decode
test vectors as //-comment lines below the JSON content, similar to
how 'git commit' appends comments to the commit message template.
The comment block is stripped before JSON parsing on save,
so it has no effect on the written data.

The feature is implemented via a new module-level JsonEditor context
manager class that encapsulates the full edit cycle:

* write JSON + examples to a TemporaryDirectory
* invoke the editor
* read back, strip //-comments, parse and return the result

Change-Id: I5a046a9c7ba7e08a98cf643d5a26bc669539b38f
Related: OS#6900

docs: auto-generate Card Filesystem Reference

Add a Sphinx extension (docs/pysim_fs_sphinx.py) that hooks into the
builder-inited event and generates docs/filesystem.rst before Sphinx
reads any source files.

The generated page contains a hierarchical listing of all implemented
EFs and DFs, organised by application/specification (UICC/TS 102 221,
ADF.USIM/TS 31.102, ADF.ISIM/TS 31.103, SIM/TS 51.011).  For each file,
the class docstring and any _test_de_encode / _test_decode vectors
are included as an encoding/decoding example table.

docs/filesystem.rst is fully generated at build time and is therefore
added to .gitignore.

Add tests/unittests/test_fs_coverage.py that walks all pySim.* modules
and verifies that every CardProfile, CardApplication, and standalone
CardDF subclass with EF/DF children is either listed in the SECTIONS
(and will appear in the docs) or explicitly EXCLUDED.

Change-Id: I06ddeefc6c11e04d7c24e116f3f39c8a6635856f
Related: OS#6316

filesystem: JsonEditor: offer interactive retry on error

When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'.  If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.

Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899

filesystem: JsonEditor: use NamedTemporaryFile

A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it.  Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.

Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e

Create missing LayerManager fsm in dynamic srv xua asps

Dynamic xUA ASPs in transport_role=server where missing initialization
+start of its LayerManager, meaning they had no xua_default_lm_fsm
instance attached.
Previous the xua_default_lm_fsm didn't do anything internally, but since
recently ir properly transitions through states in SG role too, so
we want to handle blocking/unblocking, etc.

Change-Id: Id01e7a24e92aecc61aa2850632b3712bb338e9f2

cosmetic: xua_asp_fsm: Fix wrong indentation

Change-Id: I21c13ff7bc67c060e739eae38088ac00700680de

xua_lm: Add S_INACTIVE state and improve state transitions

The S_INACTIVE state will be needed once we support administrative state
block in role ASP/(active)IPSP.

It is also useful now since sometimes we just want to leave an asp in
that state without any timer running (eg. asps in override which are not
active until the active one goes inactive).

Adding the S_INACTIVE state also allows transitioning states in role
SG, since we now have somewhere to go instead of leaving it alwas as
S_IDLE.

The S_INACTIVE state also allows staying in that state until the ASP ACT
ACK is received, moment at which the ASP is considered ACTIVE according
to specs.

Change-Id: I7e6127c150e479d0c436875b41824b3061e4fa48

asp: Fix wrong variable logged in ss7_asp_xua_srv_conn_rx_cb()

Change-Id: I24206a19fe57772caf8036464a2e1b6b8630fd7a

Support administrative block in role ASP

* Avoid activating later on if inactive when becoming blocked.
* Move to inactive state (and notify peer with ASPIA) if active when becoming blocked.
* Attempt re-activation if needed when becoming non-blocked.

Related: OS#6752
Change-Id: If32902575e874d2acc9a5fc12509af42ed151739

asp: Fix wrong variable logged in ss7_asp_xua_srv_conn_rx_cb()

Change-Id: I24206a19fe57772caf8036464a2e1b6b8630fd7a

stp: m3ua: Introduce test TC_adm_block_after_act

Related: libosmo-sigtran.git Change-Id If8e34cfb05b8b63173b8bb01a520c058daf1fe39
Related: OS#6752
Change-Id: I18672d228c1d15ef99039bccbb972e0b2496b545

Support administrative block in role ASP

* Avoid activating later on if inactive when becoming blocked.
* Move to inactive state (and notify peer with ASPIA) if active when becoming blocked.
* Attempt re-activation if needed when becoming non-blocked.

Related: OS#6752
Change-Id: If32902575e874d2acc9a5fc12509af42ed151739

Create missing LayerManager fsm in dynamic srv xua asps

Dynamic xUA ASPs in transport_role=server where missing initialization
+start of its LayerManager, meaning they had no xua_default_lm_fsm
instance attached.
Previous the xua_default_lm_fsm didn't do anything internally, but since
recently ir properly transitions through states in SG role too, so
we want to handle blocking/unblocking, etc.

Change-Id: Id01e7a24e92aecc61aa2850632b3712bb338e9f2

Support administrative block in role ASP

Change-Id: If32902575e874d2acc9a5fc12509af42ed151739

cosmetic: xua_asp_fsm: Fix wrong indentation

Change-Id: I21c13ff7bc67c060e739eae38088ac00700680de

xua_lm: Add S_INACTIVE state and improve state transitions

The S_INACTIVE state will be needed once we support administrative state
block in role ASP/(active)IPSP.

It is also useful now since sometimes we just want to leave an asp in
that state without any timer running (eg. asps in override which are not
active until the active one goes inactive).

Adding the S_INACTIVE state also allows transitioning states in role
SG, since we now have somewhere to go instead of leaving it alwas as
S_IDLE.

The S_INACTIVE state also allows staying in that state until the ASP ACT
ACK is received, moment at which the ASP is considered ACTIVE according
to specs.

Change-Id: I7e6127c150e479d0c436875b41824b3061e4fa48

xua asp block: SG: turn asp inactive when administratively blocked

This commit intentionally leaves aside implementing active->inactive
transition for asps with role ASP/(active) IPSP.

Change-Id: If8e34cfb05b8b63173b8bb01a520c058daf1fe39

xua_asp_fsm: Stop T(ack) when receiving an Error msg as response

There's no need to keep retrying retransmission if the peer already told
us something is wrong with the message.

Change-Id: Icaeb472fe67911eb15755442cf11010d3c593f1d

xua_lm: Handle M-Error.ind

Change-Id: I5bdba1ab09084b3eafcb8b4aaa9c9762b15e8583

xua_lm: Use XLM SAP to send M-SCTP_RELEASE.req

Change-Id: I582855acaced48fb838700a3013f94bee7db75ad

xua_rkm: send_reg_req: Avoid adding Routing Context with value 0

We use routing context 0 internally as "no routing context". Since the
Routing Context in the Routing KEy IE in RKM REG REQ is optional, if
local rctx is 0 (because we expect to get one allocated by the STP and
receive it through RKM REG RESP), avoid sending it with the value 0.

Change-Id: Iaa584fea3020af06951aea16e638ac6e6a84ae21

Move xUA<->LM SAP logic to its own file

Right now it's difficult to gasp what's implemented of the SAP since it
was scattered around several places.
Put all the SAP forwarding logic into its own file so it's easy to
follow and extend (eg. adding new helper functions, etc.)

Change-Id: I93d74dd7f6ea8f2e60a8bc41be15362e433e8962

sigtran_sap.h: Fix comment stating RK(DE)_REG prims are not implemented

Change-Id: I81a4df1a826fed2f24a4c7ca410b514b1251b44d

xua_lm: Fix ending up from active to idle forever

Previous transition to S_IDLE was wrong, since there's no way to recover
from that state unless the SCTP connection is closed.
Instead, try harder waiting for some new notifications to try
re-activating the ASP and see if the AS becomes active, or end up timing
out and reconnecting to re-attempt complete set of steps.

Change-Id: I5c97c0ede9c27c63f9259b43b7267a3d5d71681d

xua_lm: Use XLM SAP to send M-ASP_{UP, ACTIVE}.req

Change-Id: I5850d9ec9092ff3d099627856697a4184b0d5853

xua_lm: Handle rx of unsolicited ASPIA ACK

Related: OS#6752
Change-Id: Ica4fbaefd08cdb5c4ca2e2a608606ddbec85297c

Use osmo_ss7_asp_active() in several places

Change-Id: I172e691cb61ac68e2410b44e124737f7a4dd2775

Use osmo_ss7_asp_active() in several places

Change-Id: I172e691cb61ac68e2410b44e124737f7a4dd2775

filesystem: JsonEditor: use NamedTemporaryFile

A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it.  Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.

Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e

docs: auto-generate Card Filesystem Reference

Add a Sphinx extension (docs/pysim_fs_sphinx.py) that hooks into the
builder-inited event and generates docs/filesystem.rst before Sphinx
reads any source files.

The generated page contains a hierarchical listing of all implemented
EFs and DFs, organised by application/specification (UICC/TS 102 221,
ADF.USIM/TS 31.102, ADF.ISIM/TS 31.103, SIM/TS 51.011).  For each file,
the class docstring and any _test_de_encode / _test_decode vectors
are included as an encoding/decoding example table.

docs/filesystem.rst is fully generated at build time and is therefore
added to .gitignore.

Add tests/unittests/test_fs_coverage.py that walks all pySim.* modules
and verifies that every CardProfile, CardApplication, and standalone
CardDF subclass with EF/DF children is either listed in the SECTIONS
(and will appear in the docs) or explicitly EXCLUDED.

Change-Id: I06ddeefc6c11e04d7c24e116f3f39c8a6635856f
Related: OS#6316

filesystem: JsonEditor: offer interactive retry on error

When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'.  If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.

Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899

filesystem: edit_{binary,record}_decoded: add encode/decode examples

When invoking `edit_binary_decoded` or `edit_record_decoded`, the
temp file opened in the editor now contains the EF's encode/decode
test vectors as //-comment lines below the JSON content, similar to
how 'git commit' appends comments to the commit message template.
The comment block is stripped before JSON parsing on save,
so it has no effect on the written data.

The feature is implemented via a new module-level JsonEditor context
manager class that encapsulates the full edit cycle:

* write JSON + examples to a TemporaryDirectory
* invoke the editor
* read back, strip //-comments, parse and return the result

Change-Id: I5a046a9c7ba7e08a98cf643d5a26bc669539b38f
Related: OS#6900

xua_rkm: send_reg_req: Avoid adding Routing Context with value 0

We use routing context 0 internally as "no routing context". Since the
Routing Context in the Routing KEy IE in RKM REG REQ is optional, if
local rctx is 0 (because we expect to get one allocated by the STP and
receive it through RKM REG RESP), avoid sending it with the value 0.

Change-Id: Iaa584fea3020af06951aea16e638ac6e6a84ae21

xua_lm: Handle rx of unsolicited ASPIA ACK

Related: OS#6752
Change-Id: Ica4fbaefd08cdb5c4ca2e2a608606ddbec85297c

xua_asp_fsm: Stop T(ack) when receiving an Error msg as response

There's no need to keep retrying retransmission if the peer already told
us something is wrong with the message.

Change-Id: Icaeb472fe67911eb15755442cf11010d3c593f1d

xua_rkm: send_reg_req: Avoid adding Routing Context with value 0

We use routing context 0 internally as "no routing context". Since the
Routing Context in the Routing KEy IE in RKM REG REQ is optional, if
local rctx is 0 (because we expect to get one allocated by the STP and
receive it through RKM REG RESP), avoid sending it with the value 0.

Change-Id: Iaa584fea3020af06951aea16e638ac6e6a84ae21

xua_lm: Fix ending up from active to idle forever

Previous transition to S_IDLE was wrong, since there's no way to recover
from that state unless the SCTP connection is closed.
Instead, try harder waiting for some new notifications to try
re-activating the ASP and see if the AS becomes active, or end up timing
out and reconnecting to re-attempt complete set of steps.

Change-Id: I5c97c0ede9c27c63f9259b43b7267a3d5d71681d

m3ua: m3ua_gen_error_msg(): include Routing Context IE in cause Invalid Routing Context

As per RFC4666 3.8.1:
"""
The "Invalid Routing Context" error is sent if a message is received
from a peer with an invalid (unconfigured) Routing Context value.
For this error, the invalid Routing Context(s) MUST be included in
the Error message."
"""

Hence add the originating Routing Context IE so the error message can be
further identified.
For the same reason, also add the Diagnostic Information for the
specific cases in the switch statement, and make them more robust about
possible null pointer dereference.

Change-Id: If821109701e315d17f5334c680670ea6c7bce3bd

Move xUA<->LM SAP logic to its own file

Right now it's difficult to gasp what's implemented of the SAP since it
was scattered around several places.
Put all the SAP forwarding logic into its own file so it's easy to
follow and extend (eg. adding new helper functions, etc.)

Change-Id: I93d74dd7f6ea8f2e60a8bc41be15362e433e8962

xua_lm: Handle M-Error.ind

Change-Id: I5bdba1ab09084b3eafcb8b4aaa9c9762b15e8583

xua_as_fsm: Remove FIXME regarding queueing of msgs during state PENDING

Queueing of messags during T(r) (state PENDING) is already implemented,
see xua_as_fsm_pending() further below in the same file.

Change-Id: Ie572367493066039f7022250d94b815a4da5bd16

xua asp block: SGP/IPSP: Answer ASPAC with Err if blocked

Reject received ASPACs when the asp is administratively blocked.

Moving an already active ASP into inactive state upon user applies
adminsitrative block is not covered in this commit; it will be
implementer later.

This commit doesn't yet implement asp blocking in ASP role nor
(ASP-side) IPSP role.

Related: OS#6752
Change-Id: I23851fe05c06b6bf59b902632029a2382231427e

xua_asp_fsm: Include Routing Context IE in Err msg cuase Invalid Routing Context

As per RFC4666 3.8.1:
"""
For this error, the invalid Routing Context(s) MUST be included in
the Error message.
"""

Change-Id: Ia8bff16248967209d73cd6d5b54ac30fda283247

xua_lm: Use XLM SAP to send M-ASP_{UP, ACTIVE}.req

Change-Id: I5850d9ec9092ff3d099627856697a4184b0d5853

xua_lm: Use XLM SAP to send M-SCTP_RELEASE.req

Change-Id: I582855acaced48fb838700a3013f94bee7db75ad

sigtran_sap.h: Fix comment stating RK(DE)_REG prims are not implemented

Change-Id: I81a4df1a826fed2f24a4c7ca410b514b1251b44d

xua asp block: SG: turn asp inactive when administratively blocked

This commit intentionally leaves aside implementing active->inactive
transition for asps with role ASP/(active) IPSP.

Change-Id: If8e34cfb05b8b63173b8bb01a520c058daf1fe39

docs/conf.py: silence autosectionlabel duplicate-label warnings

sphinxarg.ext generates generic sub-headings ("Named arguments",
"Positional arguments", "Sub-commands", "General options", ...) for
every argparse command and tool.  These repeat across many files and
trigger large numbers of autosectionlabel duplicate-label warnings.

Two-pronged fix:

* `autosectionlabel_maxdepth = 3` eliminates the depth-4+ warnings
  (sub-headings inside each individual command block).
* `suppress_warnings` per file silences the residual depth-3 collisions
  ("serial reader", "decode_hex", "sub-commands", ...) that still
  appear across tool documentation files.

Cross-references into these generic argparse-generated sections are not
a supported use-case, so suppressing the warnings is appropriate.

Change-Id: I9cdf2a4f6cbd435b16b90ab668205600ffd7c3b0

docs: auto-generate Card Filesystem Reference

Add a Sphinx extension (docs/pysim_fs_sphinx.py) that hooks into the
builder-inited event and generates docs/filesystem.rst before Sphinx
reads any source files.

The generated page contains a hierarchical listing of all implemented
EFs and DFs, organised by application/specification (UICC/TS 102 221,
ADF.USIM/TS 31.102, ADF.ISIM/TS 31.103, SIM/TS 51.011).  For each file,
the class docstring and any _test_de_encode / _test_decode vectors
are included as an encoding/decoding example table.

docs/filesystem.rst is fully generated at build time and is therefore
added to .gitignore.

Change-Id: I06ddeefc6c11e04d7c24e116f3f39c8a6635856f
Related: OS#6316

filesystem: edit_{binary,record}_decoded: add encode/decode examples

When invoking `edit_binary_decoded` or `edit_record_decoded`, the
temp file opened in the editor now contains the EF's encode/decode
test vectors as //-comment lines below the JSON content, similar to
how 'git commit' appends comments to the commit message template.
The comment block is stripped before JSON parsing on save,
so it has no effect on the written data.

The feature is implemented via a new module-level JsonEditor context
manager class that encapsulates the full edit cycle:

* write JSON + examples to a TemporaryDirectory
* invoke the editor
* read back, strip //-comments, parse and return the result

Change-Id: I5a046a9c7ba7e08a98cf643d5a26bc669539b38f
Related: OS#6900

filesystem: JsonEditor: offer interactive retry on error

When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'.  If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.

Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899

filesystem: JsonEditor: use NamedTemporaryFile

A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it.  Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.

Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e

docs/conf.py: add autodoc_mock_imports for klein and twisted

The eSIM SM-DP+ server modules (`pySim.esim.es2p`, `pySim.esim.es9p`,
`pySim.esim.http_json_api`) unconditionally import optional server-side
dependencies at module level:

  pySim.esim.es2p          -- from klein import Klein
  pySim.esim.http_json_api -- from twisted.web.server import Request

Both imports fail during a docs build if the packages are absent or
broken, causing three "autodoc: failed to import" warnings and three
missing chapters in the generated manual.

Even when klein and twisted are installed, twisted 23.10.0 (the
version pulled in transitively by smpp.twisted3's `Twisted~=23.10.0`
constraint) is incompatible with Python 3.13+ because twisted.web.http
unconditionally executes `import cgi`, a module that was removed from
the standard library in Python 3.13.

Fix: add `autodoc_mock_imports = ['klein', 'twisted']` to conf.py.
Sphinx inserts mock entries into sys.modules before each autodoc import
attempt, so the modules can be imported and documented without requiring
the real packages to be importable at build time.

Change-Id: I71650466f02a6a6d150650deed167c05d2cb6e64

pysim/pcsc: do not use getProtocol for protocol selection

The documentation of the getProtocol provided by pyscard says:

"Return bit mask for the protocol of connection, or None if no
protocol set. The return value is a bit mask of
CardConnection.T0_protocol, CardConnection.T1_protocol,
CardConnection.RAW_protocol, CardConnection.T15_protocol"

This suggests that the purpose of getProtocol is not to determine
which protocols are supported. Its purpose is to determine which
protocol is currently selected (either through auto selection or
through the explicit selection made by the API user). This means
we are using getProtocol wrong.

So far this was no problem, since the auto-selected protocol
should be a supported protocol anyway. However, the automatic
protocol selection may not always return a correct result (see
bug report from THD-siegfried [1]).

Let's not trust the automatic protocol selection. Instead let's
parse the ATR and make the decision based on the TD1/TD2 bytes).

[1] https://osmocom.org/issues/6952

Related: OS#6952
Change-Id: Ib119948aa68c430e42ac84daec8b9bd542db7963

sigtran_sap.h: Document spec specifying xUA<->LM primitive

Change-Id: I14c3092a3feac5661d03b9b9114889b348b11bb1

xua_as_fsm: Remove FIXME regarding queueing of msgs during state PENDING

Queueing of messags during T(r) (state PENDING) is already implemented,
see xua_as_fsm_pending() further below in the same file.

Change-Id: Ie572367493066039f7022250d94b815a4da5bd16

xua asp block: SGP/IPSP: Answer ASPAC with Err if blocked

Reject received ASPACs when the asp is administratively blocked.

Moving an already active ASP into inactive state upon user applies
adminsitrative block is not covered in this commit; it will be
implementer later.

This commit doesn't yet implement asp blocking in ASP role nor
(ASP-side) IPSP role.

Related: OS#6752
Change-Id: I23851fe05c06b6bf59b902632029a2382231427e

xua asp block: SG: turn asp inactive when administratively blocked

This commit intentionally leaves aside implementing active->inactive
transition for asps with role ASP/(active) IPSP.

Change-Id: If8e34cfb05b8b63173b8bb01a520c058daf1fe39

pySim-read: remove import random

In pySim-read we do not have to compute any random numbers, so
we may remove random from the imports

Change-Id: Iae4ee6aafb339cc682345299b92b4ecd0bbca14e

tlv: Allow control over comprehension bit in COMPR_TLV_IE

This allows construction of a comprehension TLV with the bit set or
cleared. The default is comprehension=True to not break existing code.

```
>>> foo = Foo(decoded=b"2342", comprehension=True)
>>> bar = Foo(decoded=b"2342", comprehension=False)
```

Related: OS#6989
Change-Id: I9ca689b9b51152f3907ea470c7b42a0b12208459

tlv: Remove unused branch

0xff is checked in the first if statement and already raises a ValueError

Change-Id: Ia6a0656721a1fcaf5f16526fefe62c30b0ddb664

filesystem: JsonEditor: offer interactive retry on error

When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'.  If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.

Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899

filesystem: JsonEditor: use NamedTemporaryFile

A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it.  Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.

Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e

filesystem: edit_{binary,record}_decoded: add encode/decode examples

When invoking `edit_binary_decoded` or `edit_record_decoded`, the
temp file opened in the editor now contains the EF's encode/decode
test vectors as //-comment lines below the JSON content, similar to
how 'git commit' appends comments to the commit message template.
The comment block is stripped before JSON parsing on save,
so it has no effect on the written data.

The feature is implemented via a new module-level JsonEditor context
manager class that encapsulates the full edit cycle:

* write JSON + examples to a TemporaryDirectory
* invoke the editor
* read back, strip //-comments, parse and return the result

Change-Id: I5a046a9c7ba7e08a98cf643d5a26bc669539b38f
Related: OS#6900

docs: auto-generate Card Filesystem Reference

Add a Sphinx extension (docs/pysim_fs_sphinx.py) that hooks into the
builder-inited event and generates docs/filesystem.rst before Sphinx
reads any source files.

The generated page contains a hierarchical listing of all implemented
EFs and DFs, organised by application/specification (UICC/TS 102 221,
ADF.USIM/TS 31.102, ADF.ISIM/TS 31.103, SIM/TS 51.011).  For each file,
the class docstring and any _test_de_encode / _test_decode vectors
are included as an encoding/decoding example table.

docs/filesystem.rst is fully generated at build time and is therefore
added to .gitignore.

Change-Id: I06ddeefc6c11e04d7c24e116f3f39c8a6635856f
Related: OS#6316

stp: m3ua: Introduce test TC_adm_block_after_act

Related: OS#6752
Change-Id: I18672d228c1d15ef99039bccbb972e0b2496b545

stp: m3ua: Introduce test TC_adm_block_act_err

Related: OS#6752
Related: libosmo-sigtran.git Change-Id I23851fe05c06b6bf59b902632029a2382231427e
Change-Id: I9869ec2b71a3f6814340e181d67bab4c9266a2f4

xua asp block: SGP/IPSP: Answer ASPAC with Err if blocked

Reject received ASPACs when the asp is administratively blocked.

Moving an already active ASP into inactive state upon user applies
adminsitrative block is not covered in this commit; it will be
implementer later.

This commit doesn't yet implement asp blocking in ASP role nor
(ASP-side) IPSP role.

Related: OS#6752
Change-Id: I23851fe05c06b6bf59b902632029a2382231427e