Changes

pySim-prog/cards: fix programming of EF.SMSP

The legacy code found in legacy/cards.py does not use the modern
construct based encoder (pySim-read uses it). The card classes either
use their own implementation of update_smsp or use the generic method
provided by the SimCard class. The latter one is true for FairwavesSIM
and WavemobileSim.

Unfortunately the implementation found in the SimCard is wrong. It
adds padding at the end of the file instead of the beginning. This
completely messes up the contents of EF.SMSP for the cards using this
method. To fix this, let's use the leftpad feature provided by
the update_record. This will ensure a correct alignment of the file
contents.

Related: SYS#7765
Change-Id: Ie112418f1f1461762d61365d3863181ca6be7245

ts_51_011/EF.SMSP: fix handling of 'alpha_id' field

The field 'alpha_id' is technically not an optional field, even though
the specification describes it as optional. Once the card manufacturer
decides that the field should be present, it must be always present and
vice versa.

(see code comment for a more detailed description)

Related: SYS#7765
Change-Id: I0ec99b2648b22c56f9145345e4cd8776f9217701

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

personalization: make AlgorithmID a new EnumParam

The AlgorithmID has a few preset values, and hardly anyone knows which
is which. So instead of entering '1', '2' or '3', make it work with
prededined values 'Milenage', 'TUAK' and 'usim-test'.

Implement the enum value part abstractly in new EnumParam.

Make AlgorithmID a subclass of EnumParam and define the values as from
pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn

Related: SYS#6768
Change-Id: I71c2ec1b753c66cb577436944634f32792353240
Jenkins: skip-card-test

personalization: add get_typical_input_len() to ConfigurableParameter

The aim is to tell a user interface how wide an input text field should
be chosen to be convenient -- ideally showing the entire value in all
cases, but not too huge for fields that have no sane size limit.

Change-Id: I2568a032167a10517d4d75d8076a747be6e21890
Jenkins: skip-card-test

personalization: fix SdKey.apply_val() implementation

'securityDomain' elements are decoded to ProfileElementSD instances,
which keep higher level representations of the key data apart from the
decoded[] lists.

So far, apply_val() was dropping binary values in decoded[], which does
not work, because ProfileElementSD._pre_encode() overwrites
self.decoded[] from the higher level representation.

Implement using
- ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify
  an exsiting entry, or
- ProfileElementSD.add_key() to create a new entry.

Before this patch, SdKey parameters seemed to patch PES successfully,
but their modifications did not end up in the encoded DER.

(BTW, this does not fix any other errors that may still be present in
the various SdKey subclasses, patches coming up.)

Related: SYS#6768
Change-Id: I07dfc378705eba1318e9e8652796cbde106c6a52
Jenkins: skip-card-test

comment in uicc.py on Security Domain Keys: add SCP81

Change-Id: Ib0205880f58e78c07688b4637abd5f67ea0570d1
Jenkins: skip-card-test

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a
Jenkins: skip-card-test

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463
Jenkins: skip-card-test

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32
Jenkins: skip-card-test

contrib/jenkins.sh: separate JOB_TYPE for card tests

A separate job gives us a possibility to skip tests requiring physical
cards for specific commits that do not touch the core logic.  See the
related commits in osmo-ci.git.

Change-Id: If76d812ee43b7eb3b57fdc660c60bf31fbff5b16
Related: osmo-ci.git Ia48d1b468f65d7c2e6b4128eeac36d0f3d03c45e
Related: osmo-ci.git I986d88545f64e13cd571ba9ff56bc924822e39a0

contrib/jenkins.sh: add setup_venv()

Reduece code duplication by factoring out virtualenv setup and
activation into a shell function.

Change-Id: Ibb193d12d5502c78104ef53badc6037f08e92df1

tests: pySim-smpp2sim_test.sh: fix copy-pasted comment

Change-Id: I8167c6a3251bb6755810c96075010e920ceee8ac
Jenkins: skip-card-test

requirements: ensure safe version of PyYAML >= 5.4 (CVE-2020-1747)

PyYAML versions 5.1–5.3.1 are vulnerable to CVE-2020-1747, which allows
arbitrary code execution through yaml.FullLoader. While PyYAML 5.4+
patches this, the dependency specification (pyyaml >= 5.1) doesn't
guarantee a safe version. Let's increase the requirement to version
5.4 to ensure a safe version of is used.

This patch is based on suggestions from:
"YanTong C <chyeyantong03@gmail.com>"

Change-Id: I901c76c59e9c1bab030eab81038e04a475b32510

pySim-prog: fix Insecure PRNG for SIM Authentication Keys (CWE-338)

Root Cause:
pySim-prog.py uses Python's random module (Mersenne Twister MT19937) to
generate Ki and OPC — the root authentication keys for SIM cards. MT19937
is a deterministic PRNG that is not cryptographically secure. Its internal
state (624 × 32-bit words, 19,937 bits) can be fully recovered after
observing 624 consecutive outputs.

Impact:
1. SIM Card Cloning: An attacker who determines the PRNG state can predict
all Ki/OPC values generated before and after. With these keys, SIM cards
can be cloned.
2. Network Authentication Bypass: Ki/OPC are used in the Milenage algorithm
for 3G/4G/5G authentication. Predictable keys mean an attacker can
authenticate as any subscriber whose SIM was provisioned with the weak RNG.
3. Batch Compromise: In bulk provisioning scenarios (pySim-prog's primary
use case), hundreds or thousands of SIMs may be programmed sequentially.
Compromising one batch means recovering the PRNG state to predict all keys.

Fix:
Replace random.randrange() with os.urandom()

Change-Id: Id3e00d3ec5386f17c1525cacfc7d3f5bba43381f

filesystem: JsonEditor: use NamedTemporaryFile

A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it.  Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.

Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e

filesystem: JsonEditor: offer interactive retry on error

When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'.  If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.

Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899

tests: fix TransRecEF _test_de_encode to operate at file level

Previously, _test_de_encode vectors for TransRecEF subclasses were tested
via decode_record_hex()/encode_record_hex(), i.e. one record at a time,
with the decoded value being a scalar.

Switch test_de_encode_record() in TransRecEF_Test to use decode_hex() /
encode_hex() instead, so that vectors represent whole-file content
(decoded value is a list of records) -- consistent with how LinFixedEF
handles _test_de_encode.  Update all existing vectors accordingly.

Change-Id: I4a9610f9ee39833cd0c90f64f89f5fbdd6f0846d

pySim/ts_51_011.py: add multi-record test vector for EF_PL

Change-Id: I9f7a444b18056b1683cbd52a25af950125531746

docs: auto-generate Card Filesystem Reference

Add a Sphinx extension (docs/pysim_fs_sphinx.py) that hooks into the
builder-inited event and generates docs/filesystem.rst before Sphinx
reads any source files.

The generated page contains a hierarchical listing of all implemented
EFs and DFs, organised by application/specification (UICC/TS 102 221,
ADF.USIM/TS 31.102, ADF.ISIM/TS 31.103, SIM/TS 51.011).  For each file,
the class docstring and any _test_de_encode / _test_decode vectors
are included as an encoding/decoding example table.

docs/filesystem.rst is fully generated at build time and is therefore
added to .gitignore.

Add tests/unittests/test_fs_coverage.py that walks all pySim.* modules
and verifies that every CardProfile, CardApplication, and standalone
CardDF subclass with EF/DF children is either listed in the SECTIONS
(and will appear in the docs) or explicitly EXCLUDED.

Change-Id: I06ddeefc6c11e04d7c24e116f3f39c8a6635856f
Related: OS#6316

filesystem: edit_{binary,record}_decoded: add encode/decode examples

When invoking `edit_binary_decoded` or `edit_record_decoded`, the
temp file opened in the editor now contains the EF's encode/decode
test vectors as //-comment lines below the JSON content, similar to
how 'git commit' appends comments to the commit message template.
The comment block is stripped before JSON parsing on save,
so it has no effect on the written data.

The feature is implemented via a new module-level JsonEditor context
manager class that encapsulates the full edit cycle:

* write JSON + examples to a TemporaryDirectory
* invoke the editor
* read back, strip //-comments, parse and return the result

Change-Id: I5a046a9c7ba7e08a98cf643d5a26bc669539b38f
Related: OS#6900

pysim/pcsc: do not use getProtocol for protocol selection

The documentation of the getProtocol provided by pyscard says:

"Return bit mask for the protocol of connection, or None if no
protocol set. The return value is a bit mask of
CardConnection.T0_protocol, CardConnection.T1_protocol,
CardConnection.RAW_protocol, CardConnection.T15_protocol"

This suggests that the purpose of getProtocol is not to determine
which protocols are supported. Its purpose is to determine which
protocol is currently selected (either through auto selection or
through the explicit selection made by the API user). This means
we are using getProtocol wrong.

So far this was no problem, since the auto-selected protocol
should be a supported protocol anyway. However, the automatic
protocol selection may not always return a correct result (see
bug report from THD-siegfried [1]).

Let's not trust the automatic protocol selection. Instead let's
parse the ATR and make the decision based on the TD1/TD2 bytes).

[1] https://osmocom.org/issues/6952

Related: OS#6952
Change-Id: Ib119948aa68c430e42ac84daec8b9bd542db7963

docs/conf.py: silence autosectionlabel duplicate-label warnings

sphinxarg.ext generates generic sub-headings ("Named arguments",
"Positional arguments", "Sub-commands", "General options", ...) for
every argparse command and tool.  These repeat across many files and
trigger large numbers of autosectionlabel duplicate-label warnings.

Two-pronged fix:

* `autosectionlabel_maxdepth = 3` eliminates the depth-4+ warnings
  (sub-headings inside each individual command block).
* `suppress_warnings` per file silences the residual depth-3 collisions
  ("serial reader", "decode_hex", "sub-commands", ...) that still
  appear across tool documentation files.

Cross-references into these generic argparse-generated sections are not
a supported use-case, so suppressing the warnings is appropriate.

Change-Id: I9cdf2a4f6cbd435b16b90ab668205600ffd7c3b0

docs/conf.py: add autodoc_mock_imports for klein and twisted

The eSIM SM-DP+ server modules (`pySim.esim.es2p`, `pySim.esim.es9p`,
`pySim.esim.http_json_api`) unconditionally import optional server-side
dependencies at module level:

  pySim.esim.es2p          -- from klein import Klein
  pySim.esim.http_json_api -- from twisted.web.server import Request

Both imports fail during a docs build if the packages are absent or
broken, causing three "autodoc: failed to import" warnings and three
missing chapters in the generated manual.

Even when klein and twisted are installed, twisted 23.10.0 (the
version pulled in transitively by smpp.twisted3's `Twisted~=23.10.0`
constraint) is incompatible with Python 3.13+ because twisted.web.http
unconditionally executes `import cgi`, a module that was removed from
the standard library in Python 3.13.

Fix: add `autodoc_mock_imports = ['klein', 'twisted']` to conf.py.
Sphinx inserts mock entries into sys.modules before each autodoc import
attempt, so the modules can be imported and documented without requiring
the real packages to be importable at build time.

Change-Id: I71650466f02a6a6d150650deed167c05d2cb6e64