Changes

requirements: ensure safe version of PyYAML >= 5.4 (CVE-2020-1747)

PyYAML versions 5.1–5.3.1 are vulnerable to CVE-2020-1747, which allows
arbitrary code execution through yaml.FullLoader. While PyYAML 5.4+
patches this, the dependency specification (pyyaml >= 5.1) doesn't
guarantee a safe version. Let's increase the requirement to version
5.4 to ensure a safe version of is used.

This patch is based on suggestions from:
"YanTong C <chyeyantong03@gmail.com>"

Change-Id: I901c76c59e9c1bab030eab81038e04a475b32510

pySim-prog: fix Insecure PRNG for SIM Authentication Keys (CWE-338)

Root Cause:
pySim-prog.py uses Python's random module (Mersenne Twister MT19937) to
generate Ki and OPC — the root authentication keys for SIM cards. MT19937
is a deterministic PRNG that is not cryptographically secure. Its internal
state (624 × 32-bit words, 19,937 bits) can be fully recovered after
observing 624 consecutive outputs.

Impact:
1. SIM Card Cloning: An attacker who determines the PRNG state can predict
all Ki/OPC values generated before and after. With these keys, SIM cards
can be cloned.
2. Network Authentication Bypass: Ki/OPC are used in the Milenage algorithm
for 3G/4G/5G authentication. Predictable keys mean an attacker can
authenticate as any subscriber whose SIM was provisioned with the weak RNG.
3. Batch Compromise: In bulk provisioning scenarios (pySim-prog's primary
use case), hundreds or thousands of SIMs may be programmed sequentially.
Compromising one batch means recovering the PRNG state to predict all keys.

Fix:
Replace random.randrange() with os.urandom()

Change-Id: Id3e00d3ec5386f17c1525cacfc7d3f5bba43381f

filesystem: JsonEditor: use NamedTemporaryFile

A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it.  Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.

Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e

filesystem: JsonEditor: offer interactive retry on error

When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'.  If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.

Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899

tests: fix TransRecEF _test_de_encode to operate at file level

Previously, _test_de_encode vectors for TransRecEF subclasses were tested
via decode_record_hex()/encode_record_hex(), i.e. one record at a time,
with the decoded value being a scalar.

Switch test_de_encode_record() in TransRecEF_Test to use decode_hex() /
encode_hex() instead, so that vectors represent whole-file content
(decoded value is a list of records) -- consistent with how LinFixedEF
handles _test_de_encode.  Update all existing vectors accordingly.

Change-Id: I4a9610f9ee39833cd0c90f64f89f5fbdd6f0846d

pySim/ts_51_011.py: add multi-record test vector for EF_PL

Change-Id: I9f7a444b18056b1683cbd52a25af950125531746

docs: auto-generate Card Filesystem Reference

Add a Sphinx extension (docs/pysim_fs_sphinx.py) that hooks into the
builder-inited event and generates docs/filesystem.rst before Sphinx
reads any source files.

The generated page contains a hierarchical listing of all implemented
EFs and DFs, organised by application/specification (UICC/TS 102 221,
ADF.USIM/TS 31.102, ADF.ISIM/TS 31.103, SIM/TS 51.011).  For each file,
the class docstring and any _test_de_encode / _test_decode vectors
are included as an encoding/decoding example table.

docs/filesystem.rst is fully generated at build time and is therefore
added to .gitignore.

Add tests/unittests/test_fs_coverage.py that walks all pySim.* modules
and verifies that every CardProfile, CardApplication, and standalone
CardDF subclass with EF/DF children is either listed in the SECTIONS
(and will appear in the docs) or explicitly EXCLUDED.

Change-Id: I06ddeefc6c11e04d7c24e116f3f39c8a6635856f
Related: OS#6316

filesystem: edit_{binary,record}_decoded: add encode/decode examples

When invoking `edit_binary_decoded` or `edit_record_decoded`, the
temp file opened in the editor now contains the EF's encode/decode
test vectors as //-comment lines below the JSON content, similar to
how 'git commit' appends comments to the commit message template.
The comment block is stripped before JSON parsing on save,
so it has no effect on the written data.

The feature is implemented via a new module-level JsonEditor context
manager class that encapsulates the full edit cycle:

* write JSON + examples to a TemporaryDirectory
* invoke the editor
* read back, strip //-comments, parse and return the result

Change-Id: I5a046a9c7ba7e08a98cf643d5a26bc669539b38f
Related: OS#6900

pysim/pcsc: do not use getProtocol for protocol selection

The documentation of the getProtocol provided by pyscard says:

"Return bit mask for the protocol of connection, or None if no
protocol set. The return value is a bit mask of
CardConnection.T0_protocol, CardConnection.T1_protocol,
CardConnection.RAW_protocol, CardConnection.T15_protocol"

This suggests that the purpose of getProtocol is not to determine
which protocols are supported. Its purpose is to determine which
protocol is currently selected (either through auto selection or
through the explicit selection made by the API user). This means
we are using getProtocol wrong.

So far this was no problem, since the auto-selected protocol
should be a supported protocol anyway. However, the automatic
protocol selection may not always return a correct result (see
bug report from THD-siegfried [1]).

Let's not trust the automatic protocol selection. Instead let's
parse the ATR and make the decision based on the TD1/TD2 bytes).

[1] https://osmocom.org/issues/6952

Related: OS#6952
Change-Id: Ib119948aa68c430e42ac84daec8b9bd542db7963

docs/conf.py: silence autosectionlabel duplicate-label warnings

sphinxarg.ext generates generic sub-headings ("Named arguments",
"Positional arguments", "Sub-commands", "General options", ...) for
every argparse command and tool.  These repeat across many files and
trigger large numbers of autosectionlabel duplicate-label warnings.

Two-pronged fix:

* `autosectionlabel_maxdepth = 3` eliminates the depth-4+ warnings
  (sub-headings inside each individual command block).
* `suppress_warnings` per file silences the residual depth-3 collisions
  ("serial reader", "decode_hex", "sub-commands", ...) that still
  appear across tool documentation files.

Cross-references into these generic argparse-generated sections are not
a supported use-case, so suppressing the warnings is appropriate.

Change-Id: I9cdf2a4f6cbd435b16b90ab668205600ffd7c3b0

docs/conf.py: add autodoc_mock_imports for klein and twisted

The eSIM SM-DP+ server modules (`pySim.esim.es2p`, `pySim.esim.es9p`,
`pySim.esim.http_json_api`) unconditionally import optional server-side
dependencies at module level:

  pySim.esim.es2p          -- from klein import Klein
  pySim.esim.http_json_api -- from twisted.web.server import Request

Both imports fail during a docs build if the packages are absent or
broken, causing three "autodoc: failed to import" warnings and three
missing chapters in the generated manual.

Even when klein and twisted are installed, twisted 23.10.0 (the
version pulled in transitively by smpp.twisted3's `Twisted~=23.10.0`
constraint) is incompatible with Python 3.13+ because twisted.web.http
unconditionally executes `import cgi`, a module that was removed from
the standard library in Python 3.13.

Fix: add `autodoc_mock_imports = ['klein', 'twisted']` to conf.py.
Sphinx inserts mock entries into sys.modules before each autodoc import
attempt, so the modules can be imported and documented without requiring
the real packages to be importable at build time.

Change-Id: I71650466f02a6a6d150650deed167c05d2cb6e64

PySimLogger: add parameter to set initial log-level/verbosity

When we initialize a new PySimLogger, we always call the setup method
first and then use the set_verbose and set_level method to configure
the initial log level and the initial log verbosity. However, we
initialize the PySimLogger in all our programs the same way and we
end up with the same boilerplate code every time. Let's add a keyword
parameter to the setup method where we can pass our opts.verbose (bool)
parameter so that the setup method can do the work for the main program.

In case the caller wants a different default configuration he still can
call set_verbose and set_level methods as needed.

Change-Id: I4b8ef1e203186878910c9614a1d900d5759236a8

pySim-shell/cosmetic: remove semicolon

Change-Id: I629bacd432491211b939fcd2bed554b44ef441bc

pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument

pySim-prog and pySim-read do not integrate the pySimLogger yet. As we
may add more debug output that should not be visible on normal use, we
should ensure that the pySimLogger is correctly set up.

Change-Id: Ia2fa535fd9ce4ffa301c3f5d6f98c1f7a4716c74

pySim-read: remove import random

In pySim-read we do not have to compute any random numbers, so
we may remove random from the imports

Change-Id: Iae4ee6aafb339cc682345299b92b4ecd0bbca14e

docs/card-key-provider: fix heading levels and typo

The "ADM PIN" and "SCP02 / SCP03" sub-sections of "Field naming" used
the '~' heading character, which Sphinx resolved to level 4 - skipping
level 3 and throwing build ERRORs.  As a result, both sub-sections
had no heading at all.  Change both to '^' (level 3) to match the
other sub-sections in this file.

While at it, fix a typo: "consisting if" -> "consisting of".

Change-Id: Ia56efc7fadcc0fd62e87e63850b929d2f80851ba

docs/shell: fix copy-paste typos in editor command descriptions

Two adjacent commands (`edit_record_decoded`, `edit_binary_decoded`)
had identical copy-pasted error messages with three typos each:

  "modificatiosn" -> "modifications"
  "us the"        -> "use the"
  "comamdn"       -> "command"

Change-Id: Ie23baba4634e2cc40f81439fb11b102778aed1f6

docs/saip-tool: fix typo "insertaion" -> "insertion"

Change-Id: Ie9c9235ec964a15fab19d6ca5a83b2b1ddf07e7b

docs/legacy: fix typo "EF,IMSI" -> "EF.IMSI"

Change-Id: I1f246ec008a57b2373ed3f5531ab4166101f4dd0

docs/put_key-tutorial: fix three typos

  "verifiation"        -> "verification"
  "this is identifies" -> "this identifies" (extra word)
  "and and"            -> "and" (doubled word)

Change-Id: I1ae6b01638cc2c3dd8355ba801f85cc179ca8bd3

transport: change APDU format paradigm

Unfortunately we have mixed up the concept of TPDUs and APDUs in
earlier versions of pySim-shell. This lead to problems with
detecteding the APDU case properly (see also ISO/IEC 7816-3) and
also prevented us from adding support for T=1.

This problem has been fixed long time ago and all APDUs sent from
the pySim-shell code should be well formed and valid according to
ISO/IEC 7816-3.

To ensure that we continue to format APDUs correctly as APDUs (and
not TPDUs) we have added a mechanism to the LinkBase class that
would either raise an exception or print a warning if someone
mistakenly tries to send an APDU that is really a TPDU. Whether a
warning is printed or an exception is raised is controlled via the
apdu_strict member in the LinkBase class, which is false (print
warning only) by default.

The reason why we have implemneted the mechanism this way was
because we wanted to ensure that existing APDU scripts (pySim-shell
apdu command) keep working, even though when those scripts uses
APDUs which are formally invalid.

Sending a TPDU instead of an APDU via a T=0 link will still work
in almost all cases. This is also the reason why this problem
slipped through unnoticed for long time. However, there may still
be subtile problems araising from this practice. The root of the
problem is that it is impossible to distinguish between APDU case
3 and 4 when a TPDU instead of an APDU is sent. However in order
to handle a case 4 APDU correctly we must be able to distinguish
the APDU case correctly to handle the case correctly.
ETSI TS 102 221, section 7.3.1.1.4, clause 4 is very clear about
the fact that not (only) the status word (e.g. 61xx) but the
APDU case is what matters.

To complete the logic in LinkBaseTpdu and to maintain compatibility
(older APDU scripts), we must still be able to switch between the
'apdu_strict' mode and the non-strict mode. However, since
pySim-shell, pySim-prog and pySim-read internally use proper APDUs,
we may enable the 'apdu_strict' mode by default.

At the same time we will limit the effect of pySim-shell's
apdu_strict setable to the apdu command only. By doing so, the
bahviour of the apdu command is not altered. Users will still
have to enable the 'strict' mode explicitly. At the same time
all the internal functionality of pySim-shell will always use
the 'strict' mode.

Related: OS#6970
Change-Id: I9a531a825def318b28bf58291d811cf119003fab

global_platform: install_cap_parser: argument groups cannot be nested

pySim-shell currently does not work on systems with Python 3.14+:

  File ".../pysim/pySim/global_platform/__init__.py", line 868, in AddlShellCommands
    install_cap_parser_inst_prm_g_grp = install_cap_parser_inst_prm_g.add_argument_group()
  File "/usr/lib/python3.14/argparse.py", line 1794, in add_argument_group
    raise ValueError('argument groups cannot be nested')
  ValueError('argument groups cannot be nested')

The problem is that install_cap_parser creates a nested group inside
of mutually exclusive group.  argparse never supported group nesting
properly, so it has been deprecated since Python 3.11, and eventually
got removed in Python 3.14.

Remove group nesting, adjust the usage string, and implement the
mutual exclusiveness enforcement manually in do_install_cap().

Change-Id: Idddf72d5a745345e134b23f2f01e0257d0667579

global_platform: refactor gen_install_parameters()

gen_install_parameters() had contradictory logic: the outer guard
required all three arguments to be non-None/non-empty (making them
mutually inclusive), while the inner checks then treated each one
as optional.

Make each parameter independently optional (defaulting to None) and
remove the all-or-nothing check.  Simplify the function body to a
straightforward single-pass construction of system_specific_params.

Change-Id: I8756fb38016cdf0527fe2e21edb44381d1dc557f

docs/put_key: add tutorial that explains how to manage global platform keys

With the increased interest in using GlobalPlatform features of
UICC and eUICCs (OTA-SMS, applets, etc.), also comes an increased
interest in how the related GlobalPlatform keys can be managed
(key rotation, adding/removing keysets from/to a Security Domain).

Unfortunately, many aspects of this topic are not immediately
obvious for the average user. Let's add a tutorial that contains
some practical examples to shine some light on the topic.

Related: SYS#7881
Change-Id: I163dfedca3df572cb8442e9a4a280e6c5b00327e

pySim/EF.SMSP: fix encoding of TP-Destination Address

The TP-Destination Address in EF.SMSP uses the same encoding as the
TS-Service Centre Address field. However, even though the encoding
of both fields looks almost identical, it actually isn't.

The TS-Service Centre Address field encodes the length field as
octets required for the call_number + one octet for ton_npi.
(see also: 3GPP TS 24.011, section 8.2.5.2)

The TP-Destination Address uses the number of digits of the
call_number directly in the length field.
(see also: 3GPP TS 23.040, section 9.1.2.5)

Related: SYS#7765
Change-Id: I55c123c9e244e5a6e71a0348f5d476ef03e618e8

pySim/EF.SMSP: add an additional de_encode test for EF_SMSP

Let's add another testvector where we test what happens when we populate
none of the fields except for the tp_sc_addr.

Related: SYS#7765
Change-Id: I12b600ab17d1acfdddaffe6006095acf1a4228c9

pySim/EF.SMSP: remove superflous line break

Change-Id: Ie02e02546e708e2c339810812188bd8e8af2a720

pySim/pcsc/cosmetic: reformat comment

Change-Id: Ic04bdfbc6727cc670679c377c1afd1de53504b8f