Changes

filesystem: JsonEditor: use NamedTemporaryFile

A plain NamedTemporaryFile is sufficient here: we only need a single
file, not a directory to hold it.  Using NamedTemporaryFile is simpler
(no subdirectory to manage) and gives us a .json suffix for free,
which editors use for syntax highlighting.

Change-Id: If3b0bd0fcc90732407dbd03b9cc883f7abeb948e

filesystem: JsonEditor: offer interactive retry on error

When json.loads() fails (e.g. the user made a syntax mistake), prompt
the user with "Re-open file for editing? [y]es/[n]o:" and loop back to
the editor if they answer 'y' or 'yes'.  If the user declines, return
the original unmodified value so no write is attempted; the temp file
is still cleaned up by __exit__() in that case.

Change-Id: I9161b7becea0d8dfd3f5f740fbb253da2f061a1d
Related: OS#6899

tests: fix TransRecEF _test_de_encode to operate at file level

Previously, _test_de_encode vectors for TransRecEF subclasses were tested
via decode_record_hex()/encode_record_hex(), i.e. one record at a time,
with the decoded value being a scalar.

Switch test_de_encode_record() in TransRecEF_Test to use decode_hex() /
encode_hex() instead, so that vectors represent whole-file content
(decoded value is a list of records) -- consistent with how LinFixedEF
handles _test_de_encode.  Update all existing vectors accordingly.

Change-Id: I4a9610f9ee39833cd0c90f64f89f5fbdd6f0846d

pySim/ts_51_011.py: add multi-record test vector for EF_PL

Change-Id: I9f7a444b18056b1683cbd52a25af950125531746

docs: auto-generate Card Filesystem Reference

Add a Sphinx extension (docs/pysim_fs_sphinx.py) that hooks into the
builder-inited event and generates docs/filesystem.rst before Sphinx
reads any source files.

The generated page contains a hierarchical listing of all implemented
EFs and DFs, organised by application/specification (UICC/TS 102 221,
ADF.USIM/TS 31.102, ADF.ISIM/TS 31.103, SIM/TS 51.011).  For each file,
the class docstring and any _test_de_encode / _test_decode vectors
are included as an encoding/decoding example table.

docs/filesystem.rst is fully generated at build time and is therefore
added to .gitignore.

Add tests/unittests/test_fs_coverage.py that walks all pySim.* modules
and verifies that every CardProfile, CardApplication, and standalone
CardDF subclass with EF/DF children is either listed in the SECTIONS
(and will appear in the docs) or explicitly EXCLUDED.

Change-Id: I06ddeefc6c11e04d7c24e116f3f39c8a6635856f
Related: OS#6316

filesystem: edit_{binary,record}_decoded: add encode/decode examples

When invoking `edit_binary_decoded` or `edit_record_decoded`, the
temp file opened in the editor now contains the EF's encode/decode
test vectors as //-comment lines below the JSON content, similar to
how 'git commit' appends comments to the commit message template.
The comment block is stripped before JSON parsing on save,
so it has no effect on the written data.

The feature is implemented via a new module-level JsonEditor context
manager class that encapsulates the full edit cycle:

* write JSON + examples to a TemporaryDirectory
* invoke the editor
* read back, strip //-comments, parse and return the result

Change-Id: I5a046a9c7ba7e08a98cf643d5a26bc669539b38f
Related: OS#6900

pysim/pcsc: do not use getProtocol for protocol selection

The documentation of the getProtocol provided by pyscard says:

"Return bit mask for the protocol of connection, or None if no
protocol set. The return value is a bit mask of
CardConnection.T0_protocol, CardConnection.T1_protocol,
CardConnection.RAW_protocol, CardConnection.T15_protocol"

This suggests that the purpose of getProtocol is not to determine
which protocols are supported. Its purpose is to determine which
protocol is currently selected (either through auto selection or
through the explicit selection made by the API user). This means
we are using getProtocol wrong.

So far this was no problem, since the auto-selected protocol
should be a supported protocol anyway. However, the automatic
protocol selection may not always return a correct result (see
bug report from THD-siegfried [1]).

Let's not trust the automatic protocol selection. Instead let's
parse the ATR and make the decision based on the TD1/TD2 bytes).

[1] https://osmocom.org/issues/6952

Related: OS#6952
Change-Id: Ib119948aa68c430e42ac84daec8b9bd542db7963

docs/conf.py: silence autosectionlabel duplicate-label warnings

sphinxarg.ext generates generic sub-headings ("Named arguments",
"Positional arguments", "Sub-commands", "General options", ...) for
every argparse command and tool.  These repeat across many files and
trigger large numbers of autosectionlabel duplicate-label warnings.

Two-pronged fix:

* `autosectionlabel_maxdepth = 3` eliminates the depth-4+ warnings
  (sub-headings inside each individual command block).
* `suppress_warnings` per file silences the residual depth-3 collisions
  ("serial reader", "decode_hex", "sub-commands", ...) that still
  appear across tool documentation files.

Cross-references into these generic argparse-generated sections are not
a supported use-case, so suppressing the warnings is appropriate.

Change-Id: I9cdf2a4f6cbd435b16b90ab668205600ffd7c3b0

docs/conf.py: add autodoc_mock_imports for klein and twisted

The eSIM SM-DP+ server modules (`pySim.esim.es2p`, `pySim.esim.es9p`,
`pySim.esim.http_json_api`) unconditionally import optional server-side
dependencies at module level:

  pySim.esim.es2p          -- from klein import Klein
  pySim.esim.http_json_api -- from twisted.web.server import Request

Both imports fail during a docs build if the packages are absent or
broken, causing three "autodoc: failed to import" warnings and three
missing chapters in the generated manual.

Even when klein and twisted are installed, twisted 23.10.0 (the
version pulled in transitively by smpp.twisted3's `Twisted~=23.10.0`
constraint) is incompatible with Python 3.13+ because twisted.web.http
unconditionally executes `import cgi`, a module that was removed from
the standard library in Python 3.13.

Fix: add `autodoc_mock_imports = ['klein', 'twisted']` to conf.py.
Sphinx inserts mock entries into sys.modules before each autodoc import
attempt, so the modules can be imported and documented without requiring
the real packages to be importable at build time.

Change-Id: I71650466f02a6a6d150650deed167c05d2cb6e64

PySimLogger: add parameter to set initial log-level/verbosity

When we initialize a new PySimLogger, we always call the setup method
first and then use the set_verbose and set_level method to configure
the initial log level and the initial log verbosity. However, we
initialize the PySimLogger in all our programs the same way and we
end up with the same boilerplate code every time. Let's add a keyword
parameter to the setup method where we can pass our opts.verbose (bool)
parameter so that the setup method can do the work for the main program.

In case the caller wants a different default configuration he still can
call set_verbose and set_level methods as needed.

Change-Id: I4b8ef1e203186878910c9614a1d900d5759236a8

pySim-shell/cosmetic: remove semicolon

Change-Id: I629bacd432491211b939fcd2bed554b44ef441bc

pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument

pySim-prog and pySim-read do not integrate the pySimLogger yet. As we
may add more debug output that should not be visible on normal use, we
should ensure that the pySimLogger is correctly set up.

Change-Id: Ia2fa535fd9ce4ffa301c3f5d6f98c1f7a4716c74

pySim-read: remove import random

In pySim-read we do not have to compute any random numbers, so
we may remove random from the imports

Change-Id: Iae4ee6aafb339cc682345299b92b4ecd0bbca14e

docs/card-key-provider: fix heading levels and typo

The "ADM PIN" and "SCP02 / SCP03" sub-sections of "Field naming" used
the '~' heading character, which Sphinx resolved to level 4 - skipping
level 3 and throwing build ERRORs.  As a result, both sub-sections
had no heading at all.  Change both to '^' (level 3) to match the
other sub-sections in this file.

While at it, fix a typo: "consisting if" -> "consisting of".

Change-Id: Ia56efc7fadcc0fd62e87e63850b929d2f80851ba

docs/shell: fix copy-paste typos in editor command descriptions

Two adjacent commands (`edit_record_decoded`, `edit_binary_decoded`)
had identical copy-pasted error messages with three typos each:

  "modificatiosn" -> "modifications"
  "us the"        -> "use the"
  "comamdn"       -> "command"

Change-Id: Ie23baba4634e2cc40f81439fb11b102778aed1f6

docs/saip-tool: fix typo "insertaion" -> "insertion"

Change-Id: Ie9c9235ec964a15fab19d6ca5a83b2b1ddf07e7b

docs/legacy: fix typo "EF,IMSI" -> "EF.IMSI"

Change-Id: I1f246ec008a57b2373ed3f5531ab4166101f4dd0

docs/put_key-tutorial: fix three typos

  "verifiation"        -> "verification"
  "this is identifies" -> "this identifies" (extra word)
  "and and"            -> "and" (doubled word)

Change-Id: I1ae6b01638cc2c3dd8355ba801f85cc179ca8bd3

transport: change APDU format paradigm

Unfortunately we have mixed up the concept of TPDUs and APDUs in
earlier versions of pySim-shell. This lead to problems with
detecteding the APDU case properly (see also ISO/IEC 7816-3) and
also prevented us from adding support for T=1.

This problem has been fixed long time ago and all APDUs sent from
the pySim-shell code should be well formed and valid according to
ISO/IEC 7816-3.

To ensure that we continue to format APDUs correctly as APDUs (and
not TPDUs) we have added a mechanism to the LinkBase class that
would either raise an exception or print a warning if someone
mistakenly tries to send an APDU that is really a TPDU. Whether a
warning is printed or an exception is raised is controlled via the
apdu_strict member in the LinkBase class, which is false (print
warning only) by default.

The reason why we have implemneted the mechanism this way was
because we wanted to ensure that existing APDU scripts (pySim-shell
apdu command) keep working, even though when those scripts uses
APDUs which are formally invalid.

Sending a TPDU instead of an APDU via a T=0 link will still work
in almost all cases. This is also the reason why this problem
slipped through unnoticed for long time. However, there may still
be subtile problems araising from this practice. The root of the
problem is that it is impossible to distinguish between APDU case
3 and 4 when a TPDU instead of an APDU is sent. However in order
to handle a case 4 APDU correctly we must be able to distinguish
the APDU case correctly to handle the case correctly.
ETSI TS 102 221, section 7.3.1.1.4, clause 4 is very clear about
the fact that not (only) the status word (e.g. 61xx) but the
APDU case is what matters.

To complete the logic in LinkBaseTpdu and to maintain compatibility
(older APDU scripts), we must still be able to switch between the
'apdu_strict' mode and the non-strict mode. However, since
pySim-shell, pySim-prog and pySim-read internally use proper APDUs,
we may enable the 'apdu_strict' mode by default.

At the same time we will limit the effect of pySim-shell's
apdu_strict setable to the apdu command only. By doing so, the
bahviour of the apdu command is not altered. Users will still
have to enable the 'strict' mode explicitly. At the same time
all the internal functionality of pySim-shell will always use
the 'strict' mode.

Related: OS#6970
Change-Id: I9a531a825def318b28bf58291d811cf119003fab

global_platform: install_cap_parser: argument groups cannot be nested

pySim-shell currently does not work on systems with Python 3.14+:

  File ".../pysim/pySim/global_platform/__init__.py", line 868, in AddlShellCommands
    install_cap_parser_inst_prm_g_grp = install_cap_parser_inst_prm_g.add_argument_group()
  File "/usr/lib/python3.14/argparse.py", line 1794, in add_argument_group
    raise ValueError('argument groups cannot be nested')
  ValueError('argument groups cannot be nested')

The problem is that install_cap_parser creates a nested group inside
of mutually exclusive group.  argparse never supported group nesting
properly, so it has been deprecated since Python 3.11, and eventually
got removed in Python 3.14.

Remove group nesting, adjust the usage string, and implement the
mutual exclusiveness enforcement manually in do_install_cap().

Change-Id: Idddf72d5a745345e134b23f2f01e0257d0667579

global_platform: refactor gen_install_parameters()

gen_install_parameters() had contradictory logic: the outer guard
required all three arguments to be non-None/non-empty (making them
mutually inclusive), while the inner checks then treated each one
as optional.

Make each parameter independently optional (defaulting to None) and
remove the all-or-nothing check.  Simplify the function body to a
straightforward single-pass construction of system_specific_params.

Change-Id: I8756fb38016cdf0527fe2e21edb44381d1dc557f

docs/put_key: add tutorial that explains how to manage global platform keys

With the increased interest in using GlobalPlatform features of
UICC and eUICCs (OTA-SMS, applets, etc.), also comes an increased
interest in how the related GlobalPlatform keys can be managed
(key rotation, adding/removing keysets from/to a Security Domain).

Unfortunately, many aspects of this topic are not immediately
obvious for the average user. Let's add a tutorial that contains
some practical examples to shine some light on the topic.

Related: SYS#7881
Change-Id: I163dfedca3df572cb8442e9a4a280e6c5b00327e

pySim/EF.SMSP: fix encoding of TP-Destination Address

The TP-Destination Address in EF.SMSP uses the same encoding as the
TS-Service Centre Address field. However, even though the encoding
of both fields looks almost identical, it actually isn't.

The TS-Service Centre Address field encodes the length field as
octets required for the call_number + one octet for ton_npi.
(see also: 3GPP TS 24.011, section 8.2.5.2)

The TP-Destination Address uses the number of digits of the
call_number directly in the length field.
(see also: 3GPP TS 23.040, section 9.1.2.5)

Related: SYS#7765
Change-Id: I55c123c9e244e5a6e71a0348f5d476ef03e618e8

pySim/EF.SMSP: add an additional de_encode test for EF_SMSP

Let's add another testvector where we test what happens when we populate
none of the fields except for the tp_sc_addr.

Related: SYS#7765
Change-Id: I12b600ab17d1acfdddaffe6006095acf1a4228c9

pySim/EF.SMSP: remove superflous line break

Change-Id: Ie02e02546e708e2c339810812188bd8e8af2a720

pySim/pcsc/cosmetic: reformat comment

Change-Id: Ic04bdfbc6727cc670679c377c1afd1de53504b8f

global_platform: fix typo in SupportedTlsCipherSuitesForScp81

The attribute name is misspelled.  The BER-TLV infrastructure looks
for `_construct`; this typo means `SupportedTlsCipherSuitesForScp81`
will never decode its content.

Change-Id: I0f637951b0eeb7eca2a8b543baa737f216a935ed

global_platform: fix typo in ApplicationTemplate

The keyword argument should be `nested=`.  As written `ApplicationAID`
is silently ignored - `ApplicationTemplate` will not descend into its
nested TLVs.

Change-Id: If45dbb0c9b09fe53560d109957ce339267a9f2b0

global_platform: fix store_data() returning last chunk only

The loop builds up `response` across multiple STORE DATA blocks,
but the function returns only `data` - the response from the
*last* block.  It should return the accumulated response instead.

Change-Id: I3e15c8004d1e366e8c3896e559656622f48bb1a2

global_platform: fix s/GET/STORE/ DATA in docs

Both `do_store_data` and `store_data` have identical docstrings that
incorrectly describe the command as GET DATA.  Should be "STORE DATA".
Take a chance to fix missing space between `v2.3` and `Section`.

Change-Id: I33fc80ab8ca50fadc38217b0005eec6169c8e34e

global_platform: fix docstring for Scp03SessionKeys._get_icv()

Change-Id: I8983bc27f581295544360ba8b4ae1d28b3ea850f

ara_m: fix exceptions not being raised properly

Exceptions are meant to be thrown/raised, not returned.

Change-Id: Id799c264447e22887edcd2dc7eb991cf0af1bbfc

ara_m: fix undefined variable used in a format-string

Change-Id: I310a5d461bae2b5e4d8e07097000b079c23aa0f6

sms: fix flags_construct in SMS_DELIVER

* field `tp_rp` appears at bit positions 7 and 5
** bit 7 should be `tp_rp` (Reply Path)
** bit 5 should be `tp_sri` (Status Report Indication)
* field `tp_lp` is completely missing
** should be at bit position 3

Change-Id: I0274849f0fa07281b5e050af429ffda7d249f9e8

ota: OtaAlgo{Crypt,Auth}: fix algo_auth vs algo_crypt

* OtaAlgoCrypt.from_keyset() searches by `otak.algo_crypt`
  but the error message prints `otak.algo_auth`.  Should be
  `otak.algo_crypt` instead.

* OtaAlgoAuth.__init__() checks `algo_auth` but the error message
  prints `algo_crypt`.  Should be `otak.algo_auth` instead.

Change-Id: Ia636fffaeadc68e3f6d5b65d477e753834c95895

utils: dec_plmn(): remove redundant call

Change-Id: Ic95c3992ed57eb8fee952ec2dc7f092dd7689579

utils: DataObjectCollection.encode(): fix TypeError

`members_by_name` is a plain dictionary.  Calling it with `()` raises:

  TypeError: 'dict' object is not callable

Change-Id: I7e0c09aa7303f1506fe3a025fdc3779919dd0e6c

ts_51_011: EF.EXT[6-7]: fix typo in desc

Change-Id: I93df1c9fd8a4d588ed7ed19ec2dc1d304412fc3d

ts_31_102: fix description for EF_5GS3GPPLOCI

Change-Id: I9cf3adfce65090fedb3f0fd33c9b3d15a2c5fb8c

ts_31_102: EF_5G_PROSE_UIR: fix copy-pasted inner class name

Change-Id: I460e5ad70f35026d0d794271a4aef17323c14dfb

cdma_ruim: fix inaccurate comment for EF_AD

Change-Id: I71ea27fd30e44685ff35f49843072ca392995973

esim/saip: raise an exception properly

Change-Id: Ia3749c02120fdc16e556214d0461cbeca032447b

esim/http_json_api: allow URL rewriting

The URL used when HTTP requests are performed is defined statically
with the url_prefix passed to the constructor of JsonHttpApiClient
together with the path property in JsonHttpApiFunction.

For applications that require dynamic URLs there is no way to rewrite
the URL. Let's add a mechanism that allows API users to apply custom
URL reqriting rules by adding a rewrite_url method to
JsonHttpApiFunction. API users may then overload this method with a
custom implementation as needed.

Related: SYS#7918
Change-Id: Id2713a867079cc140517fe312189e5e2162608a5