pySim.esim.saip.File: Support pinStatusTemplateDO + lcsi
The tuples defining a DF or ADF in an eSIM template must contain a pinStatusTemplateDO. When parsing tuples into a File() instance, we must save it, and re-create it at the time we re-encode that file.
Same applies to the lcsi (life cycle state indicator), which may optionally exist for any file.
The URL used when HTTP requests are performed is defined statically with the url_prefix passed to the constructor of JsonHttpApiClient together with the path property in JsonHttpApiFunction.
For applications that require dynamic URLs there is no way to rewrite the URL. Let's add a mechanism that allows API users to apply custom URL reqriting rules by adding a rewrite_url method to JsonHttpApiFunction. API users may then overload this method with a custom implementation as needed.
global_platform: fix typo in SupportedTlsCipherSuitesForScp81
The attribute name is misspelled. The BER-TLV infrastructure looks for `_construct`; this typo means `SupportedTlsCipherSuitesForScp81` will never decode its content.
The keyword argument should be `nested=`. As written `ApplicationAID` is silently ignored - `ApplicationTemplate` will not descend into its nested TLVs.
global_platform: fix store_data() returning last chunk only
The loop builds up `response` across multiple STORE DATA blocks, but the function returns only `data` - the response from the *last* block. It should return the accumulated response instead.
Both `do_store_data` and `store_data` have identical docstrings that incorrectly describe the command as GET DATA. Should be "STORE DATA". Take a chance to fix missing space between `v2.3` and `Section`.
* field `tp_rp` appears at bit positions 7 and 5 ** bit 7 should be `tp_rp` (Reply Path) ** bit 5 should be `tp_sri` (Status Report Indication) * field `tp_lp` is completely missing ** should be at bit position 3
pySim/EF.SMSP: fix encoding of TP-Destination Address
The TP-Destination Address in EF.SMSP uses the same encoding as the TS-Service Centre Address field. However, even though the encoding of both fields looks almost identical, it actually isn't.
The TS-Service Centre Address field encodes the length field as octets required for the call_number + one octet for ton_npi. (see also: 3GPP TS 24.011, section 8.2.5.2)
The TP-Destination Address uses the number of digits of the call_number directly in the length field. (see also: 3GPP TS 23.040, section 9.1.2.5)
global_platform: install_cap_parser: argument groups cannot be nested
pySim-shell currently does not work on systems with Python 3.14+:
File ".../pysim/pySim/global_platform/__init__.py", line 868, in AddlShellCommands install_cap_parser_inst_prm_g_grp = install_cap_parser_inst_prm_g.add_argument_group() File "/usr/lib/python3.14/argparse.py", line 1794, in add_argument_group raise ValueError('argument groups cannot be nested') ValueError('argument groups cannot be nested')
The problem is that install_cap_parser creates a nested group inside of mutually exclusive group. argparse never supported group nesting properly, so it has been deprecated since Python 3.11, and eventually got removed in Python 3.14.
Remove group nesting, adjust the usage string, and implement the mutual exclusiveness enforcement manually in do_install_cap().
gen_install_parameters() had contradictory logic: the outer guard required all three arguments to be non-None/non-empty (making them mutually inclusive), while the inner checks then treated each one as optional.
Make each parameter independently optional (defaulting to None) and remove the all-or-nothing check. Simplify the function body to a straightforward single-pass construction of system_specific_params.
docs/put_key: add tutorial that explains how to manage global platform keys
With the increased interest in using GlobalPlatform features of UICC and eUICCs (OTA-SMS, applets, etc.), also comes an increased interest in how the related GlobalPlatform keys can be managed (key rotation, adding/removing keysets from/to a Security Domain).
Unfortunately, many aspects of this topic are not immediately obvious for the average user. Let's add a tutorial that contains some practical examples to shine some light on the topic.
Unfortunately we have mixed up the concept of TPDUs and APDUs in earlier versions of pySim-shell. This lead to problems with detecteding the APDU case properly (see also ISO/IEC 7816-3) and also prevented us from adding support for T=1.
This problem has been fixed long time ago and all APDUs sent from the pySim-shell code should be well formed and valid according to ISO/IEC 7816-3.
To ensure that we continue to format APDUs correctly as APDUs (and not TPDUs) we have added a mechanism to the LinkBase class that would either raise an exception or print a warning if someone mistakenly tries to send an APDU that is really a TPDU. Whether a warning is printed or an exception is raised is controlled via the apdu_strict member in the LinkBase class, which is false (print warning only) by default.
The reason why we have implemneted the mechanism this way was because we wanted to ensure that existing APDU scripts (pySim-shell apdu command) keep working, even though when those scripts uses APDUs which are formally invalid.
Sending a TPDU instead of an APDU via a T=0 link will still work in almost all cases. This is also the reason why this problem slipped through unnoticed for long time. However, there may still be subtile problems araising from this practice. The root of the problem is that it is impossible to distinguish between APDU case 3 and 4 when a TPDU instead of an APDU is sent. However in order to handle a case 4 APDU correctly we must be able to distinguish the APDU case correctly to handle the case correctly. ETSI TS 102 221, section 7.3.1.1.4, clause 4 is very clear about the fact that not (only) the status word (e.g. 61xx) but the APDU case is what matters.
To complete the logic in LinkBaseTpdu and to maintain compatibility (older APDU scripts), we must still be able to switch between the 'apdu_strict' mode and the non-strict mode. However, since pySim-shell, pySim-prog and pySim-read internally use proper APDUs, we may enable the 'apdu_strict' mode by default.
At the same time we will limit the effect of pySim-shell's apdu_strict setable to the apdu command only. By doing so, the bahviour of the apdu command is not altered. Users will still have to enable the 'strict' mode explicitly. At the same time all the internal functionality of pySim-shell will always use the 'strict' mode.
docs/card-key-provider: fix heading levels and typo
The "ADM PIN" and "SCP02 / SCP03" sub-sections of "Field naming" used the '~' heading character, which Sphinx resolved to level 4 - skipping level 3 and throwing build ERRORs. As a result, both sub-sections had no heading at all. Change both to '^' (level 3) to match the other sub-sections in this file.
While at it, fix a typo: "consisting if" -> "consisting of".
PySimLogger: add parameter to set initial log-level/verbosity
When we initialize a new PySimLogger, we always call the setup method first and then use the set_verbose and set_level method to configure the initial log level and the initial log verbosity. However, we initialize the PySimLogger in all our programs the same way and we end up with the same boilerplate code every time. Let's add a keyword parameter to the setup method where we can pass our opts.verbose (bool) parameter so that the setup method can do the work for the main program.
In case the caller wants a different default configuration he still can call set_verbose and set_level methods as needed.
pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument
pySim-prog and pySim-read do not integrate the pySimLogger yet. As we may add more debug output that should not be visible on normal use, we should ensure that the pySimLogger is correctly set up.
sphinxarg.ext generates generic sub-headings ("Named arguments", "Positional arguments", "Sub-commands", "General options", ...) for every argparse command and tool. These repeat across many files and trigger large numbers of autosectionlabel duplicate-label warnings.
Two-pronged fix:
* `autosectionlabel_maxdepth = 3` eliminates the depth-4+ warnings (sub-headings inside each individual command block). * `suppress_warnings` per file silences the residual depth-3 collisions ("serial reader", "decode_hex", "sub-commands", ...) that still appear across tool documentation files.
Cross-references into these generic argparse-generated sections are not a supported use-case, so suppressing the warnings is appropriate.
docs/conf.py: add autodoc_mock_imports for klein and twisted
The eSIM SM-DP+ server modules (`pySim.esim.es2p`, `pySim.esim.es9p`, `pySim.esim.http_json_api`) unconditionally import optional server-side dependencies at module level:
pySim.esim.es2p -- from klein import Klein pySim.esim.http_json_api -- from twisted.web.server import Request
Both imports fail during a docs build if the packages are absent or broken, causing three "autodoc: failed to import" warnings and three missing chapters in the generated manual.
Even when klein and twisted are installed, twisted 23.10.0 (the version pulled in transitively by smpp.twisted3's `Twisted~=23.10.0` constraint) is incompatible with Python 3.13+ because twisted.web.http unconditionally executes `import cgi`, a module that was removed from the standard library in Python 3.13.
Fix: add `autodoc_mock_imports = ['klein', 'twisted']` to conf.py. Sphinx inserts mock entries into sys.modules before each autodoc import attempt, so the modules can be imported and documented without requiring the real packages to be importable at build time.
pysim/pcsc: do not use getProtocol for protocol selection
The documentation of the getProtocol provided by pyscard says:
"Return bit mask for the protocol of connection, or None if no protocol set. The return value is a bit mask of CardConnection.T0_protocol, CardConnection.T1_protocol, CardConnection.RAW_protocol, CardConnection.T15_protocol"
This suggests that the purpose of getProtocol is not to determine which protocols are supported. Its purpose is to determine which protocol is currently selected (either through auto selection or through the explicit selection made by the API user). This means we are using getProtocol wrong.
So far this was no problem, since the auto-selected protocol should be a supported protocol anyway. However, the automatic protocol selection may not always return a correct result (see bug report from THD-siegfried [1]).
Let's not trust the automatic protocol selection. Instead let's parse the ATR and make the decision based on the TD1/TD2 bytes).
Add a Sphinx extension (docs/pysim_fs_sphinx.py) that hooks into the builder-inited event and generates docs/filesystem.rst before Sphinx reads any source files.
The generated page contains a hierarchical listing of all implemented EFs and DFs, organised by application/specification (UICC/TS 102 221, ADF.USIM/TS 31.102, ADF.ISIM/TS 31.103, SIM/TS 51.011). For each file, the class docstring and any _test_de_encode / _test_decode vectors are included as an encoding/decoding example table.
docs/filesystem.rst is fully generated at build time and is therefore added to .gitignore.
Add tests/unittests/test_fs_coverage.py that walks all pySim.* modules and verifies that every CardProfile, CardApplication, and standalone CardDF subclass with EF/DF children is either listed in the SECTIONS (and will appear in the docs) or explicitly EXCLUDED.
When invoking `edit_binary_decoded` or `edit_record_decoded`, the temp file opened in the editor now contains the EF's encode/decode test vectors as //-comment lines below the JSON content, similar to how 'git commit' appends comments to the commit message template. The comment block is stripped before JSON parsing on save, so it has no effect on the written data.
The feature is implemented via a new module-level JsonEditor context manager class that encapsulates the full edit cycle:
* write JSON + examples to a TemporaryDirectory * invoke the editor * read back, strip //-comments, parse and return the result
A plain NamedTemporaryFile is sufficient here: we only need a single file, not a directory to hold it. Using NamedTemporaryFile is simpler (no subdirectory to manage) and gives us a .json suffix for free, which editors use for syntax highlighting.
filesystem: JsonEditor: offer interactive retry on error
When json.loads() fails (e.g. the user made a syntax mistake), prompt the user with "Re-open file for editing? [y]es/[n]o:" and loop back to the editor if they answer 'y' or 'yes'. If the user declines, return the original unmodified value so no write is attempted; the temp file is still cleaned up by __exit__() in that case.
tests: fix TransRecEF _test_de_encode to operate at file level
Previously, _test_de_encode vectors for TransRecEF subclasses were tested via decode_record_hex()/encode_record_hex(), i.e. one record at a time, with the decoded value being a scalar.
Switch test_de_encode_record() in TransRecEF_Test to use decode_hex() / encode_hex() instead, so that vectors represent whole-file content (decoded value is a list of records) -- consistent with how LinFixedEF handles _test_de_encode. Update all existing vectors accordingly.
The current behavior we implement in the method __send_apdu_T0 is incomplete. Some details discussed in ETSI TS 102 221, section 7.3.1.1.4, clause 4 seem to be not fully implemented. We may also end up sending a GET RESPONSE in other APDU cases than case 4 (the only case that uses the GET RESPONSE command).
