Changes

pysim/pcsc: do not use getProtocol for protocol selection

The documentation of the getProtocol provided by pyscard says:

"Return bit mask for the protocol of connection, or None if no
protocol set. The return value is a bit mask of
CardConnection.T0_protocol, CardConnection.T1_protocol,
CardConnection.RAW_protocol, CardConnection.T15_protocol"

This suggests that the purpose of getProtocol is not to determine
which protocols are supported. Its purpose is to determine which
protocol is currently selected (either through auto selection or
through the explicit selection made by the API user). This means
we are using getProtocol wrong.

So far this was no problem, since the auto-selected protocol
should be a supported protocol anyway. However, the automatic
protocol selection may not always return a correct result (see
bug report from THD-siegfried [1]).

Let's not trust the automatic protocol selection. Instead let's
parse the ATR and make the decision based on the TD1/TD2 bytes).

[1] https://osmocom.org/issues/6952

Related: OS#6952
Change-Id: Ib119948aa68c430e42ac84daec8b9bd542db7963

pySim/pcsc/cosmetic: reformat comment

Change-Id: Ic04bdfbc6727cc670679c377c1afd1de53504b8f

pySim/EF.SMSP: fix encoding of TP-Destination Address

The TP-Destination Address in EF.SMSP uses the same encoding as the
TS-Service Centre Address field. However, even though the encoding
of both fields looks almost identical, it actually isn't.

The TS-Service Centre Address field encodes the length field as
octets required for the call_number + one octet for ton_npi.
(see also: 3GPP TS 24.011, section 8.2.5.2)

The TP-Destination Address uses the number of digits of the
call_number directly in the length field.
(see also: 3GPP TS 23.040, section 9.1.2.5)

Related: SYS#7765
Change-Id: I55c123c9e244e5a6e71a0348f5d476ef03e618e8

pySim/EF.SMSP: remove superflous line break

Change-Id: Ie02e02546e708e2c339810812188bd8e8af2a720

pySim/EF.SMSP: add an additional de_encode test for EF_SMSP

Let's add another testvector where we test what happens when we populate
none of the fields except for the tp_sc_addr.

Related: SYS#7765
Change-Id: I12b600ab17d1acfdddaffe6006095acf1a4228c9

transport: change APDU format paradigm

Unfortunately we have mixed up the concept of TPDUs and APDUs in
earlier versions of pySim-shell. This lead to problems with
detecteding the APDU case properly (see also ISO/IEC 7816-3) and
also prevented us from adding support for T=1.

This problem has been fixed long time ago and all APDUs sent from
the pySim-shell code should be well formed and valid according to
ISO/IEC 7816-3.

To ensure that we continue to format APDUs correctly as APDUs (and
not TPDUs) we have added a mechanism to the LinkBase class that
would either raise an exception or print a warning if someone
mistakenly tries to send an APDU that is really a TPDU. Whether a
warning is printed or an exception is raised is controlled via the
apdu_strict member in the LinkBase class, which is false (print
warning only) by default.

The reason why we have implemneted the mechanism this way was
because we wanted to ensure that existing APDU scripts (pySim-shell
apdu command) keep working, even though when those scripts uses
APDUs which are formally invalid.

Sending a TPDU instead of an APDU via a T=0 link will still work
in almost all cases. This is also the reason why this problem
slipped through unnoticed for long time. However, there may still
be subtile problems araising from this practice. The root of the
problem is that it is impossible to distinguish between APDU case
3 and 4 when a TPDU instead of an APDU is sent. However in order
to handle a case 4 APDU correctly we must be able to distinguish
the APDU case correctly to handle the case correctly.
ETSI TS 102 221, section 7.3.1.1.4, clause 4 is very clear about
the fact that not (only) the status word (e.g. 61xx) but the
APDU case is what matters.

To complete the logic in LinkBaseTpdu and to maintain compatibility
(older APDU scripts), we must still be able to switch between the
'apdu_strict' mode and the non-strict mode. However, since
pySim-shell, pySim-prog and pySim-read internally use proper APDUs,
we may enable the 'apdu_strict' mode by default.

At the same time we will limit the effect of pySim-shell's
apdu_strict setable to the apdu command only. By doing so, the
bahviour of the apdu command is not altered. Users will still
have to enable the 'strict' mode explicitly. At the same time
all the internal functionality of pySim-shell will always use
the 'strict' mode.

Related: OS#6970
Change-Id: I9a531a825def318b28bf58291d811cf119003fab

esim/saip: raise an exception properly

Change-Id: Ia3749c02120fdc16e556214d0461cbeca032447b

ara_m: fix undefined variable used in a format-string

Change-Id: I310a5d461bae2b5e4d8e07097000b079c23aa0f6

ts_31_102: fix description for EF_5GS3GPPLOCI

Change-Id: I9cf3adfce65090fedb3f0fd33c9b3d15a2c5fb8c

cdma_ruim: fix inaccurate comment for EF_AD

Change-Id: I71ea27fd30e44685ff35f49843072ca392995973

ota: OtaAlgo{Crypt,Auth}: fix algo_auth vs algo_crypt

* OtaAlgoCrypt.from_keyset() searches by `otak.algo_crypt`
  but the error message prints `otak.algo_auth`.  Should be
  `otak.algo_crypt` instead.

* OtaAlgoAuth.__init__() checks `algo_auth` but the error message
  prints `algo_crypt`.  Should be `otak.algo_auth` instead.

Change-Id: Ia636fffaeadc68e3f6d5b65d477e753834c95895

sms: fix flags_construct in SMS_DELIVER

* field `tp_rp` appears at bit positions 7 and 5
** bit 7 should be `tp_rp` (Reply Path)
** bit 5 should be `tp_sri` (Status Report Indication)
* field `tp_lp` is completely missing
** should be at bit position 3

Change-Id: I0274849f0fa07281b5e050af429ffda7d249f9e8

global_platform: fix typo in SupportedTlsCipherSuitesForScp81

The attribute name is misspelled.  The BER-TLV infrastructure looks
for `_construct`; this typo means `SupportedTlsCipherSuitesForScp81`
will never decode its content.

Change-Id: I0f637951b0eeb7eca2a8b543baa737f216a935ed

global_platform: fix docstring for Scp03SessionKeys._get_icv()

Change-Id: I8983bc27f581295544360ba8b4ae1d28b3ea850f

utils: dec_plmn(): remove redundant call

Change-Id: Ic95c3992ed57eb8fee952ec2dc7f092dd7689579

ara_m: fix exceptions not being raised properly

Exceptions are meant to be thrown/raised, not returned.

Change-Id: Id799c264447e22887edcd2dc7eb991cf0af1bbfc

utils: DataObjectCollection.encode(): fix TypeError

`members_by_name` is a plain dictionary.  Calling it with `()` raises:

  TypeError: 'dict' object is not callable

Change-Id: I7e0c09aa7303f1506fe3a025fdc3779919dd0e6c

global_platform: fix s/GET/STORE/ DATA in docs

Both `do_store_data` and `store_data` have identical docstrings that
incorrectly describe the command as GET DATA.  Should be "STORE DATA".
Take a chance to fix missing space between `v2.3` and `Section`.

Change-Id: I33fc80ab8ca50fadc38217b0005eec6169c8e34e

global_platform: fix store_data() returning last chunk only

The loop builds up `response` across multiple STORE DATA blocks,
but the function returns only `data` - the response from the
*last* block.  It should return the accumulated response instead.

Change-Id: I3e15c8004d1e366e8c3896e559656622f48bb1a2

global_platform: fix typo in ApplicationTemplate

The keyword argument should be `nested=`.  As written `ApplicationAID`
is silently ignored - `ApplicationTemplate` will not descend into its
nested TLVs.

Change-Id: If45dbb0c9b09fe53560d109957ce339267a9f2b0

ts_31_102: EF_5G_PROSE_UIR: fix copy-pasted inner class name

Change-Id: I460e5ad70f35026d0d794271a4aef17323c14dfb

ts_51_011: EF.EXT[6-7]: fix typo in desc

Change-Id: I93df1c9fd8a4d588ed7ed19ec2dc1d304412fc3d

docs/put_key: add tutorial that explains how to manage global platform keys

With the increased interest in using GlobalPlatform features of
UICC and eUICCs (OTA-SMS, applets, etc.), also comes an increased
interest in how the related GlobalPlatform keys can be managed
(key rotation, adding/removing keysets from/to a Security Domain).

Unfortunately, many aspects of this topic are not immediately
obvious for the average user. Let's add a tutorial that contains
some practical examples to shine some light on the topic.

Related: SYS#7881
Change-Id: I163dfedca3df572cb8442e9a4a280e6c5b00327e

docs/put_key: add tutorial that explains how to manage global platform keys

With the increased interest in using GlobalPlatform features of
UICC and eUICCs (OTA-SMS, applets, etc.), also comes an increased
interest in how the related GlobalPlatform keys can be managed
(key rotation, adding/removing keysets from/to a Security Domain).

Unfortunately, many aspects of this topic are not immediately
obvious for the average user. Let's add a tutorial that contains
some practical examples to shine some light on the topic.

Related: SYS#7881
Change-Id: I163dfedca3df572cb8442e9a4a280e6c5b00327e

docs/put_key: add tutorial that explains how to manage global platform keys

With the increased interest in using GlobalPlatform features of
UICC and eUICCs (OTA-SMS, applets, etc.), also comes an increased
interest in how the related GlobalPlatform keys can be managed
(key rotation, adding/removing keysets from/to a Security Domain).

Unfortunately, many aspects of this topic are not immediately
obvious for the average user. Let's add a tutorial that contains
some practical examples to shine some light on the topic.

Related: SYS#7881
Change-Id: I163dfedca3df572cb8442e9a4a280e6c5b00327e

transport: change APDU format paradigm

Unfortunately we have mixed up the concept of TPDUs and APDUs in
earlier versions of pySim-shell. This lead to problems with
detecteding the APDU case properly (see also ISO/IEC 7816-3) and
also prevented us from adding support for T=1.

This problem has been fixed long time ago and all APDUs sent from
the pySim-shell code should be well formed and valid according to
ISO/IEC 7816-3.

To ensure that we continue to format APDUs correctly as APDUs (and
not TPDUs) we have added a mechanism to the LinkBase class that
would either raise an exception or print a warning if someone
mistakenly tries to send an APDU that is really a TPDU. Whether a
warning is printed or an exception is raised is controlled via the
apdu_strict member in the LinkBase class, which is false (print
warning only) by default.

The reason why we have implemneted the mechanism this way was
because we wanted to ensure that existing APDU scripts (pySim-shell
apdu command) keep working, even though when those scripts uses
APDUs which are formally invalid.

Sending a TPDU instead of an APDU via a T=0 link will still work
in almost all cases. This is also the reason why this problem
slipped through unnoticed for long time. However, there may still
be subtile problems araising from this practice. The root of the
problem is that it is impossible to distinguish between APDU case
3 and 4 when a TPDU instead of an APDU is sent. However in order
to handle a case 4 APDU correctly we must be able to distinguish
the APDU case correctly to handle the case correctly.
ETSI TS 102 221, section 7.3.1.1.4, clause 4 is very clear about
the fact that not (only) the status word (e.g. 61xx) but the
APDU case is what matters.

To complete the logic in LinkBaseTpdu and to maintain compatibility
(older APDU scripts), we must still be able to switch between the
'apdu_strict' mode and the non-strict mode. However, it makes sense
to do this on a per-api-call basis instead globally via a class
property.

At the same time we will limit the effect of pySim-shell's
apdu_strict setable to the apdu command only. By doing so, the
bahviour of the apdu command is not altered. Users will still
have to enable the 'strict' mode explicitly. At the same time
all the internal functionality of pySim-shell will always use
the 'strict' mode.

Related: OS#6970
Change-Id: I9a531a825def318b28bf58291d811cf119003fab

utils: DataObjectCollection.encode(): fix TypeError

`members_by_name` is a plain dictionary.  Calling it with `()` raises:

  TypeError: 'dict' object is not callable

Change-Id: I7e0c09aa7303f1506fe3a025fdc3779919dd0e6c

ara_m: fix undefined variable used in a format-string

Change-Id: I310a5d461bae2b5e4d8e07097000b079c23aa0f6

global_platform: fix typo in ApplicationTemplate

The keyword argument should be `nested=`.  As written `ApplicationAID`
is silently ignored - `ApplicationTemplate` will not descend into its
nested TLVs.

Change-Id: If45dbb0c9b09fe53560d109957ce339267a9f2b0

esim/saip: raise an exception properly

Change-Id: Ia3749c02120fdc16e556214d0461cbeca032447b

ts_31_102: fix description for EF_5GS3GPPLOCI

Change-Id: I9cf3adfce65090fedb3f0fd33c9b3d15a2c5fb8c

global_platform: fix typo in SupportedTlsCipherSuitesForScp81

The attribute name is misspelled.  The BER-TLV infrastructure looks
for `_construct`; this typo means `SupportedTlsCipherSuitesForScp81`
will never decode its content.

Change-Id: I0f637951b0eeb7eca2a8b543baa737f216a935ed

global_platform: fix store_data() returning last chunk only

The loop builds up `response` across multiple STORE DATA blocks,
but the function returns only `data` - the response from the
*last* block.  It should return the accumulated response instead.

Change-Id: I3e15c8004d1e366e8c3896e559656622f48bb1a2

cdma_ruim: fix inaccurate comment for EF_AD

Change-Id: I71ea27fd30e44685ff35f49843072ca392995973

ts_51_011: EF.EXT[6-7]: fix typo in desc

Change-Id: I93df1c9fd8a4d588ed7ed19ec2dc1d304412fc3d

global_platform: fix s/GET/STORE/ DATA in docs

Both `do_store_data` and `store_data` have identical docstrings that
incorrectly describe the command as GET DATA.  Should be "STORE DATA".
Take a chance to fix missing space between `v2.3` and `Section`.

Change-Id: I33fc80ab8ca50fadc38217b0005eec6169c8e34e

utils: dec_plmn(): remove redundant call

Change-Id: Ic95c3992ed57eb8fee952ec2dc7f092dd7689579

sms: fix flags_construct in SMS_DELIVER

* field `tp_rp` appears at bit positions 7 and 5
** bit 7 should be `tp_rp` (Reply Path)
** bit 5 should be `tp_sri` (Status Report Indication)
* field `tp_lp` is completely missing
** should be at bit position 3

Change-Id: I0274849f0fa07281b5e050af429ffda7d249f9e8

ts_31_102: EF_5G_PROSE_UIR: fix copy-pasted inner class name

Change-Id: I460e5ad70f35026d0d794271a4aef17323c14dfb

global_platform: install_cap_parser: argument groups cannot be nested

pySim-shell currently does not work on systems with Python 3.14+:

  File ".../pysim/pySim/global_platform/__init__.py", line 868, in AddlShellCommands
    install_cap_parser_inst_prm_g_grp = install_cap_parser_inst_prm_g.add_argument_group()
  File "/usr/lib/python3.14/argparse.py", line 1794, in add_argument_group
    raise ValueError('argument groups cannot be nested')
  ValueError('argument groups cannot be nested')

The problem is that install_cap_parser creates a nested group inside
of mutually exclusive group.  argparse never supported group nesting
properly, so it has been deprecated since Python 3.11, and eventually
got removed in Python 3.14.

Remove group nesting, adjust the usage string, and implement the
mutual exclusiveness/inclusiveness manually in do_install_cap().

Change-Id: Idddf72d5a745345e134b23f2f01e0257d0667579

global_platform: refactor gen_install_parameters()

gen_install_parameters() had contradictory logic: the outer guard
required all three arguments to be non-None/non-empty (making them
mutually inclusive), while the inner checks then treated each one
as optional.

Make each parameter independently optional (defaulting to None) and
remove the all-or-nothing check.  Simplify the function body to a
straightforward single-pass construction of system_specific_params.

Change-Id: I8756fb38016cdf0527fe2e21edb44381d1dc557f

ota: OtaAlgo{Crypt,Auth}: fix algo_auth vs algo_crypt

* OtaAlgoCrypt.from_keyset() searches by `otak.algo_crypt`
  but the error message prints `otak.algo_auth`.  Should be
  `otak.algo_crypt` instead.

* OtaAlgoAuth.__init__() checks `algo_auth` but the error message
  prints `algo_crypt`.  Should be `otak.algo_auth` instead.

Change-Id: Ia636fffaeadc68e3f6d5b65d477e753834c95895

global_platform: fix docstring for Scp03SessionKeys._get_icv()

Change-Id: I8983bc27f581295544360ba8b4ae1d28b3ea850f

ara_m: fix exceptions not being raised properly

Exceptions are meant to be thrown/raised, not returned.

Change-Id: Id799c264447e22887edcd2dc7eb991cf0af1bbfc

esim/http_json_api: allow URL rewriting

The URL used when HTTP requests are performed is defined statically
with the url_prefix passed to the constructor of JsonHttpApiClient
together with the path property in JsonHttpApiFunction.

For applications that require dynamic URLs there is no way to rewrite
the URL. Let's add a mechanism that allows API users to apply custom
URL reqriting rules by adding a rewrite_url method to
JsonHttpApiFunction. API users may then overload this method with a
custom implementation as needed.

Related: SYS#7918
Change-Id: Id2713a867079cc140517fe312189e5e2162608a5

esim/http_json_api: allow URL rewriting

The URL used when HTTP requests are performed is defined statically
with the url_prefix passed to the constructor of JsonHttpApiClient
together with the path property in JsonHttpApiFunction.

For applications that require dynamic URLs there is no way to rewrite
the URL. Let's add a mechanism that allows API users to apply custom
URL reqriting rules by adding a rewrite_url method to
JsonHttpApiFunction. API users may then overload this method with a
custom implementation as needed.

Related: SYS#7918
Change-Id: Id2713a867079cc140517fe312189e5e2162608a5

esim/http_json_api: allow URL rewriting

The URL used when HTTP requests are performed is defined statically
with the url_prefix passed to the constructor of JsonHttpApiClient
together with the path property in JsonHttpApiFunction.

For applications that require dynamic URLs there is no way to rewrite
the URL. Let's add a mechanism that allows API users to apply custom
URL reqriting rules by adding a reqrite_url method to
JsonHttpApiFunction. API users may then overload this method with a
custom implementation as needed.

Related: SYS#7918
Change-Id: Id2713a867079cc140517fe312189e5e2162608a5

ara_m: fix undefined variable used in a format-string

Change-Id: I310a5d461bae2b5e4d8e07097000b079c23aa0f6

utils: dec_plmn(): remove redundant call

Change-Id: Ic95c3992ed57eb8fee952ec2dc7f092dd7689579

ota: OtaAlgo{Crypt,Auth}: fix algo_auth vs algo_crypt

* OtaAlgoCrypt.from_keyset() searches by `otak.algo_crypt`
  but the error message prints `otak.algo_auth`.  Should be
  `otak.algo_crypt` instead.

* OtaAlgoAuth.__init__() checks `algo_auth` but the error message
  prints `algo_crypt`.  Should be `otak.algo_auth` instead.

Change-Id: Ia636fffaeadc68e3f6d5b65d477e753834c95895

esim/saip: raise an exception properly

Change-Id: Ia3749c02120fdc16e556214d0461cbeca032447b

global_platform: fix docstring for Scp03SessionKeys._get_icv()

Change-Id: I8983bc27f581295544360ba8b4ae1d28b3ea850f

ts_31_102: EF_5G_PROSE_UIR: fix copy-pasted inner class name

Change-Id: I460e5ad70f35026d0d794271a4aef17323c14dfb

global_platform: fix typo in SupportedTlsCipherSuitesForScp81

The attribute name is misspelled.  The BER-TLV infrastructure looks
for `_construct`; this typo means `SupportedTlsCipherSuitesForScp81`
will never decode its content.

Change-Id: I0f637951b0eeb7eca2a8b543baa737f216a935ed

ts_31_102: fix description for EF_5GS3GPPLOCI

Change-Id: I9cf3adfce65090fedb3f0fd33c9b3d15a2c5fb8c

ts_51_011: EF.EXT[6-7]: fix typo in desc

Change-Id: I93df1c9fd8a4d588ed7ed19ec2dc1d304412fc3d

cdma_ruim: fix inaccurate comment for EF_AD

Change-Id: I71ea27fd30e44685ff35f49843072ca392995973

global_platform: fix typo in ApplicationTemplate

The keyword argument should be `nested=`.  As written `ApplicationAID`
is silently ignored - `ApplicationTemplate` will not descend into its
nested TLVs.

Change-Id: If45dbb0c9b09fe53560d109957ce339267a9f2b0

utils: DataObjectCollection.encode(): fix TypeError

`members_by_name` is a plain dictionary.  Calling it with `()` raises:

  TypeError: 'dict' object is not callable

Change-Id: I7e0c09aa7303f1506fe3a025fdc3779919dd0e6c

sms: fix flags_construct in SMS_DELIVER

* field `tp_rp` appears at bit positions 7 and 5
** bit 7 should be `tp_rp` (Reply Path)
** bit 5 should be `tp_sri` (Status Report Indication)
* field `tp_lp` is completely missing
** should be at bit position 3

Change-Id: I0274849f0fa07281b5e050af429ffda7d249f9e8

ara_m: fix exceptions not being raised properly

Exceptions are meant to be thrown/raised, not returned.

Change-Id: Id799c264447e22887edcd2dc7eb991cf0af1bbfc

global_platform: fix store_data() returning last chunk only

The loop builds up `response` across multiple STORE DATA blocks,
but the function returns only `data` - the response from the
*last* block.  It should return the accumulated response instead.

Change-Id: I3e15c8004d1e366e8c3896e559656622f48bb1a2

global_platform: fix s/GET/STORE/ DATA in docs

Both `do_store_data` and `store_data` have identical docstrings that
incorrectly describe the command as GET DATA.  Should be "STORE DATA".
Take a chance to fix missing space between `v2.3` and `Section`.

Change-Id: I33fc80ab8ca50fadc38217b0005eec6169c8e34e

global_platform: make --install-parameters-* independently optional

gen_install_parameters() had contradictory logic: the outer guard
required all three arguments to be non-None/non-empty (making them
mutually inclusive), while the inner checks then treated each one
as optional.

Make each parameter independently optional (defaulting to None) and
remove the all-or-nothing guard from both the function and its caller.
Simplify the function body to a straightforward single-pass
construction of system_specific_params.

Change-Id: I8756fb38016cdf0527fe2e21edb44381d1dc557f

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32

personalization: allow reading back multiple values from PES

Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32

personalization: allow reading back multiple values from PES

Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a

esim/saip: raise an exception properly

Change-Id: Ia3749c02120fdc16e556214d0461cbeca032447b

sms: fix flags_construct in SMS_DELIVER

* field `tp_rp` appears at bit positions 7 and 5
** bit 7 should be `tp_rp` (Reply Path)
** bit 5 should be `tp_sri` (Status Report Indication)
* field `tp_lp` is completely missing
** should be at bit position 3

Change-Id: I0274849f0fa07281b5e050af429ffda7d249f9e8

ara_m: fix undefined variable used in a format-string

Change-Id: I310a5d461bae2b5e4d8e07097000b079c23aa0f6

utils: dec_plmn(): remove redundant call

Change-Id: Ic95c3992ed57eb8fee952ec2dc7f092dd7689579

ota: OtaAlgo{Crypt,Auth}: fix algo_auth vs algo_crypt

* OtaAlgoCrypt.from_keyset() searches by `otak.algo_crypt`
  but the error message prints `otak.algo_auth`.  Should be
  `otak.algo_crypt` instead.

* OtaAlgoAuth.__init__() checks `algo_auth` but the error message
  prints `algo_crypt`.  Should be `otak.algo_auth` instead.

Change-Id: Ia636fffaeadc68e3f6d5b65d477e753834c95895

ts_51_011: EF.EXT[6-7]: fix typo in desc

Change-Id: I93df1c9fd8a4d588ed7ed19ec2dc1d304412fc3d

utils: DataObjectCollection.encode(): fix TypeError

`members_by_name` is a plain dictionary.  Calling it with `()` raises:

  TypeError: 'dict' object is not callable

Change-Id: I7e0c09aa7303f1506fe3a025fdc3779919dd0e6c

ts_31_102: EF_5G_PROSE_UIR: fix copy-pasted inner class name

Change-Id: I460e5ad70f35026d0d794271a4aef17323c14dfb

global_platform: refactor gen_install_parameters()

Change-Id: I8756fb38016cdf0527fe2e21edb44381d1dc557f

ara_m: fix exceptions not being raised properly

Exceptions are meant to be thrown/raised, not returned.

Change-Id: Id799c264447e22887edcd2dc7eb991cf0af1bbfc

ts_31_102: fix description for EF_5GS3GPPLOCI

Change-Id: I9cf3adfce65090fedb3f0fd33c9b3d15a2c5fb8c

ara_m: fix exceptions not being raised properly

Exceptions are meant to be thrown/raised, not returned.

Change-Id: Id799c264447e22887edcd2dc7eb991cf0af1bbfc

global_platform: fix s/GET/STORE/ DATA in docs

Both `do_store_data` and `store_data` have identical docstrings that
incorrectly describe the command as GET DATA.  Should be "STORE DATA".
Take a chance to fix missing space between `v2.3` and `Section`.

Change-Id: I33fc80ab8ca50fadc38217b0005eec6169c8e34e

ts_51_011: EF.EXT[6-7]: fix typo in desc

Change-Id: I93df1c9fd8a4d588ed7ed19ec2dc1d304412fc3d

cdma_ruim: fix inaccurate comment for EF_AD

Change-Id: I71ea27fd30e44685ff35f49843072ca392995973

ts_31_102: EF_5G_PROSE_UIR: fix copy-pasted inner class name

Change-Id: I460e5ad70f35026d0d794271a4aef17323c14dfb

global_platform: fix docstring for Scp03SessionKeys._get_icv()

Change-Id: I8983bc27f581295544360ba8b4ae1d28b3ea850f

sms: fix flags_construct in SMS_DELIVER

* field `tp_rp` appears at bit positions 7 and 5
** bit 7 should be `tp_rp` (Reply Path)
** bit 5 should be `tp_sri` (Status Report Indication)
* field `tp_lp` is completely missing
** should be at bit position 3

Change-Id: I0274849f0fa07281b5e050af429ffda7d249f9e8

utils: DataObjectCollection.encode(): fix TypeError

`members_by_name` is a plain dictionary.  Calling it with `()` raises:

  TypeError: 'dict' object is not callable

Change-Id: I7e0c09aa7303f1506fe3a025fdc3779919dd0e6c

ts_31_102: fix description for EF_5GS3GPPLOCI

Change-Id: I9cf3adfce65090fedb3f0fd33c9b3d15a2c5fb8c

global_platform: fix store_data() returning last chunk only

The loop builds up `response` across multiple STORE DATA blocks,
but the function returns only `data` - the response from the
*last* block.  It should return the accumulated response instead.

Change-Id: I3e15c8004d1e366e8c3896e559656622f48bb1a2

esim/saip: raise an exception properly

Change-Id: Ia3749c02120fdc16e556214d0461cbeca032447b

global_platform: fix typo in SupportedTlsCipherSuitesForScp81

The attribute name is misspelled.  The BER-TLV infrastructure looks
for `_construct`; this typo means `SupportedTlsCipherSuitesForScp81`
will never decode its content.

Change-Id: I0f637951b0eeb7eca2a8b543baa737f216a935ed

utils: dec_plmn(): remove redundant call

Change-Id: Ic95c3992ed57eb8fee952ec2dc7f092dd7689579

ota: OtaAlgo{Crypt,Auth}: fix algo_auth vs algo_crypt

* OtaAlgoCrypt.from_keyset() searches by `otak.algo_crypt`
  but the error message prints `otak.algo_auth`.  Should be
  `otak.algo_crypt` instead.

* OtaAlgoAuth.__init__() checks `algo_auth` but the error message
  prints `algo_crypt`.  Should be `otak.algo_auth` instead.

Change-Id: Ia636fffaeadc68e3f6d5b65d477e753834c95895

global_platform: install_cap_parser: argument groups cannot be nested

pySim-shell currently does not work on systems with Python 3.14+:

  File ".../pysim/pySim/global_platform/__init__.py", line 868, in AddlShellCommands
    install_cap_parser_inst_prm_g_grp = install_cap_parser_inst_prm_g.add_argument_group()
  File "/usr/lib/python3.14/argparse.py", line 1794, in add_argument_group
    raise ValueError('argument groups cannot be nested')
  ValueError('argument groups cannot be nested')

The problem is that install_cap_parser creates a nested group inside
of mutually exclusive group.  argparse never supported group nesting
properly, so it has been deprecated since Python 3.11, and eventually
got removed in Python 3.14.

Remove group nesting, adjust the usage string, and implement the
mutual exclusiveness/inclusiveness manually in do_install_cap().

Change-Id: Idddf72d5a745345e134b23f2f01e0257d0667579

ara_m: fix undefined variable used in a format-string

Change-Id: I310a5d461bae2b5e4d8e07097000b079c23aa0f6

global_platform: refactor gen_install_parameters()

Change-Id: I8756fb38016cdf0527fe2e21edb44381d1dc557f

global_platform: fix typo in ApplicationTemplate

The keyword argument should be `nested=`.  As written `ApplicationAID`
is silently ignored - `ApplicationTemplate` will not descend into its
nested TLVs.

Change-Id: If45dbb0c9b09fe53560d109957ce339267a9f2b0

docs/put_key: add tutorial that explains how to manage global platform keys

With the increased interest in using GlobalPlatform features of
UICC and eUICCs (OTA-SMS, applets, etc.), also comes an increased
interest in how the related GlobalPlatform keys can be managed
(key rotation, adding/removing keysets from/to a Security Domain).

Unfortunately, many aspects of this topic are not immediately
obvious for the average user. Let's add a tutorial that contains
some practical examples to shine some light on the topic.

Related: SYS#7881
Change-Id: I163dfedca3df572cb8442e9a4a280e6c5b00327e

docs/put_key: add tutorial that explains how to manage global platform keys

With the increased interest in using GlobalPlatform features of
UICC and eUICCs (OTA-SMS, applets, etc.), also comes an increased
interest in how the related GlobalPlatform keys can be managed
(key rotation, adding/removing keysets from/to a Security Domain).

Unfortunately, many aspects of this topic are not immediately
obvious for the average user. Let's add a tutorial that contains
some practical examples to shine some light on the topic.

Related: SYS#7881
Change-Id: I163dfedca3df572cb8442e9a4a280e6c5b00327e

pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument

pySim-prog and pySim-read do not integrate the pySimLogger yet. As we
may add more debug output that should not be visible on normal use, we
should ensure that the pySimLogger is correctly set up.

Change-Id: Ia2fa535fd9ce4ffa301c3f5d6f98c1f7a4716c74

PySimLogger: add parameter to set initial log-level/verbosity

When we initialize a new PySimLogger, we always call the setup method
first and then use the set_verbose and set_level method to configure
the initial log level and the initial log verbosity. However, we
initialize the PySimLogger in all our programs the same way and we
end up with the same boilerplate code every time. Let's add a keyword
parameter to the setup method where we can pass our opts.verbose (bool)
parameter so that the setup method can do the work for the main program.

In case the caller wants a different default configuration he still can
call set_verbose and set_level methods as needed.

Change-Id: I4b8ef1e203186878910c9614a1d900d5759236a8

pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument

pySim-prog and pySim-read do not integrate the pySimLogger yet. As we
may add more debug output that should not be visible on normal use, we
should ensure that the pySimLogger is correctly set up.

Change-Id: Ia2fa535fd9ce4ffa301c3f5d6f98c1f7a4716c74

pySim-shell/cosmetic: remove semicolon

Change-Id: I629bacd432491211b939fcd2bed554b44ef441bc

cdma_ruim: fix copy-pasted desc for EF.AD

Change-Id: I2338f35c21978dd6b8916c0abd57b94f5e087655

global_platform: install_cap_parser: argument groups cannot be nested

pySim-shell currently does not work on systems with Python 3.14+:

  File ".../pysim/pySim/global_platform/__init__.py", line 868, in AddlShellCommands
    install_cap_parser_inst_prm_g_grp = install_cap_parser_inst_prm_g.add_argument_group()
  File "/usr/lib/python3.14/argparse.py", line 1794, in add_argument_group
    raise ValueError('argument groups cannot be nested')
  ValueError('argument groups cannot be nested')

The problem is that install_cap_parser creates a nested group inside
of mutually exclusive group.  argparse never supported group nesting
properly, so it has been deprecated since Python 3.11, and eventually
got removed in Python 3.14.

Remove group nesting, adjust the usage string, and implement the
mutual exclusiveness/inclusiveness manually in do_install_cap().

Change-Id: Idddf72d5a745345e134b23f2f01e0257d0667579

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

contrib/csv-to_pgsl: explicitly set log level to INFO in non verbose mode

The current log level for PySimLogger currently is DEBUG. (The default
for verbose is False). Since those defaults may change over time, we
should conciously set what we want in verbose and non verbose mode, like
we already do in pySim-shell.

Change-Id: I4b8ef1e203186878910c9614a1d900d5759236a8

pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument

pySim-prog and pySim-read do not integrate the pySimLogger yet. As we
may add more debug output that should not be visible on normal use, we
should ensure that the pySimLogger is correctly set up.

Change-Id: Ia2fa535fd9ce4ffa301c3f5d6f98c1f7a4716c74

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

global_platform/scp: fix dek_encrypt/dek_decrypt for SCP02

The methods dek_encrypt/dek_decrypt use the wrong algorithm and the
wrong key material. The algorithm should be 3DES rather then single
DES and the key must be the DEK session key instead of the static
DEK key from which the DEK session key is derived.

Related: SYS#7902
Change-Id: I3d0cc7378680b346fa39152c8b7074446d2c869d

contrib/smpp-ota-tool: use '-' instead of '_' in command line args

Some commandline arguments have an underscore in their name. Let's
replace those with dashes.

Change-Id: Icbe9d753d59263997e9ca34d46ed0daca36ca16c
Related: SYS#6868

contrib/smpp-ota-tool: define commandline arguments in global scope

The commandline arguments are currently defined under __main__ in a
private scope. From there they are not reachable to the sphinx
argparse module. We have to define the arguments globally at the
top. (like in the other applications)

Related: SYS#7881
Change-Id: I2d9782e3f5b1cac78c22d206fdcac4118c7d5e7c

docs/smpp-ota-tool: Add documentation/tutorial

We already have documentation that explains how to run pySim-smpp2sim.
With smpp-ota-tool we now have a counterpart for pySim-smpp2sim, so
let's add documentation for this tool as well.

Related: SYS#7881
Change-Id: If0d18a263f5a6dc035b90f5c5c6a942d46bbba49

tests/pySim-smpp2sim_test/card_sanitizer: update card backup with new test keyset

In our test setup we run the card_sanitizer.py script regualary to ensure that
we have consistent start conditions when running our tests. In case a testcase
crashes for some reason and leaves messed up files on a test card. The
card_sanitizer.py script will ensure that any problem like that is cleaned up
over night.

For the testcases we are about to add in the patch following this one, we need
to provision a new test keyset to one of our test cards. This has been already
done manually. However since the card_sanitizer still has the old keys in its
backup we will have to update that as well.

Change-Id: I5aa8a413b19b3e43a79d03e904daab50b4b1e767
Related: OS#6868

contrib/smpp-ota-tool: fix description string (copy+paste error)

Change-Id: I559844bfa1ac372370ef9d148f2f8a6bf4ab4ef5
Related: SYS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

esim/http_json_api: allow URL rewriting

The URL used when HTTP requests are performed is defined statically
with the url_prefix passed to the constructor of JsonHttpApiClient
together with the path property in JsonHttpApiFunction.

For applications that require dynamic URLs there is no way to rewrite
the URL. Let's add a mechanism that allows API users to apply custom
URL reqriting rules by adding a reqrite_url method to
JsonHttpApiFunction. API users may then overload this method with a
custom implementation as needed.

Related: SYS#7918
Change-Id: Id2713a867079cc140517fe312189e5e2162608a5

test_configurable_parameters.py: add tests for new parameters

For:
SmspTpScAddr
MilenageRotation
MilenageXoringConstants
TuakNrOfKeccak

Change-Id: Iecbea14fe31a9ee08d871dcde7f295d26d7bd001

remove transitional name mapping

This reverts commit I974cb6c393a2ed2248a6240c2722d157e9235c33

Now, finally, all SdKey classes have a unified logical naming scheme.

Change-Id: Ic185af4a903c2211a5361d023af9e7c6fc57ae78

saip: SdKey.__doc__: update SdKey listing

Change-Id: Ib5011b0c7d76b082231744cf09077628dc4e69b7

RandomHexDigitSource: rather return in string format, not bytes

Change-Id: I4e86289f6fb72cbd4cf0c90b8b49538cfab69a7f

personalization audit: optionally audit all (unknown) SD keys

By a flag, allow to audit also all Security Domain KVN that we have
*not* created ConfigurableParameter subclasses for.

For example, SCP80 has reserved kvn 0x01..0x0f, but we offer only
Scp80Kvn01, Scp80Kvn02, Scp80Kvn03. So we would not show kvn
0x03..0x0f in an audit.

This patch includes audits of all SD key kvn there may be in the UPP.
This will help to spot SD keys that may already be present in a UPP
template, with unexpected / unusual kvn.

Change-Id: Icaf6f7b589f117868633c0968a99f2f0252cf612

ConfigurableParameter: do not magically overwrite the 'name' attribute

Change-Id: I6f631444c6addeb7ccc5f6c55b9be3dc83409169

add comment about not updating existing key_usage_qualifier

Change-Id: Ie23ae5fde17be6b37746784bf1601b4d0874397a

saip SmspTpScAddr.get_values_from_pes: allow empty values

Change-Id: Ibbdd08f96160579238b50699091826883f2e9f5a

ConfigurableParameter: safer val length check

Change-Id: Ibe91722ed1477b00d20ef5e4e7abd9068ff2f3e4

transitional name mapping

To help existing applications transition to a common naming scheme for
the SdKey classes, offer this intermediate result, where the SdKey
classes' .name are still unchanged as before generating them.

Change-Id: I974cb6c393a2ed2248a6240c2722d157e9235c33

param_source: allow plugging a random implementation (for testing)

Change-Id: Idce2b18af70c17844d6f09f7704efc869456ac39

use random.SystemRandom as random nr source (/dev/urandom)

/dev/urandom is somewhat better than python's PRNG

Change-Id: I6de38c14ac6dd55bc84d53974192509c18d02bfa

personalization.ConfigurableParameter: fix BytesIO() input

Change-Id: I0ad160eef9015e76eef10baee7c6b606fe249123

generate sdkey classes from a list

Change-Id: Ic92ddea6e1fad8167ea75baf78ffc3eb419838c4

personalization: add int as input type for BinaryParameter

Change-Id: I31d8142cb0847a8b291f8dc614d57cb4734f0190

saip SmspTpScAddr: safeguard against decoding error

Reading the TS48 V6.0 eSIM_GTP_SAIP2.1A_NoBERTLV profile results in an
exception [1] in SmspTpScAddr. I have a caller that needs to skip
erratic values instead of raising.

The underlying issue, I presume, is that either the data needs
validation before decode_record_bin(), or decode_record_bin() needs
well-defined error handling.

So far I know only of this IndexError, so, as a workaround, catch that.

[1]
  File "/pysim/pySim/esim/saip/personalization.py", line 617, in get_values_from_pes
    ef_smsp_dec = ef_smsp.decode_record_bin(f_smsp.body, 1)
  File "/pysim/pySim/filesystem.py", line 1047, in decode_record_bin
    return parse_construct(self._construct, raw_bin_data)
  File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 550, in parse_construct
    parsed = c.parse(raw_bin_data, total_len=length, **context)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 404, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 416, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 820, in _parse
    return self._decode(obj, context, path)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 268, in _decode
    if r[-1] == 'f':
       ~^^^^
  File "/application/venv/lib/python3.13/site-packages/osmocom/utils.py", line 50, in __getitem__
    return hexstr(super().__getitem__(val))
                  ~~~~~~~~~~~~~~~~~~~^^^^^
IndexError: string index out of range

Change-Id: Ic436e206776b81f24de126e8ee0ae8bf5f3e8d7a

saip/param_source: try to not repeat random values

Change-Id: I4fa743ef5677580f94b9df16a5051d1d178edeb0

saip: add numeric_base indicator to ConfigurableParameter and ParamSource

By default, numeric_base = None, to indicate that there are no explicit
limitations on the number space.

For parameters that are definitely decimal, set numeric_base = 10.
For definitely hexadecimal, set numeric_base = 16.

Do the same for ConfigurableParameter as well as ParamSource, so callers
can match them up: if a parameter is numeric_base = 10, then omit
sources that are numeric_base = 16, and vice versa.

Change-Id: Ib0977bbdd9a85167be7eb46dd331fedd529dae01

use secrets.SystemRandom as secure random nr source

secrets.SystemRandom is defined as the most secure random source
available on the given operating system.

Change-Id: I8049cd1292674b3ced82b0926569128535af6efe

SdKey KVN4X ID02: set key_usage_qual=0x48

Related: SYS#7865
Change-Id: Idc5d33a4a003801f60c95fff6931706a9aeb6692

esim.saip.personalization: fix TLSPSK keys

Add AES variant of TLSPSK DEK (SCP81 KVN40 key_id=0x02).

Change-Id: I713a008fd26bbfcf437e0f29717b753f058ce76a

UppAudit: better indicate exception cause

Change-Id: I4d986b89a473a5b12ed56b4710263b034876a33e

add test_param_src.py

Change-Id: I03087b84030fddae98b965e0075d44e04ec6ba5c

saip: SmspTpScAddr: fix get_values_from_pes

Change-Id: I2010305340499c907bb7618c04c61e194db34814

add test_configurable_parameters.py

Change-Id: Ia55f0d11f8197ca15a948a83a34b3488acf1a0b4

comment in uicc.py on Security Domain Keys: add SCP81

Change-Id: Ib0205880f58e78c07688b4637abd5f67ea0570d1

personalization: make AlgorithmID a new EnumParam

The AlgorithmID has a few preset values, and hardly anyone knows which
is which. So instead of entering '1', '2' or '3', make it work with
prededined values 'Milenage', 'TUAK' and 'usim-test'.

Implement the enum value part abstractly in new EnumParam.

Make AlgorithmID a subclass of EnumParam and define the values as from
pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn

Related: SYS#6768
Change-Id: I71c2ec1b753c66cb577436944634f32792353240

personalization: fix SdKey.apply_val() implementation

'securityDomain' elements are decoded to ProfileElementSD instances,
which keep higher level representations of the key data apart from the
decoded[] lists.

So far, apply_val() was dropping binary values in decoded[], which does
not work, because ProfileElementSD._pre_encode() overwrites
self.decoded[] from the higher level representation.

Implement using
- ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify
  an exsiting entry, or
- ProfileElementSD.add_key() to create a new entry.

Before this patch, SdKey parameters seemed to patch PES successfully,
but their modifications did not end up in the encoded DER.

(BTW, this does not fix any other errors that may still be present in
the various SdKey subclasses, patches coming up.)

Related: SYS#6768
Change-Id: I07dfc378705eba1318e9e8652796cbde106c6a52

param_source: allow input val expansion like '0 * 32'

Working with keys, we often generate 4, 8, 16, 32 digit wide random
values. Those then typically have default input values like

00000000000000000000000000000000

it is hard for humans to count the number of digits. Much easier:

00*16

Teach the ParamSource subclasses dealing with random values to
understand an expansion like this. Any expansion is carried out before
all other input value handling.

Use this expansion also in the default_value of ConfigurableParameter
subclasses that have a default_source pointing at a ParamSource that now
understand this expansion.

Related: SYS#6768
Change-Id: Ie7171c152a7b478736f8825050305606b5af5735

personalization: implement UppAudit and BatchAudit

Change-Id: Iaab336ca91b483ecdddd5c6c8e08dc475dc6bd0a

personalization: add get_typical_input_len() to ConfigurableParameter

The aim is to tell a user interface how wide an input text field should
be chosen to be convenient -- ideally showing the entire value in all
cases, but not too huge for fields that have no sane size limit.

Change-Id: I2568a032167a10517d4d75d8076a747be6e21890

personalization: allow reading back multiple values from PES

Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I497c60c101ea0eea980e8b1a4b1f36c0eda39002

rather move BatchPersonalization to separate module

Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a

MilenageRotationConstants: set example_input to 3GPP default

Change-Id: I36a9434b2f96d26d710f489d5afce1f0ef05bba1

SmspTpScAddr: set example_input

Change-Id: Ie2c367788215d746807be24051478f0032a19448

docs/smpp-ota-tool: Add documentation/tutorial

We already have documentation that explains how to run pySim-smpp2sim.
With smpp-ota-tool we now have a counterpart for pySim-smpp2sim, so
let's add documentation for this tool as well.

Related: SYS#7881
Change-Id: If0d18a263f5a6dc035b90f5c5c6a942d46bbba49

contrib/smpp-ota-tool: use '-' instead of '_' in command line args

Some commandline arguments have an underscore in their name. Let's
replace those with dashes.

Change-Id: Icbe9d753d59263997e9ca34d46ed0daca36ca16c
Related: SYS#6868

docs/smpp-ota-tool: Add documentation/tutorial

We already have documentation that explains how to run pySim-smpp2sim.
With smpp-ota-tool we now have a counterpart for pySim-smpp2sim, so
let's add documentation for this tool as well.

Related: SYS#7881
Change-Id: If0d18a263f5a6dc035b90f5c5c6a942d46bbba49

contrib/smpp-ota-tool: define commandline arguments in global scope

The commandline arguments are currently defined under __main__ in a
private scope. From there they are not reachable to the sphinx
argparse module. We have to define the arguments globally at the
top. (like in the other applications)

Related: SYS#7881
Change-Id: I2d9782e3f5b1cac78c22d206fdcac4118c7d5e7c

contrib/smpp-ota-tool: fix description string (copy+paste error)

Change-Id: I559844bfa1ac372370ef9d148f2f8a6bf4ab4ef5
Related: SYS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

contrib/smpp-ota-tool: warn about mixed up KIC/KIC indexes

Cards usually have multiple sets of KIC, KID (and KIK). The keys
are selected through an index. However, mixing keys from different
sets is concidered as a security violation and cards should reject
such configurations.

Let's print a warning to make users aware that something is off.

Change-Id: Ieb4e14145baba1c2cb4a237b612b04694940f402
Related: OS#6868

contrib/smpp-ota-tool: fix boolean commandline parameters

Boolean parameters should be false by default and use store_true when
set.

Change-Id: I0652b48d2ea5efbaaf5bc147aa8cef7ab8b0861d
Related: OS#6868

contrib/smpp-ota-tool: add missing usage helpstrings

Change-Id: Ic1521ba11b405f311a30fdb3585ad518375669ae
Related: OS#6868

contrib/smpp-ota-tool: add missing usage helpstrings

Change-Id: Ic1521ba11b405f311a30fdb3585ad518375669ae
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

personalization: fix SdKey.apply_val() implementation

'securityDomain' elements are decoded to ProfileElementSD instances,
which keep higher level representations of the key data apart from the
decoded[] lists.

So far, apply_val() was dropping binary values in decoded[], which does
not work, because ProfileElementSD._pre_encode() overwrites
self.decoded[] from the higher level representation.

Implement using
- ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify
  an exsiting entry, or
- ProfileElementSD.add_key() to create a new entry.

Before this patch, SdKey parameters seemed to patch PES successfully,
but their modifications did not end up in the encoded DER.

(BTW, this does not fix any other errors that may still be present in
the various SdKey subclasses, patches coming up.)

Related: SYS#6768
Change-Id: I07dfc378705eba1318e9e8652796cbde106c6a52

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463

personalization: make AlgorithmID a new EnumParam

The AlgorithmID has a few preset values, and hardly anyone knows which
is which. So instead of entering '1', '2' or '3', make it work with
prededined values 'Milenage', 'TUAK' and 'usim-test'.

Implement the enum value part abstractly in new EnumParam.

Make AlgorithmID a subclass of EnumParam and define the values as from
pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn

Related: SYS#6768
Change-Id: I71c2ec1b753c66cb577436944634f32792353240

personalization: add get_typical_input_len() to ConfigurableParameter

The aim is to tell a user interface how wide an input text field should
be chosen to be convenient -- ideally showing the entire value in all
cases, but not too huge for fields that have no sane size limit.

Change-Id: I2568a032167a10517d4d75d8076a747be6e21890

personalization: allow reading back multiple values from PES

Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I497c60c101ea0eea980e8b1a4b1f36c0eda39002

rather move BatchPersonalization to separate module

Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32

contrib/smpp-ota-tool: warn about mixed up KIC/KIC indexes

Cards usually have multiple sets of KIC, KID (and KIK). The keys
are selected through an index. However, mixing keys from different
sets is concidered as a security violation and cards should reject
such configurations.

Let's print a warning to make users aware that something is off.

Change-Id: Ieb4e14145baba1c2cb4a237b612b04694940f402
Related: OS#6868

contrib/smpp-ota-tool/cosmetic: use lazy formatting for logging

Change-Id: I2540472a50b7a49b5a67d088cbdd4a2228eef8f4
Related: OS#6868

contrib/smpp-ota-tool: use correct kid index

(normally KID index and KIC index should be the same since mixing keys
is a concidered as a security violation. However, in this tool we
want to allow users to specify different indexes for KIC and KIC so that
they can make tests to make sure their cards correctly reject mixed up
key indexes)

Change-Id: I8847ccc39e4779971187e7877b8902fca7f8bfc1
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

contrib/smpp-ota-tool/cosmetic: fix sourcecode formatting

Change-Id: Icbce41ffac097d2ef8714bc8963536ba54a77db2
Related: OS#6868

contrib/smpp-ota-tool: add missing usage helpstrings

Change-Id: Ic1521ba11b405f311a30fdb3585ad518375669ae
Related: OS#6868

contrib/smpp-ota-tool: fix boolean commandline parameters

Boolean parameters should be false by default and use store_true when
set.

Change-Id: I0652b48d2ea5efbaaf5bc147aa8cef7ab8b0861d
Related: OS#6868

[2/7] personalization: refactor ConfigurableParameter, Iccid, Imsi

Main points/rationales of the refactoring, details below:
1) common validation implementation
2) offer classmethods

The new features are optional, and will be heavily used by batch
personalization patches coming soon.

Implement Iccid and Imsi to use the new way, with a common abstract
DecimalParam implementation.

So far leave the other parameter classes working as they always did, to
follow suit in subsequent commits.

Details:

1) common validation implementation:
There are very common validation steps in the various parameter
implementations. It is more convenient and much more readable to
implement those once and set simple validation parameters per subclass.
So there now is a validate_val() classmethod, which subclasses can use
as-is to apply the validation parameters -- or subclasses can override
their cls.validate_val() for specialized validation.
(Those subclasses that this patch doesn't touch still override the
self.validate() instance method. Hence they still work as before this
patch, but don't use the new common features yet.)

2) offer stateless classmethods:
It is useful for...
- batch processing of multiple profiles (in upcoming patches) and
- user input validation
to be able to have classmethods that do what self.validate() and
self.apply() do, but do not modify any self.* members.
So far the paradigm was to create a class instance to keep state about
the value. This remains available, but in addition we make available the
paradigm of a singleton that is stateless (the classmethods).
Using self.validate() and self.apply() still work the same as before
this patch, i.e. via self.input_value and self.value -- but in addition,
there are now classmethods that don't touch self.* members.

Related: SYS#6768
Change-Id: I6522be4c463e34897ca9bff2309b3706a88b3ce8