Changes

pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument

pySim-prog and pySim-read do not integrate the pySimLogger yet. As we
may add more debug output that should not be visible on normal use, we
should ensure that the pySimLogger is correctly set up.

Change-Id: Ia2fa535fd9ce4ffa301c3f5d6f98c1f7a4716c74

PySimLogger: add parameter to set initial log-level/verbosity

When we initialize a new PySimLogger, we always call the setup method
first and then use the set_verbose and set_level method to configure
the initial log level and the initial log verbosity. However, we
initialize the PySimLogger in all our programs the same way and we
end up with the same boilerplate code every time. Let's add a keyword
parameter to the setup method where we can pass our opts.verbose (bool)
parameter so that the setup method can do the work for the main program.

In case the caller wants a different default configuration he still can
call set_verbose and set_level methods as needed.

Change-Id: I4b8ef1e203186878910c9614a1d900d5759236a8

pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument

pySim-prog and pySim-read do not integrate the pySimLogger yet. As we
may add more debug output that should not be visible on normal use, we
should ensure that the pySimLogger is correctly set up.

Change-Id: Ia2fa535fd9ce4ffa301c3f5d6f98c1f7a4716c74

pySim-shell/cosmetic: remove semicolon

Change-Id: I629bacd432491211b939fcd2bed554b44ef441bc

cdma_ruim: fix copy-pasted desc for EF.AD

Change-Id: I2338f35c21978dd6b8916c0abd57b94f5e087655

global_platform: install_cap_parser: argument groups cannot be nested

pySim-shell currently does not work on systems with Python 3.14+:

  File ".../pysim/pySim/global_platform/__init__.py", line 868, in AddlShellCommands
    install_cap_parser_inst_prm_g_grp = install_cap_parser_inst_prm_g.add_argument_group()
  File "/usr/lib/python3.14/argparse.py", line 1794, in add_argument_group
    raise ValueError('argument groups cannot be nested')
  ValueError('argument groups cannot be nested')

The problem is that install_cap_parser creates a nested group inside
of mutually exclusive group.  argparse never supported group nesting
properly, so it has been deprecated since Python 3.11, and eventually
got removed in Python 3.14.

Remove group nesting, adjust the usage string, and implement the
mutual exclusiveness/inclusiveness manually in do_install_cap().

Change-Id: Idddf72d5a745345e134b23f2f01e0257d0667579

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

contrib/csv-to_pgsl: explicitly set log level to INFO in non verbose mode

The current log level for PySimLogger currently is DEBUG. (The default
for verbose is False). Since those defaults may change over time, we
should conciously set what we want in verbose and non verbose mode, like
we already do in pySim-shell.

Change-Id: I4b8ef1e203186878910c9614a1d900d5759236a8

pySim-prog/pySim-read: add pySimLogger and verbose cmdline argument

pySim-prog and pySim-read do not integrate the pySimLogger yet. As we
may add more debug output that should not be visible on normal use, we
should ensure that the pySimLogger is correctly set up.

Change-Id: Ia2fa535fd9ce4ffa301c3f5d6f98c1f7a4716c74

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

pySim/transport: fix GET RESPONSE behaviour

The current behavior we implement in the method __send_apdu_T0 is
incomplete. Some details discussed in ETSI TS 102 221,
section 7.3.1.1.4, clause 4 seem to be not fully implemented. We
may also end up sending a GET RESPONSE in other APDU cases than
case 4 (the only case that uses the GET RESPONSE command).

Related: OS#6970
Change-Id: I26f0566af0cdd61dcc97f5f502479dc76adc37cc

global_platform/scp: fix dek_encrypt/dek_decrypt for SCP02

The methods dek_encrypt/dek_decrypt use the wrong algorithm and the
wrong key material. The algorithm should be 3DES rather then single
DES and the key must be the DEK session key instead of the static
DEK key from which the DEK session key is derived.

Related: SYS#7902
Change-Id: I3d0cc7378680b346fa39152c8b7074446d2c869d

contrib/smpp-ota-tool: use '-' instead of '_' in command line args

Some commandline arguments have an underscore in their name. Let's
replace those with dashes.

Change-Id: Icbe9d753d59263997e9ca34d46ed0daca36ca16c
Related: SYS#6868

contrib/smpp-ota-tool: define commandline arguments in global scope

The commandline arguments are currently defined under __main__ in a
private scope. From there they are not reachable to the sphinx
argparse module. We have to define the arguments globally at the
top. (like in the other applications)

Related: SYS#7881
Change-Id: I2d9782e3f5b1cac78c22d206fdcac4118c7d5e7c

docs/smpp-ota-tool: Add documentation/tutorial

We already have documentation that explains how to run pySim-smpp2sim.
With smpp-ota-tool we now have a counterpart for pySim-smpp2sim, so
let's add documentation for this tool as well.

Related: SYS#7881
Change-Id: If0d18a263f5a6dc035b90f5c5c6a942d46bbba49

tests/pySim-smpp2sim_test/card_sanitizer: update card backup with new test keyset

In our test setup we run the card_sanitizer.py script regualary to ensure that
we have consistent start conditions when running our tests. In case a testcase
crashes for some reason and leaves messed up files on a test card. The
card_sanitizer.py script will ensure that any problem like that is cleaned up
over night.

For the testcases we are about to add in the patch following this one, we need
to provision a new test keyset to one of our test cards. This has been already
done manually. However since the card_sanitizer still has the old keys in its
backup we will have to update that as well.

Change-Id: I5aa8a413b19b3e43a79d03e904daab50b4b1e767
Related: OS#6868

contrib/smpp-ota-tool: fix description string (copy+paste error)

Change-Id: I559844bfa1ac372370ef9d148f2f8a6bf4ab4ef5
Related: SYS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

esim/http_json_api: allow URL rewriting

The URL used when HTTP requests are performed is defined statically
with the url_prefix passed to the constructor of JsonHttpApiClient
together with the path property in JsonHttpApiFunction.

For applications that require dynamic URLs there is no way to rewrite
the URL. Let's add a mechanism that allows API users to apply custom
URL reqriting rules by adding a reqrite_url method to
JsonHttpApiFunction. API users may then overload this method with a
custom implementation as needed.

Related: SYS#7918
Change-Id: Id2713a867079cc140517fe312189e5e2162608a5

test_configurable_parameters.py: add tests for new parameters

For:
SmspTpScAddr
MilenageRotation
MilenageXoringConstants
TuakNrOfKeccak

Change-Id: Iecbea14fe31a9ee08d871dcde7f295d26d7bd001

remove transitional name mapping

This reverts commit I974cb6c393a2ed2248a6240c2722d157e9235c33

Now, finally, all SdKey classes have a unified logical naming scheme.

Change-Id: Ic185af4a903c2211a5361d023af9e7c6fc57ae78

saip: SdKey.__doc__: update SdKey listing

Change-Id: Ib5011b0c7d76b082231744cf09077628dc4e69b7

RandomHexDigitSource: rather return in string format, not bytes

Change-Id: I4e86289f6fb72cbd4cf0c90b8b49538cfab69a7f

personalization audit: optionally audit all (unknown) SD keys

By a flag, allow to audit also all Security Domain KVN that we have
*not* created ConfigurableParameter subclasses for.

For example, SCP80 has reserved kvn 0x01..0x0f, but we offer only
Scp80Kvn01, Scp80Kvn02, Scp80Kvn03. So we would not show kvn
0x03..0x0f in an audit.

This patch includes audits of all SD key kvn there may be in the UPP.
This will help to spot SD keys that may already be present in a UPP
template, with unexpected / unusual kvn.

Change-Id: Icaf6f7b589f117868633c0968a99f2f0252cf612

ConfigurableParameter: do not magically overwrite the 'name' attribute

Change-Id: I6f631444c6addeb7ccc5f6c55b9be3dc83409169

add comment about not updating existing key_usage_qualifier

Change-Id: Ie23ae5fde17be6b37746784bf1601b4d0874397a

saip SmspTpScAddr.get_values_from_pes: allow empty values

Change-Id: Ibbdd08f96160579238b50699091826883f2e9f5a

ConfigurableParameter: safer val length check

Change-Id: Ibe91722ed1477b00d20ef5e4e7abd9068ff2f3e4

transitional name mapping

To help existing applications transition to a common naming scheme for
the SdKey classes, offer this intermediate result, where the SdKey
classes' .name are still unchanged as before generating them.

Change-Id: I974cb6c393a2ed2248a6240c2722d157e9235c33

param_source: allow plugging a random implementation (for testing)

Change-Id: Idce2b18af70c17844d6f09f7704efc869456ac39

use random.SystemRandom as random nr source (/dev/urandom)

/dev/urandom is somewhat better than python's PRNG

Change-Id: I6de38c14ac6dd55bc84d53974192509c18d02bfa

personalization.ConfigurableParameter: fix BytesIO() input

Change-Id: I0ad160eef9015e76eef10baee7c6b606fe249123

generate sdkey classes from a list

Change-Id: Ic92ddea6e1fad8167ea75baf78ffc3eb419838c4

personalization: add int as input type for BinaryParameter

Change-Id: I31d8142cb0847a8b291f8dc614d57cb4734f0190

saip SmspTpScAddr: safeguard against decoding error

Reading the TS48 V6.0 eSIM_GTP_SAIP2.1A_NoBERTLV profile results in an
exception [1] in SmspTpScAddr. I have a caller that needs to skip
erratic values instead of raising.

The underlying issue, I presume, is that either the data needs
validation before decode_record_bin(), or decode_record_bin() needs
well-defined error handling.

So far I know only of this IndexError, so, as a workaround, catch that.

[1]
  File "/pysim/pySim/esim/saip/personalization.py", line 617, in get_values_from_pes
    ef_smsp_dec = ef_smsp.decode_record_bin(f_smsp.body, 1)
  File "/pysim/pySim/filesystem.py", line 1047, in decode_record_bin
    return parse_construct(self._construct, raw_bin_data)
  File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 550, in parse_construct
    parsed = c.parse(raw_bin_data, total_len=length, **context)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 404, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 416, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse
    return self.subcon._parsereport(stream, context, path)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport
    obj = self._parse(stream, context, path)
  File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 820, in _parse
    return self._decode(obj, context, path)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 268, in _decode
    if r[-1] == 'f':
       ~^^^^
  File "/application/venv/lib/python3.13/site-packages/osmocom/utils.py", line 50, in __getitem__
    return hexstr(super().__getitem__(val))
                  ~~~~~~~~~~~~~~~~~~~^^^^^
IndexError: string index out of range

Change-Id: Ic436e206776b81f24de126e8ee0ae8bf5f3e8d7a

saip/param_source: try to not repeat random values

Change-Id: I4fa743ef5677580f94b9df16a5051d1d178edeb0

saip: add numeric_base indicator to ConfigurableParameter and ParamSource

By default, numeric_base = None, to indicate that there are no explicit
limitations on the number space.

For parameters that are definitely decimal, set numeric_base = 10.
For definitely hexadecimal, set numeric_base = 16.

Do the same for ConfigurableParameter as well as ParamSource, so callers
can match them up: if a parameter is numeric_base = 10, then omit
sources that are numeric_base = 16, and vice versa.

Change-Id: Ib0977bbdd9a85167be7eb46dd331fedd529dae01

use secrets.SystemRandom as secure random nr source

secrets.SystemRandom is defined as the most secure random source
available on the given operating system.

Change-Id: I8049cd1292674b3ced82b0926569128535af6efe

SdKey KVN4X ID02: set key_usage_qual=0x48

Related: SYS#7865
Change-Id: Idc5d33a4a003801f60c95fff6931706a9aeb6692

esim.saip.personalization: fix TLSPSK keys

Add AES variant of TLSPSK DEK (SCP81 KVN40 key_id=0x02).

Change-Id: I713a008fd26bbfcf437e0f29717b753f058ce76a

UppAudit: better indicate exception cause

Change-Id: I4d986b89a473a5b12ed56b4710263b034876a33e

add test_param_src.py

Change-Id: I03087b84030fddae98b965e0075d44e04ec6ba5c

saip: SmspTpScAddr: fix get_values_from_pes

Change-Id: I2010305340499c907bb7618c04c61e194db34814

add test_configurable_parameters.py

Change-Id: Ia55f0d11f8197ca15a948a83a34b3488acf1a0b4

comment in uicc.py on Security Domain Keys: add SCP81

Change-Id: Ib0205880f58e78c07688b4637abd5f67ea0570d1

personalization: make AlgorithmID a new EnumParam

The AlgorithmID has a few preset values, and hardly anyone knows which
is which. So instead of entering '1', '2' or '3', make it work with
prededined values 'Milenage', 'TUAK' and 'usim-test'.

Implement the enum value part abstractly in new EnumParam.

Make AlgorithmID a subclass of EnumParam and define the values as from
pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn

Related: SYS#6768
Change-Id: I71c2ec1b753c66cb577436944634f32792353240

personalization: fix SdKey.apply_val() implementation

'securityDomain' elements are decoded to ProfileElementSD instances,
which keep higher level representations of the key data apart from the
decoded[] lists.

So far, apply_val() was dropping binary values in decoded[], which does
not work, because ProfileElementSD._pre_encode() overwrites
self.decoded[] from the higher level representation.

Implement using
- ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify
  an exsiting entry, or
- ProfileElementSD.add_key() to create a new entry.

Before this patch, SdKey parameters seemed to patch PES successfully,
but their modifications did not end up in the encoded DER.

(BTW, this does not fix any other errors that may still be present in
the various SdKey subclasses, patches coming up.)

Related: SYS#6768
Change-Id: I07dfc378705eba1318e9e8652796cbde106c6a52

param_source: allow input val expansion like '0 * 32'

Working with keys, we often generate 4, 8, 16, 32 digit wide random
values. Those then typically have default input values like

00000000000000000000000000000000

it is hard for humans to count the number of digits. Much easier:

00*16

Teach the ParamSource subclasses dealing with random values to
understand an expansion like this. Any expansion is carried out before
all other input value handling.

Use this expansion also in the default_value of ConfigurableParameter
subclasses that have a default_source pointing at a ParamSource that now
understand this expansion.

Related: SYS#6768
Change-Id: Ie7171c152a7b478736f8825050305606b5af5735

personalization: implement UppAudit and BatchAudit

Change-Id: Iaab336ca91b483ecdddd5c6c8e08dc475dc6bd0a

personalization: add get_typical_input_len() to ConfigurableParameter

The aim is to tell a user interface how wide an input text field should
be chosen to be convenient -- ideally showing the entire value in all
cases, but not too huge for fields that have no sane size limit.

Change-Id: I2568a032167a10517d4d75d8076a747be6e21890

personalization: allow reading back multiple values from PES

Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I497c60c101ea0eea980e8b1a4b1f36c0eda39002

rather move BatchPersonalization to separate module

Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a

MilenageRotationConstants: set example_input to 3GPP default

Change-Id: I36a9434b2f96d26d710f489d5afce1f0ef05bba1

SmspTpScAddr: set example_input

Change-Id: Ie2c367788215d746807be24051478f0032a19448

docs/smpp-ota-tool: Add documentation/tutorial

We already have documentation that explains how to run pySim-smpp2sim.
With smpp-ota-tool we now have a counterpart for pySim-smpp2sim, so
let's add documentation for this tool as well.

Related: SYS#7881
Change-Id: If0d18a263f5a6dc035b90f5c5c6a942d46bbba49

contrib/smpp-ota-tool: use '-' instead of '_' in command line args

Some commandline arguments have an underscore in their name. Let's
replace those with dashes.

Change-Id: Icbe9d753d59263997e9ca34d46ed0daca36ca16c
Related: SYS#6868

docs/smpp-ota-tool: Add documentation/tutorial

We already have documentation that explains how to run pySim-smpp2sim.
With smpp-ota-tool we now have a counterpart for pySim-smpp2sim, so
let's add documentation for this tool as well.

Related: SYS#7881
Change-Id: If0d18a263f5a6dc035b90f5c5c6a942d46bbba49

contrib/smpp-ota-tool: define commandline arguments in global scope

The commandline arguments are currently defined under __main__ in a
private scope. From there they are not reachable to the sphinx
argparse module. We have to define the arguments globally at the
top. (like in the other applications)

Related: SYS#7881
Change-Id: I2d9782e3f5b1cac78c22d206fdcac4118c7d5e7c

contrib/smpp-ota-tool: fix description string (copy+paste error)

Change-Id: I559844bfa1ac372370ef9d148f2f8a6bf4ab4ef5
Related: SYS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

contrib/smpp-ota-tool: warn about mixed up KIC/KIC indexes

Cards usually have multiple sets of KIC, KID (and KIK). The keys
are selected through an index. However, mixing keys from different
sets is concidered as a security violation and cards should reject
such configurations.

Let's print a warning to make users aware that something is off.

Change-Id: Ieb4e14145baba1c2cb4a237b612b04694940f402
Related: OS#6868

contrib/smpp-ota-tool: fix boolean commandline parameters

Boolean parameters should be false by default and use store_true when
set.

Change-Id: I0652b48d2ea5efbaaf5bc147aa8cef7ab8b0861d
Related: OS#6868

contrib/smpp-ota-tool: add missing usage helpstrings

Change-Id: Ic1521ba11b405f311a30fdb3585ad518375669ae
Related: OS#6868

contrib/smpp-ota-tool: add missing usage helpstrings

Change-Id: Ic1521ba11b405f311a30fdb3585ad518375669ae
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

personalization: fix SdKey.apply_val() implementation

'securityDomain' elements are decoded to ProfileElementSD instances,
which keep higher level representations of the key data apart from the
decoded[] lists.

So far, apply_val() was dropping binary values in decoded[], which does
not work, because ProfileElementSD._pre_encode() overwrites
self.decoded[] from the higher level representation.

Implement using
- ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify
  an exsiting entry, or
- ProfileElementSD.add_key() to create a new entry.

Before this patch, SdKey parameters seemed to patch PES successfully,
but their modifications did not end up in the encoded DER.

(BTW, this does not fix any other errors that may still be present in
the various SdKey subclasses, patches coming up.)

Related: SYS#6768
Change-Id: I07dfc378705eba1318e9e8652796cbde106c6a52

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463

personalization: make AlgorithmID a new EnumParam

The AlgorithmID has a few preset values, and hardly anyone knows which
is which. So instead of entering '1', '2' or '3', make it work with
prededined values 'Milenage', 'TUAK' and 'usim-test'.

Implement the enum value part abstractly in new EnumParam.

Make AlgorithmID a subclass of EnumParam and define the values as from
pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn

Related: SYS#6768
Change-Id: I71c2ec1b753c66cb577436944634f32792353240

personalization: add get_typical_input_len() to ConfigurableParameter

The aim is to tell a user interface how wide an input text field should
be chosen to be convenient -- ideally showing the entire value in all
cases, but not too huge for fields that have no sane size limit.

Change-Id: I2568a032167a10517d4d75d8076a747be6e21890

personalization: allow reading back multiple values from PES

Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I497c60c101ea0eea980e8b1a4b1f36c0eda39002

rather move BatchPersonalization to separate module

Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32

contrib/smpp-ota-tool: warn about mixed up KIC/KIC indexes

Cards usually have multiple sets of KIC, KID (and KIK). The keys
are selected through an index. However, mixing keys from different
sets is concidered as a security violation and cards should reject
such configurations.

Let's print a warning to make users aware that something is off.

Change-Id: Ieb4e14145baba1c2cb4a237b612b04694940f402
Related: OS#6868

contrib/smpp-ota-tool/cosmetic: use lazy formatting for logging

Change-Id: I2540472a50b7a49b5a67d088cbdd4a2228eef8f4
Related: OS#6868

contrib/smpp-ota-tool: use correct kid index

(normally KID index and KIC index should be the same since mixing keys
is a concidered as a security violation. However, in this tool we
want to allow users to specify different indexes for KIC and KIC so that
they can make tests to make sure their cards correctly reject mixed up
key indexes)

Change-Id: I8847ccc39e4779971187e7877b8902fca7f8bfc1
Related: OS#6868

tests/pySim-smpp2sim_test: add testcases for AES128 and AES256

Extend the existing test script so that it can handle multiple
testcases. Also add support for switching eUICC profiles.
Finally, add a testcases to test OTA-SMS (RFM) with AES128 and
AES256 encryption.

Change-Id: I1f10504f3a29a8c74a17991632d932819fecfa5a
Related: OS#6868

contrib/smpp-ota-tool/cosmetic: fix sourcecode formatting

Change-Id: Icbce41ffac097d2ef8714bc8963536ba54a77db2
Related: OS#6868

contrib/smpp-ota-tool: add missing usage helpstrings

Change-Id: Ic1521ba11b405f311a30fdb3585ad518375669ae
Related: OS#6868

contrib/smpp-ota-tool: fix boolean commandline parameters

Boolean parameters should be false by default and use store_true when
set.

Change-Id: I0652b48d2ea5efbaaf5bc147aa8cef7ab8b0861d
Related: OS#6868

esim/http_json_api.py: support text/plain response Content-Type

Allow returning text/plain Content-Types as 'data' output argument.

So far, all the esim/http_json_api functions require a JSON response.
However, a specific vendor has a list function where the request is JSON
but the response is text/plain CSV data. Allow and return in a dict.

Change-Id: Iba6e4cef1048b376050a435a900c0f395655a790

personalization: add param_source.py, add batch.py

Implement pySim.esim.saip.batch.BatchPersonalization,
generating N eSIM profiles from a preset configuration.

Batch parameters can be fed by a constant, incrementing, random or from
CSV rows: add pySim.esim.saip.param_source.* classes to feed such input
to each of the BatchPersonalization's ConfigurableParameter instances.

Related: SYS#6768
Change-Id: I497c60c101ea0eea980e8b1a4b1f36c0eda39002

rather move BatchPersonalization to separate module

Change-Id: I01ae40a06605eb205bfb409189fcd2b3a128855a

personalization: indicate default ParamSource per ConfigurableParameter

Add default_source class members pointing to ParamSource classes to all
ConfigurableParameter subclasses.

This is useful to automatically set up a default ParamSource for a given
ConfigurableParameter subclass, during user interaction to produce a
batch personalization.

For example, if the user selects a Pin1 parameter, a calling program can
implicitly set this to a RandomDigitSource, which will magically make it
work the way that most users need.

BTW, default_source and default_value can be combined to configure a
matching ParamSource instance:

  my_source = MyParam.default_source.from_str( MyParam.default_value )

Change-Id: Ie58d13bce3fa1aa2547cf3cee918c2f5b30a8b32

personalization: allow reading back multiple values from PES

Change-Id: Iecb68af7c216c6b9dc3add469564416b6f37f7b2

personalization: implement reading back values from a PES

Implement get_values_from_pes(), the reverse direction of apply_val():
read back and return values from a ProfileElementSequence. Implement for
all ConfigurableParameter subclasses.

Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works
fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this
implementation to use the higher level ProfileElementSD members.

Implementation detail:

Implement get_values_from_pes() as classmethod that returns a generator.
Subclasses should yield all occurences of their parameter in a given
PES.

For example, the ICCID can appear in multiple places.
Iccid.get_values_from_pes() yields all of the individual values. A set()
of the results quickly tells whether the PES is consistent.

Rationales for reading back values:

This allows auditing an eSIM profile, particularly for producing an
output.csv from a batch personalization (that generated lots of random
key material which now needs to be fed to an HLR...).

Reading back from a binary result is more reliable than storing the
values that were fed into a personalization.
By auditing final DER results with this code, I discovered:
- "oh, there already was some key material in my UPP template."
- "all IMSIs ended up the same, forgot to set up the parameter."
- the SdKey.apply() implementations currently don't work, see
  I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.

Change-Id: I234fc4317f0bdc1a486f0cee4fa432c1dce9b463

MilenageRotationConstants: set example_input to 3GPP default

Change-Id: I36a9434b2f96d26d710f489d5afce1f0ef05bba1

pySim.esim.saip.File: Support pinStatusTemplateDO + lcsi

The tuples defining a DF or ADF in an eSIM template must contain a
pinStatusTemplateDO.  When parsing tuples into a File() instance, we
must save it, and re-create it at the time we re-encode that file.

Same applies to the lcsi (life cycle state indicator), which may
optionally exist for any file.

Change-Id: I073aa4374f2cd664d07fa0224bf0d4c809cdf4aa
Closes: OS#6955

compile_asn1_subdir: filter compiled files by .asn suffix

When I open the .asn file in vim, pySim should not attempt to read the
vim .swp file as asn.1.

  File "/home/moi/osmo-dev/src/pysim/pySim/esim/saip/__init__.py", line 45, in <module>
    asn1 = compile_asn1_subdir('saip')
[...]
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 21: invalid start byte

Related: OS#6937
Change-Id: I37df3fc081e51e2ed2198876c63f6e68ecc8fcd8

esim/http_json_api: add alternative API interface (follow up)

This is a follow up patch to change:
I2a5d4b59b12e08d5eae7a1215814d3a69c8921f6

- do not ignore length of kwargs
- fix role parameter (roles other than 'legacy_client' can be used now)
- use startswith instead of match

Related: SYS#7866
Change-Id: Ifae13e82d671ff09bddf771f063a388d2ab283eb

pySim-shell_test/euicc: ensure test-profile is enabled

When testing commands like get_profile_info, enable_profile,
disable_profile or the commands to manage notifications, we
should ensure that the correct profile is enabled before
executing the actual testcase.

Change-Id: Ie57b0305876bc5001ab3a9c3a3b5711408161b74

pySim/euicc: fix encoding/decoding of Iccid

The class Iccid uses a BcdAdapter to encoded/decode the ICCID. This
works fine for ICCIDs that have an even (20) number of digits. In case
the digit count is odd (19), the ICCID the last digit requires padding.

Let's switch to PaddedBcdAdapter for encoding/decoding, to ensure that
odd-length ICCIDs are padded automatically.

Change-Id: I527a44ba454656a0d682ceb590eec6d9d0ac883a
Related: OS#6868

pySim-shell_test/euicc: ensure test-profile is enabled

When testing commands like get_profile_info, enable_profile,
disable_profile or the commands to manage notifications, we
should ensure that the correct profile is enabled before
executing the actual testcase.

Change-Id: Ie57b0305876bc5001ab3a9c3a3b5711408161b74

pySim/global_platform: replace deprecated argument group() creation

The global platform code used install_cap_parser.add_mutually_exclusive_group().add_argument_group()
But calling add_argument_group() on an exclusive group was never an intended feature nor
according to python docs was a working feature.

Change-Id: I4f73d3417a12b7fe94e33a265cdae244f3c9a1e9

pysim.utils.decomposeATR: Fix docutils warning

pySim/utils.py:docstring of pySim.utils.decomposeATR:9: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]

Change-Id: Ifda4ba15014ba97634fd5bd5c9b19d9110f4670e

pySim.esim.saip: Fix docstring warnings:

this fixes the following two warnings:

pySim/esim/saip/__init__.py:docstring of pySim.esim.saip.FsNode.walk:1: WARNING: Inline strong start-string without end-string. [docutils]
pySim/esim/saip/__init__.py:docstring of pySim.esim.saip.FsNodeDF.walk:1: WARNING: Inline strong start-string without end-string. [docutils]

Change-Id: Id7debf9296923b735f76623808cee68967a1ece7

pySim.esim.saip.personalization: Fix docstring errors + warnings

pysim/pySim/esim/saip/personalization.py:docstring of pySim.esim.saip.personalization.ConfigurableParameter:27: ERROR: Unexpected indentation. [docutils]
pysim/pySim/esim/saip/personalization.py:docstring of pySim.esim.saip.personalization.ConfigurableParameter:29: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
pysim/pySim/esim/saip/personalization.py:docstring of pySim.esim.saip.personalization.ConfigurableParameter:34: ERROR: Unexpected indentation. [docutils]
pysim/pySim/esim/saip/personalization.py:docstring of pySim.esim.saip.personalization.ConfigurableParameter:35: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
pysim/pySim/esim/saip/personalization.py:docstring of pySim.esim.saip.personalization.ConfigurableParameter:52: ERROR: Unexpected indentation. [docutils]
pysim/pySim/esim/saip/personalization.py:docstring of pySim.esim.saip.personalization.ConfigurableParameter:53: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]

Change-Id: I3918308856c3a1a5e6e90561c3e2a6b88040670d

pySim.esim.saip.personalization: Fix docstring error

pySim/esim/saip/personalization.py:docstring of pySim.esim.saip.personalization.MilenageXoringConstants:4: ERROR: Unexpected indentation. [docutils]

Change-Id: If6ae360b7f74c095fa9075ae9aa988440496e6de

pySim-shell_test/euicc: ensure test-profile is enabled

When testing commands like get_profile_info, enable_profile,
disable_profile or the commands to manage notifications, we
should ensure that the correct profile is enabled before
executing the actual testcase.

Change-Id: Ie57b0305876bc5001ab3a9c3a3b5711408161b74

pySim-shell_test/euicc: ensure test-profile is enabled

When testing commands like get_profile_info, enable_profile
or disable_profile, we should ensure that the correct profile
is enabled befor executing the actual testcase.

Change-Id: Ie57b0305876bc5001ab3a9c3a3b5711408161b74

pySim-shell_test/euicc: ensure test-profile is enabled

When testing the enable_profile / disable_profile commands we expect that
the profile we test with is enabled before the test begins. To be more
robust lets send an enable_profile command before the actual test begins.

Change-Id: Ie57b0305876bc5001ab3a9c3a3b5711408161b74

ModemATCommandLink: fix SyntaxWarning: invalid escape sequence '\+'

Change-Id: If8de5299a4dc5a8525ef6657213db95d30e3c83b
Fixes: OS#6948

pySim-shell_test/euicc: fix testcase method name

We have two test_enable_disable_profile method, the second one should
be called test_set_nickname.

Change-Id: I5ff79218fdafc8c42c8b58cc00be3e56e09d808b

[2/7] personalization: refactor ConfigurableParameter, Iccid, Imsi

Main points/rationales of the refactoring, details below:
1) common validation implementation
2) offer classmethods

The new features are optional, and will be heavily used by batch
personalization patches coming soon.

Implement Iccid and Imsi to use the new way, with a common abstract
DecimalParam implementation.

So far leave the other parameter classes working as they always did, to
follow suit in subsequent commits.

Details:

1) common validation implementation:
There are very common validation steps in the various parameter
implementations. It is more convenient and much more readable to
implement those once and set simple validation parameters per subclass.
So there now is a validate_val() classmethod, which subclasses can use
as-is to apply the validation parameters -- or subclasses can override
their cls.validate_val() for specialized validation.
(Those subclasses that this patch doesn't touch still override the
self.validate() instance method. Hence they still work as before this
patch, but don't use the new common features yet.)

2) offer stateless classmethods:
It is useful for...
- batch processing of multiple profiles (in upcoming patches) and
- user input validation
to be able to have classmethods that do what self.validate() and
self.apply() do, but do not modify any self.* members.
So far the paradigm was to create a class instance to keep state about
the value. This remains available, but in addition we make available the
paradigm of a singleton that is stateless (the classmethods).
Using self.validate() and self.apply() still work the same as before
this patch, i.e. via self.input_value and self.value -- but in addition,
there are now classmethods that don't touch self.* members.

Related: SYS#6768
Change-Id: I6522be4c463e34897ca9bff2309b3706a88b3ce8