esim/http_json_api.py: support text/plain response Content-Type
Allow returning text/plain Content-Types as 'data' output argument.
So far, all the esim/http_json_api functions require a JSON response. However, a specific vendor has a list function where the request is JSON but the response is text/plain CSV data. Allow and return in a dict.
ts_51_011/EF.SMSP: fix handling of 'alpha_id' field
The field 'alpha_id' is technically not an optional field, even though the specification describes it as optional. Once the card manufacturer decides that the field should be present, it must be always present and vice versa.
(see code comment for a more detailed description)
The legacy code found in legacy/cards.py does not use the modern construct based encoder (pySim-read uses it). The card classes either use their own implementation of update_smsp or use the generic method provided by the SimCard class. The latter one is true for FairwavesSIM and WavemobileSim.
Unfortunately the implementation found in the SimCard is wrong. It adds padding at the end of the file instead of the beginning. This completely messes up the contents of EF.SMSP for the cards using this method. To fix this, let's use the leftpad feature provided by the update_record. This will ensure a correct alignment of the file contents.
The current behavior we implement in the method __send_apdu_T0 is incomplete. Some details discussed in ETSI TS 102 221, section 7.3.1.1.4, clause 4 seem to be not fully implemented. We may also end up sending a GET RESPONSE in other APDU cases than case 4 (the only case that uses the GET RESPONSE command).
validate_val() calls len() to check the value against allow_len, min_len and max_len. len() requires the object to have a __len__() method, which integers do not — calling len() on an int raises TypeError.
Fix this by checking for __len__ first: if present, use len(val) as usual; otherwise fall back to len(str(val)), which gives the number of decimal digits for integer values.
validate_val() calls len() to check the value against allow_len, min_len and max_len. len() requires the object to have a __len__() method, which integers do not — calling len() on an int raises TypeError.
Fix this by checking for __len__ first: if present, use len(val) as usual; otherwise fall back to len(str(val)), which gives the number of decimal digits for integer values.
saip: add numeric_base indicator to ConfigurableParameter
By default, numeric_base = None, to indicate that there are no explicit limitations on the number space.
For parameters that are definitely decimal, set numeric_base = 10. For definitely hexadecimal, set numeric_base = 16.
Do the same for ConfigurableParameter as well as ParamSource, so callers can match them up: if a parameter is numeric_base = 10, then omit sources that are numeric_base = 16, and vice versa.
Add ConfigurableParameterTest, which applies each parameter to a real UPP DER template and reads it back, comparing results against a stored expected-output snapshot (xo/test_configurable_parameters).
Add TestValidateVal covering validate_val() for Iccid, Imsi, Pin1, Puk1 and K, testing both valid inputs and invalid ones expected to raise ValueError.
Add TestEnumParam covering the EnumParam methods (validate_val, map_name_to_val, map_val_to_name, name_normalize, clean_name_str) using AlgorithmID as the concrete subclass, including fuzzy name matching.
Also add get_value_from_pes() to ConfigurableParameter as a convenience wrapper around get_values_from_pes() that asserts all returned values are identical and returns the single result.
personalization audit: optionally audit all (unknown) SD keys
By a flag, allow to audit also all Security Domain KVN that we have *not* created ConfigurableParameter subclasses for.
For example, SCP80 has reserved kvn 0x01..0x0f, but we offer only Scp80Kvn01, Scp80Kvn02, Scp80Kvn03. So we would not show kvn 0x03..0x0f in an audit.
This patch includes audits of all SD key kvn there may be in the UPP. This will help to spot SD keys that may already be present in a UPP template, with unexpected / unusual kvn.
To help existing applications transition to a common naming scheme for the SdKey classes, offer this intermediate result, where the SdKey classes' .name are still unchanged as before generating them.
'securityDomain' elements are decoded to ProfileElementSD instances, which keep higher level representations of the key data apart from the decoded[] lists.
So far, apply_val() was dropping binary values in decoded[], which does not work, because ProfileElementSD._pre_encode() overwrites self.decoded[] from the higher level representation.
Implement using - ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify an exsiting entry, or - ProfileElementSD.add_key() to create a new entry.
Before this patch, SdKey parameters seemed to patch PES successfully, but their modifications did not end up in the encoded DER.
(BTW, this does not fix any other errors that may still be present in the various SdKey subclasses, patches coming up.)
personalization audit: optionally audit all (unknown) SD keys
By a flag, allow to audit also all Security Domain KVN that we have *not* created ConfigurableParameter subclasses for.
For example, SCP80 has reserved kvn 0x01..0x0f, but we offer only Scp80Kvn01, Scp80Kvn02, Scp80Kvn03. So we would not show kvn 0x03..0x0f in an audit.
This patch includes audits of all SD key kvn there may be in the UPP. This will help to spot SD keys that may already be present in a UPP template, with unexpected / unusual kvn.
saip SmspTpScAddr: safeguard against decoding error
Reading the TS48 V6.0 eSIM_GTP_SAIP2.1A_NoBERTLV profile results in an exception [1] in SmspTpScAddr. I have a caller that needs to skip erratic values instead of raising.
The underlying issue, I presume, is that either the data needs validation before decode_record_bin(), or decode_record_bin() needs well-defined error handling.
So far I know only of this IndexError, so, as a workaround, catch that.
[1] File "/pysim/pySim/esim/saip/personalization.py", line 617, in get_values_from_pes ef_smsp_dec = ef_smsp.decode_record_bin(f_smsp.body, 1) File "/pysim/pySim/filesystem.py", line 1047, in decode_record_bin return parse_construct(self._construct, raw_bin_data) File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 550, in parse_construct parsed = c.parse(raw_bin_data, total_len=length, **context) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 404, in parse return self.parse_stream(io.BytesIO(data), **contextkw) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 416, in parse_stream return self._parsereport(stream, context, "(parsing)") ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse subobj = sc._parsereport(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse return self.subcon._parsereport(stream, context, path) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse subobj = sc._parsereport(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse return self.subcon._parsereport(stream, context, path) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 820, in _parse return self._decode(obj, context, path) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 268, in _decode if r[-1] == 'f': ~^^^^ File "/application/venv/lib/python3.13/site-packages/osmocom/utils.py", line 50, in __getitem__ return hexstr(super().__getitem__(val)) ~~~~~~~~~~~~~~~~~~~^^^^^ IndexError: string index out of range
Add ConfigurableParameterTest, which applies each parameter to a real UPP DER template and reads it back, comparing results against a stored expected-output snapshot (xo/test_configurable_parameters).
Add TestValidateVal covering validate_val() for Iccid, Imsi, Pin1, Puk1 and K, testing both valid inputs and invalid ones expected to raise ValueError.
Add TestEnumParam covering the EnumParam methods (validate_val, map_name_to_val, map_val_to_name, name_normalize, clean_name_str) using AlgorithmID as the concrete subclass, including fuzzy name matching.
Also add get_value_from_pes() to ConfigurableParameter as a convenience wrapper around get_values_from_pes() that asserts all returned values are identical and returns the single result.
personalization: add get_typical_input_len() to ConfigurableParameter
The aim is to tell a user interface how wide an input text field should be chosen to be convenient -- ideally showing the entire value in all cases, but not too huge for fields that have no sane size limit.
saip: add numeric_base indicator to ConfigurableParameter
By default, numeric_base = None, to indicate that there are no explicit limitations on the number space.
For parameters that are definitely decimal, set numeric_base = 10. For definitely hexadecimal, set numeric_base = 16.
Do the same for ConfigurableParameter as well as ParamSource, so callers can match them up: if a parameter is numeric_base = 10, then omit sources that are numeric_base = 16, and vice versa.
personalization: indicate default ParamSource per ConfigurableParameter
Add default_source class members pointing to ParamSource classes to all ConfigurableParameter subclasses.
This is useful to automatically set up a default ParamSource for a given ConfigurableParameter subclass, during user interaction to produce a batch personalization.
For example, if the user selects a Pin1 parameter, a calling program can implicitly set this to a RandomDigitSource, which will magically make it work the way that most users need.
BTW, default_source and default_value can be combined to configure a matching ParamSource instance:
Implement pySim.esim.saip.batch.BatchPersonalization, generating N eSIM profiles from a preset configuration.
Batch parameters can be fed by a constant, incrementing, random or from CSV rows: add pySim.esim.saip.param_source.* classes to feed such input to each of the BatchPersonalization's ConfigurableParameter instances.
The AlgorithmID has a few preset values, and hardly anyone knows which is which. So instead of entering '1', '2' or '3', make it work with prededined values 'Milenage', 'TUAK' and 'usim-test'.
Implement the enum value part abstractly in new EnumParam.
Make AlgorithmID a subclass of EnumParam and define the values as from pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn
personalization: implement reading back values from a PES
Implement get_values_from_pes(), the reverse direction of apply_val(): read back and return values from a ProfileElementSequence. Implement for all ConfigurableParameter subclasses.
Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this implementation to use the higher level ProfileElementSD members.
Implementation detail:
Implement get_values_from_pes() as classmethod that returns a generator. Subclasses should yield all occurences of their parameter in a given PES.
For example, the ICCID can appear in multiple places. Iccid.get_values_from_pes() yields all of the individual values. A set() of the results quickly tells whether the PES is consistent.
Rationales for reading back values:
This allows auditing an eSIM profile, particularly for producing an output.csv from a batch personalization (that generated lots of random key material which now needs to be fed to an HLR...).
Reading back from a binary result is more reliable than storing the values that were fed into a personalization. By auditing final DER results with this code, I discovered: - "oh, there already was some key material in my UPP template." - "all IMSIs ended up the same, forgot to set up the parameter." - the SdKey.apply() implementations currently don't work, see I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.
saip: add numeric_base indicator to ConfigurableParameter
By default, numeric_base = None, to indicate that there are no explicit limitations on the number space.
For parameters that are definitely decimal, set numeric_base = 10. For definitely hexadecimal, set numeric_base = 16.
Do the same for ConfigurableParameter as well as ParamSource, so callers can match them up: if a parameter is numeric_base = 10, then omit sources that are numeric_base = 16, and vice versa.
saip SmspTpScAddr: safeguard against decoding error
Reading the TS48 V6.0 eSIM_GTP_SAIP2.1A_NoBERTLV profile results in an exception [1] in SmspTpScAddr. I have a caller that needs to skip erratic values instead of raising.
The underlying issue, I presume, is that either the data needs validation before decode_record_bin(), or decode_record_bin() needs well-defined error handling.
So far I know only of this IndexError, so, as a workaround, catch that.
[1] File "/pysim/pySim/esim/saip/personalization.py", line 617, in get_values_from_pes ef_smsp_dec = ef_smsp.decode_record_bin(f_smsp.body, 1) File "/pysim/pySim/filesystem.py", line 1047, in decode_record_bin return parse_construct(self._construct, raw_bin_data) File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 550, in parse_construct parsed = c.parse(raw_bin_data, total_len=length, **context) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 404, in parse return self.parse_stream(io.BytesIO(data), **contextkw) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 416, in parse_stream return self._parsereport(stream, context, "(parsing)") ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse subobj = sc._parsereport(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse return self.subcon._parsereport(stream, context, path) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse subobj = sc._parsereport(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse return self.subcon._parsereport(stream, context, path) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 820, in _parse return self._decode(obj, context, path) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 268, in _decode if r[-1] == 'f': ~^^^^ File "/application/venv/lib/python3.13/site-packages/osmocom/utils.py", line 50, in __getitem__ return hexstr(super().__getitem__(val)) ~~~~~~~~~~~~~~~~~~~^^^^^ IndexError: string index out of range
personalization: add get_typical_input_len() to ConfigurableParameter
The aim is to tell a user interface how wide an input text field should be chosen to be convenient -- ideally showing the entire value in all cases, but not too huge for fields that have no sane size limit.
personalization: implement reading back values from a PES
Implement get_values_from_pes(), the reverse direction of apply_val(): read back and return values from a ProfileElementSequence. Implement for all ConfigurableParameter subclasses.
Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this implementation to use the higher level ProfileElementSD members.
Implementation detail:
Implement get_values_from_pes() as classmethod that returns a generator. Subclasses should yield all occurences of their parameter in a given PES.
For example, the ICCID can appear in multiple places. Iccid.get_values_from_pes() yields all of the individual values. A set() of the results quickly tells whether the PES is consistent.
Rationales for reading back values:
This allows auditing an eSIM profile, particularly for producing an output.csv from a batch personalization (that generated lots of random key material which now needs to be fed to an HLR...).
Reading back from a binary result is more reliable than storing the values that were fed into a personalization. By auditing final DER results with this code, I discovered: - "oh, there already was some key material in my UPP template." - "all IMSIs ended up the same, forgot to set up the parameter." - the SdKey.apply() implementations currently don't work, see I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.
To help existing applications transition to a common naming scheme for the SdKey classes, offer this intermediate result, where the SdKey classes' .name are still unchanged as before generating them.
The AlgorithmID has a few preset values, and hardly anyone knows which is which. So instead of entering '1', '2' or '3', make it work with prededined values 'Milenage', 'TUAK' and 'usim-test'.
Implement the enum value part abstractly in new EnumParam.
Make AlgorithmID a subclass of EnumParam and define the values as from pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn
'securityDomain' elements are decoded to ProfileElementSD instances, which keep higher level representations of the key data apart from the decoded[] lists.
So far, apply_val() was dropping binary values in decoded[], which does not work, because ProfileElementSD._pre_encode() overwrites self.decoded[] from the higher level representation.
Implement using - ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify an exsiting entry, or - ProfileElementSD.add_key() to create a new entry.
Before this patch, SdKey parameters seemed to patch PES successfully, but their modifications did not end up in the encoded DER.
(BTW, this does not fix any other errors that may still be present in the various SdKey subclasses, patches coming up.)
Implement pySim.esim.saip.batch.BatchPersonalization, generating N eSIM profiles from a preset configuration.
Batch parameters can be fed by a constant, incrementing, random or from CSV rows: add pySim.esim.saip.param_source.* classes to feed such input to each of the BatchPersonalization's ConfigurableParameter instances.
personalization audit: optionally audit all (unknown) SD keys
By a flag, allow to audit also all Security Domain KVN that we have *not* created ConfigurableParameter subclasses for.
For example, SCP80 has reserved kvn 0x01..0x0f, but we offer only Scp80Kvn01, Scp80Kvn02, Scp80Kvn03. So we would not show kvn 0x03..0x0f in an audit.
This patch includes audits of all SD key kvn there may be in the UPP. This will help to spot SD keys that may already be present in a UPP template, with unexpected / unusual kvn.
personalization: indicate default ParamSource per ConfigurableParameter
Add default_source class members pointing to ParamSource classes to all ConfigurableParameter subclasses.
This is useful to automatically set up a default ParamSource for a given ConfigurableParameter subclass, during user interaction to produce a batch personalization.
For example, if the user selects a Pin1 parameter, a calling program can implicitly set this to a RandomDigitSource, which will magically make it work the way that most users need.
BTW, default_source and default_value can be combined to configure a matching ParamSource instance:
Implement pySim.esim.saip.batch.BatchPersonalization, generating N eSIM profiles from a preset configuration.
Batch parameters can be fed by a constant, incrementing, random or from CSV rows: add pySim.esim.saip.param_source.* classes to feed such input to each of the BatchPersonalization's ConfigurableParameter instances.
Implement pySim.esim.saip.batch.BatchPersonalization, generating N eSIM profiles from a preset configuration.
Batch parameters can be fed by a constant, incrementing, random or from CSV rows: add pySim.esim.saip.param_source.* classes to feed such input to each of the BatchPersonalization's ConfigurableParameter instances.
personalization: implement reading back values from a PES
Implement get_values_from_pes(), the reverse direction of apply_val(): read back and return values from a ProfileElementSequence. Implement for all ConfigurableParameter subclasses.
Future: SdKey.get_values_from_pes() is reading pe.decoded[], which works fine, but I07dfc378705eba1318e9e8652796cbde106c6a52 will change this implementation to use the higher level ProfileElementSD members.
Implementation detail:
Implement get_values_from_pes() as classmethod that returns a generator. Subclasses should yield all occurences of their parameter in a given PES.
For example, the ICCID can appear in multiple places. Iccid.get_values_from_pes() yields all of the individual values. A set() of the results quickly tells whether the PES is consistent.
Rationales for reading back values:
This allows auditing an eSIM profile, particularly for producing an output.csv from a batch personalization (that generated lots of random key material which now needs to be fed to an HLR...).
Reading back from a binary result is more reliable than storing the values that were fed into a personalization. By auditing final DER results with this code, I discovered: - "oh, there already was some key material in my UPP template." - "all IMSIs ended up the same, forgot to set up the parameter." - the SdKey.apply() implementations currently don't work, see I07dfc378705eba1318e9e8652796cbde106c6a52 for a fix.
personalization: add get_typical_input_len() to ConfigurableParameter
The aim is to tell a user interface how wide an input text field should be chosen to be convenient -- ideally showing the entire value in all cases, but not too huge for fields that have no sane size limit.
To help existing applications transition to a common naming scheme for the SdKey classes, offer this intermediate result, where the SdKey classes' .name are still unchanged as before generating them.
saip SmspTpScAddr: safeguard against decoding error
Reading the TS48 V6.0 eSIM_GTP_SAIP2.1A_NoBERTLV profile results in an exception [1] in SmspTpScAddr. I have a caller that needs to skip erratic values instead of raising.
The underlying issue, I presume, is that either the data needs validation before decode_record_bin(), or decode_record_bin() needs well-defined error handling.
So far I know only of this IndexError, so, as a workaround, catch that.
[1] File "/pysim/pySim/esim/saip/personalization.py", line 617, in get_values_from_pes ef_smsp_dec = ef_smsp.decode_record_bin(f_smsp.body, 1) File "/pysim/pySim/filesystem.py", line 1047, in decode_record_bin return parse_construct(self._construct, raw_bin_data) File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 550, in parse_construct parsed = c.parse(raw_bin_data, total_len=length, **context) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 404, in parse return self.parse_stream(io.BytesIO(data), **contextkw) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 416, in parse_stream return self._parsereport(stream, context, "(parsing)") ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse subobj = sc._parsereport(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse return self.subcon._parsereport(stream, context, path) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2236, in _parse subobj = sc._parsereport(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 2770, in _parse return self.subcon._parsereport(stream, context, path) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 428, in _parsereport obj = self._parse(stream, context, path) File "/application/venv/lib/python3.13/site-packages/construct/core.py", line 820, in _parse return self._decode(obj, context, path) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/application/venv/lib/python3.13/site-packages/osmocom/construct.py", line 268, in _decode if r[-1] == 'f': ~^^^^ File "/application/venv/lib/python3.13/site-packages/osmocom/utils.py", line 50, in __getitem__ return hexstr(super().__getitem__(val)) ~~~~~~~~~~~~~~~~~~~^^^^^ IndexError: string index out of range
'securityDomain' elements are decoded to ProfileElementSD instances, which keep higher level representations of the key data apart from the decoded[] lists.
So far, apply_val() was dropping binary values in decoded[], which does not work, because ProfileElementSD._pre_encode() overwrites self.decoded[] from the higher level representation.
Implement using - ProfileElementSD.find_key() and SecurityDomainKeyComponent to modify an exsiting entry, or - ProfileElementSD.add_key() to create a new entry.
Before this patch, SdKey parameters seemed to patch PES successfully, but their modifications did not end up in the encoded DER.
(BTW, this does not fix any other errors that may still be present in the various SdKey subclasses, patches coming up.)
The AlgorithmID has a few preset values, and hardly anyone knows which is which. So instead of entering '1', '2' or '3', make it work with prededined values 'Milenage', 'TUAK' and 'usim-test'.
Implement the enum value part abstractly in new EnumParam.
Make AlgorithmID a subclass of EnumParam and define the values as from pySim/esim/asn1/saip/PE_Definitions-3.3.1.asn
personalization: indicate default ParamSource per ConfigurableParameter
Add default_source class members pointing to ParamSource classes to all ConfigurableParameter subclasses.
This is useful to automatically set up a default ParamSource for a given ConfigurableParameter subclass, during user interaction to produce a batch personalization.
For example, if the user selects a Pin1 parameter, a calling program can implicitly set this to a RandomDigitSource, which will magically make it work the way that most users need.
BTW, default_source and default_value can be combined to configure a matching ParamSource instance:
personalization audit: optionally audit all (unknown) SD keys
By a flag, allow to audit also all Security Domain KVN that we have *not* created ConfigurableParameter subclasses for.
For example, SCP80 has reserved kvn 0x01..0x0f, but we offer only Scp80Kvn01, Scp80Kvn02, Scp80Kvn03. So we would not show kvn 0x03..0x0f in an audit.
This patch includes audits of all SD key kvn there may be in the UPP. This will help to spot SD keys that may already be present in a UPP template, with unexpected / unusual kvn.
saip: add numeric_base indicator to ConfigurableParameter
By default, numeric_base = None, to indicate that there are no explicit limitations on the number space.
For parameters that are definitely decimal, set numeric_base = 10. For definitely hexadecimal, set numeric_base = 16.
Do the same for ConfigurableParameter as well as ParamSource, so callers can match them up: if a parameter is numeric_base = 10, then omit sources that are numeric_base = 16, and vice versa.
contrib/jenkins.sh: separate JOB_TYPE for card tests
A separate job gives us a possibility to skip tests requiring physical cards for specific commits that do not touch the core logic. See the related commits in osmo-ci.git.
contrib/jenkins.sh: separate JOB_TYPE for card tests
A separate job gives us a possibility to skip tests requiring physical cards for specific commits that do not touch the core logic. See the related commits in osmo-ci.git.
contrib/jenkins.sh: separate JOB_TYPE for card tests
A separate job gives us a possibility to skip tests requiring physical cards for specific commits that do not touch the core logic. See the related commits in osmo-ci.git.
contrib/jenkins.sh: separate JOB_TYPE for card tests
A separate job gives us a possibility to skip tests requiring physical cards for specific commits that do not touch the core logic. See the related commits in osmo-ci.git.
The legacy code found in legacy/cards.py does not use the modern construct based encoder (pySim-read uses it). The card classes either use their own implementation of update_smsp or use the generic method provided by the SimCard class. The latter one is true for FairwavesSIM and WavemobileSim.
Unfortunately the implementation found in the SimCard is wrong. It adds padding at the end of the file instead of the beginning. This completely messes up the contents of EF.SMSP for the cards using this method. To fix this, let's use the leftpad feature provided by the update_record. This will ensure a correct alignment of the file contents.
ts_51_011/EF.SMSP: fix handling of 'alpha_id' field
The field 'alpha_id' is technically not an optional field, even though the specification describes it as optional. Once the card manufacturer decides that the field should be present, it must be always present and vice versa.
(see code comment for a more detailed description)
The legacy code found in legacy/cards.py does not use the modern construct based encoder (pySim-read uses it). The card classes either use their own implementation of update_smsp or use the generic method provided by the SimCard class. The latter one is true for FairwavesSIM and WavemobileSim.
Unfortunately the implementation found in the SimCard is wrong. It adds padding at the end of the file instead of the beginning. This completely messes up the contents of EF.SMSP for the cards using this method. To fix this, let's use the leftpad feature provided by the update_record. This will ensure a correct alignment of the file contents.
ts_51_011/EF.SMSP: fix handling of 'alpha_id' field
The field 'alpha_id' is technically not an optional field, even though the specification describes it as optional. Once the card manufacturer decides that the field should be present, it must be always present and vice versa.
(see code comment for a more detailed description)
ts_51_011/EF.SMSP: fix handling of 'alpha_id' field
The field 'alpha_id' is technically not an optional field, even though the specification describes it as optional. Once the card manufacturer decides that the field should be present, it must be always present and vice versa.
(see code comment for a more detailed description)
The legacy code found in legacy/cards.py does not use the modern construct based encoder (pySim-read uses it). The card classes either use their own implementation of update_smsp or use the generic method provided by the SimCard class. The latter one is true for FairwavesSIM and WavemobileSim.
Unfortunately the implementation found in the SimCard is wrong. It adds padding at the end of the file instead of the beginning. This completely messes up the contents of EF.SMSP for the cards using this method. To fix this, let's use the leftpad feature provided by the update_record. This will ensure a correct alignment of the file contents.
ts_51_011/EF.SMSP: fix handling of 'alpha_id' field
The field 'alpha_id' is technically not an optional field, even though the specification describes it as optional. Once the card manufacturer decides that the field should be present, it must be always present and vice versa.
(see code comment for a more detailed description)
ts_51_011/EF.SMSP: fix handling of 'alpha_id' field
The field 'alpha_id' is technically not an optional field, even though the specification describes it as optional. Once the card manufacturer decides that the field should be present, it must be always present and vice versa.
(see code comment for a more detailed description)
ts_51_011/EF.SMSP: fix handling of 'alpha_id' field
The field 'alpha_id' is technically not an optional field, even though the specification describes it as optional. Once the card manufacturer decides that the field should be present, it must be always present and vice versa.
(see code comment for a more detailed description)
contrib/jenkins.sh: separate JOB_TYPE for card tests
A separate job gives us a possibility to skip tests requiring physical cards for specific commits that do not touch the core logic. See the related commits in osmo-ci.git.
contrib/jenkins.sh: add support for "Jenkins: skip-card-tests" marker
Add a have_jenkins_marker() helper that checks the commit message for a given value on the "Jenkins:" footer line, for instance:
Jenkins: skip-card-tests
When this marker is present, the real-card integration tests (pySim-prog_test and pySim-shell_test) are skipped, allowing commits that don't affect card I/O to bypass long card-dependent test runs.
contrib/jenkins.sh: add support for "Jenkins: skip-card-tests" marker
Add a have_jenkins_marker() helper that checks the commit message for a given value on the "Jenkins:" footer line, for instance:
Jenkins: skip-card-tests
When this marker is present, the real-card integration tests (pySim-prog_test and pySim-shell_test) are skipped, allowing commits that don't affect card I/O to bypass long card-dependent test runs.
requirements: ensure safe version of PyYAML >= 5.4 (CVE-2020-1747)
PyYAML versions 5.1–5.3.1 are vulnerable to CVE-2020-1747, which allows arbitrary code execution through yaml.FullLoader. While PyYAML 5.4+ patches this, the dependency specification (pyyaml >= 5.1) doesn't guarantee a safe version. Let's increase the requirement to version 5.4 to ensure a safe version of is used.
This patch is based on suggestions from: "YanTong C <chyeyantong03@gmail.com>"
osmo-smdpp.py: fix path Traversal Bypass in SM-DP+ (CWE-22)
Root Cause: os.path.commonprefix() compares strings character-by-character, NOT by path components. This is a well-known Python antipattern (Python docs explicitly warn: "this function may return invalid paths because it works a character at a time").
Attack Context: The matchingId parameter is received from a network client via the GSMA ES9+ authenticateClient API endpoint (POST to /gsma/rsp2/es9plus/authenticateClient). The SM-DP+ server is a Twisted web application listening on port 443. An unauthenticated eUICC client sends the matchingId in the ctxParamsForCommonAuthentication ASN.1 structure.
Fix: Replace os.path.commonprefix() with proper path component checking:
requirements: ensure safe version of PyYAML >= 5.4 (CVE-2020-1747)
PyYAML versions 5.1–5.3.1 are vulnerable to CVE-2020-1747, which allows arbitrary code execution through yaml.FullLoader. While PyYAML 5.4+ patches this, the dependency specification (pyyaml >= 5.1) doesn't guarantee a safe version. Let's increase the requirement to version 5.4 to ensure a safe version of is used.
This patch is based on suggestions from: "YanTong C <chyeyantong03@gmail.com>"
pySim-prog: fix Insecure PRNG for SIM Authentication Keys (CWE-338)
Root Cause: pySim-prog.py uses Python's random module (Mersenne Twister MT19937) to generate Ki and OPC — the root authentication keys for SIM cards. MT19937 is a deterministic PRNG that is not cryptographically secure. Its internal state (624 × 32-bit words, 19,937 bits) can be fully recovered after observing 624 consecutive outputs.
Impact: 1. SIM Card Cloning: An attacker who determines the PRNG state can predict all Ki/OPC values generated before and after. With these keys, SIM cards can be cloned. 2. Network Authentication Bypass: Ki/OPC are used in the Milenage algorithm for 3G/4G/5G authentication. Predictable keys mean an attacker can authenticate as any subscriber whose SIM was provisioned with the weak RNG. 3. Batch Compromise: In bulk provisioning scenarios (pySim-prog's primary use case), hundreds or thousands of SIMs may be programmed sequentially. Compromising one batch means recovering the PRNG state to predict all keys.
The current behavior we implement in the method __send_apdu_T0 is incomplete. Some details discussed in ETSI TS 102 221, section 7.3.1.1.4, clause 4 seem to be not fully implemented. We may also end up sending a GET RESPONSE in other APDU cases than case 4 (the only case that uses the GET RESPONSE command).
The current behavior we implement in the method __send_apdu_T0 is incomplete. Some details discussed in ETSI TS 102 221, section 7.3.1.1.4, clause 4 seem to be not fully implemented. We may also end up sending a GET RESPONSE in other APDU cases than case 4 (the only case that uses the GET RESPONSE command).
The current behavior we implement in the method __send_apdu_T0 is incomplete. Some details discussed in ETSI TS 102 221, section 7.3.1.1.4, clause 4 seem to be not fully implemented. We may also end up sending a GET RESPONSE in other APDU cases than case 4 (the only case that uses the GET RESPONSE command).
filesystem: JsonEditor: offer interactive retry on error
When json.loads() fails (e.g. the user made a syntax mistake), prompt the user with "Re-open file for editing? [y]es/[n]o:" and loop back to the editor if they answer 'y' or 'yes'. If the user declines, return the original unmodified value so no write is attempted; the temp file is still cleaned up by __exit__() in that case.
tests: fix TransRecEF _test_de_encode to operate at file level
Previously, _test_de_encode vectors for TransRecEF subclasses were tested via decode_record_hex()/encode_record_hex(), i.e. one record at a time, with the decoded value being a scalar.
Switch test_de_encode_record() in TransRecEF_Test to use decode_hex() / encode_hex() instead, so that vectors represent whole-file content (decoded value is a list of records) -- consistent with how LinFixedEF handles _test_de_encode. Update all existing vectors accordingly.
A plain NamedTemporaryFile is sufficient here: we only need a single file, not a directory to hold it. Using NamedTemporaryFile is simpler (no subdirectory to manage) and gives us a .json suffix for free, which editors use for syntax highlighting.
A plain NamedTemporaryFile is sufficient here: we only need a single file, not a directory to hold it. Using NamedTemporaryFile is simpler (no subdirectory to manage) and gives us a .json suffix for free, which editors use for syntax highlighting.
filesystem: JsonEditor: offer interactive retry on error
When json.loads() fails (e.g. the user made a syntax mistake), prompt the user with "Re-open file for editing? [y]es/[n]o:" and loop back to the editor if they answer 'y' or 'yes'. If the user declines, return the original unmodified value so no write is attempted; the temp file is still cleaned up by __exit__() in that case.
When invoking `edit_binary_decoded` or `edit_record_decoded`, the temp file opened in the editor now contains the EF's encode/decode test vectors as //-comment lines below the JSON content, similar to how 'git commit' appends comments to the commit message template. The comment block is stripped before JSON parsing on save, so it has no effect on the written data.
The feature is implemented via a new module-level JsonEditor context manager class that encapsulates the full edit cycle:
* write JSON + examples to a TemporaryDirectory * invoke the editor * read back, strip //-comments, parse and return the result
Main points/rationales of the refactoring, details below: 1) common validation implementation 2) offer classmethods
The new features are optional, and will be heavily used by batch personalization patches coming soon.
Implement Iccid and Imsi to use the new way, with a common abstract DecimalParam implementation.
So far leave the other parameter classes working as they always did, to follow suit in subsequent commits.
Details:
1) common validation implementation: There are very common validation steps in the various parameter implementations. It is more convenient and much more readable to implement those once and set simple validation parameters per subclass. So there now is a validate_val() classmethod, which subclasses can use as-is to apply the validation parameters -- or subclasses can override their cls.validate_val() for specialized validation. (Those subclasses that this patch doesn't touch still override the self.validate() instance method. Hence they still work as before this patch, but don't use the new common features yet.)
2) offer stateless classmethods: It is useful for... - batch processing of multiple profiles (in upcoming patches) and - user input validation to be able to have classmethods that do what self.validate() and self.apply() do, but do not modify any self.* members. So far the paradigm was to create a class instance to keep state about the value. This remains available, but in addition we make available the paradigm of a singleton that is stateless (the classmethods). Using self.validate() and self.apply() still work the same as before this patch, i.e. via self.input_value and self.value -- but in addition, there are now classmethods that don't touch self.* members.
