OpCerts chains are required to be in memory and storage as a block, preventing memory optimizations

[tcarmelveilleux]

Problem

AdminPairingInfo has a large linear blob of storage for one set of opcerts for a given fabric:

    struct StorableAdminPairingInfo
    {
        uint16_t mAdmin;    /* This field is serialized in LittleEndian byte order */
        uint64_t mNodeId;   /* This field is serialized in LittleEndian byte order */
        uint64_t mFabricId; /* This field is serialized in LittleEndian byte order */
        uint16_t mVendorId; /* This field is serialized in LittleEndian byte order */

        uint16_t mRootCertLen; /* This field is serialized in LittleEndian byte order */
        uint16_t mOpCertLen;   /* This field is serialized in LittleEndian byte order */

        Crypto::P256SerializedKeypair mOperationalKey;
        uint8_t mRootCert[kMaxChipCertSize];
        // The operationa credentials set can have up to two certs -> ICAC and NOC
        uint8_t mOperationalCert[kMaxChipCertSize * 2];
        char mFabricLabel[kFabricLabelMaxLengthInBytes + 1] = { '\0' };
    };

This contains 3x opcerts at kMaxChipCertSize (600).

Furthermore, the OperationalCredentialSet expects to keep all of these in-memory, per fabric, in a way that they cannot be used in place.

Both of these together cause a large amount of data to be maintained in RAM on embedded platform, when they are actually very infrequently used (only during case), and only ever used one at a time.

This prevents memory optimizations on memory-starved devices and is an obstacle to fitting multi-fabric support on some low-RAM targets. Already some Silabs parts testing commissioning flow with real opcerts are running out of heap due to fragmentation because of these large blocks.

Proposed Solution

This particular problem can be addressed in several steps:

1. Communicating to the dev team that operational certs are infrequently used, and their lookup from non-volatile storage on-demand is a very small cost compared to the time cost of signature verifications in which they take part, and memory cost in maintaining them in RAM working set permanently.
2. Split kMaxChipCertSize from 600 (which is the max for DER certs for Device Attestation and for DER version of opcerts) to two values:

• kMaxChipOpCertSize = 400 -> The maximum size of a TLV cert
• kMaxCHIPDerCertSize = 600 -> The guaranteed max size of a DER cert like for DAC/PAI/PAA, and opcert at commissioner prior to TLV conversion
• This is a savings of 33%

3. Refactor OperationalCredentialSet and code using it, to never expect an "already in-place" version, and rather to look them up from persisted storage every time. Currently this keeps copies of all elements of the opcert chain in HEAP (see AdminPairingInfo::SetXXXCert which allocs into heap). Release is done at the end of CASE, but the whole set (3x certs) is needed in memory at a time. If N incoming CASE sessions are in progress, Nx3 certs are maintained.
4. Introduce a TlvCertificateStore interface, which can be provided by platform to enable optimization of storage/lookup of operational certs.

All primitives in the spec (and their related implementations) should be able to work with OpCerts one by one. They do not need to be maintained in memory permanently, or contiguously.

The following interface sketch would fulfill all requirements of Matter Operational Certs:

• Allow removal of roots by SKID
• Allow lookup of "next in chain" by AKID/SKID
• Allow lookup of NOC by full node reference if needed (PK + fabric id + node id)
• Add simple implementation of AddTrustedRootCertificate/RemoveTrustedRootCertificate/AddOpCert/UpdateOpCert/RemoveFabric to spec with no dangling issues and no duplication of certificates
• Allow optimization of large blob storage by platform, since OpCerts are the largest contributor to non-volatile stored state in quantity and size
• Make clear that you can just look-up the certs in a temp buffer.

// Store/Retrieve TLV certificates by properties.
// Has all the primitives that may be needed to implement
// Matter operational certs
class TlvCertificateStore
{
 public:
  // Search by SKID --> Good to find known cert / next in chain
  virtual CHIP_ERROR GetBySkid(
    const uint8_t* cert_skid,
    size_t skid_len,
    uint8_t * cert_buffer_out,
    size_t cert_buffer_len) = 0;

  // Search by Public Key --> Used by GetByFullFabricRef
  virtual CHIP_ERROR GetByPublicKey(
    const uint8_t* public_key,
    size_t public_key_len,
    uint8_t * cert_buffer_out,
    size_t cert_buffer_len) = 0;

  // Good to find a NOC, so that chain can be followed
  virtual CHIP_ERROR GetByFullFabricRef(
    const uint8_t* public_key,
    size_t public_key_len,
    uint64_t fabric_id,
    uint64_t node_id,
    uint8_t * cert_buffer_out,
    size_t cert_buffer_len) = 0;

  // Insert any cert
  virtual CHIP_ERROR Insert(uint8_t * cert_buffer, size_t cert_buffer_len) = 0;

  // Remove by public key. Unambiguous.
  virtual CHIP_ERROR RemoveByPublicKey(const uint8_t* public_key, size_t public_key_len) = 0;
  // Remove by content. Unambiguous.
  virtual CHIP_ERROR RemoveByCertContent(const uint8_t* cert_buffer, size_t cert_buffer_len) = 0;
};

[tcarmelveilleux]

The trust store concepts for roots mentioned in this issue would help solve #17208 and #17209

[woody-apple]

@tcarmelveilleux is this now complete? What remains here?