In OpenSSL 3, ASN1_item_verify*() can return 2 on error.

[botovq]

Introduced in #10942 as part of extracting ASN1_item_verify_ctx() from ASN1_item_verify() when a ret = -1; was deleted after the goto err here:

openssl/crypto/asn1/a_verify.c

Lines 143 to 154 in 5bbdbce

	ret = pkey->ameth->item_verify(ctx, it, data, alg, signature, pkey);
	/*
	* Return values meaning:
	* <=0: error.
	* 1: method does everything.
	* 2: carry on as normal, method has called EVP_DigestVerifyInit()
	*/
	if (ret <= 0)
	ERR_raise(ERR_LIB_ASN1, ERR_R_EVP_LIB);
	if (ret <= 1)
	goto err;
	} else {

If item_verify() returns 2 (which it usually will for RSA-PSS for example) the control flow would skip here and return 2 if ASN1_item_i2d() errors:

openssl/crypto/asn1/a_verify.c

Lines 202 to 210 in 5bbdbce

	inl = ASN1_item_i2d(data, &buf_in, it);
	if (inl <= 0) {
	ERR_raise(ERR_LIB_ASN1, ERR_R_INTERNAL_ERROR);
	goto err;
	}
	if (buf_in == NULL) {
	ERR_raise(ERR_LIB_ASN1, ERR_R_ASN1_LIB);
	goto err;
	}

[t8m]

Fix in #24576