Allow serial number 0 for root CA certificates

[jackctj117]

Fixes #8615

This pull request updates the logic for validating X.509 certificate serial numbers in wolfcrypt/src/asn.c. The main change is to improve compliance with RFC 5280 while allowing for real-world exceptions involving root CAs. The previous strict check for zero serial numbers is now more nuanced, permitting serial number 0 for self-signed CA certificates but still rejecting it for other certificates.

Certificate serial number validation improvements:

• Moved and updated the check for serial numbers of 0 to allow self-signed CA (root) certificates to have a serial number of 0, while still treating serial 0 as an error for all other certificates. This better aligns with RFC 5280 and real-world trust store practices. [1] [2]
• Removed the previous check that always treated a serial number of 0 as an error, regardless of certificate type.

Testing

Tested with certificates generated using OpenSSL to verify all scenarios:

• Root CA with serial 0 loads successfully (previously failed, now passes)
• End-entity cert with serial 0 is correctly rejected
• Normal end-entity cert signed by root CA with serial 0 verifies successfully
• Self-signed non-CA cert with serial 0 is correctly rejected

[kareem-wolfssl]

Changes look good.

Can you add some test cases with a CA and leaf cert with a serial of 0?

[jackctj117]

@kareem-wolfssl it looks like the failures are looking for an expected failure due to a self-signed CA certificate with serial number 0.

[kareem-wolfssl]

@kareem-wolfssl it looks like the failures are looking for an expected failure due to a self-signed CA certificate with serial number 0.

Yes, you will need to update the failing test test_MakeCertWith0Ser since we no longer expect a root CA with serial 0 to fail.

[dgarske]

@jackctj117 looks like some valid unit test failures you will need to look into. All related to --enable-asn=original.

FAILURES:
   390: test_SerialNumber0_RootCA