LibCrypto: Make PKSystem methods return a ByteBuffer directly

It used to be that the caller would supply a buffer to write the output
to. This created an anti-pattern in multiple places where the caller
would allocate a `ByteBuffer` and then use `.bytes()` to provide it to
the `PKSystem` method. Then the callee would resize the output buffer
and reassign it, but because the resize was on `Bytes` and not on
`ByteBuffer`, the caller using the latter would cause a bug.

Additionally, in pretty much all cases the buffer was pre-allocated
shortly before.
This commit is contained in:
devgianlu 2024-12-25 22:04:38 +01:00 committed by Ali Mohammad Pur
commit 0fc02d4d00
Notes: github-actions[bot] 2025-01-13 16:02:16 +00:00
9 changed files with 69 additions and 112 deletions

View file

@ -191,11 +191,7 @@ void TLSv12::build_rsa_pre_master_secret(PacketBuilder& builder)
}
Crypto::PK::RSA_PKCS1_EME rsa(certificate.public_key.rsa);
Vector<u8, 32> out;
out.resize(rsa.output_size());
auto outbuf = out.span();
MUST(rsa.encrypt(m_context.premaster_key, outbuf));
auto outbuf = MUST(rsa.encrypt(m_context.premaster_key));
if constexpr (TLS_DEBUG) {
dbgln("Encrypted: ");