Description

Currently Lighthouse takes a long time to start up while it reloads the public key cache from disk.

The fundamental reason for this is that it stores the BLS pubkeys in their compressed form and then has to decompress every one of them for the in-memory cache:

Just changing that PublicKeyBytes to PublicKey would not be sufficient, because the two types share the same compressed SSZ representation:

So we would need to create new methods on bls::GenericPublicKey like serialize_uncompressed and deserialize_uncompressed which use the 96-byte serialization of the PublicKey rather than the 48-byte compressed serialization. For blst, we can use the PublicKey::serialize function. For milagro we can use PublicKey::as_uncompressed_bytes. Both libraries also include deserialisation functions acting on the uncompressed bytes.

Once these methods are in place in the bls wrapper lib, we can call them from the DatabasePubkey wrapper. We'll need two versions of DatabasePubkey in order to implement the schema migration from compressed keys to uncompressed keys.

By storing the bytes uncompressed we'll double the disk space required for the pubkey cache from 500k * 48 bytes = 24 MB to 48 MB. Given the overall size of the database I believe this is an acceptable trade-off. We could attempt to apply a faster compression algorithm like zstd to the data, but seeing as the input is pseudo-random bytes I suspect this naive compression would be ineffective.

Version

Lighthouse v3.0.0