> Don't sign abstract messages, sign bags of bytes.
Isn't the relevant question, "what are the bytes that are being signed?" That's what the specification seems to be answering here.
I believe it's implicit that the signature is actually of a hash of the message itself, which is why they need to serialize the message in a deterministic fashion, because otherwise the hash would be different in different circumstances for the same message. If you want to validate the signature, you need to make sure that the hash being signed is of the same exact string that you are verifying.
> requires canonical representations, a major PITA and source of bugs
Yes, absolutely, especially with regards to JSON, which doesn't have a fully deterministic serialization standard. The scuttlebutt specification seems to assume that JSON.stringify() will produce a consistent output, but that is not really the case, per my understanding, at least not with regards to objects.
From the Scuttlebutt specification:
> The canonical format is defined by the ECMA-262 6th Edition section JSON.stringify. For an example, see how the above message is formatted.
JSON object serialization in Javascript depends on the insertion order of the members of an object, so if you somehow change the ordering in which your keys are updated as the object is built, you will get a different string output, even if the data of the underlying object is the same. And this is just within the JavaScript runtime—if you are implementing this using another programming language, you can't just rely on the ECMA standard to determine the ordering of your object members at the time of serialization.
If I recall properly, JSON serialization is ambiguous in the specification in a variety of ways, which results in different implementations outputting subtly different strings, even under ordinary circumstances.
It seems weird that you would write a formal, generalized protocol specification relying on the idiosyncratic implementation details of JavaScript, for such an important thing as cryptographic signatures, as the Scuttlebutt specification seems to do here.
> It seems weird that you would write a formal, generalized protocol specification relying on the idiosyncratic implementation details of JavaScript, for such an important thing as cryptographic signatures, as the Scuttlebutt specification seems to do here.
The Protocol Guide was created after the initial implementation and its protocol were already in wide use, and the quirks were discovered while re-implementing it.
More info about implementations here (Node.js, Go, Rust x2, and Python; additionally there are implementations of varying states in Java, C, Haskell, Erlang and probably others):
https://dev.scuttlebutt.nz/#/?id=implementations
---
If making a new protocol using signatures over JSON objects, one might use this:
JWS Clear Text JSON Signature Option (JWS/CT)
https://datatracker.ietf.org/doc/html/draft-jordan-jws-ct
JWS/CT uses JSON Web Signature (JWS) [RFC7515], JSON Canonicalizion Scheme (JCS) [RFC8785], and I-JSON [RFC7493] (subset of JSON for interoperability)
Matrix does it like this as well. Servers have to serialize an object as JSON (specified as Python code), then add the signature to the object and serialize it again to send it on the network. https://spec.matrix.org/v1.1/appendices/#signing-json
Exactly this. You need deterministic serialization, because you need to be sure that when the _same_ object is constructed in different settings, it is going to hash consistently. In Hyper Hyper Space [1], the set of basic types as well as the composition primitives used to construct all data structures have built-in deterministic serialization, just for this reason (e.g. a set will serialize into a deterministically ordered list, etc.)
> Exactly this. You need deterministic serialization, because you need to be sure that when the _same_ object is constructed in different settings, it is going to hash consistently.
I can see how this might matter in some specific systems, but when we're talking about signatures only the signer constructs the object. Stuff like the "JWS/CT using JWS and JSON Canonicalization" recommended in a sibling comment is a complete misdesign for virtually all signing use cases. That's why "our signature scheme requires canonical representations" is a red flag.
But "the signer" here is a cryptographic identity, that may be present in more than one device. So, even when conceptually it is just one entity, in practice it may be several computers doing something independently, and one may need the result to be the same given identical inputs.
Offhand that sounds really cursed: Signing the "same" thing multiple times with the same key shared among multiple devices and also expecting the signatures to be identical. I honestly can't think of anything that would require something like this and it sounds like a really bad idea as well.
Just to clarify something I've seen a few siblings confused about: Saying your signature scheme should not rely on canonical representations doesn't mean you shouldn't parse the input properly before processing it (e.g. checking that the input corresponds to a schema). It just means those two things are separate things best done separately. It's a little like NATting and firewalling: often confused with each other, or people arguing one acts like the other, but they're really separate things, and everything is much easier once you understand and heed that.
Isn't the relevant question, "what are the bytes that are being signed?" That's what the specification seems to be answering here.
I believe it's implicit that the signature is actually of a hash of the message itself, which is why they need to serialize the message in a deterministic fashion, because otherwise the hash would be different in different circumstances for the same message. If you want to validate the signature, you need to make sure that the hash being signed is of the same exact string that you are verifying.
> requires canonical representations, a major PITA and source of bugs
Yes, absolutely, especially with regards to JSON, which doesn't have a fully deterministic serialization standard. The scuttlebutt specification seems to assume that JSON.stringify() will produce a consistent output, but that is not really the case, per my understanding, at least not with regards to objects.
From the Scuttlebutt specification:
> The canonical format is defined by the ECMA-262 6th Edition section JSON.stringify. For an example, see how the above message is formatted.
JSON object serialization in Javascript depends on the insertion order of the members of an object, so if you somehow change the ordering in which your keys are updated as the object is built, you will get a different string output, even if the data of the underlying object is the same. And this is just within the JavaScript runtime—if you are implementing this using another programming language, you can't just rely on the ECMA standard to determine the ordering of your object members at the time of serialization.
If I recall properly, JSON serialization is ambiguous in the specification in a variety of ways, which results in different implementations outputting subtly different strings, even under ordinary circumstances.
It seems weird that you would write a formal, generalized protocol specification relying on the idiosyncratic implementation details of JavaScript, for such an important thing as cryptographic signatures, as the Scuttlebutt specification seems to do here.