zecfoundation
/
zebra
mirror of https://github.com/ZcashFoundation/zebra.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Create CompactSize64 for non-message-length fields (#3008) 

* Add CompactSize64 and CompactSizeMessage types

But don't remove read_compactsize and write_compactsize yet.

* Fix CompactSize capitalisation

```sh
fastmod compactSize CompactSize zebra* book
fastmod compactsize CompactSize zebra* book
```

* Make CompactSize patterns consistent with integer lengths

* Replace unwrap_err with asserting is_err

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>

* Replace a panic with an assertion

* Make generic serialization use CompactSizeMessage

* Fix type inference and borrow-checker errors

* Doctest fixes

Co-authored-by: Janito Vaqueiro Ferreira Filho <janito.vff@gmail.com>
This commit is contained in:
teor 2021-11-05 01:57:30 +10:00 committed by GitHub
parent d03161c63f
commit 01e63da3fe
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 461 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
book/src/dev/rfcs/0010-v5-transaction.md
View File

@ -53,7 +53,7 @@ Orchard uses `Halo2Proof`s with corresponding signature type changes. Each Orcha
[other-transaction-v5-changes]: #other-transaction-v5-changes
V5 transactions split `Spend`s, `Output`s, and `AuthorizedAction`s into multiple arrays,
with a single `compactsize` count before the first array. We add new
with a single `CompactSize` count before the first array. We add new
`zcash_deserialize_external_count` and `zcash_serialize_external_count` utility functions,
which make it easier to serialize and deserialize these arrays correctly.

16
zebra-chain/src/serialization.rs
View File

@ -6,6 +6,7 @@
//! `ReadZcashExt`, extension traits for `io::Read` and `io::Write` with utility functions
//! for reading and writing data (e.g., the Bitcoin variable-integer format).
mod compact_size;
mod constraint;
mod date_time;
mod error;
@ -14,13 +15,17 @@ mod write_zcash;
mod zcash_deserialize;
mod zcash_serialize;
pub(crate) mod serde_helpers;
pub mod sha256d;
pub(crate) mod serde_helpers;
#[cfg(any(test, feature = "proptest-impl"))]
pub mod arbitrary;
#[cfg(test)]
pub mod tests;
pub use compact_size::{CompactSize64, CompactSizeMessage};
pub use constraint::AtLeastOne;
pub use date_time::{DateTime32, Duration32};
pub use error::SerializationError;
@ -31,9 +36,6 @@ pub use zcash_deserialize::{
ZcashDeserialize, ZcashDeserializeInto,
};
pub use zcash_serialize::{
zcash_serialize_bytes, zcash_serialize_bytes_external_count, zcash_serialize_external_count,
FakeWriter, ZcashSerialize, MAX_PROTOCOL_MESSAGE_LEN,
zcash_serialize_bytes, zcash_serialize_bytes_external_count, zcash_serialize_empty_list,
zcash_serialize_external_count, FakeWriter, ZcashSerialize, MAX_PROTOCOL_MESSAGE_LEN,
};
#[cfg(test)]
pub mod tests;

23
zebra-chain/src/serialization/arbitrary.rs
View File

@ -1,9 +1,13 @@
//! Arbitrary data generation for serialization proptests
use super::{read_zcash::canonical_socket_addr, DateTime32};
use std::{convert::TryInto, net::SocketAddr};
use chrono::{TimeZone, Utc, MAX_DATETIME, MIN_DATETIME};
use proptest::{arbitrary::any, prelude::*};
use std::net::SocketAddr;
use super::{
read_zcash::canonical_socket_addr, CompactSizeMessage, DateTime32, MAX_PROTOCOL_MESSAGE_LEN,
};
impl Arbitrary for DateTime32 {
type Parameters = ();
@ -62,3 +66,18 @@ pub fn datetime_u32() -> impl Strategy<Value = chrono::DateTime<Utc>> {
pub fn canonical_socket_addr_strategy() -> impl Strategy<Value = SocketAddr> {
any::<SocketAddr>().prop_map(canonical_socket_addr)
}
impl Arbitrary for CompactSizeMessage {
type Parameters = ();
fn arbitrary_with(_args: Self::Parameters) -> Self::Strategy {
(0..=MAX_PROTOCOL_MESSAGE_LEN)
.prop_map(|size| {
size.try_into()
.expect("MAX_PROTOCOL_MESSAGE_LEN fits in CompactSizeMessage")
})
.boxed()
}
type Strategy = BoxedStrategy<Self>;
}

363
zebra-chain/src/serialization/compact_size.rs Normal file
View File

@ -0,0 +1,363 @@
//! Types and serialization for the Bitcoin `CompactSize` format.
//!
//! Zebra decodes `CompactSize` fields into two different types,
//! depending on the consensus rules that apply to that type:
//! - [`CompactSizeMessage`] for sizes that must be less than the network message limit, and
//! - [`CompactSize64`] for flags, arbitrary counts, and sizes that span multiple blocks.
use std::convert::{TryFrom, TryInto};
use byteorder::{LittleEndian, ReadBytesExt, WriteBytesExt};
use crate::serialization::{
SerializationError, ZcashDeserialize, ZcashDeserializeInto, ZcashSerialize,
MAX_PROTOCOL_MESSAGE_LEN,
};
#[cfg(any(test, feature = "proptest-impl"))]
use proptest_derive::Arbitrary;
/// A CompactSize-encoded field that is limited to [`MAX_PROTOCOL_MESSAGE_LEN`].
/// Used for sizes or counts of objects that are sent in network messages.
///
/// # Security
///
/// Deserialized sizes must be validated before being used to allocate memory.
///
/// Preallocating vectors using untrusted `CompactSize`s allows memory
/// denial of service attacks. Valid sizes must be less than
/// `MAX_PROTOCOL_MESSAGE_LEN / min_serialized_item_bytes` (or a lower limit
/// specified by the Zcash consensus rules or Bitcoin network protocol).
///
/// As a defence-in-depth for memory preallocation attacks,
/// Zebra rejects sizes greater than the protocol message length limit.
/// (These sizes should be impossible, because each array items takes at
/// least one byte.)
///
/// # Serialization Examples
///
/// ```
/// use zebra_chain::serialization::{CompactSizeMessage, ZcashSerialize, MAX_PROTOCOL_MESSAGE_LEN};
/// use std::convert::{TryFrom, TryInto};
///
/// let size = CompactSizeMessage::try_from(0x12).unwrap();
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\x12");
///
/// let size = CompactSizeMessage::try_from(0xfd).unwrap();
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\xfd\xfd\x00");
///
/// let size = CompactSizeMessage::try_from(0xaafd).unwrap();
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\xfd\xfd\xaa");
///
/// let max_size: usize = MAX_PROTOCOL_MESSAGE_LEN.try_into().unwrap();
/// let size = CompactSizeMessage::try_from(max_size).unwrap();
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\xfe\x00\x00\x20\x00");
/// ```
///
/// [`CompactSizeMessage`]s greater than the maximum network message length
/// can not be constructed:
/// ```
/// # use zebra_chain::serialization::{CompactSizeMessage, MAX_PROTOCOL_MESSAGE_LEN};
/// # use std::convert::{TryFrom, TryInto};
/// # let max_size: usize = MAX_PROTOCOL_MESSAGE_LEN.try_into().unwrap();
/// assert!(CompactSizeMessage::try_from(max_size + 1).is_err());
///
/// assert!(CompactSizeMessage::try_from(0xbbaafd).is_err());
///
/// assert!(CompactSizeMessage::try_from(0x22ccbbaafd).is_err());
/// ```
///
/// # Deserialization Examples
///
/// ```
/// use zebra_chain::serialization::{CompactSizeMessage, ZcashDeserializeInto, MAX_PROTOCOL_MESSAGE_LEN};
/// use std::{convert::{TryFrom, TryInto}, io::Cursor};
///
/// assert_eq!(
/// CompactSizeMessage::try_from(0x12).unwrap(),
/// Cursor::new(b"\x12").zcash_deserialize_into().unwrap(),
/// );
///
/// assert_eq!(
/// CompactSizeMessage::try_from(0xfd).unwrap(),
/// Cursor::new(b"\xfd\xfd\x00").zcash_deserialize_into().unwrap(),
/// );
///
/// assert_eq!(
/// CompactSizeMessage::try_from(0xaafd).unwrap(),
/// Cursor::new(b"\xfd\xfd\xaa").zcash_deserialize_into().unwrap(),
/// );
///
/// let max_size: usize = MAX_PROTOCOL_MESSAGE_LEN.try_into().unwrap();
/// assert_eq!(
/// CompactSizeMessage::try_from(max_size).unwrap(),
/// Cursor::new(b"\xfe\x00\x00\x20\x00").zcash_deserialize_into().unwrap(),
/// );
/// ```
///
/// [`CompactSizeMessage`]s greater than the maximum network message length are invalid.
/// They return a [`SerializationError::Parse`]:
/// ```
/// # use zebra_chain::serialization::{CompactSizeMessage, ZcashDeserialize, MAX_PROTOCOL_MESSAGE_LEN};
/// # use std::{convert::TryFrom, io::Cursor};
/// let max_size_plus_one = Cursor::new(b"\xfe\x01\x00\x20\x00");
/// assert!(CompactSizeMessage::zcash_deserialize(max_size_plus_one).is_err());
///
/// let bytes = Cursor::new(b"\xfe\xfd\xaa\xbb\x00");
/// assert!(CompactSizeMessage::zcash_deserialize(bytes).is_err());
///
/// let bytes = Cursor::new(b"\xff\xfd\xaa\xbb\xcc\x22\x00\x00\x00");
/// assert!(CompactSizeMessage::zcash_deserialize(bytes).is_err());
/// ```
#[derive(Copy, Clone, Debug, Eq, PartialEq, Ord, PartialOrd, Hash)]
pub struct CompactSizeMessage(
/// The number of items in a Zcash message.
///
/// This field is private to enforce the [`MAX_PROTOCOL_MESSAGE_LEN`] limit.
u32,
);
/// An arbitrary CompactSize-encoded field.
/// Used for flags, arbitrary counts, and sizes that span multiple blocks.
///
/// # Security
///
/// Deserialized sizes must be validated before being used to allocate memory.
///
/// Most allocation sizes should use [`CompactSizeMessage`],
/// because it is limited to [`MAX_PROTOCOL_MESSAGE_LEN`].
///
/// # Serialization Examples
///
/// ```
/// use zebra_chain::serialization::{CompactSize64, ZcashSerialize, MAX_PROTOCOL_MESSAGE_LEN};
/// use std::convert::TryFrom;
///
/// let size = CompactSize64::from(0x12);
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\x12");
///
/// let size = CompactSize64::from(0xfd);
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\xfd\xfd\x00");
///
/// let size = CompactSize64::from(0xaafd);
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\xfd\xfd\xaa");
///
/// let max_size = u64::try_from(MAX_PROTOCOL_MESSAGE_LEN).unwrap();
/// let size = CompactSize64::from(max_size);
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\xfe\x00\x00\x20\x00");
///
/// let size = CompactSize64::from(max_size + 1);
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\xfe\x01\x00\x20\x00");
///
/// let size = CompactSize64::from(0xbbaafd);
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\xfe\xfd\xaa\xbb\x00");
///
/// let size = CompactSize64::from(0x22ccbbaafd);
/// let buf = size.zcash_serialize_to_vec().unwrap();
/// assert_eq!(buf, b"\xff\xfd\xaa\xbb\xcc\x22\x00\x00\x00");
/// ```
///
/// # Deserialization Examples
///
/// ```
/// use zebra_chain::serialization::{CompactSize64, ZcashDeserializeInto, MAX_PROTOCOL_MESSAGE_LEN};
/// use std::{convert::TryFrom, io::Cursor};
///
/// assert_eq!(
/// CompactSize64::from(0x12),
/// Cursor::new(b"\x12").zcash_deserialize_into().unwrap(),
/// );
///
/// assert_eq!(
/// CompactSize64::from(0xfd),
/// Cursor::new(b"\xfd\xfd\x00").zcash_deserialize_into().unwrap(),
/// );
///
/// assert_eq!(
/// CompactSize64::from(0xaafd),
/// Cursor::new(b"\xfd\xfd\xaa").zcash_deserialize_into().unwrap(),
/// );
///
/// let max_size = u64::try_from(MAX_PROTOCOL_MESSAGE_LEN).unwrap();
/// assert_eq!(
/// CompactSize64::from(max_size),
/// Cursor::new(b"\xfe\x00\x00\x20\x00").zcash_deserialize_into().unwrap(),
/// );
///
/// assert_eq!(
/// CompactSize64::from(max_size + 1),
/// Cursor::new(b"\xfe\x01\x00\x20\x00").zcash_deserialize_into().unwrap(),
/// );
///
/// assert_eq!(
/// CompactSize64::from(0xbbaafd),
/// Cursor::new(b"\xfe\xfd\xaa\xbb\x00").zcash_deserialize_into().unwrap(),
/// );
///
/// assert_eq!(
/// CompactSize64::from(0x22ccbbaafd),
/// Cursor::new(b"\xff\xfd\xaa\xbb\xcc\x22\x00\x00\x00").zcash_deserialize_into().unwrap(),
/// );
///```
#[derive(Copy, Clone, Debug, Eq, PartialEq, Ord, PartialOrd, Hash)]
#[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))]
pub struct CompactSize64(
/// A numeric value.
///
/// This field is private for consistency with [`CompactSizeMessage`].
u64,
);
// `CompactSizeMessage` are item counts, so we expect them to be used with `usize`
// `CompactSize64` are arbitrary integers, so we expect it to be used with `u64`
//
// We don't implement conversions between CompactSizeMessage and CompactSize64,
// because we want to distinguish fields with different numeric constraints.
//
// We don't implement From<CompactSizeMessage> for u16 or u8,
// because we want all values to go through the same code paths.
// (And we don't want to accidentally truncate using `as`.)
// It would also make integer literal type inference fail.
//
// We don't implement `PartialEq` or `Ord` with integers,
// because it makes type inference fail.
impl TryFrom<usize> for CompactSizeMessage {
type Error = SerializationError;
#[inline]
fn try_from(size: usize) -> Result<Self, Self::Error> {
use SerializationError::Parse;
let size: u32 = size.try_into()?;
// # Security
// Defence-in-depth for memory DoS via preallocation.
if size
> MAX_PROTOCOL_MESSAGE_LEN
.try_into()
.expect("MAX_PROTOCOL_MESSAGE_LEN fits in u32")
{
// This could be invalid data from peers, so we return a parse error.
Err(Parse("CompactSize larger than protocol message limit"))?;
}
Ok(CompactSizeMessage(size))
}
}
impl From<CompactSizeMessage> for usize {
#[inline]
fn from(size: CompactSizeMessage) -> Self {
size.0.try_into().expect("u32 fits in usize")
}
}
impl From<CompactSize64> for u64 {
#[inline]
fn from(size: CompactSize64) -> Self {
size.0
}
}
impl From<u64> for CompactSize64 {
#[inline]
fn from(size: u64) -> Self {
CompactSize64(size)
}
}
impl ZcashSerialize for CompactSizeMessage {
/// Serialize a CompactSizeMessage into the Zcash consensus-critical format.
///
/// # Panics
///
/// If the value exceeds `MAX_PROTOCOL_MESSAGE_LEN`.
#[inline]
fn zcash_serialize<W: std::io::Write>(&self, writer: W) -> Result<(), std::io::Error> {
// # Security
// Defence-in-depth for memory DoS via preallocation.
//
// This is validated data inside Zebra, so we panic.
assert!(
self.0
<= MAX_PROTOCOL_MESSAGE_LEN
.try_into()
.expect("usize fits in u64"),
"CompactSize larger than protocol message limit"
);
// Use the same serialization format as CompactSize64.
let size: u64 = self.0.into();
CompactSize64(size).zcash_serialize(writer)
}
}
impl ZcashDeserialize for CompactSizeMessage {
#[inline]
fn zcash_deserialize<R: std::io::Read>(reader: R) -> Result<Self, SerializationError> {
// Use the same deserialization format as CompactSize64.
let size: CompactSize64 = reader.zcash_deserialize_into()?;
let size: usize = size.0.try_into()?;
size.try_into()
}
}
impl ZcashSerialize for CompactSize64 {
#[inline]
fn zcash_serialize<W: std::io::Write>(&self, mut writer: W) -> Result<(), std::io::Error> {
let n = self.0;
match n {
0x00..=0xfc => writer.write_u8(n as u8),
0x00fd..=0xffff => {
writer.write_u8(0xfd)?;
writer.write_u16::<LittleEndian>(n as u16)
}
0x0001_0000..=0xffff_ffff => {
writer.write_u8(0xfe)?;
writer.write_u32::<LittleEndian>(n as u32)
}
_ => {
writer.write_u8(0xff)?;
writer.write_u64::<LittleEndian>(n)
}
}
}
}
impl ZcashDeserialize for CompactSize64 {
#[inline]
fn zcash_deserialize<R: std::io::Read>(mut reader: R) -> Result<Self, SerializationError> {
use SerializationError::Parse;
let flag_byte = reader.read_u8()?;
let size = match flag_byte {
n @ 0x00..=0xfc => Ok(n as u64),
0xfd => match reader.read_u16::<LittleEndian>()? {
n @ 0x00fd..=u16::MAX => Ok(n as u64),
_ => Err(Parse("non-canonical CompactSize")),
},
0xfe => match reader.read_u32::<LittleEndian>()? {
n @ 0x0001_0000..=u32::MAX => Ok(n as u64),
_ => Err(Parse("non-canonical CompactSize")),
},
0xff => match reader.read_u64::<LittleEndian>()? {
n @ 0x0000_0001_0000_0000..=u64::MAX => Ok(n),
_ => Err(Parse("non-canonical CompactSize")),
},
}?;
Ok(CompactSize64(size))
}
}

2
zebra-chain/src/serialization/error.rs
View File

@ -26,7 +26,7 @@ pub enum SerializationError {
TryFromSliceError(#[from] TryFromSliceError),
/// The length of a vec is too large to convert to a usize (and thus, too large to allocate on this platform)
#[error("compactsize too large: {0}")]
#[error("CompactSize too large: {0}")]
TryFromIntError(#[from] TryFromIntError),
/// An error caused when validating a zatoshi `Amount`

10
zebra-chain/src/serialization/read_zcash.rs
View File

@ -1,6 +1,6 @@
use std::io;
use std::{
convert::TryInto,
io,
net::{IpAddr, Ipv6Addr, SocketAddr},
};
@ -67,15 +67,15 @@ pub trait ReadZcashExt: io::Read {
n @ 0x00..=0xfc => Ok(n as u64),
0xfd => match self.read_u16::<LittleEndian>()? {
n @ 0x0000_00fd..=0x0000_ffff => Ok(n as u64),
_ => Err(Parse("non-canonical compactsize")),
_ => Err(Parse("non-canonical CompactSize")),
},
0xfe => match self.read_u32::<LittleEndian>()? {
n @ 0x0001_0000..=0xffff_ffff => Ok(n as u64),
_ => Err(Parse("non-canonical compactsize")),
_ => Err(Parse("non-canonical CompactSize")),
},
0xff => match self.read_u64::<LittleEndian>()? {
n @ 0x1_0000_0000..=0xffff_ffff_ffff_ffff => Ok(n),
_ => Err(Parse("non-canonical compactsize")),
_ => Err(Parse("non-canonical CompactSize")),
},
}?;
@ -86,7 +86,7 @@ pub trait ReadZcashExt: io::Read {
.try_into()
.expect("usize fits in u64")
{
Err(Parse("compactsize larger than protocol message limit"))?;
Err(Parse("CompactSize larger than protocol message limit"))?;
}
Ok(size)

4
zebra-chain/src/serialization/tests/prop.rs
View File

@ -14,7 +14,7 @@ proptest! {
fn compactsize_write_then_read_round_trip(s in 0u64..0x2_0000u64) {
zebra_test::init();
// Maximum encoding size of a compactsize is 9 bytes.
// Maximum encoding size of a CompactSize is 9 bytes.
let mut buf = [0u8; 8+1];
Cursor::new(&mut buf[..]).write_compactsize(s).unwrap();
let expect_s = Cursor::new(&buf[..]).read_compactsize().unwrap();
@ -27,7 +27,7 @@ proptest! {
// Only do the test if the bytes were valid.
if let Ok(s) = Cursor::new(&bytes[..]).read_compactsize() {
// The compactsize encoding is variable-length, so we may not even
// The CompactSize encoding is variable-length, so we may not even
// read all of the input bytes, and therefore we can't expect that
// the encoding will reproduce bytes that were never read. Instead,
// copy the input bytes, and overwrite them with the encoding of s,

17
zebra-chain/src/serialization/write_zcash.rs
View File

@ -1,12 +1,13 @@
use std::io;
use std::{
convert::TryInto,
io,
net::{IpAddr, SocketAddr},
};
use super::MAX_PROTOCOL_MESSAGE_LEN;
use byteorder::{BigEndian, LittleEndian, WriteBytesExt};
use super::MAX_PROTOCOL_MESSAGE_LEN;
/// Extends [`Write`] with methods for writing Zcash/Bitcoin types.
///
/// [`Write`]: https://doc.rust-lang.org/std/io/trait.Write.html
@ -51,12 +52,12 @@ pub trait WriteZcashExt: io::Write {
// # Security
// Defence-in-depth for memory DoS via preallocation.
//
if n > MAX_PROTOCOL_MESSAGE_LEN
.try_into()
.expect("usize fits in u64")
{
panic!("compactsize larger than protocol message limit");
}
assert!(
n <= MAX_PROTOCOL_MESSAGE_LEN
.try_into()
.expect("usize fits in u64"),
"CompactSize larger than protocol message limit"
);
match n {
0x0000_0000..=0x0000_00fc => self.write_u8(n as u8),

22
zebra-chain/src/serialization/zcash_deserialize.rs
View File

@ -3,7 +3,7 @@ use std::{
io,
};
use super::{AtLeastOne, ReadZcashExt, SerializationError, MAX_PROTOCOL_MESSAGE_LEN};
use super::{AtLeastOne, CompactSizeMessage, SerializationError, MAX_PROTOCOL_MESSAGE_LEN};
/// Consensus-critical serialization for Zcash.
///
@ -20,20 +20,20 @@ pub trait ZcashDeserialize: Sized {
fn zcash_deserialize<R: io::Read>(reader: R) -> Result<Self, SerializationError>;
}
/// Deserialize a `Vec`, where the number of items is set by a compactsize
/// Deserialize a `Vec`, where the number of items is set by a CompactSize
/// prefix in the data. This is the most common format in Zcash.
///
/// See `zcash_deserialize_external_count` for more details, and usage
/// information.
impl<T: ZcashDeserialize + TrustedPreallocate> ZcashDeserialize for Vec<T> {
fn zcash_deserialize<R: io::Read>(mut reader: R) -> Result<Self, SerializationError> {
let len = reader.read_compactsize()?.try_into()?;
zcash_deserialize_external_count(len, reader)
let len: CompactSizeMessage = (&mut reader).zcash_deserialize_into()?;
zcash_deserialize_external_count(len.into(), reader)
}
}
/// Deserialize an `AtLeastOne` vector, where the number of items is set by a
/// compactsize prefix in the data. This is the most common format in Zcash.
/// CompactSize prefix in the data. This is the most common format in Zcash.
impl<T: ZcashDeserialize + TrustedPreallocate> ZcashDeserialize for AtLeastOne<T> {
fn zcash_deserialize<R: io::Read>(mut reader: R) -> Result<Self, SerializationError> {
let v: Vec<T> = (&mut reader).zcash_deserialize_into()?;
@ -48,16 +48,16 @@ impl<T: ZcashDeserialize + TrustedPreallocate> ZcashDeserialize for AtLeastOne<T
/// This allows the optimization without relying on specialization.
impl ZcashDeserialize for Vec<u8> {
fn zcash_deserialize<R: io::Read>(mut reader: R) -> Result<Self, SerializationError> {
let len = reader.read_compactsize()?.try_into()?;
zcash_deserialize_bytes_external_count(len, reader)
let len: CompactSizeMessage = (&mut reader).zcash_deserialize_into()?;
zcash_deserialize_bytes_external_count(len.into(), reader)
}
}
/// Deserialize a `Vec` containing `external_count` items.
///
/// In Zcash, most arrays are stored as a compactsize, followed by that number
/// In Zcash, most arrays are stored as a CompactSize, followed by that number
/// of items of type `T`. But in `Transaction::V5`, some types are serialized as
/// multiple arrays in different locations, with a single compactsize before the
/// multiple arrays in different locations, with a single CompactSize before the
/// first array.
///
/// ## Usage
@ -65,7 +65,7 @@ impl ZcashDeserialize for Vec<u8> {
/// Use `zcash_deserialize_external_count` when the array count is determined by
/// other data, or a consensus rule.
///
/// Use `Vec::zcash_deserialize` for data that contains compactsize count,
/// Use `Vec::zcash_deserialize` for data that contains CompactSize count,
/// followed by the data array.
///
/// For example, when a single count applies to multiple arrays:
@ -159,7 +159,7 @@ pub trait TrustedPreallocate {
/// The length of the longest valid `Vec<u8>` that can be received over the network
///
/// It takes 5 bytes to encode a compactsize representing any number netween 2^16 and (2^32 - 1)
/// It takes 5 bytes to encode a CompactSize representing any number netween 2^16 and (2^32 - 1)
/// MAX_PROTOCOL_MESSAGE_LEN is ~2^21, so the largest Vec<u8> that can be received from an honest peer is
/// (MAX_PROTOCOL_MESSAGE_LEN - 5);
pub(crate) const MAX_U8_ALLOCATION: usize = MAX_PROTOCOL_MESSAGE_LEN - 5;

61
zebra-chain/src/serialization/zcash_serialize.rs
View File

@ -1,6 +1,11 @@
use std::io;
use std::{convert::TryInto, io};
use super::{AtLeastOne, WriteZcashExt};
use super::{AtLeastOne, CompactSizeMessage};
/// The maximum length of a Zcash message, in bytes.
///
/// This value is used to calculate safe preallocation limits for some types
pub const MAX_PROTOCOL_MESSAGE_LEN: usize = 2 * 1024 * 1024;
/// Consensus-critical serialization for Zcash.
///
@ -52,18 +57,31 @@ impl std::io::Write for FakeWriter {
}
}
/// Serialize a `Vec` as a compactsize number of items, then the items. This is
/// Serialize a `Vec` as a CompactSize number of items, then the items. This is
/// the most common format in Zcash.
///
/// See `zcash_serialize_external_count` for more details, and usage information.
impl<T: ZcashSerialize> ZcashSerialize for Vec<T> {
fn zcash_serialize<W: io::Write>(&self, mut writer: W) -> Result<(), io::Error> {
writer.write_compactsize(self.len() as u64)?;
let len: CompactSizeMessage = self
.len()
.try_into()
.expect("len fits in MAX_PROTOCOL_MESSAGE_LEN");
len.zcash_serialize(&mut writer)?;
zcash_serialize_external_count(self, writer)
}
}
/// Serialize a byte vector as a compactsize number of items, then the items.
/// Serialize an `AtLeastOne` vector as a CompactSize number of items, then the
/// items. This is the most common format in Zcash.
impl<T: ZcashSerialize> ZcashSerialize for AtLeastOne<T> {
fn zcash_serialize<W: io::Write>(&self, mut writer: W) -> Result<(), io::Error> {
self.as_vec().zcash_serialize(&mut writer)
}
}
/// Serialize a byte vector as a CompactSize number of items, then the items.
///
/// # Correctness
///
@ -75,24 +93,28 @@ impl<T: ZcashSerialize> ZcashSerialize for Vec<T> {
// we specifically want to serialize `Vec`s here, rather than generic slices
#[allow(clippy::ptr_arg)]
pub fn zcash_serialize_bytes<W: io::Write>(vec: &Vec<u8>, mut writer: W) -> Result<(), io::Error> {
writer.write_compactsize(vec.len() as u64)?;
let len: CompactSizeMessage = vec
.len()
.try_into()
.expect("len fits in MAX_PROTOCOL_MESSAGE_LEN");
len.zcash_serialize(&mut writer)?;
zcash_serialize_bytes_external_count(vec, writer)
}
/// Serialize an `AtLeastOne` vector as a compactsize number of items, then the
/// items. This is the most common format in Zcash.
impl<T: ZcashSerialize> ZcashSerialize for AtLeastOne<T> {
fn zcash_serialize<W: io::Write>(&self, mut writer: W) -> Result<(), io::Error> {
self.as_vec().zcash_serialize(&mut writer)
}
/// Serialize an empty list of items, by writing a zero CompactSize length.
/// (And no items.)
pub fn zcash_serialize_empty_list<W: io::Write>(writer: W) -> Result<(), io::Error> {
let len: CompactSizeMessage = 0.try_into().expect("zero fits in MAX_PROTOCOL_MESSAGE_LEN");
len.zcash_serialize(writer)
}
/// Serialize a typed `Vec` **without** writing the number of items as a
/// compactsize.
/// CompactSize.
///
/// In Zcash, most arrays are stored as a compactsize, followed by that number
/// In Zcash, most arrays are stored as a CompactSize, followed by that number
/// of items of type `T`. But in `Transaction::V5`, some types are serialized as
/// multiple arrays in different locations, with a single compactsize before the
/// multiple arrays in different locations, with a single CompactSize before the
/// first array.
///
/// ## Usage
@ -100,7 +122,7 @@ impl<T: ZcashSerialize> ZcashSerialize for AtLeastOne<T> {
/// Use `zcash_serialize_external_count` when the array count is determined by
/// other data, or a consensus rule.
///
/// Use `Vec::zcash_serialize` for data that contains compactsize count,
/// Use `Vec::zcash_serialize` for data that contains CompactSize count,
/// followed by the data array.
///
/// For example, when a single count applies to multiple arrays:
@ -125,7 +147,7 @@ pub fn zcash_serialize_external_count<W: io::Write, T: ZcashSerialize>(
}
/// Serialize a raw byte `Vec` **without** writing the number of items as a
/// compactsize.
/// CompactSize.
///
/// This is a convenience alias for `writer.write_all(&vec)`.
//
@ -152,8 +174,3 @@ impl ZcashSerialize for String {
self.as_str().zcash_serialize(&mut writer)
}
}
/// The maximum length of a Zcash message, in bytes.
///
/// This value is used to calculate safe preallocation limits for some types
pub const MAX_PROTOCOL_MESSAGE_LEN: usize = 2 * 1024 * 1024;