zips/zip-0244.rst

::

  ZIP: 244
  Title: Transaction Non-Malleability
  Owners: Kris Nuttycombe <kris@electriccoin.co>
          Daira Hopwood <daira@electriccoin.co>
  Status: Draft
  Category: Consensus
  Created: 2021-01-06
  License: MIT
  Discussions-To: <https://github.com/zcash/zips/issues/411>

===========
Terminology
===========

The key words "MUST" and "MUST NOT" in this document are to be interpreted as described in RFC 2119. [#RFC2119]_

The terms "consensus branch", "epoch", and "network upgrade" in this document are to be interpreted as
described in ZIP 200. [#zip-0200]_

========
Abstract
========

This proposal defines a new transaction digest algorithm for the <TBD> network upgrade
onward, in order to introduce non-malleable transaction identifiers that commit to
all transaction data except for attestations to transaction validity.

This proposal also defines a new transaction digest algorithm for signature validation,
which shares all available structure produced during the construction of transaction 
identifiers, in order to minimize redundant data hashing in validation.

This proposal also defines new semantics for the ``hashLightClientRoot`` field of the
block header, to enable additional commitments to be represented in this hash and to
provide a mechanism for future extensibility of the set of commitments represented.

==========
Motivation
==========

In all cases, but particularly in order to support the use of transactions in
higher-level protocols, any modification of the transaction that has not been
explicitly permitted (such as via anyone-can-spend inputs) should invalidate
attestations to spend authority or to the included outputs. Following the activation
of this proposed change, transaction identifiers will be stable irrespective of 
any possible malleation of "witness data" such as proofs and transaction
signatures.

============
Requirements
============

- Continue to support existing functionality of the protocol (multisig, 
  signing modes for transparent inputs).

- Allow the use of transaction ids, and pairs of the form (transaction id,
  output index) as stable identifiers. 

- A sender must be able to recognize their own transaction, even given allowed
  forms of malleability such as recomputation of transaction signatures.

- In the case of transparent inputs, it should be possible to create a
  transaction (B) that spends the outputs from a previous transaction (A) even
  before (A) has been mined. This should also be possible in the case that the
  creator of (B) does not wait for confirmations of (A). That is, (B) should remain
  valid so long as any variant of (A) is eventually mined.

- It should not be possible for an attacker to malleate a transaction in a
  fashion that would result in the transaction being interpreted as a
  double-spend.

- It should be possible in the future to upgrade the protocol in such a fashion
  that only non-malleable transactions are accepted.

- It should be possible to use the transaction id unmodified as the value that
  is used to produce a signature hash in the case that the transaction contains
  no transparent inputs, or in the case that only the ``SIGHASH_ALL`` flag is
  used. 


================
Non-requirements
================

In order to support backwards-compatibility with parts of the ecosystem that
have not yet upgraded to the non-malleable transaction format, it is not an
initial requirement that all transactions be non-malleable.

=============
Specification
=============

-------
Digests
-------

All digests are personalized BLAKE2b-256 hashes. In cases where no elements are
available for hashing (for example, if there are no transparent inputs) the resulting hash
will be over just the personalization string, providing domain separation even for
empty data fields.

TxId Digest
===========

A new transaction digest algorithm is defined that constructs the identifier for
a transaction from a tree of hashes. Each branch of the tree of hashes will
correspond to a specific subset of transaction data. The overall structure of 
the hash is as follows; each name referenced here will be described in detail
below:

    txid_digest
    ├── header_digest
    ├── transparent_digest
    │   ├── prevouts_digest
    │   ├── sequence_digest
    │   └── outputs_digest
    ├── sprout_digest
    └── sapling_digest
        ├── sapling_spends_digest
        │   ├── sapling_spends_compact_digest
        │   └── sapling_spends_noncompact_digest
        ├── sapling_outputs_digest
        │   ├── sapling_outputs_compact_digest
        │   ├── sapling_outputs_memos_digest
        │   └── sapling_outputs_noncompact_digest
        └── valueBalance

Each node written as ``snake_case`` in this tree is a BLAKE2b-256 hash of its 
children, initialized with a personalization string specific to that branch 
of the tree. Nodes that are not themselves digests are written in ``camelCase``. 
In the specification below, nodes of the tree are presented in depth-first order.

``txid_digest``
--------------
A BLAKE2b-256 hash of the following values ::

   * T.1: ``header_digest`` (32-byte hash output)
   * T.2: ``transparent_digest`` (32-byte hash output)
   * T.3: ``sprout_digest (32-byte hash output)
   * T.4: ``sapling_digest (32-byte hash output)

The personalization field of this hash is set to::

  "ZcashTxHash_" || CONSENSUS_BRANCH_ID

As in ZIP 143 [#zip-0143]_, CONSENSUS_BRANCH_ID is the 4-byte little-endian encoding of
the consensus branch ID for the epoch of the block containing the transaction. Domain
separation of the transaction id hash across parallel consensus branches provides replay
protection: transactions targeted for one consensus branch will not have the same
transaction identifier on other consensus branches.

T.1: ``header_digest``
`````````````````````````
A BLAKE2b-256 hash of the following values ::

* T.1a. ``version`` (4-byte little-endian version identifier including overwinter flag)
* T.1b. ``version_group_id`` (4-byte little-endian version group identifier)
* T.1c. ``consensus_branch_id`` (4-byte little-endian consensus branch id)
* T.1d. ``lock_time``  (4-byte little-endian nLockTime value)
* T.1e. ``expiry_height``  (4-byte little-endian block height)

The personalization field of this hash is set to::

  "ZTxIdHeadersHash"

T.2: ``transparent_digest``
`````````````````````````
A BLAKE2b-256 hash of the following values ::

* T.2a. ``prevouts_digest`` (32-byte hash)
* T.2b. ``sequence_digest`` (32-byte hash)
* T.2c. ``outputs_digest``  (32-byte hash)

The personalization field of this hash is set to::

  "ZTxIdTranspaHash"

T.2a: ``prevouts_digest``
'''''''''''''''''''''''
A BLAKE2b-256 hash of the field encoding of all ``outpoint``
field values of transparent inputs to the transaction.

The personalization field of this hash is set to::

  "ZTxIdPrevoutHash"

T.2b: ``sequence_digest``
'''''''''''''''''''''''
A BLAKE2b-256 hash of the 32-bit little-endian representation of all ``nSequence``
field values of transparent inputs to the transaction.

The personalization field of this hash is set to::

  "ZTxIdSequencHash"

T.2c: ``outputs_digest``
''''''''''''''''''''''
A BLAKE2b-256 hash of the field encodings of all ``prevout`` field values of 
transparent inputs belonging to the transaction.

The personalization field of this hash is set to::

  "ZTxIdOutputsHash"

T.3: ``sprout_digest``
`````````````````````````
A BLAKE2b-256 hash of the non-authorizing components of Sprout ``JSDescription`` values
belonging to the transaction. For each ``JSDescription``, the following elements are
appended to the hash ::

* T.3a. ``vpub_old`` (8-byte signed little-endian)
* T.3b. ``vpub_new`` (8-byte signed little-endian)
* T.3c. ``anchor``   (32 bytes)
* T.3d. ``nullifiers``  (2 x 32 bytes)
* T.3e. ``commitments`` (2 x 32 bytes)
* T.3f. ``ephemeral_key`` (32 bytes)
* T.3g. ``random_seed`` (32 bytes)
* T.3h. ``macs`` (2 x 32 bytes)
* T.3i. ``ciphertexts`` (2 x 601 bytes)

The personalization field of this hash is set to::

  "ZTxIdJSplitsHash"

T.4: ``sapling_digest``
`````````````````````
The digest of Sapling components is composed of two subtrees which are organized to 
permit easy interoperability with the ``CompactBlock`` representation of Sapling data
specified by the ZIP 307 Light Client Protocol [#zip-0307]_.

This digest is a BLAKE2b-256 hash of the following values ::

   * T.4a. ``sapling_spends_digest`` (32-byte hash)
   * T.4b. ``sapling_outputs_digest`` (32-byte hash)
   * T.4c. ``valueBalance`` (64-bit signed little-endian)

The personalization field of this hash is set to::

  "ZTxIdSaplingHash"

T.4a: ``sapling_spends_digest``
''''''''''''''''''''''''''''''
This digest is a BLAKE2b-256 hash of the following values ::

   * T.4a.i.   ``sapling_spends_compact_digest`` (32-byte hash)
   * T.4a.ii.  ``sapling_spends_noncompact_digest`` (32-byte hash)

The personalization field of this hash is set to::

  "ZTxIdSSpendsHash"

T.4a.i: ``sapling_spends_compact_digest``
.......................................
A BLAKE2b-256 hash of the field encoding of all nullifier field
values of Sapling shielded spends belonging to the transaction.

The personalization field of this hash is set to::

  "ZTxIdSSpendCHash"

T.4a.ii: ``sapling_spends_noncompact_digest``
...........................................
A BLAKE2b-256 hash of the non-nullifier information for all Sapling shielded spends
belonging to the transaction, excluding zkproof data. For each spend, the following
elements are included in the hash:: 

   * T.4a.ii.1 ``cv``      (field encoding bytes)
   * T.4a.ii.2 ``anchor``  (field encoding bytes)
   * T.4a.ii.3 ``rk``      (field encoding bytes)

The personalization field of this hash is set to::

  "ZTxIdSSpendNHash"

T.4b: ``sapling_outputs_digest``
'''''''''''''''''''''''''''''''
This digest is a BLAKE2b-256 hash of the following values ::

   * T.4a.i.   ``sapling_outputs_compact_digest`` (32-byte hash)
   * T.4b.ii.  ``sapling_outputs_memos_digest`` (32-byte hash)
   * T.4b.iii. ``sapling_outputs_noncompact_digest`` (32-byte hash)

The personalization field of this hash is set to::

  "ZTxIdSOutputHash"

T.4b.i: ``sapling_outputs_compact_digest``
.........................................
A BLAKE2b-256 hash of the subset of Sapling output information included in the 
ZIP-307 [#zip-0307]_ ``CompactBlock`` format for all Sapling shielded outputs
belonging to the transaction. For each output, the following elements are included
in the hash:: 

   * T.4b.i.1 ``cmu``                  (field encoding bytes)
   * T.4b.i.2 ``ephemeral_key``        (field encoding bytes)
   * T.4b.i.3 ``enc_ciphertext[..52]`` (First 52 bytes of field encoding)

The personalization field of this hash is set to::

  "ZTxIdSOutC__Hash"

T.4a.ii: ``sapling_outputs_memos_digest``
........................................
A BLAKE2b-256 hash of the subset of Sapling shielded memo field data for all Sapling 
shielded outputs belonging to the transaction. For each output, the following elements 
are included in the hash:: 

   * T.4b.ii.1 ``enc_ciphertext[52..564] (contents of the encrypted memo field)

The personalization field of this hash is set to::

  "ZTxIdSOutM__Hash"

T.4a.iii: ``sapling_outputs_noncompact_digest``
..............................................
A BLAKE2b-256 hash of the remaining subset of Sapling output information **not** included
in the ZIP 307 [#zip-0307]_ ``CompactBlock`` format, excluding zkproof data, for all
Sapling shielded outputs belonging to the transaction. For each output, the following
elements are included in the hash:: 

   * T.4b.iii.1 ``cv``                    (field encoding bytes)
   * T.4b.iii.2 ``enc_ciphertext[564..]`` (post-memo suffix of field encoding)
   * T.4b.iii.3 ``out_ciphertext``        (field encoding bytes)

The personalization field of this hash is set to::

  "ZTxIdSOutN__Hash" (2 underscore characters)

Signature Digest
================

A new per-input transaction digest algorithm that constructs a hash that may be signed
by a transaction creator to commit to the effects of the transaction. In the
case that the transaction consumes no transparent inputs, it should be possible
to just sign the transaction identifier produced by the ``TxId Digest`` algorithm.
In the case that transparent inputs are present, this algorithm follows closely
the ZIP 143 [#zip-0143]_ algorithm.

The overall structure of the hash is as follows; each name referenced here will be
described in detail below:

    signature_digest
    ├── header_digest
    ├── transparent_digest
    ├── sprout_digest
    └── sapling_digest

``signature_digest``
--------------------
A BLAKE2b-256 hash of the following values ::

   * S.1: ``header_digest`` (32-byte hash output)
   * S.2: ``transparent_digest`` (32-byte hash output)
   * S.3: ``sprout_digest (32-byte hash output)
   * S.4: ``sapling_digest (32-byte hash output)

The personalization field of this hash is set to::

  "ZcashTxHash_" || CONSENSUS_BRANCH_ID

This value must have the same personalization as the top hash of the transaction
identifier digest tree, in order to make it possible to sign the transaction id
in the case that there are no transparent inputs.

S.1: ``header_digest``
`````````````````````````
Identical to that specified for the transaction identifier.

S.2: ``transparent_digest``
`````````````````````````
If we are producing a hash for the signature over a transparent input,
the value of the digest produced here depends upon the value of a ``hash_type``
flag as in ZIP 143 [#zip-0143]_ and BIP 143 [#bip-0143]_.

The construction of each component below depends upon the values of the 
hash_type flag bits. Each component will be described separately

This digest is a BLAKE2b-256 hash of the following values ::

   * S.2a. ``prevouts_digest`` (32-byte hash)
   * S.2b. ``sequence_digest`` (32-byte hash)
   * S.2c. ``outputs_digest`` (32-byte hash)
   * S.2d. ``txin_sig_digest`` (32-byte hash)

The personalization field of this hash is set to::

   "ZTxIdTranspaHash"

S.2a: ``prevouts_digest``
'''''''''''''''''''''''
This is a BLAKE2b-256 hash initialized with the personalization field value 
"ZTxIdPrevoutHash". 

If the ``SIGHASH_ANYONECANPAY`` flag is set::

   * the hash is immediately finalized, without being updated with any
     additional data

otherwise::

   * identical to the value of ``prevouts_digest`` as specified for the
     transaction identifier in section T.2a.

S.2b: ``sequence_digest``
'''''''''''''''''''''''
This is a BLAKE2b-256 hash initialized with the personalization field value 
"ZTxIdSequencHash". 

If any of the ``SIGHASH_ANYONECANPAY``, ``SIGHASH_SINGLE``, or ``SIGHASH_NONE`` flags are
set::

   * the hash is immediately finalized, without being updated with any
     additional data

otherwise::

   * identical to the value of ``sequence_digest`` as specified for the
     transaction identifier in section T.2b.

S.2c: ``outputs_digest``
'''''''''''''''''''''''
This is a BLAKE2b-256 hash initialized with the personalization field value 
"ZTxIdOutputsHash". 

If the ``SIGHASH_SINGLE`` flag is set and the signature hash is being computed for a
transparent input::

   * the hash is updated with the field encoding of the ``prevout`` field value 
     of the input, then finalized

If the ``SIGHASH_SINGLE`` flag is set and the signature is being computed for
a shielded input, or if the ``SIGHASH_NONE`` flag is set::

   * the hash is immediately finalized, without being updated with any
     additional data

otherwise::

   * identical to the value of ``outputs_digest`` as specified for the
     transaction identifier in section T.2c.

S.2d: ``txin_sig_digest``
'''''''''''''''''''''''''
This is a BLAKE2b-256 hash initialized with the personalization field value
"Zcash___TxInHash" (3 underscores).

If the signature hash is being computed for a transparent input, the hash
is updated with the following properties of that input::

   * S.2d.i.   ``prevout``     (field encoding)
   * S.2d.ii.  ``script_code`` (field encoding)
   * S.2d.iii. ``value``       (8-byte signed little-endian)
   * S.2d.iv.  ``nSequence``   (4-byte unsigned little-endian)

otherwise::
   
   * the hash is immediately finalized, without being updated with any
     additional data

S.3: ``sprout_digest``
`````````````````````````
Identical to that specified for the transaction identifier.

S.4: ``sapling_digest``
`````````````````````````
Identical to that specified for the transaction identifier.

Authorizing Data Commitment
===========================

A new transaction digest algorithm is defined that constructs a digest which commits
to the authorizing data of a transaction from a tree of BLAKE2b-256 hashes. 
The overall structure of the hash is as follows:

    auth_digest
    ├── transparent_scripts_digest
    ├── sprout_sigs_digest
    └── sapling_sigs_digest

Each node written as ``snake_case`` in this tree is a BLAKE2b-256 hash of authorizing
data of the transaction.

The pair (Transaction Identifier, Auth Commitment) constitutes a commitment to all the
data of a serialized transaction that may be included in a block. 

``auth_digest``
--------------
A BLAKE2b-256 hash of the following values ::

   * ``transparent_scripts_digest`` (32-byte hash output)
   * ``sprout_auth_digest (32-byte hash output)
   * ``sapling_auth_digest (32-byte hash output)

The personalization field of this hash is set to::

  "ZTxAuth_____Hash" (5 underscore characters)

1: ``transparent_scripts_digest``
`````````````````````````````````
A BLAKE2b-256 hash of the field encoding of the Bitcoin script associated 
with each transparent input belonging to the transaction.

The personalization field of this hash is set to::

  "ZTxAuthTransHash"

2: ``sprout_auth_digest``
```````````````````````````
A BLAKE2b-256 hash of the field encoding of the zkproof values of each
``JSDescription`` belonging to the transaction.

   * 2a. ``zkproofs``         (field encoding bytes)
   * 2b. ``joinsplit_pubkey``    
   * 2b. ``joinsplit_sig``     

The personalization field of this hash is set to::

  "ZTxAuthSprouHash"

3: ``sapling_auth_digest``
```````````````````````````
A BLAKE2b-256 hash of the field encoding of the Sapling zkproof values
of each Sapling spend description and output description belonging to 
the transaction, followed by the field encoding of the binding signature ::

   * 3a. ``spend_zkproofs``  (field encoding bytes)
   * 3b. ``output_zkproofs`` (field encoding bytes)
   * 3c. ``binding_sig``     

The personalization field of this hash is set to::

  "ZTxAuthSapliHash"

--------------------
Block Header Changes
--------------------

The nonmalleable transaction identifier specified by this ZIP will be used
in the place of the current malleable transaction identifier within the
Merkel tree committed to by the ``hashMerkleRoot`` value. However, this 
change now means that ``hashMerkleRoot`` is not sufficient to fully commit
to the transaction data, including witnesses, that appear within the block.

As a consequence, we now need to add a new commitment to the block header.
This commitment will be the root of a Merkle tree that has parallel structure
to the tree committed to by ``hashMerkleRoot`` (a path through this merkle
tree to a transaction identifies the same transaction as that path reaches
in the tree rooted at ``hashMerkleRoot``) but where the leaves are hashes
produced according to the :ref:`Authorizing Data Commitment<Authorizing Data Commitment>` 
specification.

This new commitment is named ``hashAuthDataRoot`` and is the root of a left-dense
binary merkle tree of transaction authorizing data commitments. Empty internal nodes 
and leaves in the merkle tree (nodes without children) have the "null" hash value 
``[0u8; 32]``. Hashes in this tree are BLAKE2b-256 hashes personalized by the string
``"ZcashAuthDatHash"``.

Changing the block header format to allow space for an additional 
commitment is somewhat invasive. Instead, the name and meaning of the 
``hashLightClientRoot`` field is changed.

``hashLightClientRoot`` is renamed to ``hashBlockCommitments``. The value
of this hash is the BLAKE2b-256 hash personalized by the string ``"ZcashBlockCommit"``
of the following elements::

   * ``hashLightClientRoot`` as described in ZIP 221 [#zip-0221]_
   * ``hashAuthDataRoot`` 


========================
Reference implementation
========================

- https://github.com/zcash/librustzcash/pull/319/files

==========
References
==========

.. [#RFC2119] `RFC 2119: Key words for use in RFCs to Indicate Requirement Levels <https://www.rfc-editor.org/rfc/rfc2119.html>`_
.. [#zip-0200] `ZIP 200: Network Upgrade Activation Mechanism <https://zips.z.cash/zip-0200>`_
.. [#zip-0221] `ZIP 221: FlyClient - Consensus Layer Changes <https://zips.z.cash/zip-0221>`_
.. [#zip-0076] `ZIP 76: Transaction Signature Validation before Overwinter <https://zips.z.cash/zip-0076>`_
.. [#zip-0143] `ZIP 143: Transaction Signature Validation for Overwinter <https://zips.z.cash/zip-0143>`_
.. [#bip-0143] `Transaction Signature Verification for Version 0 Witness Program <https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki>`_
.. [#zip-0307] `ZIP 307: Light Client Protocol for Payment Detection <https://zips.z.cash/zip-0307>`_