Title: Transaction Signature Verification for Overwinter
Owners: Jack Grigg <str4d@electriccoin.co>
Daira Hopwood <daira@electriccoin.co>
Credits: Johnson Lau <jl2012@xbt.hk>
Pieter Wuille <pieter.wuille@gmail.com>
Bitcoin-ABC
Status: Final
Category: Consensus
Created: 2017-12-27
License: MIT</code></pre>
<h1id="terminology">Terminology</h1>
<p>The key words "MUST" and "MUST NOT" in this document are to be interpreted as described in RFC 2119.<ahref="#fn1"class="footnote-ref"id="fnref1"><sup>1</sup></a></p>
<p>The terms "branch" and "network upgrade" in this document are to be interpreted as described in ZIP 200.<ahref="#fn2"class="footnote-ref"id="fnref2"><sup>2</sup></a></p>
<p>The term "Overwinter" in this document is to be interpreted as described in ZIP 201.<ahref="#fn3"class="footnote-ref"id="fnref3"><sup>3</sup></a></p>
<h1id="abstract">Abstract</h1>
<p>This proposal defines a new transaction digest algorithm for signature verification from the Overwinter network upgrade, in order to minimize redundant data hashing in verification, and to cover the input value by the signature.</p>
<h1id="motivation">Motivation</h1>
<p>There are 4 ECDSA signature verification codes in the original Zcash script system: <code>CHECKSIG</code>, <code>CHECKSIGVERIFY</code>, <code>CHECKMULTISIG</code>, <code>CHECKMULTISIGVERIFY</code> ("sigops"). According to the sighash type (<code>ALL</code>, <code>NONE</code>, or <code>SINGLE</code>, possibly modified by <code>ANYONECANPAY</code>), a transaction digest is generated with a double SHA256 of a serialized subset of the transaction, and the signature is verified against this digest with a given public key. The detailed procedure is described in a Bitcoin Wiki article.<ahref="#fn4"class="footnote-ref"id="fnref4"><sup>4</sup></a> The transaction digest is additionally used for the JoinSplit signature (solely with sighash type <code>ALL</code>). <ahref="#fn5"class="footnote-ref"id="fnref5"><sup>5</sup></a></p>
<p>Unfortunately, there are at least 2 weaknesses in the original SignatureHash transaction digest algorithm:</p>
<ul>
<li>For the verification of each signature, the amount of data hashing is proportional to the size of the transaction. Therefore, data hashing grows in O(n<sup>2</sup>) as the number of sigops in a transaction increases. While a 1 MB block would normally take 2 seconds to verify with an average computer in 2015, a 1MB transaction with 5569 sigops may take 25 seconds to verify. This could be fixed by optimizing the digest algorithm by introducing some reusable "midstate", so the time complexity becomes O(n).<ahref="#fn6"class="footnote-ref"id="fnref6"><sup>6</sup></a></li>
<li>The algorithm does not involve the value being spent by the input. This is usually not a problem for online network nodes as they could request for the specified transaction to acquire the output value. For an offline transaction signing device ("cold wallet"), however, the lack of knowledge of input amount makes it impossible to calculate the exact amount being spent and the transaction fee. To cope with this problem a cold wallet must also acquire the full transaction being spent, which could be a big obstacle in the implementation of a lightweight, air-gapped wallet. By including the input value of part of the transaction digest, a cold wallet may safely sign a transaction by learning the value from an untrusted source. In the case that a wrong value is provided and signed, the signature would be invalid and no funds would be lost. <ahref="#fn7"class="footnote-ref"id="fnref7"><sup>7</sup></a></li>
</ul>
<p>Deploying the aforementioned fixes in the original script system is not a simple task.</p>
<h1id="specification">Specification</h1>
<p>A new transaction digest algorithm is defined:</p>
<pre><code>BLAKE2b-256 hash of the serialization of:
1. header of the transaction (4-byte little endian)
2. nVersionGroupId of the transaction (4-byte little endian)
3. hashPrevouts (32-byte hash)
4. hashSequence (32-byte hash)
5. hashOutputs (32-byte hash)
6. hashJoinSplits (32-byte hash)
7. nLockTime of the transaction (4-byte little endian)
8. nExpiryHeight of the transaction (4-byte little endian)
9. sighash type of the signature (4-byte little endian)
10. If we are serializing an input (i.e. this is not a JoinSplit signature hash):
a. outpoint (32-byte hash + 4-byte little endian)
b. scriptCode of the input (serialized as scripts inside CTxOuts)
c. value of the output spent by this input (8-byte little endian)
d. nSequence of the input (4-byte little endian)</code></pre>
<p>The new algorithm is based on the Bitcoin transaction digest algorithm defined in BIP 143,<ahref="#fn8"class="footnote-ref"id="fnref8"><sup>8</sup></a> with replay protection inspired by BUIP-HF v1.2.<ahref="#fn9"class="footnote-ref"id="fnref9"><sup>9</sup></a></p>
<p>The new algorithm MUST be used for signatures created over the Overwinter transaction format.<ahref="#fn10"class="footnote-ref"id="fnref10"><sup>10</sup></a> Combined with the new consensus rule that v1 and v2 transaction formats will be invalid from the Overwinter upgrade,<ahref="#fn11"class="footnote-ref"id="fnref11"><sup>11</sup></a> this effectively means that all transaction signatures from the Overwinter activation height will use the new algorithm.<ahref="#fn12"class="footnote-ref"id="fnref12"><sup>12</sup></a></p>
<p>The BLAKE2b-256 personalization field<ahref="#fn13"class="footnote-ref"id="fnref13"><sup>13</sup></a> is set to:</p>
<p><code>CONSENSUS_BRANCH_ID</code> is the little-endian encoding of <code>BRANCH_ID</code> for the epoch of the block containing the transaction.<ahref="#fn14"class="footnote-ref"id="fnref14"><sup>14</sup></a> Domain separation of the signature hash across parallel branches provides replay protection: transactions targeted for one branch will have invalid signatures on other branches.</p>
<p>Transaction creators MUST specify the epoch they want their transaction to be mined in. Across a network upgrade, this means that if a transaction is not mined before the activation height, it will never be mined.</p>
<p>Semantics of the original sighash types remain unchanged, except the following:</p>
<ol>
<li>The serialization format is changed;</li>
<li>All sighash types commit to the amount being spent by the signed input;</li>
<li><code>SINGLE</code> does not commit to the input index. When <code>ANYONECANPAY</code> is not set, the semantics are unchanged since <code>hashPrevouts</code> and <code>outpoint</code> together implictly commit to the input index. When <code>SINGLE</code> is used with <code>ANYONECANPAY</code>, omission of the index commitment allows permutation of the input-output pairs, as long as each pair is located at an equivalent index.</li>
</ol>
<h2id="field-definitions">Field definitions</h2>
<p>The items 7, 9, 10a, 10d have the same meaning as the original algorithm from Bitcoin.<ahref="#fn15"class="footnote-ref"id="fnref15"><sup>15</sup></a></p>
<h3id="header">1: <code>header</code></h3>
<p>Deserialized into two transaction properties: <code>fOverwintered</code> and <code>nVersion</code>. For transactions that use this transaction digest algorithm, <code>fOverwintered</code> is always set.<ahref="#fn16"class="footnote-ref"id="fnref16"><sup>16</sup></a></p>
<p>Provides domain separation of <code>nVersion</code>. It is only defined if <code>fOverwintered</code> is set, which means that it is always defined for transactions that use this algorithm.<ahref="#fn17"class="footnote-ref"id="fnref17"><sup>17</sup></a></p>
<li>If none of the <code>ANYONECANPAY</code>, <code>SINGLE</code>, <code>NONE</code> sighash type is set, <code>hashSequence</code> is the BLAKE2b-256 hash of the serialization of <code>nSequence</code> of all inputs;
<ul>
<li>The BLAKE2b-256 personalization field is set to <code>ZcashSequencHash</code>.</li>
</ul></li>
<li>Otherwise, <code>hashSequence</code> is a <code>uint256</code> of <code>0x0000......0000</code>.</li>
</ul>
<p>Note: the encoding of the sighash type is masked with <code>0x1F</code> when checking whether or not the <code>SINGLE</code> and <code>NONE</code> flags are set.</p>
<li>If the sighash type is neither <code>SINGLE</code> nor <code>NONE</code>, <code>hashOutputs</code> is the BLAKE2b-256 hash of the serialization of all output amount (8-byte little endian) with <code>scriptPubKey</code> (serialized as scripts inside CTxOuts);</li>
<li>If sighash type is <code>SINGLE</code> and the input index is smaller than the number of outputs, <code>hashOutputs</code> is the BLAKE2b-256 hash of the output (serialized as above) with the same index as the input;
<ul>
<li>The BLAKE2b-256 personalization field is set to <code>ZcashOutputsHash</code> in both cases above.</li>
</ul></li>
<li>Otherwise, <code>hashOutputs</code> is a <code>uint256</code> of <code>0x0000......0000</code>.<ahref="#fn18"class="footnote-ref"id="fnref18"><sup>18</sup></a></li>
</ul>
<p>Note: the encoding of the sighash type is masked with <code>0x1F</code> when checking whether or not the SINGLE` and <code>NONE</code> flags are set.</p>
<li>If <code>vjoinsplits</code> is non-empty, <code>hashJoinSplits</code> is the BLAKE2b-256 hash of the serialization of all JoinSplits (in their canonical transaction serialization format) concatenated with the joinSplitPubKey;
<ul>
<li>The BLAKE2b-256 personalization field is set to <code>ZcashJSplitsHash</code>.</li>
<li>Note that while signatures are omitted, the JoinSplit proofs are included in the signature hash, as with v1 and v2 transactions.</li>
</ul></li>
<li>Otherwise, <code>hashJoinSplits</code> is a <code>uint256</code> of <code>0x0000......0000</code>.</li>
<p>The block height after which the transaction becomes unilaterally invalid, and can never be mined.<ahref="#fn19"class="footnote-ref"id="fnref19"><sup>19</sup></a></p>
<p>The script being currently executed: <code>redeemScript</code> for P2SH, or <code>scriptPubKey</code> in the general case. This is the same script as serialized in the Sprout transaction digest algorithm.</p>
<h3id="c-value">10c: value</h3>
<p>An 8-byte little-endian value of the amount, in zatoshi, spent in this input.</p>
<h2id="notes">Notes</h2>
<p>The <code>hashPrevouts</code>, <code>hashSequence</code>, <code>hashOutputs</code>, and <code>hashJoinSplits</code> calculated in an earlier verification can be reused in other inputs of the same transaction, so that the time complexity of the whole hashing process reduces from O(n<sup>2</sup>) to O(n).</p>
<p>Refer to the reference implementation, reproduced below, for the precise algorithm:</p>
<aclass="sourceLine"id="cb4-18"title="18"><spanclass="cf">for</span> (<spanclass="dt">unsigned</span><spanclass="dt">int</span> n = <spanclass="dv">0</span>; n < txTo.vin.size(); n++) {</a>
<aclass="sourceLine"id="cb4-19"title="19"> ss << txTo.vin[n].prevout;</a>
<aclass="sourceLine"id="cb4-26"title="26"><spanclass="cf">for</span> (<spanclass="dt">unsigned</span><spanclass="dt">int</span> n = <spanclass="dv">0</span>; n < txTo.vin.size(); n++) {</a>
<aclass="sourceLine"id="cb4-27"title="27"> ss << txTo.vin[n].nSequence;</a>
<aclass="sourceLine"id="cb4-34"title="34"><spanclass="cf">for</span> (<spanclass="dt">unsigned</span><spanclass="dt">int</span> n = <spanclass="dv">0</span>; n < txTo.vout.size(); n++) {</a>
<aclass="sourceLine"id="cb4-35"title="35"> ss << txTo.vout[n];</a>
<aclass="sourceLine"id="cb4-46"title="46"><spanclass="cf">for</span> (<spanclass="dt">unsigned</span><spanclass="dt">int</span> n = <spanclass="dv">0</span>; n < txTo.vjoinsplit.size(); n++) {</a>
<aclass="sourceLine"id="cb4-47"title="47"> ss << txTo.vjoinsplit[n];</a>
<aclass="sourceLine"id="cb4-48"title="48"> }</a>
<aclass="sourceLine"id="cb4-49"title="49"> ss << txTo.joinSplitPubKey;</a>
<aclass="sourceLine"id="cb4-78"title="78"><spanclass="co">// The input being signed (replacing the scriptSig with scriptCode + amount)</span></a>
<aclass="sourceLine"id="cb4-79"title="79"><spanclass="co">// The prevout may already be contained in hashPrevout, and the nSequence</span></a>
<aclass="sourceLine"id="cb4-80"title="80"><spanclass="co">// may already be contained in hashSequence.</span></a>
<aclass="sourceLine"id="cb4-81"title="81"> ss << txTo.vin[nIn].prevout;</a>
<aclass="sourceLine"id="cb4-82"title="82"> ss <<<spanclass="kw">static_cast</span><<spanclass="at">const</span> CScriptBase&>(scriptCode);</a>
<aclass="sourceLine"id="cb4-83"title="83"> ss << amount;</a>
<aclass="sourceLine"id="cb4-84"title="84"> ss << txTo.vin[nIn].nSequence;</a>
<p>To ensure consistency in consensus-critical behaviour, developers should test their implementations against the ZIP 143 test vectors<ahref="#fn20"class="footnote-ref"id="fnref20"><sup>20</sup></a>. The first two test vectors are broken out below for clarity. Note that 32-byte values below are exactly as the hash function returns, and are not reversed. Further examples can be found in the SignatureHash test data<ahref="#fn21"class="footnote-ref"id="fnref21"><sup>21</sup></a>.</p>
<p>This proposal is backwards-compatible with old UTXOs. It is <strong>not</strong> backwards-compatible with older software. All transactions will be required to use this transaction digest algorithm for signatures, and so transactions created by older software will be rejected by the network.</p>
<liid="fn1"><p><ahref="https://tools.ietf.org/html/rfc2119">Key words for use in RFCs to Indicate Requirement Levels</a><ahref="#fnref1"class="footnote-back">↩</a></p></li>
<li><ahref="https://bitcointalk.org/?topic=140078">New Bitcoin vulnerability: A transaction that takes at least 3 minutes to verify</a></li>
<li><ahref="http://rusty.ozlabs.org/?p=522">The Megatransaction: Why Does It Take 25 Seconds?</a></li>
</ul>
<ahref="#fnref6"class="footnote-back">↩</a></li>
<liid="fn7"><p><ahref="https://bitcointalk.org/index.php?topic=181734.0">SIGHASH_WITHINPUTVALUE: Super-lightweight HW wallets and offline data</a><ahref="#fnref7"class="footnote-back">↩</a></p></li>
<liid="fn8"><p><ahref="https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki">BIP 143: Transaction Signature Verification for Version 0 Witness Program</a><ahref="#fnref8"class="footnote-back">↩</a></p></li>
<liid="fn9"><p><ahref="https://github.com/Bitcoin-ABC/bitcoin-abc/blob/master/doc/abc/replay-protected-sighash.md">BUIP-HF Digest for replay protected signature verification across hard forks, version 1.2</a><ahref="#fnref9"class="footnote-back">↩</a></p></li>
<liid="fn10"><p><ahref="https://github.com/zcash/zips/blob/master/zip-0202.rst">ZIP 202: Version 3 Transaction Format for Overwinter</a><ahref="#fnref10"class="footnote-back">↩</a></p></li>
<liid="fn11"><p><ahref="https://github.com/zcash/zips/blob/master/zip-0202.rst">ZIP 202: Version 3 Transaction Format for Overwinter</a><ahref="#fnref11"class="footnote-back">↩</a></p></li>
<liid="fn12"><p><ahref="https://github.com/zcash/zips/blob/master/zip-0201.rst">ZIP 201: Network Peer Management for Overwinter</a><ahref="#fnref12"class="footnote-back">↩</a></p></li>
<liid="fn13"><p><ahref="https://blake2.net/blake2.pdf">"BLAKE2: simpler, smaller, fast as MD5", Section 2.8</a><ahref="#fnref13"class="footnote-back">↩</a></p></li>
<liid="fn16"><p><ahref="https://github.com/zcash/zips/blob/master/zip-0202.rst">ZIP 202: Version 3 Transaction Format for Overwinter</a><ahref="#fnref16"class="footnote-back">↩</a></p></li>
<liid="fn17"><p><ahref="https://github.com/zcash/zips/blob/master/zip-0202.rst">ZIP 202: Version 3 Transaction Format for Overwinter</a><ahref="#fnref17"class="footnote-back">↩</a></p></li>
<liid="fn18"><p>In the original algorithm, a <code>uint256</code> of <code>0x0000......0001</code> is committed if the input index for a <code>SINGLE</code> signature is greater than or equal to the number of outputs. In this ZIP a <code>0x0000......0000</code> is commited, without changing the semantics.<ahref="#fnref18"class="footnote-back">↩</a></p></li>
<liid="fn20"><p><ahref="https://github.com/zcash-hackworks/zcash-test-vectors/blob/master/zip_0143.py">ZIP 143 Test Vectors</a><ahref="#fnref20"class="footnote-back">↩</a></p></li>
<liid="fn21"><p><ahref="https://github.com/zcash/zcash/blob/master/src/test/data/sighash.json">SignatureHash Test Vectors</a><ahref="#fnref21"class="footnote-back">↩</a></p></li>
<liid="fn22"><p><ahref="https://github.com/zcash/zips/blob/master/zip-0201.rst">ZIP 201: Network Peer Management for Overwinter</a><ahref="#fnref22"class="footnote-back">↩</a></p></li>
