Merge pull request #247 from zcash/add-darksidewalletd-docs

Add initial darksidewalletd docs

README.md

@@ -132,6 +132,11 @@ If lightwalletd detects corruption in these cache files, it will log
 a message containing the string `CORRUPTION` and also indicate the
 nature of the corruption.
 
+## Darksidewalletd & Testing
+
+Lightwalletd now supports a mode that enables integration testing of itself and
+wallets that connect to it. See the [darksidewalletd
+docs](docs/darksidewalletd.md) for more information.
 
 # Pull Requests
 

docs/darksidewalletd.md

@@ -0,0 +1,168 @@
+# Intro to darksidewalletd
+
+Darksidewalletd is a feature included in lightwalletd, enabled by the
+`--darkside-very-insecure` flag, which can serve arbitrary blocks to a zcash
+light client wallet. This is useful for security and reorg testing. It includes
+a minimally-functional mock zcashd which comes with a gRPC API for controlling
+which blocks it will serve.
+
+This means that you can use darksidewalletd to alter the sequence of blocks, the
+blocks and information inside the blocks, and much more--then serve it to
+a light client wallet to see how it behaves. Multiple wallets can connect to the
+same darksidewalletd at the same time. Darksidewalletd should only be used for
+testing, and therefore is hard-coded to shut down after 30 minutes of operation
+to prevent accidental deployment as a server.
+
+## Dependencies 
+
+Lightwalletd and most dependencies of lightwalletd, including Go version 1.11 or
+later, but not zcashd. Since Darksidewalletd mocks zcashd, it can run standalone
+and does use zcashd to get blocks or send and receive transactions.
+
+For the tutorial (and further testing) the tool grpcurl will be used to call the
+API and set blocks.
+
+## Overview
+### Running darksidewalletd
+
+To start darksidewalletd, you run lightwalletd with a flag:
+
+`./lightwalletd --darkside-very-insecure`
+
+To prevent accidental deployment in production, it will automatically shut off
+after 30 minutes.
+
+### Default set of blocks
+
+There’s a file in the repo called ./testdata/default-darkside-blocks. This
+contains the blocks darksidewalletd loads by default. The format of the file is
+one hex-encoded block per line.
+
+### Generating fake blocks with genblocks
+
+Lightwalletd and the wallets themselves don’t actually perform any validation of
+the blocks (beyond checking the blocks’ prevhashes, which is used to detect
+reorgs). For information on block headers, see the zcash protocol specification.
+That means the blocks we give darksidewalletd don’t need to be fully valid, see
+table:
+
+
+Block component|Must be valid|Must be partially valid|Not checked for validity 
+:-----|:-----|:-----|:-----
+nVersion|x| | 
+hashPrevBlock|x| | 
+hashMerkleRoot| | |x 
+hashFinalSaplingRoot| | |x 
+nTime| | |x 
+nBits| | |x 
+nNonce| | |x 
+Equihash solution| | |x 
+Transaction Data*| |x|  
+
+\*Transactions in blocks must conform to the transaction format, but not need
+valid zero-knowledge proofs etc.
+
+There’s a tool to help with generating these fake just-barely-valid-enough
+blocks, it’s called genblocks. To use it you create a directory of text files,
+one file per block, and each line in the file is a hex-encoded transaction that
+should go into that block:
+
+```
+mkdir blocksA
+touch blocksA/{1000,1001,1002,1003,1004,1005}.txt
+echo “some hex-encoded transaction you want to put in block 1003” > blocksA/1003.txt
+```
+
+This will output the blocks, one hex-encoded block per line (the same format as
+./testdata/default-darkside-blocks). 
+
+Tip: Because nothing is checking the full validity of transactions, you can get
+any hex-encoded transaction you want from a block explorer and put those in the
+block files. The sochain block explorer makes it easy to obtain the raw
+transaction hex, by viewing the transaction (example), clicking “Raw Data”, then
+copying the “tx_hex” field.
+
+### Using DarksideSetState to submit a new set of blocks
+
+As mentioned, the darksidewalletd PR adds an RPC, it’s called DarksideSetState,
+which lets you control the blocks the mock zcashd is serving. Well, it would let
+you if you could speak gRPC. If you want to do it manually (not part of some
+test code) you can use a tool called grpcurl to call the API and set the blocks.
+Once you have that installed, there’s a script in utils/submitblocks.sh to
+submit the blocks, which internally uses grpcurl, e.g.:
+
+```
+./genblocks --blocks-dir blocksA > blocksA.txt
+./utils/submitblocks.sh 1000 1000 blocksA.txt
+```
+
+In the submitblocks.sh command, the first “1000” is the height to serve the
+first block at (so that if blocksA.txt contains 6 blocks they will be served as
+heights 1000, 1001, 1002, 1003, 1004, and 1005. If the genblocks tool was used
+to create the blocksA file, then this argument must match what was given to
+genblocks, otherwise the heights in the coinbase transactions will not match up
+with the height lightwalletd is serving the blocks as. The second “1000” sets
+the value that lightwalletd will report the sapling activation height to be.
+
+Tip: The DarksideSetState expects a complete set of blocks for the mock zcashd
+to serve, if you want to just add one block, for example, you need to re-submit
+all of the blocks including the new one.
+
+## Tutorial
+### Triggering a Reorg
+
+To begin following these instructions, build lightwalletd but do not start
+darksidewalletd yet. If you started it during the overview above, kill the
+server before starting the tutorial. 
+
+We’ll use genblocks to generate the hex-encoded blocks, then
+./utils/submitblocks.sh to get them into darksidewalletd. We’ll call the blocks
+before the reorg “blocksA” and the blocks after the reorg “blocksB”:
+
+```
+mkdir blocksA
+touch blocksA/{1000,1001,1002,1003,1004,1005}.txt
+mkdir blocksB
+touch blocksB/{1000,1001,1002,1003,1004,1005,1006}.txt
+echo "0400008085202f8901950521a79e89ed418a4b506f42e9829739b1ca516d4c590bddb4465b4b347bb2000000006a4730440220142920f2a9240c5c64406668c9a16d223bd01db33a773beada7f9c9b930cf02b0220171cbee9232f9c5684eb918db70918e701b86813732871e1bec6fbfb38194f53012102975c020dd223263d2a9bfff2fa6004df4c07db9f01c531967546ef941e2fcfbffeffffff026daf9b00000000001976a91461af073e7679f06677c83aa48f205e4b98feb8d188ac61760356100000001976a91406f6b9a7e1525ee12fd77af9b94a54179785011b88ac4c880b007f880b000000000000000000000000" > blocksB/1004.txt
+```
+
+Use genblocks to put together the fake blocks:
+
+```
+./genblocks --blocks-dir blocksA > testdata/default-darkside-blocks
+./genblocks --blocks-dir blocksB > testdata/darkside-blocks-reorg
+```
+
+(note: this is overwrites the file darksidewalletd loads by default, testdata/default-darkside-blocks)
+
+Now you can start darksidewalletd and it’ll load the blocksA blocks:
+
+`./lightwalletd --darkside-very-insecure`
+
+That will have loaded and be serving the blocksA blocks. We can push up the
+blocksB blocks using ./utils/submitblocks.sh:
+
+`./utils/submitblocks.sh 1000 1000 testdata/darkside-blocks-reorg`
+
+We should now see a reorg in server.log:
+
+```
+{"app":"frontend-grpc","duration":442279,"error":null,"level":"info","method":"/cash.z.wallet.sdk.rpc.CompactTxStreamer/DarksideSetState","msg":"method called","peer_addr":{"IP":"127.0.0.1","Port":47636,"Zone":""},"time":"2020-03-23T13:59:41-06:00"}
+{"app":"frontend-grpc","hash":"a244942179988ea6e56a3a55509fcf22673df26200c67bebd93504385a1a7c4f","height":1004,"level":"warning","msg":"REORG","phash":"06e7c72646e3d51417de25bd83896c682b72bdf5be680908d621cba86d222798","reorg":1,"time":"2020-03-23T13:59:44-06:00"}
+```
+
+## Use cases
+
+Check out some of the potential security test cases here: [wallet <->
+lightwalletd integration
+tests](https://github.com/zcash/lightwalletd/blob/master/docs/integration-tests.md)
+
+## Source Code
+* cmd/genblocks -- tool for generating fake block sets.
+* testdata/default-darkside-blocks -- the set of blocks loaded by default
+* common/darkside.go -- implementation of darksidewalletd
+* frontend/service.go -- entrypoints for darksidewalletd GRPC APIs
+
+
+

docs/integration-tests.md

@@ -0,0 +1,71 @@
+# Wallet ⟷ Lightwalletd Integration Tests
+
+## High-priority tests
+Funds are at risk if these tests fail. 
+
+**Reorged-Away Transaction** 
+A transparent/shielded transaction is sent to the wallet in block N containing value v. There's a reorg to height N-k for some k >= 1, and after the reorg the original transaction is no longer there but there is a new transaction with a different value u. Before the reorg, the wallet should detect the transaction and show unconfirmed balance v. After the reorg, the wallet should show unconfirmed balance u. Some number of blocks later, the balance is marked as confirmed.
+
+Consequences if this test fails: An attacker could take advantage of regular/accidental reorgs to confuse the wallet about its balance.
+
+**Dropped from Mempool**
+Similar to the reorged-away transaction test, except the transaction only enters the mempool and is never mined.
+
+Consequences: An attacker could confuse wallets about their balance by arranging for a transaction to enter the mempool but not be mined.
+
+**Transparent TXID Malleated**
+The wallet sends a transparent transaction. Its transaction ID is malleated to change its transaction ID, and then mined. After sending the transaction, the wallet’s balance should be reduced by the value of the transaction. 100 blocks after the transparent transaction was mined, the wallet’s balance should still be reduced by that amount.
+
+Consequences if this test fails: An attacker could malleate one of the wallet’s transparent transactions, and if it times out thinking it was never mined, the wallet would think it has balance when it doesn’t.
+
+**Transaction Never Mined**
+The wallet broadcasts a transparent/shielded transaction optionally with an expiry height. For 100 blocks (or at least until the expiry height), the transaction is never mined. After sending the transaction, the wallet’s balance should be reduced by the value of the transaction, and it should stay reduced by that amount until the expiry height (if any).
+
+Consequences if this test fails: If the wallet concludes the transaction will never be mined before the expiry height, then an attacker can delay mining the transaction to cause the wallet to think it has more funds than it does.
+
+**Transaction Created By Other Wallet**
+A seed is imported into three wallets with transparent/shielded funds. Wallet A sends a payment to some address. At the same time, Wallet B sends a payment of a different amount using some of the same UTXOs or notes. Wallet C does not send any payments. Wallet B’s transaction gets mined instead of Wallet A’s. The balances of all three wallets are decreased by the value of Wallet B’s transaction.
+
+Consequences if this test fails: A user importing their seed into multiple wallets and making simultaneous transactions could lead them to be confused about their balance.
+
+**Anchor Invalidation**
+A wallet broadcasts a sapling transaction using a recent anchor. A reorg occurs which invalidates that anchor, i.e. some of the previous shielded transactions changed. (Depending on how we want to handle this) the wallet either detects this and re-broadcasts the transaction or marks the transaction as failed and the funds become spendable again.
+
+Consequences if this test fails: Wallets might get confused about their balance if this ever occurs.
+
+**Secret Transactions**
+Lightwalletd has some shielded/transparent funds. It creates a real transaction sending these funds to the wallet, such that if the transaction were broadcast on the Zcash network, the wallet really would get the funds. However, instead of broadcasting the transaction, the lightwalletd operator includes the transaction in a compact block, but does not broadcast the transaction to the actual network. The wallet should detect that the transaction has not really been mined by miners on the Zcash network, and not show an increased balance.
+
+(Currently, this test will fail, since the wallet is not checking the PoW or block headers at all. Worse, even with PoW/header checks, lightwalletd could mine down the difficulty to have their wallets follow an invalid chain. To comabt this wallets would need to reach out to multiple independent lightwalletds to verify it has the highest PoW chain. Or Larry’s idea, to warn when the PoW is below a certain threshold (chosen to be above what most attackers could do but low enough we’d legitimately want to warn users if it drops that low), I like a lot better.)
+
+Consequences if this test fails: lightwalletd can make it appear as though a wallet received funds when it didn’t.
+
+## Medium-priority tests
+Funds aren’t at risk if these fail but there’s a severe problem. 
+
+**Normal Payments**
+Wallet A sends a shielded/transparent transaction to Wallet B. Wallet B receives the transaction and sends half back to wallet A. Wallet A receives the transaction, and B receives change. All of the balances end up as expected.
+
+Consequences if this test fails: Normal functionality of the wallet is broken.
+
+**Mempool DoS**
+The transactions in the mempool constantly churn. The wallet should limit its bandwidth used to fetch new transactions in the mempool, rather than using an unlimited amount.
+
+Consequences if this test fails: It’s possible to run up the bandwidth bills of wallet users.
+
+
+## Low-priority tests
+ These won’t occur unless lightwalletd is evil. 
+
+**High Block Number**
+Lightwalletd announces that the latest block is some very large number, much larger than the actual block height. The wallet syncs up to that point (with lightwalletd providing fake blocks all the way). Lightwalletd then stops lying about the block height and blocks. This should trigger the wallet’s reorg limit and the wallet should be unusable.
+
+**Repeated Note**
+A shielded transaction is sent to the wallet. Lightwalletd simply repeats the transaction in a compact block sent to the wallet. The wallet should not think it has twice as much money. From this point, no shielded transactions the wallet sends can be mined, since they will use invalid anchors.
+
+**Invalid Note**
+Same as repeated note above, but random data. The results should be exactly the same.
+
+**Omitted Note**
+A shielded transaction is sent to the wallet. Lightwalletd simply does not send the transaction to the wallet (omits it from the compact block).  From this point, no shielded transactions the wallet sends can be mined, since they will use invalid anchors.
+