Commit Graph

98 Commits

Author SHA1 Message Date
Felix Lange 693d9ccbfb trie: more node iterator improvements (#14615)
* ethdb: remove Set

Set deadlocks immediately and isn't part of the Database interface.

* trie: add Err to Iterator

This is useful for testing because the underlying NodeIterator doesn't
need to be kept in a separate variable just to get the error.

* trie: add LeafKey to iterator, panic when not at leaf

LeafKey is useful for callers that can't interpret Path.

* trie: retry failed seek/peek in iterator Next

Instead of failing iteration irrecoverably, make it so Next retries the
pending seek or peek every time.

Smaller changes in this commit make this easier to test:

* The iterator previously returned from Next on encountering a hash
  node. This caused it to visit the same path twice.
* Path returned nibbles with terminator symbol for valueNode attached
  to fullNode, but removed it for valueNode attached to shortNode. Now
  the terminator is always present. This makes Path unique to each node
  and simplifies Leaf.

* trie: add Path to MissingNodeError

The light client trie iterator needs to know the path of the node that's
missing so it can retrieve a proof for it. NodeIterator.Path is not
sufficient because it is updated when the node is resolved and actually
visited by the iterator.

Also remove unused fields. They were added a long time ago before we
knew which fields would be needed for the light client.
2017-06-20 18:26:09 +02:00
S. Matthew English 061889d4ea rlp, trie, contracts, compression, consensus: improve comments (#14580) 2017-06-12 14:45:17 +02:00
Felix Lange 4047ccad2f trie: add start key to NodeIterator constructors
The 'step' method is split into two parts, 'peek' and 'push'. peek
returns the next state but doesn't make it current.

The end of iteration was previously tracked by setting 'trie' to nil.
End of iteration is now tracked using the 'iteratorEnd' error, which is
slightly cleaner and requires less code.
2017-04-25 02:14:31 +02:00
Felix Lange a13e920af0 trie: clean up iterator constructors
Make it so each iterator has exactly one public constructor:

- NodeIterators can be created through a method.
- Iterators can be created through NewIterator on any NodeIterator.
2017-04-25 02:14:31 +02:00
Felix Lange f958d7d482 trie: rework and document key encoding
'encode' and 'decode' are meaningless because the code deals with three
encodings. Document the encodings and give a name to each one.
2017-04-25 02:14:31 +02:00
Felix Lange a31d268b76 trie: remove Key in MissingNodeError
The key was constructed from nibbles, which isn't possible for all
nodes. Remove the only use of Key in LightTrie by always retrying with
the original key that was looked up.
2017-04-18 14:52:11 +02:00
Nick Johnson b35aa21f9f trie: implement unionIterator (#14312) 2017-04-13 11:14:19 +02:00
Péter Szilágyi 09777952ee core, consensus: pluggable consensus engines (#3817)
This commit adds pluggable consensus engines to go-ethereum. In short, it
introduces a generic consensus interface, and refactors the entire codebase to
use this interface.
2017-04-05 00:16:29 +02:00
Felix Lange f1534f5797 trie, whisper/whisperv5: use math/rand Read function 2017-03-22 20:49:15 +01:00
Péter Szilágyi d4fd06c3dc
all: blidly swap out glog to our log15, logs need rework 2017-02-23 12:16:44 +02:00
Nick Johnson 555273495b trie: add difference iterator (#3637)
This PR implements a differenceIterator, which allows iterating over trie nodes
that exist in one trie but not in another. This is a prerequisite for most GC
strategies, in order to find obsolete nodes.
2017-02-22 23:49:34 +01:00
Felix Lange b9b3efb09f all: fix ineffectual assignments and remove uses of crypto.Sha3
go get github.com/gordonklaus/ineffassign
ineffassign .
2017-01-09 16:24:42 +01:00
Péter Szilágyi 18c77744ff
all: fix spelling errors 2017-01-06 19:44:35 +02:00
Felix Lange f2da6581ba all: fix issues reported by honnef.co/go/simple/cmd/gosimple 2017-01-06 18:18:07 +01:00
Felix Lange 35a7dcb162 all: gofmt -w -s 2017-01-06 15:52:03 +01:00
Felix Lange d3b751e4d9 trie: remove dependency on ethdb
This removes the core/types -> leveldb dependency.
2017-01-06 14:15:22 +01:00
Péter Szilágyi 90b16a3e85
core/state, eth/downloader, trie: reset fast-failure on progress 2016-10-31 14:19:14 +02:00
Péter Szilágyi f8608a5228 trie: while fast syncing, don't keep trie nodes in memory (#3186) 2016-10-21 17:34:17 +02:00
Péter Szilágyi 88a593d559
cmd/geth, trie: report on trie cache unloads, also add debug log 2016-10-19 17:31:19 +03:00
Felix Lange 5d9bb0a050 trie: add metric to track cache unloading (#3175) 2016-10-19 16:07:19 +02:00
Felix Lange d48e6ae66f trie: make the cache miss counter a real metric 2016-10-19 15:22:37 +02:00
Jeffrey Wilcke 25ac04a444 Merge pull request #3153 from fjl/trie-unload-fix
trie: improve cache unloading mechanism
2016-10-19 13:35:49 +02:00
Péter Szilágyi 2bb5ec1e41
cmd/geth, trie: track and report trie cache misses 2016-10-19 14:12:35 +03:00
Felix Lange 8d56bf5ceb trie: ensure dirty flag is unset for embedded child nodes
This was caught by the new invariant check.
2016-10-18 04:57:47 +02:00
Felix Lange 177cab5fe7 trie: ensure resolved nodes stay loaded
Commit 40cdcf1183 broke the optimisation which kept nodes resolved
during Get in the trie. The decoder assigned cache generation 0
unconditionally, causing resolved nodes to get flushed on Commit.

This commit fixes it and adds two tests.
2016-10-18 04:57:47 +02:00
Felix Lange 187d6a66a5 trie: avoid loading the root node twice
New checks whether the root node is present by loading it from the
database. Keep the node around instead of discarding it.
2016-10-18 04:57:47 +02:00
Felix Lange 2cd7a0395d trie: fix regression that linked all downloaded nodes together
The trie sync code links subtries using pointers into node structs.
Since commit 40cdcf1183 nodes are no longer copied when unpacking from
an interface value, causing all nodes to get linked up as the sync
progresses. Fix it by breaking the pointer chain with an explicit copy.
2016-10-15 00:42:52 +02:00
Felix Lange 40cdcf1183 trie, core/state: improve memory usage and performance (#3135)
* trie: store nodes as pointers

This avoids memory copies when unwrapping node interface values.

name      old time/op  new time/op  delta
Get        388ns ± 8%   215ns ± 2%  -44.56%  (p=0.000 n=15+15)
GetDB      363ns ± 3%   202ns ± 2%  -44.21%  (p=0.000 n=15+15)
UpdateBE  1.57µs ± 2%  1.29µs ± 3%  -17.80%  (p=0.000 n=13+15)
UpdateLE  1.92µs ± 2%  1.61µs ± 2%  -16.25%  (p=0.000 n=14+14)
HashBE    2.16µs ± 6%  2.18µs ± 6%     ~     (p=0.436 n=15+15)
HashLE    7.43µs ± 3%  7.21µs ± 3%   -2.96%  (p=0.000 n=15+13)

* trie: close temporary databases in GetDB benchmark

* trie: don't keep []byte from DB load around

Nodes decoded from a DB load kept hashes and values as sub-slices of
the DB value. This can be a problem because loading from leveldb often
returns []byte with a cap that's larger than necessary, increasing
memory usage.

* trie: unload old cached nodes

* trie, core/state: use cache unloading for account trie

* trie: use explicit private flags (fixes Go 1.5 reflection issue).

* trie: fixup cachegen overflow at request of nick

* core/state: rename journal size constant
2016-10-14 19:04:33 +03:00
Felix Lange c3a77d6268 trie: fix delete bug for values contained in fullNode
Delete crashed if a fullNode contained a valueNode directly. This bug is
very unlikely to occur with SecureTrie, but can happen with regular
tries. This commit also introduces a randomised test which triggers all
trie operations, which should prevent such bugs in the future.

Credit for finding this bug goes to Github user @rjl493456442.
2016-09-29 22:53:59 +02:00
Felix Lange ba8c4c6b1a trie: remove dependency on package crypto
Package crypto needs cgo, which is inconvenient for some build
configurations.
2016-09-29 22:23:24 +02:00
Péter Szilágyi 710435b51b core, eth, trie: reuse trie journals in all our code 2016-09-28 11:27:31 +03:00
Felix Lange cd791bd855 core, trie: replace state caches with trie journal 2016-09-28 11:27:28 +03:00
Péter Szilágyi 1d5d21726a Merge pull request #2627 from karalabe/concurrent-head-sync
eth/downloader, trie: pull head state concurrently with chain
2016-05-31 11:52:16 +03:00
Péter Szilágyi 4f1d92b332 eth/downloader, trie: pull head state concurrently with chain 2016-05-27 14:26:00 +03:00
Péter Szilágyi 748d1c171d core, core/state, trie: enterprise hand-tuned multi-level caching 2016-05-26 16:33:09 +03:00
Felix Lange 6fdd0893c3 all: fix go vet warnings 2016-04-15 11:17:27 +02:00
Felix Lange d04a2e7557 all: update license information 2016-04-15 09:48:05 +02:00
Péter Szilágyi e90958cd29 cmd, eth, ethdb, node: prioritise chaindata for resources, bump cache 2016-03-09 10:33:39 +02:00
Jeffrey Wilcke 483feb0d3f Merge pull request #2242 from jimenezrick/upstream-crypto
Closes #2241: Use Keccak-256 from golang.org/x/crypto/sha3 and mention explicitly
2016-02-24 12:57:57 +01:00
Ricardo Catalinas Jiménez 436fc8d76a all: Rename crypto.Sha3{,Hash}() to crypto.Keccak256{,Hash}()
As we aren't really using the standarized SHA-3
2016-02-21 22:34:34 +00:00
Péter Szilágyi b8d59d9c98 core/state, trie: switch iterator panics to error fields 2016-02-16 12:37:00 +02:00
Péter Szilágyi 151c7bef41 core/state, trie: node iterator reports parent hashes too 2016-02-16 12:21:08 +02:00
Péter Szilágyi 5a057a8ded core/state, trie: surface iterator entry hashes 2016-02-16 12:21:08 +02:00
Péter Szilágyi 7e29b0b5b4 core/state, trie: add node iterator, test state/trie sync consistency 2016-02-16 12:21:08 +02:00
Péter Szilágyi f3d4ce0d16 core/state, ethdb, trie: test intermediate secure key leak, fix memdb bug 2016-01-20 16:06:28 +02:00
Péter Szilágyi 886478b18b core/state, trie: don't leak database writes before commit 2016-01-20 12:09:24 +02:00
zsfelfoldi ef422ee1e1 light: implemented odr-capable trie and state structures 2015-12-17 16:07:54 +01:00
zsfelfoldi 52904ae32f trie: added error handling
Created alternate versions of Trie and SecureTrie functions that can return a MissingNodeError (used by ODR services)
2015-12-01 01:38:32 +01:00
Péter Szilágyi 5b0ee8ec30 core, eth, trie: fix data races and merge/review issues 2015-10-21 16:49:55 +03:00
Péter Szilágyi b97e34a8e4 eth/downloader: concurrent receipt and state processing 2015-10-19 10:03:10 +03:00