* StoreKVPair protobuf message definition and generated go types
* store WriteListener
* update MultiStore, CacheWrap, CacheWrapper interfaces
* adjust KVStores to fit new CacheWrapper interface
* new ListenKVStore
* adjust multistores to fit new MultiStore interface and enable wrapping returned KVStores with the new ListenKVStore
* typo fixes in adr
* ListenKV Store test
* update server mock KVStore and MultiStore
* multistore unit test; fix multistore constructor
* update changelog
* fix bug identified in CI
* improve codecov, minor fixes/adjustments
* review fixes
* review updates; flip set to delete in KVStorePair, updated proto-docs from running 'make proto-gen'
* Skips very noisy benchmarks that end up running only for b.N=1 because
their entire time is spent in setup, and varying parameters doesn't change
much given that the number of stores is what dominates the expense. To
ensure we can provide reliable benchmarks, progressively for the project,
skip these until there is a proper re-work of what the benchmarks need to do
* Previously sub-benchmarks: b.Run(...) did not b.ReportAllocs() due to a faulty
assumption that invoking b.ReportAllocs() at the top would be inherited by
all sub-benchmarks. This change fixes that
Fixes#8779Fixes#8855
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Reduces CPU burn by using a typed List to avoid the expensive type
assertions from using an interface. This is the only option for now
until Go makes generics generally available.
The change brings time spent on the time assertion cummulatively to:
580ms down from 6.88s
Explanation:
The type assertions were showing up heavily in profiles:
* Before this commit
```shell
Total: 42.18s
ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator
in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go
14.01s 18.87s (flat, cum) 44.74% of Total
. . 17: items []*kv.Pair
. . 18: ascending bool
. . 19:}
. . 20:
. . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator {
. 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len())
. . 23:
. . 24: var entered bool
. . 25:
510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() {
6.85s 6.88s 27: item := e.Value.(*kv.Pair)
5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) {
120ms 120ms 29: if entered {
. . 30: break
. . 31: }
. . 32:
. . 33: continue
. . 34: }
. . 35:
820ms 980ms 36: itemsInDomain = append(itemsInDomain, item)
. . 37: entered = true
. . 38: }
. . 39:
. 1.21s 40: return &memIterator{
. . 41: start: start,
. . 42: end: end,
. . 43: items: itemsInDomain,
. . 44: ascending: ascending,
. . 45: }
```
and given that the list only uses that type, it is only right to lift the
code from container/list.List, and only modify Element.Value's type.
For emphasis, the code is basically just a retrofit of
container/list/list.go but with a typing, and we'll keep it as is
until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after
generics have landed.
* After this commit
```shell
Total: 45.25s
ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator
in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go
4.84s 6.77s (flat, cum) 14.96% of Total
. . 16: items []*kv.Pair
. . 17: ascending bool
. . 18:}
. . 19:
. . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator {
. 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len())
. . 22:
. . 23: var entered bool
. . 24:
60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() {
580ms 580ms 26: item := e.Value
3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) {
80ms 80ms 28: if entered {
. . 29: break
. . 30: }
. . 31:
. . 32: continue
. . 33: }
. . 34:
440ms 580ms 35: itemsInDomain = append(itemsInDomain, item)
. . 36: entered = true
. . 37: }
. . 38:
. 260ms 39: return &memIterator{
. . 40: start: start,
. . 41: end: end,
. . 42: items: itemsInDomain,
. . 43: ascending: ascending,
. . 44: }
```
Fixes#8810
After continuously profiling InitGensis with 100K accounts, it showed
pathologically slow code, that was the result of a couple of patterns:
* Unconditional and not always necessary map lookups
* O(n^2) sdk.AccAddressFromBech32 retrievals when the code is expensive,
during a quicksort
The remedy involved 4 parts:
* O(n) sdk.AccAddressFromBech32 invocations, down from O(n^2) in the quicksort
* Only doing map lookups when the domain key check has passed
* Using a black magic compiler technique of the map clearing idiom
* Zero allocation []byte<->string conversion
With 100K accounts, this brings InitGenesis down to ~6min, instead of
20+min, it reduces the sort code from ~7sec down to 50ms.
Also some simple benchmark reflect the change:
```shell
name old time/op new time/op delta
SanitizeBalances500-8 19.3ms ±10% 1.5ms ± 5% -92.46% (p=0.000 n=20+20)
SanitizeBalances1000-8 41.9ms ± 8% 3.0ms ±12% -92.92% (p=0.000 n=20+20)
name old alloc/op new alloc/op delta
SanitizeBalances500-8 9.05MB ± 6% 0.56MB ± 0% -93.76% (p=0.000 n=20+18)
SanitizeBalances1000-8 20.2MB ± 3% 1.1MB ± 0% -94.37% (p=0.000 n=20+19)
name old allocs/op new allocs/op delta
SanitizeBalances500-8 72.4k ± 6% 4.5k ± 0% -93.76% (p=0.000 n=20+20)
SanitizeBalances1000-8 162k ± 3% 9k ± 0% -94.40% (p=0.000 n=20+20)
```
The CPU profiles show the radical change as per
https://github.com/cosmos/cosmos-sdk/issues/7766#issuecomment-786671734
Later on, we shall do more profiling and fixes but for now this brings
down the run-time for InitGenesis.
Fixes#7766
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
With this change, we'll get details on the number of
allocations performed by code. Later on when we have
continuous benchmarking infrastructure, this change
will prove useful to flag regressions.
Fixes#8459
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
* Add state sync support
* fix incorrect test tempdir
* proto: move and update Protobuf schemas
* proto: lint fixes
* comment tweaks
* don't use type aliasing
* don't call .Error() when logging errors
* use create terminology instead of take for snapshots
* reuse chunk hasher
* simplify key encoding code
* track chunk index in Manager
* add restoreDone message for Manager
* add a ready channel to Snapshotter.Restore()
* add comment on streaming IO API
* use sdkerrors for error handling
* fix incorrect error
* tweak changelog
* syntax fix
* update test code after merge
* Add height in exported genesis
* +1
* Add test
* Refactor ctx in setupApp
* Use amino in export
* Use tmjson
* Add custom initialVersion (set to 0 for now)
* Add comment
* Add mount in initChainer
* app.LastBlockheight
* InitializeAndSeal in InitChain?
* Revert create store with initial version
* Update to latest iavl
* Check height in test
* Make it work
* Add more tests
* Rename interface
* Use struct isntead of 6 args
* Fix lint
* Remove stray fmt
* Revert go mod/sum
* Install iavl rc3
* Update comments
* Add fee in network
* Typo
* Fix logic in commit
* Fix tests
* Only set initial version on > 1
* Genesis block num = 1
* Fresh chain, genesis block = 0
* Add comments
* Revert Mutable/ImmutableTree
* Allow for zero height
* Fix restart
* Add comments
* Add comments, fix test
* Fix remaining one test
* Add panic test
* Update comment
* Add test for --height
* No cast
* Add check that genesis file exists
* Remove duplicate imports
* Fail early
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Jack Zampolin <jack.zampolin@gmail.com>
Co-authored-by: Cory <cjlevinson@gmail.com>
* Replaced hardcoded 'cosmos' and 'cosmosvaloper' help references
* Ran make format target
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* create base proto package, move ValAddresses to staking
* move tx, crypto, query proto packages into base; mark module level packages as beta
* new cosmos/base/abci.proto file, refactor some message names
* add comments to TxResponse
* refactor RegisterInterface calls to new proto package names
* fix bug in merge commit
* move missing module proto packages to beta, move cosmos.kv to cosmos.base.kv
* add tx.proto files, holding proto messages for module Msgs
* rm old generated pb.go files
* cosmos/base/base.proto -> cosmos/base/coin.proto
* mark genutil proto package as beta
* Fix conflicts
* Restructure proto files
* Re put signing
* Fix test
* Update slashing genesis
* Ignore confio proto files
* Fix interface register
* Make proto-gen
* Fix lint
* Put tx service into tx.v1beta1
* Use v1beta1 in interface registry
* Remove to cosmos.base.store
* Remove extra confio in buf
Co-authored-by: Amaury Martiny <amaury.martiny@protonmail.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
Noticed during my audit, the code for cache.CommitKVStoreCacheManager.Reset()
discarded the prior on every invocation i.e.
m = make(map[T(key)]T(value))
However, this can be made fast and conserve memory by using the map clearing idiom
that the Go compiler recognizes i.e.
for key := range m {
delete(m, key)
}
and turns into very fast code, instead of the extraneous map discarding.
The speed up generated is:
```shell
$ benchstat before after
name old time/op new time/op delta
Reset-8 204ns ± 2% 66ns ± 9% -67.86% (p=0.000 n=20+20)
name old alloc/op new alloc/op delta
Reset-8 384B ± 0% 0B -100.00% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
Reset-8 3.00 ± 0% 0.00 -100.00% (p=0.000 n=20+20)
```
Fixes#6681
Noticed during an audit, we've got duplicated code that's unused.
There were 2 type definitions:
* type kvPair types.Pair
* type KVPair types.Pair
and each had a .bytes() and .Bytes() method respectively that
then used some extra code.
This change deletes the duplicated/unnecessary code but also
while here improves the performance by removing the use of a
bytes.Buffer which is unnecessary given that we are only
length prefixed the key, length prefixing the value hence
the various helpers are unnecessary.
The added benchmarks shows the performance boost
```shell
$ benchstat before after
name old time/op new time/op delta
KVPairBytes-8 146µs ± 1% 142µs ± 2% -3.05% (p=0.000 n=18+17)
name old speed new speed delta
KVPairBytes-8 6.84GB/s ± 1% 7.06GB/s ± 2% +3.15% (p=0.000 n=18+17)
name old alloc/op new alloc/op delta
KVPairBytes-8 1.01MB ± 0% 1.01MB ± 0% -0.04% (p=0.000 n=17+20)
name old allocs/op new allocs/op delta
KVPairBytes-8 6.00 ± 0% 1.00 ± 0% -83.33% (p=0.000 n=20+20)
```
Closes: #6688
* Add basic grpc query service for x/evidence
* Add grpc query test for AllEvidences query
* linting
* Add AnyUnpacker to query test helper and some var renaming
* Add test to check Evidence query result
* Update proto/cosmos/evidence/query.proto
Co-authored-by: Amaury Martiny <amaury.martiny@protonmail.com>
* Use table tests
* Use NewQueryEvidenceRequest in place of QueryEvidenceParams
* Remove ConvertEvidence
Co-authored-by: Amaury Martiny <amaury.martiny@protonmail.com>
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
* add ics23-tendermint to sdk
* fix errors and prealloc
* move ics-23 to internal
* minor changes and move maps to internal
* remvoe some usage of tendermint merkle
* fix linter
* dont use named returns
* move internal to store/rootmulti
* fix imports
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
* cleanup using CommitmentProof.Calculate()
* get correct version of confio/ics23
* Update store/types/proof.go
Co-authored-by: colin axner <25233464+colin-axner@users.noreply.github.com>
* revert review commit
Co-authored-by: colin axner <25233464+colin-axner@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* switch iavl store to use ics proof
* fix proofs to return for correct height
* appease linter
* Register commitment op correctly
* Make CommitmentOp generic over all ics23 specs (#6331)
* Make the CommitmentOp generic over all ics23 ProofSpecs, using Type to distinguish
* Register SimpleMerkle ics23 proof op as well
* Addressed linter issues
* move commitment proof to types
* Apply suggestions from code review
Co-authored-by: colin axner <25233464+colin-axner@users.noreply.github.com>
* address review comments:
* Apply suggestions from code review
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
* allow proofs against empty store
* address review comments
* Update store/types/proof.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* add changelog
Co-authored-by: Ethan Frey <ethanfrey@users.noreply.github.com>
Co-authored-by: colin axner <25233464+colin-axner@users.noreply.github.com>
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Marko <marbar3778@yahoo.com>
Ian Norden
Emmanuel T Odeke
Albert Chon
Marko
Robert Zaremba
Alessio Treglia
Riccardo Montagnin
Aaron Craelius
Anil Kumar Kammari
Aleksandr Bezobchuk
vincent
Cory
Erik Grinaker
Amaury Martiny
dependabot[bot]
Neeraj Murarka
billy rennekamp
Adam Bozanich
Marie
Aditya
Ethan Frey