We can shave off some milliseconds, but also cut down some Megabytes of
RAM consumed by only requesting from the cache if needed, but also using
the map clearing idiom which is recognized by the compiler to make fast
code.
Noticed in profiles from Tharsis' Ethermint per https://github.com/tharsis/ethermint/issues/710
- Before
* Memory profiles
```shell
19.50MB 19.50MB 134: store.cache = make(map[string]*cValue)
18.50MB 18.50MB 135: store.deleted = make(map[string]struct{})
15.50MB 15.50MB 136: store.unsortedCache = make(map[string]struct{})
```
* CPU profiles
```go
. . 118: // TODO: Consider allowing usage of Batch, which would allow the write to
. . 119: // at least happen atomically.
150ms 150ms 120: for _, key := range keys {
220ms 3.64s 121: cacheValue := store.cache[key]
. . 122:
. . 123: switch {
. 250ms 124: case store.isDeleted(key):
. . 125: store.parent.Delete([]byte(key))
210ms 210ms 126: case cacheValue.value == nil:
. . 127: // Skip, it already doesn't exist in parent.
. . 128: default:
240ms 27.94s 129: store.parent.Set([]byte(key), cacheValue.value)
. . 130: }
. . 131: }
...
10ms 60ms 134: store.cache = make(map[string]*cValue)
. 40ms 135: store.deleted = make(map[string]struct{})
. 50ms 136: store.unsortedCache = make(map[string]struct{})
. 110ms 137: store.sortedCache = dbm.NewMemDB()
```
- After
* Memory profiles
```shell
. . 130: // Clear the cache using the map clearing idiom
. . 131: // and not allocating fresh objects.
. . 132: // Please see https://bencher.orijtech.com/perfclinic/mapclearing/
. . 133: for key := range store.cache {
. . 134: delete(store.cache, key)
. . 135: }
. . 136: for key := range store.deleted {
. . 137: delete(store.deleted, key)
. . 138: }
. . 139: for key := range store.unsortedCache {
. . 140: delete(store.unsortedCache, key)
. . 141: }
```
* CPU profiles
```shell
. . 111: // TODO: Consider allowing usage of Batch, which would allow the write to
. . 112: // at least happen atomically.
110ms 110ms 113: for _, key := range keys {
. 210ms 114: if store.isDeleted(key) {
. . 115: // We use []byte(key) instead of conv.UnsafeStrToBytes because we cannot
. . 116: // be sure if the underlying store might do a save with the byteslice or
. . 117: // not. Once we get confirmation that .Delete is guaranteed not to
. . 118: // save the byteslice, then we can assume only a read-only copy is sufficient.
. . 119: store.parent.Delete([]byte(key))
. . 120: continue
. . 121: }
. . 122:
50ms 2.45s 123: cacheValue := store.cache[key]
910ms 920ms 124: if cacheValue.value != nil {
. . 125: // It already exists in the parent, hence delete it.
120ms 29.56s 126: store.parent.Set([]byte(key), cacheValue.value)
. . 127: }
. . 128: }
. . 129:
. . 130: // Clear the cache using the map clearing idiom
. . 131: // and not allocating fresh objects.
. . 132: // Please see https://bencher.orijtech.com/perfclinic/mapclearing/
. 210ms 133: for key := range store.cache {
. . 134: delete(store.cache, key)
. . 135: }
. 10ms 136: for key := range store.deleted {
. . 137: delete(store.deleted, key)
. . 138: }
. 170ms 139: for key := range store.unsortedCache {
. . 140: delete(store.unsortedCache, key)
. . 141: }
. 260ms 142: store.sortedCache = dbm.NewMemDB()
. 10ms 143:}
```
Fixes#10487
Updates https://github.com/tharsis/ethermint/issues/710
This change takes the observation that previous dbm.IsKeyInDomain
which searches for [start, end) was performing too many byteslice
comparisons. Instead we start off by sorting all the values in the
store.unsortedCache, and then apply a modified binary search to
look for values that fall within the domain [start, end)
The procedure involves:
* iterating over all items to build a list of all keys -- O(n)
* invoking sort.Strings immediately, of which
we anyways eventually invoke sort.Slice(unsorted, ...) which uses
Quicksort -- O(nlog(n)) or O(n^2) worst case
* invoking modified binary search which is O(log(n)) * 2 ~ O(log(n))
to search for the [start, end) range indices
for a total approximate complexity of:
Best case: O(n) + O(n(log(n))) + O(log(n)) ~= O(nlog(n))
Worst case: O(n) + O(n^2) + O(log(n)) ~= O(n^2)
instead of previously:
* iterating over all the unsorted items and invoking dbm.IsKeyInDomain:
bytes.Compare ~ O(n) + O(n*s*e) where s -- len(start), e -- len(end)
for overall complexity of O(n*s*e)
* invoking sort.Slice(unsorted, ...) which uses
Quicksort -- O(nlog(n)) or O(n^2) worst case
for a total approximate complexity of:
Best case: O(n) + O(n*s*e) + O(nlog(n)) ~= O(n*s*e) ~ O(n^2)
Worst case: O(n) + O(n*s*e) + O(n^2) ~= O(n*s*e) ~ O(n^2)
Ordinarily we'd combine the n*s*e to be n*m, but really the comparisons
between (start & key, end & key) are profound that it makes sense to
keep them as factors. The overall benchmark results vindicate our choice
of isolating the factors (n*s*e)
The benchmarks show that as the number of keys to iterate grows, the
new code grows gracefully in a somewhat linear growth, notice for
CAcheKVStoreIterator*, when we go from:
* 1,000 to 10,000 keys: 120us->1,600us (13X) old vs 95us->900us (9.47X) new
* 50,000 to 100,000 keys: 19ms->100ms (5.3X) old vs 5.5ms->17ms (3X) new
```shell
time/op
GetValidator-8 5.8ms ± 2% 4.7ms ± 1% -17.69% (p=0.000 n=10+10)
OneBankSendTxPerBlock-8 3.2ms ± 2% 2.8ms ± 1% -10.80% (p=0.000 n=7+10)
OneBankMultiSendTxPerBlock-8 3.1ms ± 3% 2.9ms ± 2% -8.36% (p=0.000 n=10+10)
AccountMapperSetAccount-8 8.6µs ± 1% 7.8µs ± 1% -9.74% (p=0.000 n=10+10)
CacheKVStoreIterator500-8 64µs ± 6% 51µs ± 6% -19.22% (p=0.000 n=10+9)
CacheKVStoreIterator1000-8 0.12ms ± 4% 95µs ± 4% -19.55% (p=0.000 n=10+10)
CacheKVStoreIterator10000-8 1.6ms ± 4% 0.90ms ± 1% -42.11% (p=0.000 n=10+10)
CacheKVStoreIterator50000-8 19ms ± 5% 5.5ms ± 1% -71.35% (p=0.000 n=10+10)
CacheKVStoreIterator100000-8 0.10s ± 23% 17ms ± 7% -83.44% (p=0.000 n=10+10)
CacheKVStoreGetNoKeyFound-8 1.3µs ± 6% 0.90µs ± 3% -31.19% (p=0.000 n=9+9)
CacheKVStoreGetKeyFound-8 0.66µs ± 6% 0.56µs ± 2% -14.81% (p=0.000 n=10+9)
alloc/op
B/op
BlockProvision-8 0.11kB ± 0% 0.10kB ± 0% -7.14% (p=0.000 n=10+10)
CacheKVStoreIterator50000-8 0.89MB ± 6% 0.53MB ± 1% -40.85% (p=0.000 n=10+10)
CacheKVStoreIterator100000-8 6.3MB ± 23% 1.6MB ± 6% -74.17% (p=0.000 n=10+10)
CacheKVStoreGetNoKeyFound-8 0.26kB ± 0% 0.23kB ± 1% -11.53% (p=0.000 n=10+8)
allocs/op (count)
AccountMapperSetAccount-8 42 ± 0% 38 ± 0% -9.52% (p=0.000 n=10+10)
BlockProvision-8 6.0 ± 0% 5.0 ± 0% -16.67% (p=0.000 n=10+10)
CacheKVStoreIterator1000-8 14 ± 0% 13 ± 0% -7.14% (p=0.002 n=8+10)
CacheKVStoreIterator10000-8 0.15k ± 2% 76 ± 1% -49.00% (p=0.000 n=7+10)
CacheKVStoreIterator50000-8 8.9k ± 11% 2.0k ± 2% -77.60% (p=0.000 n=10+10)
CacheKVStoreIterator100000-8 0.10M ± 26% 13k ± 12% -86.89% (p=0.000 n=10+10)
CacheKVStoreGetNoKeyFound-8 5.0 ± 0% 4.0 ± 0% -20.00% (p=0.000 n=10+10)
```
Note: Purposefully using a commit off master that doesn't
include the buggy code that caused x/bank.BenchmarkOneBank* to fail
per issue https://github.com/cosmos/cosmos-sdk/issues/10023
Updates #9876
/cc @cuonglm @kirbyquerby
<!--
The default pull request template is for types feat, fix, or refactor.
For other templates, add one of the following parameters to the url:
- template=docs.md
- template=other.md
-->
## Description
Closes: #XXXX
<!-- Add a description of the changes that this PR introduces and the files that
are the most critical to review. -->
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [ ] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
<!--
The default pull request template is for types feat, fix, or refactor.
For other templates, add one of the following parameters to the url:
- template=docs.md
- template=other.md
-->
## Description
Followup from #10077
Closes: #XXXX
<!-- Add a description of the changes that this PR introduces and the files that
are the most critical to review. -->
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [x] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [x] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
## Description
Closes: #10072
Significantly speeds up all the GasKvStore and CacheKVStore operations. Now Get/Set for these stores no longer even appears on my Osmosis benchmarks, saving ~8% of those benchmark's times. (Only one of the internal methods for setCacheValue appear -- will try to separately get a PR for reducing those memory allocations if #10026 is merged)
Talked to @alexanderbez about this on Discord, and he seemed in agreement with the approach of removing telemetry from these stores.
This does technically change telemetry, but I don't know if this is / should be considered a breaking change?
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification - is there something I should be updating?
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
## Description
So we don't burn un-necessary CPU in case of dirty store/delete.
```
name old time/op new time/op delta
CacheKVStoreIterator500-8 23.3µs ± 2% 23.4µs ± 1% ~ (p=0.151 n=5+5)
CacheKVStoreIterator1000-8 47.0µs ± 2% 46.3µs ± 0% -1.50% (p=0.008 n=5+5)
CacheKVStoreIterator10000-8 462µs ± 3% 458µs ± 1% ~ (p=0.690 n=5+5)
CacheKVStoreIterator50000-8 2.63ms ± 5% 2.51ms ± 2% -4.63% (p=0.032 n=5+5)
CacheKVStoreIterator100000-8 8.09ms ±21% 6.98ms ± 4% ~ (p=0.151 n=5+5)
CacheKVStoreGetNoKeyFound-8 393ns ± 2% 397ns ± 2% ~ (p=0.421 n=5+5)
CacheKVStoreGetKeyFound-8 269ns ± 5% 268ns ± 3% ~ (p=1.000 n=5+5)
```
Fixes#9991
---
### Author Checklist
*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*
I have...
- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [ ] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed
### Reviewers Checklist
*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*
I have...
- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
* StoreKVPair protobuf message definition and generated go types
* store WriteListener
* update MultiStore, CacheWrap, CacheWrapper interfaces
* adjust KVStores to fit new CacheWrapper interface
* new ListenKVStore
* adjust multistores to fit new MultiStore interface and enable wrapping returned KVStores with the new ListenKVStore
* typo fixes in adr
* ListenKV Store test
* update server mock KVStore and MultiStore
* multistore unit test; fix multistore constructor
* update changelog
* fix bug identified in CI
* improve codecov, minor fixes/adjustments
* review fixes
* review updates; flip set to delete in KVStorePair, updated proto-docs from running 'make proto-gen'
Reduces CPU burn by using a typed List to avoid the expensive type
assertions from using an interface. This is the only option for now
until Go makes generics generally available.
The change brings time spent on the time assertion cummulatively to:
580ms down from 6.88s
Explanation:
The type assertions were showing up heavily in profiles:
* Before this commit
```shell
Total: 42.18s
ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator
in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go
14.01s 18.87s (flat, cum) 44.74% of Total
. . 17: items []*kv.Pair
. . 18: ascending bool
. . 19:}
. . 20:
. . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator {
. 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len())
. . 23:
. . 24: var entered bool
. . 25:
510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() {
6.85s 6.88s 27: item := e.Value.(*kv.Pair)
5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) {
120ms 120ms 29: if entered {
. . 30: break
. . 31: }
. . 32:
. . 33: continue
. . 34: }
. . 35:
820ms 980ms 36: itemsInDomain = append(itemsInDomain, item)
. . 37: entered = true
. . 38: }
. . 39:
. 1.21s 40: return &memIterator{
. . 41: start: start,
. . 42: end: end,
. . 43: items: itemsInDomain,
. . 44: ascending: ascending,
. . 45: }
```
and given that the list only uses that type, it is only right to lift the
code from container/list.List, and only modify Element.Value's type.
For emphasis, the code is basically just a retrofit of
container/list/list.go but with a typing, and we'll keep it as is
until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after
generics have landed.
* After this commit
```shell
Total: 45.25s
ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator
in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go
4.84s 6.77s (flat, cum) 14.96% of Total
. . 16: items []*kv.Pair
. . 17: ascending bool
. . 18:}
. . 19:
. . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator {
. 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len())
. . 22:
. . 23: var entered bool
. . 24:
60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() {
580ms 580ms 26: item := e.Value
3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) {
80ms 80ms 28: if entered {
. . 29: break
. . 30: }
. . 31:
. . 32: continue
. . 33: }
. . 34:
440ms 580ms 35: itemsInDomain = append(itemsInDomain, item)
. . 36: entered = true
. . 37: }
. . 38:
. 260ms 39: return &memIterator{
. . 40: start: start,
. . 41: end: end,
. . 42: items: itemsInDomain,
. . 43: ascending: ascending,
. . 44: }
```
Fixes#8810
After continuously profiling InitGensis with 100K accounts, it showed
pathologically slow code, that was the result of a couple of patterns:
* Unconditional and not always necessary map lookups
* O(n^2) sdk.AccAddressFromBech32 retrievals when the code is expensive,
during a quicksort
The remedy involved 4 parts:
* O(n) sdk.AccAddressFromBech32 invocations, down from O(n^2) in the quicksort
* Only doing map lookups when the domain key check has passed
* Using a black magic compiler technique of the map clearing idiom
* Zero allocation []byte<->string conversion
With 100K accounts, this brings InitGenesis down to ~6min, instead of
20+min, it reduces the sort code from ~7sec down to 50ms.
Also some simple benchmark reflect the change:
```shell
name old time/op new time/op delta
SanitizeBalances500-8 19.3ms ±10% 1.5ms ± 5% -92.46% (p=0.000 n=20+20)
SanitizeBalances1000-8 41.9ms ± 8% 3.0ms ±12% -92.92% (p=0.000 n=20+20)
name old alloc/op new alloc/op delta
SanitizeBalances500-8 9.05MB ± 6% 0.56MB ± 0% -93.76% (p=0.000 n=20+18)
SanitizeBalances1000-8 20.2MB ± 3% 1.1MB ± 0% -94.37% (p=0.000 n=20+19)
name old allocs/op new allocs/op delta
SanitizeBalances500-8 72.4k ± 6% 4.5k ± 0% -93.76% (p=0.000 n=20+20)
SanitizeBalances1000-8 162k ± 3% 9k ± 0% -94.40% (p=0.000 n=20+20)
```
The CPU profiles show the radical change as per
https://github.com/cosmos/cosmos-sdk/issues/7766#issuecomment-786671734
Later on, we shall do more profiling and fixes but for now this brings
down the run-time for InitGenesis.
Fixes#7766
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
* enable the wsl linter
Fix various wsl-related warnings.
x/ibc/04-channel/keeper/handshake.go: fix missing return statement in ChanOpenTry().
* goimports -w files
* remove unknown linter references
* run make format
* Revert "run make format"
This reverts commit f810b62b9e4993f08506663d4e5f2ec2228a9863.
* run make format
* Bump Tendermint version to v0.33.0
* Deprecate old cmn package with new packages
* Update update DB APIs
* More DB updates
* Bump IAVL to v0.13.0
* Handle error returned by iavl.NewMutableTree
* Fix some IAVL stuffs
* Update IAVL
* More updates
* Passing tests
* Fix unit tests
Co-authored-by: Jack Zampolin <jack.zampolin@gmail.com>
Emmanuel T Odeke
Dev Ojha
Cuong Manh Le
Ian Norden
Robert Zaremba
Marko
Adam Bozanich
Alexander Bezobchuk
Alessio Treglia
Joon