cosmos-sdk/db/rocksdb/iterator.go

150 lines
2.8 KiB
Go
Raw Normal View History

feat: ADR-040: Add `RootStore` implementation (#10430) ## Description Part of: https://github.com/cosmos/cosmos-sdk/issues/10192 Introduces a new `RootStore` type in the `store/v2` package and an implementation, without yet replacing the `MultiStore` or refactoring its use within the SDK (which will happen in the follow up: https://github.com/cosmos/cosmos-sdk/pull/10174). Specified by [ADR-040](https://github.com/cosmos/cosmos-sdk/blob/1326fa2a7dfc3d83cf23dc1c1f33ff131152ad60/docs/architecture/adr-040-storage-and-smt-state-commitments.md). Fixes https://github.com/cosmos/cosmos-sdk/issues/10651 Fixes https://github.com/cosmos/cosmos-sdk/issues/10263 --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
2021-12-16 05:09:57 -08:00
//go:build rocksdb_build
feat: ADR-040: Implement RocksDB backend (#9851) ## Description Partially resolves: https://github.com/vulcanize/cosmos-sdk/issues/14 Implements a [RocksDB](https://github.com/facebook/rocksdb)-based backend for the DB interface introduced by https://github.com/cosmos/cosmos-sdk/pull/9573 and specified by [ADR-040](https://github.com/cosmos/cosmos-sdk/blob/eb7d939f86c6cd7b4218492364cdda3f649f06b5/docs/architecture/adr-040-storage-and-smt-state-commitments.md). * Historical versioning is implemented with [Checkpoints](https://github.com/facebook/rocksdb/wiki/Checkpoints). * Uses `OptimisticTransactionDB` to allow concurrent transactions with write conflict detection. This depends on some additional CGo bindings - see https://github.com/tecbot/gorocksdb/pull/216, https://github.com/facebook/rocksdb/pull/8526. We'll need to replace the `gorocksdb` module until these are upstream. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - n/a - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
2021-10-05 08:39:55 -07:00
package rocksdb
import (
"bytes"
feat: use cosmos/rocksdb rather than replace trick (#10927) ## Description A follow up to https://github.com/cosmos/gorocksdb/pull/4 (see description there). Also: * remove import aliasing in rocksdb package. Closes: #10915 TODO: - [x] merge https://github.com/cosmos/gorocksdb/pull/4 - [x] tag new release in https://github.com/cosmos/gorocksdb - [x] Update this PR to include new release - [ ] tag `db/v1.0.0` (or `db/v0.1.0)` --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
2022-01-11 14:29:53 -08:00
"github.com/cosmos/cosmos-sdk/db"
"github.com/cosmos/gorocksdb"
feat: ADR-040: Implement RocksDB backend (#9851) ## Description Partially resolves: https://github.com/vulcanize/cosmos-sdk/issues/14 Implements a [RocksDB](https://github.com/facebook/rocksdb)-based backend for the DB interface introduced by https://github.com/cosmos/cosmos-sdk/pull/9573 and specified by [ADR-040](https://github.com/cosmos/cosmos-sdk/blob/eb7d939f86c6cd7b4218492364cdda3f649f06b5/docs/architecture/adr-040-storage-and-smt-state-commitments.md). * Historical versioning is implemented with [Checkpoints](https://github.com/facebook/rocksdb/wiki/Checkpoints). * Uses `OptimisticTransactionDB` to allow concurrent transactions with write conflict detection. This depends on some additional CGo bindings - see https://github.com/tecbot/gorocksdb/pull/216, https://github.com/facebook/rocksdb/pull/8526. We'll need to replace the `gorocksdb` module until these are upstream. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - n/a - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
2021-10-05 08:39:55 -07:00
)
type rocksDBIterator struct {
source *gorocksdb.Iterator
start, end []byte
isReverse bool
isInvalid bool
// Whether iterator has been advanced to the first element (is fully initialized)
primed bool
}
feat: use cosmos/rocksdb rather than replace trick (#10927) ## Description A follow up to https://github.com/cosmos/gorocksdb/pull/4 (see description there). Also: * remove import aliasing in rocksdb package. Closes: #10915 TODO: - [x] merge https://github.com/cosmos/gorocksdb/pull/4 - [x] tag new release in https://github.com/cosmos/gorocksdb - [x] Update this PR to include new release - [ ] tag `db/v1.0.0` (or `db/v0.1.0)` --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
2022-01-11 14:29:53 -08:00
var _ db.Iterator = (*rocksDBIterator)(nil)
feat: ADR-040: Implement RocksDB backend (#9851) ## Description Partially resolves: https://github.com/vulcanize/cosmos-sdk/issues/14 Implements a [RocksDB](https://github.com/facebook/rocksdb)-based backend for the DB interface introduced by https://github.com/cosmos/cosmos-sdk/pull/9573 and specified by [ADR-040](https://github.com/cosmos/cosmos-sdk/blob/eb7d939f86c6cd7b4218492364cdda3f649f06b5/docs/architecture/adr-040-storage-and-smt-state-commitments.md). * Historical versioning is implemented with [Checkpoints](https://github.com/facebook/rocksdb/wiki/Checkpoints). * Uses `OptimisticTransactionDB` to allow concurrent transactions with write conflict detection. This depends on some additional CGo bindings - see https://github.com/tecbot/gorocksdb/pull/216, https://github.com/facebook/rocksdb/pull/8526. We'll need to replace the `gorocksdb` module until these are upstream. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - n/a - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
2021-10-05 08:39:55 -07:00
func newRocksDBIterator(source *gorocksdb.Iterator, start, end []byte, isReverse bool) *rocksDBIterator {
if isReverse {
if end == nil {
source.SeekToLast()
} else {
source.Seek(end)
if source.Valid() {
eoakey := moveSliceToBytes(source.Key()) // end or after key
if bytes.Compare(end, eoakey) <= 0 {
source.Prev()
}
} else {
source.SeekToLast()
}
}
} else {
if start == nil {
source.SeekToFirst()
} else {
source.Seek(start)
}
}
return &rocksDBIterator{
source: source,
start: start,
end: end,
isReverse: isReverse,
isInvalid: false,
primed: false,
}
}
// Domain implements Iterator.
func (itr *rocksDBIterator) Domain() ([]byte, []byte) {
return itr.start, itr.end
}
// Valid implements Iterator.
func (itr *rocksDBIterator) Valid() bool {
if !itr.primed {
return false
}
if itr.isInvalid {
return false
}
if !itr.source.Valid() {
itr.isInvalid = true
return false
}
var (
start = itr.start
end = itr.end
key = moveSliceToBytes(itr.source.Key())
)
// If key is end or past it, invalid.
if itr.isReverse {
if start != nil && bytes.Compare(key, start) < 0 {
itr.isInvalid = true
return false
}
} else {
if end != nil && bytes.Compare(key, end) >= 0 {
itr.isInvalid = true
return false
}
}
return true
}
// Key implements Iterator.
func (itr *rocksDBIterator) Key() []byte {
itr.assertIsValid()
return moveSliceToBytes(itr.source.Key())
}
// Value implements Iterator.
func (itr *rocksDBIterator) Value() []byte {
itr.assertIsValid()
return moveSliceToBytes(itr.source.Value())
}
// Next implements Iterator.
func (itr *rocksDBIterator) Next() bool {
if !itr.primed {
itr.primed = true
} else {
if itr.isReverse {
itr.source.Prev()
} else {
itr.source.Next()
}
}
return itr.Valid()
}
// Error implements Iterator.
func (itr *rocksDBIterator) Error() error {
return itr.source.Err()
}
// Close implements Iterator.
func (itr *rocksDBIterator) Close() error {
itr.source.Close()
return nil
}
func (itr *rocksDBIterator) assertIsValid() {
if !itr.Valid() {
panic("iterator is invalid")
}
}
// moveSliceToBytes will free the slice and copy out a go []byte
// This function can be applied on *Slice returned from Key() and Value()
// of an Iterator, because they are marked as freed.
func moveSliceToBytes(s *gorocksdb.Slice) []byte {
defer s.Free()
if !s.Exists() {
return nil
}
v := make([]byte, s.Size())
copy(v, s.Data())
return v
}