1326fa2a7d
<!-- < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < ☺ v ✰ Thanks for creating a PR! ✰ v Before smashing the submit button please review the checkboxes. v If a checkbox is n/a - please still include it but + a little note why ☺ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --> ## Description <!-- Add a description of the changes that this PR introduces and the files that are the most critical to review. --> Hello 👋 this PR introduces the second stage of changes to support [ADR-038](https://github.com/cosmos/cosmos-sdk/pull/8012) state listening. This is rebased on top of the [first segment](https://github.com/cosmos/cosmos-sdk/pull/8551), which introduces the low level changes to the MultiStore and KVStore interfaces and implementations, the new WriteListener types, and the new listen.KVStore type. In this segment we introduce the StreamingService interface, an implementation that writes out to files, and it's integration and configuration at the BaseApp level. The idea was to have the first segment reviewed independently first but if people think it is easier/more time efficient to review both at the same time then we could start here. Thanks! This work is towards satisfying [ADR-038](https://github.com/cosmos/cosmos-sdk/blob/master/docs/architecture/adr-038-state-listening.md) --- Before we can merge this PR, please make sure that all the following items have been checked off. If any of the checklist items are not applicable, please leave them but write a little note why. - [x] Targeted PR against correct branch (see [CONTRIBUTING.md](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] Linked to Github issue with discussion and accepted design OR link to spec that describes this work. - [x] Code follows the [module structure standards](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules/structure.md). - [x] Wrote unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] Updated relevant documentation (`docs/`) or specification (`x/<module>/spec/`) - [x] Added relevant `godoc` [comments](https://blog.golang.org/godoc-documenting-go-code). - [x] Added a relevant changelog entry to the `Unreleased` section in `CHANGELOG.md` - [x] Re-reviewed `Files changed` in the Github PR explorer - [x] Review `Codecov Report` in the comment section below once CI passes |
||
---|---|---|
.. | ||
file | ||
README.md | ||
constructor.go | ||
constructor_test.go |
README.md
State Streaming Service
This package contains the constructors for the StreamingService
s used to write state changes out from individual KVStores to a
file or stream, as described in ADR-038 and defined in types/streaming.go.
The child directories contain the implementations for specific output destinations.
Currently, a StreamingService
implementation that writes state changes out to files is supported, in the future support for additional
output destinations can be added.
The StreamingService
is configured from within an App using the AppOptions
loaded from the app.toml file:
[store]
streamers = [ # if len(streamers) > 0 we are streaming
"file", # name of the streaming service, used by constructor
]
[streamers]
[streamers.file]
keys = ["list", "of", "store", "keys", "we", "want", "to", "expose", "for", "this", "streaming", "service"]
write_dir = "path to the write directory"
prefix = "optional prefix to prepend to the generated file names"
store.streamers
contains a list of the names of the StreamingService
implementations to employ which are used by ServiceTypeFromString
to return the ServiceConstructor
for that particular implementation:
listeners := cast.ToStringSlice(appOpts.Get("store.streamers"))
for _, listenerName := range listeners {
constructor, err := ServiceTypeFromString(listenerName)
if err != nil {
// handle error
}
}
streamers
contains a mapping of the specific StreamingService
implementation name to the configuration parameters for that specific service.
streamers.x.keys
contains the list of StoreKey
names for the KVStores to expose using this service and is required by every type of StreamingService
.
In order to expose all KVStores, we can include *
in this list. An empty list is equivalent to turning the service off.
Additional configuration parameters are optional and specific to the implementation.
In the case of the file streaming service, streamers.file.write_dir
contains the path to the
directory to write the files to, and streamers.file.prefix
contains an optional prefix to prepend to the output files to prevent potential collisions
with other App StreamingService
output files.
The ServiceConstructor
accepts AppOptions
, the store keys collected using streamers.x.keys
, a BinaryMarshaller
and
returns a StreamingService
implementation. The AppOptions
are passed in to provide access to any implementation specific configuration options,
e.g. in the case of the file streaming service the streamers.file.write_dir
and streamers.file.prefix
.
streamingService, err := constructor(appOpts, exposeStoreKeys, appCodec)
if err != nil {
// handler error
}
The returned StreamingService
is loaded into the BaseApp using the BaseApp's SetStreamingService
method.
The Stream
method is called on the service to begin the streaming process. Depending on the implementation this process
may be synchronous or asynchronous with the message processing of the state machine.
bApp.SetStreamingService(streamingService)
wg := new(sync.WaitGroup)
quitChan := make(chan struct{})
streamingService.Stream(wg, quitChan)