Simulation docs (#5033)
* simulation docs * update docs with the latest simulation changes * minor imporvments * clean up of simulation.md * expand section on weights * minor reword * minor wording fix Co-authored-by: Marko <marbar3778@yahoo.com>
This commit is contained in:
Federico Kunze
GitHub
parent
6890feb3d2
commit
c7a6299eb2
|
|
@ -0,0 +1,123 @@
|
|||
# Module Simulation
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* [Cosmos Blockchain Simulator](./../using-the-sdk/simulation.md)
|
||||
|
||||
## Synopsis
|
||||
|
||||
This document details how to define each module simulation functions to be
|
||||
integrated with the application `SimulationManager`.
|
||||
|
||||
* [Simulation package](#simulation-package)
|
||||
* [Store decoders](#store-decoders)
|
||||
* [Randomized genesis](#randomized-genesis)
|
||||
* [Randomized parameters](#randomized-parameters)
|
||||
* [Random weighted operations](#random-weighted-operations)
|
||||
* [Random proposal contents](#random-proposal-contents)
|
||||
* [Registering the module simulation functions](#registering-simulation-functions)
|
||||
* [App simulator manager](#app-simulator-manager)
|
||||
* [Simulation tests](#simulation-tests)
|
||||
|
||||
## Simulation package
|
||||
|
||||
Every module that implements the SDK simulator needs to have a `x/<module>/simulation`
|
||||
package which contains the primary functions required by the fuzz tests: store
|
||||
decoders, randomized genesis state and parameters, weighted operations and proposal
|
||||
contents.
|
||||
|
||||
### Store decoders
|
||||
|
||||
Registering the store decoders is required for the `AppImportExport`. This allows
|
||||
for the key-value pairs from the stores to be decoded (_i.e_ unmarshalled)
|
||||
to their corresponding types. In particular, it matches the key to a concrete type
|
||||
and then unmarshals the value from the `KVPair` to the type provided.
|
||||
|
||||
You can use the example [here](https://github.com/cosmos/cosmos-sdk/blob/release%2Fv0.38.0/x/distribution/simulation/decoder.go) from the distribution module to implement your store decoders.
|
||||
|
||||
### Randomized genesis
|
||||
|
||||
The simulator tests different scenarios and values for genesis parameters
|
||||
in order to fully test the edge cases of specific modules. The `simulator` package from each module must expose a `RandomizedGenState` function to generate the initial random `GenesisState` from a given seed. In
|
||||
|
||||
Once the module genesis parameter are generated randomly (or with the key and
|
||||
values defined in a `params` file), they are marshaled to JSON format and added
|
||||
to the app genesis JSON to use it on the simulations.
|
||||
|
||||
You can check an example on how to create the randomized genesis [here](https://github.com/cosmos/cosmos-sdk/blob/release%2Fv0.38.0/x/staking/simulation/genesis.go).
|
||||
|
||||
### Randomized parameter changes
|
||||
|
||||
The simulator is able to test parameter changes at random. The simulator package from each module must contain a `RandomizedParams` func that will simulate parameter changes of the module throughout the simulations lifespan.
|
||||
|
||||
You can see how an example of what is needed to fully test parameter changes [here](https://github.com/cosmos/cosmos-sdk/blob/release%2Fv0.38.0/x/staking/simulation/params.go)
|
||||
|
||||
### Random weighted operations
|
||||
|
||||
Operations are one of the crucial parts of the SDK simulation. They are the transactions
|
||||
(`Msg`) that are simulated with random field values. The sender of the operation
|
||||
is also assigned randomly.
|
||||
|
||||
Operations on the simulation are simulated using the full [transaction cycle](../core/transactions.md) of a
|
||||
`ABCI` application that exposes the `BaseApp`.
|
||||
|
||||
Shown below is how weights are set:
|
||||
|
||||
+++ https://github.com/cosmos/cosmos-sdk/blob/release%2Fv0.38.0/x/staking/simulation/operations.go#L18-L92
|
||||
|
||||
As you can see the weights are predefined in this case but there are options on how to override this behavior with different weights. One is allowing `*rand.Rand` to define a random weight for the operation, or you can inject your own predefined weights.
|
||||
|
||||
Here is how one can override the above package `simappparams`.
|
||||
|
||||
+++ https://github.com/cosmos/gaia/blob/master/sims.mk#L9-L22
|
||||
|
||||
For the last test a tool called runsim <!-- # TODO: add link to runsim readme when its created --> is used, this is used to parallelize go test instances, provide info to Github and slack integrations to provide information to your team on how the simulations are running.
|
||||
|
||||
### Random proposal contents
|
||||
|
||||
Randomized governance proposals are also supported on the SDK simulator. Each
|
||||
module must define the governance proposal `Content`s that they expose and register
|
||||
them to be used on the parameters.
|
||||
|
||||
## Registering simulation functions
|
||||
|
||||
Now that all the required functions are defined, we need to integrate them into the module pattern within the `module.go`:
|
||||
|
||||
+++ https://github.com/cosmos/cosmos-sdk/blob/release%2Fv0.38.0/x/distribution/module.go#L156-L185
|
||||
|
||||
## App Simulator manager
|
||||
|
||||
The following step is setting up the `SimulatorManager` at the app level. This
|
||||
is required for the simulation test files on the next step.
|
||||
|
||||
```go
|
||||
type CustomApp struct {
|
||||
...
|
||||
sm *module.SimulationManager
|
||||
}
|
||||
```
|
||||
|
||||
Then at the instantiation of the application, we create the `SimulationManager`
|
||||
instance in the same way we create the `ModuleManager` but this time we only pass
|
||||
the modules that implement the simulation functions from the `AppModuleSimulation`
|
||||
interface described above.
|
||||
|
||||
```go
|
||||
func NewCustomApp(...) {
|
||||
// create the simulation manager and define the order of the modules for deterministic simulations
|
||||
app.sm = module.NewSimulationManager(
|
||||
auth.NewAppModule(app.accountKeeper),
|
||||
bank.NewAppModule(app.bankKeeper, app.accountKeeper),
|
||||
supply.NewAppModule(app.supplyKeeper, app.accountKeeper),
|
||||
ov.NewAppModule(app.govKeeper, app.accountKeeper, app.supplyKeeper),
|
||||
mint.NewAppModule(app.mintKeeper),
|
||||
distr.NewAppModule(app.distrKeeper, app.accountKeeper, app.supplyKeeper, app.stakingKeeper),
|
||||
staking.NewAppModule(app.stakingKeeper, app.accountKeeper, app.supplyKeeper),
|
||||
slashing.NewAppModule(app.slashingKeeper, app.accountKeeper, app.stakingKeeper),
|
||||
)
|
||||
|
||||
// register the store decoders for simulation tests
|
||||
app.sm.RegisterStoreDecoders()
|
||||
...
|
||||
}
|
||||
```
|
||||
|
|
@ -37,13 +37,18 @@ x/{module}
|
|||
│ ├── params.go
|
||||
│ ├── ...
|
||||
│ └── querier.go
|
||||
├── simulation
|
||||
│ ├── decoder.go
|
||||
│ ├── genesis.go
|
||||
│ ├── operations.go
|
||||
│ ├── params.go
|
||||
│ └── proposals.go
|
||||
├── abci.go
|
||||
├── alias.go
|
||||
├── genesis.go
|
||||
├── handler.go
|
||||
├── module.go
|
||||
├── ...
|
||||
└── simulation.go
|
||||
└── module.go
|
||||
```
|
||||
|
||||
- `abci.go`: The module's `BeginBlocker` and `EndBlocker` implementations (if any).
|
||||
|
|
@ -53,7 +58,7 @@ there is nothing preventing developers from importing other packages from the mo
|
|||
(excluding`internal/`) but it is recommended that `alias.go` have everything
|
||||
exposed that other modules may need. The majority of the exported values here will
|
||||
typically come from `internal/` (see below).
|
||||
- `client/`: The module's CLI and REST client functionality implementation and
|
||||
- `client/`: The module's CLI and REST client functionality implementation and
|
||||
testing.
|
||||
- `exported/`: The module's exported types -- typically type interfaces. If a module
|
||||
relies on other module keepers, it is expected to receive them as interface
|
||||
|
|
@ -79,8 +84,10 @@ allows for greater freedom of development while maintaining API stability.
|
|||
implementations such as the querier and invariants.
|
||||
- `module.go`: The module's implementation of the `AppModule` and `AppModuleBasic`
|
||||
interfaces.
|
||||
- `simulation.go`: The module's simulation messages and related types (if any).
|
||||
- `simulation/`: The module's simulation package defines all the required functions
|
||||
used on the blockchain simulator: randomized genesis state, parameters, weigthed
|
||||
operations, proposal contents and types decoders.
|
||||
|
||||
## Next {hide}
|
||||
|
||||
Learn about [interfaces](../interfaces/interfaces-intro.md) {hide}
|
||||
Learn about [interfaces](../interfaces/interfaces-intro.md) {hide}
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -19,4 +19,4 @@
|
|||
"vuepress-plugin-smooth-scroll": "0.0.9",
|
||||
"vuepress-theme-cosmos": "^1.0.113"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -7,4 +7,3 @@ parent:
|
|||
|
||||
- [Modules](../../x/README.md)
|
||||
- [Simulation](./simulation.md)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,49 +1,65 @@
|
|||
# Cosmos Blockchain Simulator
|
||||
|
||||
The Cosmos SDK offers a full fledged simulation framework to fuzz test every message defined by a module.
|
||||
The Cosmos SDK offers a full fledged simulation framework to fuzz test every
|
||||
message defined by a module.
|
||||
|
||||
This functionality is provided by the[`SimApp`](https://github.com/cosmos/cosmos-sdk/blob/master/simapp/app.go),
|
||||
which is a dummy application that is used for running the [`simulation`](https://github.com/cosmos/cosmos-sdk/tree/master/x/simulation) module.
|
||||
This module defines all the simulation logic as well as the operations for randomized parameters like accounts, balances etc.
|
||||
On the SDK, this functionality is provided by the[`SimApp`](https://github.com/cosmos/cosmos-sdk/blob/master/simapp/app.go), which is a
|
||||
`Baseapp` application that is used for running the [`simulation`](https://github.com/cosmos/cosmos-sdk/tree/master/x/simulation) module.
|
||||
This module defines all the simulation logic as well as the operations for
|
||||
randomized parameters like accounts, balances etc.
|
||||
|
||||
## Goals
|
||||
|
||||
The blockchain simulator tests how the blockchain application would behave under real life circumstances by generating and sending randomized messages.
|
||||
The goal of this is to detect and debug failures that could halt a live chain, by providing logs and statistics about the operations run by the simulator as well as exporting the latest application state when a failure was found.
|
||||
The blockchain simulator tests how the blockchain application would behave under
|
||||
real life circumstances by generating and sending randomized messages.
|
||||
The goal of this is to detect and debug failures that could halt a live chain,
|
||||
by providing logs and statistics about the operations run by the simulator as
|
||||
well as exporting the latest application state when a failure was found.
|
||||
|
||||
Its main difference with integration testing is that the simulator app allows you to pass parameters to customize the chain that's being simulated.
|
||||
This comes in handy when trying to reproduce bugs that were generated in the provided operations (randomized or not).
|
||||
Its main difference with integration testing is that the simulator app allows
|
||||
you to pass parameters to customize the chain that's being simulated.
|
||||
This comes in handy when trying to reproduce bugs that were generated in the
|
||||
provided operations (randomized or not).
|
||||
|
||||
## Simulation commands
|
||||
|
||||
The simulation app has different commands, each of which tests a different failure type:
|
||||
The simulation app has different commands, each of which tests a different
|
||||
failure type:
|
||||
|
||||
- `AppImportExport`: The simulator exports the initial app state and then it creates a new app with the exported `genesis.json` as an input, checking for inconsistencies between the stores.
|
||||
- `AppSimulationAfterImport`: Queues two simulations together. The first one provides the app state (_i.e_ genesis) to the second. Useful to test software upgrades or hard-forks from a live chain.
|
||||
- `AppStateDeterminism`: Checks that all the nodes return the same values, in the same order.
|
||||
- `BenchmarkInvariants`: Analyses the performance of running all the modules' invariants (_i.e_ secuentially runs a [benchmark](https://golang.org/pkg/testing/#hdr-Benchmarks) test). An invariant checks for differences between the values that are on the store and the passive tracker. Eg: total coins held by accounts vs total supply tracker.
|
||||
- `FullAppSimulation`: General simulation mode. Runs the chain and the specified operations for a given number of blocks. Tests that there're no `panics` on the simulation. It does also run invariant checks on every `Period` but they are not benchmarked.
|
||||
* `AppImportExport`: The simulator exports the initial app state and then it
|
||||
creates a new app with the exported `genesis.json` as an input, checking for
|
||||
inconsistencies between the stores.
|
||||
* `AppSimulationAfterImport`: Queues two simulations together. The first one provides the app state (_i.e_ genesis) to the second. Useful to test software upgrades or hard-forks from a live chain.
|
||||
* `AppStateDeterminism`: Checks that all the nodes return the same values, in the same order.
|
||||
* `BenchmarkInvariants`: Analysis of the performance of running all modules' invariants (_i.e_ sequentially runs a [benchmark](https://golang.org/pkg/testing/#hdr-Benchmarks) test). An invariant checks for
|
||||
differences between the values that are on the store and the passive tracker. Eg: total coins held by accounts vs total supply tracker.
|
||||
* `FullAppSimulation`: General simulation mode. Runs the chain and the specified operations for a given number of blocks. Tests that there're no `panics` on the simulation. It does also run invariant checks on every `Period` but they are not benchmarked.
|
||||
|
||||
Each simulation must receive a set of inputs (_i.e_ flags) such as the number of blocks that the simulation is run, seed, block size, etc.
|
||||
Each simulation must receive a set of inputs (_i.e_ flags) such as the number of
|
||||
blocks that the simulation is run, seed, block size, etc.
|
||||
Check the full list of flags [here](https://github.com/cosmos/cosmos-sdk/blob/adf6ddd4a807c8363e33083a3281f6a5e112ab89/simapp/sim_test.go#L34-L50).
|
||||
|
||||
## Simulator Modes
|
||||
|
||||
In addition to the various inputs and commands, the simulator runs in three modes:
|
||||
|
||||
1. Completely random where the initial state, module parameters and simulation parameters are **pseudo-randomly generated**.
|
||||
1. Completely random where the initial state, module parameters and simulation
|
||||
parameters are **pseudo-randomly generated**.
|
||||
2. From a `genesis.json` file where the initial state and the module parameters are defined.
|
||||
This mode is helpful for running simulations on a known state such as a live network export where a new (mostly likely breaking) version of the application needs to be tested.
|
||||
3. From a `params.json` file where the initial state is pseudo-randomly generated but the module and simulation parameters can be provided manually.
|
||||
This allows for a more controlled and deterministic simulation setup while allowing the state space to still be pseudo-randomly simulated. The list of available parameters is listed [here](https://github.com/cosmos/cosmos-sdk/blob/adf6ddd4a807c8363e33083a3281f6a5e112ab89/x/simulation/params.go#L170-L178).
|
||||
This allows for a more controlled and deterministic simulation setup while allowing the state space to still be pseudo-randomly simulated.
|
||||
The list of available parameters are listed [here](https://github.com/cosmos/cosmos-sdk/blob/adf6ddd4a807c8363e33083a3281f6a5e112ab89/x/simulation/params.go#L170-L178).
|
||||
|
||||
::: tip
|
||||
These modes are not mutually exclusive. So you can for example run a randomly generated genesis state (`1`) with manually generated simulation params (`3`).
|
||||
These modes are not mutually exclusive. So you can for example run a randomly
|
||||
generated genesis state (`1`) with manually generated simulation params (`3`).
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
This is a general example of how simulations are run. For more specific examples check the SDK [Makefile](https://github.com/cosmos/cosmos-sdk/blob/adf6ddd4a807c8363e33083a3281f6a5e112ab89/Makefile#L88-L123).
|
||||
This is a general example of how simulations are run. For more specific examples
|
||||
check the SDK [Makefile](https://github.com/cosmos/cosmos-sdk/blob/adf6ddd4a807c8363e33083a3281f6a5e112ab89/Makefile#L88-L123).
|
||||
|
||||
```bash
|
||||
$ go test -mod=readonly github.com/cosmos/cosmos-sdk/simapp \
|
||||
|
|
@ -56,15 +72,26 @@ This is a general example of how simulations are run. For more specific examples
|
|||
|
||||
Here are some suggestions when encountering a simulation failure:
|
||||
|
||||
- Export the app state at the height were the failure was found. You can do this by passing the `-ExportStatePath` flag to the simulator.
|
||||
- Use `-Verbose` logs. They could give you a better hint on all the operations involved.
|
||||
* Export the app state at the height were the failure was found. You can do this
|
||||
by passing the `-ExportStatePath` flag to the simulator.
|
||||
* Use `-Verbose` logs. They could give you a better hint on all the operations
|
||||
involved.
|
||||
* Reduce the simulation `-Period`. This will run the invariants checks more
|
||||
frequently.
|
||||
* Print all the failed invariants at once with `-PrintAllInvariants`.
|
||||
* Try using another `-Seed`. If it can reproduce the same error and if it fails
|
||||
sooner you will spend less time running the simulations.
|
||||
* Reduce the `-NumBlocks` . How's the app state at the height previous to the
|
||||
failure?
|
||||
* Run invariants on every operation with `-SimulateEveryOperation`. _Note_: this
|
||||
will slow down your simulation **a lot**.
|
||||
* Try adding logs to operations that are not logged. You will have to define a
|
||||
[Logger](https://github.com/cosmos/cosmos-sdk/blob/adf6ddd4a807c8363e33083a3281f6a5e112ab89/x/staking/keeper/keeper.go#L65:17) on your `Keeper`.
|
||||
|
||||
- Reduce the simulation `-Period`. This will run the invariants checks more frequently.
|
||||
- Print all the failed invariants at once with `-PrintAllInvariants`.
|
||||
- Try using another `-Seed`. If it can reproduce the same error and if it fails sooner you will spend less time running the simulations.
|
||||
- Reduce the `-NumBlocks` . How's the app state at the height previous to the failure?
|
||||
- Run invariants on every operation with `-SimulateEveryOperation`. _Note_: this will slow down your simulation **a lot**.
|
||||
- Try adding logs to operations that are not logged. You will have to define a [Logger](https://github.com/cosmos/cosmos-sdk/blob/adf6ddd4a807c8363e33083a3281f6a5e112ab89/x/staking/keeper/keeper.go#L65:17) on your `Keeper`.
|
||||
## Use simulation in your SDK-based application
|
||||
|
||||
<!-- ## Use simulation in your SDK-based application -->
|
||||
<!-- TODO: link to the simulation section on the tutorial for how to add your own simulation messages -->
|
||||
Learn how you can integrate the simulation into your SDK-based application:
|
||||
|
||||
* Application Simulation Manager
|
||||
* [Building modules: Simulator](../building-modules/simulator.md)
|
||||
* Simulator tests
|
||||
|
|
|
|||
Loading…
Reference in New Issue