Merge pull request #1618 from tendermint/1494-production-notes
[docs] notes about running a production system
This commit is contained in:
commit
178e357d7f
|
@ -58,6 +58,7 @@ Tendermint 102
|
||||||
subscribing-to-events-via-websocket.rst
|
subscribing-to-events-via-websocket.rst
|
||||||
indexing-transactions.rst
|
indexing-transactions.rst
|
||||||
how-to-read-logs.rst
|
how-to-read-logs.rst
|
||||||
|
running-in-production.rst
|
||||||
|
|
||||||
Tendermint 201
|
Tendermint 201
|
||||||
--------------
|
--------------
|
||||||
|
|
|
@ -0,0 +1,203 @@
|
||||||
|
Running in production
|
||||||
|
=====================
|
||||||
|
|
||||||
|
Logging
|
||||||
|
-------
|
||||||
|
|
||||||
|
Default logging level (``main:info,state:info,*:``) should suffice for normal
|
||||||
|
operation mode. Read `this post
|
||||||
|
<https://blog.cosmos.network/one-of-the-exciting-new-features-in-0-10-0-release-is-smart-log-level-flag-e2506b4ab756>`__
|
||||||
|
for details on how to configure ``log_level`` config variable. Some of the
|
||||||
|
modules can be found `here <./how-to-read-logs.html#list-of-modules>`__. If
|
||||||
|
you're trying to debug Tendermint or asked to provide logs with debug logging
|
||||||
|
level, you can do so by running tendermint with ``--log_level="*:debug"``.
|
||||||
|
|
||||||
|
DOS Exposure and Mitigation
|
||||||
|
---------------------------
|
||||||
|
|
||||||
|
Validators are supposed to setup `Sentry Node Architecture
|
||||||
|
<https://blog.cosmos.network/tendermint-explained-bringing-bft-based-pos-to-the-public-blockchain-domain-f22e274a0fdb>`__
|
||||||
|
to prevent Denial-of-service attacks. You can read more about it `here
|
||||||
|
<https://github.com/tendermint/aib-data/blob/develop/medium/TendermintBFT.md>`__.
|
||||||
|
|
||||||
|
P2P
|
||||||
|
~~~
|
||||||
|
|
||||||
|
The core of the Tendermint peer-to-peer system is ``MConnection``. Each
|
||||||
|
connection has ``MaxPacketMsgPayloadSize``, which is the maximum packet size
|
||||||
|
and bounded send & receive queues. One can impose restrictions on send &
|
||||||
|
receive rate per connection (``SendRate``, ``RecvRate``).
|
||||||
|
|
||||||
|
RPC
|
||||||
|
~~~
|
||||||
|
|
||||||
|
Endpoints returning multiple entries are limited by default to return 30
|
||||||
|
elements (100 max).
|
||||||
|
|
||||||
|
Rate-limiting and authentication are another key aspects to help protect
|
||||||
|
against DOS attacks. While in the future we may implement these features, for
|
||||||
|
now, validators are supposed to use external tools like `NGINX
|
||||||
|
<https://www.nginx.com/blog/rate-limiting-nginx/>`__ or `traefik
|
||||||
|
<https://docs.traefik.io/configuration/commons/#rate-limiting>`__ to achieve
|
||||||
|
the same things.
|
||||||
|
|
||||||
|
Debugging Tendermint
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
If you ever have to debug Tendermint, the first thing you should probably do is
|
||||||
|
to check out the logs. See `"How to read logs" <./how-to-read-logs.html>`__,
|
||||||
|
where we explain what certain log statements mean.
|
||||||
|
|
||||||
|
If, after skimming through the logs, things are not clear still, the second
|
||||||
|
TODO is to query the `/status` RPC endpoint. It provides the necessary info:
|
||||||
|
whenever the node is syncing or not, what height it is on, etc.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ curl http(s)://{ip}:{rpcPort}/status
|
||||||
|
```
|
||||||
|
|
||||||
|
`/dump_consensus_state` will give you a detailed overview of the consensus
|
||||||
|
state (proposer, lastest validators, peers states). From it, you should be able
|
||||||
|
to figure out why, for example, the network had halted.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ curl http(s)://{ip}:{rpcPort}/dump_consensus_state
|
||||||
|
```
|
||||||
|
|
||||||
|
There is a reduced version of this endpoint - `/consensus_state`, which
|
||||||
|
returns just the votes seen at the current height.
|
||||||
|
|
||||||
|
- `Github Issues <https://github.com/tendermint/tendermint/issues>`__
|
||||||
|
- `StackOverflow questions <https://stackoverflow.com/questions/tagged/tendermint>`__
|
||||||
|
|
||||||
|
Monitoring Tendermint
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
Each Tendermint instance has a standard `/health` RPC endpoint, which responds
|
||||||
|
with 200 (OK) if everything is fine and 500 (or no response) - if something is
|
||||||
|
wrong.
|
||||||
|
|
||||||
|
Other useful endpoints include mentioned earlier `/status`, `/net_info` and
|
||||||
|
`/validators`.
|
||||||
|
|
||||||
|
We have a small tool, called tm-monitor, which outputs information from the
|
||||||
|
endpoints above plus some statistics. The tool can be found `here
|
||||||
|
<https://github.com/tendermint/tools/tree/master/tm-monitor>`__.
|
||||||
|
|
||||||
|
What happens when my app dies?
|
||||||
|
------------------------------
|
||||||
|
|
||||||
|
You are supposed to run Tendermint under a `process supervisor
|
||||||
|
<https://en.wikipedia.org/wiki/Process_supervision>`__ (like systemd or runit).
|
||||||
|
It will ensure Tendermint is always running (despite possible errors).
|
||||||
|
|
||||||
|
Getting back to the original question, if your application dies, Tendermint
|
||||||
|
will panic. After a process supervisor restarts your application, Tendermint
|
||||||
|
should be able to reconnect successfully. The order of restart does not matter
|
||||||
|
for it.
|
||||||
|
|
||||||
|
Signal handling
|
||||||
|
---------------
|
||||||
|
|
||||||
|
We catch SIGINT and SIGTERM and try to clean up nicely. For other signals we
|
||||||
|
use the default behaviour in Go: `Default behavior of signals in Go programs
|
||||||
|
<https://golang.org/pkg/os/signal/#hdr-Default_behavior_of_signals_in_Go_programs>`__.
|
||||||
|
|
||||||
|
Hardware
|
||||||
|
--------
|
||||||
|
|
||||||
|
Processor and Memory
|
||||||
|
~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
While actual specs vary depending on the load and validators count, minimal requirements are:
|
||||||
|
|
||||||
|
- 1GB RAM
|
||||||
|
- 25GB of disk space
|
||||||
|
- 1.4 GHz CPU
|
||||||
|
|
||||||
|
SSD disks are preferable for applications with high transaction throughput.
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
|
||||||
|
- 2GB RAM
|
||||||
|
- 100GB SSD
|
||||||
|
- x64 2.0 GHz 2v CPU
|
||||||
|
|
||||||
|
While for now, Tendermint stores all the history and it may require significant
|
||||||
|
disk space over time, we are planning to implement state syncing (See `#828
|
||||||
|
<https://github.com/tendermint/tendermint/issues/828>`__). So, storing all the
|
||||||
|
past blocks will not be necessary.
|
||||||
|
|
||||||
|
Operating Systems
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Tendermint can be compiled for a wide range of operating systems thanks to Go
|
||||||
|
language (the list of $OS/$ARCH pairs can be found `here
|
||||||
|
<https://golang.org/doc/install/source#environment>`__).
|
||||||
|
|
||||||
|
While we do not favor any operation system, more secure and stable Linux server
|
||||||
|
distributions (like Centos) should be preferred over desktop operation systems
|
||||||
|
(like Mac OS).
|
||||||
|
|
||||||
|
Misc.
|
||||||
|
~~~~~
|
||||||
|
|
||||||
|
NOTE: if you are going to use Tendermint in a public domain, make sure you read
|
||||||
|
`hardware recommendations (see "4. Hardware")
|
||||||
|
<https://cosmos.network/validators>`__ for a validator in the Cosmos network.
|
||||||
|
|
||||||
|
Configuration parameters
|
||||||
|
------------------------
|
||||||
|
|
||||||
|
- ``p2p.flush_throttle_timeout``
|
||||||
|
``p2p.max_packet_msg_payload_size``
|
||||||
|
``p2p.send_rate``
|
||||||
|
``p2p.recv_rate``
|
||||||
|
|
||||||
|
If you are going to use Tendermint in a private domain and you have a private
|
||||||
|
high-speed network among your peers, it makes sense to lower flush throttle
|
||||||
|
timeout and increase other params.
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
[p2p]
|
||||||
|
|
||||||
|
send_rate=20000000 # 2MB/s
|
||||||
|
recv_rate=20000000 # 2MB/s
|
||||||
|
flush_throttle_timeout=10
|
||||||
|
max_packet_msg_payload_size=10240 # 10KB
|
||||||
|
|
||||||
|
- ``mempool.recheck``
|
||||||
|
|
||||||
|
After every block, Tendermint rechecks every transaction left in the mempool to
|
||||||
|
see if transactions committed in that block affected the application state, so
|
||||||
|
some of the transactions left may become invalid. If that does not apply to
|
||||||
|
your application, you can disable it by setting ``mempool.recheck=false``.
|
||||||
|
|
||||||
|
- ``mempool.broadcast``
|
||||||
|
|
||||||
|
Setting this to false will stop the mempool from relaying transactions to other
|
||||||
|
peers until they are included in a block. It means only the peer you send the
|
||||||
|
tx to will see it until it is included in a block.
|
||||||
|
|
||||||
|
- ``consensus.skip_timeout_commit``
|
||||||
|
|
||||||
|
We want skip_timeout_commit=false when there is economics on the line because
|
||||||
|
proposers should wait to hear for more votes. But if you don't care about that
|
||||||
|
and want the fastest consensus, you can skip it. It will be kept false by
|
||||||
|
default for public deployments (e.g. `Cosmos Hub
|
||||||
|
<https://cosmos.network/intro/hub>`__) while for enterprise applications,
|
||||||
|
setting it to true is not a problem.
|
||||||
|
|
||||||
|
- ``consensus.peer_gossip_sleep_duration``
|
||||||
|
|
||||||
|
You can try to reduce the time your node sleeps before checking if theres something to send its peers.
|
||||||
|
|
||||||
|
- ``consensus.timeout_commit``
|
||||||
|
|
||||||
|
You can also try lowering ``timeout_commit`` (time we sleep before proposing the next block).
|
||||||
|
|
||||||
|
- ``consensus.max_block_size_txs``
|
||||||
|
|
||||||
|
By default, the maximum number of transactions per a block is 10_000. Feel free
|
||||||
|
to change it to suit your needs.
|
|
@ -0,0 +1,33 @@
|
||||||
|
# WAL
|
||||||
|
|
||||||
|
Consensus module writes every message to the WAL (write-ahead log).
|
||||||
|
|
||||||
|
It also issues fsync syscall through
|
||||||
|
[File#Sync](https://golang.org/pkg/os/#File.Sync) for messages signed by this
|
||||||
|
node (to prevent double signing).
|
||||||
|
|
||||||
|
Under the hood, it uses
|
||||||
|
[autofile.Group](https://godoc.org/github.com/tendermint/tmlibs/autofile#Group),
|
||||||
|
which rotates files when those get too big (> 10MB).
|
||||||
|
|
||||||
|
The total maximum size is 1GB. We only need the latest block and the block before it,
|
||||||
|
but if the former is dragging on across many rounds, we want all those rounds.
|
||||||
|
|
||||||
|
## Replay
|
||||||
|
|
||||||
|
Consensus module will replay all the messages of the last height written to WAL
|
||||||
|
before a crash (if such occurs).
|
||||||
|
|
||||||
|
The private validator may try to sign messages during replay because it runs
|
||||||
|
somewhat autonomously and does not know about replay process.
|
||||||
|
|
||||||
|
For example, if we got all the way to precommit in the WAL and then crash,
|
||||||
|
after we replay the proposal message, the private validator will try to sign a
|
||||||
|
prevote. But it will fail. That's ok because we’ll see the prevote later in the
|
||||||
|
WAL. Then it will go to precommit, and that time it will work because the
|
||||||
|
private validator contains the `LastSignBytes` and then we’ll replay the
|
||||||
|
precommit from the WAL.
|
||||||
|
|
||||||
|
Make sure to read about [WAL
|
||||||
|
corruption](https://tendermint.readthedocs.io/projects/tools/en/master/specification/corruption.html#wal-corruption)
|
||||||
|
and recovery strategies.
|
|
@ -47,3 +47,13 @@ type bcStatusResponseMessage struct {
|
||||||
## Protocol
|
## Protocol
|
||||||
|
|
||||||
TODO
|
TODO
|
||||||
|
|
||||||
|
## Channels
|
||||||
|
|
||||||
|
Defines `maxMsgSize` for the maximum size of incoming messages,
|
||||||
|
`SendQueueCapacity` and `RecvBufferCapacity` for maximum sending and
|
||||||
|
receiving buffers respectively. These are supposed to prevent amplification
|
||||||
|
attacks by setting up the upper limit on how much data we can receive & send to
|
||||||
|
a peer.
|
||||||
|
|
||||||
|
Sending incorrectly encoded data will result in stopping the peer.
|
||||||
|
|
|
@ -342,3 +342,11 @@ It broadcasts `NewRoundStepMessage` or `CommitStepMessage` upon new round state
|
||||||
broadcasting these messages does not depend on the PeerRoundState; it is sent on the StateChannel.
|
broadcasting these messages does not depend on the PeerRoundState; it is sent on the StateChannel.
|
||||||
Upon receiving VoteMessage it broadcasts `HasVoteMessage` message to its peers on the StateChannel.
|
Upon receiving VoteMessage it broadcasts `HasVoteMessage` message to its peers on the StateChannel.
|
||||||
`ProposalHeartbeatMessage` is sent the same way on the StateChannel.
|
`ProposalHeartbeatMessage` is sent the same way on the StateChannel.
|
||||||
|
|
||||||
|
## Channels
|
||||||
|
|
||||||
|
Defines 4 channels: state, data, vote and vote_set_bits. Each channel
|
||||||
|
has `SendQueueCapacity` and `RecvBufferCapacity` and
|
||||||
|
`RecvMessageCapacity` set to `maxMsgSize`.
|
||||||
|
|
||||||
|
Sending incorrectly encoded data will result in stopping the peer.
|
||||||
|
|
|
@ -0,0 +1,10 @@
|
||||||
|
# Evidence Reactor
|
||||||
|
|
||||||
|
## Channels
|
||||||
|
|
||||||
|
[#1503](https://github.com/tendermint/tendermint/issues/1503)
|
||||||
|
|
||||||
|
Sending invalid evidence will result in stopping the peer.
|
||||||
|
|
||||||
|
Sending incorrectly encoded data or data exceeding `maxMsgSize` will result
|
||||||
|
in stopping the peer.
|
|
@ -0,0 +1,14 @@
|
||||||
|
# Mempool Reactor
|
||||||
|
|
||||||
|
## Channels
|
||||||
|
|
||||||
|
[#1503](https://github.com/tendermint/tendermint/issues/1503)
|
||||||
|
|
||||||
|
Mempool maintains a cache of the last 10000 transactions to prevent
|
||||||
|
replaying old transactions (plus transactions coming from other
|
||||||
|
validators, who are continually exchanging transactions). Read [Replay
|
||||||
|
Protection](https://tendermint.readthedocs.io/projects/tools/en/master/app-development.html?#replay-protection)
|
||||||
|
for details.
|
||||||
|
|
||||||
|
Sending incorrectly encoded data or data exceeding `maxMsgSize` will result
|
||||||
|
in stopping the peer.
|
|
@ -0,0 +1,12 @@
|
||||||
|
# PEX Reactor
|
||||||
|
|
||||||
|
## Channels
|
||||||
|
|
||||||
|
Defines only `SendQueueCapacity`. [#1503](https://github.com/tendermint/tendermint/issues/1503)
|
||||||
|
|
||||||
|
Implements rate-limiting by enforcing minimal time between two consecutive
|
||||||
|
`pexRequestMessage` requests. If the peer sends us addresses we did not ask,
|
||||||
|
it is stopped.
|
||||||
|
|
||||||
|
Sending incorrectly encoded data or data exceeding `maxMsgSize` will result
|
||||||
|
in stopping the peer.
|
Loading…
Reference in New Issue