diff --git a/docs/abci-cli.md b/docs/abci-cli.md new file mode 100644 index 00000000..f8092d80 --- /dev/null +++ b/docs/abci-cli.md @@ -0,0 +1,329 @@ +# Using ABCI-CLI + +To facilitate testing and debugging of ABCI servers and simple apps, we +built a CLI, the `abci-cli`, for sending ABCI messages from the command +line. + +## Install + +Make sure you [have Go installed](https://golang.org/doc/install). + +Next, install the `abci-cli` tool and example applications: + + go get -u github.com/tendermint/abci/cmd/abci-cli + +If this fails, you may need to use [dep](https://github.com/golang/dep) +to get vendored dependencies: + + cd $GOPATH/src/github.com/tendermint/abci + make get_tools + make get_vendor_deps + make install + +Now run `abci-cli` to see the list of commands: + + Usage: + abci-cli [command] + + Available Commands: + batch Run a batch of abci commands against an application + check_tx Validate a tx + commit Commit the application state and return the Merkle root hash + console Start an interactive abci console for multiple commands + counter ABCI demo example + deliver_tx Deliver a new tx to the application + kvstore ABCI demo example + echo Have the application echo a message + help Help about any command + info Get some info about the application + query Query the application state + set_option Set an options on the application + + Flags: + --abci string socket or grpc (default "socket") + --address string address of application socket (default "tcp://127.0.0.1:46658") + -h, --help help for abci-cli + -v, --verbose print the command and results as if it were a console session + + Use "abci-cli [command] --help" for more information about a command. + +## KVStore - First Example + +The `abci-cli` tool lets us send ABCI messages to our application, to +help build and debug them. + +The most important messages are `deliver_tx`, `check_tx`, and `commit`, +but there are others for convenience, configuration, and information +purposes. + +We'll start a kvstore application, which was installed at the same time +as `abci-cli` above. The kvstore just stores transactions in a merkle +tree. + +Its code can be found +[here](https://github.com/tendermint/abci/blob/master/cmd/abci-cli/abci-cli.go) +and looks like: + + func cmdKVStore(cmd *cobra.Command, args []string) error { + logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout)) + + // Create the application - in memory or persisted to disk + var app types.Application + if flagPersist == "" { + app = kvstore.NewKVStoreApplication() + } else { + app = kvstore.NewPersistentKVStoreApplication(flagPersist) + app.(*kvstore.PersistentKVStoreApplication).SetLogger(logger.With("module", "kvstore")) + } + + // Start the listener + srv, err := server.NewServer(flagAddrD, flagAbci, app) + if err != nil { + return err + } + srv.SetLogger(logger.With("module", "abci-server")) + if err := srv.Start(); err != nil { + return err + } + + // Wait forever + cmn.TrapSignal(func() { + // Cleanup + srv.Stop() + }) + return nil + } + +Start by running: + + abci-cli kvstore + +And in another terminal, run + + abci-cli echo hello + abci-cli info + +You'll see something like: + + -> data: hello + -> data.hex: 68656C6C6F + +and: + + -> data: {"size":0} + -> data.hex: 7B2273697A65223A307D + +An ABCI application must provide two things: + +- a socket server +- a handler for ABCI messages + +When we run the `abci-cli` tool we open a new connection to the +application's socket server, send the given ABCI message, and wait for a +response. + +The server may be generic for a particular language, and we provide a +[reference implementation in +Golang](https://github.com/tendermint/abci/tree/master/server). See the +[list of other ABCI implementations](./ecosystem.html) for servers in +other languages. + +The handler is specific to the application, and may be arbitrary, so +long as it is deterministic and conforms to the ABCI interface +specification. + +So when we run `abci-cli info`, we open a new connection to the ABCI +server, which calls the `Info()` method on the application, which tells +us the number of transactions in our Merkle tree. + +Now, since every command opens a new connection, we provide the +`abci-cli console` and `abci-cli batch` commands, to allow multiple ABCI +messages to be sent over a single connection. + +Running `abci-cli console` should drop you in an interactive console for +speaking ABCI messages to your application. + +Try running these commands: + + > echo hello + -> code: OK + -> data: hello + -> data.hex: 0x68656C6C6F + + > info + -> code: OK + -> data: {"size":0} + -> data.hex: 0x7B2273697A65223A307D + + > commit + -> code: OK + -> data.hex: 0x0000000000000000 + + > deliver_tx "abc" + -> code: OK + + > info + -> code: OK + -> data: {"size":1} + -> data.hex: 0x7B2273697A65223A317D + + > commit + -> code: OK + -> data.hex: 0x0200000000000000 + + > query "abc" + -> code: OK + -> log: exists + -> height: 0 + -> value: abc + -> value.hex: 616263 + + > deliver_tx "def=xyz" + -> code: OK + + > commit + -> code: OK + -> data.hex: 0x0400000000000000 + + > query "def" + -> code: OK + -> log: exists + -> height: 0 + -> value: xyz + -> value.hex: 78797A + +Note that if we do `deliver_tx "abc"` it will store `(abc, abc)`, but if +we do `deliver_tx "abc=efg"` it will store `(abc, efg)`. + +Similarly, you could put the commands in a file and run +`abci-cli --verbose batch < myfile`. + +## Counter - Another Example + +Now that we've got the hang of it, let's try another application, the +"counter" app. + +Like the kvstore app, its code can be found +[here](https://github.com/tendermint/abci/blob/master/cmd/abci-cli/abci-cli.go) +and looks like: + + func cmdCounter(cmd *cobra.Command, args []string) error { + + app := counter.NewCounterApplication(flagSerial) + + logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout)) + + // Start the listener + srv, err := server.NewServer(flagAddrC, flagAbci, app) + if err != nil { + return err + } + srv.SetLogger(logger.With("module", "abci-server")) + if err := srv.Start(); err != nil { + return err + } + + // Wait forever + cmn.TrapSignal(func() { + // Cleanup + srv.Stop() + }) + return nil + } + +The counter app doesn't use a Merkle tree, it just counts how many times +we've sent a transaction, asked for a hash, or committed the state. The +result of `commit` is just the number of transactions sent. + +This application has two modes: `serial=off` and `serial=on`. + +When `serial=on`, transactions must be a big-endian encoded incrementing +integer, starting at 0. + +If `serial=off`, there are no restrictions on transactions. + +We can toggle the value of `serial` using the `set_option` ABCI message. + +When `serial=on`, some transactions are invalid. In a live blockchain, +transactions collect in memory before they are committed into blocks. To +avoid wasting resources on invalid transactions, ABCI provides the +`check_tx` message, which application developers can use to accept or +reject transactions, before they are stored in memory or gossipped to +other peers. + +In this instance of the counter app, `check_tx` only allows transactions +whose integer is greater than the last committed one. + +Let's kill the console and the kvstore application, and start the +counter app: + + abci-cli counter + +In another window, start the `abci-cli console`: + + > set_option serial on + -> code: OK + -> log: OK (SetOption doesn't return anything.) + + > check_tx 0x00 + -> code: OK + + > check_tx 0xff + -> code: OK + + > deliver_tx 0x00 + -> code: OK + + > check_tx 0x00 + -> code: BadNonce + -> log: Invalid nonce. Expected >= 1, got 0 + + > deliver_tx 0x01 + -> code: OK + + > deliver_tx 0x04 + -> code: BadNonce + -> log: Invalid nonce. Expected 2, got 4 + + > info + -> code: OK + -> data: {"hashes":0,"txs":2} + -> data.hex: 0x7B22686173686573223A302C22747873223A327D + +This is a very simple application, but between `counter` and `kvstore`, +its easy to see how you can build out arbitrary application states on +top of the ABCI. [Hyperledger's +Burrow](https://github.com/hyperledger/burrow) also runs atop ABCI, +bringing with it Ethereum-like accounts, the Ethereum virtual-machine, +Monax's permissioning scheme, and native contracts extensions. + +But the ultimate flexibility comes from being able to write the +application easily in any language. + +We have implemented the counter in a number of languages (see the +example directory <https://github.com/tendermint/abci/tree/master/example\_\_). + +To run the Node JS version, `cd` to `example/js` and run + + node app.js + +(you'll have to kill the other counter application process). In another +window, run the console and those previous ABCI commands. You should get +the same results as for the Go version. + +## Bounties + +Want to write the counter app in your favorite language?! We'd be happy +to add you to our [ecosystem](https://tendermint.com/ecosystem)! We're +also offering [bounties](https://hackerone.com/tendermint/) for +implementations in new languages! + +The `abci-cli` is designed strictly for testing and debugging. In a real +deployment, the role of sending messages is taken by Tendermint, which +connects to the app using three separate connections, each with its own +pattern of messages. + +For more information, see the [application developers +guide](./app-development.html). For examples of running an ABCI app with +Tendermint, see the [getting started guide](./getting-started.html). +Next is the ABCI specification. diff --git a/docs/abci-cli.rst b/docs/abci-cli.rst deleted file mode 100644 index d4a73723..00000000 --- a/docs/abci-cli.rst +++ /dev/null @@ -1,371 +0,0 @@ -Using ABCI-CLI -============== - -To facilitate testing and debugging of ABCI servers and simple apps, we -built a CLI, the ``abci-cli``, for sending ABCI messages from the -command line. - -Install -------- - -Make sure you `have Go installed `__. - -Next, install the ``abci-cli`` tool and example applications: - -:: - - go get -u github.com/tendermint/abci/cmd/abci-cli - -If this fails, you may need to use `dep `__ to get vendored -dependencies: - -:: - - cd $GOPATH/src/github.com/tendermint/abci - make get_tools - make get_vendor_deps - make install - -Now run ``abci-cli`` to see the list of commands: - -:: - - Usage: - abci-cli [command] - - Available Commands: - batch Run a batch of abci commands against an application - check_tx Validate a tx - commit Commit the application state and return the Merkle root hash - console Start an interactive abci console for multiple commands - counter ABCI demo example - deliver_tx Deliver a new tx to the application - kvstore ABCI demo example - echo Have the application echo a message - help Help about any command - info Get some info about the application - query Query the application state - set_option Set an options on the application - - Flags: - --abci string socket or grpc (default "socket") - --address string address of application socket (default "tcp://127.0.0.1:46658") - -h, --help help for abci-cli - -v, --verbose print the command and results as if it were a console session - - Use "abci-cli [command] --help" for more information about a command. - - -KVStore - First Example ------------------------ - -The ``abci-cli`` tool lets us send ABCI messages to our application, to -help build and debug them. - -The most important messages are ``deliver_tx``, ``check_tx``, and -``commit``, but there are others for convenience, configuration, and -information purposes. - -We'll start a kvstore application, which was installed at the same time as -``abci-cli`` above. The kvstore just stores transactions in a merkle tree. - -Its code can be found `here `__ and looks like: - -.. container:: toggle - - .. container:: header - - **Show/Hide KVStore Example** - - .. code-block:: go - - func cmdKVStore(cmd *cobra.Command, args []string) error { - logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout)) - - // Create the application - in memory or persisted to disk - var app types.Application - if flagPersist == "" { - app = kvstore.NewKVStoreApplication() - } else { - app = kvstore.NewPersistentKVStoreApplication(flagPersist) - app.(*kvstore.PersistentKVStoreApplication).SetLogger(logger.With("module", "kvstore")) - } - - // Start the listener - srv, err := server.NewServer(flagAddrD, flagAbci, app) - if err != nil { - return err - } - srv.SetLogger(logger.With("module", "abci-server")) - if err := srv.Start(); err != nil { - return err - } - - // Wait forever - cmn.TrapSignal(func() { - // Cleanup - srv.Stop() - }) - return nil - } - -Start by running: - -:: - - abci-cli kvstore - -And in another terminal, run - -:: - - abci-cli echo hello - abci-cli info - -You'll see something like: - -:: - - -> data: hello - -> data.hex: 68656C6C6F - -and: - -:: - - -> data: {"size":0} - -> data.hex: 7B2273697A65223A307D - -An ABCI application must provide two things: - -- a socket server -- a handler for ABCI messages - -When we run the ``abci-cli`` tool we open a new connection to the -application's socket server, send the given ABCI message, and wait for a -response. - -The server may be generic for a particular language, and we provide a -`reference implementation in -Golang `__. See -the `list of other ABCI -implementations <./ecosystem.html>`__ for servers in -other languages. - -The handler is specific to the application, and may be arbitrary, so -long as it is deterministic and conforms to the ABCI interface -specification. - -So when we run ``abci-cli info``, we open a new connection to the ABCI -server, which calls the ``Info()`` method on the application, which -tells us the number of transactions in our Merkle tree. - -Now, since every command opens a new connection, we provide the -``abci-cli console`` and ``abci-cli batch`` commands, to allow multiple -ABCI messages to be sent over a single connection. - -Running ``abci-cli console`` should drop you in an interactive console -for speaking ABCI messages to your application. - -Try running these commands: - -:: - - > echo hello - -> code: OK - -> data: hello - -> data.hex: 0x68656C6C6F - - > info - -> code: OK - -> data: {"size":0} - -> data.hex: 0x7B2273697A65223A307D - - > commit - -> code: OK - -> data.hex: 0x0000000000000000 - - > deliver_tx "abc" - -> code: OK - - > info - -> code: OK - -> data: {"size":1} - -> data.hex: 0x7B2273697A65223A317D - - > commit - -> code: OK - -> data.hex: 0x0200000000000000 - - > query "abc" - -> code: OK - -> log: exists - -> height: 0 - -> value: abc - -> value.hex: 616263 - - > deliver_tx "def=xyz" - -> code: OK - - > commit - -> code: OK - -> data.hex: 0x0400000000000000 - - > query "def" - -> code: OK - -> log: exists - -> height: 0 - -> value: xyz - -> value.hex: 78797A - -Note that if we do ``deliver_tx "abc"`` it will store ``(abc, abc)``, -but if we do ``deliver_tx "abc=efg"`` it will store ``(abc, efg)``. - -Similarly, you could put the commands in a file and run -``abci-cli --verbose batch < myfile``. - -Counter - Another Example -------------------------- - -Now that we've got the hang of it, let's try another application, the -"counter" app. - -Like the kvstore app, its code can be found `here `__ and looks like: - -.. container:: toggle - - .. container:: header - - **Show/Hide Counter Example** - - .. code-block:: go - - func cmdCounter(cmd *cobra.Command, args []string) error { - - app := counter.NewCounterApplication(flagSerial) - - logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout)) - - // Start the listener - srv, err := server.NewServer(flagAddrC, flagAbci, app) - if err != nil { - return err - } - srv.SetLogger(logger.With("module", "abci-server")) - if err := srv.Start(); err != nil { - return err - } - - // Wait forever - cmn.TrapSignal(func() { - // Cleanup - srv.Stop() - }) - return nil - } - - -The counter app doesn't use a Merkle tree, it just counts how many times -we've sent a transaction, asked for a hash, or committed the state. The -result of ``commit`` is just the number of transactions sent. - -This application has two modes: ``serial=off`` and ``serial=on``. - -When ``serial=on``, transactions must be a big-endian encoded -incrementing integer, starting at 0. - -If ``serial=off``, there are no restrictions on transactions. - -We can toggle the value of ``serial`` using the ``set_option`` ABCI -message. - -When ``serial=on``, some transactions are invalid. In a live blockchain, -transactions collect in memory before they are committed into blocks. To -avoid wasting resources on invalid transactions, ABCI provides the -``check_tx`` message, which application developers can use to accept or -reject transactions, before they are stored in memory or gossipped to -other peers. - -In this instance of the counter app, ``check_tx`` only allows -transactions whose integer is greater than the last committed one. - -Let's kill the console and the kvstore application, and start the counter -app: - -:: - - abci-cli counter - -In another window, start the ``abci-cli console``: - -:: - - > set_option serial on - -> code: OK - -> log: OK (SetOption doesn't return anything.) - - > check_tx 0x00 - -> code: OK - - > check_tx 0xff - -> code: OK - - > deliver_tx 0x00 - -> code: OK - - > check_tx 0x00 - -> code: BadNonce - -> log: Invalid nonce. Expected >= 1, got 0 - - > deliver_tx 0x01 - -> code: OK - - > deliver_tx 0x04 - -> code: BadNonce - -> log: Invalid nonce. Expected 2, got 4 - - > info - -> code: OK - -> data: {"hashes":0,"txs":2} - -> data.hex: 0x7B22686173686573223A302C22747873223A327D - -This is a very simple application, but between ``counter`` and -``kvstore``, its easy to see how you can build out arbitrary application -states on top of the ABCI. `Hyperledger's -Burrow `__ also runs atop ABCI, -bringing with it Ethereum-like accounts, the Ethereum virtual-machine, -Monax's permissioning scheme, and native contracts extensions. - -But the ultimate flexibility comes from being able to write the -application easily in any language. - -We have implemented the counter in a number of languages (see the -`example directory `__! -We're also offering `bounties `__ for -implementations in new languages! - -The ``abci-cli`` is designed strictly for testing and debugging. In a -real deployment, the role of sending messages is taken by Tendermint, -which connects to the app using three separate connections, each with -its own pattern of messages. - -For more information, see the `application developers -guide <./app-development.html>`__. For examples of running an ABCI -app with Tendermint, see the `getting started -guide <./getting-started.html>`__. Next is the ABCI specification. diff --git a/docs/app-architecture.md b/docs/app-architecture.md new file mode 100644 index 00000000..242bea64 --- /dev/null +++ b/docs/app-architecture.md @@ -0,0 +1,50 @@ +# Application Architecture Guide + +Here we provide a brief guide on the recommended architecture of a +Tendermint blockchain application. + +The following diagram provides a superb example: + + + +The end-user application here is the Cosmos Voyager, at the bottom left. +Voyager communicates with a REST API exposed by a local Light-Client +Daemon. The Light-Client Daemon is an application specific program that +communicates with Tendermint nodes and verifies Tendermint light-client +proofs through the Tendermint Core RPC. The Tendermint Core process +communicates with a local ABCI application, where the user query or +transaction is actually processed. + +The ABCI application must be a deterministic result of the Tendermint +consensus - any external influence on the application state that didn't +come through Tendermint could cause a consensus failure. Thus *nothing* +should communicate with the application except Tendermint via ABCI. + +If the application is written in Go, it can be compiled into the +Tendermint binary. Otherwise, it should use a unix socket to communicate +with Tendermint. If it's necessary to use TCP, extra care must be taken +to encrypt and authenticate the connection. + +All reads from the app happen through the Tendermint `/abci_query` +endpoint. All writes to the app happen through the Tendermint +`/broadcast_tx_*` endpoints. + +The Light-Client Daemon is what provides light clients (end users) with +nearly all the security of a full node. It formats and broadcasts +transactions, and verifies proofs of queries and transaction results. +Note that it need not be a daemon - the Light-Client logic could instead +be implemented in the same process as the end-user application. + +Note for those ABCI applications with weaker security requirements, the +functionality of the Light-Client Daemon can be moved into the ABCI +application process itself. That said, exposing the application process +to anything besides Tendermint over ABCI requires extreme caution, as +all transactions, and possibly all queries, should still pass through +Tendermint. + +See the following for more extensive documentation: +- [Interchain Standard for the Light-Client REST API](https://github.com/cosmos/cosmos-sdk/pull/1028) +- [Tendermint RPC Docs](https://tendermint.github.io/slate/) +- [Tendermint in Production](https://github.com/tendermint/tendermint/pull/1618) +- [Tendermint Basics](https://tendermint.readthedocs.io/en/master/using-tendermint.html) +- [ABCI spec](https://github.com/tendermint/abci/blob/master/specification.rst) diff --git a/docs/app-architecture.rst b/docs/app-architecture.rst deleted file mode 100644 index c303ba4a..00000000 --- a/docs/app-architecture.rst +++ /dev/null @@ -1,42 +0,0 @@ -Application Architecture Guide -============================== - -Here we provide a brief guide on the recommended architecture of a Tendermint blockchain -application. - -The following diagram provides a superb example: - -https://drive.google.com/open?id=1yR2XpRi9YCY9H9uMfcw8-RMJpvDyvjz9 - -The end-user application here is the Cosmos Voyager, at the bottom left. -Voyager communicates with a REST API exposed by a local Light-Client Daemon. -The Light-Client Daemon is an application specific program that communicates with -Tendermint nodes and verifies Tendermint light-client proofs through the Tendermint Core RPC. -The Tendermint Core process communicates with a local ABCI application, where the -user query or transaction is actually processed. - -The ABCI application must be a deterministic result of the Tendermint consensus - any external influence -on the application state that didn't come through Tendermint could cause a -consensus failure. Thus *nothing* should communicate with the application except Tendermint via ABCI. - -If the application is written in Go, it can be compiled into the Tendermint binary. -Otherwise, it should use a unix socket to communicate with Tendermint. -If it's necessary to use TCP, extra care must be taken to encrypt and authenticate the connection. - -All reads from the app happen through the Tendermint `/abci_query` endpoint. -All writes to the app happen through the Tendermint `/broadcast_tx_*` endpoints. - -The Light-Client Daemon is what provides light clients (end users) with nearly all the security of a full node. -It formats and broadcasts transactions, and verifies proofs of queries and transaction results. -Note that it need not be a daemon - the Light-Client logic could instead be implemented in the same process as the end-user application. - -Note for those ABCI applications with weaker security requirements, the functionality of the Light-Client Daemon can be moved -into the ABCI application process itself. That said, exposing the application process to anything besides Tendermint over ABCI -requires extreme caution, as all transactions, and possibly all queries, should still pass through Tendermint. - -See the following for more extensive documentation: -- [Interchain Standard for the Light-Client REST API](https://github.com/cosmos/cosmos-sdk/pull/1028) -- [Tendermint RPC Docs](https://tendermint.github.io/slate/) -- [Tendermint in Production](https://github.com/tendermint/tendermint/pull/1618) -- [Tendermint Basics](https://tendermint.readthedocs.io/en/master/using-tendermint.html) -- [ABCI spec](https://github.com/tendermint/abci/blob/master/specification.rst) diff --git a/docs/index.rst b/docs/index.rst index f9d71429..75d150e1 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -18,10 +18,10 @@ Tendermint 101 .. toctree:: :maxdepth: 2 - introduction.rst - install.rst - getting-started.rst - using-tendermint.rst + introduction.md + install.md + getting-started.md + using-tendermint.md Tendermint Ecosystem -------------------- @@ -29,7 +29,7 @@ Tendermint Ecosystem .. toctree:: :maxdepth: 2 - ecosystem.rst + ecosystem.md Tendermint Tools ---------------- @@ -39,26 +39,28 @@ Tendermint Tools .. toctree:: :maxdepth: 2 - deploy-testnets.rst - terraform-and-ansible.rst + deploy-testnets.md + terraform-and-ansible.md tools/docker.rst tools/benchmarking.rst tools/monitoring.rst +.. TODO fix tools + Tendermint 102 -------------- .. toctree:: :maxdepth: 2 - abci-cli.rst - abci-spec.rst - app-architecture.rst - app-development.rst - subscribing-to-events-via-websocket.rst - indexing-transactions.rst - how-to-read-logs.rst - running-in-production.rst + abci-cli.md + abci-spec.rst + app-architecture.md + app-development.md + subscribing-to-events-via-websocket.md + indexing-transactions.md + how-to-read-logs.md + running-in-production.md Tendermint 201 -------------- @@ -66,9 +68,9 @@ Tendermint 201 .. toctree:: :maxdepth: 2 - specification.rst - determinism.rst - transactional-semantics.rst + specification.md + determinism.md + transactional-semantics.md * For a deeper dive, see `this thesis `__. * There is also the `original whitepaper `__, though it is now quite outdated.