diff --git a/README.md b/README.md index 354bb1d..d3cddc8 100644 --- a/README.md +++ b/README.md @@ -1,184 +1,22 @@ -# Overview +# mango-geyser-services -This project is about streaming Solana account updates for a specific program -into other databases or event queues. - -Having an up to date version of all account data data in a database is -particularly useful for queries that need access to all accounts. For example, -retrieving the addresses of Mango Markets accounts with the largest unrealized -PnL goes from "getProgramAccounts from a Solana node for 50MB of data and -compute locally (3-10s total)" to "run a SQL query (150ms total)". - -The database could also be used as a backend for serving `getMultipleAccounts` -and `getProgramAccounts` queries generally. That would reduce load on Solana -RPCt nodes while decreasing response times. - -Supported Solana sources: - -- Geyser plugin (preferred) plus JSONRPC HTTP API (for initial snapshots) - -Unfinished Solana sources: - -- JSONRPC websocket subscriptions plus JSONRPC HTTP API (for initial snapshots) - -Supported targets: - -- PostgreSQL +Mango v4 Geyser Services # Components -- [`geyser-plugin-grpc/`](geyser-plugin-grpc/) - - The Solana Geyser plugin. It opens a gRPC server (see [`proto/`](proto/)) and - broadcasts account and slot updates to all clients that connect. - - [`lib/`](lib/) - The connector abstractions that the connector service is built from. - - Projects may want to use it to build their own connector service and decode - their specific account data before sending it into target systems. - -- [`connector-raw/`](connector-raw/) - - A connector binary built on lib/ that stores raw binary account data in - PostgreSQL. - -- [`connector-mango/`](connector-mango/) - - A connector binary built on lib/ that decodes Mango account types before - storing them in PostgeSQL. + Tools for building services - [`service-mango-fills/`](service-mango-fills/) - A service providing lowest-latency, bandwidth conserving access to fill events + A service providing lowest-latency, bandwidth conserving access to fill events for Mango V4 Perp and Openbook markets as they are processed by the rpc node. -# Setup Tutorial + - [`service-mango-pnl/`](service-mango-pnl/) -1. Compile the project. + A service providing pre-computed account lists ordered by unsettled PnL per market - Make sure that you are using _exactly_ the same Rust version for compiling - the Geyser plugin that was used for compiling your `solana-validator`! - Otherwise the plugin will crash the validator during startup! + - [`service-mango-orderbook/`](service-mango-pnl/) -2. Prepare the plugin configuration file. - - [Here is an example](geyser-plugin-grpc/example-config.json). This file - points the validator to your plugin shared library, controls which accounts - will be exported, which address the gRPC server will bind to and internal - queue sizes. - -3. Run `solana-validator` with `--geyser-plugin-config myconfig.json`. - - Check the logs to ensure the plugin was loaded. - -4. Prepare the connector configuration file. - - [Here is an example](connector-raw/example-config.toml). - - - `rpc_ws_url` is unused and can stay empty. - - `connection_string` for your `grpc_sources` must point to the gRPC server - address configured for the plugin. - - `rpc_http_url` must point to the JSON-RPC URL. - - `connection_string` for your `posgres_target` uses - [the tokio-postgres syntax](https://docs.rs/tokio-postgres/0.7.5/tokio_postgres/config/struct.Config.html) - - `program_id` must match what is configured for the gRPC plugin - -5. Prepare the PostgreSQL schema. - - Use [this example script](connector-raw/scripts/create_schema.sql). - -6. Start the connector service binary. - - Pass the path to the config file as the first argument. It logs to stdout. It - should be restarted on exit. (it intentionally terminates when postgres is - unreachable for too long, for example) - -7. Monitor the logs - - `WARN` messages can be recovered from. `ERROR` messages need attention. - - Check the metrics for `account_write_queue` and `slot_update_queue`: They - should be around 0. If they keep growing the service can't keep up and you'll - need to figure out what's up. - -# Design and Reliability - -``` -Solana ---------------> Connector -----------> PostgreSQL - nodes jsonrpc/gRPC nodes -``` - -For reliability it is recommended to feed data from multiple Solana nodes into -each Connector node. - -It is also allowed to run multiple Connector nodes that target the same -PostgeSQL target database. - -The Connector service is stateless (except for some caches). Restarting it is -always safe. - -If the Solana node is down, the Connector service attempts to reconnect and then -requests a new data snapshot if necessary. - -If PostgeSQL is down temporarily, the Connector service caches updates and -applies them when the database is back up. - -If PostgreSQL is down for a longer time, the Connector service exits with an -error. On restart, it pauses until PostgreSQL is back up, and then starts -pulling data from the Solana nodes again. - -# PostgreSQL data layout - -See `scripts/` for SQL that creates the target schema. - -The Connector streams data into the `account_write` and `slot` tables. When -slots become "rooted", older `account_write` data rooted slots is deleted. That -way the current account data for the latest rooted, confirmed or processed slot -can be queried, but older data is forgotten. - -When new slots arrive, the `uncle` column is updated for "processed" and -"confirmed" slots to allow easy filtering of slots that are no longer part of -the chain. - -Example for querying confirmed data: - -``` -SELECT DISTINCT ON(pubkey_id) - pubkey, account_write.* -FROM account_write -LEFT JOIN slot USING(slot) -INNER JOIN pubkey USING(pubkey_id) -WHERE status = 'Rooted' OR status IS NULL OR (uncle = FALSE AND status = 'Confirmed') -ORDER BY pubkey_id, slot DESC, write_version DESC; -``` - -For each pubkey, this gets the latest (most recent slot, most recent -write_version) account data; limited to slots that are either rooted or -(confirmed and not an uncle). - -# Fill Service Setup - -1. Prepare the connector configuration file. - - [Here is an example](service-mango-fills/example-config.toml). - - - `bind_ws_addr` is the listen port for the websocket clients - - `rpc_ws_url` is unused and can stay empty. - - `connection_string` for your `grpc_sources` must point to the gRPC server - address configured for the plugin. - - `rpc_http_url` must point to the JSON-RPC URL. - - `program_id` must match what is configured for the gRPC plugin - - `markets` need to contain all observed perp markets - -2. Start the service binary. - - Pass the path to the config file as the first argument. It logs to stdout. It - should be restarted on exit. - -3. Monitor the logs - - `WARN` messages can be recovered from. `ERROR` messages need attention. The - logs are very spammy changing the default log level is recommended when you - dont want to analyze performance of the service. + A service providing Orderbook L2 state and delta updates for Mango V4 Perp and Openbook Spot markets diff --git a/cd/pnl.toml b/cd/pnl.toml new file mode 100644 index 0000000..8560210 --- /dev/null +++ b/cd/pnl.toml @@ -0,0 +1,23 @@ +app = "mango-pnl" +kill_signal = "SIGINT" +kill_timeout = 5 + +[build] + dockerfile = "../Dockerfile" + +[experimental] + cmd = ["service-mango-pnl", "pnl-config.toml"] + +[[services]] + internal_port = 8081 + processes = ["app"] + protocol = "tcp" + + [services.concurrency] + hard_limit = 1024 + soft_limit = 1024 + type = "connections" + +[metrics] + path = "/metrics" + port = 9091 diff --git a/fly.toml b/cd/ws.toml similarity index 66% rename from fly.toml rename to cd/ws.toml index 0c7ed13..2a8e004 100644 --- a/fly.toml +++ b/cd/ws.toml @@ -3,12 +3,8 @@ app = "mango-geyser-services" kill_signal = "SIGINT" kill_timeout = 5 -# [build] -# image = "us-docker.pkg.dev/mango-markets/gcr.io/mango-geyser-services:latest" - [processes] fills = "service-mango-fills fills-config.toml" - pnl = "service-mango-pnl pnl-config.toml" orderbook = "service-mango-orderbook orderbook-config.toml" [[services]] @@ -25,20 +21,6 @@ kill_timeout = 5 hard_limit = 1024 soft_limit = 1024 -[[services]] - processes = ["pnl"] - internal_port = 8081 - protocol = "tcp" - - [[services.ports]] - handlers = ["tls", "http"] - port = 8081 - - [services.concurrency] - type = "connections" - hard_limit = 1024 - soft_limit = 1024 - [[services]] processes = ["orderbook"] internal_port = 8082 diff --git a/service-mango-fills/Cargo.toml b/service-mango-fills/Cargo.toml index b56c273..2e94ff9 100644 --- a/service-mango-fills/Cargo.toml +++ b/service-mango-fills/Cargo.toml @@ -24,8 +24,8 @@ tokio = { version = "1", features = ["full"] } tokio-tungstenite = "0.17" bytemuck = "1.7.2" -mango-v4 = { git = "https://github.com/blockworks-foundation/mango-v4", branch = "ckamm/accountfetcher-send" } -client = { git = "https://github.com/blockworks-foundation/mango-v4", branch = "ckamm/accountfetcher-send" } +mango-v4 = { git = "https://github.com/blockworks-foundation/mango-v4", branch = "dev" } +client = { git = "https://github.com/blockworks-foundation/mango-v4", branch = "dev" } serum_dex = { git = "https://github.com/jup-ag/openbook-program", branch = "feat/expose-things" } anchor-lang = "0.25.0" anchor-client = "0.25.0" diff --git a/service-mango-fills/README.md b/service-mango-fills/README.md index 32d9d47..7d8968e 100644 --- a/service-mango-fills/README.md +++ b/service-mango-fills/README.md @@ -1,13 +1,35 @@ -connector-fills +# service-mango-fills This module parses event queues and exposes individual fills on a websocket. -TODO: +## Setup -- [] early filter out all account writes we dont care about -- [] startup logic, dont accept websockets before first snapshot +1. Prepare the connector configuration file. + + [Here is an example](service-mango-fills/example-config.toml). + + - `bind_ws_addr` is the listen port for the websocket clients + - `rpc_ws_url` is unused and can stay empty. + - `connection_string` for your `grpc_sources` must point to the gRPC server + address configured for the plugin. + - `rpc_http_url` must point to the JSON-RPC URL. + - `program_id` must match what is configured for the gRPC plugin + - `markets` need to contain all observed perp markets + +2. Start the service binary. + + Pass the path to the config file as the first argument. It logs to stdout. It + should be restarted on exit. + +3. Monitor the logs + + `WARN` messages can be recovered from. `ERROR` messages need attention. The + logs are very spammy changing the default log level is recommended when you + dont want to analyze performance of the service. + +## TODO +- [] startup logic, dont accept market subscriptions before first snapshot - [] failover logic, kill all websockets when we receive a later snapshot, more frequent when running on home connections -- [] track websocket connect / disconnect - [] track latency accountwrite -> websocket -- [] track queue length +- [] create new model for fills so snapshot maps can be combined per market diff --git a/service-mango-orderbook/README.md b/service-mango-orderbook/README.md new file mode 100644 index 0000000..cb668a9 --- /dev/null +++ b/service-mango-orderbook/README.md @@ -0,0 +1,35 @@ +# service-mango-orderbook + +This module parses bookside accounts and exposes L2 data and updates on a websocket + +## Setup + +1. Prepare the connector configuration file. + + [Here is an example](service-mango-fills/example-config.toml). + + - `bind_ws_addr` is the listen port for the websocket clients + - `rpc_ws_url` is unused and can stay empty. + - `connection_string` for your `grpc_sources` must point to the gRPC server + address configured for the plugin. + - `rpc_http_url` must point to the JSON-RPC URL. + - `program_id` must match what is configured for the gRPC plugin + - `markets` need to contain all observed perp markets + +2. Start the service binary. + + Pass the path to the config file as the first argument. It logs to stdout. It + should be restarted on exit. + +3. Monitor the logs + + `WARN` messages can be recovered from. `ERROR` messages need attention. The + logs are very spammy changing the default log level is recommended when you + dont want to analyze performance of the service. + +## TODO +- [] startup logic, dont accept market subscriptions before first snapshot +- [] failover logic, kill all websockets when we receive a later snapshot, more + frequent when running on home connections +- [] track latency accountwrite -> websocket +- [] create new model for fills so snapshot maps can be combined per market