Update CD, dependencies, READMEs
This commit is contained in:
parent
464946dc16
commit
f45e614765
178
README.md
178
README.md
|
@ -1,184 +1,22 @@
|
||||||
# Overview
|
# mango-geyser-services
|
||||||
|
|
||||||
This project is about streaming Solana account updates for a specific program
|
Mango v4 Geyser Services
|
||||||
into other databases or event queues.
|
|
||||||
|
|
||||||
Having an up to date version of all account data data in a database is
|
|
||||||
particularly useful for queries that need access to all accounts. For example,
|
|
||||||
retrieving the addresses of Mango Markets accounts with the largest unrealized
|
|
||||||
PnL goes from "getProgramAccounts from a Solana node for 50MB of data and
|
|
||||||
compute locally (3-10s total)" to "run a SQL query (150ms total)".
|
|
||||||
|
|
||||||
The database could also be used as a backend for serving `getMultipleAccounts`
|
|
||||||
and `getProgramAccounts` queries generally. That would reduce load on Solana
|
|
||||||
RPCt nodes while decreasing response times.
|
|
||||||
|
|
||||||
Supported Solana sources:
|
|
||||||
|
|
||||||
- Geyser plugin (preferred) plus JSONRPC HTTP API (for initial snapshots)
|
|
||||||
|
|
||||||
Unfinished Solana sources:
|
|
||||||
|
|
||||||
- JSONRPC websocket subscriptions plus JSONRPC HTTP API (for initial snapshots)
|
|
||||||
|
|
||||||
Supported targets:
|
|
||||||
|
|
||||||
- PostgreSQL
|
|
||||||
|
|
||||||
# Components
|
# Components
|
||||||
|
|
||||||
- [`geyser-plugin-grpc/`](geyser-plugin-grpc/)
|
|
||||||
|
|
||||||
The Solana Geyser plugin. It opens a gRPC server (see [`proto/`](proto/)) and
|
|
||||||
broadcasts account and slot updates to all clients that connect.
|
|
||||||
|
|
||||||
- [`lib/`](lib/)
|
- [`lib/`](lib/)
|
||||||
|
|
||||||
The connector abstractions that the connector service is built from.
|
Tools for building services
|
||||||
|
|
||||||
Projects may want to use it to build their own connector service and decode
|
|
||||||
their specific account data before sending it into target systems.
|
|
||||||
|
|
||||||
- [`connector-raw/`](connector-raw/)
|
|
||||||
|
|
||||||
A connector binary built on lib/ that stores raw binary account data in
|
|
||||||
PostgreSQL.
|
|
||||||
|
|
||||||
- [`connector-mango/`](connector-mango/)
|
|
||||||
|
|
||||||
A connector binary built on lib/ that decodes Mango account types before
|
|
||||||
storing them in PostgeSQL.
|
|
||||||
|
|
||||||
- [`service-mango-fills/`](service-mango-fills/)
|
- [`service-mango-fills/`](service-mango-fills/)
|
||||||
|
|
||||||
A service providing lowest-latency, bandwidth conserving access to fill events
|
A service providing lowest-latency, bandwidth conserving access to fill events for Mango V4 Perp and Openbook markets
|
||||||
as they are processed by the rpc node.
|
as they are processed by the rpc node.
|
||||||
|
|
||||||
# Setup Tutorial
|
- [`service-mango-pnl/`](service-mango-pnl/)
|
||||||
|
|
||||||
1. Compile the project.
|
A service providing pre-computed account lists ordered by unsettled PnL per market
|
||||||
|
|
||||||
Make sure that you are using _exactly_ the same Rust version for compiling
|
- [`service-mango-orderbook/`](service-mango-pnl/)
|
||||||
the Geyser plugin that was used for compiling your `solana-validator`!
|
|
||||||
Otherwise the plugin will crash the validator during startup!
|
|
||||||
|
|
||||||
2. Prepare the plugin configuration file.
|
A service providing Orderbook L2 state and delta updates for Mango V4 Perp and Openbook Spot markets
|
||||||
|
|
||||||
[Here is an example](geyser-plugin-grpc/example-config.json). This file
|
|
||||||
points the validator to your plugin shared library, controls which accounts
|
|
||||||
will be exported, which address the gRPC server will bind to and internal
|
|
||||||
queue sizes.
|
|
||||||
|
|
||||||
3. Run `solana-validator` with `--geyser-plugin-config myconfig.json`.
|
|
||||||
|
|
||||||
Check the logs to ensure the plugin was loaded.
|
|
||||||
|
|
||||||
4. Prepare the connector configuration file.
|
|
||||||
|
|
||||||
[Here is an example](connector-raw/example-config.toml).
|
|
||||||
|
|
||||||
- `rpc_ws_url` is unused and can stay empty.
|
|
||||||
- `connection_string` for your `grpc_sources` must point to the gRPC server
|
|
||||||
address configured for the plugin.
|
|
||||||
- `rpc_http_url` must point to the JSON-RPC URL.
|
|
||||||
- `connection_string` for your `posgres_target` uses
|
|
||||||
[the tokio-postgres syntax](https://docs.rs/tokio-postgres/0.7.5/tokio_postgres/config/struct.Config.html)
|
|
||||||
- `program_id` must match what is configured for the gRPC plugin
|
|
||||||
|
|
||||||
5. Prepare the PostgreSQL schema.
|
|
||||||
|
|
||||||
Use [this example script](connector-raw/scripts/create_schema.sql).
|
|
||||||
|
|
||||||
6. Start the connector service binary.
|
|
||||||
|
|
||||||
Pass the path to the config file as the first argument. It logs to stdout. It
|
|
||||||
should be restarted on exit. (it intentionally terminates when postgres is
|
|
||||||
unreachable for too long, for example)
|
|
||||||
|
|
||||||
7. Monitor the logs
|
|
||||||
|
|
||||||
`WARN` messages can be recovered from. `ERROR` messages need attention.
|
|
||||||
|
|
||||||
Check the metrics for `account_write_queue` and `slot_update_queue`: They
|
|
||||||
should be around 0. If they keep growing the service can't keep up and you'll
|
|
||||||
need to figure out what's up.
|
|
||||||
|
|
||||||
# Design and Reliability
|
|
||||||
|
|
||||||
```
|
|
||||||
Solana ---------------> Connector -----------> PostgreSQL
|
|
||||||
nodes jsonrpc/gRPC nodes
|
|
||||||
```
|
|
||||||
|
|
||||||
For reliability it is recommended to feed data from multiple Solana nodes into
|
|
||||||
each Connector node.
|
|
||||||
|
|
||||||
It is also allowed to run multiple Connector nodes that target the same
|
|
||||||
PostgeSQL target database.
|
|
||||||
|
|
||||||
The Connector service is stateless (except for some caches). Restarting it is
|
|
||||||
always safe.
|
|
||||||
|
|
||||||
If the Solana node is down, the Connector service attempts to reconnect and then
|
|
||||||
requests a new data snapshot if necessary.
|
|
||||||
|
|
||||||
If PostgeSQL is down temporarily, the Connector service caches updates and
|
|
||||||
applies them when the database is back up.
|
|
||||||
|
|
||||||
If PostgreSQL is down for a longer time, the Connector service exits with an
|
|
||||||
error. On restart, it pauses until PostgreSQL is back up, and then starts
|
|
||||||
pulling data from the Solana nodes again.
|
|
||||||
|
|
||||||
# PostgreSQL data layout
|
|
||||||
|
|
||||||
See `scripts/` for SQL that creates the target schema.
|
|
||||||
|
|
||||||
The Connector streams data into the `account_write` and `slot` tables. When
|
|
||||||
slots become "rooted", older `account_write` data rooted slots is deleted. That
|
|
||||||
way the current account data for the latest rooted, confirmed or processed slot
|
|
||||||
can be queried, but older data is forgotten.
|
|
||||||
|
|
||||||
When new slots arrive, the `uncle` column is updated for "processed" and
|
|
||||||
"confirmed" slots to allow easy filtering of slots that are no longer part of
|
|
||||||
the chain.
|
|
||||||
|
|
||||||
Example for querying confirmed data:
|
|
||||||
|
|
||||||
```
|
|
||||||
SELECT DISTINCT ON(pubkey_id)
|
|
||||||
pubkey, account_write.*
|
|
||||||
FROM account_write
|
|
||||||
LEFT JOIN slot USING(slot)
|
|
||||||
INNER JOIN pubkey USING(pubkey_id)
|
|
||||||
WHERE status = 'Rooted' OR status IS NULL OR (uncle = FALSE AND status = 'Confirmed')
|
|
||||||
ORDER BY pubkey_id, slot DESC, write_version DESC;
|
|
||||||
```
|
|
||||||
|
|
||||||
For each pubkey, this gets the latest (most recent slot, most recent
|
|
||||||
write_version) account data; limited to slots that are either rooted or
|
|
||||||
(confirmed and not an uncle).
|
|
||||||
|
|
||||||
# Fill Service Setup
|
|
||||||
|
|
||||||
1. Prepare the connector configuration file.
|
|
||||||
|
|
||||||
[Here is an example](service-mango-fills/example-config.toml).
|
|
||||||
|
|
||||||
- `bind_ws_addr` is the listen port for the websocket clients
|
|
||||||
- `rpc_ws_url` is unused and can stay empty.
|
|
||||||
- `connection_string` for your `grpc_sources` must point to the gRPC server
|
|
||||||
address configured for the plugin.
|
|
||||||
- `rpc_http_url` must point to the JSON-RPC URL.
|
|
||||||
- `program_id` must match what is configured for the gRPC plugin
|
|
||||||
- `markets` need to contain all observed perp markets
|
|
||||||
|
|
||||||
2. Start the service binary.
|
|
||||||
|
|
||||||
Pass the path to the config file as the first argument. It logs to stdout. It
|
|
||||||
should be restarted on exit.
|
|
||||||
|
|
||||||
3. Monitor the logs
|
|
||||||
|
|
||||||
`WARN` messages can be recovered from. `ERROR` messages need attention. The
|
|
||||||
logs are very spammy changing the default log level is recommended when you
|
|
||||||
dont want to analyze performance of the service.
|
|
||||||
|
|
|
@ -0,0 +1,23 @@
|
||||||
|
app = "mango-pnl"
|
||||||
|
kill_signal = "SIGINT"
|
||||||
|
kill_timeout = 5
|
||||||
|
|
||||||
|
[build]
|
||||||
|
dockerfile = "../Dockerfile"
|
||||||
|
|
||||||
|
[experimental]
|
||||||
|
cmd = ["service-mango-pnl", "pnl-config.toml"]
|
||||||
|
|
||||||
|
[[services]]
|
||||||
|
internal_port = 8081
|
||||||
|
processes = ["app"]
|
||||||
|
protocol = "tcp"
|
||||||
|
|
||||||
|
[services.concurrency]
|
||||||
|
hard_limit = 1024
|
||||||
|
soft_limit = 1024
|
||||||
|
type = "connections"
|
||||||
|
|
||||||
|
[metrics]
|
||||||
|
path = "/metrics"
|
||||||
|
port = 9091
|
|
@ -3,12 +3,8 @@ app = "mango-geyser-services"
|
||||||
kill_signal = "SIGINT"
|
kill_signal = "SIGINT"
|
||||||
kill_timeout = 5
|
kill_timeout = 5
|
||||||
|
|
||||||
# [build]
|
|
||||||
# image = "us-docker.pkg.dev/mango-markets/gcr.io/mango-geyser-services:latest"
|
|
||||||
|
|
||||||
[processes]
|
[processes]
|
||||||
fills = "service-mango-fills fills-config.toml"
|
fills = "service-mango-fills fills-config.toml"
|
||||||
pnl = "service-mango-pnl pnl-config.toml"
|
|
||||||
orderbook = "service-mango-orderbook orderbook-config.toml"
|
orderbook = "service-mango-orderbook orderbook-config.toml"
|
||||||
|
|
||||||
[[services]]
|
[[services]]
|
||||||
|
@ -25,20 +21,6 @@ kill_timeout = 5
|
||||||
hard_limit = 1024
|
hard_limit = 1024
|
||||||
soft_limit = 1024
|
soft_limit = 1024
|
||||||
|
|
||||||
[[services]]
|
|
||||||
processes = ["pnl"]
|
|
||||||
internal_port = 8081
|
|
||||||
protocol = "tcp"
|
|
||||||
|
|
||||||
[[services.ports]]
|
|
||||||
handlers = ["tls", "http"]
|
|
||||||
port = 8081
|
|
||||||
|
|
||||||
[services.concurrency]
|
|
||||||
type = "connections"
|
|
||||||
hard_limit = 1024
|
|
||||||
soft_limit = 1024
|
|
||||||
|
|
||||||
[[services]]
|
[[services]]
|
||||||
processes = ["orderbook"]
|
processes = ["orderbook"]
|
||||||
internal_port = 8082
|
internal_port = 8082
|
|
@ -24,8 +24,8 @@ tokio = { version = "1", features = ["full"] }
|
||||||
tokio-tungstenite = "0.17"
|
tokio-tungstenite = "0.17"
|
||||||
bytemuck = "1.7.2"
|
bytemuck = "1.7.2"
|
||||||
|
|
||||||
mango-v4 = { git = "https://github.com/blockworks-foundation/mango-v4", branch = "ckamm/accountfetcher-send" }
|
mango-v4 = { git = "https://github.com/blockworks-foundation/mango-v4", branch = "dev" }
|
||||||
client = { git = "https://github.com/blockworks-foundation/mango-v4", branch = "ckamm/accountfetcher-send" }
|
client = { git = "https://github.com/blockworks-foundation/mango-v4", branch = "dev" }
|
||||||
serum_dex = { git = "https://github.com/jup-ag/openbook-program", branch = "feat/expose-things" }
|
serum_dex = { git = "https://github.com/jup-ag/openbook-program", branch = "feat/expose-things" }
|
||||||
anchor-lang = "0.25.0"
|
anchor-lang = "0.25.0"
|
||||||
anchor-client = "0.25.0"
|
anchor-client = "0.25.0"
|
||||||
|
|
|
@ -1,13 +1,35 @@
|
||||||
connector-fills
|
# service-mango-fills
|
||||||
|
|
||||||
This module parses event queues and exposes individual fills on a websocket.
|
This module parses event queues and exposes individual fills on a websocket.
|
||||||
|
|
||||||
TODO:
|
## Setup
|
||||||
|
|
||||||
- [] early filter out all account writes we dont care about
|
1. Prepare the connector configuration file.
|
||||||
- [] startup logic, dont accept websockets before first snapshot
|
|
||||||
|
[Here is an example](service-mango-fills/example-config.toml).
|
||||||
|
|
||||||
|
- `bind_ws_addr` is the listen port for the websocket clients
|
||||||
|
- `rpc_ws_url` is unused and can stay empty.
|
||||||
|
- `connection_string` for your `grpc_sources` must point to the gRPC server
|
||||||
|
address configured for the plugin.
|
||||||
|
- `rpc_http_url` must point to the JSON-RPC URL.
|
||||||
|
- `program_id` must match what is configured for the gRPC plugin
|
||||||
|
- `markets` need to contain all observed perp markets
|
||||||
|
|
||||||
|
2. Start the service binary.
|
||||||
|
|
||||||
|
Pass the path to the config file as the first argument. It logs to stdout. It
|
||||||
|
should be restarted on exit.
|
||||||
|
|
||||||
|
3. Monitor the logs
|
||||||
|
|
||||||
|
`WARN` messages can be recovered from. `ERROR` messages need attention. The
|
||||||
|
logs are very spammy changing the default log level is recommended when you
|
||||||
|
dont want to analyze performance of the service.
|
||||||
|
|
||||||
|
## TODO
|
||||||
|
- [] startup logic, dont accept market subscriptions before first snapshot
|
||||||
- [] failover logic, kill all websockets when we receive a later snapshot, more
|
- [] failover logic, kill all websockets when we receive a later snapshot, more
|
||||||
frequent when running on home connections
|
frequent when running on home connections
|
||||||
- [] track websocket connect / disconnect
|
|
||||||
- [] track latency accountwrite -> websocket
|
- [] track latency accountwrite -> websocket
|
||||||
- [] track queue length
|
- [] create new model for fills so snapshot maps can be combined per market
|
||||||
|
|
|
@ -0,0 +1,35 @@
|
||||||
|
# service-mango-orderbook
|
||||||
|
|
||||||
|
This module parses bookside accounts and exposes L2 data and updates on a websocket
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
1. Prepare the connector configuration file.
|
||||||
|
|
||||||
|
[Here is an example](service-mango-fills/example-config.toml).
|
||||||
|
|
||||||
|
- `bind_ws_addr` is the listen port for the websocket clients
|
||||||
|
- `rpc_ws_url` is unused and can stay empty.
|
||||||
|
- `connection_string` for your `grpc_sources` must point to the gRPC server
|
||||||
|
address configured for the plugin.
|
||||||
|
- `rpc_http_url` must point to the JSON-RPC URL.
|
||||||
|
- `program_id` must match what is configured for the gRPC plugin
|
||||||
|
- `markets` need to contain all observed perp markets
|
||||||
|
|
||||||
|
2. Start the service binary.
|
||||||
|
|
||||||
|
Pass the path to the config file as the first argument. It logs to stdout. It
|
||||||
|
should be restarted on exit.
|
||||||
|
|
||||||
|
3. Monitor the logs
|
||||||
|
|
||||||
|
`WARN` messages can be recovered from. `ERROR` messages need attention. The
|
||||||
|
logs are very spammy changing the default log level is recommended when you
|
||||||
|
dont want to analyze performance of the service.
|
||||||
|
|
||||||
|
## TODO
|
||||||
|
- [] startup logic, dont accept market subscriptions before first snapshot
|
||||||
|
- [] failover logic, kill all websockets when we receive a later snapshot, more
|
||||||
|
frequent when running on home connections
|
||||||
|
- [] track latency accountwrite -> websocket
|
||||||
|
- [] create new model for fills so snapshot maps can be combined per market
|
Loading…
Reference in New Issue