Christian Kamm
185bf4f9cf
Postgres: Fix account_write cleanup query
...
Avoid misbehaving in situations where there's an account write for a
slot that doesn't yet have an entry in the slots table by limiting the
slot numbers.
2022-01-20 15:28:16 +01:00
Christian Kamm
287308daf7
Readme updates
2022-01-20 14:56:34 +01:00
Christian Kamm
dc80d8266c
Process postgres cleanup jobs separately
...
Executing them for every rooted slot may overwhelm the database.
2022-01-20 14:49:51 +01:00
Christian Kamm
08b99875c2
Postgres: Add an index to account write tables
...
This index matches exactly the most frequent query type of
SELECT DISTINCT(pubkey) ...
ORDER BY pubkey, slot DESC, write_version DESC
2022-01-20 11:23:02 +01:00
Christian Kamm
cf26557f7d
Update to solana 1.8.12
2022-01-12 12:00:06 +01:00
Christian Kamm
afc08d8266
Fix metadata
2022-01-07 08:21:00 +01:00
Christian Kamm
7d8fd45de4
Add support for updating a monitoring table
...
That way connector interruptions can be seen directly form the target
database.
2022-01-06 12:54:35 +01:00
Christian Kamm
b1f18c0b13
Fix data deletion not being enableable
2022-01-05 16:02:34 +01:00
Christian Kamm
fb7e0d14e2
Update to solana 1.8.11
2022-01-04 18:39:48 +01:00
Christian Kamm
0755c7ed08
Update dependencies
2022-01-04 10:38:49 +01:00
Christian Kamm
5b5eaba4ff
Pipeline SQL inserts for better performance
...
Previously insert speed was primarily limited by roundtrip time to the
database since they were fully sequential in each connection. Now a full
batch of inserts is done per connection in parallel.
This could still be improved a lot. Currently each connection will fully
wait for all inserts to be done in the current batch before starting
another one.
2022-01-04 10:15:57 +01:00
Christian Kamm
3f242aa3b6
Fix startup issue around rooted slots
2022-01-04 10:14:30 +01:00
Christian Kamm
d5395a3b75
Add config flag for deleting old data
...
Keeping it is very useful for testing.
2022-01-03 10:36:11 +01:00
Christian Kamm
f0c26bb8fb
Fix write_version from multiple servers and startup
...
Each node has an internal write_version counter. To deduplicate events
and produce a consistent database snapshot independent of the internal
count, we map it a per-slot-and-pubkey write version.
To do this correctly, we can only process account writes for slots where
all account writes will be received. This is a problem during startup,
where it's unclear what slots we have missed data for.
To make startup easier, let the plugin keep track of the highest slot
number that account writes have happened for, and send it when a new
consumer connects.
2022-01-03 10:36:11 +01:00
Christian Kamm
4236805b29
Fix problem with write_version being node-specific
...
To work around that, build our own write_version based on the number of
writes to a pubkey each slot.
2021-12-20 16:08:36 +01:00
Christian Kamm
a7a53dd6f7
Heroku: Have two running instances
2021-12-20 12:40:39 +01:00
Christian Kamm
1b885355fd
Fix log messages for slot update retries
2021-12-20 11:28:53 +01:00
Christian Kamm
76e14d0300
Fix initial snapshot #1
...
Previously the initial snapshot could be returned for a discarded slot.
Now a finalized-commitment snapshot is retrieved.
2021-12-13 16:48:33 +01:00
Christian Kamm
17d6787bf1
Add TLS domain name argument
2021-11-30 17:15:41 +01:00
Christian Kamm
60ca647650
Do the slot updates before wiping non-rooted account writes
2021-11-26 11:59:23 +01:00
Christian Kamm
c65baee04f
If a slot becomes rooted, mark preceeding non-uncle slots rooted.
...
This fixes the _rooted views, which previously would skip data from
slots that were never actively confirmed to be rooted.
2021-11-26 11:54:08 +01:00
Christian Kamm
a941f486ed
Fix bug introduced by deleting older slots
...
Deleting older slots meant that older (but still most recent) account
writes that had their slot deleted would no longer show up in views.
Now the guarantee is that an account write for a missing slot that is
smaller than the top slot in the slots table is rooted.
To guarantee that, account writes for non-rooted slots that preceed
rooted slots are deleted now.
2021-11-25 10:38:55 +01:00
Christian Kamm
3cdbeddf8e
Update solana version to 1.8.5
2021-11-19 11:59:59 +01:00
Christian Kamm
e34ee0489c
Update solana version to 1.8.3
2021-11-19 11:57:47 +01:00
Christian Kamm
2abe2cca03
Heroku: Add aptfile to pull in build-time dependencies
2021-11-19 11:32:45 +01:00
Christian Kamm
c82caf58c8
Fix dependency on jsonrpc fork
2021-11-19 11:17:25 +01:00
Christian Kamm
3912f6142f
Add setup for heroku
2021-11-19 10:44:30 +01:00
Christian Kamm
b65815e4d4
gRPC: Optionally use TLS
2021-11-18 14:21:17 +01:00
Christian Kamm
4c9b782d1f
Mango: Drop all views in drop_views.sql
2021-11-15 16:18:27 +01:00
Christian Kamm
5ef1a19300
Mango: Update views and PnL query
2021-11-15 16:15:18 +01:00
Christian Kamm
dbfbae1a49
Delay snapshot more, add metric on slot insertion queue
2021-11-15 15:49:59 +01:00
Christian Kamm
dde006243b
Minor clippy fixes
2021-11-15 15:37:29 +01:00
Christian Kamm
8f6bd7bc22
Postgres: Allow parallel slot update connections
...
It would be nicer if many of the writes could happen on the same
connection. Separate connections are only necessary if transactions are
used.
2021-11-15 15:28:18 +01:00
Christian Kamm
343b5408bc
Postgres: Add option to allow invalid certificates
2021-11-14 13:14:19 +01:00
Christian Kamm
c4bf05b1d2
Postgres: Use native-tls to allow TLS connections
2021-11-13 09:32:32 +01:00
Christian Kamm
5e3db3203a
Readme: Add more motivation / context
2021-11-09 18:41:34 +01:00
Christian Kamm
f1608409ca
Documentation
2021-11-09 15:07:42 +01:00
Christian Kamm
71071d9b52
Add example plugin config file
2021-11-09 14:52:05 +01:00
Christian Kamm
456b19ea06
Add example SQL script for computing mango PnL
2021-11-09 14:50:08 +01:00
Christian Kamm
d889b5ff5b
Move postgres numeric support to separate file
2021-11-09 14:32:58 +01:00
Christian Kamm
3a0e2535f0
Use anyhow::Result over Result<T, anyhow::Error>
2021-11-09 14:27:09 +01:00
Christian Kamm
0cdee40851
Metrics cleanup
2021-11-09 14:23:55 +01:00
Christian Kamm
4d5f0c5d10
Basic metrics
2021-11-09 09:00:00 +01:00
Christian Kamm
029cbd2fb5
Allow multiple grpc sources
2021-11-08 15:48:50 +01:00
Christian Kamm
f43aaf0f55
Don't process old account writes
...
This will help when receiving data from multiple validator nodes
2021-11-08 15:31:35 +01:00
Christian Kamm
35c0c5da23
Fix delayed snapshots
...
The snapshot was requested too early before.
2021-11-08 14:26:37 +01:00
Christian Kamm
11f4d6bd67
Use separate pubkey table
2021-11-08 14:10:48 +01:00
Christian Kamm
a17f142875
Use enum for slot status
2021-11-08 12:44:01 +01:00
Christian Kamm
7f6e25e107
Wipe old slots from table, now nothing accumulates
2021-11-08 12:09:14 +01:00
Christian Kamm
47f8bb823d
Remove unnecessary trailing backslashes from multiline string literals
2021-11-08 11:59:57 +01:00