190528b722
Add process name label, fix some metric types
2022-10-08 12:57:47 +01:00
bd7f80f201
Sanitize metric names
2022-10-07 13:15:27 +01:00
e2729306b8
cargo fmt
2022-10-07 11:46:19 +01:00
2881378b7f
Refactor metrics
2022-10-07 11:44:53 +01:00
1d1f462cf6
Rename count metrics
2022-10-06 01:33:07 +01:00
d14b4081aa
cargo fmt
2022-10-05 23:53:20 +01:00
dbf28b74be
Merge branch 'master' of github.com:blockworks-foundation/solana-accountsdb-connector
2022-10-05 23:50:28 +01:00
ae7ffc1dcb
Add prometheus metrics to fills
2022-10-05 23:50:21 +01:00
1bebc2f6bc
Aggregate t event metrics ( #3 )
...
* Aggregate t event metrics
* Misc
* Misc
2022-09-21 16:45:55 +02:00
2f09d0a8f2
Add docker build and fly deployment actions ( #2 )
...
* Test dockerfile with caching
* Fix action
* Remove tags
* Use sccache release
* Use max mode
* idk
* Try no runtime build
* Use cargo-chef
* Test rebuild
* Revert "Test rebuild"
This reverts commit d7e1654c63a0cc99a3898c1b17df0cddd13ac0bd.
* Publish image to GCR
* Trigger build
* Read config from env
* Fix typo
* Bind to correct address
* Lint
* Add fly config
* Switch trigger branch to main
* Use both common branch names
2022-09-20 23:01:24 +02:00
3dfb430416
Add zstd decoding
2022-08-25 11:51:38 +01:00
6b280445cc
Try 1.10.35
2022-08-24 17:19:40 +01:00
f39fcdfba9
Fix lib for 1.10 compatibility
2022-08-24 13:47:16 +01:00
18fc4afd93
Bump ci rust version
2022-08-24 13:16:34 +01:00
495b5371ec
Update solana and mango deps
2022-08-24 12:56:50 +01:00
8dde08d377
service-fills: add slote and write version to fill update
2022-04-08 05:23:52 +02:00
b41c094aba
merge fill service
2022-03-28 20:01:16 +02:00
c2f0c0ac05
Fix write_version sanity checking for 1.9
...
In 1.9, the write_version will no longer be strictly monotonic across
accounts.
2022-03-28 15:37:53 +02:00
1be0cc8fb6
pnl-service: first working version
2022-03-23 18:39:42 +01:00
cdf6d54156
Config: Separate out source config
...
We want to be able to not have a postgres target
2022-03-23 08:51:17 +01:00
3beaac671d
Add memory_target and chain_data
...
This will be useful for storing account data in memory.
2022-03-23 08:19:19 +01:00
42f07d58a6
Keep slot, write_version etc as u64, convert only for postgres
...
This makes the conversion to i64 something that's local to the postgres
target, and doesn't affect unrelated code.
2022-03-23 08:18:02 +01:00
83e9c54fc0
Fix talking to older plugins
...
The previous commit only looked like it worked due to stale generated
code.
2022-03-23 08:16:02 +01:00
2b1bf634c7
grpc proto: Change back to AccountsDb name to avoid incompatibility
...
Existing plugin deployments must continue to work with the new version
of the connector.
2022-03-22 12:54:38 +01:00
b6cefddef2
Compatibility fixes for 1.9
...
- Renamed to geyser everywhere and updated the crates
- Changed FullRpcClient to AccountsDataClient
2022-03-22 08:36:59 +00:00
f25c49d983
Configurable max account write queue size
2022-03-16 11:04:30 +01:00
80423fdac9
Track "is_selected" to denote closed/reused accounts
2022-01-28 10:12:01 +01:00
66266a3469
Update solana version to 1.8.14
2022-01-26 10:08:53 +01:00
185bf4f9cf
Postgres: Fix account_write cleanup query
...
Avoid misbehaving in situations where there's an account write for a
slot that doesn't yet have an entry in the slots table by limiting the
slot numbers.
2022-01-20 15:28:16 +01:00
dc80d8266c
Process postgres cleanup jobs separately
...
Executing them for every rooted slot may overwhelm the database.
2022-01-20 14:49:51 +01:00
cf26557f7d
Update to solana 1.8.12
2022-01-12 12:00:06 +01:00
afc08d8266
Fix metadata
2022-01-07 08:21:00 +01:00
7d8fd45de4
Add support for updating a monitoring table
...
That way connector interruptions can be seen directly form the target
database.
2022-01-06 12:54:35 +01:00
b1f18c0b13
Fix data deletion not being enableable
2022-01-05 16:02:34 +01:00
fb7e0d14e2
Update to solana 1.8.11
2022-01-04 18:39:48 +01:00
5b5eaba4ff
Pipeline SQL inserts for better performance
...
Previously insert speed was primarily limited by roundtrip time to the
database since they were fully sequential in each connection. Now a full
batch of inserts is done per connection in parallel.
This could still be improved a lot. Currently each connection will fully
wait for all inserts to be done in the current batch before starting
another one.
2022-01-04 10:15:57 +01:00
3f242aa3b6
Fix startup issue around rooted slots
2022-01-04 10:14:30 +01:00
d5395a3b75
Add config flag for deleting old data
...
Keeping it is very useful for testing.
2022-01-03 10:36:11 +01:00
f0c26bb8fb
Fix write_version from multiple servers and startup
...
Each node has an internal write_version counter. To deduplicate events
and produce a consistent database snapshot independent of the internal
count, we map it a per-slot-and-pubkey write version.
To do this correctly, we can only process account writes for slots where
all account writes will be received. This is a problem during startup,
where it's unclear what slots we have missed data for.
To make startup easier, let the plugin keep track of the highest slot
number that account writes have happened for, and send it when a new
consumer connects.
2022-01-03 10:36:11 +01:00
4236805b29
Fix problem with write_version being node-specific
...
To work around that, build our own write_version based on the number of
writes to a pubkey each slot.
2021-12-20 16:08:36 +01:00
1b885355fd
Fix log messages for slot update retries
2021-12-20 11:28:53 +01:00
76e14d0300
Fix initial snapshot #1
...
Previously the initial snapshot could be returned for a discarded slot.
Now a finalized-commitment snapshot is retrieved.
2021-12-13 16:48:33 +01:00
17d6787bf1
Add TLS domain name argument
2021-11-30 17:15:41 +01:00
60ca647650
Do the slot updates before wiping non-rooted account writes
2021-11-26 11:59:23 +01:00
c65baee04f
If a slot becomes rooted, mark preceeding non-uncle slots rooted.
...
This fixes the _rooted views, which previously would skip data from
slots that were never actively confirmed to be rooted.
2021-11-26 11:54:08 +01:00
a941f486ed
Fix bug introduced by deleting older slots
...
Deleting older slots meant that older (but still most recent) account
writes that had their slot deleted would no longer show up in views.
Now the guarantee is that an account write for a missing slot that is
smaller than the top slot in the slots table is rooted.
To guarantee that, account writes for non-rooted slots that preceed
rooted slots are deleted now.
2021-11-25 10:38:55 +01:00
3cdbeddf8e
Update solana version to 1.8.5
2021-11-19 11:59:59 +01:00
e34ee0489c
Update solana version to 1.8.3
2021-11-19 11:57:47 +01:00
b65815e4d4
gRPC: Optionally use TLS
2021-11-18 14:21:17 +01:00
dbfbae1a49
Delay snapshot more, add metric on slot insertion queue
2021-11-15 15:49:59 +01:00
dde006243b
Minor clippy fixes
2021-11-15 15:37:29 +01:00
8f6bd7bc22
Postgres: Allow parallel slot update connections
...
It would be nicer if many of the writes could happen on the same
connection. Separate connections are only necessary if transactions are
used.
2021-11-15 15:28:18 +01:00
343b5408bc
Postgres: Add option to allow invalid certificates
2021-11-14 13:14:19 +01:00
c4bf05b1d2
Postgres: Use native-tls to allow TLS connections
2021-11-13 09:32:32 +01:00
d889b5ff5b
Move postgres numeric support to separate file
2021-11-09 14:32:58 +01:00
3a0e2535f0
Use anyhow::Result over Result<T, anyhow::Error>
2021-11-09 14:27:09 +01:00
0cdee40851
Metrics cleanup
2021-11-09 14:23:55 +01:00
4d5f0c5d10
Basic metrics
2021-11-09 09:00:00 +01:00
029cbd2fb5
Allow multiple grpc sources
2021-11-08 15:48:50 +01:00
f43aaf0f55
Don't process old account writes
...
This will help when receiving data from multiple validator nodes
2021-11-08 15:31:35 +01:00
35c0c5da23
Fix delayed snapshots
...
The snapshot was requested too early before.
2021-11-08 14:26:37 +01:00
11f4d6bd67
Use separate pubkey table
2021-11-08 14:10:48 +01:00
a17f142875
Use enum for slot status
2021-11-08 12:44:01 +01:00
7f6e25e107
Wipe old slots from table, now nothing accumulates
2021-11-08 12:09:14 +01:00
47f8bb823d
Remove unnecessary trailing backslashes from multiline string literals
2021-11-08 11:59:57 +01:00
0424c3c9d6
grpc configurability
2021-11-08 11:42:22 +01:00
d366581826
Make more postgres settings configurable
2021-11-08 11:30:17 +01:00
d9cb30c2b3
Make parallel postgres connections configurable
2021-11-08 10:05:17 +01:00
cbe9e16b62
Configurable program id
2021-11-08 09:57:56 +01:00
e67dd6c235
Split into connector library, and raw and mango binaries
2021-11-08 09:45:41 +01:00
Riordan Panayides
waterquarks
riordanp
Maximilian Schneider
Christian Kamm
Linus Kendall
Christian Kamm