f25c49d983
Configurable max account write queue size
2022-03-16 11:04:30 +01:00
80423fdac9
Track "is_selected" to denote closed/reused accounts
2022-01-28 10:12:01 +01:00
185bf4f9cf
Postgres: Fix account_write cleanup query
...
Avoid misbehaving in situations where there's an account write for a
slot that doesn't yet have an entry in the slots table by limiting the
slot numbers.
2022-01-20 15:28:16 +01:00
dc80d8266c
Process postgres cleanup jobs separately
...
Executing them for every rooted slot may overwhelm the database.
2022-01-20 14:49:51 +01:00
7d8fd45de4
Add support for updating a monitoring table
...
That way connector interruptions can be seen directly form the target
database.
2022-01-06 12:54:35 +01:00
b1f18c0b13
Fix data deletion not being enableable
2022-01-05 16:02:34 +01:00
5b5eaba4ff
Pipeline SQL inserts for better performance
...
Previously insert speed was primarily limited by roundtrip time to the
database since they were fully sequential in each connection. Now a full
batch of inserts is done per connection in parallel.
This could still be improved a lot. Currently each connection will fully
wait for all inserts to be done in the current batch before starting
another one.
2022-01-04 10:15:57 +01:00
d5395a3b75
Add config flag for deleting old data
...
Keeping it is very useful for testing.
2022-01-03 10:36:11 +01:00
1b885355fd
Fix log messages for slot update retries
2021-12-20 11:28:53 +01:00
60ca647650
Do the slot updates before wiping non-rooted account writes
2021-11-26 11:59:23 +01:00
c65baee04f
If a slot becomes rooted, mark preceeding non-uncle slots rooted.
...
This fixes the _rooted views, which previously would skip data from
slots that were never actively confirmed to be rooted.
2021-11-26 11:54:08 +01:00
a941f486ed
Fix bug introduced by deleting older slots
...
Deleting older slots meant that older (but still most recent) account
writes that had their slot deleted would no longer show up in views.
Now the guarantee is that an account write for a missing slot that is
smaller than the top slot in the slots table is rooted.
To guarantee that, account writes for non-rooted slots that preceed
rooted slots are deleted now.
2021-11-25 10:38:55 +01:00
dbfbae1a49
Delay snapshot more, add metric on slot insertion queue
2021-11-15 15:49:59 +01:00
dde006243b
Minor clippy fixes
2021-11-15 15:37:29 +01:00
8f6bd7bc22
Postgres: Allow parallel slot update connections
...
It would be nicer if many of the writes could happen on the same
connection. Separate connections are only necessary if transactions are
used.
2021-11-15 15:28:18 +01:00
343b5408bc
Postgres: Add option to allow invalid certificates
2021-11-14 13:14:19 +01:00
c4bf05b1d2
Postgres: Use native-tls to allow TLS connections
2021-11-13 09:32:32 +01:00
3a0e2535f0
Use anyhow::Result over Result<T, anyhow::Error>
2021-11-09 14:27:09 +01:00
0cdee40851
Metrics cleanup
2021-11-09 14:23:55 +01:00
4d5f0c5d10
Basic metrics
2021-11-09 09:00:00 +01:00
11f4d6bd67
Use separate pubkey table
2021-11-08 14:10:48 +01:00
a17f142875
Use enum for slot status
2021-11-08 12:44:01 +01:00
7f6e25e107
Wipe old slots from table, now nothing accumulates
2021-11-08 12:09:14 +01:00
47f8bb823d
Remove unnecessary trailing backslashes from multiline string literals
2021-11-08 11:59:57 +01:00
d366581826
Make more postgres settings configurable
2021-11-08 11:30:17 +01:00
d9cb30c2b3
Make parallel postgres connections configurable
2021-11-08 10:05:17 +01:00
e67dd6c235
Split into connector library, and raw and mango binaries
2021-11-08 09:45:41 +01:00
Christian Kamm
Christian Kamm