Commit Graph

15 Commits

Author SHA1 Message Date
Christian Kamm cdf6d54156 Config: Separate out source config
We want to be able to not have a postgres target
2022-03-23 08:51:17 +01:00
Christian Kamm f25c49d983 Configurable max account write queue size 2022-03-16 11:04:30 +01:00
Christian Kamm e4e3fb2cac Update example config files 2022-01-20 16:20:16 +01:00
Christian Kamm dc80d8266c Process postgres cleanup jobs separately
Executing them for every rooted slot may overwhelm the database.
2022-01-20 14:49:51 +01:00
Christian Kamm 7d8fd45de4 Add support for updating a monitoring table
That way connector interruptions can be seen directly form the target
database.
2022-01-06 12:54:35 +01:00
Christian Kamm 5b5eaba4ff Pipeline SQL inserts for better performance
Previously insert speed was primarily limited by roundtrip time to the
database since they were fully sequential in each connection. Now a full
batch of inserts is done per connection in parallel.

This could still be improved a lot. Currently each connection will fully
wait for all inserts to be done in the current batch before starting
another one.
2022-01-04 10:15:57 +01:00
Christian Kamm d5395a3b75 Add config flag for deleting old data
Keeping it is very useful for testing.
2022-01-03 10:36:11 +01:00
Christian Kamm 17d6787bf1 Add TLS domain name argument 2021-11-30 17:15:41 +01:00
Christian Kamm b65815e4d4 gRPC: Optionally use TLS 2021-11-18 14:21:17 +01:00
Christian Kamm 8f6bd7bc22 Postgres: Allow parallel slot update connections
It would be nicer if many of the writes could happen on the same
connection. Separate connections are only necessary if transactions are
used.
2021-11-15 15:28:18 +01:00
Christian Kamm 343b5408bc Postgres: Add option to allow invalid certificates 2021-11-14 13:14:19 +01:00
Christian Kamm f1608409ca Documentation 2021-11-09 15:07:42 +01:00
Christian Kamm 029cbd2fb5 Allow multiple grpc sources 2021-11-08 15:48:50 +01:00
Christian Kamm 0424c3c9d6 grpc configurability 2021-11-08 11:42:22 +01:00
Christian Kamm d366581826 Make more postgres settings configurable 2021-11-08 11:30:17 +01:00