solana-accountsdb-connector/connector-raw/example-config.toml

[source]
dedup_queue_size = 50000
rpc_ws_url = ""

[[source.grpc_sources]]
name = "server"
connection_string = "http://[::1]:10000"
retry_connection_sleep_secs = 30

#[source.grpc_sources.tls]
#ca_cert_path = "ca.pem"
#client_cert_path = "client.pem"
#client_key_path = "client.pem"
#domain_name = "example.com"

[source.snapshot]
rpc_http_url = ""
program_id = ""

[postgres_target]
connection_string =  "host=/var/run/postgresql"
account_write_connection_count = 4
account_write_max_batch_size = 10
account_write_max_queue_size = 10000
slot_update_connection_count = 2
retry_query_max_count = 3
retry_query_sleep_secs = 5
retry_connection_sleep_secs = 30
fatal_connection_timeout_secs = 600
allow_invalid_certs = false
delete_old_data = true
monitoring_name = "example"
monitoring_update_interval_secs = 30
cleanup_interval_secs = 10
Config: Separate out source config We want to be able to not have a postgres target 2022-03-23 00:51:17 -07:00			`[source]`
			`dedup_queue_size = 50000`
Make more postgres settings configurable 2021-11-08 02:26:01 -08:00			`rpc_ws_url = ""`
grpc configurability 2021-11-08 02:42:22 -08:00
Config: Separate out source config We want to be able to not have a postgres target 2022-03-23 00:51:17 -07:00			`[[source.grpc_sources]]`
Documentation 2021-11-09 06:07:42 -08:00			`name = "server"`
grpc configurability 2021-11-08 02:42:22 -08:00			`connection_string = "http://[::1]:10000"`
			`retry_connection_sleep_secs = 30`

Config: Separate out source config We want to be able to not have a postgres target 2022-03-23 00:51:17 -07:00			`#[source.grpc_sources.tls]`
gRPC: Optionally use TLS 2021-11-18 05:21:17 -08:00			`#ca_cert_path = "ca.pem"`
			`#client_cert_path = "client.pem"`
			`#client_key_path = "client.pem"`
Add TLS domain name argument 2021-11-30 08:15:41 -08:00			`#domain_name = "example.com"`
gRPC: Optionally use TLS 2021-11-18 05:21:17 -08:00
Config: Separate out source config We want to be able to not have a postgres target 2022-03-23 00:51:17 -07:00			`[source.snapshot]`
grpc configurability 2021-11-08 02:42:22 -08:00			`rpc_http_url = ""`
Make more postgres settings configurable 2021-11-08 02:26:01 -08:00			`program_id = ""`

			`[postgres_target]`
			`connection_string = "host=/var/run/postgresql"`
			`account_write_connection_count = 4`
Pipeline SQL inserts for better performance Previously insert speed was primarily limited by roundtrip time to the database since they were fully sequential in each connection. Now a full batch of inserts is done per connection in parallel. This could still be improved a lot. Currently each connection will fully wait for all inserts to be done in the current batch before starting another one. 2022-01-04 01:15:22 -08:00			`account_write_max_batch_size = 10`
Configurable max account write queue size 2022-03-16 03:04:30 -07:00			`account_write_max_queue_size = 10000`
Postgres: Allow parallel slot update connections It would be nicer if many of the writes could happen on the same connection. Separate connections are only necessary if transactions are used. 2021-11-15 06:25:04 -08:00			`slot_update_connection_count = 2`
Make more postgres settings configurable 2021-11-08 02:26:01 -08:00			`retry_query_max_count = 3`
			`retry_query_sleep_secs = 5`
			`retry_connection_sleep_secs = 30`
			`fatal_connection_timeout_secs = 600`
Postgres: Add option to allow invalid certificates 2021-11-14 04:14:19 -08:00			`allow_invalid_certs = false`
Add config flag for deleting old data Keeping it is very useful for testing. 2022-01-03 01:33:26 -08:00			`delete_old_data = true`
Add support for updating a monitoring table That way connector interruptions can be seen directly form the target database. 2022-01-06 03:54:35 -08:00			`monitoring_name = "example"`
Process postgres cleanup jobs separately Executing them for every rooted slot may overwhelm the database. 2022-01-20 05:49:51 -08:00			`monitoring_update_interval_secs = 30`
Update example config files 2022-01-20 07:20:16 -08:00			`cleanup_interval_secs = 10`