5bca8add90
Merge branch 'fix/account-key-sql' into improvement/schema-fixes-hetzner
2024-02-24 18:55:25 +01:00
d7f0864e89
fix account_key vs key on insert
2024-02-24 18:52:04 +01:00
bab25959f6
postgres schema fixes after moving to self-operated PostgreSQL ( #57 )
...
* remove indices from migration.sql which here never created
* BREAKING: make transaction_id the primary key in transactions lookup table
* use varchar in postgres.rs
* add "NOT NULL" constraint wherever possible
* fix small accounts lookup table naming
* fly app for migration
see https://github.com/blockworks-foundation/BankingStageErrorsTrackingSidecar/issues/56
* autovacuum settings
* databse init script
* explain query_user role
* tune toast_tuple_target and comment on page size
* minor fixes
* dba queries
* jot down the fly machine config
* restore create schema
* tweak storage param for postgres 15
2024-02-24 18:29:52 +01:00
477a8b5b7a
Merge remote-tracking branch 'origin/main' into improvement/schema-fixes-hetzner
2024-02-24 18:29:25 +01:00
2b13bab8d7
tweak storage param for postgres 15
2024-02-24 18:16:39 +01:00
f48d06664a
restore create schema
2024-02-23 17:54:29 +01:00
efe1036e91
jot down the fly machine config
2024-02-23 16:44:52 +01:00
718345a271
dba queries
2024-02-23 16:41:34 +01:00
8966b1ac28
minor fixes
2024-02-23 16:41:29 +01:00
cb5f7dea64
tune toast_tuple_target and comment on page size
2024-02-23 15:48:19 +01:00
bba046a0a0
explain query_user role
2024-02-23 15:08:23 +01:00
74af277ea0
databse init script
2024-02-23 14:15:57 +01:00
50dadc0071
fix percentiles missing if values are dense ( #60 )
2024-02-23 10:33:47 +01:00
2c51ea8abf
issue #58 : fix INSERT INTO deadlock by ordering data ( #59 )
...
* issue #58 : fix INSERT INTO deadlock by ordering data
* order transactions store in INSERT INTO batch to fix https://github.com/blockworks-foundation/BankingStageErrorsTrackingSidecar/issues/58
* log tx info batch size
* workaround for cargo-chef MSRV issue
2024-02-12 17:10:40 +01:00
0627a1caea
autovacuum settings
2024-02-08 13:56:44 +01:00
81e5580a7c
fly app for migration
...
see https://github.com/blockworks-foundation/BankingStageErrorsTrackingSidecar/issues/56
2024-02-08 11:13:05 +01:00
2fc9b2f3f2
fix small accounts lookup table naming
2024-02-08 09:02:59 +01:00
b33925e2b3
add "NOT NULL" constraint wherever possible
2024-02-08 08:32:43 +01:00
8fa88d7377
use varchar in postgres.rs
2024-02-08 08:14:46 +01:00
9c7d769afc
BREAKING: make transaction_id the primary key in transactions lookup table
2024-02-08 08:02:03 +01:00
c636847d90
remove indices from migration.sql which here never created
2024-02-08 07:35:44 +01:00
9ca22c9a46
reduce LIMIT_LATEST_TXS_PER_ACCOUNT ( #54 )
2024-02-03 09:19:10 +01:00
99427a9253
Revert "partition accounts_map_transaction_latest table ( #47 )"
...
This reverts commit 7a577a43fa
2024-02-01 17:56:20 +01:00
a065914e62
Revert "build amt_latest tx array with latest at leftmost position ( #51 )"
...
This reverts commit a981ff4b2b
2024-02-01 17:55:38 +01:00
6383a8a24e
Revert "clear postgres session and use hardcoded temptable names ( #52 )" ( #53 )
...
This reverts commit 7afa8fc45f
2024-02-01 09:44:23 +01:00
7afa8fc45f
clear postgres session and use hardcoded temptable names ( #52 )
...
* clear pg session, redesign insert_accounts_for_transaction
* redesign insert_transactions_for_block
* redesign save_account_usage_in_block
* redesign create_transaction_ids
* redesign create_accounts_for_transaction
* narrow resources cleared in clear_session
2024-02-01 08:07:19 +01:00
7a577a43fa
partition accounts_map_transaction_latest table ( #47 )
2024-01-31 10:49:16 +01:00
a981ff4b2b
build amt_latest tx array with latest at leftmost position ( #51 )
2024-01-31 10:39:29 +01:00
65b27ce930
Adding metrics for gSFA old and new
2024-01-29 18:19:22 +01:00
797f638f44
Adding metrics to save block info
2024-01-29 15:33:05 +01:00
a9aebba6b5
Fixing build issues
2024-01-29 15:11:16 +01:00
343bb70428
Fix minor bug
2024-01-29 15:09:03 +01:00
f0e37643c5
Adding more metrics, and having multiple postgres sessions
2024-01-29 14:51:21 +01:00
681b042ba1
Optimize account saving
2024-01-29 14:14:07 +01:00
e8cebe89fd
Increasing limit for number of blocks that could be saved
2024-01-29 12:45:13 +01:00
92bf5f84d2
Limiting the number of heavy accounts in the block
2024-01-29 12:03:28 +01:00
295277fbf0
Reverting adding transaction data
2024-01-29 11:54:33 +01:00
8b37b3e2d9
Adding more metrics
2024-01-29 11:29:16 +01:00
829cc7952a
Alt processing 2024 01 26 ( #50 )
...
* Banking stage sidecar deadlock (#48 )
* Properly loading the ALTs
* Adding a timeout for ALTs task list
* More optimization for ALTs
* Adding timeout for getMA
* Avoid RwLocks because they cause deadlocks
* Fixing the deadlock
* cargo fmt and minor changes
* Fixing up metrics
* minor changes
* Solving issue with accounts metric
* Avoid waiting alts while processing block
* Fixing the metrics, minor optimizations
* fix double insertion of transaction accounts
2024-01-29 10:54:11 +01:00
566a107145
removed FK contraint on error_code ( #49 )
2024-01-27 09:35:10 +01:00
64d607155c
Creating seperate task to save transaction accounts
2024-01-22 17:48:34 +01:00
abbf805726
reduce LIMIT_LATEST_TXS_PER_ACCOUNT to 100 ( #43 )
2024-01-11 18:15:38 +01:00
fbfef4db06
slow sqlfunction ( #42 )
...
* rework array_dedup_append function
* array_dedup_append: implemented faster version
uses nested array remove for merge+deduplication
2024-01-11 17:39:44 +01:00
fd99ea1f0a
simplify insertinto queries ( #41 )
...
* remove extra subquery
* handle block already exist case
* remove trailing semicolons
2024-01-10 16:56:35 +01:00
f5f9c99375
multiple write dbsessions ( #40 )
...
* postgres connection handling:
- use dedicated connections
- set work_mem
* serialize writes
* brush send_block_info_to_buffer logging
* remove sleep
* brush log
* cleanu
2024-01-10 16:39:05 +01:00
f0727de608
log metrics for postgres write statements ( #39 )
...
* add debug logs for query execution
* use tokio instant
* words
2024-01-10 14:13:19 +01:00
527fe545c1
resubscribe banking stage errors every 10 mins incase of inactivity
2024-01-09 15:35:15 +01:00
5c3edc836f
add table with latest 1000 txs per acc_id ( #38 )
...
* wip
* use fancy postgres method
* cleanup
* fix logs
* format
* simpler function
* Revert "simpler function"
This reverts commit 57120b9ce8
2024-01-09 15:12:03 +01:00
77ae2a38a2
Making missing sig into char(88)
2024-01-08 16:14:02 +01:00
c8c59135f8
Reverting sigature to char(88) and account key to char(44)
2024-01-05 17:08:23 +01:00
GroovieGermanikus
Groovie | Mango
godmodegalactus
galactus