d7f0864e89
fix account_key vs key on insert
2024-02-24 18:52:04 +01:00
bab25959f6
postgres schema fixes after moving to self-operated PostgreSQL ( #57 )
...
* remove indices from migration.sql which here never created
* BREAKING: make transaction_id the primary key in transactions lookup table
* use varchar in postgres.rs
* add "NOT NULL" constraint wherever possible
* fix small accounts lookup table naming
* fly app for migration
see https://github.com/blockworks-foundation/BankingStageErrorsTrackingSidecar/issues/56
* autovacuum settings
* databse init script
* explain query_user role
* tune toast_tuple_target and comment on page size
* minor fixes
* dba queries
* jot down the fly machine config
* restore create schema
* tweak storage param for postgres 15
2024-02-24 18:29:52 +01:00
50dadc0071
fix percentiles missing if values are dense ( #60 )
2024-02-23 10:33:47 +01:00
2c51ea8abf
issue #58 : fix INSERT INTO deadlock by ordering data ( #59 )
...
* issue #58 : fix INSERT INTO deadlock by ordering data
* order transactions store in INSERT INTO batch to fix https://github.com/blockworks-foundation/BankingStageErrorsTrackingSidecar/issues/58
* log tx info batch size
* workaround for cargo-chef MSRV issue
2024-02-12 17:10:40 +01:00
9ca22c9a46
reduce LIMIT_LATEST_TXS_PER_ACCOUNT ( #54 )
2024-02-03 09:19:10 +01:00
99427a9253
Revert "partition accounts_map_transaction_latest table ( #47 )"
...
This reverts commit 7a577a43fa
2024-02-01 17:56:20 +01:00
a065914e62
Revert "build amt_latest tx array with latest at leftmost position ( #51 )"
...
This reverts commit a981ff4b2b
2024-02-01 17:55:38 +01:00
6383a8a24e
Revert "clear postgres session and use hardcoded temptable names ( #52 )" ( #53 )
...
This reverts commit 7afa8fc45f
2024-02-01 09:44:23 +01:00
7afa8fc45f
clear postgres session and use hardcoded temptable names ( #52 )
...
* clear pg session, redesign insert_accounts_for_transaction
* redesign insert_transactions_for_block
* redesign save_account_usage_in_block
* redesign create_transaction_ids
* redesign create_accounts_for_transaction
* narrow resources cleared in clear_session
2024-02-01 08:07:19 +01:00
7a577a43fa
partition accounts_map_transaction_latest table ( #47 )
2024-01-31 10:49:16 +01:00
a981ff4b2b
build amt_latest tx array with latest at leftmost position ( #51 )
2024-01-31 10:39:29 +01:00
65b27ce930
Adding metrics for gSFA old and new
2024-01-29 18:19:22 +01:00
797f638f44
Adding metrics to save block info
2024-01-29 15:33:05 +01:00
a9aebba6b5
Fixing build issues
2024-01-29 15:11:16 +01:00
343bb70428
Fix minor bug
2024-01-29 15:09:03 +01:00
f0e37643c5
Adding more metrics, and having multiple postgres sessions
2024-01-29 14:51:21 +01:00
681b042ba1
Optimize account saving
2024-01-29 14:14:07 +01:00
e8cebe89fd
Increasing limit for number of blocks that could be saved
2024-01-29 12:45:13 +01:00
92bf5f84d2
Limiting the number of heavy accounts in the block
2024-01-29 12:03:28 +01:00
295277fbf0
Reverting adding transaction data
2024-01-29 11:54:33 +01:00
8b37b3e2d9
Adding more metrics
2024-01-29 11:29:16 +01:00
829cc7952a
Alt processing 2024 01 26 ( #50 )
...
* Banking stage sidecar deadlock (#48 )
* Properly loading the ALTs
* Adding a timeout for ALTs task list
* More optimization for ALTs
* Adding timeout for getMA
* Avoid RwLocks because they cause deadlocks
* Fixing the deadlock
* cargo fmt and minor changes
* Fixing up metrics
* minor changes
* Solving issue with accounts metric
* Avoid waiting alts while processing block
* Fixing the metrics, minor optimizations
* fix double insertion of transaction accounts
2024-01-29 10:54:11 +01:00
566a107145
removed FK contraint on error_code ( #49 )
2024-01-27 09:35:10 +01:00
64d607155c
Creating seperate task to save transaction accounts
2024-01-22 17:48:34 +01:00
abbf805726
reduce LIMIT_LATEST_TXS_PER_ACCOUNT to 100 ( #43 )
2024-01-11 18:15:38 +01:00
fbfef4db06
slow sqlfunction ( #42 )
...
* rework array_dedup_append function
* array_dedup_append: implemented faster version
uses nested array remove for merge+deduplication
2024-01-11 17:39:44 +01:00
fd99ea1f0a
simplify insertinto queries ( #41 )
...
* remove extra subquery
* handle block already exist case
* remove trailing semicolons
2024-01-10 16:56:35 +01:00
f5f9c99375
multiple write dbsessions ( #40 )
...
* postgres connection handling:
- use dedicated connections
- set work_mem
* serialize writes
* brush send_block_info_to_buffer logging
* remove sleep
* brush log
* cleanu
2024-01-10 16:39:05 +01:00
f0727de608
log metrics for postgres write statements ( #39 )
...
* add debug logs for query execution
* use tokio instant
* words
2024-01-10 14:13:19 +01:00
527fe545c1
resubscribe banking stage errors every 10 mins incase of inactivity
2024-01-09 15:35:15 +01:00
5c3edc836f
add table with latest 1000 txs per acc_id ( #38 )
...
* wip
* use fancy postgres method
* cleanup
* fix logs
* format
* simpler function
* Revert "simpler function"
This reverts commit 57120b9ce8
2024-01-09 15:12:03 +01:00
77ae2a38a2
Making missing sig into char(88)
2024-01-08 16:14:02 +01:00
c8c59135f8
Reverting sigature to char(88) and account key to char(44)
2024-01-05 17:08:23 +01:00
050bbbbb76
Fix/fly cleanup2 ( #36 )
...
* clippy+fmt
* add run-service-and-cleanup.sh to docker
2024-01-05 11:16:27 +01:00
f877a3a0a9
run cleanupjob on fly ( #35 )
...
* clippy+fmt
* poor-man's integration of cleanup job
- runs every 5 hours
- deletes all but 2mn slots
* change slots back
2024-01-05 11:12:44 +01:00
7a3480e044
Fix/cleanupdb delete amb ( #31 )
...
* clippy+fmt
* cleanup also from amb table
2024-01-05 11:06:47 +01:00
6b0e2d269f
Loading all ALTS in bulk, saving stats of priority fees by cus ( #34 )
...
* Loading all ALTS in bulk, saving stats of priority fees by cus
* cargo fmt
2024-01-03 11:43:55 +01:00
3a783316b3
Copying alts from build and using it in docker file
2023-12-26 22:25:24 +01:00
c2e5ccae34
Update docker file with location of alts
2023-12-26 17:56:17 +01:00
739c8d2a67
Preloading ALTs from a predetermined list
2023-12-26 14:40:33 +01:00
ff0d1ebc8c
disabling alts
2023-12-24 10:31:45 +01:00
af07d199dd
Loading all alts before starting
2023-12-24 10:21:53 +01:00
31da6d2b5b
Reverting ATLs and changing signature and account key to text
2023-12-24 10:07:49 +01:00
921dce82bd
Changing back signature type to char(88) and account_key to char(44)
2023-12-22 21:42:48 +01:00
a47f4254aa
making temp tables back char 88 and char 44
2023-12-22 21:03:17 +01:00
c79801ebeb
disable alts
2023-12-22 18:00:20 +01:00
bf15a3cd14
Avoid deserializing alts all the time
2023-12-22 16:42:46 +01:00
c51d8f87b5
Improve/use text insteadof varchar char ( #33 )
...
* clippy+fmt
* use varchar for signature+account_address
* cluster amt table
* Postgres: use TEXT everywhere
2023-12-22 15:24:06 +01:00
481c1e2d06
Updating prometheus counter name
2023-12-22 15:02:29 +01:00
dda7329282
Adding ALTs counter
2023-12-22 15:01:06 +01:00
GroovieGermanikus
Groovie | Mango
godmodegalactus
galactus