7d39a947ac
Minor changes, increasing default lite-rpc limits
2024-03-26 17:36:11 +01:00
29bd6deae4
Fixing accounts on demand ( #371 )
...
* Fixing accounts on demand
* Adding missing file and renaming gprc to grpc
* Adding missing grpc_accounts_streaming
2024-03-26 14:32:50 +01:00
087cc7f204
Merge prod main 2024 03 25 ( #369 )
...
* Use jemalloc
* Solving issue of finalized meta after processed block (#365 )
* Solving issue of finalized meta after processed block
* Fixing the broken test
* Removing unwanted logs
* Add logs on block queues and reception time
* reduce level of block reception log
* Revert "Solving issue of finalized meta after processed block (#365 )"
This reverts commit c09700fd79
2024-03-25 17:44:57 +01:00
cd9df11501
use geyser blockmeta to progress finalizedslot ( #367 )
...
introduce BlockInfo on a new BlockInfoStream; maps from yellowstone BlockMeta
fix: last_finalized_slot must progress unconditionally
2024-03-22 19:22:38 +01:00
defdc20dd5
Getting finalized meta before processed main ( #366 )
...
* Use jemalloc
* Solving issue of finalized meta after processed block (#365 )
* Solving issue of finalized meta after processed block
* Fixing the broken test
* Minor changes
---------
Co-authored-by: Christian Kamm <mail@ckamm.de>
2024-03-22 12:30:14 +01:00
ae495bed53
Merge branch 'production'
...
using 1.17.15 geyser
2024-03-22 10:57:20 +01:00
7e6a1a4003
Reverting back the geyser version ( #364 )
2024-03-22 09:55:46 +01:00
50b81135c3
Merge branch 'production'
2024-03-18 20:01:02 +01:00
85fcb6097e
Adding some more metrics for quic tpu, and changing some parameters
2024-03-18 17:25:52 +01:00
9be86919ba
Merge branch 'production'
2024-03-16 16:18:01 +01:00
1e3a9cdcbe
update geyser-grpc-connector with attempt fix
2024-03-11 18:26:59 +01:00
33ad19ed47
Removing verbose log for grpc multiplexing ( #354 )
2024-03-11 13:48:15 +01:00
c59f3f8f38
Updating docker file with correct version info
2024-03-08 11:40:55 +01:00
18a0ab7062
from grpc block ProduceBlock mapping - Performance+Typing ( #348 )
...
* remove all Pubkey::default();
1. the value does not make sense
2. we can safely assume that bytes can be deserialized
* use Signature instead of String
* use solana_sdk:#️⃣ :Hash for blockhash
* store VersionedMessage instead of serialized bas64 string
* refactor compute_budget extraction
* fix test data
* simplify map_compute_budget_instructions
* fmt
---------
Co-authored-by: godmodegalactus <godmodegalactus@gmail.com>
2024-03-01 17:21:48 +01:00
8ef1b56cdb
account_subscription_into_multiple_channels_and_merge_them ( #344 )
...
* account_subscription_into_multiple_channels_and_merge_them
* Removing unecessary changes
* Minor fix after groovies comments
2024-03-01 14:45:28 +01:00
49bb7ff9f3
Implementing account on demand ( #338 )
...
* Implementing account on demand
* Resubscribing accounts after every sometime
* Changing account stream commitment to processed instead
* Fixing account on demand, and account subscription bugs
* Changing variable name for commitment from confirmed to processed
* Some account optimizations
* Adding get balance call
* Changes after groovies review.
* On demand accounts subscribe first then delete old subscription
2024-02-28 15:40:11 +01:00
855c0190eb
prepare new "util" crate ( #333 )
...
prepare new "util" crate
2024-02-22 16:54:42 +01:00
7bba02f3c5
wrap producedblock struct and avoid cloning it ( #336 )
...
wrap ProducedBlock in Arc
2024-02-22 14:28:29 +01:00
2c42a536b6
Improve/geyser replace streams ( #330 )
...
* add env ENABLE_INSTRUMENT_TRACING to enable tracing
* add tracing dependency
* example tracings
* log recent_processed_blocks.len
* bound channels
* replace fastestwins with task version
* revert fake error
* fix meta stream
* disble almost all
* disable more
* keep slot channel up
* log memory usage using CAP
* log CAP memory usage
* add span tracing to mutliplex tasks
* merge
* fix import
* remove useless spans
* fix span logger
* log emitted data from multiplexer
* improve naming of mpsc from autoconnection task
* WIP: use task/mpsc autoconnect for full blocks
* use task/mpsc autoconnect for block meta too
* tag the channels
* add spans
* bump to use updated geyser-grpc-connector with mpsc
* remove unused functions
* extract constant
* split plugger for one and many downstreams
* clippy+fmt
* use task for slots too
* remove broadcast intermediate channel
2024-02-15 21:49:15 +01:00
bcc3c37346
Accounts support ( #327 )
...
* Slitting rpc and pubsub services, adding few class and rpc, pubsub callbacks for accounts
* Implementing account storage and service writing some tests
* Chainging to interface, adding new tests
* Implementing accounts related rpc endpoints
* Adding tests for account filters
* Minor bug fixed and errors
* Fixing issue with get program accounts
* Implementing subscription
* use auto reconnect (#329 )
* groovie comments part_1
* Fixing tests, fixing groovies comments
* Adding workspace to test filters, minor changes
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-02-14 19:20:25 +01:00
67641f6d40
fix bug causing endless geyser restart loops ( #324 )
...
fix restart logic
2024-02-12 11:53:52 +01:00
38da1ed987
Improvement/check block sequencing ( #299 )
...
use grpc multiplex and improve sequencing
2024-02-07 15:51:22 +01:00
196b400e15
Adding support for address lookup tables ( #319 )
...
* Initial implementation of address lookup tables
* Implementing fetching of address lookup tables
* Use address lookup tables to calculate account prio fees
* Adding some logs and renaming classes
* preloading all the alts correctly
* Changes after groovies review
* Fixing issues after rebase
2024-02-06 17:59:46 +01:00
7e7992bbc3
Fixing memory issue ( #320 )
2024-02-06 10:08:18 +01:00
27b1e6ad16
Calculating prio fees by account ( #310 )
...
* Calculating readable and writable accounts for transaction and saving in transaction info, declaration of primary strutures
Implementing calculation of prioritization by accounts
Optimize calculation of prioritization fees stats (#301 )
* Optimize calculation of prioritization fees stats
* comment on arrays in response
* Fix the calculation of supp
* Fix fmt
---------
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
Implementing calculation of prioritization by accounts
Adding endpoint methods for prioritization fees by account
* Implement account stats subscription
* Fixing the serialization and deserialization of last
* Changes after groovies review
2024-02-01 18:49:51 +01:00
bc7d89ce4e
Production ( #313 )
...
* fix: panic on geyser close, multiplex bug
https://github.com/blockworks-foundation/geyser-grpc-connector/issues/3
* update Cargo.lock
* reverting cargo.lock
* Fix issues with grpc and postgres
* Solving merge issues
* Fixing cargo fmt
* Increase finish quic timeout (#280 ) (#281 )
* integrate geyser slot subscription (#283 )
* Increase finish quic timeout (#280 )
* Moving geyser slot subscription from stream to channels (#282 )
* Moving geyser slot subscription from stream to channels
* Closing all the slot subscription tasks incase of restart
* Making slot channel unbounded (bug)
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* Update cargolock file
* Fixing clippy removing grpc_inspect
* merging main with production (#290 )
* remove block_debug_listen (#286 )
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* clippy
* Fixing message too long and overflow panics (#288 )
* Update geyser grpc connector commit (#289 )
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
* Merging MTU changes and setting up transportation config (#293 )
* remove block_debug_listen (#286 )
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* clippy
* Fixing message too long and overflow panics (#288 )
* Update geyser grpc connector commit (#289 )
* Updating the transport config to match with solana endpoint (#292 )
* Updating the transport config to match with solana endpoint
* Setting max MTU after groovies comments
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
* Making block subscription processed and moving confirmed block subscr… (#291 )
* Making block subscription processed and moving confirmed block subscription to meta
* Sending both processed and confirmed blocks, if block has already been confirmed
* Minor bug, subscribing to processed blocks instead of confirmed (#295 )
* reduce log
* Restart connection if any stream fails (#306 )
* Main into production 01/02/2024 (#312 )
* Production into main (#303 )
* fix: panic on geyser close, multiplex bug
https://github.com/blockworks-foundation/geyser-grpc-connector/issues/3
* update Cargo.lock
* reverting cargo.lock
* Fix issues with grpc and postgres
* Solving merge issues
* Fixing cargo fmt
* Increase finish quic timeout (#280 ) (#281 )
* integrate geyser slot subscription (#283 )
* Increase finish quic timeout (#280 )
* Moving geyser slot subscription from stream to channels (#282 )
* Moving geyser slot subscription from stream to channels
* Closing all the slot subscription tasks incase of restart
* Making slot channel unbounded (bug)
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* Update cargolock file
* Fixing clippy removing grpc_inspect
* merging main with production (#290 )
* remove block_debug_listen (#286 )
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* clippy
* Fixing message too long and overflow panics (#288 )
* Update geyser grpc connector commit (#289 )
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
* Merging MTU changes and setting up transportation config (#293 )
* remove block_debug_listen (#286 )
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* clippy
* Fixing message too long and overflow panics (#288 )
* Update geyser grpc connector commit (#289 )
* Updating the transport config to match with solana endpoint (#292 )
* Updating the transport config to match with solana endpoint
* Setting max MTU after groovies comments
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
* Making block subscription processed and moving confirmed block subscr… (#291 )
* Making block subscription processed and moving confirmed block subscription to meta
* Sending both processed and confirmed blocks, if block has already been confirmed
* Minor bug, subscribing to processed blocks instead of confirmed (#295 )
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
* Restart connection if any stream fails
* Updating to version 0.2.4
* Updating the change logs
* Fixing all the sub overflows by using saturating sub (#309 )
* Fixing by cu computation adding more tests to check by CU (#311 )
* Fixing by cu computation adding more tests to check by CU
* Adding more tests
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
2024-02-01 15:16:24 +01:00
60388667f5
Main into production 01/02/2024 ( #312 )
...
* Production into main (#303 )
* fix: panic on geyser close, multiplex bug
https://github.com/blockworks-foundation/geyser-grpc-connector/issues/3
* update Cargo.lock
* reverting cargo.lock
* Fix issues with grpc and postgres
* Solving merge issues
* Fixing cargo fmt
* Increase finish quic timeout (#280 ) (#281 )
* integrate geyser slot subscription (#283 )
* Increase finish quic timeout (#280 )
* Moving geyser slot subscription from stream to channels (#282 )
* Moving geyser slot subscription from stream to channels
* Closing all the slot subscription tasks incase of restart
* Making slot channel unbounded (bug)
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* Update cargolock file
* Fixing clippy removing grpc_inspect
* merging main with production (#290 )
* remove block_debug_listen (#286 )
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* clippy
* Fixing message too long and overflow panics (#288 )
* Update geyser grpc connector commit (#289 )
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
* Merging MTU changes and setting up transportation config (#293 )
* remove block_debug_listen (#286 )
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* clippy
* Fixing message too long and overflow panics (#288 )
* Update geyser grpc connector commit (#289 )
* Updating the transport config to match with solana endpoint (#292 )
* Updating the transport config to match with solana endpoint
* Setting max MTU after groovies comments
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
* Making block subscription processed and moving confirmed block subscr… (#291 )
* Making block subscription processed and moving confirmed block subscription to meta
* Sending both processed and confirmed blocks, if block has already been confirmed
* Minor bug, subscribing to processed blocks instead of confirmed (#295 )
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
* Restart connection if any stream fails
* Updating to version 0.2.4
* Updating the change logs
* Fixing all the sub overflows by using saturating sub (#309 )
* Fixing by cu computation adding more tests to check by CU (#311 )
* Fixing by cu computation adding more tests to check by CU
* Adding more tests
---------
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
2024-02-01 15:08:47 +01:00
118f5435e3
Fixing all the sub overflows by using saturating sub ( #309 )
2024-01-30 15:24:35 +01:00
a44923c334
Updating to version 0.2.4
2024-01-29 13:11:54 +01:00
64e86862c1
Restart connection if any stream fails ( #306 )
2024-01-26 22:49:10 +01:00
6c61e64948
Restart connection if any stream fails
2024-01-26 22:25:19 +01:00
847ce74ab6
reduce log
2024-01-26 19:46:51 +01:00
7171b524f0
Feature/block priofees ( #274 )
...
* boilerplate
* add dummy endpoint
* simple logic with highes block
* endpoint works
* remove cu calculus
* log get_latest_block startup timing
* websocket subscription basics
* WIP
* propagate fees via channel
* cleanup
* use broadcast sender
* proper handling of broadcast channel issues
* rename stuff
* move to dedicated crate
* cleanup deps
* move data definition
* hide private types
* percentile math
* code format
* integrated stats by cu
* add cu stats
* restart inline asserts
* warn about DashMap access
* switch to BTreeMap
* remove vote transactions from calculus
* return vec
* array
* flat format
* reformat
* split arrays
* rename keys
* enable experimental tag
* code format
* more logging
* reset log level
* rename websocket method to blockPrioritizationFeesSubscribe
* HACK: use processed blocks
* add is_vote_transaction
* do not fail if fees goes down
* udp message size example
* imrpove 100 handling
* simplify inital sort
* Revert "HACK: use processed blocks"
This reverts commit ff17d9475c
2024-01-25 10:29:14 +01:00
705a08c2c6
fix some namings ( #298 )
...
* fix some namings
* clipp
* make run_clippy_fmt.sh executable
2024-01-23 08:17:48 +01:00
de26490e9c
Squash and merging all the commits from groovies branch groovie/post… ( #300 )
...
* Squash and merging all the commits from groovies branch groovie/postgres_saving_blocksII
* Ingore tests which require postgres
2024-01-22 15:22:11 +01:00
0c5aa196c1
Minor bug, subscribing to processed blocks instead of confirmed ( #295 )
2024-01-22 10:54:39 +01:00
33cca71d49
Making block subscription processed and moving confirmed block subscr… ( #291 )
...
* Making block subscription processed and moving confirmed block subscription to meta
* Sending both processed and confirmed blocks, if block has already been confirmed
2024-01-21 13:56:40 +01:00
e3bfaac726
remove block_debug_listen ( #286 )
...
* remove block_debug_listen
caused a panic - need more time to investigate
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])
* clippy
2024-01-18 13:57:57 +01:00
d589d90999
Moving geyser slot subscription from stream to channels ( #284 )
...
Closing all the slot subscription tasks incase of restart
Making slot channel unbounded because of errors
2024-01-17 19:45:10 +01:00
cfb5e1ac66
Moving geyser slot subscription from stream to channels ( #282 )
...
* Moving geyser slot subscription from stream to channels
* Closing all the slot subscription tasks incase of restart
2024-01-17 15:17:49 +01:00
36c62763c0
Fixing merging of confirmed block streams ( #276 )
2024-01-16 10:13:12 +01:00
6f61d89743
Fixing the performance issue with geyser multiplexing ( #275 )
2024-01-15 20:22:59 +01:00
bea39d8c6a
Migrating to solana v1.17.15 ( #273 )
2024-01-15 18:43:48 +01:00
78e1bb2b79
More clippy changes
2024-01-12 16:39:19 +01:00
c3da6ad1b6
cargo fmt and clippy issues
2024-01-12 14:40:48 +01:00
3f69de5247
Resarting geyser slot and block stream in case they disconnect
2024-01-12 14:21:54 +01:00
185cbd379b
Merge pull request #267 from blockworks-foundation/reducing_network_usage_by_fetching_only_finalized_blockmeta-groovie
...
Reduce network by only fetching block meta instead of whole block
2024-01-11 13:16:52 +01:00
b49e51da9c
Merge pull request #266 from blockworks-foundation/bootstrap_leader_schedule_from_rpc
...
Bootstrap leader schedule from rpc
2024-01-08 16:14:35 +01:00
079ff113c4
fix cleanup logic
2024-01-08 14:51:32 +01:00
8996a97424
change .send error handling on broadcast channel
2024-01-08 14:21:05 +01:00
godmodegalactus
galactus
Groovie | Mango
GroovieGermanikus