Commit Graph

136 Commits

Author SHA1 Message Date
Groovie | Mango d10910e737
reducing yellowstone dependencies (#389)
* WIP - not compiling

* remove connect_hacked

* use reimported types

* cleanup

* fix fmt
2024-04-17 15:11:51 +02:00
godmodegalactus 91cf06436a
Changing exit signal from notify to broadcast channel in grpc multiplexer 2024-04-02 14:45:41 +02:00
galactus 681334197f
Adding custom tpu send transaction example (#380)
* Adding an example for custom tpu send transaction

* Fixing the custom tpu example

* Optimizing SentTransactionInfo, and calculating TPS

* Reverting unwanted changes

* After groovies review
2024-04-02 14:26:54 +02:00
godmodegalactus 1e956a2b4f
Revert "Replacing notify by cancellation token to correctly shutdown tasks"
This reverts commit d1abc345cf.
2024-03-28 11:49:20 +01:00
godmodegalactus d1abc345cf
Replacing notify by cancellation token to correctly shutdown tasks 2024-03-28 09:09:42 +01:00
godmodegalactus dfe8d972eb
Porting back the exit notify change 2024-03-27 17:57:40 +01:00
galactus 79748c5a58
Merge production main 2024 03 27 (#374)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Removing unwanted logs

* Add logs on block queues and reception time

* reduce level of block reception log

* Revert "Solving issue of finalized meta after processed block (#365)"

This reverts commit c09700fd79.

Production runs show more stable memory use but worse performance on
landing transactions.

* use yellowstone grpc with hacked-windowsize

* upgraded geyser-grpc-connector + increased timeout

* Fix block stream throughput problems

By temporarily pasting in a function to connect to block streams via a
more generously configured endpoint.

* Enabling unstable tokio on fly

* Fixing prioritization heap metrics (#370)

* Fixing prioritization heap metrics

* Fixing transaction sizes tests

* Correctly ending the block streams instead of aborting them

* Replacing exit signal with exit notification (#372)

* Replacing exit signal with exit notification

* Deprecitating nightly version

* Increase connection size

* Fixing issue with multiple notify channels

* Joining heap task instead of aborting

* fix postgres ssl algo issue - RC2-40-CBC  (#373)

* fix 373: openssl-legacy.cnf

* Remove replace console subscriber with tracing subscriber

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-03-27 17:09:38 +01:00
Lou-Kamades 4f51c27a08
TC-1 (#359)
add benchnew with confirmation-slot and confirmation-rate (TC-1 and TC-2)
2024-03-27 11:45:30 +01:00
Groovie | Mango 3c994597b4
benchrunner service (#363)
benchrunner service
+ minor changes in bench code
2024-03-26 13:41:46 +01:00
galactus 087cc7f204
Merge prod main 2024 03 25 (#369)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Removing unwanted logs

* Add logs on block queues and reception time

* reduce level of block reception log

* Revert "Solving issue of finalized meta after processed block (#365)"

This reverts commit c09700fd79.

Production runs show more stable memory use but worse performance on
landing transactions.

* use yellowstone grpc with hacked-windowsize

* upgraded geyser-grpc-connector + increased timeout

* Fix block stream throughput problems

By temporarily pasting in a function to connect to block streams via a
more generously configured endpoint.

* Enabling unstable tokio on fly

* update to v0.10.3+yellowstone.1.12+solana.1.17.15-hacked-windowsize3

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
2024-03-25 17:44:57 +01:00
galactus defdc20dd5
Getting finalized meta before processed main (#366)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Minor changes

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
2024-03-22 12:30:14 +01:00
GroovieGermanikus ae495bed53
Merge branch 'production'
using 1.17.15 geyser
2024-03-22 10:57:20 +01:00
galactus 7e6a1a4003
Reverting back the geyser version (#364) 2024-03-22 09:55:46 +01:00
godmodegalactus 50b81135c3
Merge branch 'production' 2024-03-18 20:01:02 +01:00
godmodegalactus 85fcb6097e
Adding some more metrics for quic tpu, and changing some parameters 2024-03-18 17:25:52 +01:00
godmodegalactus b51a9ab818
prioritization heap bugfix 2024-03-18 15:26:42 +01:00
godmodegalactus 8c31e83ef9
Bug fix prioritization heap 2024-03-18 15:18:32 +01:00
Lou-Kamades 583cee1aa4
restore old benchmarks (#362)
restore old benchmarks
2024-03-18 13:47:43 +01:00
godmodegalactus 9be86919ba
Merge branch 'production' 2024-03-16 16:18:01 +01:00
Lou-Kamades 41c40a8128
Use CLI instead of binaries for configuring benchmarks (#357)
refactor: use CLI for benchmarks
2024-03-14 17:49:34 +01:00
Lou-Kamades 573c2da966
Refactor benchmarks into separate binaries (#350)
make the new benchmark test cases individual bins
2024-03-13 17:29:23 +01:00
GroovieGermanikus a1dbf1d77c
Cargo.lock 2024-03-11 18:28:16 +01:00
godmodegalactus 90874c73ac
Updating rust version to 1.75.0 2024-03-08 11:33:39 +01:00
galactus 8ef1b56cdb
account_subscription_into_multiple_channels_and_merge_them (#344)
* account_subscription_into_multiple_channels_and_merge_them

* Removing unecessary changes

* Minor fix after groovies comments
2024-03-01 14:45:28 +01:00
galactus 49bb7ff9f3
Implementing account on demand (#338)
* Implementing account on demand

* Resubscribing accounts after every sometime

* Changing account stream commitment to processed instead

* Fixing account on demand, and account subscription bugs

* Changing variable name for commitment from confirmed to processed

* Some account optimizations

* Adding get balance call

* Changes after groovies review.

* On demand accounts subscribe first then delete old subscription
2024-02-28 15:40:11 +01:00
galactus 064d47719e
Creating an example directory and adding example to be implemented (#340) 2024-02-26 20:43:02 +01:00
Groovie | Mango 855c0190eb
prepare new "util" crate (#333)
prepare new "util" crate
2024-02-22 16:54:42 +01:00
galactus 0ab947b7d7
Updating jsonrpsee and fixing cors (#332)
* Updating json rpsee and fixing CORS

* Fixing cors and updating jsonrpsee and fixing a bug in address lookup table store

* Adding rpc errors, and fixing groovies comments

* for moment get performance data from rpc client
2024-02-21 14:55:44 +01:00
Groovie | Mango 2c42a536b6
Improve/geyser replace streams (#330)
* add env ENABLE_INSTRUMENT_TRACING to enable tracing

* add tracing dependency

* example tracings

* log recent_processed_blocks.len

* bound channels

* replace fastestwins with task version

* revert fake error

* fix meta stream

* disble almost all

* disable more

* keep slot channel up

* log memory usage using CAP

* log CAP memory usage

* add span tracing to mutliplex tasks

* merge

* fix import

* remove useless spans

* fix span logger

* log emitted data from multiplexer

* improve naming of mpsc from autoconnection task

* WIP: use task/mpsc autoconnect for full blocks

* use task/mpsc autoconnect for block meta too

* tag the channels

* add spans

* bump to use updated geyser-grpc-connector with mpsc

* remove unused functions

* extract constant

* split plugger for one and many downstreams

* clippy+fmt

* use task for slots too

* remove broadcast intermediate channel
2024-02-15 21:49:15 +01:00
galactus bcc3c37346
Accounts support (#327)
* Slitting rpc and pubsub services, adding few class and rpc, pubsub callbacks for accounts

* Implementing account storage and service writing some tests

* Chainging to interface, adding new tests

* Implementing accounts related rpc endpoints

* Adding tests for account filters

* Minor bug fixed and errors

* Fixing issue with get program accounts

* Implementing subscription

* use auto reconnect (#329)

* groovie comments part_1

* Fixing tests, fixing groovies comments

* Adding workspace to test filters, minor changes

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-02-14 19:20:25 +01:00
Groovie | Mango 38da1ed987
Improvement/check block sequencing (#299)
use grpc multiplex and improve sequencing
2024-02-07 15:51:22 +01:00
galactus 196b400e15
Adding support for address lookup tables (#319)
* Initial implementation of address lookup tables

* Implementing fetching of address lookup tables

* Use address lookup tables to calculate account prio fees

* Adding some logs and renaming classes

* preloading all the alts correctly

* Changes after groovies review

* Fixing issues after rebase
2024-02-06 17:59:46 +01:00
Groovie | Mango fae4cecc0e
Improvement/saving blocks use lookup tables (#318)
block store basic implementation
2024-02-05 20:20:15 +01:00
galactus 9e031902e3
Implementing missing endpoints (#316)
* Implementing missing endpoints

* Adding slot subscription, and implementing prio fee method

* Changes after groovie and anikets review
2024-02-02 16:42:42 +01:00
galactus 27b1e6ad16
Calculating prio fees by account (#310)
* Calculating readable and writable accounts for transaction and saving in transaction info, declaration of primary strutures

Implementing calculation of prioritization by accounts

Optimize calculation of prioritization fees stats (#301)

* Optimize calculation of prioritization fees stats

* comment on arrays in response

* Fix the calculation of supp

* Fix fmt

---------

Co-authored-by: GroovieGermanikus <groovie@mango.markets>

Implementing calculation of prioritization by accounts

Adding endpoint methods for prioritization fees by account

* Implement account stats subscription

* Fixing the serialization and deserialization of last

* Changes after groovies review
2024-02-01 18:49:51 +01:00
galactus 60388667f5
Main into production 01/02/2024 (#312)
* Production into main (#303)

* fix: panic on geyser close, multiplex bug

https://github.com/blockworks-foundation/geyser-grpc-connector/issues/3

* update Cargo.lock

* reverting cargo.lock

* Fix issues with grpc and postgres

* Solving merge issues

* Fixing cargo fmt

* Increase finish quic timeout (#280) (#281)

* integrate geyser slot subscription (#283)

* Increase finish quic timeout (#280)

* Moving geyser slot subscription from stream to channels (#282)

* Moving geyser slot subscription from stream to channels

* Closing all the slot subscription tasks incase of restart

* Making slot channel unbounded (bug)

* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* Update cargolock file

* Fixing clippy removing grpc_inspect

* merging main with production (#290)

* remove block_debug_listen (#286)

* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* clippy

* Fixing message too long and overflow panics (#288)

* Update geyser grpc connector commit (#289)

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>

* Merging MTU changes and setting up transportation config (#293)

* remove block_debug_listen (#286)

* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* clippy

* Fixing message too long and overflow panics (#288)

* Update geyser grpc connector commit (#289)

* Updating the transport config to match with solana endpoint (#292)

* Updating the transport config to match with solana endpoint

* Setting max MTU after groovies comments

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>

* Making block subscription processed and moving confirmed block subscr… (#291)

* Making block subscription processed and moving confirmed block subscription to meta

* Sending both processed and confirmed blocks, if block has already been confirmed

* Minor bug, subscribing to processed blocks instead of confirmed (#295)

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>

* Restart connection if any stream fails

* Updating to version 0.2.4

* Updating the change logs

* Fixing all the sub overflows by using saturating sub (#309)

* Fixing by cu computation adding more tests to check by CU (#311)

* Fixing by cu computation adding more tests to check by CU

* Adding more tests

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
2024-02-01 15:08:47 +01:00
godmodegalactus a44923c334
Updating to version 0.2.4 2024-01-29 13:11:54 +01:00
Groovie | Mango 7171b524f0
Feature/block priofees (#274)
* boilerplate

* add dummy endpoint

* simple logic with highes block

* endpoint works

* remove cu calculus

* log get_latest_block startup timing

* websocket subscription basics

* WIP

* propagate fees via channel

* cleanup

* use broadcast sender

* proper handling of broadcast channel issues

* rename stuff

* move to dedicated crate

* cleanup deps

* move data definition

* hide private types

* percentile math

* code format

* integrated stats by cu

* add cu stats

* restart inline asserts

* warn about DashMap access

* switch to BTreeMap

* remove vote transactions from calculus

* return vec

* array

* flat format

* reformat

* split arrays

* rename keys

* enable experimental tag

* code format

* more logging

* reset log level

* rename websocket method to blockPrioritizationFeesSubscribe

* HACK: use processed blocks

* add is_vote_transaction

* do not fail if fees goes down

* udp message size example

* imrpove 100 handling

* simplify inital sort

* Revert "HACK: use processed blocks"

This reverts commit ff17d9475c.

* add percentile test case

* add per block cu_consumed+tx_count(nonvote)

* fix supid cleanup bug

* add TxAggregateStats

* add by_cu test

* add alternative cu percentlie impl

* cleanup cu tests

* add check for step

* saturating_sub

* remove useless file

* add is_vote to history crate

* fix fmt+clippy (nightly)

* clippy

* clippy

* clippy

* clippy
2024-01-25 10:29:14 +01:00
galactus de26490e9c
Squash and merging all the commits from groovies branch groovie/post… (#300)
* Squash and merging all the commits from  groovies branch groovie/postgres_saving_blocksII

* Ingore tests which require postgres
2024-01-22 15:22:11 +01:00
galactus e00683884a
Update geyser grpc connector commit (#289) 2024-01-18 15:53:02 +01:00
galactus ba77b0b389
Updating cargo file (#287)
* Updating cargo file

* Updating tag for grpc connector
2024-01-18 11:28:35 +01:00
galactus 6f61d89743
Fixing the performance issue with geyser multiplexing (#275) 2024-01-15 20:22:59 +01:00
galactus bea39d8c6a
Migrating to solana v1.17.15 (#273) 2024-01-15 18:43:48 +01:00
galactus d91e2e9316
disable geyser stakes and votes for refactoring (#268) 2024-01-11 14:37:06 +01:00
GroovieGermanikus ef5716932f
fix: panic on geyser close, multiplex bug
https://github.com/blockworks-foundation/geyser-grpc-connector/issues/3
2024-01-07 20:08:20 +01:00
musitdev dbc152bb64 rebase from master 2024-01-03 16:50:16 +01:00
musitdev dc37b865f6 update cargo.toml with workplace 2024-01-03 10:20:18 +01:00
musitdev 3e4c81c567 Stake and vote management. Integration of the main loop in current architecture 2024-01-03 10:20:14 +01:00
Groovie | Mango 5cfb6de72a
grpc multiplexing integrate (#255)
allow multiple GRPC sources
2023-12-22 14:42:20 +01:00
Aniket Prajapati 3eecb89b81
simple config, env and cli (#247)
* simple config, env and cli

* fix comp

* init env first

* added env override

* postgres config

* fmt

* updated env in readme

* examples

* a
2023-11-27 14:24:59 +05:30