Commit Graph

135 Commits

Author SHA1 Message Date
Groovie | Mango 0d1d856305
Update to solana 1.18 (#397)
* update solana from 1.17.28 -> 1.18.15
* remove legacy cu
* use yellowstone 1.15
* use tagged geyser-grpc-connector version
2024-06-13 13:45:52 +02:00
Groovie | Mango 944a93114c
Bugfix,379 identity key pair loading must be optional (#381)
fix identity keypair loading logic

reverting to this logic
1. check if env variable IDENTITY is present
2. check if cli arg --identity-keypair is present
3. assume no identity
2024-04-02 15:52:06 +02:00
galactus 681334197f
Adding custom tpu send transaction example (#380)
* Adding an example for custom tpu send transaction

* Fixing the custom tpu example

* Optimizing SentTransactionInfo, and calculating TPS

* Reverting unwanted changes

* After groovies review
2024-04-02 14:26:54 +02:00
galactus 79748c5a58
Merge production main 2024 03 27 (#374)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Removing unwanted logs

* Add logs on block queues and reception time

* reduce level of block reception log

* Revert "Solving issue of finalized meta after processed block (#365)"

This reverts commit c09700fd79.

Production runs show more stable memory use but worse performance on
landing transactions.

* use yellowstone grpc with hacked-windowsize

* upgraded geyser-grpc-connector + increased timeout

* Fix block stream throughput problems

By temporarily pasting in a function to connect to block streams via a
more generously configured endpoint.

* Enabling unstable tokio on fly

* Fixing prioritization heap metrics (#370)

* Fixing prioritization heap metrics

* Fixing transaction sizes tests

* Correctly ending the block streams instead of aborting them

* Replacing exit signal with exit notification (#372)

* Replacing exit signal with exit notification

* Deprecitating nightly version

* Increase connection size

* Fixing issue with multiple notify channels

* Joining heap task instead of aborting

* fix postgres ssl algo issue - RC2-40-CBC  (#373)

* fix 373: openssl-legacy.cnf

* Remove replace console subscriber with tracing subscriber

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-03-27 17:09:38 +01:00
Groovie | Mango 3c994597b4
benchrunner service (#363)
benchrunner service
+ minor changes in bench code
2024-03-26 13:41:46 +01:00
Groovie | Mango cd9df11501
use geyser blockmeta to progress finalizedslot (#367)
introduce BlockInfo on a new BlockInfoStream; maps from yellowstone BlockMeta
fix: last_finalized_slot must progress unconditionally
2024-03-22 19:22:38 +01:00
galactus defdc20dd5
Getting finalized meta before processed main (#366)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Minor changes

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
2024-03-22 12:30:14 +01:00
GroovieGermanikus ae495bed53
Merge branch 'production'
using 1.17.15 geyser
2024-03-22 10:57:20 +01:00
godmodegalactus 5be4d2e755
Bug fix update finalized blockhash 2024-03-22 09:22:51 +01:00
godmodegalactus 9946abb12d
Minor performance fix 2024-03-21 18:43:29 +01:00
godmodegalactus f219e5eee1
Minor updates to block information store 2024-03-21 16:44:17 +01:00
godmodegalactus 35d13a25de
Merge branch 'production' 2024-03-19 18:09:59 +01:00
godmodegalactus 4bcc9ddb6c
Adding quic timers metrics, more quic bugfixes 2024-03-19 14:37:08 +01:00
godmodegalactus 73d0b069be
Fixing memory leak, and metrics 2024-03-19 10:35:29 +01:00
godmodegalactus 50b81135c3
Merge branch 'production' 2024-03-18 20:01:02 +01:00
godmodegalactus 85fcb6097e
Adding some more metrics for quic tpu, and changing some parameters 2024-03-18 17:25:52 +01:00
godmodegalactus b51a9ab818
prioritization heap bugfix 2024-03-18 15:26:42 +01:00
godmodegalactus 8c31e83ef9
Bug fix prioritization heap 2024-03-18 15:18:32 +01:00
godmodegalactus 9be86919ba
Merge branch 'production' 2024-03-16 16:18:01 +01:00
godmodegalactus f0442ce50d
Implementing prioritiztion heap, and using it to sort transactions by priority 2024-03-16 15:38:05 +01:00
galactus ce215e0239
After fanatids review (#356)
* After fanatids review

* After fanatids updated comments
2024-03-12 14:02:57 +01:00
godmodegalactus 3e01ebb7cc
Fixing perofmrance issues after checking last block height 2024-03-08 17:58:49 +01:00
godmodegalactus c59f3f8f38
Updating docker file with correct version info 2024-03-08 11:40:55 +01:00
galactus 73973ba0f5
Check blockhash is expired or not (#351) 2024-03-08 11:27:37 +01:00
galactus 3f7130335a
Check blockhash is expired or not (#351) 2024-03-07 16:09:22 +01:00
Groovie | Mango 18a0ab7062
from grpc block ProduceBlock mapping - Performance+Typing (#348)
* remove all Pubkey::default();
1. the value does not make sense
2. we can safely assume that bytes can be deserialized

* use Signature instead of String

* use solana_sdk:#️⃣:Hash for blockhash

* store VersionedMessage instead of serialized bas64 string

* refactor compute_budget extraction

* fix test data

* simplify map_compute_budget_instructions

* fmt

---------

Co-authored-by: godmodegalactus <godmodegalactus@gmail.com>
2024-03-01 17:21:48 +01:00
galactus 8ef1b56cdb
account_subscription_into_multiple_channels_and_merge_them (#344)
* account_subscription_into_multiple_channels_and_merge_them

* Removing unecessary changes

* Minor fix after groovies comments
2024-03-01 14:45:28 +01:00
galactus 49bb7ff9f3
Implementing account on demand (#338)
* Implementing account on demand

* Resubscribing accounts after every sometime

* Changing account stream commitment to processed instead

* Fixing account on demand, and account subscription bugs

* Changing variable name for commitment from confirmed to processed

* Some account optimizations

* Adding get balance call

* Changes after groovies review.

* On demand accounts subscribe first then delete old subscription
2024-02-28 15:40:11 +01:00
Groovie | Mango 7bba02f3c5
wrap producedblock struct and avoid cloning it (#336)
wrap ProducedBlock in Arc
2024-02-22 14:28:29 +01:00
galactus bcc3c37346
Accounts support (#327)
* Slitting rpc and pubsub services, adding few class and rpc, pubsub callbacks for accounts

* Implementing account storage and service writing some tests

* Chainging to interface, adding new tests

* Implementing accounts related rpc endpoints

* Adding tests for account filters

* Minor bug fixed and errors

* Fixing issue with get program accounts

* Implementing subscription

* use auto reconnect (#329)

* groovie comments part_1

* Fixing tests, fixing groovies comments

* Adding workspace to test filters, minor changes

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-02-14 19:20:25 +01:00
galactus 196b400e15
Adding support for address lookup tables (#319)
* Initial implementation of address lookup tables

* Implementing fetching of address lookup tables

* Use address lookup tables to calculate account prio fees

* Adding some logs and renaming classes

* preloading all the alts correctly

* Changes after groovies review

* Fixing issues after rebase
2024-02-06 17:59:46 +01:00
galactus 9e031902e3
Implementing missing endpoints (#316)
* Implementing missing endpoints

* Adding slot subscription, and implementing prio fee method

* Changes after groovie and anikets review
2024-02-02 16:42:42 +01:00
galactus 27b1e6ad16
Calculating prio fees by account (#310)
* Calculating readable and writable accounts for transaction and saving in transaction info, declaration of primary strutures

Implementing calculation of prioritization by accounts

Optimize calculation of prioritization fees stats (#301)

* Optimize calculation of prioritization fees stats

* comment on arrays in response

* Fix the calculation of supp

* Fix fmt

---------

Co-authored-by: GroovieGermanikus <groovie@mango.markets>

Implementing calculation of prioritization by accounts

Adding endpoint methods for prioritization fees by account

* Implement account stats subscription

* Fixing the serialization and deserialization of last

* Changes after groovies review
2024-02-01 18:49:51 +01:00
galactus 60388667f5
Main into production 01/02/2024 (#312)
* Production into main (#303)

* fix: panic on geyser close, multiplex bug

https://github.com/blockworks-foundation/geyser-grpc-connector/issues/3

* update Cargo.lock

* reverting cargo.lock

* Fix issues with grpc and postgres

* Solving merge issues

* Fixing cargo fmt

* Increase finish quic timeout (#280) (#281)

* integrate geyser slot subscription (#283)

* Increase finish quic timeout (#280)

* Moving geyser slot subscription from stream to channels (#282)

* Moving geyser slot subscription from stream to channels

* Closing all the slot subscription tasks incase of restart

* Making slot channel unbounded (bug)

* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* Update cargolock file

* Fixing clippy removing grpc_inspect

* merging main with production (#290)

* remove block_debug_listen (#286)

* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* clippy

* Fixing message too long and overflow panics (#288)

* Update geyser grpc connector commit (#289)

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>

* Merging MTU changes and setting up transportation config (#293)

* remove block_debug_listen (#286)

* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* clippy

* Fixing message too long and overflow panics (#288)

* Update geyser grpc connector commit (#289)

* Updating the transport config to match with solana endpoint (#292)

* Updating the transport config to match with solana endpoint

* Setting max MTU after groovies comments

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>

* Making block subscription processed and moving confirmed block subscr… (#291)

* Making block subscription processed and moving confirmed block subscription to meta

* Sending both processed and confirmed blocks, if block has already been confirmed

* Minor bug, subscribing to processed blocks instead of confirmed (#295)

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>

* Restart connection if any stream fails

* Updating to version 0.2.4

* Updating the change logs

* Fixing all the sub overflows by using saturating sub (#309)

* Fixing by cu computation adding more tests to check by CU (#311)

* Fixing by cu computation adding more tests to check by CU

* Adding more tests

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
2024-02-01 15:08:47 +01:00
godmodegalactus a44923c334
Updating to version 0.2.4 2024-01-29 13:11:54 +01:00
Groovie | Mango 7171b524f0
Feature/block priofees (#274)
* boilerplate

* add dummy endpoint

* simple logic with highes block

* endpoint works

* remove cu calculus

* log get_latest_block startup timing

* websocket subscription basics

* WIP

* propagate fees via channel

* cleanup

* use broadcast sender

* proper handling of broadcast channel issues

* rename stuff

* move to dedicated crate

* cleanup deps

* move data definition

* hide private types

* percentile math

* code format

* integrated stats by cu

* add cu stats

* restart inline asserts

* warn about DashMap access

* switch to BTreeMap

* remove vote transactions from calculus

* return vec

* array

* flat format

* reformat

* split arrays

* rename keys

* enable experimental tag

* code format

* more logging

* reset log level

* rename websocket method to blockPrioritizationFeesSubscribe

* HACK: use processed blocks

* add is_vote_transaction

* do not fail if fees goes down

* udp message size example

* imrpove 100 handling

* simplify inital sort

* Revert "HACK: use processed blocks"

This reverts commit ff17d9475c.

* add percentile test case

* add per block cu_consumed+tx_count(nonvote)

* fix supid cleanup bug

* add TxAggregateStats

* add by_cu test

* add alternative cu percentlie impl

* cleanup cu tests

* add check for step

* saturating_sub

* remove useless file

* add is_vote to history crate

* fix fmt+clippy (nightly)

* clippy

* clippy

* clippy

* clippy
2024-01-25 10:29:14 +01:00
Groovie | Mango 705a08c2c6
fix some namings (#298)
* fix some namings

* clipp

* make run_clippy_fmt.sh executable
2024-01-23 08:17:48 +01:00
galactus de26490e9c
Squash and merging all the commits from groovies branch groovie/post… (#300)
* Squash and merging all the commits from  groovies branch groovie/postgres_saving_blocksII

* Ingore tests which require postgres
2024-01-22 15:22:11 +01:00
galactus 33cca71d49
Making block subscription processed and moving confirmed block subscr… (#291)
* Making block subscription processed and moving confirmed block subscription to meta

* Sending both processed and confirmed blocks, if block has already been confirmed
2024-01-21 13:56:40 +01:00
galactus 4d4019264e
Fixing message too long and overflow panics (#288) 2024-01-18 15:11:29 +01:00
galactus 4ed7eead8b
Adding prometheus metrics to follow quic states (#277) 2024-01-16 11:30:40 +01:00
galactus bea39d8c6a
Migrating to solana v1.17.15 (#273) 2024-01-15 18:43:48 +01:00
godmodegalactus 78e1bb2b79
More clippy changes 2024-01-12 16:39:19 +01:00
godmodegalactus c3da6ad1b6
cargo fmt and clippy issues 2024-01-12 14:40:48 +01:00
godmodegalactus 2498950583
Reconnecing to block stream and slot stream incase of error 2024-01-12 14:33:00 +01:00
Groovie | Mango f910587553
fix prioritization_fees overflow (#245)
* fix prioritization_fees overflow

* suggest inline

* Cargo fmt

---------

Co-authored-by: galactus <96341601+godmodegalactus@users.noreply.github.com>
Co-authored-by: godmodegalactus <godmodegalactus@gmail.com>
2024-01-11 14:16:32 +01:00
galactus 185cbd379b
Merge pull request #267 from blockworks-foundation/reducing_network_usage_by_fetching_only_finalized_blockmeta-groovie
Reduce network by only fetching block meta instead of whole block
2024-01-11 13:16:52 +01:00
GroovieGermanikus 9ef773c129
refactoring 2024-01-08 14:14:59 +01:00
musitdev 032232f6f3 add rpc leader shedule loading if the bootstrap file is not present 2024-01-05 15:58:25 +01:00
musitdev 402ebd69b1 correct from PR remarks 2024-01-04 13:36:13 +01:00