Commit Graph

43 Commits

Author SHA1 Message Date
GroovieGermanikus a109037223
fix fmt 2024-04-16 15:54:38 +02:00
GroovieGermanikus 274dc76125
cleanup 2024-04-16 13:44:07 +02:00
GroovieGermanikus bc081c0695
remove connect_hacked 2024-04-16 13:35:43 +02:00
GroovieGermanikus a3ad81cf39
WIP - not compiling 2024-04-16 10:53:45 +02:00
Groovie | Mango b84e880961
remove async_channel and bring back tokio mpsc§ (#384) 2024-04-03 11:22:36 +02:00
godmodegalactus 12a6832c56
Swapping notify channel with broadcast channel 2024-04-02 15:07:01 +02:00
godmodegalactus ca3fa46139
Using broadcast channels instead of notify 2024-04-02 14:48:54 +02:00
godmodegalactus 1e956a2b4f
Revert "Replacing notify by cancellation token to correctly shutdown tasks"
This reverts commit d1abc345cf.
2024-03-28 11:49:20 +01:00
godmodegalactus 02229060cf
Revert "Fix clippy fmt"
This reverts commit bac036ceaa.
2024-03-28 11:49:17 +01:00
godmodegalactus bac036ceaa
Fix clippy fmt 2024-03-28 09:12:12 +01:00
godmodegalactus d1abc345cf
Replacing notify by cancellation token to correctly shutdown tasks 2024-03-28 09:09:42 +01:00
galactus 79748c5a58
Merge production main 2024 03 27 (#374)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Removing unwanted logs

* Add logs on block queues and reception time

* reduce level of block reception log

* Revert "Solving issue of finalized meta after processed block (#365)"

This reverts commit c09700fd79.

Production runs show more stable memory use but worse performance on
landing transactions.

* use yellowstone grpc with hacked-windowsize

* upgraded geyser-grpc-connector + increased timeout

* Fix block stream throughput problems

By temporarily pasting in a function to connect to block streams via a
more generously configured endpoint.

* Enabling unstable tokio on fly

* Fixing prioritization heap metrics (#370)

* Fixing prioritization heap metrics

* Fixing transaction sizes tests

* Correctly ending the block streams instead of aborting them

* Replacing exit signal with exit notification (#372)

* Replacing exit signal with exit notification

* Deprecitating nightly version

* Increase connection size

* Fixing issue with multiple notify channels

* Joining heap task instead of aborting

* fix postgres ssl algo issue - RC2-40-CBC  (#373)

* fix 373: openssl-legacy.cnf

* Remove replace console subscriber with tracing subscriber

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-03-27 17:09:38 +01:00
galactus 29bd6deae4
Fixing accounts on demand (#371)
* Fixing accounts on demand

* Adding missing file and renaming gprc to grpc

* Adding missing grpc_accounts_streaming
2024-03-26 14:32:50 +01:00
galactus 087cc7f204
Merge prod main 2024 03 25 (#369)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Removing unwanted logs

* Add logs on block queues and reception time

* reduce level of block reception log

* Revert "Solving issue of finalized meta after processed block (#365)"

This reverts commit c09700fd79.

Production runs show more stable memory use but worse performance on
landing transactions.

* use yellowstone grpc with hacked-windowsize

* upgraded geyser-grpc-connector + increased timeout

* Fix block stream throughput problems

By temporarily pasting in a function to connect to block streams via a
more generously configured endpoint.

* Enabling unstable tokio on fly

* update to v0.10.3+yellowstone.1.12+solana.1.17.15-hacked-windowsize3

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
2024-03-25 17:44:57 +01:00
Groovie | Mango cd9df11501
use geyser blockmeta to progress finalizedslot (#367)
introduce BlockInfo on a new BlockInfoStream; maps from yellowstone BlockMeta
fix: last_finalized_slot must progress unconditionally
2024-03-22 19:22:38 +01:00
Groovie | Mango 18a0ab7062
from grpc block ProduceBlock mapping - Performance+Typing (#348)
* remove all Pubkey::default();
1. the value does not make sense
2. we can safely assume that bytes can be deserialized

* use Signature instead of String

* use solana_sdk:#️⃣:Hash for blockhash

* store VersionedMessage instead of serialized bas64 string

* refactor compute_budget extraction

* fix test data

* simplify map_compute_budget_instructions

* fmt

---------

Co-authored-by: godmodegalactus <godmodegalactus@gmail.com>
2024-03-01 17:21:48 +01:00
Groovie | Mango 7bba02f3c5
wrap producedblock struct and avoid cloning it (#336)
wrap ProducedBlock in Arc
2024-02-22 14:28:29 +01:00
Groovie | Mango 2c42a536b6
Improve/geyser replace streams (#330)
* add env ENABLE_INSTRUMENT_TRACING to enable tracing

* add tracing dependency

* example tracings

* log recent_processed_blocks.len

* bound channels

* replace fastestwins with task version

* revert fake error

* fix meta stream

* disble almost all

* disable more

* keep slot channel up

* log memory usage using CAP

* log CAP memory usage

* add span tracing to mutliplex tasks

* merge

* fix import

* remove useless spans

* fix span logger

* log emitted data from multiplexer

* improve naming of mpsc from autoconnection task

* WIP: use task/mpsc autoconnect for full blocks

* use task/mpsc autoconnect for block meta too

* tag the channels

* add spans

* bump to use updated geyser-grpc-connector with mpsc

* remove unused functions

* extract constant

* split plugger for one and many downstreams

* clippy+fmt

* use task for slots too

* remove broadcast intermediate channel
2024-02-15 21:49:15 +01:00
galactus bcc3c37346
Accounts support (#327)
* Slitting rpc and pubsub services, adding few class and rpc, pubsub callbacks for accounts

* Implementing account storage and service writing some tests

* Chainging to interface, adding new tests

* Implementing accounts related rpc endpoints

* Adding tests for account filters

* Minor bug fixed and errors

* Fixing issue with get program accounts

* Implementing subscription

* use auto reconnect (#329)

* groovie comments part_1

* Fixing tests, fixing groovies comments

* Adding workspace to test filters, minor changes

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-02-14 19:20:25 +01:00
Groovie | Mango 38da1ed987
Improvement/check block sequencing (#299)
use grpc multiplex and improve sequencing
2024-02-07 15:51:22 +01:00
galactus 196b400e15
Adding support for address lookup tables (#319)
* Initial implementation of address lookup tables

* Implementing fetching of address lookup tables

* Use address lookup tables to calculate account prio fees

* Adding some logs and renaming classes

* preloading all the alts correctly

* Changes after groovies review

* Fixing issues after rebase
2024-02-06 17:59:46 +01:00
galactus 27b1e6ad16
Calculating prio fees by account (#310)
* Calculating readable and writable accounts for transaction and saving in transaction info, declaration of primary strutures

Implementing calculation of prioritization by accounts

Optimize calculation of prioritization fees stats (#301)

* Optimize calculation of prioritization fees stats

* comment on arrays in response

* Fix the calculation of supp

* Fix fmt

---------

Co-authored-by: GroovieGermanikus <groovie@mango.markets>

Implementing calculation of prioritization by accounts

Adding endpoint methods for prioritization fees by account

* Implement account stats subscription

* Fixing the serialization and deserialization of last

* Changes after groovies review
2024-02-01 18:49:51 +01:00
godmodegalactus 6c61e64948
Restart connection if any stream fails 2024-01-26 22:25:19 +01:00
Groovie | Mango 7171b524f0
Feature/block priofees (#274)
* boilerplate

* add dummy endpoint

* simple logic with highes block

* endpoint works

* remove cu calculus

* log get_latest_block startup timing

* websocket subscription basics

* WIP

* propagate fees via channel

* cleanup

* use broadcast sender

* proper handling of broadcast channel issues

* rename stuff

* move to dedicated crate

* cleanup deps

* move data definition

* hide private types

* percentile math

* code format

* integrated stats by cu

* add cu stats

* restart inline asserts

* warn about DashMap access

* switch to BTreeMap

* remove vote transactions from calculus

* return vec

* array

* flat format

* reformat

* split arrays

* rename keys

* enable experimental tag

* code format

* more logging

* reset log level

* rename websocket method to blockPrioritizationFeesSubscribe

* HACK: use processed blocks

* add is_vote_transaction

* do not fail if fees goes down

* udp message size example

* imrpove 100 handling

* simplify inital sort

* Revert "HACK: use processed blocks"

This reverts commit ff17d9475c.

* add percentile test case

* add per block cu_consumed+tx_count(nonvote)

* fix supid cleanup bug

* add TxAggregateStats

* add by_cu test

* add alternative cu percentlie impl

* cleanup cu tests

* add check for step

* saturating_sub

* remove useless file

* add is_vote to history crate

* fix fmt+clippy (nightly)

* clippy

* clippy

* clippy

* clippy
2024-01-25 10:29:14 +01:00
Groovie | Mango e3bfaac726
remove block_debug_listen (#286)
* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* clippy
2024-01-18 13:57:57 +01:00
galactus cfb5e1ac66
Moving geyser slot subscription from stream to channels (#282)
* Moving geyser slot subscription from stream to channels

* Closing all the slot subscription tasks incase of restart
2024-01-17 15:17:49 +01:00
galactus 6f61d89743
Fixing the performance issue with geyser multiplexing (#275) 2024-01-15 20:22:59 +01:00
godmodegalactus 78e1bb2b79
More clippy changes 2024-01-12 16:39:19 +01:00
Groovie | Mango 5cfb6de72a
grpc multiplexing integrate (#255)
allow multiple GRPC sources
2023-12-22 14:42:20 +01:00
Aniket Prajapati a4fdc1554a
-D clippy (#252)
* -D clippy

* scripts

* fmt

* commented out

* diff clippy and build

* fix

* push -> main
2023-12-08 03:26:41 +05:30
godmodegalactus 7113046d66
Resubscribing grpc block subsciption on failure, get vote accounts every 10 minutes instead of 1s 2023-12-01 11:12:54 +01:00
Godmode Galactus e6d2ab897e
Update grpc version to 0.11.0 so that we can use the latest triton service 2023-11-14 10:42:58 +01:00
Arrowana 15ffb44f6e allow x-token 2023-11-07 21:04:17 +11:00
godmodegalactus fc1810cc89
Removing postgres_session and moving it to history, start to implement saving blocks 2023-10-04 15:25:06 +02:00
godmodegalactus 2730ea504e
changing name blockhash to recent_blockhash 2023-10-03 12:51:18 +02:00
godmodegalactus 480631de72
Save all transactions in the cache, save transaction messages too, Increase transaction store size 2023-10-03 11:36:38 +02:00
godmodegalactus 886c80faa6
adding previous blockhash and renaming txs to transactions 2023-09-21 16:08:41 +02:00
Godmode Galactus 7ce00a2642
changes after groovies review 2023-09-19 16:49:46 +02:00
GroovieGermanikus e2351d7a95 warn on signature error 2023-09-04 18:21:35 +02:00
GroovieGermanikus 6cfaa77a99 replace deprecated Signature::new 2023-09-04 15:16:04 +02:00
GroovieGermanikus d91b1d9d20 opt for borsh0_10 instead of borsh (0.9) as suggested 2023-09-04 15:09:51 +02:00
Godmode Galactus 3a8ce5e95c
Bugfix adding gprc support 2023-08-31 13:56:33 +02:00
Godmode Galactus a418fe7fc7
Refactoring the code from gyser branch and merging into master 2023-08-31 12:34:13 +02:00