Commit Graph

123 Commits

Author SHA1 Message Date
godmodegalactus f954980e40
using solana-account-manager without tokio 2024-07-16 13:19:37 +02:00
godmodegalactus e6cc877cff
resubscribing to quic blocks, slots and block metas 2024-07-15 16:47:23 +02:00
godmodegalactus f5f2833e24
Incrementing correctly accounts counter and cargo update 2024-07-15 11:08:05 +02:00
godmodegalactus a7bfdf8ff1
subscribe to deleted accounts, change lz4 compression level 2024-07-11 22:37:17 +02:00
godmodegalactus 83a422b43a
Creating a new repository for accounts related code and removing it from lite-rpc 2024-07-11 10:33:15 +02:00
godmodegalactus 9419d005e2
adding prometheus metrics to quic_geyser_plugin 2024-06-25 16:29:14 +02:00
godmodegalactus 6f2e89adb6
increasing size of accounts braodcast channel 2024-06-21 16:40:15 +02:00
godmodegalactus be8cbb3269
adding more tests for simple_filter_store 2024-06-21 16:26:15 +02:00
godmodegalactus 6927a65530
implementing comparison for account filters, implemeting gpa fallback,moving filter to pubkey 2024-06-20 16:34:31 +02:00
godmodegalactus 337ee91a0e
rebasing and version updates 2024-06-17 15:59:33 +02:00
godmodegalactus 50f304e19a
adding more tests, related to accounts processing 2024-06-17 15:20:50 +02:00
godmodegalactus a7b935126c
adding quic geyser plugin cluster endpoint 2024-06-17 15:20:49 +02:00
godmodegalactus e16a516b45
Adding filtering by program id in AccountStorageItself 2024-06-17 15:20:46 +02:00
godmodegalactus 76aded7ed0
Making blocks and accounts subscription compressed, slot and blockinfo uncompressed
Reducing log levels

Increasing size of channels

Setting max slot for account update

Updating libraries, moving grpc, other changes
2024-06-17 15:20:08 +02:00
godmodegalactus 8003dab58f
Saving compressed accounts in Inmemory, adding lz4 and zstd compression methods
Saving account updates during a slot inmemory store

Getting compressed accounts with tonic gzip compression

cargo fmt and clippy
2024-06-17 15:19:09 +02:00
Groovie | Mango 0d1d856305
Update to solana 1.18 (#397)
* update solana from 1.17.28 -> 1.18.15
* remove legacy cu
* use yellowstone 1.15
* use tagged geyser-grpc-connector version
2024-06-13 13:45:52 +02:00
godmodegalactus 5dd65155bf
processed slots should be always increasing 2024-05-30 13:43:49 +02:00
Groovie | Mango d10910e737
reducing yellowstone dependencies (#389)
* WIP - not compiling

* remove connect_hacked

* use reimported types

* cleanup

* fix fmt
2024-04-17 15:11:51 +02:00
Groovie | Mango b84e880961
remove async_channel and bring back tokio mpsc§ (#384) 2024-04-03 11:22:36 +02:00
Groovie | Mango 69d7dbb123
optimize mapping of produced block (#382)
optimize mapping of produced block
2024-04-03 10:40:53 +02:00
godmodegalactus 12a6832c56
Swapping notify channel with broadcast channel 2024-04-02 15:07:01 +02:00
godmodegalactus ca3fa46139
Using broadcast channels instead of notify 2024-04-02 14:48:54 +02:00
godmodegalactus 91cf06436a
Changing exit signal from notify to broadcast channel in grpc multiplexer 2024-04-02 14:45:41 +02:00
galactus 681334197f
Adding custom tpu send transaction example (#380)
* Adding an example for custom tpu send transaction

* Fixing the custom tpu example

* Optimizing SentTransactionInfo, and calculating TPS

* Reverting unwanted changes

* After groovies review
2024-04-02 14:26:54 +02:00
godmodegalactus 1e956a2b4f
Revert "Replacing notify by cancellation token to correctly shutdown tasks"
This reverts commit d1abc345cf.
2024-03-28 11:49:20 +01:00
godmodegalactus 02229060cf
Revert "Fix clippy fmt"
This reverts commit bac036ceaa.
2024-03-28 11:49:17 +01:00
godmodegalactus bac036ceaa
Fix clippy fmt 2024-03-28 09:12:12 +01:00
godmodegalactus d1abc345cf
Replacing notify by cancellation token to correctly shutdown tasks 2024-03-28 09:09:42 +01:00
godmodegalactus dfe8d972eb
Porting back the exit notify change 2024-03-27 17:57:40 +01:00
galactus 79748c5a58
Merge production main 2024 03 27 (#374)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Removing unwanted logs

* Add logs on block queues and reception time

* reduce level of block reception log

* Revert "Solving issue of finalized meta after processed block (#365)"

This reverts commit c09700fd79.

Production runs show more stable memory use but worse performance on
landing transactions.

* use yellowstone grpc with hacked-windowsize

* upgraded geyser-grpc-connector + increased timeout

* Fix block stream throughput problems

By temporarily pasting in a function to connect to block streams via a
more generously configured endpoint.

* Enabling unstable tokio on fly

* Fixing prioritization heap metrics (#370)

* Fixing prioritization heap metrics

* Fixing transaction sizes tests

* Correctly ending the block streams instead of aborting them

* Replacing exit signal with exit notification (#372)

* Replacing exit signal with exit notification

* Deprecitating nightly version

* Increase connection size

* Fixing issue with multiple notify channels

* Joining heap task instead of aborting

* fix postgres ssl algo issue - RC2-40-CBC  (#373)

* fix 373: openssl-legacy.cnf

* Remove replace console subscriber with tracing subscriber

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-03-27 17:09:38 +01:00
godmodegalactus 7d39a947ac
Minor changes, increasing default lite-rpc limits 2024-03-26 17:36:11 +01:00
galactus 29bd6deae4
Fixing accounts on demand (#371)
* Fixing accounts on demand

* Adding missing file and renaming gprc to grpc

* Adding missing grpc_accounts_streaming
2024-03-26 14:32:50 +01:00
galactus 087cc7f204
Merge prod main 2024 03 25 (#369)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Removing unwanted logs

* Add logs on block queues and reception time

* reduce level of block reception log

* Revert "Solving issue of finalized meta after processed block (#365)"

This reverts commit c09700fd79.

Production runs show more stable memory use but worse performance on
landing transactions.

* use yellowstone grpc with hacked-windowsize

* upgraded geyser-grpc-connector + increased timeout

* Fix block stream throughput problems

By temporarily pasting in a function to connect to block streams via a
more generously configured endpoint.

* Enabling unstable tokio on fly

* update to v0.10.3+yellowstone.1.12+solana.1.17.15-hacked-windowsize3

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
2024-03-25 17:44:57 +01:00
Groovie | Mango cd9df11501
use geyser blockmeta to progress finalizedslot (#367)
introduce BlockInfo on a new BlockInfoStream; maps from yellowstone BlockMeta
fix: last_finalized_slot must progress unconditionally
2024-03-22 19:22:38 +01:00
galactus defdc20dd5
Getting finalized meta before processed main (#366)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Minor changes

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
2024-03-22 12:30:14 +01:00
GroovieGermanikus ae495bed53
Merge branch 'production'
using 1.17.15 geyser
2024-03-22 10:57:20 +01:00
galactus 7e6a1a4003
Reverting back the geyser version (#364) 2024-03-22 09:55:46 +01:00
godmodegalactus 50b81135c3
Merge branch 'production' 2024-03-18 20:01:02 +01:00
godmodegalactus 85fcb6097e
Adding some more metrics for quic tpu, and changing some parameters 2024-03-18 17:25:52 +01:00
godmodegalactus 9be86919ba
Merge branch 'production' 2024-03-16 16:18:01 +01:00
GroovieGermanikus 1e3a9cdcbe
update geyser-grpc-connector with attempt fix 2024-03-11 18:26:59 +01:00
galactus 33ad19ed47
Removing verbose log for grpc multiplexing (#354) 2024-03-11 13:48:15 +01:00
godmodegalactus c59f3f8f38
Updating docker file with correct version info 2024-03-08 11:40:55 +01:00
Groovie | Mango 18a0ab7062
from grpc block ProduceBlock mapping - Performance+Typing (#348)
* remove all Pubkey::default();
1. the value does not make sense
2. we can safely assume that bytes can be deserialized

* use Signature instead of String

* use solana_sdk:#️⃣:Hash for blockhash

* store VersionedMessage instead of serialized bas64 string

* refactor compute_budget extraction

* fix test data

* simplify map_compute_budget_instructions

* fmt

---------

Co-authored-by: godmodegalactus <godmodegalactus@gmail.com>
2024-03-01 17:21:48 +01:00
galactus 8ef1b56cdb
account_subscription_into_multiple_channels_and_merge_them (#344)
* account_subscription_into_multiple_channels_and_merge_them

* Removing unecessary changes

* Minor fix after groovies comments
2024-03-01 14:45:28 +01:00
galactus 49bb7ff9f3
Implementing account on demand (#338)
* Implementing account on demand

* Resubscribing accounts after every sometime

* Changing account stream commitment to processed instead

* Fixing account on demand, and account subscription bugs

* Changing variable name for commitment from confirmed to processed

* Some account optimizations

* Adding get balance call

* Changes after groovies review.

* On demand accounts subscribe first then delete old subscription
2024-02-28 15:40:11 +01:00
Groovie | Mango 855c0190eb
prepare new "util" crate (#333)
prepare new "util" crate
2024-02-22 16:54:42 +01:00
Groovie | Mango 7bba02f3c5
wrap producedblock struct and avoid cloning it (#336)
wrap ProducedBlock in Arc
2024-02-22 14:28:29 +01:00
Groovie | Mango 2c42a536b6
Improve/geyser replace streams (#330)
* add env ENABLE_INSTRUMENT_TRACING to enable tracing

* add tracing dependency

* example tracings

* log recent_processed_blocks.len

* bound channels

* replace fastestwins with task version

* revert fake error

* fix meta stream

* disble almost all

* disable more

* keep slot channel up

* log memory usage using CAP

* log CAP memory usage

* add span tracing to mutliplex tasks

* merge

* fix import

* remove useless spans

* fix span logger

* log emitted data from multiplexer

* improve naming of mpsc from autoconnection task

* WIP: use task/mpsc autoconnect for full blocks

* use task/mpsc autoconnect for block meta too

* tag the channels

* add spans

* bump to use updated geyser-grpc-connector with mpsc

* remove unused functions

* extract constant

* split plugger for one and many downstreams

* clippy+fmt

* use task for slots too

* remove broadcast intermediate channel
2024-02-15 21:49:15 +01:00
galactus bcc3c37346
Accounts support (#327)
* Slitting rpc and pubsub services, adding few class and rpc, pubsub callbacks for accounts

* Implementing account storage and service writing some tests

* Chainging to interface, adding new tests

* Implementing accounts related rpc endpoints

* Adding tests for account filters

* Minor bug fixed and errors

* Fixing issue with get program accounts

* Implementing subscription

* use auto reconnect (#329)

* groovie comments part_1

* Fixing tests, fixing groovies comments

* Adding workspace to test filters, minor changes

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-02-14 19:20:25 +01:00