Commit Graph

106 Commits

Author SHA1 Message Date
Groovie | Mango 0d1d856305
Update to solana 1.18 (#397)
* update solana from 1.17.28 -> 1.18.15
* remove legacy cu
* use yellowstone 1.15
* use tagged geyser-grpc-connector version
2024-06-13 13:45:52 +02:00
Groovie | Mango d10910e737
reducing yellowstone dependencies (#389)
* WIP - not compiling

* remove connect_hacked

* use reimported types

* cleanup

* fix fmt
2024-04-17 15:11:51 +02:00
godmodegalactus 1e956a2b4f
Revert "Replacing notify by cancellation token to correctly shutdown tasks"
This reverts commit d1abc345cf.
2024-03-28 11:49:20 +01:00
godmodegalactus d1abc345cf
Replacing notify by cancellation token to correctly shutdown tasks 2024-03-28 09:09:42 +01:00
Groovie | Mango 3c994597b4
benchrunner service (#363)
benchrunner service
+ minor changes in bench code
2024-03-26 13:41:46 +01:00
galactus 087cc7f204
Merge prod main 2024 03 25 (#369)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Removing unwanted logs

* Add logs on block queues and reception time

* reduce level of block reception log

* Revert "Solving issue of finalized meta after processed block (#365)"

This reverts commit c09700fd79.

Production runs show more stable memory use but worse performance on
landing transactions.

* use yellowstone grpc with hacked-windowsize

* upgraded geyser-grpc-connector + increased timeout

* Fix block stream throughput problems

By temporarily pasting in a function to connect to block streams via a
more generously configured endpoint.

* Enabling unstable tokio on fly

* update to v0.10.3+yellowstone.1.12+solana.1.17.15-hacked-windowsize3

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
2024-03-25 17:44:57 +01:00
galactus defdc20dd5
Getting finalized meta before processed main (#366)
* Use jemalloc

* Solving issue of finalized meta after processed block (#365)

* Solving issue of finalized meta after processed block

* Fixing the broken test

* Minor changes

---------

Co-authored-by: Christian Kamm <mail@ckamm.de>
2024-03-22 12:30:14 +01:00
GroovieGermanikus ae495bed53
Merge branch 'production'
using 1.17.15 geyser
2024-03-22 10:57:20 +01:00
galactus 7e6a1a4003
Reverting back the geyser version (#364) 2024-03-22 09:55:46 +01:00
godmodegalactus 50b81135c3
Merge branch 'production' 2024-03-18 20:01:02 +01:00
godmodegalactus 85fcb6097e
Adding some more metrics for quic tpu, and changing some parameters 2024-03-18 17:25:52 +01:00
galactus 8ef1b56cdb
account_subscription_into_multiple_channels_and_merge_them (#344)
* account_subscription_into_multiple_channels_and_merge_them

* Removing unecessary changes

* Minor fix after groovies comments
2024-03-01 14:45:28 +01:00
galactus 49bb7ff9f3
Implementing account on demand (#338)
* Implementing account on demand

* Resubscribing accounts after every sometime

* Changing account stream commitment to processed instead

* Fixing account on demand, and account subscription bugs

* Changing variable name for commitment from confirmed to processed

* Some account optimizations

* Adding get balance call

* Changes after groovies review.

* On demand accounts subscribe first then delete old subscription
2024-02-28 15:40:11 +01:00
galactus 064d47719e
Creating an example directory and adding example to be implemented (#340) 2024-02-26 20:43:02 +01:00
Groovie | Mango 855c0190eb
prepare new "util" crate (#333)
prepare new "util" crate
2024-02-22 16:54:42 +01:00
galactus 0ab947b7d7
Updating jsonrpsee and fixing cors (#332)
* Updating json rpsee and fixing CORS

* Fixing cors and updating jsonrpsee and fixing a bug in address lookup table store

* Adding rpc errors, and fixing groovies comments

* for moment get performance data from rpc client
2024-02-21 14:55:44 +01:00
galactus bcc3c37346
Accounts support (#327)
* Slitting rpc and pubsub services, adding few class and rpc, pubsub callbacks for accounts

* Implementing account storage and service writing some tests

* Chainging to interface, adding new tests

* Implementing accounts related rpc endpoints

* Adding tests for account filters

* Minor bug fixed and errors

* Fixing issue with get program accounts

* Implementing subscription

* use auto reconnect (#329)

* groovie comments part_1

* Fixing tests, fixing groovies comments

* Adding workspace to test filters, minor changes

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
2024-02-14 19:20:25 +01:00
Groovie | Mango 38da1ed987
Improvement/check block sequencing (#299)
use grpc multiplex and improve sequencing
2024-02-07 15:51:22 +01:00
galactus 196b400e15
Adding support for address lookup tables (#319)
* Initial implementation of address lookup tables

* Implementing fetching of address lookup tables

* Use address lookup tables to calculate account prio fees

* Adding some logs and renaming classes

* preloading all the alts correctly

* Changes after groovies review

* Fixing issues after rebase
2024-02-06 17:59:46 +01:00
Groovie | Mango fae4cecc0e
Improvement/saving blocks use lookup tables (#318)
block store basic implementation
2024-02-05 20:20:15 +01:00
galactus 27b1e6ad16
Calculating prio fees by account (#310)
* Calculating readable and writable accounts for transaction and saving in transaction info, declaration of primary strutures

Implementing calculation of prioritization by accounts

Optimize calculation of prioritization fees stats (#301)

* Optimize calculation of prioritization fees stats

* comment on arrays in response

* Fix the calculation of supp

* Fix fmt

---------

Co-authored-by: GroovieGermanikus <groovie@mango.markets>

Implementing calculation of prioritization by accounts

Adding endpoint methods for prioritization fees by account

* Implement account stats subscription

* Fixing the serialization and deserialization of last

* Changes after groovies review
2024-02-01 18:49:51 +01:00
galactus 60388667f5
Main into production 01/02/2024 (#312)
* Production into main (#303)

* fix: panic on geyser close, multiplex bug

https://github.com/blockworks-foundation/geyser-grpc-connector/issues/3

* update Cargo.lock

* reverting cargo.lock

* Fix issues with grpc and postgres

* Solving merge issues

* Fixing cargo fmt

* Increase finish quic timeout (#280) (#281)

* integrate geyser slot subscription (#283)

* Increase finish quic timeout (#280)

* Moving geyser slot subscription from stream to channels (#282)

* Moving geyser slot subscription from stream to channels

* Closing all the slot subscription tasks incase of restart

* Making slot channel unbounded (bug)

* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* Update cargolock file

* Fixing clippy removing grpc_inspect

* merging main with production (#290)

* remove block_debug_listen (#286)

* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* clippy

* Fixing message too long and overflow panics (#288)

* Update geyser grpc connector commit (#289)

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>

* Merging MTU changes and setting up transportation config (#293)

* remove block_debug_listen (#286)

* remove block_debug_listen

caused a panic - need more time to investigate

2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] thread 'tokio-runtime-worker' panicked at cluster-endpoints/src/grpc_inspect.rs:59:21:
2024-01-17T20:31:42.913 app[683d392fd45368] ams [info] Error receiving block: Closed
2024-01-17T20:31:42.922 app[683d392fd45368] ams [info] 2024-01-17T20:31:42.912597Z ERROR lite_rpc: Services quit unexpectedly Err(cluster endpoint failure (Err(JoinError::Panic(Id(20), ...)), 1, [JoinHandle { id: Id(19) }, JoinHandle { id: Id(23) }])

* clippy

* Fixing message too long and overflow panics (#288)

* Update geyser grpc connector commit (#289)

* Updating the transport config to match with solana endpoint (#292)

* Updating the transport config to match with solana endpoint

* Setting max MTU after groovies comments

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>

* Making block subscription processed and moving confirmed block subscr… (#291)

* Making block subscription processed and moving confirmed block subscription to meta

* Sending both processed and confirmed blocks, if block has already been confirmed

* Minor bug, subscribing to processed blocks instead of confirmed (#295)

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>

* Restart connection if any stream fails

* Updating to version 0.2.4

* Updating the change logs

* Fixing all the sub overflows by using saturating sub (#309)

* Fixing by cu computation adding more tests to check by CU (#311)

* Fixing by cu computation adding more tests to check by CU

* Adding more tests

---------

Co-authored-by: Groovie | Mango <95291500+grooviegermanikus@users.noreply.github.com>
Co-authored-by: GroovieGermanikus <groovie@mango.markets>
2024-02-01 15:08:47 +01:00
godmodegalactus a44923c334
Updating to version 0.2.4 2024-01-29 13:11:54 +01:00
Groovie | Mango 7171b524f0
Feature/block priofees (#274)
* boilerplate

* add dummy endpoint

* simple logic with highes block

* endpoint works

* remove cu calculus

* log get_latest_block startup timing

* websocket subscription basics

* WIP

* propagate fees via channel

* cleanup

* use broadcast sender

* proper handling of broadcast channel issues

* rename stuff

* move to dedicated crate

* cleanup deps

* move data definition

* hide private types

* percentile math

* code format

* integrated stats by cu

* add cu stats

* restart inline asserts

* warn about DashMap access

* switch to BTreeMap

* remove vote transactions from calculus

* return vec

* array

* flat format

* reformat

* split arrays

* rename keys

* enable experimental tag

* code format

* more logging

* reset log level

* rename websocket method to blockPrioritizationFeesSubscribe

* HACK: use processed blocks

* add is_vote_transaction

* do not fail if fees goes down

* udp message size example

* imrpove 100 handling

* simplify inital sort

* Revert "HACK: use processed blocks"

This reverts commit ff17d9475c.

* add percentile test case

* add per block cu_consumed+tx_count(nonvote)

* fix supid cleanup bug

* add TxAggregateStats

* add by_cu test

* add alternative cu percentlie impl

* cleanup cu tests

* add check for step

* saturating_sub

* remove useless file

* add is_vote to history crate

* fix fmt+clippy (nightly)

* clippy

* clippy

* clippy

* clippy
2024-01-25 10:29:14 +01:00
galactus de26490e9c
Squash and merging all the commits from groovies branch groovie/post… (#300)
* Squash and merging all the commits from  groovies branch groovie/postgres_saving_blocksII

* Ingore tests which require postgres
2024-01-22 15:22:11 +01:00
galactus bea39d8c6a
Migrating to solana v1.17.15 (#273) 2024-01-15 18:43:48 +01:00
musitdev 21a8f12ed4 correct clippy errors 2024-01-04 14:05:11 +01:00
musitdev dc37b865f6 update cargo.toml with workplace 2024-01-03 10:20:18 +01:00
musitdev 3e4c81c567 Stake and vote management. Integration of the main loop in current architecture 2024-01-03 10:20:14 +01:00
Godmode Galactus e6d2ab897e
Update grpc version to 0.11.0 so that we can use the latest triton service 2023-11-14 10:42:58 +01:00
aniketfuryrocks d68ba7e3f0 implement rpc methods with a todo!() clause 2023-10-30 10:38:16 +05:30
galactus b6d0777528
Merge branch 'main' into remove_force_version_of_rustup 2023-10-14 13:33:08 +02:00
Steve 12c97b88d1
Merge pull request #228 from grooviegermanikus/groovie/disable-gso-flag
disable gso (generic segmentation offload)
2023-10-13 23:22:50 +02:00
godmodegalactus cb2982c65c
removing = from rustup version 2023-10-13 16:18:55 +02:00
GroovieGermanikus eb0fb9c6f4
cleanup cargo file 2023-10-09 17:45:08 +02:00
GroovieGermanikus d353569dbb
WIP: add DISABLE_GSO env flag 2023-10-04 20:44:52 +02:00
musitdev c6aba8f396 Epoch integration 2023-10-04 11:28:18 +02:00
godmodegalactus 7ef29e1af6
implementing history, block storage method
changes after groovies review
2023-09-21 15:39:25 +02:00
riordanp 9b4778373b
Deploy QUIC Proxy (#187)
* Load listen addr from env

* Add deployment config for quic-forward-proxy

* ipv6 support

* Read QUIC_PROXY_ADDR from env
2023-09-08 14:13:20 +01:00
GroovieGermanikus ffb9f3130f use solana 1.16.3+ (1.16.12 effectively) 2023-09-04 15:02:12 +02:00
Godmode Galactus a418fe7fc7
Refactoring the code from gyser branch and merging into master 2023-08-31 12:34:13 +02:00
GroovieGermanikus 0ad0563a65 Merge remote-tracking branch 'origin/main' into groovie/tpu-proxy-as-module-with-integration-test 2023-08-02 13:32:01 +02:00
GroovieGermanikus 9e064001f8 extract integration test 2023-07-20 14:08:38 +02:00
GroovieGermanikus 0490e557de merge integration test branch into proxy branch 2023-07-20 13:45:54 +02:00
GroovieGermanikus dea39eebda fix cargo 2023-07-20 09:31:59 +02:00
GroovieGermanikus b4b7a2a583 reset non-test code 2023-07-20 09:31:59 +02:00
GroovieGermanikus cdc3d1444d add proper tracing to test 2023-07-20 09:31:58 +02:00
GroovieGermanikus 77c801ff45 proxy forwarding works 2023-06-26 22:10:06 +02:00
Godmode Galactus 70eb250b10
Updating Cargo files with new version 2023-06-23 15:40:50 +02:00
GroovieGermanikus 56743e7695 integrate proxy with lite-rpc for testing 2023-06-22 18:54:09 +02:00