* save progress
* rename threads handler
* added writer for txs
* after extracting structure to handle tx confirmations
* extract LogWriter
* Replace pair TimestampedTransaction with struct
* add compute_unit_price to TimestampedTransaction
* add cu_price to LogWriter
* add block time to the logs
* Fix warnings
* add comments and restructure code
* some small improvements
* Renamed conformation_processing.rs to log_transaction_service.rs
* address numerous PR comments
* split LogWriter into two structs
* simplify code of LogWriters
* extract process_blocks
* specify commitment in LogTransactionService
* break thread loop if receiver happens to be dropped
* update start_slot when processing blocks
* address pr comments
* fix clippy error
* minor changes
* fix ms problem
* fix bug with time in clear transaction map
This is port of firedancer's implementation of weighted shuffle:
https://github.com/firedancer-io/firedancer/blob/3401bfc26/src/ballet/wsample/fd_wsample.chttps://github.com/anza-xyz/agave/pull/185
implemented weighted shuffle using binary tree. Though asymptotically a
binary tree has better performance, compared to a Fenwick tree, it has
less cache locality resulting in smaller improvements and in particular
slower WeightedShuffle::new.
In order to improve cache locality and reduce the overheads of
traversing the tree, this commit instead uses a generalized N-ary tree
with fanout of 16, showing significant improvements in both
WeightedShuffle::new and WeightedShuffle::shuffle.
With 4000 weights:
N-ary tree (fanout 16):
test bench_weighted_shuffle_new ... bench: 36,244 ns/iter (+/- 243)
test bench_weighted_shuffle_shuffle ... bench: 149,082 ns/iter (+/- 1,474)
Binary tree:
test bench_weighted_shuffle_new ... bench: 58,514 ns/iter (+/- 229)
test bench_weighted_shuffle_shuffle ... bench: 269,961 ns/iter (+/- 16,446)
Fenwick tree:
test bench_weighted_shuffle_new ... bench: 39,413 ns/iter (+/- 179)
test bench_weighted_shuffle_shuffle ... bench: 364,771 ns/iter (+/- 2,078)
The improvements become even more significant as there are more items to
shuffle. With 20_000 weights:
N-ary tree (fanout 16):
test bench_weighted_shuffle_new ... bench: 200,659 ns/iter (+/- 4,395)
test bench_weighted_shuffle_shuffle ... bench: 941,928 ns/iter (+/- 26,492)
Binary tree:
test bench_weighted_shuffle_new ... bench: 881,114 ns/iter (+/- 12,343)
test bench_weighted_shuffle_shuffle ... bench: 1,822,257 ns/iter (+/- 12,772)
Fenwick tree:
test bench_weighted_shuffle_new ... bench: 276,936 ns/iter (+/- 14,692)
test bench_weighted_shuffle_shuffle ... bench: 2,644,713 ns/iter (+/- 49,252)
#### Problem
As #72 introduced AccountsFile::TieredStorage, it also performs
file-type check when opening an accounts-file to determine whether
it is a tiered-storage or an append-vec. But before tiered-storage is
enabled, this opening check is unnecessary.
#### Summary of Changes
Remove the accounts-file type check code and simply assume everything
is append-vec on AccountsFile::new_from_file().
#### Problem
AccountsFile currently doesn't have an implementation for TieredStorage.
To enable AccountsDB tests for the TieredStorage, we need AccountsFile
to support TieredStorage.
#### Summary of Changes
This PR implements a AccountsFile::TieredStorage, a thin wrapper between
AccountsFile and TieredStorage.
#### Problem
The TieredStorage has not yet implemented the AccountsFile::capacity()
API.
#### Summary of Changes
Implement capacity() API for TieredStorage and limit file size to 16GB,
same as the append-vec file.
This is partial port of firedancer's implementation of weighted shuffle:
https://github.com/firedancer-io/firedancer/blob/3401bfc26/src/ballet/wsample/fd_wsample.c
Though Fenwick trees use less space, inverse queries require an
additional O(log n) factor for binary search resulting an overall
O(n log n log n) performance for weighted shuffle.
This commit instead uses a binary tree where each node contains the sum
of all weights in its left sub-tree. The weights themselves are
implicitly stored at the leaves. Inverse queries and updates to the tree
all can be done O(log n) resulting an overall O(n log n) weighted
shuffle implementation.
Based on benchmarks, this results in 24% improvement in
WeightedShuffle::shuffle:
Fenwick tree:
test bench_weighted_shuffle_new ... bench: 36,686 ns/iter (+/- 191)
test bench_weighted_shuffle_shuffle ... bench: 342,625 ns/iter (+/- 4,067)
Binary tree:
test bench_weighted_shuffle_new ... bench: 59,131 ns/iter (+/- 362)
test bench_weighted_shuffle_shuffle ... bench: 260,194 ns/iter (+/- 11,195)
Though WeightedShuffle::new is now slower, it generally can be cached
and reused as in Turbine:
https://github.com/anza-xyz/agave/blob/b3fd87fe8/turbine/src/cluster_nodes.rs#L68
Additionally the new code has better asymptotic performance. For
example with 20_000 weights WeightedShuffle::shuffle is 31% faster:
Fenwick tree:
test bench_weighted_shuffle_new ... bench: 255,071 ns/iter (+/- 9,591)
test bench_weighted_shuffle_shuffle ... bench: 2,466,058 ns/iter (+/- 9,873)
Binary tree:
test bench_weighted_shuffle_new ... bench: 830,727 ns/iter (+/- 10,210)
test bench_weighted_shuffle_shuffle ... bench: 1,696,160 ns/iter (+/- 75,271)
#### Problem
TieredStorage::file_size() essentially supports AccountsFile::len(),
but its API is inconsistent with AccountsFile's.
#### Summary of Changes
Refactor TieredStorage::file_size() to ::len() and share the same API
as AccountsFile's.
#### Test Plan
Build
Existing unit-tests.
#### Problem
The current implementation of TieredStorage::file_size() requires
a sys-call to provide the file size.
#### Summary of Changes
Add len() API to TieredStorageReader, and have HotStorageReader()
implement the API using Mmap::len().
#### Test Plan
Update existing unit-test to also verify HotStorageReader::len().
#### Problem
The current AppendVecId actually refers to an accounts file id.
#### Summary of Changes
Rename AppendVecId to AccountsFileId.
#### Test Plan
Build
* add set compute units arg for program deploy
* update master changes
* remove duplicates
* fixes and tests
* remove extra lines
* feedback
* Use simulation to determine compute units consumed
* feedback
---------
Co-authored-by: NagaprasadVr <nagaprasadvr246@gmail.com>
* Fix: deploy program on last slot of epoch during environment change
* solana-runtime: deploy at last epoch slot test
* disable deployment of sol_alloc_free
* Move tx-batch-constructor to its own function
* use new_from_cache
---------
Co-authored-by: Alessandro Decina <alessandro.d@gmail.com>
* Remove incorrect check
* Move to closure
* Use match statement instead
* Adjust rent_exempt_reserve by existing balance
* Only transfer lamports if rent_exempt_reserve needs are greater than 0
* Rename variable for clarity
* Add minimum-delegation check
* Bump test split amount to meet arbitrary mock minimum-delegation amount
* Add functions to collect executed transactions fee in details;
* remove unnecessary derive attributes
* change function name from add to accumulate; remove collector_fee_details from PartialEq
* add AbiExample
* add test
* share function to withdraw errored transaction
* more tests
* runtime: do fewer syscalls in remap_append_vec_file
Use renameat2(src, dest, NOREPLACE) as an atomic version of if
statx(dest).is_err() { rename(src, dest) }.
We have high inode contention during storage rebuild and this saves 1
fs syscall for each appendvec.
* Address review feedback
* Pass the final result of LastVotedForkSlots aggregation to next
stage and find the heaviest fork we will Gossip to others.
* Change comments.
* Small fixes to address PR comments.
* Move correctness proof to SIMD.
* Fix a broken merge.
* Use blockstore to check parent slot of any block in FindHeaviestFork
* Change error message.
* Add special message when first slot in the list doesn't link to root.
Bubble up the constants to the CLI that control the sizes of the
following two thread pools:
- The thread pool used to replay multiple forks in parallel
- The thread pool used to execute transactions in parallel
#### Problem
The TieredStorageFooter has the min_account_address and
max_account_address fields to describe the account address
range in its file. But the current implementation hasn't updated
the fields yet.
#### Summary of Changes
This PR enables the TieredStorage to persist address range
information into its footer via min_account_address and
max_account_address.
#### Test Plan
Updated tiered-storage test to verify persisted account address range.
#### Problem
The TieredStorage::new_readonly() function currently has the following
problems:
* It opens the file without checking the magic number before checking and loading the footer.
* It opens the file twice: first to load the footer, then open again by the reader.
#### Summary of Changes
This PR refactors TieredStorage::new_readonly() so that it first performs all
checks inside the constructor of TieredReadableFile. The TieredReadableFile
instance is then passed to the proper reader (currently HotStorageReader)
when all checks are passed.
#### Test Plan
* Added a new test to check MagicNumberMismatch.
* Existing tiered-storage tests
Joe C
samkim-crypto
kirill lykov
behzad nouri
carllin
Yueh-Hsuan Chiang
Lucas Steuernagel
Pankaj Garg
Tyera
Brooks
Justin Starry
Andrew Fitzgerald
Jon C
Alexander Meißner
blake
Greg Cusack
Tao Zhu
Alessandro Decina
Wen
Dmitri Makarov
steviez