This commit is contained in:
Greg Fitzgerald 2021-03-02 06:36:49 -08:00 committed by GitHub
parent d73af9c1dd
commit 2463cc1e6a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 16 additions and 18 deletions

View File

@ -310,7 +310,7 @@ impl RepairWeight {
// Attempts to chain the orphan subtree rooted at `orphan_tree_root`
// to any earlier subtree with new any ancestry information in `blockstore`.
// Returns the earliest known ancestor of `heavest_tree_root`.
// Returns the earliest known ancestor of `heaviest_tree_root`.
fn update_orphan_ancestors(
&mut self,
blockstore: &Blockstore,

View File

@ -4,9 +4,9 @@ title: Persistent Account Storage
## Persistent Account Storage
The set of Accounts represent the current computed state of all the transactions that have been processed by a validator. Each validator needs to maintain this entire set. Each block that is proposed by the network represents a change to this set, and since each block is a potential rollback point the changes need to be reversible.
The set of accounts represent the current computed state of all the transactions that have been processed by a validator. Each validator needs to maintain this entire set. Each block that is proposed by the network represents a change to this set, and since each block is a potential rollback point, the changes need to be reversible.
Persistent storage like NVMEs are 20 to 40 times cheaper than DDR. The problem with persistent storage is that write and read performance is much slower than DDR and care must be taken in how data is read or written to. Both reads and writes can be split between multiple storage drives and accessed in parallel. This design proposes a data structure that allows for concurrent reads and concurrent writes of storage. Writes are optimized by using an AppendVec data structure, which allows a single writer to append while allowing access to many concurrent readers. The accounts index maintains a pointer to a spot where the account was appended to every fork, thus removing the need for explicit checkpointing of state.
Persistent storage like NVMEs are 20 to 40 times cheaper than DDR. The problem with persistent storage is that write and read performance is much slower than DDR. Care must be taken in how data is read or written to. Both reads and writes can be split between multiple storage drives and accessed in parallel. This design proposes a data structure that allows for concurrent reads and concurrent writes of storage. Writes are optimized by using an AppendVec data structure, which allows a single writer to append while allowing access to many concurrent readers. The accounts index maintains a pointer to a spot where the account was appended to every fork, thus removing the need for explicit checkpointing of state.
## AppendVec

View File

@ -37,11 +37,6 @@ use std::{
sync::{Arc, Mutex},
};
#[derive(Default, Debug, AbiExample)]
pub(crate) struct ReadonlyLock {
lock_count: Mutex<u64>,
}
#[derive(Debug, Default, AbiExample)]
pub struct AccountLocks {
write_locks: HashSet<Pubkey>,
@ -1041,8 +1036,6 @@ pub fn update_accounts_bench(accounts: &Accounts, pubkeys: &[Pubkey], slot: u64)
#[cfg(test)]
mod tests {
// TODO: all the bank tests are bank specific, issue: 2194
use super::*;
use crate::rent_collector::RentCollector;
use solana_sdk::{

View File

@ -58,8 +58,8 @@ impl SlotCacheInner {
self.cache
.get(pubkey)
// 1) Maybe can eventually use a Cow to avoid a clone on every read
// 2) Popping is only safe if its guaranteed only replay/banking threads
// are reading from the AccountsDb
// 2) Popping is only safe if it's guaranteed that only
// replay/banking threads are reading from the AccountsDb
.map(|account_ref| account_ref.value().clone())
}

View File

@ -5002,7 +5002,6 @@ impl AccountsDb {
#[cfg(test)]
pub mod tests {
// TODO: all the bank tests are bank specific, issue: 2194
use super::*;
use crate::{
accounts_hash::MERKLE_FANOUT, accounts_index::tests::*, accounts_index::RefCount,

View File

@ -91,7 +91,7 @@ impl CumulativeOffset {
}
}
// Allow retreiving &[start..end] from a logical src: Vec<T>, where src is really Vec<Vec<T>> (or later Vec<Vec<Vec<T>>>)
// Allow retrieving &[start..end] from a logical src: Vec<T>, where src is really Vec<Vec<T>> (or later Vec<Vec<Vec<T>>>)
// This model prevents callers from having to flatten which saves both working memory and time.
#[derive(Default, Debug)]
pub struct CumulativeOffsets {
@ -618,7 +618,6 @@ impl AccountsHash {
#[cfg(test)]
pub mod tests {
// TODO: all the bank tests are bank specific, issue: 2194
use super::*;
use std::str::FromStr;

View File

@ -306,7 +306,7 @@ impl<T: 'static + Clone + IsCached + ZeroLamport> AccountsIndex<T> {
// First we show that for any bank `B` that is a descendant of
// the current `max_root`, it must be true that and `B.ancestors.contains(max_root)`,
// regardless of the pattern of `squash()` behavior, `where` `ancestors` is the set
// regardless of the pattern of `squash()` behavior, where `ancestors` is the set
// of ancestors that is tracked in each bank.
//
// Proof: At startup, if starting from a snapshot, generate_index() adds all banks
@ -338,7 +338,7 @@ impl<T: 'static + Clone + IsCached + ZeroLamport> AccountsIndex<T> {
// BankForks before the `set_root`.
//
// This means by the guarantees of `R_descendants` described above, because
// `R_new` is an ancestor of `B`, and `R < R_new < B`, then B.ancestors.contains(R_new)`.
// `R_new` is an ancestor of `B`, and `R < R_new < B`, then `B.ancestors.contains(R_new)`.
//
// Now until the next `set_root`, any new banks constructed from `new_from_parent` will
// also have `max_root == R_new` in their ancestor set, so the claim holds for those descendants

View File

@ -1,3 +1,6 @@
//! Persistent storage for accounts. For more information, see:
//! https://docs.solana.com/implemented-proposals/persistent-account-storage
use log::*;
use memmap2::MmapMut;
use serde::{Deserialize, Serialize};
@ -132,7 +135,7 @@ impl Drop for AppendVec {
if let Err(_e) = remove_file(&self.path) {
// promote this to panic soon.
// disabled due to many false positive warnings while running tests.
// blocked by rpc's updrade to jsonrpc v17
// blocked by rpc's upgrade to jsonrpc v17
//error!("AppendVec failed to remove {:?}: {:?}", &self.path, e);
}
}
@ -165,10 +168,14 @@ impl AppendVec {
})
.unwrap();
// Theoretical performance optimization: write a zero to the end of
// the file so that we won't have to resize it later, which may be
// expensive.
data.seek(SeekFrom::Start((size - 1) as u64)).unwrap();
data.write_all(&[0]).unwrap();
data.seek(SeekFrom::Start(0)).unwrap();
data.flush().unwrap();
//UNSAFE: Required to create a Mmap
let map = unsafe { MmapMut::map_mut(&data) };
let map = map.unwrap_or_else(|e| {