7e65289729
Convert blockstore TransactionStatus column family to protobufs ( #15733 )
...
* Prevent panic if TransactionStatus can't be deserialized
* Convert Blockstore TransactionStatus column to protobuf
* Add compatability test
2021-03-05 09:05:35 -07:00
e92cff3efa
create-snapshot subcommad now accepts the ROOT keyword
2021-02-26 16:40:10 -08:00
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
170cb792eb
Return blockstore error if previous_blockhash cannot be determined ( #15382 )
...
* Return blockstore error if previous_blockhash cannot be determined
* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
0812931c38
Log if unsanitary transactions are read from blockstore ( #15319 )
2021-02-14 06:32:43 +00:00
da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks ( #15204 )
...
* Remove timestamp_correction feature gating
* Remove timestamp_bounding feature gating
* Remove unused deprecated ledger code
* Remove unused deprecated unbounded-timestamp code
* Enable independent adjustment of fast/slow timestamp bounding
* Update timestamp bounds to 25% fast, 80% slow; warp timestamp
* Update bank hash test
* Add PR number to feature
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
42bf6dc2ab
Revert hard nofile limit back to 500000
2021-02-03 19:43:55 +00:00
8376781ec8
Don't squash caught errors, please ( #15046 )
...
* Don't squash caught errors, please
* Update blockstore.rs
* Update blockstore.rs
2021-02-03 14:42:34 +00:00
894b412aef
docs: bump nofiles recommendations to match maps
2021-02-02 21:59:21 -07:00
1de6d28eaf
Add block_time to getConfirmedSignaturesForAddress2 and getConfirmedTransaction ( #14572 )
...
* add block_time to get_confirmed_signatures_for_address2 and protobuf implementation for tx_by_addr
* add tests for convert
* update cargo lock
* run cargo format after rebase
* introduce legacy TransactionByAddrInfo
* move LegacyTransactionByAddrInfo back to storage-bigtable
2021-01-20 22:10:35 -08:00
e1f2d384db
Don't require increased open file limit at ledger creation
...
Follow-up to 0b92720fdb
2020-12-17 08:46:13 -08:00
0b92720fdb
Don't require increased open file limit in solana-test-validator
...
Travis CI in particular does not allow the open file limit to be
increased.
2020-12-16 20:43:08 -08:00
ac0d32bc7e
Add blockstore skipped api ( #14145 )
...
* Add blockstore api to determine if a slot was skipped
* Return custom rpc error if slot is skipped
2020-12-16 13:40:36 -07:00
7143aaa89b
Clippy
2020-12-14 08:03:29 -08:00
13db3eca9f
SPL token balance in transaction metadata ( #13673 )
...
* feat: store pre / post token balances
* move helper functions into separate include
* move token balance functionality to transaction-status crate
* fix blockstore processor test
* fix bigtable legacy test
* add caching to decimals
2020-12-10 19:25:07 -08:00
aa2751e614
Check shred type in is_duplicate ( #14050 )
2020-12-10 18:20:08 -08:00
c5fe076432
Better dupe detection ( #13992 )
2020-12-09 23:14:31 -08:00
e1793e5a13
caches vote-state de-serialized from vote accounts ( #13795 )
...
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.
This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.
Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.
2020-11-30 17:18:33 +00:00
afc1b59475
Fix assertion failure ( #13626 )
...
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 21:30:38 -08:00
80db6c0980
Add bounded timestamp-estimation method
2020-10-31 21:30:42 -07:00
39686ef098
Use bank timestamp to populate Blockstore::blocktime_cf when correction active ( #13158 )
2020-10-26 19:23:45 +00:00
7bc073defe
Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand`
2020-10-21 19:08:13 -07:00
359707c85e
Convert Blockstore Rewards cf to protobuf ( #12860 )
...
* Add Blockstore protobuf cf type
* Add Rewards message to proto and make generated pub
* Convert Rewards cf to ProtobufColumn
* Add bench
* Adjust tags
* Move solana proto definitions and conversion methods to new crate
2020-10-15 18:04:10 -06:00
b028c47d2b
Correct Bank timestamp drift every slot ( #12737 )
...
* Move timestamp helper to sdk
* Add Bank method for getting timestamp estimate
* Return sysvar info from Bank::clock
* Add feature-gated timestamp correction
* Rename unix_timestamp method to be more descriptive
* Review comments
* Add timestamp metric
2020-10-09 21:53:41 +00:00
8f5431551e
Store program logs in blockstore / bigtable (TransactionWithStatusMeta) ( #12678 )
...
* introduce store program logs in blockstore / bigtable
* fix test, transaction logs created for successful transactions
* fix test for legacy bincode implementation around log_messages
* only api nodes should record logs
* truncate transaction logs to 100KB
* refactor log truncate for improved coverage
2020-10-08 12:06:15 -07:00
71a308affd
Use datapoint instead of print for is_full signal ( #12673 )
2020-10-07 13:36:54 -07:00
ce98088457
Track inserted repair shreds ( #12455 )
2020-09-29 14:13:21 -07:00
6601ec8f26
Record and store invoked instructions in transaction meta ( #12311 )
...
* Record invoked instructions and store in transaction meta
* Enable cpi recording if transaction sender is some
* Rename invoked to innerInstructions
2020-09-24 22:36:22 +08:00
731a943239
Remove transaction encoding from storage layer ( #12404 )
2020-09-24 13:10:29 +08:00
65a6bfad09
Add blockstore column to store performance sampling data ( #12251 )
...
* Add blockstore column to store performance sampling data
* introduce timer and write performance metrics to blockstore
* introduce getRecentPerformanceSamples rpc
* only run on rpc nodes enabled with transaction history
* add unit tests for get_recent_performance_samples
* remove RpcResponse from rpc call
* refactor to use Instant::now and elapsed for timer
* switch to root bank and ensure not negative subraction
* Add PerfSamples to purge/compaction
* refactor to use Instant::now and elapsed for timer
* switch to root bank and ensure not negative subraction
* remove duplicate constants
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-09-22 12:26:32 -07:00
05db41fe9c
Cache block time in Blockstore ( #11955 )
...
* Add blockstore column to cache block times
* Add method to cache block time
* Add service to cache block time
* Update rpc getBlockTime to use new method, and refactor blockstore slightly
* Return block_time with confirmed block, if available
* Add measure and warning to cache-block-time
2020-09-09 09:33:14 -06:00
1c1a3f979d
Detect and notify when deserializable shreds are available ( #11816 )
...
* Add logic to check for complete data ranges
* Add RPC signature notification
Co-authored-by: Carl <carl@solana.com>
2020-09-01 22:06:06 -07:00
247f27af37
Submit a vote timestamp every vote ( #10630 )
...
* Submit a timestamp for every vote
* Submit at most one vote timestamp per second
* Submit a timestamp for every new vote
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-08-21 11:10:51 -06:00
8841c3398c
Bump MacOS nofile recommendation message
2020-08-20 23:57:07 +00:00
b7ed06b17a
Cleanup test utilities ( #11723 )
...
* Add voting utility
* Add blockstore utility
Co-authored-by: Carl <carl@solana.com>
2020-08-20 05:04:38 +00:00
11951eb009
Sync FD limit and max maps to 500k
2020-08-19 16:36:03 +00:00
d15173ad9d
Address latest nightly clippy lints, but globally disable stable_sort_primitive
2020-08-17 22:36:10 -07:00
adc984a225
Rename Binary64 to Base64. Establish Base58 encoding
2020-08-17 17:26:29 -07:00
b5f3ced860
Add base64 (binary64) encoding for getConfirmedTransaction/getConfirmedBlock
2020-08-17 17:26:29 -07:00
6c5b8f324a
Rpc: Add until parameter for getConfirmedSignaturesForAddress2 ( #11644 )
...
* Refactor bigtable apis to accept start and end keys
* Make helper fn to deserialize cell data
* Refactor get_confirmed_signatures_for_address to use get_row_data range
* Add until param to get_confirmed_signatures_for_address
* Add until param to blockstore api
* Plumb until through client/cli
* Simplify client params
2020-08-15 10:42:17 -06:00
99fb36fe45
RPC: getConfirmedSignaturesForAddress2 only returns confirmed signatures ( #11615 )
...
* Add failing test case
* Limit to only rooted slots
2020-08-13 11:07:42 -06:00
de5fb3ba0e
Blockstore address signatures: handle slots that cross primary indexes, and refactor get_confirmed_signatures_for_address2 ( #11497 )
...
* Freeze address-signature index in the middle of slot to show failure case
* Secondary filter on signature
* Use AddressSignatures iterator instead of manually decrementing slots
* Remove unused method
* Add metrics
* Add transaction-status-index doccumentation
2020-08-10 10:27:38 -06:00
5530ee4c95
Add address-based lower bound to get_confirmed_signatures_for_address2 loop ( #11426 )
2020-08-06 22:29:25 +00:00
1061b50665
Fix blockstore empty panic ( #11423 )
...
* Add panicking test
* Add failing test: fresh transaction-status column shouldn't point at valid root 0
* Prevent transaction status match outside of primary-index bounds
* Initialize transaction-status and address-signature primer entries with Slot::MAX
* Revert "Add failing test: fresh transaction-status column shouldn't point at valid root 0"
This reverts commit cbad2a9fae22e5531e3b4ff1b0a9d6a223826c71.
* Revert "Initialize transaction-status and address-signature primer entries with Slot::MAX"
This reverts commit ffaeac0669d0cbe18dd68b5ce177e15a92360b72.
2020-08-06 22:21:46 +00:00
02c0981ecf
Rename startAfter to before
2020-08-05 14:33:14 -07:00
a11f137810
Rework get_confirmed_signatures_for_address2
2020-08-05 14:33:14 -07:00
1b2276520b
Add getConfirmedSignaturesForAddress2 RPC method
2020-08-05 14:33:14 -07:00
494968be66
UiTransactionEncoding is now copy
2020-08-05 14:27:12 -07:00
eefcf484cb
clippy
2020-08-03 18:35:15 +00:00
491f5ae61a
Add block time placeholder to getConfirmedBlock
2020-07-10 14:47:11 +00:00
3f6042d8b3
Add RepairWeight to track votes seen in gossip for weighted repair ( #10903 )
...
* Add RepairWeight
Co-authored-by: Carl <carl@solana.com>
2020-07-06 22:49:40 -07:00
58a475b789
Add db recovery methods ( #10838 )
2020-07-06 12:43:45 -07:00
f014c94369
Add breakdown of erasure blobs ( #10912 )
...
Co-authored-by: Carl <carl@solana.com>
2020-07-03 17:44:32 -07:00
f17ac70bb2
Add weighted traversal ( #10877 )
...
Co-authored-by: Carl <carl@solana.com>
2020-07-02 14:33:04 -07:00
2669ccb864
Update struct prefixes to Ui ( #10874 )
...
* Update comments
* Use Ui prefix
2020-07-01 20:06:40 +00:00
b89e506cbb
Consistently create temp dirs under ledger/farf ( #10848 )
2020-07-01 11:38:59 +09:00
583cec922b
Remove ledger purge batching ( #10830 )
2020-06-29 14:44:35 -07:00
1c498369b5
Remove fee-payer guesswork from Message and Transaction ( #10776 )
...
* Make Message::new_with_payer the default constructor
* Remove Transaction::new_[un]signed_instructions
These guess the fee-payer instead of stating it explicitly
2020-06-24 14:52:38 -06:00
77b8de193c
Add utility functions for testing ( #10749 )
...
* Add ancestor iterator
* Add blockstore generation from trees
Co-authored-by: Carl <carl@solana.com>
2020-06-23 12:05:00 -07:00
a33fef9af2
Add repair breakdown by slot and index ( #10717 )
...
* Slot full logging
* Repair stats logging
Co-authored-by: Carl <carl@solana.com>
2020-06-19 18:28:15 -07:00
0550b893b0
Fix typos ( #10675 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
6ee222363e
Move BankForks to solana_runtime ( #10637 )
...
* Move BankForks to solana_runtime
* Update imports
2020-06-17 15:27:03 +00:00
e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) ( #10445 )
...
* address warnings from 'rustup run beta cargo clippy --workspace'
minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs
expect some performance improvement AccountsDB::clean_accounts()
* address warnings from 'rustup run beta cargo clippy --workspace --tests'
* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'
* rustfmt
* fix warning stragglers
* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned
* Remove lock around JsonRpcRequestProcessor (#10417 )
automerge
* make ancestors parameter optional to avoid forcing construction of empty hash maps
Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
caa7f7a0c9
Support opening an in-use rocksdb as secondary ( #10209 )
...
automerge
2020-06-02 21:32:44 -07:00
eee9a08376
Purge TransactionStatus and AddressSignatures exactly from ledger-tool ( #10358 )
...
* Add failing test
* Add execution path to purge primary-index columns exactly
* Fail gracefully if older TransactionStatus rocksdb keys are present
* Remove columns_empty check for special columns
* Move blockstore purge methods to submodule
* Remove unused column empty check
2020-06-02 19:49:31 -06:00
19d11800bf
Remove WithSigner ( #10325 )
...
automerge
2020-05-29 23:17:44 -07:00
bc37dfc4c1
log leader ( #10316 )
...
automerge
2020-05-29 04:35:20 -07:00
fb4d8e1f62
cleanup clippy tests ( #10172 )
...
automerge
2020-05-29 00:26:06 -07:00
5ac2ae1178
Purge next slots to avoid a blockstore_processor panic on restart ( #10281 )
2020-05-27 19:55:40 -07:00
156387aba4
LedgerCleanupService no longer causes an OOM and actually purges ( #10199 )
...
* cleanup_ledger() now services new_root_receiver while purging
* purge_slots() now fully deletes before compacting
* Add ledger pruning grafana graph
2020-05-24 21:41:54 -07:00
97f2bcff69
master: Add nonce to shreds repairs, add shred data size to header ( #10109 )
...
* Add nonce to shreds/repairs
* Add data shred size to header
Co-authored-by: Carl <carl@solana.com>
2020-05-19 12:38:18 -07:00
7080fb9b37
Abort if the open fd limit cannot be increased ( #10064 )
...
automerge
2020-05-15 12:14:21 -07:00
58ef02f02b
9951 clippy errors in the test suite ( #10030 )
...
automerge
2020-05-15 09:35:43 -07:00
9575afc8fa
Refactor blockstore recovery code ( #10008 )
2020-05-13 10:09:38 -07:00
903a8a3196
Add retransmit packets_by_slot metrics ( #9975 )
2020-05-11 13:49:10 -07:00
ad3c8fb812
More logging around failure ( #9967 )
...
automerge
2020-05-10 16:01:20 -07:00
3442f36f8a
Repair alternate versions of dead slots ( #9805 )
...
Co-authored-by: Carl <carl@solana.com>
2020-05-05 14:07:21 -07:00
a91236012d
Pass around --max-genesis-archive-unpacked-size ( #9161 )
...
automerge
2020-04-29 18:53:34 -07:00
a12428a5b8
Use Blockstore lowest_slot to start root iterator ( #9738 )
2020-04-28 10:22:10 -06:00
fa20963b93
Revert shred fs ( #9712 )
...
* Revert "Untar is called for shred archives that do not exist. (#9565 )"
This reverts commit 729cb5eec6#9366 )"
This reverts commit 5ed39de8c5
2020-04-24 15:04:23 -07:00
76b1c2baf0
One less alloc per transaction ( #9705 )
...
* One less alloc per transaction
* Fix benches
* Fix compiler warnings in bench build
* Fix move build
* Fix bench
2020-04-24 13:03:46 -06:00
ecb343c23b
reduce errors ( #9669 )
...
Co-authored-by: Anatoly Yakovenko <anatoly@solana.com>
2020-04-22 19:46:06 -07:00
5d2f488004
Add getLowestNonpurgedBlock rpc; use blockstore api in getConfirmedBlocks ( #9656 )
...
automerge
2020-04-22 13:33:06 -07:00
729cb5eec6
Untar is called for shred archives that do not exist. ( #9565 )
...
automerge
2020-04-18 08:34:55 -07:00
5ed39de8c5
Dont insert shred payload into rocksdb ( #9366 )
...
automerge
2020-04-16 18:20:55 -07:00
530c542002
Rpc: Speed up getBlockTime ( #9510 )
...
* Add get-block-time metrics
* Add datapoints to blockstore rpc apis
* Tune timestamp_slot_range
* Refactor get_timestamp_slots
* Cargo.lock
2020-04-15 18:09:14 -06:00
91159ea8e3
Rpc: Add getConfirmedSignaturesForAddress ( #9407 )
...
automerge
2020-04-09 20:21:31 -07:00
d5a9ee97f2
Add --allow-dead-slots argument to `slot`/`print`/`json` commands ( #9408 )
...
automerge
2020-04-09 20:10:51 -07:00
f3afe5c99c
Remove dead code ( #9404 )
...
automerge
2020-04-09 13:09:59 -07:00
e1aa247548
Rpc: Add getConfirmedTransaction ( #9381 )
...
* Add blockstore method to return a complete transaction by signature
* Plumb getConfirmedTransaction rpc
* Add doc
2020-04-09 00:57:30 -06:00
f655b3f0fd
Fix lowest_cleanup_slot check in Blockstore ( #9382 )
...
automerge
2020-04-08 18:47:16 -07:00
36e73cada4
Add blockstore address-to-signature index ( #9367 )
...
automerge
2020-04-08 12:50:39 -07:00
acf64f8476
Update getSignatureStatuses to return historical statuses ( #9314 )
...
automerge
2020-04-06 03:04:54 -07:00
49e2cc6593
Rework TransactionStatus index in blockstore ( #9281 )
...
automerge
2020-04-04 20:24:06 -07:00
ad0997e15f
RPC: add `err` field to TransactionStatus, alongside the now deprecated `status` field ( #9296 )
...
automerge
2020-04-04 16:13:26 -07:00
b557b3170e
Add log before opening database
2020-04-03 14:55:06 -07:00
b7b4aa5d4d
move rpc types from client to client-types crate ( #9039 )
...
* Separate client types into own crate, so ledger does not need it
Removes about 50 crates of dependency from ledger
* Drop Rpc name from transaction-status types
2020-03-26 13:29:30 -07:00
ed036b978d
Accumulate blockstore metrics and submit every 2s ( #9075 )
2020-03-26 12:51:41 -07:00
b8fda9d730
Log how much data the ledger holds before processing it ( #9079 )
2020-03-25 21:41:50 -07:00
70c167182a
ledger tool now outputs transaction status information if available ( #9024 )
...
automerge
2020-03-23 12:49:21 -07:00
1b8f9e75dd
Update getSignatureStatus: support multiple signatures, include slot in each response item ( #9022 )
...
* Rename enable-rpc-get-confirmed-block
* Rename RpcTransactionStatus -> RpcTransactionStatusMeta
* Return simplified RpcTransactionStatus; Add support for multiple transactions
* Update docs
* typo
2020-03-23 11:25:39 -06:00
4d2b83d01f
Add option to disable rocks compaction ( #9011 )
2020-03-23 08:42:32 -07:00
dc1db33ec9
Add Capabilities to Signal BroadcastStage to Retransmit ( #8899 )
2020-03-19 23:35:01 -07:00
b7b36bb0a4
Upgrade to Rust 1.42 ( #8836 )
...
* Upgrade to Rust 1.42
* deref
* parens
Co-authored-by: Trent Nelson <trent@solana.com>
2020-03-13 14:15:22 -06:00
f47a789b15
Add find_incomplete_slots ( #8654 )
...
* Add find_incomplete_slots
* Add live slots iterator
2020-03-05 10:58:00 -08:00
dec3da8f9d
Add orphan iterator ( #8636 )
2020-03-04 18:10:30 -08:00
73063544bd
Move shred_version module to sdk/
2020-02-24 14:46:12 -07:00
ab361a8073
Rename KeypairUtil to Signer ( #8360 )
...
automerge
2020-02-20 13:28:55 -08:00
ab475e4849
get_confirmed_block: expect() less
2020-02-13 10:56:34 -07:00
72b11081a4
Report validator rewards in getConfirmedBlock JSON RPC
2020-02-11 17:25:45 -07:00
8436457e75
Rename program_utils.rs ( #8127 )
2020-02-05 12:48:30 -08:00
ed5101b031
Generate max coding shreds when necessary ( #8099 )
...
* Generate max coding shreds when necessary
* test
2020-02-04 15:45:01 -08:00
4ffd7693d6
Add lock to make sure slot-based locktree calls are safe ( #7993 )
2020-01-28 13:45:41 -08:00
37d1daf58e
Revert "Generate MAX_DATA_SHREDS_PER_FEC_BLOCK coding shreds for each FEC block ( #7474 )" ( #7898 )
...
automerge
2020-01-21 11:48:09 -08:00
2dd8ab197d
Remove redundant threadpools in sigverify ( #7888 )
...
* Limit the number of thread pools sigverify creates
* Name local threadpools
2020-01-20 20:08:19 -08:00
21d5fe6272
Fix timestamp overflow ( #7886 )
...
* Split timestamp calculation into separate fn for math unit testing
* Add failing test
* Fix failing test; also bump stakes to near expected cluster max supply
* Don't error on timestamp of slot 0
2020-01-20 17:54:44 -07:00
0526d4ff21
Add logging surrounding failure in `get_slot_entries_with_shred_info()` ( #7846 )
...
* Add logging surrounding failure
2020-01-16 16:03:24 -08:00
76e20015a4
Add separate thread to check for and store duplicate slot proofs ( #7834 )
2020-01-16 15:27:54 -08:00
6d3b8b6d7d
Remove tuples from JSON RPC responses ( #7806 )
...
* Remove RpcConfirmedBlock tuple
* Remove getRecentBlockhash tuple
* Remove getProgramAccounts tuple
* Remove tuple from get_signature_confirmation_status
* Collect Rpc response types
* Camel-case epoch schedule for rpc response
* Remove getBlockCommitment tuple
* Remove getStorageTurn tuple
* Update json-rpc docs
2020-01-15 00:25:45 -07:00
ff1ca1e0d3
Consolidate entry tick verification into one function ( #7740 )
...
* Consolidate entry tick verification into one function
* Mark bad slots as dead in blocktree processor
* more feedback
* Add bank.is_complete
* feedback
2020-01-15 09:15:26 +08:00
721c4378c1
Plumb ability to handle duplicate shreds into shred insertion functions ( #7784 )
2020-01-14 15:37:53 -08:00
f91ffbbfdf
Add support in BlockStore for tracking duplicate slots ( #7761 )
...
* Add test
* Add new column family to track duplicate slots
* Fix clippy errors
* Introduce new SlotColumn for common implementation of Column trait
2020-01-13 17:21:39 -08:00
b5dba77056
Rename blocktree to blockstore ( #7757 )
...
automerge
2020-01-13 13:13:52 -08:00
Tyera Eulberg
Michael Vines
Ryo Onodera
Trent Nelson
Josh
sakridge
behzad nouri
carllin
Tyera Eulberg
Justin Starry
Greg Fitzgerald
Kristofer Peterson
Ryo Onodera
anatoly yakovenko
Jack May
Pankaj Garg
Sagar Dhawan