Commit Graph

124 Commits

Author SHA1 Message Date
Justin Starry 918d04e3f0
Add more slot update notifications (#15734)
* Add more slot update notifications

* fix merge

* Address feedback and add integration test

* switch to datapoint

* remove unused shred method

* fix clippy

* new thread for rpc completed slots

* remove extra constant

* fixes

* rely on channel closing

* fix check
2021-03-12 21:44:06 +08:00
Tyera Eulberg 7e65289729
Convert blockstore TransactionStatus column family to protobufs (#15733)
* Prevent panic if TransactionStatus can't be deserialized

* Convert Blockstore TransactionStatus column to protobuf

* Add compatability test
2021-03-05 09:05:35 -07:00
Michael Vines e92cff3efa create-snapshot subcommad now accepts the ROOT keyword 2021-02-26 16:40:10 -08:00
Michael Vines 5df36aec7d Pacify clippy 2021-02-19 20:08:41 -08:00
Tyera Eulberg 170cb792eb
Return blockstore error if previous_blockhash cannot be determined (#15382)
* Return blockstore error if previous_blockhash cannot be determined

* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
Tyera Eulberg 0812931c38
Log if unsanitary transactions are read from blockstore (#15319) 2021-02-14 06:32:43 +00:00
Tyera Eulberg da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks (#15204)
* Remove timestamp_correction feature gating

* Remove timestamp_bounding feature gating

* Remove unused deprecated ledger code

* Remove unused deprecated unbounded-timestamp code

* Enable independent adjustment of fast/slow timestamp bounding

* Update timestamp bounds to 25% fast, 80% slow; warp timestamp

* Update bank hash test

* Add PR number to feature

Co-authored-by: Michael Vines <mvines@gmail.com>

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
Michael Vines 42bf6dc2ab Revert hard nofile limit back to 500000 2021-02-03 19:43:55 +00:00
Ryo Onodera 8376781ec8
Don't squash caught errors, please (#15046)
* Don't squash caught errors, please

* Update blockstore.rs

* Update blockstore.rs
2021-02-03 14:42:34 +00:00
Trent Nelson 894b412aef docs: bump nofiles recommendations to match maps 2021-02-02 21:59:21 -07:00
Josh 1de6d28eaf
Add block_time to getConfirmedSignaturesForAddress2 and getConfirmedTransaction (#14572)
* add block_time to get_confirmed_signatures_for_address2 and protobuf implementation for tx_by_addr

* add tests for convert

* update cargo lock

* run cargo format after rebase

* introduce legacy TransactionByAddrInfo

* move LegacyTransactionByAddrInfo back to storage-bigtable
2021-01-20 22:10:35 -08:00
Michael Vines e1f2d384db Don't require increased open file limit at ledger creation
Follow-up to 0b92720fdb, `create_new_ledger()` does not require a higher fd limit
2020-12-17 08:46:13 -08:00
Michael Vines 0b92720fdb Don't require increased open file limit in solana-test-validator
Travis CI in particular does not allow the open file limit to be
increased.
2020-12-16 20:43:08 -08:00
Tyera Eulberg ac0d32bc7e
Add blockstore skipped api (#14145)
* Add blockstore api to determine if a slot was skipped

* Return custom rpc error if slot is skipped
2020-12-16 13:40:36 -07:00
Michael Vines 7143aaa89b Clippy 2020-12-14 08:03:29 -08:00
Josh 13db3eca9f
SPL token balance in transaction metadata (#13673)
* feat: store pre / post token balances

* move helper functions into separate include

* move token balance functionality to transaction-status crate

* fix blockstore processor test

* fix bigtable legacy test

* add caching to decimals
2020-12-10 19:25:07 -08:00
sakridge aa2751e614
Check shred type in is_duplicate (#14050) 2020-12-10 18:20:08 -08:00
sakridge c5fe076432
Better dupe detection (#13992) 2020-12-09 23:14:31 -08:00
behzad nouri e1793e5a13
caches vote-state de-serialized from vote accounts (#13795)
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.

This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.

Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.
2020-11-30 17:18:33 +00:00
carllin afc1b59475
Fix assertion failure (#13626)
Co-authored-by: Carl Lin <carl@solana.com>
2020-11-16 21:30:38 -08:00
Tyera Eulberg 80db6c0980 Add bounded timestamp-estimation method 2020-10-31 21:30:42 -07:00
Tyera Eulberg 39686ef098
Use bank timestamp to populate Blockstore::blocktime_cf when correction active (#13158) 2020-10-26 19:23:45 +00:00
Michael Vines 7bc073defe Run `codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand` 2020-10-21 19:08:13 -07:00
Tyera Eulberg 359707c85e
Convert Blockstore Rewards cf to protobuf (#12860)
* Add Blockstore protobuf cf type

* Add Rewards message to proto and make generated pub

* Convert Rewards cf to ProtobufColumn

* Add bench

* Adjust tags

* Move solana proto definitions and conversion methods to new crate
2020-10-15 18:04:10 -06:00
Tyera Eulberg b028c47d2b
Correct Bank timestamp drift every slot (#12737)
* Move timestamp helper to sdk

* Add Bank method for getting timestamp estimate

* Return sysvar info from Bank::clock

* Add feature-gated timestamp correction

* Rename unix_timestamp method to be more descriptive

* Review comments

* Add timestamp metric
2020-10-09 21:53:41 +00:00
Josh 8f5431551e
Store program logs in blockstore / bigtable (TransactionWithStatusMeta) (#12678)
* introduce store program logs in blockstore / bigtable

* fix test, transaction logs created for successful transactions

* fix test for legacy bincode implementation around log_messages

* only api nodes should record logs

* truncate transaction logs to 100KB

* refactor log truncate for improved coverage
2020-10-08 12:06:15 -07:00
sakridge 71a308affd
Use datapoint instead of print for is_full signal (#12673) 2020-10-07 13:36:54 -07:00
sakridge ce98088457
Track inserted repair shreds (#12455) 2020-09-29 14:13:21 -07:00
Justin Starry 6601ec8f26
Record and store invoked instructions in transaction meta (#12311)
* Record invoked instructions and store in transaction meta

* Enable cpi recording if transaction sender is some

* Rename invoked to innerInstructions
2020-09-24 22:36:22 +08:00
Justin Starry 731a943239
Remove transaction encoding from storage layer (#12404) 2020-09-24 13:10:29 +08:00
Josh 65a6bfad09
Add blockstore column to store performance sampling data (#12251)
* Add blockstore column to store performance sampling data

* introduce timer and write performance metrics to blockstore

* introduce getRecentPerformanceSamples rpc

* only run on rpc nodes enabled with transaction history

* add unit tests for get_recent_performance_samples

* remove RpcResponse from rpc call

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* Add PerfSamples to purge/compaction

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* remove duplicate constants

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-09-22 12:26:32 -07:00
Tyera Eulberg 05db41fe9c
Cache block time in Blockstore (#11955)
* Add blockstore column to cache block times

* Add method to cache block time

* Add service to cache block time

* Update rpc getBlockTime to use new method, and refactor blockstore slightly

* Return block_time with confirmed block, if available

* Add measure and warning to cache-block-time
2020-09-09 09:33:14 -06:00
carllin 1c1a3f979d
Detect and notify when deserializable shreds are available (#11816)
* Add logic to check for complete data ranges

* Add RPC signature notification

Co-authored-by: Carl <carl@solana.com>
2020-09-01 22:06:06 -07:00
Michael Vines 247f27af37
Submit a vote timestamp every vote (#10630)
* Submit a timestamp for every vote

* Submit at most one vote timestamp per second

* Submit a timestamp for every new vote

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-08-21 11:10:51 -06:00
Trent Nelson 8841c3398c Bump MacOS nofile recommendation message 2020-08-20 23:57:07 +00:00
carllin b7ed06b17a
Cleanup test utilities (#11723)
* Add voting utility

* Add blockstore utility

Co-authored-by: Carl <carl@solana.com>
2020-08-20 05:04:38 +00:00
Trent Nelson 11951eb009 Sync FD limit and max maps to 500k 2020-08-19 16:36:03 +00:00
Michael Vines d15173ad9d Address latest nightly clippy lints, but globally disable stable_sort_primitive 2020-08-17 22:36:10 -07:00
Michael Vines adc984a225 Rename Binary64 to Base64. Establish Base58 encoding 2020-08-17 17:26:29 -07:00
Michael Vines b5f3ced860 Add base64 (binary64) encoding for getConfirmedTransaction/getConfirmedBlock 2020-08-17 17:26:29 -07:00
Tyera Eulberg 6c5b8f324a
Rpc: Add until parameter for getConfirmedSignaturesForAddress2 (#11644)
* Refactor bigtable apis to accept start and end keys

* Make helper fn to deserialize cell data

* Refactor get_confirmed_signatures_for_address to use get_row_data range

* Add until param to get_confirmed_signatures_for_address

* Add until param to blockstore api

* Plumb until through client/cli

* Simplify client params
2020-08-15 10:42:17 -06:00
Tyera Eulberg 99fb36fe45
RPC: getConfirmedSignaturesForAddress2 only returns confirmed signatures (#11615)
* Add failing test case

* Limit to only rooted slots
2020-08-13 11:07:42 -06:00
Tyera Eulberg de5fb3ba0e
Blockstore address signatures: handle slots that cross primary indexes, and refactor get_confirmed_signatures_for_address2 (#11497)
* Freeze address-signature index in the middle of slot to show failure case

* Secondary filter on signature

* Use AddressSignatures iterator instead of manually decrementing slots

* Remove unused method

* Add metrics

* Add transaction-status-index doccumentation
2020-08-10 10:27:38 -06:00
Tyera Eulberg 5530ee4c95
Add address-based lower bound to get_confirmed_signatures_for_address2 loop (#11426) 2020-08-06 22:29:25 +00:00
Tyera Eulberg 1061b50665
Fix blockstore empty panic (#11423)
* Add panicking test

* Add failing test: fresh transaction-status column shouldn't point at valid root 0

* Prevent transaction status match outside of primary-index bounds

* Initialize transaction-status and address-signature primer entries with Slot::MAX

* Revert "Add failing test: fresh transaction-status column shouldn't point at valid root 0"

This reverts commit cbad2a9fae22e5531e3b4ff1b0a9d6a223826c71.

* Revert "Initialize transaction-status and address-signature primer entries with Slot::MAX"

This reverts commit ffaeac0669d0cbe18dd68b5ce177e15a92360b72.
2020-08-06 22:21:46 +00:00
Michael Vines 02c0981ecf Rename startAfter to before 2020-08-05 14:33:14 -07:00
Michael Vines a11f137810 Rework get_confirmed_signatures_for_address2 2020-08-05 14:33:14 -07:00
Michael Vines 1b2276520b Add getConfirmedSignaturesForAddress2 RPC method 2020-08-05 14:33:14 -07:00
Michael Vines 494968be66 UiTransactionEncoding is now copy 2020-08-05 14:27:12 -07:00
Michael Vines eefcf484cb clippy 2020-08-03 18:35:15 +00:00