Commit Graph

262 Commits

Author SHA1 Message Date
sakridge 1caeea8bc2
Refactor new bank paths into common function (#8454) 2020-02-25 15:49:59 -08:00
Pankaj Garg aa80f69171
Promote some datapoints to `info` to fix dashboard (#8381)
automerge
2020-02-21 13:41:49 -08:00
Tyera Eulberg ab361a8073
Rename KeypairUtil to Signer (#8360)
automerge
2020-02-20 13:28:55 -08:00
anatoly yakovenko ccad5d5aaf
change warnings to infos (#8322) 2020-02-19 14:25:49 -08:00
carllin 73a278dc64
Factor out creating genesis with vote accounts into a utility function (#8315)
automerge
2020-02-18 02:39:47 -08:00
anatoly yakovenko 17fb8258e5
Datapoints overwhelm the metrics queue and blow up ram usage. (#8272)
automerge
2020-02-14 11:11:55 -08:00
Michael Vines c4fd81fc1c The getConfirmedBlock RPC API is now disabled by default
The --enable-rpc-get-confirmed-block flag allows validators to opt-in to
the higher disk usage and IOPS.
2020-02-11 22:24:08 -07:00
Michael Vines 72b11081a4 Report validator rewards in getConfirmedBlock JSON RPC 2020-02-11 17:25:45 -07:00
carllin 0c8cee8c4a
Refactor select_fork() to avoid clones and for clarity (#8081)
* Refactor select_fork() to avoid clones and for clarity

* Add test that fork weights are increasing
2020-02-03 16:48:24 -08:00
carllin 4197cce8c9
Tower tests (#7974)
* Add testing framework for voting
2020-01-28 16:02:28 -08:00
carllin 4ffd7693d6
Add lock to make sure slot-based locktree calls are safe (#7993) 2020-01-28 13:45:41 -08:00
Sunny Gleason 5cf090c896
feat: implement RPC notification queue (#7863) 2020-01-20 16:08:29 -05:00
Tyera Eulberg 6d3b8b6d7d
Remove tuples from JSON RPC responses (#7806)
* Remove RpcConfirmedBlock tuple

* Remove getRecentBlockhash tuple

* Remove getProgramAccounts tuple

* Remove tuple from get_signature_confirmation_status

* Collect Rpc response types

* Camel-case epoch schedule for rpc response

* Remove getBlockCommitment tuple

* Remove getStorageTurn tuple

* Update json-rpc docs
2020-01-15 00:25:45 -07:00
Justin Starry ff1ca1e0d3
Consolidate entry tick verification into one function (#7740)
* Consolidate entry tick verification into one function

* Mark bad slots as dead in blocktree processor

* more feedback

* Add bank.is_complete

* feedback
2020-01-15 09:15:26 +08:00
Greg Fitzgerald b5dba77056 Rename blocktree to blockstore (#7757)
automerge
2020-01-13 13:13:52 -08:00
Tyera Eulberg a17d5795fb getConfirmedBlock: add encoding optional parameter (#7756)
automerge
2020-01-12 21:34:30 -08:00
sakridge 73c93cc345
Print bank hash and hash inputs. (#7733) 2020-01-09 16:33:10 -08:00
Michael Vines 4fe0b116ae Measure heap usage while processing the ledger 2020-01-03 13:25:37 -07:00
Michael Vines a0fb9de515 Move thread_mem_usage module into measure/ 2020-01-03 13:25:37 -07:00
Pankaj Garg 87b2525e03
Limit maximum number of shreds in a slot to 32K (#7584)
* Limit maximum number of shreds in a slot to 32K

* mark dead slot replay as fatal error
2019-12-30 07:42:09 -08:00
Parth 727be309b2 fix entryverification state (#7169)
automerge
2019-12-23 23:26:27 -08:00
Sagar Dhawan 6a9005645a
Update "limit-ledger-size" to use DeleteRange for much faster deletes (#7515)
* Update "limit-ledger-size" to use DeleteRange for much faster deletes

* Update core/src/ledger_cleanup_service.rs

Co-Authored-By: Michael Vines <mvines@gmail.com>

* Rewrite more idiomatically

* Move max_ledger_slots to a fn for clippy

* Remove unused import

* Detect when all columns have been purged and fix a bug in deletion

* Check that more than 1 column is actually deleted

* Add helper to test that ledger meets minimum slot bounds

* Remove manual batching of deletes

* Refactor to keep some N slots older than the highest root

* Define MAX_LEDGER_SLOTS that ledger_cleanup_service will try to keep around

* Refactor compact range
2019-12-18 11:50:09 -08:00
sakridge 98b80288ed
Optimize bank_forks critical section (#7477) 2019-12-13 17:20:31 -08:00
sakridge dd54fff978
Use pinned memory for entry verify (#7440) 2019-12-12 10:36:27 -08:00
Parth 6d2861f358
add unit test for minority fork overcommit attack (#7292)
* add unit test for minority fork overcommit attack

* add generic function to simulate fork selection
2019-12-10 22:06:16 +05:30
TristanDebrunner 9ecb844de7 Split up ReplayStageConfig to make it derive Default (#7334)
automerge
2019-12-06 14:39:35 -08:00
Tyera Eulberg 3ab8185777
Add intermittent Timestamping to Votes (#7233)
* Add intermittent timestamp to Vote

* Add timestamp to VoteState, add timestamp processing to program

* Print recent timestamp with solana show-vote-account

* Add offset of 1 to timestamp Vote interval to initialize at node boot (slot 1)

* Review comments

* Cache last_timestamp in Tower and use for interval check

* Move work into Tower method

* Clarify timestamping interval

* Replace tuple with struct
2019-12-06 14:38:49 -07:00
TristanDebrunner fae9c08815
Add ReplayStageConfig (#7195) 2019-12-04 11:17:17 -07:00
carllin f0a40862d6
Support local cluster edge case testing (#7135)
* Refactor local cluster to support killing a partition

* Rework run_network_partition

* Introduce fixed leader schedule

* Plumb fixed schedule into test
2019-12-03 16:31:59 -08:00
Sunny Gleason 0a0f15baca
RPC subscriptions for new slot notifications (#7114)
* feat: slot notifications via pubsub rpc w/ tests
2019-11-26 03:42:54 -05:00
Tyera Eulberg 0f66e5e49b Add getConfirmedBlock test to rpc (#7120)
automerge
2019-11-25 11:08:03 -08:00
Michael Vines 0bd41f98ed Avoid jemalloc in windows build (#7089)
automerge
2019-11-21 18:39:29 -08:00
anatoly yakovenko d9e7a5fcbe
Use fork weight instead of individual bank weight for fork selection. (#7079)
* Fix weight calculation

* Fix tests

* fork weight

* wait until nodes are in the leader schedule

* enable sanity

* fewer long tests
2019-11-21 15:47:08 -08:00
Tyera Eulberg c965a110f2
Use unbounded channel (#7081) 2019-11-21 14:23:40 -07:00
sakridge e66b29943b
datapoint for best fork weight and slot in replay (#7066) 2019-11-20 17:26:52 -08:00
Tyera Eulberg 97ca6858b7
Write transaction status and fee into persistent store (#7030)
* Pass blocktree into execute_batch, if persist_transaction_status

* Add validator arg to enable persistent transaction status store

* Pass blocktree into banking_stage, if persist_transaction_status

* Add validator params to bash scripts

* Expose actual transaction statuses outside Bank; add tests

* Fix benches

* Offload transaction status writes to a separate thread

* Enable persistent transaction status along with rpc service

* nudge

* Review comments
2019-11-20 16:43:10 -07:00
anatoly yakovenko b150da837a Use epoch as the gossip purge timeout for staked nodes. (#7005)
automerge
2019-11-20 11:25:18 -08:00
Jack May d184d3a732
Merge native programs parts into one unit (#7047) 2019-11-20 10:12:43 -08:00
Pankaj Garg d32a072190
Use ticks_per_slot to calculate maximum grace ticks (#7024)
* Use ticks_per_slot to calculate maximum grace ticks

* fix test

* fix votable candidate ordering

* fixes to pick_best_fork() and a unit test

* fixes
2019-11-19 17:55:42 -08:00
Dan Albert b958bf9086
Fix confirmation metrics (#7035) 2019-11-19 09:51:50 -05:00
carllin 43144cfe8b
Make banks that fail threshhold check resettable (#7027) 2019-11-19 02:36:30 -08:00
carllin 11d2d2eccd
Fix progress map losing banks and recomputing stats (#7026)
* Fix progress map missing banks

* Fix confirmations

* Fix test

* Initialiize progress with frozen banks atartup
2019-11-19 02:36:00 -08:00
Sagar Dhawan 6bfe0fca1f
Add a version field to shreds (#7023)
* Add a version field to shreds

* Clippy

* Fix Chacha Golden

* Fix shredder bench compile

* Fix blocktree bench compile
2019-11-18 18:05:02 -08:00
carllin b66c03667c
Log for threshold failure (#7008) 2019-11-17 17:10:16 -08:00
carllin 70322d1ff8
Add error logging to dead slots (#7000) 2019-11-16 02:54:51 -08:00
anatoly yakovenko 59413b3124 Fix rules for fork selection (#6906)
automerge
2019-11-15 08:36:33 -08:00
Sagar Dhawan 79d7090867
Remove obsolete references to Blob (#6957)
* Remove the name "blob" from archivers

* Remove the name "blob" from broadcast

* Remove the name "blob" from Cluset Info

* Remove the name "blob" from Repair

* Remove the name "blob" from a bunch more places

* Remove the name "blob" from tests and book
2019-11-14 11:49:31 -08:00
carllin 43e2301e2c
Fix roots overrunning broadcast (#6884)
* Add trusted pathway for insert_shreds to avoid checks
2019-11-14 00:32:07 -08:00
Greg Fitzgerald a3a830e1ab
Delete Service trait (#6921) 2019-11-13 11:12:09 -07:00
Greg Fitzgerald 30a08f4282 Cleanup ledger macros (#6916)
automerge
2019-11-13 07:14:09 -08:00
Justin Starry 9807f47d4e
Rename genesis block to genesis config (#6816) 2019-11-08 23:56:57 -05:00
Justin Starry 02058ea699
Reject blocks with invalid last ticks in replay stage (#6833)
* Reject blocks with invalid last ticks in replay stage

* slot_full
2019-11-08 20:21:54 -05:00
Michael Vines b800642fa4
Add new fork log message for when the node is leader for consistency (#6808) 2019-11-08 12:30:25 -07:00
Pankaj Garg 0ace79939b
Add reference tick to data shreds (#6772)
* Add reference tick to data shreds

* fix tests
2019-11-06 13:27:58 -08:00
carllin 24102a7435
Allow voting on empty banks (#6719)
* Allow votes on empty banks

* Remove making first bank is_delta true, no longer necessary for idling

* Remove votable from ledger tool
2019-11-06 01:02:26 -08:00
Michael Vines 09e648f957
ledger-tool/: Include full validator voting history in fork-graph (#6756) 2019-11-05 19:40:00 -07:00
anatoly yakovenko f54cfcdb8f
Store and persists full stack of tower votes in gossip (#6695)
* vote array

wip

wip

wip

update

gossip index should match tower index

tests build

clippy

test index after expired vote

test

bank specific last vote sync time

* verify

* we are likely to see many more warnings about old votes now
2019-11-04 16:19:54 -08:00
sakridge 57983980a7 Lower verify-batch-size to debug (#6722)
automerge
2019-11-04 16:00:59 -08:00
Tyera Eulberg 33f4aaf3fd
Rename confidence to commitment (#6714) 2019-11-04 16:44:27 -07:00
Michael Vines 50a17fc00b Use Slot and Epoch type aliases instead of raw u64 (#6693)
automerge
2019-11-02 00:38:30 -07:00
Justin Starry e8e5ddc55d
Verify number of hashes for each block of entries (#6262)
* Verify number of hashes for each block of entries

* Fix blocktree processor tick check

* Rebase once more
2019-10-31 16:38:50 -04:00
sakridge 38491c8c4b
Reduce verify-batch-size log (#6623) 2019-10-30 13:41:11 -07:00
Rob Walker b4119c454a
credit_only credits forwarding (#6509)
* credit_only_credits_forwarding

* whack transfer_now()

* fixup

* bench should retry the airdrop TX

* fixup

* try to make bench-exchange a bit more robust, informative
2019-10-23 22:01:22 -07:00
sakridge 03d29a8311
Async poh verify (#6353)
* Async poh verify

* Up ticks_per_s to 160

GPU poh verify needs shorter poh sequences or it takes forever to
verify. Keep slot time the same at 400ms.

* Fix stats

* Don't halt on ticks

* Increase retries for local_cluster tests and make repairman test serial
2019-10-23 12:11:04 -07:00
carllin b38bf90de7
Deshred blocks in parallel (#6461)
* Deshred in parallel

* Add tests for corrupt slots and parallel deshred

* Rename load_blocktree_entries to load_blocktree_entries_with_shred_count
2019-10-21 16:15:10 -07:00
Pankaj Garg 84e911361a
Use constants instead of lazy_static for shred header sizes (#6472) 2019-10-21 12:46:16 -07:00
Greg Fitzgerald 2636418659
Move blocktree_processor to solana_ledger (#6460)
* Drop core::result dependency in bank_forks

* Move blocktree_processor into solana_ledger
2019-10-20 09:54:38 -06:00
Rob Walker e2c316d2d0
system_instruction_processor updates (#6448)
* zero lamport account creation

* whack create_user_account, take 2

* target->to

* ..

* ..

* update chacha golden

* update chacha golden

* ..

* ..
2019-10-19 18:23:27 -07:00
Pankaj Garg badeb4d31a
Rework shred headers to fix position of signature (#6451)
* Rework shred headers to fix position of signature

* fix clippy
2019-10-18 22:55:59 -07:00
Greg Fitzgerald 5468be2ef9 Add solana-ledger crate (#6415)
automerge
2019-10-18 09:28:51 -07:00
carllin 27c0d30a07
Fix logging (#6417) 2019-10-18 02:06:41 -07:00
Greg Fitzgerald d3068c3918
Remove circular dependencies in core (#6408)
* Remove core::result dependency from blocktree

* Remove core::result dependency from shred

* Move Packet from core::packet to sdk::packet

This way we don't need to split perf_libs yet.

* Disable packet when compiling BPF programs
2019-10-17 11:37:30 -06:00
Pankaj Garg 83c1831a01
Fix replay stage test (#6406) 2019-10-16 15:41:43 -07:00
carllin ccb4e32ee0
ReplayStage metrics (#6358)
* ReplayStage metrics

* Add more metrics

* Refactor get_slot_entries_with_shred_count() to detect wasted work

* Update dashboard

* Update broadcast slots to micros

* Add broadcast dashboard
2019-10-16 14:32:18 -07:00
Justin Starry 7e6e7e8406
Remove special handling of first ledger tick (#6263)
* Remove special handling of first ledger tick

* Fix subtraction overflow

* @garious feedback

* Back to height

* More tick_height name changes

* Fix off-by-one

* Fix leader tick error

* Fix merge conflict

* Fix recently added test
2019-10-16 15:53:11 -04:00
sakridge dfca2b510b
Lower shred/receiver stats (#6365)
too many messages
2019-10-15 11:43:52 -07:00
Rob Walker 7cf90766a3
add epoch_schedule sysvar (#6256)
* add epoch_schedule sysvar

* book sheesh!
2019-10-08 22:34:26 -07:00
Justin Starry 5e31565574
Expand blocktree processor options (#6248)
* Refactor blocktree processor args and support full leader cache

* Add entry callback option

* Rename num_threads to override_num_threads

* Add test for entry callback

* Refactor cached leader schedule changes

* Add tests for blocktree process options

* Refactor test

* @mvines feedback
2019-10-08 17:58:49 -04:00
sakridge 5ef012b2c1
Tweak debug to remove unreadable datapoints (#6060) 2019-10-04 16:25:22 -07:00
Michael Vines f9f5bc2eb5
More clippy 2019-10-02 21:21:07 -07:00
Sagar Dhawan 84f74807d4 Skip considering banks older than the latest vote slot (#6037)
automerge
2019-09-23 19:40:03 -07:00
Sagar Dhawan 62c22c6cb1
Fix really old banks triggering log spam (#6025) 2019-09-23 13:59:16 -07:00
carllin 8240d1fe0a
Confidence implementation (#5993)
* Change confidence parameters

* Add status_cache_ancestors to get all relevant ancestors of a bank including roots from status cache

* Fix and add tests

* Clippy
2019-09-20 19:38:56 -07:00
Pankaj Garg ca9d4e34df
Broadcast stage tuning (#5989) 2019-09-19 16:29:52 -07:00
Pankaj Garg 783e8672e7
Removed Shred enum (#5963)
* Remove shred enum and it's references

* rename ShredInfo to Shred

* clippy
2019-09-18 16:24:30 -07:00
carllin 2f50d0e145
Refactor confidence from replay stage (#5938) 2019-09-17 19:43:40 -07:00
Pankaj Garg ff608992ee
Replace Shred usage with ShredInfo (#5939)
* Replace Shred usage with ShredInfo

* Fix tests

* fix clippy
2019-09-17 18:22:46 -07:00
Michael Vines 84c8a5bbec
Add replay-stage-mark_dead_slot datapoint (#5907) 2019-09-14 08:50:53 -07:00
Pankaj Garg 3d3b03a123
Verify signature of recovered shred before adding them to blocktree (#5811)
* Verify signature of recovered shred before adding them to blocktree

* fix failing tests, and review comments
2019-09-05 18:20:30 -07:00
carllin bd74e63702
Offload remaining confidence cache computation to separate thread (#5792)
* Move remaining confidence cache computation to separate thread

* Move confidence cache out of bank forks
2019-09-04 23:10:25 -07:00
Pankaj Garg 02ee2a601c
Further cleanup of blocktree after Blob deprecation (#5780) 2019-09-04 12:47:09 -07:00
carllin dfa6238342
Remove unnecessary construction of descendants (#5742) 2019-09-04 01:49:42 -07:00
Pankaj Garg 3b0d48e3b8
Remove blocktree blob references (#5691)
* Remove blocktree blob references

* fixes and cleanup

* replace uninitialized() call with MaybeUninit

* fix bench
2019-09-03 21:32:51 -07:00
anatoly yakovenko 475f6fe666
votes only need slots and the last bank hash (#5499)
churn

cleanup

reverse test slot hashes

test check_slots_are_valid

updates

only send the minimum bank vote difference

fixup! only send the minimum bank vote difference

some banks may not have a voting account setup

fixup! votes only need slots and the last bank hash

fixup! fixup! votes only need slots and the last bank hash

fmt

fixed compare

fixed vote

fixup! fixed vote

poke ci

filter the local votes via the last bank vote
2019-09-02 12:01:09 -07:00
Parth 7dfb735db9
randomize tx ordering (#4978)
Summary of Changes:
This change adds functionality to randomize tx execution for every entry. It does this by implementing OrderedIterator that iterates tx slice as per the order specified. The order is generated randomly for every entry.
2019-08-28 21:08:32 +05:30
carllin 8b9c3a2561
Blocktree last_root to enforce a slot floor (#5593)
* Add last_root to blocktree

* Don't repair earlier than last_root

* Add integration test to make sure blocktree floor is enforced
2019-08-27 15:09:41 -07:00
Pankaj Garg 4ac1213c9c
Integrate coding shreds and recovery (#5625)
* Integrate coding shreds and recovery

* More tests for shreds and some fixes

* address review comments

* fixes to code shred generation

* unignore tests

* fixes to recovery
2019-08-26 18:27:45 -07:00
Michael Vines faafee6b42
to to/the the (#5590) 2019-08-21 17:46:59 -07:00
Pankaj Garg 4798e7fa73
Integrate data shreds (#5541)
* Insert data shreds in blocktree and database

* Integrate data shreds with rest of the code base

* address review comments, and some clippy fixes

* Fixes to some tests

* more test fixes

* ignore some local cluster tests

* ignore replicator local cluster tests
2019-08-20 17:16:06 -07:00
Michael Vines ccee6241a6
Revert "publish fixes"
This reverts commit 4d13d3871d.
2019-08-16 17:28:07 -07:00