Commit Graph

132 Commits

Author SHA1 Message Date
Justin Starry bf5bce50a4
Fix stake pruning test (#5124) 2019-07-16 13:20:03 -04:00
Pankaj Garg 1c966aac25 Facility to generate a blocktree prune list using ledger tool (#5041)
automerge
2019-07-12 16:58:13 -07:00
Sagar Dhawan 35ec7a5156
Decouple turns from segments in PoRep (#5004)
* Decouple Segments from Turns in Storage

* Get replicator local cluster tests running in a reasonable amount of time

* Fix unused imports

* Document new RPC APIs

* Check for exit while polling
2019-07-10 13:33:29 -07:00
Sagar Dhawan b8e7736af2 Move SLOTS_PER_SEGMENT to genesis (#4992)
automerge
2019-07-09 16:48:40 -07:00
Rob Walker f777a1a74c
groom replay_stage and poh_recorder (#4961)
* groom replay_stage and poh_recorder

* fixup

* fixup

* don't freeze() parent, need to review bank_forks and maybe vote...
2019-07-09 15:36:30 -07:00
carllin 22ef3c7c54
Blob verify (#4951)
* Ensure signable data is not out of range

* Add a broadcast stage that puts bad sizes in blobs

* Resign blob after modifyign size

* Remove assertions that fail when size != meta.size
2019-07-08 18:21:52 -07:00
carllin fc180f4cbf
Halve stake of malicious validator (#4937) 2019-07-05 15:45:39 -07:00
Sagar Dhawan 55b8ff72d0
Enable parallel tests to reduce test time (#4919)
* Add crate to serialize some tests

* Ignore unused attribute warning

* Enable parallel run in CI

* Try to fix lograte tests

* Fix interdependent counter tests
2019-07-02 17:35:03 -07:00
Pankaj Garg 3615445a12
Broadcast run for injecting fake blobs in turbine (#4889)
* Broadcast run for injecting fake blobs in turbine

* address review comments

* new local cluster test that uses fake blob broadcast

* added a test to make sure tvu_peers ordering is guaranteed
2019-07-01 17:54:03 -07:00
carllin 6cec61dcfc
Re-enable tests (#4848) 2019-06-27 12:09:14 -07:00
Justin Starry 861d6468ca Stake weighted pruning for the gossip network (#4769)
* Stake weighted pruning

* Fix compile error

* Fix clippy errors

* Add helper for creating a connected staked network

* Bug fixes and test groundwork

* Small refactor

* Anatoly's feedback and tests

* Doc updates

* @rob-solana's feedback

* Fix test bug and add log trace

* @rob-solana's feedback
2019-06-26 00:30:16 -07:00
Michael Vines 206e62271b Ignore flaky test_two_unbalanced_stakes (#4794)
automerge
2019-06-23 20:55:43 -07:00
carllin aacb38864c
Mark dead forks in replay stage (#4715)
* Add DeadSlots column family

* Filter dead forks from get_slots_since

* Mark erroring slots as dead in replay stage, add test

* Mark dead forks in progress instead of removing them

* Fix logging process_entries failures in replay_stage

* Unignore test_fail_entry_verification_leader
2019-06-20 15:50:41 -07:00
Michael Vines 4a32bc48d2 ignore unstable test_repairman_catchup 2019-06-20 09:24:11 -07:00
carllin 46bb79df29
Support for custom BroadcastStage in local cluster tests (#4716)
* Refactor BroadcastStage to support custom implementations, add FailEntryVerificationBroadcastRun implementation

* Plumb switch on broadcast type through validator

* Add test for validator generating non-verifiable entries to local_cluster

* Fix bad initializers

* Refactor broadcast run code into utils
2019-06-19 00:13:19 -07:00
Sagar Dhawan cc48773b03 Add "download from replicator" utility (#4709)
automerge
2019-06-17 18:12:13 -07:00
Rob Walker 4c6a6d63bf
add MiningPools, fund validator MiningPools from inflation (#4676)
* add MiningPool fund validator MinigPools from inflation

* fixup

* finish rename of MINIMUM_SLOT_LENGTH to MINIMUM_SLOTS_PER_EPOCH

* deterministic miningpool location

* point_value, not credit_value... use f64
2019-06-14 11:38:37 -07:00
carllin 8c1b9a0b67
Data plane verification (#4639)
* Add signature to blob

* Change Signable trait to support returning references to signable data

* Add signing to broadcast

* Verify signatures in window_service

* Add testing for signatures to erasure

* Add RPC for getting current slot, consume RPC call in test_repairman_catchup for more deterministic results
2019-06-12 16:43:05 -07:00
anatoly yakovenko 82df267ec9
s/avalanche/turbine (#4561)
* s/avalanche/turbine/g
2019-06-06 12:48:40 -07:00
Pankaj Garg 70a16e91a5
Randomize avalanche broadcast peer table for each blob (#4529)
* fix clippy warnings
2019-06-03 20:38:05 -07:00
Pankaj Garg 3574469052
Add random distribution for avalanche peers (#4493)
* Add random distribution for avalanche peers

* fix clippy warnings

* bug fixes

* nits
2019-06-01 07:55:43 -07:00
Sathish 182096dc1a
Create bank snapshots (#4244)
* Revert "Revert "Create bank snapshots (#3671)" (#4243)"

This reverts commit 81fa69d347.

* keep saved and unsaved copies of status cache

* fix format check

* bench for status cache serialize

* misc cleanup

* remove appendvec storage on purge

* fix accounts restore

* cleanup

* Pass snapshot path as args

* Fix clippy
2019-05-30 21:31:35 -07:00
carllin 045d4d5294
Unignore test test_repairman_catchup (#4484) 2019-05-30 13:21:12 -07:00
carllin c82f4a1b6d
Unignore test_repairman_catchup 2019-05-29 21:59:41 -07:00
carllin 348c2263ba
Remove genesis blockhash (#4471)
* Remove genesis blockhash

* Remove genesis blockhash from tests

* Fix golden
2019-05-29 17:29:02 -07:00
Sagar Dhawan 335dfdc4d5
Fix Gossip skipping push for some values (#4463)
* Make gossip skip over values from Pruned nodes

* Add test and init blooms to contain the origin
2019-05-28 18:39:40 -07:00
Michael Vines 2adb98a4a0 Ignore flaky test_repairman_catchup (#4439)
automerge
2019-05-26 12:24:20 -07:00
carllin d772a27936 Plumb ClusterInfoRepairListener (#4428)
automerge
2019-05-24 19:20:09 -07:00
Michael Vines cfe5afd34c
_id => _pubkey variable renaming (#4419)
* wallet: rename *_account_id to *_account_pubkey

* s/from_id/from_pubkey/g

* s/node_id/node_pubkey/g

* s/stake_id/stake_pubkey/g

* s/voter_id/voter_pubkey/g

* s/vote_id/vote_pubkey/g

* s/delegate_id/delegate_pubkey/g

* s/account_id/account_pubkey/g

* s/to_id/to_pubkey/g

* s/my_id/my_pubkey/g

* cargo fmt

* s/staker_id/staker_pubkey/g

* s/mining_pool_id/mining_pool_pubkey/g

* s/leader_id/leader_pubkey/g

* cargo fmt

* s/funding_id/funding_pubkey/g
2019-05-23 23:20:04 -07:00
Michael Vines 94beb4b8c2
More fullnode -> validator renaming (#4414)
* s/fullnode_config/validator_config/g

* s/FullnodeConfig/ValidatorConfig/g

* mv core/lib/fullnode.rs core/lib/validator.rs

* s/Fullnode/Validator/g

* Add replicator-x.sh

* Rename fullnode.md to validator.md

* cargo fmt
2019-05-23 22:05:16 -07:00
Michael Vines 2ed77b040a
create_genesis_block() now returns a struct (#4403) 2019-05-22 20:39:00 -07:00
Michael Vines f079a78c5e Remove fee arg from system_transaction::* (#4346)
automerge
2019-05-20 10:03:19 -07:00
Michael Vines 392a39dd54
Poh subsystem cleanup, genesis plumbing, enable real PoH on edge testnet (#4292)
* Remove unused PohServiceConfig::Step

* Clarify variable name

* Poh::hash() now takes an iteration counter

* man -> max

* Inline functions with single call site

* Move PohServiceConfig into GenesisBlock

* Add plumbing to enable real PoH on testnets

* Batch hashes to improve PoH hash rate

* Ensure a constant hashes_per_tick

* Remove PohEntry mixin field

* Poh/PohEntry no longer maintains tick_height

* Ensure a constant hashes_per_tick

* ci/localnet-sanity.sh: Use real PoH

* Rework Poh/PohService to keep PohRecorder unlocked as much as possible while hashing
2019-05-18 14:01:36 -07:00
Rob Walker 39e85a3e53
kill some bs58 (#4316)
* kill some bs58

* fixup
2019-05-16 21:43:18 -07:00
Sagar Dhawan a0ffbf50a5
Correctly remove replicator from data plane after its done repairing (#4301)
* Correctly remove replicator from data plane after its done repairing

* Update discover to report nodes and replicators separately

* Fix print and condition to be spy
2019-05-16 07:14:58 -07:00
Sagar Dhawan fd3f2cb910
Add Storage accounts for all nodes (#4298)
* Setup storage keypairs for all nodes

* Clean up naming

* clippy

* Update arg value_names
2019-05-15 15:19:29 -07:00
Rob Walker 0c1191c3ee
rework staking_utils (#4283) 2019-05-14 16:15:51 -07:00
carllin 3bd921264a
Move EpochSchedule into own module (#4272) 2019-05-13 16:24:32 -07:00
Michael Vines a2e3a92b01
Extend GetBlockHash RPC API to include the fee scehdule for using the returned blockhash (#4222) 2019-05-13 12:49:37 -07:00
Rob Walker 81fa69d347
Revert "Create bank snapshots (#3671)" (#4243)
This reverts commit abf2b300da.
2019-05-09 19:27:27 -07:00
Sathish abf2b300da Create bank snapshots (#3671)
* Be able to create bank snapshots

* fix clippy

* load snapshot on start

* regenerate account index from the storage

* Remove rc feature dependency

* cleanup

* save snapshot for slot 0
2019-05-09 19:27:06 -07:00
carllin 575a0e318b
Add newly completed slots signal to Blocktree (#4225)
* Add channel to blocktree for communicating when slots are completed

* Refactor RepairService options into a RepairStrategy
2019-05-09 14:10:04 -07:00
carllin b8fd51e97d
Add new gossip structure for supporting repairs (#4205)
* Add Epoch Slots to gossip

* Add new gossip structure to support Repair

* remove unnecessary clones

* Setup dummy fast repair in repair_service

* PR comments
2019-05-08 13:50:32 -07:00
Michael Vines 55e3b7d380
Storage transactions are now paid for by a system account (#4193)
* Pay program loading fees from a system account

* Pay transaction fees from a system account
2019-05-07 15:01:10 -07:00
Sagar Dhawan 2107e15bd3
Reduce Avalanche redundancy and implement traditional fanout (#4174)
* Reduce Avalanche redundancy and implement traditional fanout

* Revert tiny fanout

* Update diagrams and docs based on review comments
2019-05-07 13:24:58 -07:00
Michael Vines 4f3b22d04e
Minor code restyling, no functional changes 2019-05-07 12:35:29 -07:00
Rob Walker 8e400fc4bd
rework genesis (passive staking groundwork) (#4187)
* rework genesis

* fixup
2019-05-07 11:16:22 -07:00
carllin 297328ff9a Fix improper tick sleeping time in test (#4155)
automerge
2019-05-03 20:15:10 -07:00
Sagar Dhawan a7b695c27a
Change replicators to slot-based (#4118) 2019-05-03 16:27:53 -07:00
carllin 73f250f03a
Make minimum warmup period 32 slots long (#4031)
* Make minimum warmup period 32 slots long

* PR fixes
2019-04-29 15:26:52 -07:00
Sagar Dhawan 0cbac26591
Add genesis blockhash to blobs (#3953) 2019-04-23 16:24:44 -07:00
Sagar Dhawan 43f7cd8149
Fix Retransmit slamming the leader with its own blobs (#3938) 2019-04-22 18:41:01 -07:00
Sagar Dhawan 349e8a9462
Ensure forwarded Blobs don't break Erasure (#3907) 2019-04-20 16:44:06 -07:00
Sagar Dhawan c0bffb56df
Update Avalanche to use Epoch Stakes (#3900) 2019-04-19 22:31:40 -07:00
Sagar Dhawan 970cc32e65
Fix Data Plane computation when stakes are equal (#3913) 2019-04-19 21:07:21 -07:00
Rob Walker efbb573316
add an env::var driven localcluster test (#3906) 2019-04-19 15:47:03 -07:00
carllin 512bfc93cb
Add a cache for leader schedules (#3841)
* Add a cache for leader schedules
2019-04-19 02:39:44 -07:00
Tyera Eulberg a9cfae486c
Revert-revert migrate to ed25519-dalek crate (#3877)
* Revert "Revert "Migrate from ring to ed25519-dalek, take 2 (#3844)" (#3868)"

This reverts commit 6a878602f2.

* Fix Signature::verify method
2019-04-18 14:37:20 -06:00
Tyera Eulberg 6a878602f2
Revert "Migrate from ring to ed25519-dalek, take 2 (#3844)" (#3868)
This reverts commit e9b82bacda.
2019-04-18 11:47:34 -06:00
Tyera Eulberg e9b82bacda
Migrate from ring to ed25519-dalek, take 2 (#3844)
* Migrate from ring to ed25519-dalek

* Move gen_keypair_file test to a more appropriate location

* Fixup bench-exchange and add helper fn for single deterministic keypair

* Update golden
2019-04-18 10:38:32 -06:00
anatoly yakovenko 68fc303b9b
Rework Accounts for fast squash, hashing state and checkpoint recovery. (#3613)
* accounts rewrite

* ignore grow tests

* skip duplicate roots

* allow for a root race

* logger

* accounts_index tests

* tests

* tests
2019-04-15 17:15:50 -07:00
Sagar Dhawan 3fcf03ff3e
Refactor LocalCluster and add support for listener nodes (#3790) 2019-04-15 15:27:45 -07:00
Jack May 44b391096d
Configurable local cluster native processors (#3676) 2019-04-08 11:15:58 -07:00
Greg Fitzgerald 35298e01a8 Remove Instruction wrapper structs and name functions after enum fields 2019-04-03 13:34:27 -07:00
Greg Fitzgerald 867f6f107b Rename SystemInstruction::Move to SystemInstruction::Transfer 2019-04-03 08:35:57 -06:00
Greg Fitzgerald 43bb813cbe Rename 'new_account' to 'new_user_account'
And 'new_program_account' to 'new_account'
2019-04-02 21:24:42 -06:00
Tyera Eulberg 6ff2a0a75e Rework discover to handle additional parameters, and be unit-testable 2019-04-01 23:05:25 -06:00
Greg Fitzgerald fcef54d062 Add a constructor to generate random pubkeys 2019-03-31 16:23:18 -06:00
carllin f886b3b12b
Fix resetting PohRecorder to wrong bank (#3553)
* Check whether future slot already has transmission
2019-03-29 20:00:36 -07:00
Sagar Dhawan 7d0ff8e713
Re-enable Replicator test (#3534) 2019-03-27 17:21:49 -07:00
Sagar Dhawan e8cc566b2b
Storage Account setup for replicators and validators (#3516)
* Setup Storage Accounts for replicators

* Setup Storage Accounts for validators

* Add Replicator Info to Local Cluster and Add test
2019-03-27 15:54:09 -07:00
Sagar Dhawan 36ea088387
Fix Storage Stage not receiving entries when node is leader (#3528) 2019-03-27 13:10:33 -07:00
carllin 68c35bfde6
Restart node test (#3459) (#3465)
* Restart node test (#3459)

* Add test to local_cluster for restarting a node

* fix so that we don't hit end of epoch - leader not found before trying to transfer

* Do not look for confirmations, b/c nobody is voting on empty transmissions in this single node test
2019-03-23 19:19:55 -07:00
anatoly yakovenko 148e08a8a5
Enable cluster tests (#3372)
* Cluster tests

* stable!

* fixup! stable!

* fixup! fixup! stable!

* fixup! fixup! fixup! stable!

* fixup! fixup! fixup! fixup! stable!

* fixed space

* add getNumBlocksSinceSignatureConfirmation entry for the json rpc docs

* Check in upcoming epochs for potential leadership slots in next_leader_slot()
2019-03-21 07:43:21 -07:00
Stephen Akridge 682b1b89b3 Adjust for vector of entries in blobs. 2019-03-19 13:49:48 -07:00
Stephen Akridge ee58c1f960 Add test for replicator ledger download
Add an interface to query the storage slot a
  replicator is holding on storage_addr port.
Fix logic to poll blocktree for all slots
  replicated being filled.
Add test logic to ask replicator what slot it
  is replicating and then download an entry in
  the slot.
2019-03-19 13:49:48 -07:00
Stephen Akridge 07f4dd385d Cleanup replicator sockets
Add optional UdpSocket for storage interface.
Add new_localhost_replicator to create a new replicator local node.
2019-03-19 13:49:48 -07:00
Pankaj Garg 56fcc93ef5
Schedule node for consecutive slots as leader (#3353)
* Also tweak epoch and slot duration

* new test for leader schedule
2019-03-19 06:36:45 -07:00
Rob Walker c70412d7bb
move core tests to core (#3355)
* move core tests to core

* remove window

* fix up flaky tests

* test_entryfication needs a singly-threaded banking_stage

* move core benches to core

* remove unnecessary dependencies

* remove core as a member for now, test it like runtime

* stop running tests twice

* remove duplicate runs of tests in perf
2019-03-18 22:08:21 -07:00
Mark 656fb173f9
Extract kvstore into separate crate (#3327)
* extract kvstore into new crate

* add kvstore crate to CI publishing list
2019-03-15 18:42:47 -05:00
Mark fa3e1fa7c9
Add error correction to write-log (#3323) 2019-03-15 15:04:34 -05:00
Mark 56b0ba2601
KvStore - A data-store to support BlockTree (#2897)
* Mostly implement key-value store and add integration points

Essential key-value store functionality is implemented, needs more work to be integrated, tested, and activated.

Behind the `kvstore` feature.
2019-03-11 17:53:14 -05:00