Commit Graph

104 Commits

Author SHA1 Message Date
Michael Vines da5b777ee7 Purge Default::default() 2019-02-09 10:12:32 -08:00
carllin fd7db7a954
Support multiple forks in the ledger (#2277)
* Modify db_ledger to support per_slot metadata, add signal for updates, and add chaining to slots in db_ledger

* Modify replay stage to ask db_ledger for updates based on slots

* Add repair send/receive metrics

* Add repair service, remove old repair code

* Fix tmp_copy_ledger and setup for tests to account for multiple slots and tick limits within slots
2019-02-07 15:10:54 -08:00
Greg Fitzgerald dad0bfe447 Replace transaction traits with structs
Also:
* SystemTransaction::new -> new_account
* SystemTransaction::new_create -> new_program_account
2019-02-01 11:38:25 -08:00
Rob Walker 1b50fbbc90
remove Result<> from Blob accessors, add parent (#2608)
* remove Result<> from Blob accessors, add parent
* update chacha's golden
* fixup benches
2019-01-30 20:18:28 -08:00
Sathish e0f046b7a5
Optimize Transaction/Instruction serialization with custom routine (#2515)
* Optimize transaction serialization with custom routine to reduce the serialized size.

* Update serialized_size to accept self as parameter

* Optimize serialize / deserialize operations
2019-01-24 21:14:15 -08:00
Pankaj Garg a9b083e585
Set fetch stage socket non blocking to false while during recv (#2542)
* Set fetch stage socket non blocking to false while during recv

* remove ProgramError changes from this PR
2019-01-24 12:46:40 -08:00
Rob Walker 022a97da99
revert revert of kill window (#2427)
* remove window code from most places
* window used only for testing
* remove unecessary clippy directives
2019-01-15 10:51:53 -08:00
Rob Walker 447243f994
Revert "remove window code from most places" (#2417)
* Revert "Fix link to book in Local Testnet section (#2416)"

This reverts commit 710c0c9980.

* Revert "Add current leader information to dashboard (#2413)"

This reverts commit f0300c1711.

* Revert "remove window code from most places (#2389)"

This reverts commit e3c0bd5a3f.
2019-01-14 15:11:18 -08:00
Rob Walker e3c0bd5a3f
remove window code from most places (#2389)
* remove window code from most places
* window used only for testing
* remove unecessary clippy directives
2019-01-14 12:11:55 -08:00
Greg Fitzgerald 7341298a11
Cleanup tpu forwarder (#2377)
* Use unwrap() on locks

An error there generally indicates a programmer error, not a
runtime error, so a detailed runtime message is not generally useful.

* Only clone Arcs when passing them across thread boundaries

* Cleanup TPU forwarder

By separating the query from the update, all the branches get easier to
test. Also, the update operation gets so simple, that we see it
belongs over in packet.rs.

* Thanks clippy

cute
2019-01-10 13:34:48 -07:00
carllin 58a4905916
Make reconstruct_entries_from_blobs() support Blobs and borrowed SharedBlobs, make distinction between to_blobs and to_shared_blobs (#2270) 2018-12-22 19:30:30 -08:00
Rob Walker 4f48f1a850
add db_ledger genesis, rework to_blob(), to_blobs() (#2135) 2018-12-12 20:42:12 -08:00
Carl 245362db96 Make a dummy version of serving repairs from db_ledger 2018-12-12 19:12:51 -08:00
carllin ae903f190e
Broadcast for slots (#2081)
* Insert blobs into db_ledger in broadcast stage to support leader to validator transitions

* Add transmitting real slots to broadcast stage

* Handle real slots instead of default slots in window

* Switch to dummy repair on slots and modify erasure to support leader rotation

* Shorten length of holding locks

* Remove logger from replicator test
2018-12-12 15:58:29 -08:00
Rob Walker aeee25e703
add tick_height to Entry to be able to repair by period, chain forks of Entries, etc. (#2096) 2018-12-10 20:03:04 -08:00
Greg Fitzgerald a8d6c75a24 cargo +nightly fix --features=bpf_c,cuda,erasure,chacha --edition-idioms 2018-12-08 23:19:55 -07:00
Greg Fitzgerald ec5a8141eb cargo fix --edition 2018-12-08 23:19:55 -07:00
carllin 9c30bddb88
Rocks db erasure decoding (#1900)
* Change erasure to consume new RocksDb window

* Change tests for erasure

* Remove erasure from window

* Integrate erasure decoding back into window

* Remove corrupted blobs from ledger

* Replace Erasure result with result module's Result
2018-12-05 12:47:19 -08:00
Michael Vines 42689d4842 cargo fmt 2018-12-05 10:49:06 -08:00
Michael Vines 214ed3667c Move system_transaction out of src/ 2018-12-05 10:49:06 -08:00
Michael Vines c99f93e40a Remove signature.rs indirection 2018-12-03 13:50:00 -08:00
Michael Vines 0878bd53d9 Delete stub src/transaction.rs 2018-11-29 23:07:57 -08:00
carllin d9dabdfc74
Rocks db window utils (#1851)
* Implement new ledger module based on RocksDb

* Add db_window module for windowing functions from RocksDb
2018-11-19 23:20:18 -08:00
Michael Vines d96a6b42a5 Move drone into its own crate 2018-11-16 20:42:21 -08:00
Sagar Dhawan 729d28d910 Add poh verification before processing entries
- Replicate stage now verifies entries delivered
  by the window
- Minor refactor of entries_from_blobs
2018-11-13 14:17:00 -08:00
Rob Walker 6c10458b5b
leader slots in Blobs (#1732)
* add leader slot to Blobs
* remove get_X() methods in favor of X() methods for Blob
* add slot to get_scheduled_leader()
2018-11-07 13:18:14 -08:00
Tyera Eulberg f683817b48 Remove RPU; replace with RPC 2018-11-05 20:30:47 -07:00
carllin 298bd6479a
Add first leader to genesis (#1681)
* Add first leader to genesis entries, consume in genesis.sh

* Set bootstrap leader in the bank on startup, remove instantiation of bootstrap leader from bin/fullnode

* Remove need to initialize bootstrap leader in leader_scheduler, now can be read from genesis entries

* Add separate interface new_with_leader() in mint for creating genesis leader entries
2018-11-02 14:32:05 -07:00
Michael Vines e47fcb196b s/solana_program_interface/solana[_-]sdk/g 2018-10-25 12:31:45 -07:00
carllin 0bd1412562
Switch leader scheduler to use PoH ticks instead of Entry height (#1519)
* Add PoH height to process_ledger()

* Moved broadcast_stage Leader Scheduling logic to use Poh height instead of entry_height

* Moved LeaderScheduler logic to PoH in ReplicateStage

* Fix Leader scheduling tests to use PoH instead of entry height

* Change is_leader detection in repair() to use PoH instead of entry height

* Add tests to LeaderScheduler for new functionality

* fix Entry::new and genesis block PoH counts

* Moved LeaderScheduler to PoH ticks

* Cleanup to resolve PR comments
2018-10-18 22:57:48 -07:00
carllin 9931ac9780
Leader scheduler plumbing (#1440)
* Added LeaderScheduler module and tests

* plumbing for LeaderScheduler in Fullnode + tests. Add vote processing for active set to ReplicateStage and WriteStage

* Add LeaderScheduler plumbing for Tvu, window, and tests

* Fix bank and switch tests to use new LeaderScheduler

* move leader rotation check from window service to replicate stage

* Add replicate_stage leader rotation exit test

* removed leader scheduler from the window service and associated modules/tests

* Corrected is_leader calculation in repair() function in window.rs

* Integrate LeaderScheduler with write_stage for leader to validator transitions

* Integrated LeaderScheduler with BroadcastStage

* Removed gossip leader rotation from crdt

* Add multi validator, leader test

* Comments and cleanup

* Remove unneeded checks from broadcast stage

* Fix case where a validator/leader need to immediately transition on startup after reading ledger and seeing they are not in the correct role

* Set new leader in validator -> validator transitions

* Clean up for PR comments, refactor LeaderScheduler from process_entry/process_ledger_tail

* Cleaned out LeaderScheduler options, implemented LeaderScheduler strategy that only picks the bootstrap leader to support existing tests, drone/airdrops

* Ignore test_full_leader_validator_network test due to bug where the next leader in line fails to get the last entry before rotation (b/c it hasn't started up yet). Added a test test_dropped_handoff_recovery go track this bug
2018-10-10 16:49:41 -07:00
Pankaj Garg e10574c64d Remove recycler and it's usage
- The memory usage due to recycler was high, and incrementing with
  time.
2018-09-27 10:42:37 -06:00
jackcmay 9c47e022dc
break dependency of programs on solana core (#1371)
* break dependency of programs on Solana core
2018-09-27 07:49:26 -07:00
Greg Fitzgerald b7ae5b712a Move Pubkey into its own module 2018-09-26 20:40:40 -06:00
carllin e7383a7e66
Validator to leader (#1303)
* Add check in window_service to exit in checks for leader rotation, and propagate that service exit up to fullnode

* Added logic to shutdown Tvu once ReplicateStage finishes

* Added test for successfully shutting down validator and starting up leader

* Add test for leader validator interaction

* fix streamer to check for exit signal before checking socket again to prevent busy leaders from never returning

* PR comments - Rewrite make_consecutive_blobs() function, revert genesis function change
2018-09-25 15:41:29 -07:00
Rob Walker 9e3d2956d8 remove last recycle? 2018-09-24 08:09:41 -06:00
Greg Fitzgerald 6073cd57fa Boot Recycler::recycle() 2018-09-20 17:08:51 -06:00
Anatoly Yakovenko 431692d9d0 Use a Drop trait to keep track of lifetimes for recycled objects.
* Move recycler instances to the point of allocation
* sinks no longer need to call `recycle`
* Remove the recycler arguments from all the apis that no longer need them
2018-09-19 16:59:42 -06:00
anatoly yakovenko 6ec0e42220
budget as separate contract and system call contract (#1189)
* budget and system contracts and verification

* contract check_id methods
* system call contract
* verify contract execution rules
* move system into its own file
* allocate before transfer for budget
* store error in budget context
* budget contract and tests without bank
* moved budget of of bank
2018-09-17 13:36:31 -07:00
sakridge 4dc30ea104
Add recycler stats (#1187) 2018-09-13 14:49:48 -07:00
Pankaj Garg 90df6237c6
Implements recvmmsg() for UDP packets (#1161)
* Implemented recvmmsg() for UDP packets

- This change implements binding between libc API for recvmmsg()
- The function can receive multiple packets using one system call

Fixes #1141

* Added unit tests for recvmmsg()

* Added recv_mmsg() wrapper for non Linux OS

* Address review comments for recvmmsg()

* Remove unnecessary imports

* Moved target specific dependencies to the function
2018-09-13 14:41:28 -07:00
sakridge 1a4cd763f8
Fix missing recycle in recv_from (#1205)
In the error case that i>0 (we have blobs to send)
we break out of the loop and do not push the allocated r
to the v array. We should recycle this blob, otherwise it
will be dropped.
2018-09-13 08:29:18 -07:00
Rob Walker 9ab5692acf
fix "leak" in Blob::recv_from (#1198)
* fix "leak" in Blob::recv_from

fixes #1199
2018-09-12 14:45:43 -07:00
sakridge 297f859631
Change '>=' back to '>' to fix recycling of blobs/packets (#1192)
Recycler will have a strong ref to the item so it will be at
least 1, >= will always prevent recycling.
2018-09-11 16:52:45 -07:00
anatoly yakovenko c34d911eaf
Migrate Budget DSL to use the Account state (#979)
* Migrate Budget DSL to use the Account state instead of global bank data structures.

* Serialize Instruction into Transaction::userdata.
* Store the pending set in the Account::userdata
* Enforce the token balance rules on contract execution. This becomes the entry point for generic contracts.
* This pr will have a performance impact on the bank. The next set of changes will fix this by locking each account during multi threaded execution of all the contracts.
* With this change a contract transaction needs to store its state under an address. That address could be the destination of the tokens, or any random address. For the latter, an extra step would be needed to claim the tokens which isn't implemented by budget_dsl at the moment.
* test tracking issue 1157
2018-09-07 20:18:36 -07:00
Rob Walker 226d3b9471
Trace recycle() calls (#968)
* trace recycle() calls fixes #810
2018-09-05 05:07:02 +09:00
Greg Fitzgerald 8cc030ef84 Use Vec instead of VecDeque for SharedBlobs 2018-09-04 07:50:23 -10:00
Rob Walker 63e44dcc35 continue rendezvous refactor for gossip and repair
* remove trailing whitespace in ci/audit.sh

  * code review fixups
     * rename GOSSIP_PORT_RANGE => SOLANA_PORT_RANGE
     * remove out-of-date TODO in localnet-sanity.sh

  * remove features=test and code that was using it (localhost prohibitions in
      crdt) added TODO in crdt.rs, maybe we should boot localhost in production
      networks?

  * boot tvu_window from NodeInfo: instead, send repair requests from the repair
      socket (to gossip on peer) and answer repair requests via the sockaddr
      from the repair request

  * remove various unused pub functions

  * banish SocketAddr parse().unwrap() to a macro that can also accept simpler stuff
2018-08-31 23:21:07 +09:00
Greg Fitzgerald 90ae662e4d Fix packet header offset
And update transaction offsets to use the same approach as packet.rs.
Maybe this should be serialized_size(), but thanks to this
GenericArray update, those values are the same.
2018-08-26 14:27:19 -06:00
sakridge cabe80b129
Increment counter by number of packets received (#1049)
So that we can see the total packets/s
2018-08-23 12:32:50 -07:00