Commit Graph

146 Commits

Author SHA1 Message Date
Greg Fitzgerald 0fa27f65bb Use the default Pubkey formatter instead of debug_id() 2018-09-06 16:31:47 -06:00
Greg Fitzgerald c9a1ac9b8c Don't propogate errors we'll never handle 2018-09-04 06:01:32 -10:00
Rob Walker 176e806d94 rework of netwrk rendezvous
* rename NodeInfo field of Node from "data" to "info"
      (touches a lot of files)

  * update client to use gossip to find leader, a la drone

  * rework multinode scripts
      * move more stuff into rust
      * added usage to all
      * no more rsync unless you're a validator (TODO: whack that, too)
  * fullnode doesn't bail if drone isn't up yet, just keeps trying
  * drone doesn't bail if network isn't up yet, just keeps trying
2018-08-31 23:21:07 +09:00
Rob Walker 63e44dcc35 continue rendezvous refactor for gossip and repair
* remove trailing whitespace in ci/audit.sh

  * code review fixups
     * rename GOSSIP_PORT_RANGE => SOLANA_PORT_RANGE
     * remove out-of-date TODO in localnet-sanity.sh

  * remove features=test and code that was using it (localhost prohibitions in
      crdt) added TODO in crdt.rs, maybe we should boot localhost in production
      networks?

  * boot tvu_window from NodeInfo: instead, send repair requests from the repair
      socket (to gossip on peer) and answer repair requests via the sockaddr
      from the repair request

  * remove various unused pub functions

  * banish SocketAddr parse().unwrap() to a macro that can also accept simpler stuff
2018-08-31 23:21:07 +09:00
Rob Walker 1af4cee63b fix #1079
* move gossip/NCP off assuming anything about its address
  * use a single socket to send and receive gossip
  * remove --addr/-a from CLIs
  * rearrange networking utility code
  * use Arc<UdpSocket> to share the Sync-safe UdpSocket among threads
  * rename TestNode to Node

TODO:

  * re-enable 127.0.0.1 as a valid address in crdt
  * change repair request/response to a similar, single socket
  * pick cloned sockets or Arc<UdpSocket> for all these (rpu uses tryclone())
  * update contact_info with network truthiness instead of what the node
      says?
2018-08-31 23:21:07 +09:00
anatoly yakovenko f7c0d30167
Disallow localhost in deployment (#1064)
* disallow localhost in deployment

* tests

* fmt

* integration tests do not have a flag to check

* fmt
2018-08-25 21:09:18 -07:00
Greg Fitzgerald be58fdf1bb Less constructors 2018-08-23 13:42:54 -07:00
Greg Fitzgerald 9c5e69bf3d Don't offer two ways to specify a leader 2018-08-23 13:42:54 -07:00
Pankaj Garg 24b0031925 Reduce number of nodes in multinode test (#1003) 2018-08-20 13:40:42 -07:00
Tyera Eulberg f73f3941cd Revert ill-advised jsonrpc marker, and handle jsonrpc server close 2018-08-18 12:29:49 -06:00
Tyera Eulberg 1eb8724a89 Disable Rpc module for other tests to prevent port conflicts 2018-08-18 12:29:49 -06:00
Rob Walker 3ae867bdd6 fixups 2018-08-18 02:22:52 -07:00
Pankaj Garg f6f0900506
Large network test to not poll validator for sigs (#998)
- The finality is already reached. The test will check the signature
  in validators once, instead of polling. This will help speed up the test.
2018-08-17 14:38:19 -07:00
Anatoly Yakovenko a284030ecc Account type with state
comments

fixups!

fixups!

fixups for a real Result<> from get_balance()

on 2nd thought, be more rigorous

Merge branch 'rob-solana-accounts_with_state' into accounts_with_state

update

review comments

comments

get rid of option
2018-08-16 14:44:51 -07:00
Pankaj Garg c2983f824e
Refactored large network test to use finality to assert success (#978) 2018-08-15 20:05:43 -07:00
anatoly yakovenko 88d6fea999 Revert "Accounts with state (#954)"
This reverts commit c23fa289c3.
2018-08-15 19:44:39 -07:00
anatoly yakovenko c23fa289c3
Accounts with state (#954)
* Account type with state

* fixed test according to @rob-solana
2018-08-15 14:32:11 -07:00
Pankaj Garg db35f220f7
Run multinode test for enough iterations for a small node count test (#971) 2018-08-15 10:44:14 -07:00
Pankaj Garg 982afa87a6
Retransmit blobs from leader from window (#975)
- Some nodes don't have leader information while leader is broadcasting
  blobs to those nodes. Such blobs are not retransmitted. This change
  rertansmits the blobs once the leader's identity is know.
2018-08-14 21:51:37 -07:00
Greg Fitzgerald b801ca477d Declare fullnode a word 2018-08-13 20:17:16 -06:00
Greg Fitzgerald 2727067b94 Move winow into its own module 2018-08-13 20:17:16 -06:00
Pankaj Garg bf15cad36b
Add get_finality request and use it from multinode test (#941) 2018-08-13 08:55:13 -07:00
anatoly yakovenko f07c038266
Fix bank coalescing (#949)
* fix bank coalescing

* comments

* fix bench

* fix bench

* backout banking stage coalescing

* 120 nodes

* 100
2018-08-12 10:04:21 -07:00
Rob Walker 96d6985895 rework read_ledger, LedgerWriter, and LedgerWindow for recover()
fixes #910
2018-08-10 18:07:23 -07:00
Greg Fitzgerald ad331e6d56 Rename PublicKey type to Pubkey
Recognize pubkey as a noun meaning the public key of a keypair.
2018-08-09 13:41:37 -06:00
Greg Fitzgerald c2bbe4344e Rename KeyPair to Keypair 2018-08-09 13:41:37 -06:00
Greg Fitzgerald 8567253833 Ignore flaky test 2018-08-09 10:15:10 -06:00
Pankaj Garg a6857dbaaa Updated node count to 230. Increased wmem on CI large 2018-08-08 13:13:18 -07:00
pgarg66 75854cc234
Update dynamic network test with more nodes (#904)
- Check for correct OS params in test-large-network.sh
2018-08-08 06:52:57 -07:00
Rob Walker 9783d47fd1 write a "unit" test for WindowLedger (it was working ;)
clear flags on fresh blobs, lest they sometimes impersonate coding blobs...

fix bug: advance *received whether the blob_index is in the window or not,
  failure to do so results in a stalled repair request pipeline
2018-08-08 04:28:09 -07:00
Stephen Akridge 38be61bd22 Check for log level before doing perf counter work
Perf counters, especially when running the dynamic test can cause
functions like crdt::apply_updates to be really slow (>500ms).
2018-08-08 00:16:53 -07:00
Rob Walker fbc754ea25 plug in LedgerWindow
fixes #872
2018-08-07 17:27:53 -07:00
pgarg66 16772d3d51
Coalesce multiple blobs if received close together (#869)
- This helps reduce unnecessary growth of window if small blobs
  are received in small space of time
2018-08-07 10:29:57 -07:00
Kazuyoshi Kato 9c1b6288a4 Use ? instead of unwrap()
This change addresses #833, while there are still some unwrap() though.
2018-08-07 08:10:22 -07:00
Rob Walker 3beb38ac8a /tmp/farf no good on multi-user machine 2018-08-06 14:53:40 -07:00
Rob Walker 039ed01abf on 2nd thought: do not copy_ledger() for this test 2018-08-06 08:51:41 -07:00
Rob Walker ead0eb2754 move copy_ledger() back into ledger.rs
Don't recover() for copy(), as copy() is already tolerant of things
    recover() guards against.  Note: recover() is problematic if the ledger is
    "live", i.e. is currently being written to.
2018-08-06 08:51:41 -07:00
Rob Walker c3db2df7eb tweak random access ledger
* add recover_ledger() to deal with expected common ledger corruptions
  * add verify_ledger() for future use cases (ledger-tool)
  * increase ledger testing
  * allow replicate stage to run without a ledger
  * ledger-tool to output valid json
2018-08-06 08:51:41 -07:00
Rob Walker 5193819d8e Revert "Revert "plug in new ledger""
This reverts commit 57e928d1d0.
2018-08-06 08:51:41 -07:00
Rob Walker f2b1a04bca cargo fmt fixups 2018-08-03 11:59:25 -07:00
Rob Walker 57e928d1d0 Revert "plug in new ledger"
This reverts commit 46d9ba5ca0.
2018-08-03 10:24:51 -07:00
Rob Walker 46d9ba5ca0 plug in new ledger 2018-08-02 21:50:47 -07:00
Tyera Eulberg fda3b9bbd4 Use new PublicKey format instead of hex 2018-08-01 16:15:14 -07:00
Pankaj Garg d622742b84 Mark test-multinode-basic as ignore 2018-08-01 10:13:05 -06:00
sakridge 2ea6f86199 Submit leader's vote after observing 2/3 validator votes (#780)
* fixup!

* fixups!

* send the vote and count it

* actually vote

* test

* Spelling fixes

* Process the voting transaction in the leader's bank

* Send tokens to the leader

* Give leader tokens in more cases

* Test for write_stage::leader_vote

* Request airdrop inside fullnode and not the script

* Change readme to indicate that drone should be up before leader

And start drone before leader in snap scripts

* Rename _kp => _keypair for keypairs and other review fixups

* Remove empty else
* tweak test_leader_vote numbers to be closer to testing 2/3 boundary
* combine creating blob and transaction for leader/validator
2018-07-31 22:07:53 -07:00
pgarg66 7c5172a65e
Converted sigverify disable flag to runtime check instead of "cfg" (#799) 2018-07-31 16:54:24 -07:00
pgarg66 0c33c9e0d7
Dynamic network test changes (#795)
- No sigverify if feature sigverify_cpu_disable is used
- Purge validators in the test if lag count increases beyond
  SOLANA_DYNAMIC_NODES_PURGE_LAG environment variable
- Other useful log messages in the test
2018-07-30 13:57:10 -07:00
sakridge 0b47404ba6 Check for default leader and use cmp::max for a bit nicer code (#779) 2018-07-27 15:53:31 -07:00
Stephen Akridge 7f4844f426 More stats in dynamic multinode test 2018-07-27 11:55:09 -07:00
Michael Vines c52d4eca0b Stop validator first to stop voting before the leader stops 2018-07-26 17:29:32 -07:00
Michael Vines 7672506b45 Validators now vote once a second regardless 2018-07-26 17:07:42 -07:00
Rob Walker 297896bc49 honor environment variable SOLANA_DYNAMIC_NODES, obviating need to edit+compile for re-test 2018-07-25 20:45:16 -07:00
pgarg66 9daa7bdbe2
Replace rayon with threads for dynamic network test (#745) 2018-07-24 17:54:29 -07:00
pgarg66 166099b9d9
Start validators in parallel in multinode test (#727) 2018-07-23 09:27:06 -07:00
Martius Lim 84225beeef replace 'daddr' checks with 'is_valid_address()' 2018-07-20 13:09:01 -04:00
Greg Fitzgerald 28af9a39b4 Don't clone before borrowing
Clippy told us to change function parameters to references, but
wasn't able to then tell us that the clone() before borrowing
was superfluous. This patch removes those by hand.

No expectation of a performance improvement here, since we were
just cloning reference counts. Just removes a bunch of noise.
2018-07-18 08:04:31 -04:00
Anatoly Yakovenko 3d45b04da8 review comments 2018-07-17 15:51:32 -04:00
Anatoly Yakovenko 9e2f26a5d2 review comments 2018-07-17 15:51:32 -04:00
Anatoly Yakovenko a016f6e82e bulds 2018-07-17 15:51:32 -04:00
Anatoly Yakovenko eb3e5fd204 server too 2018-07-17 15:51:32 -04:00
Anatoly Yakovenko 72282dc493 fast exit dynamic test 2018-07-17 15:51:32 -04:00
anatoly yakovenko d8c9655128
Dynamic test assert (#643)
* log responder error to warn

* log responder error to warn

* fixup!

* fixed assert

* fixed bad ports issue

* comments

* test for dummy address in Crdt::new instaad of NodeInfo::new

* return error if ContactInfo supplied to Crdt::new cannot be used to connect to network

* comments
2018-07-16 19:31:52 -07:00
Rob Walker 710fa822a0 fixups 2018-07-12 22:51:55 -07:00
Rob Walker a3ff40476e Banish stdin/stdout for ledger
step one: accept a "ledger file" argument instead of "outfile"
2018-07-12 22:51:55 -07:00
Greg Fitzgerald 30f0c25b65 Fix all remaining clippy warnings
Fixes #586
2018-07-12 09:40:40 -06:00
OEM Configuration (temporary user) 468ac9facd Refactor the "ReplicatedData" struct
Rename the "ReplicatedData" struct to the "NodeInfo" struct.
Also refactors and renames the members in this struct.
2018-07-11 11:38:41 -06:00
Anatoly Yakovenko 7687436bef some cleanup on messages 2018-07-10 13:32:31 -06:00
Anatoly Yakovenko 6a1b5a222a rebase builds 2018-07-10 13:32:31 -06:00
Anatoly Yakovenko be2bf69c93 initial vote stage
wip

voting

wip

move voting into the replicate stage

update

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

fixup!

update

fixup!

fixup!

fixup!

tpu processing votes in entries before write stage

fixup!

fixup!

txs

make sure validators have an account

fixup!

fixup!

fixup!

exit fullnode correctly

exit on exit not err

try 50

add delay for voting

300

300

startup logs

par start

100

no rayon

retry longer

log leader drop

fix distance

50 nodes

100

handle deserialize error

update

fix broadcast

new table every time

tweaks

table

update

try shuffle table

skip kill

skip add

purge test

fixed tests

rebase 2

fixed tests

fixed rebase

cleanup

ok for blobs to be longer then window

fix init window

60 nodes
2018-07-10 13:32:31 -06:00
Greg Fitzgerald 0672794692 Cleanup leader restarts
Try to avoid adding a runtime codepath when all paths are
statically known.
2018-07-10 11:11:36 -06:00
Greg Fitzgerald c65c0d9b23 Expose fewer exit variables 2018-07-10 11:11:36 -06:00
Anatoly Yakovenko 63985d4595 renamed to contact_info 2018-07-09 20:40:14 -06:00
Anatoly Yakovenko 97dd1834d7 fix tests, more logs 2018-07-09 20:40:14 -06:00
Anatoly Yakovenko 2ea030be48 stick all the addrs into one struct 2018-07-09 20:40:14 -06:00
Rob Walker 90a4ab7e57 fixes issue #299 2018-07-09 14:50:14 -07:00
Rob Walker 412e15fbdc add test for populated window 2018-07-09 14:50:14 -07:00
Rob Walker ed0a590549 support an initial window filled with last up-to-WINDOW_SIZE blobs 2018-07-09 14:50:14 -07:00
Greg Fitzgerald 9c456b2fb0 Fixup the integration tests 2018-07-04 16:40:34 -06:00
Anatoly Yakovenko 22d2c962b2 ignore 2018-07-03 18:10:16 -06:00
Anatoly Yakovenko e771d36278 Better logs 2018-07-03 18:10:16 -06:00
Rob Walker 800c2dd370 make the leader append to the ledger file 2018-07-03 17:17:52 -06:00
Anatoly Yakovenko ae309f80f7 boot from file test 2018-07-03 14:00:39 -06:00
Anatoly Yakovenko fa70b3bf70 split out files, fixed a bug @garious! 2018-07-03 14:00:39 -06:00
Anatoly Yakovenko 3a90f138b2 dynamit network test
* cleaned up fullnode api
* added debug_id to ReplicatedData and crdt for debugging
2018-07-03 14:00:39 -06:00
Anatoly Yakovenko b8fe5ae076 rename server to fullnode 2018-07-02 16:34:49 -06:00
Michael Vines 3e0e09555a Undo UPnP UDP port binding 2018-07-02 14:38:01 -07:00
Michael Vines 0b56d603c2 Client NAT traversal 0.1
UPnP is now used to request a port on the NAT be forwarded to the local machine.
This obviously only works for NATs that support UPnP, and thus is not a panacea
for all NAT-related connectivity issues.

Notable hacks in this patch include a transmit/receive UDP socket pair to work
around current protocol limitations whereby the full node assumes its peer can
receive on the same UDP port it transmitted from.
2018-06-29 17:36:26 -07:00
Stephen Akridge f3df5df52c add validator catchup to multi-node test 2018-06-29 10:39:41 -07:00
Rob Walker 2f42658cd4 ... 2018-06-27 14:51:18 -07:00
Greg Fitzgerald 7aa05618a3 data_replicator -> ncp
Fixes #327
2018-06-07 17:11:17 -06:00
anatoly yakovenko 216510c573
repair socket and receiver thread (#303)
repair socket and receiver thread
2018-06-02 08:32:51 -07:00
Greg Fitzgerald c9fd5d74b5 Boot futures 0.1
We added them thinking it'd be a good stepping stone towards an
asynchronous thin client, but it's used inconsistently and where
it used, the function is still synchronous, which is just confusing.
2018-05-31 14:13:09 -06:00
Greg Fitzgerald 60609a44ba Initialize recorder from bank's last_id 2018-05-31 12:13:18 -06:00
Anatoly Yakovenko cef1c208a5 Crdt pipeline, coalesce window repair requests in the listener by examining all of them at once, and ublock those threads from doing io. 2018-05-30 14:04:48 -06:00
Greg Fitzgerald 9c9c63572b cargo fmt
rustfmt was updated with 1.26.1
2018-05-29 20:33:45 -07:00
Greg Fitzgerald fc00594ea4 Move multinode test to integration tests 2018-05-26 00:36:50 -06:00