Justin Starry
85eb37fab0
Merge pull request from GHSA-8v47-8c53-wwrc
...
* Track transaction check time separately from account loads
* banking packet process metrics
* Remove signature clone in status cache lookup
* Reduce allocations when converting packets to transactions
* Add blake3 hash of transaction messages in status cache
* Bug fixes
* fix tests and run fmt
* Address feedback
* fix simd tx entry verification
* Fix rebase
* Feedback
* clean up
* Add tests
* Remove feature switch and fall back to signature check
* Bump programs/bpf Cargo.lock
* clippy
* nudge benches
* Bump `BankSlotDelta` frozen ABI hash`
* Add blake3 to sdk/programs/Cargo.lock
* nudge bpf tests
* short circuit status cache checks
Co-authored-by: Trent Nelson <trent@solana.com>
2021-04-13 00:28:08 -06:00
Christian Drappi
54a04bac3d
Apple M1 compatibility ( #16346 )
...
Co-authored-by: Christian Drappi <christiandrappi@Christians-MacBook-Pro.local>
2021-04-09 17:21:01 -07:00
behzad nouri
22a18a68e3
stops consuming pinned vectors with a recycler ( #16441 )
...
If the vector is pinned and has a recycler, From<PinnedVec>
implementation of Vec should clone (instead of consuming) the underlying
vector so that the next allocation of a PinnedVec will recycle an
already pinned one.
2021-04-09 16:55:24 +00:00
François Garillot
b08cff9e77
Simplify some pattern-matches ( #16402 )
...
When those match an exact combinator on Option / Result.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust ).
2021-04-08 12:40:37 -06:00
Michael Vines
43feef7362
Adjust tokio version to just "1"
2021-04-05 17:13:20 +00:00
Tyera Eulberg
b8b6777262
Only get Blockstore::last_root once ( #16362 )
2021-04-05 04:14:02 +00:00
Tyera Eulberg
1a13d22984
Fixup iterator method ( #16357 )
2021-04-04 23:32:51 +00:00
carllin
4e5ef6bce2
Add cluster state verifier logging ( #16330 )
...
* Add cluster state verifier logging
* Add duplicate-slots iterator to ledger tool
2021-04-02 21:48:44 -07:00
Tyera Eulberg
da27acabcc
Rpc: enable getConfirmedSignaturesForAddress2 to return confirmed (not yet finalized) data ( #16281 )
...
* Update blockstore method to allow return of unfinalized signature
* Support confirmed sigs in getConfirmedSignaturesForAddress2
* Add deprecated comments
* Update docs
* Enable confirmed transaction-history in cli
* Return real confirmation_status; fill in not-yet-finalized block time if possible
2021-04-01 04:35:57 +00:00
Tyera Eulberg
18bd47dbe1
Rpc: fix getConfirmedTransaction slot ( #16288 )
...
* Fix transaction blockstore apis
* Update blockstore apis in rpc
2021-03-31 21:04:00 -06:00
behzad nouri
3f63ed9a72
removes OrderedIterator and transaction batch iteration order ( #16153 )
...
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817
However load_and_execute_transactions is iterating over
lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.
If iteratorion_order is [1, 2, 0], and i is 0, then:
lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).
This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
Tyera Eulberg
433f1ead1c
Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data ( #16142 )
...
* Add Blockstore block and tx apis that allow unrooted responses
* Add TransactionStatusMessage, and send on bank freeze; also refactor TransactionStatusSender
* Track highest slot with tx-status writes complete
* Rename and unpub fn
* Add commitment to GetConfirmed input configs
* Support confirmed blocks in getConfirmedBlock
* Support confirmed txs in getConfirmedTransaction
* Update sigs-for-addr2 comment
* Enable confirmed block in cli
* Enable confirmed transaction in cli
* Review comments
* Rename blockstore method
2021-03-26 16:47:35 -06:00
Jeff Washington (jwash)
4f4cffbd03
Throttle PoH ticks by cumulative slot time ( #16139 )
...
* Throttle PoH ticks by cumulative slot time
* respond to pr feedback
* saturating sub
* updated comment
2021-03-26 18:54:16 +00:00
Justin Starry
e7fd7d46cf
rpc: add getSlotLeaders method ( #16057 )
2021-03-23 17:48:54 +00:00
behzad nouri
4f82b897bc
buffers data shreds to make larger erasure coded sets ( #15849 )
...
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.
This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
2021-03-23 14:52:38 +00:00
carllin
d76ad33597
Handle blockstore insert dup checks ( #16051 )
2021-03-22 16:18:22 -07:00
Tyera Eulberg
654449ce91
Bump tokio to 1.1 ( #15926 )
2021-03-16 21:32:50 +00:00
behzad nouri
3b85cbc504
renames is_last_in_fec_set back to is_last_data ( #15848 )
...
https://github.com/solana-labs/solana/pull/10095
renamed is_last_data to is_last_in_fec_set. However, the code shows that
this is actually meant to indicate where the serialized data is
complete:
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L599-L600
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L229-L231
There are multiple FEC sets for each `&[Entry]` serialized and this flag
does not represent shreds last in FEC sets (only the very last one by
overlap). So the name is wrong and confusing
2021-03-16 10:09:16 +00:00
Michael Vines
a2eb655322
=1.7.0
2021-03-16 07:51:07 +00:00
Michael Vines
0c9ca5522c
Bump version to v1.7.0
2021-03-13 09:01:21 +00:00
Justin Starry
918d04e3f0
Add more slot update notifications ( #15734 )
...
* Add more slot update notifications
* fix merge
* Address feedback and add integration test
* switch to datapoint
* remove unused shred method
* fix clippy
* new thread for rpc completed slots
* remove extra constant
* fixes
* rely on channel closing
* fix check
2021-03-12 21:44:06 +08:00
Trent Nelson
24d18b3cf2
docs: add docs links for crates published to crates.io
2021-03-11 08:38:18 +00:00
Jeff Washington (jwash)
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
Jeff Washington (jwash)
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
Jeff Washington (jwash)
9448540ea6
log timing after ledger process ( #15716 )
2021-03-05 11:01:52 -06:00
Tyera Eulberg
7e65289729
Convert blockstore TransactionStatus column family to protobufs ( #15733 )
...
* Prevent panic if TransactionStatus can't be deserialized
* Convert Blockstore TransactionStatus column to protobuf
* Add compatability test
2021-03-05 09:05:35 -07:00
Jeff Washington (jwash)
be35c1c1b7
add execute detail timings ( #15638 )
2021-03-03 17:07:45 -06:00
sakridge
f1223fb783
Lower blockstore processor error severity ( #15578 )
2021-03-01 14:57:37 -08:00
Trent Nelson
ef63ffce7d
Bump ed25519-dalek to 1.0.1
2021-03-01 21:10:46 +00:00
Michael Vines
33eaa2b238
Sort forks in "ledger processed..." log message
2021-02-28 17:32:50 -08:00
Michael Vines
e92cff3efa
create-snapshot subcommad now accepts the ROOT keyword
2021-02-26 16:40:10 -08:00
carllin
c2e8814dce
Add limit and shrink policy for recycler ( #15320 )
2021-02-24 00:15:58 -08:00
Michael Vines
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
Tyera Eulberg
170cb792eb
Return blockstore error if previous_blockhash cannot be determined ( #15382 )
...
* Return blockstore error if previous_blockhash cannot be determined
* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
Trent Nelson
7f7370c306
Re-allow clippy::integer_arithmetic at crate-level
2021-02-17 13:55:08 -07:00
Michael Vines
9ba69a7381
Adapt to fs_extra 1.2.0
2021-02-17 12:44:58 -08:00
Tyera Eulberg
98e3e570d2
Add --force arg for bigtable upload
2021-02-16 17:24:07 -08:00
behzad nouri
e403aeaf05
adds an inverted index to leader schedule ( #15249 )
...
next_leader_slot is doing a linear search for slots in which a pubkey is
the leader:
https://github.com/solana-labs/solana/blob/e59a24d9f/ledger/src/leader_schedule_cache.rs#L123-L157
This can be done more efficiently by adding an inverted index to leader
schedule.
2021-02-15 00:52:52 +00:00
Tyera Eulberg
0812931c38
Log if unsanitary transactions are read from blockstore ( #15319 )
2021-02-14 06:32:43 +00:00
Tyera Eulberg
da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks ( #15204 )
...
* Remove timestamp_correction feature gating
* Remove timestamp_bounding feature gating
* Remove unused deprecated ledger code
* Remove unused deprecated unbounded-timestamp code
* Enable independent adjustment of fast/slow timestamp bounding
* Update timestamp bounds to 25% fast, 80% slow; warp timestamp
* Update bank hash test
* Add PR number to feature
Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-02-09 15:49:00 -07:00
sakridge
bbae23358c
ledger-tool cleanup and additions ( #15179 )
...
* Plumb allow-dead-slots to ledger-tool verify
* ledger-tool cleanup and add some useful missing args
Print root slots and how many unrooted past last root.
2021-02-06 17:26:42 -08:00
sakridge
774416a546
Ledger purge printing ( #15176 )
2021-02-06 15:45:08 -08:00
Ryo Onodera
c69027e5aa
chore: bump serde from 1.0.118 to 1.0.122 ( #15126 )
...
* chore: bump serde from 1.0.118 to 1.0.122
* Update programs/bpf/Cargo.lock
2021-02-05 19:23:24 +09:00
Tyera Eulberg
d1563f0ccd
Bump tonic, prost, tarpc, tokio ( #15013 )
...
* Update tonic & prost, and regenerate proto
* Reignore doc code
* Revert pull #14367 , but pin tokio to v0.2 for jsonrpc
* Bump backoff and goauth -> and therefore tokio
* Bump tokio in faucet, net-utils
* Bump remaining tokio, plus tarpc
2021-02-05 00:21:53 -07:00
Michael Vines
42bf6dc2ab
Revert hard nofile limit back to 500000
2021-02-03 19:43:55 +00:00
Ryo Onodera
8376781ec8
Don't squash caught errors, please ( #15046 )
...
* Don't squash caught errors, please
* Update blockstore.rs
* Update blockstore.rs
2021-02-03 14:42:34 +00:00
Trent Nelson
894b412aef
docs: bump nofiles recommendations to match maps
2021-02-02 21:59:21 -07:00
dependabot[bot]
1df93fa2be
chore: bump serde from 1.0.112 to 1.0.118 ( #14828 )
...
* chore: bump serde from 1.0.112 to 1.0.122
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.122.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.122 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Update frozen_abi digest following serde update
* Revert "chore: bump serde from 1.0.112 to 1.0.122"
This reverts commit a3ef4442a4c985144ae2bd7ceaf8899a7ab8d7c0.
* Revert "[auto-commit] Update all Cargo lock files"
This reverts commit c41c3b005fb1ccade55155302c52cd5736c4b55f.
* chore: bump serde from 1.0.112 to 1.0.118
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.112 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.118 )
Signed-off-by: dependabot[bot] <support@github.com>
* [auto-commit] Update all Cargo lock files
* Remove serum-dex pinning
* blind commit!
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-02-02 23:28:16 +09:00
Tyera Eulberg
cbb8b79a60
Add validator flag to opt in to cpi and logs storage ( #14922 )
...
* Add validator flag to opt in to cpi and logs storage
* Default TestValidator to opt-in; allow using in multinode-demo
* No clone
Co-authored-by: Carl Lin <carl@solana.com>
2021-02-01 14:00:51 -07:00
Michael Vines
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00