Commit Graph

58 Commits

Author SHA1 Message Date
Brooks Prumo ec7e17787e
Compute accounts data len during generate_index() (#21757) 2021-12-10 13:27:59 -06:00
Justin Starry fd175c1ea9
Add StakesCache struct to abstract away locking (#21738) 2021-12-10 10:51:33 -05:00
Brooks Prumo 46fe56171b
Make StakeHistory clone-on-write (#21573) 2021-12-03 12:10:29 -06:00
Brooks Prumo 0ef1b25e4b
Make StakeDelegations clone-on-write (#21542) 2021-12-03 08:54:38 -06:00
Alexander Meißner b78f5b6032
Refactor: Cleanup InstructionProcessor (#21404)
* Moves create_message(), native_invoke() and process_cross_program_instruction()
from the InstructionProcessor to the InvokeContext so that they can have a useful "self" parameter.

* Moves InstructionProcessor into InvokeContext and Bank.

* Moves ExecuteDetailsTimings into its own file.

* Moves Executor into invoke_context.rs

* Moves PreAccount into its own file.

* impl AbiExample for BuiltinPrograms
2021-12-01 08:54:42 +01:00
Michael Vines ba9dfa0d22 Remove frozen account support 2021-11-29 08:38:11 -08:00
Jeff Washington (jwash) a8e000a2a6
add filler accounts to bloat validator and predict failure (#20491)
* add filler accounts to bloat validator and predict failure

* assert no accounts match filler

* cleanup magic numbers

* panic if can't load from snapshot with filler accounts specified

* some renames

* renames

* into_par_iter

* clean filler accts, too
2021-10-11 12:46:27 -05:00
Lijun Wang fe97cb2ddf
AccountsDb plugin framework (#20047)
Summary of Changes

Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows

Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.

The code comprises 4 major parts:

accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
Jeff Washington (jwash) 456bf15012
AccountsIndexConfig -> AccountsDbConfig (#19687) 2021-09-08 04:30:38 +00:00
Justin Starry 2d7f036afd
Add solana-program-runtime crate (#19438) 2021-08-27 00:30:36 +00:00
Brooks Prumo 234461f779
Add base_slot to get_snapshot_storages() (#19348) 2021-08-20 16:23:43 -05:00
Jeff Washington (jwash) 7c70f2158b
accounts_index_bins to AccountsIndexConfig (#19257)
* accounts_index_bins to AccountsIndexConfig

* rename param bins -> config

* rename BINS_FOR* to ACCOUNTS_INDEX_CONFIG_FOR*
2021-08-17 14:50:01 -05:00
behzad nouri 00e5e12906 renames solana_runtime::vote_account::VoteAccount
Rename:
  VoteAccount    -> VoteAccountInner  # the private type
  ArcVoteAccount -> VoteAccount       # the public type
2021-08-10 22:54:17 +00:00
Jeff Washington (jwash) 651343688d
Use option for account index bucket parameter (#19150) 2021-08-10 11:32:25 -05:00
Jeff Washington (jwash) 47e0d9aa95
plumb more accounts_index bins (#19123) 2021-08-10 05:45:46 -05:00
Jeff Washington (jwash) c7b986220e
Accounts::new_with_config_for_tests (#19104) 2021-08-09 09:01:12 -05:00
Jeff Washington (jwash) 14361906ca
for all tests, bank::new -> bank::new_for_tests (#19064) 2021-08-05 08:42:38 -05:00
Brooks Prumo d1debcd971
Add incremental snapshot utils (#18504)
This commit adds high-level functions for creating and loading-from
incremental snapshots, plus all low-level functions required to perform
those tasks.  This commit **does not** add taking incremental snapshots
as part of a running validator, nor starting up a node with an
incremental snapshot; just laying ground work.

Additionally, `snapshot_utils` and `serde_snapshot` have been
refactored to use a common code paths for the different snapshots.

Also of note, some renaming has happened:
  1. Snapshots are now either `full_` or `incremental_` throughout the
     codebase.  If not specified, the code applies to both.
  2. Bank snapshots now are called "bank snapshots"
     (before they were called "slot snapshots", "bank snapshots", or
      just "snapshots").  The one exception is within `Bank`, where they
     are still just "snapshots", because they are already "bank
     snapshots".
  3. Snapshot archives now have `_archive` in the code.  This
     should clear up an ambiguity between bank snapshots and snapshot
     archives.
2021-07-22 14:40:37 -05:00
Justin Starry 207c90bd8b
Shorten long SerializeWith type paths in abi digest (#18734) 2021-07-20 08:59:50 -05:00
Jeff Washington (jwash) d092fa1f03
add ledger-tool verify verify-accounts-index option (#18375)
* add ledger-tool verify verify-accounts-index option

* comment, merge, respond to feedback, cleanup
2021-07-13 11:06:18 -05:00
Alexander Meißner 6514096a67 chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
Jon Cinque 1b1d34da59
Refactor stake program into solana_program (#17906)
* Move stake state / instructions into solana_program

* Update account-decoder

* Update cli and runtime

* Update all other parts

* Commit Cargo.lock changes in programs/bpf

* Update cli stake instruction import

* Allow integer arithmetic

* Update ABI digest

* Bump rust mem instruction count

* Remove useless structs

* Move stake::id() -> stake::program::id()

* Re-export from solana_sdk and mark deprecated

* Address feedback

* Run cargo fmt
2021-06-15 18:04:00 +02:00
Lijun Wang 269d995832
Make account shrink configurable #17544 (#17778)
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true

Fixes #17544
2021-06-09 21:21:32 -07:00
Jeff Washington (jwash) 738cc9549f
parallel get_snapshot_storages (#17589) 2021-06-01 19:33:12 -05:00
Jeff Washington (jwash) c8fdf0762c
add Ancestors to get_snapshot_storages to prepare for the future (#17591) 2021-06-01 15:14:35 -05:00
Jeff Washington (jwash) 6b9d8d41a3
add --limit_load_slot_count_from_snapshot to ledger-tool (#17417) 2021-05-26 10:36:12 -05:00
Jeff Washington (jwash) 9d6837c904
fix comment (#17286) 2021-05-17 14:06:25 -05:00
Jeff Washington (jwash) f39dda00e0
type AccountSecondaryIndexes = HashSet (#17108) 2021-05-10 14:22:48 +00:00
Jeff Washington (jwash) 1a9954f85b
bank deposit checked_add error (#16917)
* bank deposit checked_add error

* add id

* rename variables

* rename error and metric
2021-04-30 16:22:17 -05:00
Jeff Washington (jwash) 5591cd6ef3
.owner -> .owner() (#16757)
* .owner -> .get_owner()

* use
2021-04-23 09:35:09 -05:00
carllin d747614b27
Account for possibility of cache flush in load() (#15454)
* Account for possibility of cache flush in load()

* More cleaning

* More cleaning

* Remove unused method and some comment cleaning

* Fix typo

* Make the detected impossible purge race panic()!

* Finally revert to original .expect()

* Fix typos...

* Add assertion for max_root for easier reasoning

* Reframe races with LoadHint as possible opt.

* Fix test

* Make race bug tests run longer for less flaky

* Delay the clone-in-lock slow path even for RPC

* Make get_account panic-free & add its onchain ver.

* Fix rebase conflicts...

* Clean up

* Clean up comment

* Revert fn name change

* Fix flaky test...

* fmt...

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-04-17 00:23:32 +09:00
Michael Vines 1061d021c9 Unpack snapshot AppendVecs directly into account paths 2021-03-11 08:25:24 -08:00
Michael Vines 65dd177601 AppendVec cleanup: remove dead code, rename non-idiomatic "new" function 2021-03-10 14:50:59 -08:00
Jeff Washington (jwash) 8a3135d17b
Account->AccountSharedData (#15691) 2021-03-09 15:06:07 -06:00
Michael Vines 5df36aec7d Pacify clippy 2021-02-19 20:08:41 -08:00
dependabot[bot] 1df93fa2be
chore: bump serde from 1.0.112 to 1.0.118 (#14828)
* chore: bump serde from 1.0.112 to 1.0.122

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.112 to 1.0.122.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.122)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Update frozen_abi digest following serde update

* Revert "chore: bump serde from 1.0.112 to 1.0.122"

This reverts commit a3ef4442a4c985144ae2bd7ceaf8899a7ab8d7c0.

* Revert "[auto-commit] Update all Cargo lock files"

This reverts commit c41c3b005fb1ccade55155302c52cd5736c4b55f.

* chore: bump serde from 1.0.112 to 1.0.118

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.112 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.112...v1.0.118)

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Remove serum-dex pinning

* blind commit!

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-02-02 23:28:16 +09:00
carllin 6dfad0652f
Cache account stores, flush from AccountsBackgroundService (#13140) 2021-01-11 17:00:23 -08:00
carllin 5affd8aa72
Add secondary indexes (#14212) 2020-12-31 18:06:03 -08:00
behzad nouri d6d76219b6
caches staked nodes computed from vote-accounts (#13929) 2020-12-17 21:22:50 +00:00
Michael Vines 7143aaa89b Clippy 2020-12-14 08:03:29 -08:00
behzad nouri e1793e5a13
caches vote-state de-serialized from vote accounts (#13795)
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.

This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.

Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.
2020-11-30 17:18:33 +00:00
sakridge 43053dcc90
Re-use accounts stores (#12885)
* Re-use accounts_db stores

Creating files and dropping mmap areas can be expensive

* Add test for storage finder

Can encounter an infinite loop when the store is too small, but
smaller than the normal store size.

* Fix storage finding

* Check for strong_count == 1

* try_recycle helper
2020-11-04 09:17:05 -08:00
Jack May c458d4b213
move Account to solana-sdk (#13198) 2020-10-28 22:01:07 -07:00
Michael Vines a4956844bd Update frozen_abi hashes
The movement of files in sdk/ caused ABI hashes to change
2020-10-24 08:37:55 -07:00
Michael Vines 6858950f76 Remove frozen ABI modules from solana-sdk 2020-10-20 16:11:30 -07:00
carllin c879e7c1ad
Fix fee mismatch on snapshot deserialize (#12697)
Co-authored-by: Carl Lin <carl@solana.com>
2020-10-08 23:44:41 -07:00
Michael Vines 31696a1d72 Port BPFLoader2 activation to FeatureSet and rework built-in program activation 2020-09-28 12:50:19 -07:00
Michael Vines aa5c008fa8 Add pico-inflation feature 2020-09-28 08:25:00 -07:00
sakridge 68e5a2ef56
Add RPC notify and banking keys debug (#12396) 2020-09-23 18:46:42 -07:00
Ryo Onodera 53b8ea4464
Rename to ClusterType and restore devnet compat. (#12068)
* Rename to ClusterType and restore devnet compat.

* De-duplicate parse code and add comments

* Adjust default Devnet genesis & reduce it in tests
2020-09-08 23:55:09 +09:00