Brennan Watt
a2e7bdf50a
Rust v1.63.0 ( #27148 )
...
* Upgrade to Rust v1.63.0
* Add nightly_clippy_allows
* Resolve some new clippy nightly lints
* Increase QUIC packets completion timeout
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-08-17 15:48:33 -07:00
Jeff Washington (jwash)
225cddcffb
serialize incremental_snapshot_hash ( #26839 )
...
* serialize incremental_snapshot_hash
* pr feedback
2022-08-17 15:14:31 -05:00
Justin Starry
69598ed4c0
Refactor: Add `RuntimeConfig` field to Bank ( #26946 )
...
* Refactor: Simplify arguments for bank constructor methods
* Refactor: Add RuntimeConfig to Bank fields
* Arc wrap runtime_config
* Arc wrap all runtime config usages
* Remove Copy trait derivation from RuntimeConfig
* Remove some arc wrapping
2022-08-05 20:49:00 +01:00
apfitzge
a7e4df5d6e
Separate remapping append_vecs into functions ( #26696 )
...
Separate remapping append_vecs into remap_and_reconstruct_storages and helpers
2022-07-25 14:55:30 -05:00
apfitzge
f2b81fe665
Separate field_from_stream to get bank/account fields from individual snapshot stream ( #26679 )
2022-07-21 12:27:31 -05:00
Jeff Washington (jwash)
3e9bd6170c
use OnceCell instead of RwLock for rent_paying_accounts_by_partition ( #26601 )
2022-07-14 08:52:41 -05:00
Jeff Washington (jwash)
6b0eb5a42b
handle rent paying accounts differently ( #26487 )
...
* handle rent paying accounts differently
* restore collection
* downgrade assert to metric and warn until we get more runtime
2022-07-12 17:51:33 -05:00
carllin
90ef2cd02a
Parse snapshot for bank fields ( #26016 )
2022-07-06 17:30:30 -05:00
Jeff Washington (jwash)
17a99d98dd
Revert "avoid adding to 'uncleaned_roots' when generating index and c… ( #26441 )
...
Revert "avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink (#25936 )"
This reverts commit e24cc537a4
.
2022-07-06 11:32:45 -05:00
Brooks Prumo
53b9420562
Remove `INTO!` macros in serde_snapshot ( #26409 )
2022-07-05 16:37:20 -05:00
Jeff Washington (jwash)
e24cc537a4
avoid adding to 'uncleaned_roots' when generating index and caller passes accounts-db-skip-shrink ( #25936 )
2022-06-14 10:10:44 -05:00
Ashwin Sekar
8caced68ce
Serialize lamports per signature ( #25364 )
...
* Serialize lamports per signature
* Add full snapshot archive test, enable features in previous tests
2022-06-06 13:07:02 -07:00
Jeff Washington (jwash)
dd0bf4ff4b
add filler accounts gradually with cache flush ( #25678 )
2022-06-01 15:47:21 -05:00
Jeff Washington (jwash)
de03a3d3aa
filler accounts get a non-zero rent_epoch ( #25386 )
...
* fill accounts get a non-zero rent_epoch
* use snapshot_slot
* get rid of perf simulation code
2022-05-20 09:03:55 -05:00
Giorgio Gambino
60b2155bd3
Add accounts-filler-size command line option ( #23896 )
2022-04-11 13:10:09 -05:00
Jeff Washington (jwash)
c27150b1a3
reserialize_bank_fields_with_hash ( #23916 )
...
* reserialize_bank_with_new_accounts_hash
* Update runtime/src/serde_snapshot.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/serde_snapshot/tests.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-07 14:05:57 -05:00
Jeff Washington (jwash)
550ca7bf92
compare contents of serialized banks instead of exact file format ( #24141 )
...
* compare contents of serialized banks instead of exact file format
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* Update runtime/src/snapshot_utils.rs
Co-authored-by: Brooks Prumo <brooks@prumo.org>
* pr feedback
* get rid of clone
* pr feedback
Co-authored-by: Brooks Prumo <brooks@prumo.org>
2022-04-06 21:55:44 -05:00
Jeff Washington (jwash)
fddd162645
reserialize bank in ahv by first writing to temp file in abs ( #23947 )
2022-04-06 21:39:26 -05:00
Jeff Washington (jwash)
6a7f6585ce
persist historical_roots ( #24029 )
2022-04-04 13:13:11 -05:00
Jeff Washington (jwash)
0ca5a0ec68
prior_roots -> historical_roots ( #24064 )
2022-04-02 12:01:43 -05:00
Jeff Washington (jwash)
396b49a7c1
Start saving/loading prior_roots(_with_hash) to snapshot ( #23844 )
...
* Start saving/loading prior_roots(_with_hash) to snapshot
* Update runtime/src/accounts_index.rs
Co-authored-by: Michael Vines <mvines@gmail.com>
* Update runtime/src/accounts_index.rs
Co-authored-by: Michael Vines <mvines@gmail.com>
* update comment
Co-authored-by: Michael Vines <mvines@gmail.com>
2022-03-24 10:06:24 -05:00
Jeff Washington (jwash)
cc4d75a16f
fix typo ( #23535 )
2022-03-08 18:28:00 -06:00
Brooks Prumo
fb0e71946f
Cleanup use declaration ( #22875 )
2022-02-01 23:34:11 +00:00
Brooks Prumo
22f6db2e4f
Cleanup serde snapshot common.rs ( #22854 )
2022-02-01 05:39:53 -06:00
Jeff Washington (jwash)
d3b95b9226
Cleanup AccountStorage struct ( #22463 )
...
* Revert "chore: bump dashmap from 4.0.2 to 5.0.0 (#21824 )"
This reverts commit 8aa3d690b5
.
* Cleanup AccountStorage struct
2022-01-13 17:51:53 -06:00
Brooks Prumo
2756abce39
More serde snapshot cleanup ( #22449 )
2022-01-13 09:20:20 -06:00
Brooks Prumo
9c3144e286
Refactor serde snapshot's "future" to "newer" ( #22431 )
2022-01-13 07:12:09 -06:00
Jeff Washington (jwash)
fb62407232
AcctIdx: appendvecid: u32 ( #21842 )
2022-01-03 10:35:17 -06:00
Jeff Washington (jwash)
af53d2f692
simplify api on reconstruct_single_storage ( #21970 )
2021-12-17 09:06:23 -06:00
Jeff Washington (jwash)
4c0373b72a
introduce AtomicAppendVecId ( #21793 )
2021-12-11 21:34:35 -06:00
Brooks Prumo
eeb97fe7ce
Add accounts_data_len to Bank ( #21781 )
2021-12-11 09:34:38 -06:00
Brooks Prumo
ec7e17787e
Compute accounts data len during generate_index() ( #21757 )
2021-12-10 13:27:59 -06:00
Brooks Prumo
1528f85112
Add return types to generate_index() ( #21735 )
2021-12-09 17:15:13 -06:00
Michael Vines
b8837c04ec
Reformat imports to a consistent style for imports
...
rustfmt.toml configuration:
imports_granularity = "One"
group_imports = "One"
2021-12-03 09:19:13 -08:00
Alexander Meißner
b78f5b6032
Refactor: Cleanup InstructionProcessor ( #21404 )
...
* Moves create_message(), native_invoke() and process_cross_program_instruction()
from the InstructionProcessor to the InvokeContext so that they can have a useful "self" parameter.
* Moves InstructionProcessor into InvokeContext and Bank.
* Moves ExecuteDetailsTimings into its own file.
* Moves Executor into invoke_context.rs
* Moves PreAccount into its own file.
* impl AbiExample for BuiltinPrograms
2021-12-01 08:54:42 +01:00
Michael Vines
ba9dfa0d22
Remove frozen account support
2021-11-29 08:38:11 -08:00
Brooks Prumo
c24e30f689
Constrain memory ordering on AccountsDb::next_id ( #21311 )
2021-11-18 20:34:37 -06:00
Brooks Prumo
c3e5927d16
Constrain memory ordering on AccountsDb::write_version ( #21312 )
2021-11-17 07:11:16 -06:00
Alexander Meißner
7200c5106e
Replaces MockInvokeContext by ThisInvokeContext in tests ( #20881 )
...
* Replaces MockInvokeContext by ThisInvokeContext in BpfLoader, SystemInstructionProcessor, CLIs, ConfigProcessor, StakeProcessor and VoteProcessor.
* Finally, removes MockInvokeContext, MockComputeMeter and MockLogger.
* Adjusts assert_instruction_count test.
* Moves ThisInvokeContext to the program-runtime crate.
2021-11-04 21:47:32 +01:00
Jeff Washington (jwash)
b1e3a82c11
refactor rent collector and logs ( #21155 )
2021-11-04 06:28:41 -05:00
Jeff Washington (jwash)
c59c56c0f8
log rent exempt accounts ( #21137 )
2021-11-03 14:44:52 -05:00
Jeff Washington (jwash)
9d330fc638
FillerAccts: use variable cycle partitions ( #20963 )
2021-10-27 11:18:27 -05:00
Lijun Wang
f14365f4b4
Accountsdb plugin postgres -- bulk insertion at startup ( #20763 )
...
Use bulk insertion to Postgres at startup to reduce time taken for initial snapshot restore for postgres plugin. Avoid duplicate writes of accounts at startup. Doing account plugin notification and indexing in parallel.
Improved error handling for postgres plugin to show the real db issues for debug purpose
Added more metrics for postgres plugin.
Refactored plugin centric code out to a sub module from accounts_db and added unit tests
2021-10-24 12:43:33 -07:00
Jeff Washington (jwash)
a8e000a2a6
add filler accounts to bloat validator and predict failure ( #20491 )
...
* add filler accounts to bloat validator and predict failure
* assert no accounts match filler
* cleanup magic numbers
* panic if can't load from snapshot with filler accounts specified
* some renames
* renames
* into_par_iter
* clean filler accts, too
2021-10-11 12:46:27 -05:00
Lijun Wang
fe97cb2ddf
AccountsDb plugin framework ( #20047 )
...
Summary of Changes
Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows
Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.
The code comprises 4 major parts:
accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
Brooks Prumo
1347b507da
Handle duplicate AppendVec IDs ( #20096 )
...
When reconstructing the AccountsDb, if the storages came from full and
incremental snapshots generated on different nodes, it's possible that
the AppendVec IDs could overlap/have duplicates, which would cause the
reconstruction to fail.
This commit handles this issue by unconditionally remapping the
AppendVec ID for every AppendVec.
Fixes #17088
2021-09-23 17:34:32 -05:00
Jeff Washington (jwash)
456bf15012
AccountsIndexConfig -> AccountsDbConfig ( #19687 )
2021-09-08 04:30:38 +00:00
Jack May
170927b7c4
nit: Move builtins types to builtins file ( #19597 )
2021-09-02 21:29:11 -07:00
Justin Starry
2d7f036afd
Add solana-program-runtime crate ( #19438 )
2021-08-27 00:30:36 +00:00
Jeff Washington (jwash)
7c70f2158b
accounts_index_bins to AccountsIndexConfig ( #19257 )
...
* accounts_index_bins to AccountsIndexConfig
* rename param bins -> config
* rename BINS_FOR* to ACCOUNTS_INDEX_CONFIG_FOR*
2021-08-17 14:50:01 -05:00