Use bulk insertion to Postgres at startup to reduce time taken for initial snapshot restore for postgres plugin. Avoid duplicate writes of accounts at startup. Doing account plugin notification and indexing in parallel.
Improved error handling for postgres plugin to show the real db issues for debug purpose
Added more metrics for postgres plugin.
Refactored plugin centric code out to a sub module from accounts_db and added unit tests
* Move blockhash and fee_calculator in ThisInvokeContext instead of using a reference.
* Moves tx_wide_compute_cap into InvokeContext::push().
* Adds ThisInvokeContext::new_mock() constructor.
* Adds missing loader account in uses of MockInvokeContext.
* Use keyed_account_at_index() when accessing keyed_accounts.
* Makes sysvar interface consistent between ThisInvokeContext and MockInvokeContext,
in order to add InvokeContext::get_sysvars().
* Adds InvokeContext::set_blockhash() and InvokeContext ::set_fee_calculator().
* Adds new_mock_with_features.
* Makes ancestors optional in ThisInvokeContext.
* Adds prepare_mock_invoke_context() and mock_process_instruction().
- decouple cost_model from cost_tracker; allowing one cost_model
instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
* Replaces program_id field in InvokeContextStackFrame by index.
* Swaps order of program account and programdata account.
* Removes program_id parameter from InvokeContext::push().
* add filler accounts to bloat validator and predict failure
* assert no accounts match filler
* cleanup magic numbers
* panic if can't load from snapshot with filler accounts specified
* some renames
* renames
* into_par_iter
* clean filler accts, too
Now that CRDS supports incremental snapshot hashes,
SnapshotPackagerService needs to push 'em!
This commit does two main things:
1. SnapshotPackagerService now knows about incremental snapshot hashes,
and will push SnapshotPackage::IncrementalSnapshot hashes to CRDS.
2. At startup, when loading from a full + incremental snapshot, the
hashes need to be passed all the way to SnapshotPackagerService so it
can push these starting hashes to CRDS. Those values have been piped
through.
Fixes#20441 and #20423
* Adds first_instruction_account parameter to process_instruction().
* Removes InvokeContext::remove_first_keyed_account() from all BPF loaders.
* Removes InvokeContext::remove_first_keyed_account() from all builtin programs.
* Removes InvokeContext::remove_first_keyed_account() from all mock ups.
* Deprecates InvokeContext::remove_first_keyed_account().
* Documents index base of keyed_account_at_index().
* Adds dynamic offset to call sites of "keyed_account_at_index()".
* Add separate vote processing tpu port
* Add feature to send to tpu vote port
* Add vote rejecting sigverify mode
* use packet.meta.is_simple_vote_tx in place of deserialization
* consolidate code that identifies vote tx atcommon path for cpu and gpu
* new key for feature set
* banking forward tpu vote
* add tpu vote port to dockerfile and other review changes
* Simplify thread id compare
* fix a test; updated cluster_info ABI change
Co-authored-by: Tao Zhu <tao@solana.com>
Co-authored-by: sakridge <sakridge@gmail.com>
Summary of Changes
Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows
Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.
The code comprises 4 major parts:
accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
* Inlines MessageProcessor::execute_instruction() in MessageProcessor::process_message().
* Moves MessageProcessor::create_pre_accounts() into ThisInvokeContext::push().
* Move instruction_recorders slice into InvokeContext.
* Makes account_indices optional in InvokeContext::push().
* Separates initial InvokeContext::push() from ThisInvokeContext::new().
* invoke_context.pop() the base invocation frame.
* Zip message.instructions.iter() and program_indices.iter().
* Moves ThisInvokeContext::new() to the beginning of the loop inside MessageProcessor::process_message().
* Hoists ThisInvokeContext::new() out of loop inside MessageProcessor::process_message().
* Removes unnecessary clone() from ThisInvokeContext::new() in MessageProcessor ::process_message().
* Stop ignoring errors from MessageProcessor::create_pre_accounts().
* Moves MessageProcessor::verify_account_references() and MessageProcessor::verify() into InvokeContext::verify().
Brooks Prumo
Jeff Washington (jwash)
Justin Starry
sakridge
Jack May
Lijun Wang
Alexander Meißner
Tao Zhu
dependabot[bot]
Jon Cinque
Michael Vines
Trent Nelson