Commit Graph

2588 Commits

Author SHA1 Message Date
dependabot[bot] 157f165a3d
Bump memmap2 from 0.5.1 to 0.5.2 (#22414)
* Bump memmap2 from 0.5.1 to 0.5.2

Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases)
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.5.1...v0.5.2)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-11 15:08:13 -07:00
Tyera Eulberg 637e366b18
Prevent rent-paying account creation (#22292)
* Fixup typo

* Add new feature

* Add new TransactionError

* Add framework for checking account state before and after transaction processing

* Fail transactions that leave new rent-paying accounts

* Only check rent-state of writable tx accounts

* Review comments: combine process_result success behavior; log and metrics before feature activation

* Fix tests that assume rent-exempt accounts are okay

* Remove test no longer relevant

* Remove native/sysvar special case

* Move metrics submission to report legacy->legacy rent paying transitions as well
2022-01-11 11:32:25 -07:00
behzad nouri a49ef49f87
optimizes ReadOnlyAccountsCache LRU eviction implementation (#22403)
ReadOnlyAccountsCache is using a background thread, table scan and sort
to implement LRU eviction policy:
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L66-L73
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L186-L191
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L222

DashMap internally locks each shard when accessed; so a table scan in
the background thread can create a lot of lock contention.

This commit adds an index-list queue containing cached keys in the order
that they are accessed. Each hash-map entry also includes its index into
this queue.
When an item is first entered into the cache, it is added to the end of
the queue. Also each time an entry is looked up from the cache it is
moved to the end of queue. As a result, items in the queue are always
sorted in the order that they have last been accessed. When doing LRU
eviction, cache entries are evicted from the front of the queue.
Using index-list, all queue operations above are O(1) with low overhead
and so above achieves an efficient implementation of LRU cache eviction
policy.
2022-01-11 17:25:28 +00:00
Jeff Biseda 8b66625c95
convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
dependabot[bot] 04a2c19c87
Bump memmap2 from 0.5.0 to 0.5.1 (#22409)
* Bump memmap2 from 0.5.0 to 0.5.1

Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.5.0 to 0.5.1.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases)
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.5.0...v0.5.1)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-10 11:14:56 -07:00
Brooks Prumo 9bc2592da1
Use lazy_rent_collection directly (#22410) 2022-01-10 10:47:08 -06:00
dependabot[bot] 28275a33d6
Bump tempfile from 3.2.0 to 3.3.0 (#22401)
* Bump tempfile from 3.2.0 to 3.3.0

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases)
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.2.0...v3.3.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-09 12:26:08 -07:00
Trent Nelson 3b4aad9df1 bank: fix executor cache metrics 2022-01-09 04:22:39 +00:00
dependabot[bot] eaa52bc935
Bump zstd from 0.9.1+zstd.1.5.1 to 0.9.2+zstd.1.5.1 (#22388)
* Bump zstd from 0.9.1+zstd.1.5.1 to 0.9.2+zstd.1.5.1

Bumps [zstd](https://github.com/gyscos/zstd-rs) from 0.9.1+zstd.1.5.1 to 0.9.2+zstd.1.5.1.
- [Release notes](https://github.com/gyscos/zstd-rs/releases)
- [Commits](https://github.com/gyscos/zstd-rs/compare/0.9.1...v0.9.2)

---
updated-dependencies:
- dependency-name: zstd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-08 22:21:56 +00:00
dependabot[bot] d8ec8ca2e6
Bump blake3 from 1.2.0 to 1.3.0 (#22373)
* Bump blake3 from 1.2.0 to 1.3.0

Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 1.2.0 to 1.3.0.
- [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases)
- [Commits](https://github.com/BLAKE3-team/BLAKE3/compare/1.2.0...1.3.0)

---
updated-dependencies:
- dependency-name: blake3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-08 20:28:04 +00:00
Brooks Prumo d90d5ee9b6
Refactor Rent::due() with RentDue enum (#22346) 2022-01-08 09:03:46 -06:00
Jack May 4a9f4e2505 improve multi executor cache addition 2022-01-08 03:47:23 -07:00
Trent Nelson ad3cb0bc93 --amend 2022-01-08 08:19:27 +00:00
Trent Nelson 4ce48307bb bank: prime new executor cache entry use-counts 2022-01-08 08:19:27 +00:00
Trent Nelson 6d76db1de5 bank: Add executors cache metrics 2022-01-07 16:53:27 -07:00
behzad nouri c2389fc209
removes CowCachedExecutors (#22343)
Copy-on-write semantics for cached executors can be implemented by a
simple Arc<CachedExecutors> as opposed to CowCachedExecutors:
https://github.com/solana-labs/solana/blob/f1e2598ba/runtime/src/bank.rs#L244-L247

This will also avoid the need for double locking as in:
https://github.com/solana-labs/solana/blob/f1e2598ba/runtime/src/bank.rs#L3490-L3491
https://github.com/solana-labs/solana/blob/f1e2598ba/runtime/src/bank.rs#L3525-L3526
2022-01-07 14:06:29 +00:00
Justin Starry 52d12cc802
Add runtime support for address table lookups (#22223)
* Add support for address table lookups in runtime

* feedback

* feedback
2022-01-07 11:59:09 +08:00
Jack May f1e2598baa
Retain executor cache counts (#22322) 2022-01-06 08:56:00 -08:00
Jeff Washington (jwash) e0c091a9f4
factor out code in do_shrink_slot_stores (#22306) 2022-01-06 10:49:24 -06:00
Jeff Washington (jwash) 100293c4b5
refactor get_store_for_shrink (#22307) 2022-01-06 10:49:01 -06:00
Trent Nelson 390ef0fbcd Consolidate process instruction execution timings to own struct 2022-01-06 03:56:46 -07:00
Trent Nelson 72fc6096a0 Use saturating_add_assign macro 2022-01-06 03:56:46 -07:00
Carl Lin b25e4a200b Add execute metrics 2022-01-06 03:56:46 -07:00
Trent Nelson 7d32909e17 move `ExecuteTimings` from `runtime::bank` to `program_runtime::timings` 2022-01-06 03:56:46 -07:00
Jack May 12e160269e
cache executors on failed transactions (#22308) 2022-01-05 22:09:03 -08:00
Brooks Prumo ab13e39518
Only sum accounts data len from non-zero lamport accounts (#22309) 2022-01-05 23:23:37 +00:00
Justin Starry f2ed6f09ee Skip updating already cached executors if unmodified 2022-01-05 16:08:10 -07:00
Michael Vines bb3a1b6b31 Add zk_token_sdk_enabled feature to gate Zk Token proof program and `sol_zk_token_elgamal_op` syscalls 2022-01-05 11:57:37 -08:00
Alexander Meißner 9f63493789
Refactor: Remove KeyedAccounts (2) (#22274)
* Adds InstructionContext::get_signers().
Improves error messages when modifying borrowed accounts.

* Removes keyed_accounts from InvokeContext tests.

* Removes keyed_accounts from message_processor.rs

* Removes keyed_accounts from bank.rs

* Removes keyed_accounts from bpf serialization.
2022-01-05 09:39:37 +01:00
Justin Starry 45458e7139
Refactor: Improve type safety and readability of transaction execution (#22215)
* Refactor Bank::load_and_execute_transactions

* Refactor: improve type safety of TransactionExecutionResult

* Add enum for extra type safety in execution results

* feedback
2022-01-05 10:15:15 +08:00
Brooks Prumo 1460f00e0f
Consume from AccountsDataMeter (#21994) 2022-01-04 10:00:21 -06:00
Justin Starry 2b5e00d36d
Limit number of accounts that a transaction can lock (#22201) 2022-01-04 14:25:23 +08:00
Alexander Meißner 73e6038986
Refactor: Remove `KeyedAccount` from program runtime (#22226)
* Makes error handling in BorrowedAccount optional.
Adds BorrowedAccount ::get_rent_epoch().
Exposes InstructionContext::get_index_in_transaction().
Turns accounts and account_keys into pinned boxed slices.

* Introduces "unsafe" to InvokeContext::push().

* Turns &TransactionContext into &mut TransactionContext in InvokeContext.

* Push and pop InstructionContext in InvokeContext.
Makes test_process_cross_program and test_native_invoke symmetric.
Removes the borrow check from test_invoke_context_verify.

* Removes keyed_accounts from prepare_instruction()

* Removes usage of invoke_stack.

* Removes keyed_accounts from program-test.

* Removes caller_write_privileges.

* Removes keyed_accounts from BPF parameter (de-)serialization.
2022-01-03 23:30:56 +01:00
dependabot[bot] 672fed04cb
Bump serde from 1.0.132 to 1.0.133 (#22233)
* Bump serde from 1.0.132 to 1.0.133

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.132 to 1.0.133.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.132...v1.0.133)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2022-01-03 22:19:39 +00:00
Brooks Prumo 04b76eb066
Use AtomicUsize for next_bucket_to_flush (#22095) 2022-01-03 10:35:35 -06:00
Jeff Washington (jwash) fb62407232
AcctIdx: appendvecid: u32 (#21842) 2022-01-03 10:35:17 -06:00
Tao Zhu a2a7e91ad6 set secp256k1 cost similar to sigverify 2021-12-31 17:45:50 -06:00
Tao Zhu d743c2917c re-calibrate limit based on mainnet data, see issue #21917 2021-12-31 17:45:50 -06:00
Alexander Meißner e529d03c11
Cleanup #22182 (#22205)
* Turns compute_units_consumed of ProcessInstructionResult into a &mut parameter.

* Removes second nesting level from test_process_instruction_compute_budget().

* Makes test_process_cross_program and test_native_invoke symmetric.

* Unifies test_process_cross_program(), test_native_invoke() and test_process_instruction_compute_budget() into test_process_instruction().
2021-12-31 17:55:27 +01:00
carllin d06e6c7425
Count compute units even when transaction errors (#22182) 2021-12-30 21:21:42 -05:00
Alexander Meißner edb20d6909
Splits index of InstructionAccount into index_in_transaction and index_in_caller. (#22165) 2021-12-30 15:46:36 +01:00
carllin 33d0b5e011
Revert "Count compute units even when transaction errors (#22059)" (#22174)
This reverts commit eaa8c67bde.
2021-12-30 02:42:32 -05:00
Justin Starry b1d9a2e60e
Don't forward packets received from TPU forwards port (#22078)
* Don't forward packets received from TPU forwards port

* Add banking stage test
2021-12-29 19:34:31 +01:00
carllin eaa8c67bde
Count compute units even when transaction errors (#22059) 2021-12-28 17:05:11 -05:00
Tyera Eulberg 422a095647
Add (preflight) simulation to BanksClient (#22084)
* Add more-legitimate conversion from legacy Transaction to SanitizedTransaction

* Add Banks method with preflight checks

* Expose BanksClient method with preflight checks

* Unwrap simulation err

* Add Bank simulation method that works on unfrozen Banks

* Add simpler api

* Better name: BanksTransactionResultWithSimulation
2021-12-28 19:25:46 +00:00
Brooks Prumo e3b20c443a
Use load_accounts_data_len() instead of Atomic .load() (#22144) 2021-12-28 13:38:20 +00:00
Brooks Prumo 800472ddf5
Add AccountsDataMeter to InvokeContext (#21813) 2021-12-28 05:14:48 -06:00
Alexander Meißner a06646631c
Feature: TransactionContext, InstructionContext and BorrowedAccount (#21706)
* Adds TransactionContext, InstructionContext and BorrowedAccount.

* Redirects the usage of accounts in InvokeContext through TransactionContext.
Also use the types declared in transaction_context.rs everywhere.

* Adjusts all affected tests.
2021-12-27 18:49:32 +01:00
dependabot[bot] bb97c8fdcd
Bump zstd from 0.9.0+zstd.1.5.0 to 0.9.1+zstd.1.5.1 (#22105)
* Bump zstd from 0.9.0+zstd.1.5.0 to 0.9.1+zstd.1.5.1

Bumps [zstd](https://github.com/gyscos/zstd-rs) from 0.9.0+zstd.1.5.0 to 0.9.1+zstd.1.5.1.
- [Release notes](https://github.com/gyscos/zstd-rs/releases)
- [Commits](https://github.com/gyscos/zstd-rs/compare/0.9.0...0.9.1)

---
updated-dependencies:
- dependency-name: zstd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-25 21:26:55 -07:00
Alexander Meißner cc947cad03
Refactor: CPI Instruction Recording (#22111)
* Unifies all InstructionRecorders of a transaction into one.

* Stops explicitly compiling CPI instructions for recording,
uses the indices gathered from instruction_accounts instead.
2021-12-25 13:35:43 +01:00
Alexander Meißner 2ab4f34c02
Refactor: Remove `Message` and `CompiledInstruction` from `InvokeContext` interfaces (#22102)
* Introduces InstructionAccount which is like AccountMeta but uses an index instead of a Pubkey

* Renames InvokeContext::create_message() to InvokeContext::prepare_instruction()

* Removes Message and CompiledInstruction from InvokeContext interfaces.

* Resolves TODOs of sol_invoke_signed() in program-test.

* Moves CompiledInstruction::visit_each_account() into invoke_context.rs
2021-12-24 16:17:55 +01:00
behzad nouri f67ecd5c18
removes unused Cargo dependencies (#22022)
Also moving some to [dev-dependencies] so that they are not propagated
to other packages which depend on the package.
2021-12-22 17:01:33 +00:00
Jeff Washington (jwash) 61cc7b10a9
AcctIdx: respect disk idx mem size param (#22050) 2021-12-22 09:54:05 -06:00
Jeff Washington (jwash) bdae2993e0
AcctIdx: hold ranges in memory uses multiple threads (#22031) 2021-12-21 17:31:48 -06:00
dependabot[bot] 41ec7c8be9
chore: bump num_cpus from 1.13.0 to 1.13.1 (#22044)
* chore: bump num_cpus from 1.13.0 to 1.13.1

Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.13.0 to 1.13.1.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.13.0...v1.13.1)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-21 00:06:14 -07:00
dependabot[bot] 8f547a6c98
chore: bump serde from 1.0.131 to 1.0.132 (#21989)
* chore: bump serde from 1.0.131 to 1.0.132

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.131 to 1.0.132.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.131...v1.0.132)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-18 07:45:20 +00:00
segfaultdoctor 76098dd42a
RPC Block Subscription (#21787)
* add stuff

* compiling

* add notify block

* wip

* feat: add blockSubscribe pubsub method

* address PR comments

Co-authored-by: Lucas B <buffalu@jito.network>
Co-authored-by: Zano <segfaultdoctor@protonmail.com>
2021-12-17 16:03:09 -07:00
mooori 0f6e8d3385
Check file size of `snapshot_version` when unarchiving snapshot (#21925) 2021-12-17 12:27:54 -06:00
Jeff Washington (jwash) 8ed7ad5fa7
AcctIdx: hold range recognizes already held ranges (#21937) 2021-12-17 10:04:41 -06:00
Jeff Washington (jwash) 729698e815
AcctIdx: items() uses held ranges (#21954) 2021-12-17 09:59:29 -06:00
Jeff Washington (jwash) af53d2f692
simplify api on reconstruct_single_storage (#21970) 2021-12-17 09:06:23 -06:00
Alexander Meißner 66fa8f9667
Refactor: Removes `Rc` from `Refcell<AccountSharedData>` in the program-runtime (#21927)
* Removes Rc from Rc<RefCell<AccountSharedData>> in the program-runtime.

* Adjusts tests in bpf_loader, system_instruction_processor, config_processor, vote_instruction and stake_instruction
2021-12-17 14:01:12 +01:00
Jeff Washington (jwash) c4389a6675
AcctIdx: held ranges search in lifo order (#21964) 2021-12-16 23:25:22 -06:00
Jeff Washington (jwash) 5d40da5688
log "next_id" to track append vec ids (#21971) 2021-12-16 22:41:34 -06:00
Jeff Washington (jwash) 6374995522
AcctIdx: share bucket map size for perf (#21935) 2021-12-16 21:25:54 -06:00
Jeff Washington (jwash) ba777f4f56
AcctIdx: remove Option from held ranges (#21958) 2021-12-16 21:22:04 -06:00
Jeff Washington (jwash) e11a1911ad
load_accounts_index_for_shrink ignores cached entries (#21951) 2021-12-16 16:37:08 -06:00
Justin Starry 6ff0be6a82
Clean up demote program write lock feature (#21949)
* Clean up demote program write lock feature

* fix test
2021-12-16 17:27:22 -05:00
dependabot[bot] a5769c029f
chore: bump tar from 0.4.37 to 0.4.38 (#21921)
* chore: bump tar from 0.4.37 to 0.4.38

Bumps [tar](https://github.com/alexcrichton/tar-rs) from 0.4.37 to 0.4.38.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases)
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.37...0.4.38)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-16 15:06:56 -07:00
Jeff Washington (jwash) 347323cbb2
use bg thread pool for shrink (#21950) 2021-12-16 15:54:38 -06:00
Jeff Washington (jwash) e97da0ea15
AcctIdx: define type for serialized AppendVecId (#21938) 2021-12-16 08:41:01 -06:00
Jeff Washington (jwash) 18417e410e
AcctIdx: remove troublesome assert (#21947) 2021-12-16 08:37:05 -06:00
Alexander Meißner 49cb161203
Fixes the calculation of the "compute_meter_consumption" across process_instruction() and process_message(). (#21944) 2021-12-16 15:15:58 +01:00
Jeff Washington (jwash) 82672b40fd
AcctIdx: streamline metric update (#21936) 2021-12-15 19:52:23 -06:00
Jeff Washington (jwash) 1e0d3f13e6
AcctIdx: fix metrics bug (#21934) 2021-12-15 17:05:38 -06:00
Brooks Prumo 635337d2ff
Bank gets accounts data len delta from MessageProcessor::process_message() 2021-12-15 16:41:38 -06:00
Jeff Washington (jwash) 46e5350d8c
AcctInfo: store offset in AccountInfo as u32 (#21895) 2021-12-15 15:41:11 -06:00
Jeff Washington (jwash) e5be96d8bf
AcctIdx: consolidate next_id calls (#21929) 2021-12-15 15:39:54 -06:00
Jeff Washington (jwash) 41dd31e5f4
AcctIdx: move cached store id to bit (#21892)
* AcctIdx: move cached store id to bit

* add comments/rename
2021-12-15 11:49:24 -06:00
Jeff Washington (jwash) 02fa135815
AcctIdx: create test fn get_test() to isolate changes to AcctIdx::get() (#21909) 2021-12-15 09:09:56 -06:00
Jeff Washington (jwash) f4308bdb64
AcctIdx: StoredSize is u32 (#21894) 2021-12-14 13:50:19 -06:00
Jeff Washington (jwash) e694acaf5f
AcctIdx: better types for AccountInfo (#21893) 2021-12-14 13:08:49 -06:00
Jeff Washington (jwash) ec583bd12d
AcctIdex: use StorageLocation (#21853) 2021-12-14 10:27:17 -06:00
Jeff Washington (jwash) a86fe899ac
AcctIdx: move zero lamport out of accounts index (#21526) 2021-12-14 09:31:42 -06:00
Justin Starry c92c09a8be
Remove activated feature for removing inactive delegations from stakes cache (#21732)
* Remove activated feature for removing inactive delegations from stakes cache

* Fix builtin purging
2021-12-14 09:23:36 -05:00
Justin Starry e5476913fe
Remove activated feature that checks tx signature len (#21747) 2021-12-14 09:23:05 -05:00
Jeff Washington (jwash) 6fc6673ead
AcctInfo: store_id private and add accessor (#21839) 2021-12-13 21:35:30 -06:00
Jeff Washington (jwash) 98e5ea9dce
AcctIdx: simplify AccountIndexGetResult (#21857) 2021-12-13 21:16:17 -06:00
Jeff Washington (jwash) 50f26ea9c0
AcctInfo: create AcctInfo with cache explicitly (#21847) 2021-12-13 17:35:07 -06:00
Jeff Washington (jwash) 6155ef6377
AcctInfo: make offset private, add accessor (#21838) 2021-12-13 14:43:26 -06:00
dependabot[bot] 8aa3d690b5
chore: bump dashmap from 4.0.2 to 5.0.0 (#21824)
* chore: bump dashmap from 4.0.2 to 5.0.0

Bumps [dashmap](https://github.com/xacrimon/dashmap) from 4.0.2 to 5.0.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases)
- [Commits](https://github.com/xacrimon/dashmap/commits/v5.0.0)

---
updated-dependencies:
- dependency-name: dashmap
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-13 13:23:58 -07:00
Jeff Washington (jwash) 9e9a1a4876
AcctIdx: AccountInfo.lamports private (#21819) 2021-12-13 10:59:33 -06:00
Jeff Washington (jwash) c9e0bde407
AcctIdx: AccountInfo::stored_size private (#21821) 2021-12-13 10:10:06 -06:00
Jeff Washington (jwash) 90f41fd9b7
use cost model to limit new account creation (#21369)
* use cost model to limit new account creation

* handle every system instruction

* remove &

* simplify match

* simplify match

* add datapoint for account data size

* add postgres error handling

* handle accounts:unlock_accounts
2021-12-12 14:57:18 -06:00
Jeff Washington (jwash) 025a5a3b9c
AccountInfo: construct with new() (#21788) 2021-12-12 14:36:05 -06:00
Jeff Washington (jwash) 825f8bcea4
use AppendVecId instead of usize (#21792) 2021-12-11 21:38:13 -06:00
Jeff Washington (jwash) c5b6ea74ef
replace some .lamports checks with trait (#21783) 2021-12-11 21:37:33 -06:00
Jeff Washington (jwash) 4c0373b72a
introduce AtomicAppendVecId (#21793) 2021-12-11 21:34:35 -06:00
Jeff Washington (jwash) ef7f49a21d
move AccountInfo to its own file (#21784) 2021-12-11 11:47:05 -06:00
Brooks Prumo eeb97fe7ce
Add accounts_data_len to Bank (#21781) 2021-12-11 09:34:38 -06:00
Justin Starry 2bbe1d875a
Deserialize accounts before acquiring stakes cache lock (#21733)
* Deserialize stored accounts before locking stakes cache

* fix test
2021-12-10 16:02:35 -05:00
Justin Starry 9b41ddd9ba
Add address lookup table program (#21616)
* Add address lookup table program

* feedback
2021-12-10 16:02:16 -05:00
Brooks Prumo ec7e17787e
Compute accounts data len during generate_index() (#21757) 2021-12-10 13:27:59 -06:00
Tyera Eulberg 80eae20610 Fix master fmt 2021-12-10 11:33:39 -07:00
Tyera Eulberg 350845c513
Move type alias and use it more broadly (#21763) 2021-12-10 10:46:21 -07:00
Justin Starry fd175c1ea9
Add StakesCache struct to abstract away locking (#21738) 2021-12-10 10:51:33 -05:00
axleiro 622fd7c7ec
testing for the latest changes 2021-12-10 19:44:22 +05:30
dependabot[bot] d870f566ef
chore: bump serde from 1.0.130 to 1.0.131 (#21758)
* chore: bump serde from 1.0.130 to 1.0.131

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.130 to 1.0.131.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.130...v1.0.131)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-10 08:47:46 +00:00
Tyera Eulberg c1386d66e6
Nits in message-processor (#21755)
* Fixup typo

* Simplify types slightly
2021-12-09 23:38:36 -07:00
Jeff Washington (jwash) 16a6dceb6b
impl better debug for Ancestors (#21737) 2021-12-09 18:36:13 -06:00
Brooks Prumo 1528f85112
Add return types to generate_index() (#21735) 2021-12-09 17:15:13 -06:00
Justin Starry 6fc329180b
Add more reporting for invalid stake cache members and prune them (#21654)
* Add more reporting for invalid stake cache members

* feedback
2021-12-09 14:12:11 -05:00
Jeff Washington (jwash) a2df1eb502
AcctIdx: disk generate index and filler accounts use more threads (#21566) 2021-12-09 11:54:14 -06:00
Jeff Washington (jwash) 58a7d3fc0e
AcctIdx: disable direct to disk for filler account upsert (#21710) 2021-12-09 10:14:57 -06:00
Jeff Washington (jwash) 0090916735
skip clean on startup, too (#21718) 2021-12-09 09:36:02 -06:00
Jeff Washington (jwash) e020960f49
AcctIdx: generate index goes to mem not disk (#21709) 2021-12-08 19:48:12 -06:00
Jeff Washington (jwash) 54862eba0d
AcctIdx: env var to enable testing of disk buckets (#21494) 2021-12-08 19:47:25 -06:00
Jeff Washington (jwash) 181c0092d6
AcctIdx: resize in-mem after startup for disk index (#21676) 2021-12-08 16:52:22 -06:00
Jeff Washington (jwash) 8d1e5ac294
refactor and test scan abort code (#21390) 2021-12-08 14:09:34 -06:00
Ashwin Sekar f0acf7681e
Add vote instructions that directly update on chain vote state (#21531)
* Add vote state instructions

UpdateVoteState and UpdateVoteStateSwitch

* cargo tree

* extract vote state version conversion to common fn
2021-12-07 16:47:26 -08:00
Alexander Meißner 1df88837c8
- Implicitly fixes invoke_context.return_data not being reset between instructions in process_message. (#21671)
- Lets InvokeContext::process_cross_program_instruction() handle the first invocation depth too.
- Marks InvokeContext::verify(), InvokeContext::verify_and_update() and InvokeContext::process_executable_chain() private.
- Renames InvokeContext::process_cross_program_instruction() to InvokeContext::process_instruction().
- Removes InvokeContext::new_mock_with_sysvars().
2021-12-07 23:00:04 +01:00
dependabot[bot] f6801a4af4
chore: bump itertools from 0.10.1 to 0.10.3 (#21643)
* chore: bump itertools from 0.10.1 to 0.10.3

Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/rust-itertools/itertools/releases)
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.1...v0.10.3)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-12-06 23:23:26 -07:00
Brooks Prumo da4015a959
Ensure that StakeDelegations and StakeHistory serde (#21640)
Add tests to StakeDelegations and StakeHistory to ensure that the outer
types serialize and deserialize correctly to/from the inner types.
2021-12-06 17:34:10 -06:00
Justin Starry a1adcb23b6
Remove activated feature for filtering invalid stakes from rewards (#21641) 2021-12-06 17:56:05 -05:00
Justin Starry 3dab1e711d
Move transaction error code into new module (#21635) 2021-12-06 12:45:33 -05:00
Trent Nelson d6f22433d0 Bump version to v1.10.0 2021-12-04 20:17:54 +00:00
Brooks Prumo 46fe56171b
Make StakeHistory clone-on-write (#21573) 2021-12-03 12:10:29 -06:00
Justin Starry 1430b58a6d
Remove deprecated slow epoch boundary methods (#21568) 2021-12-03 17:59:10 +00:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
Brooks Prumo 0ef1b25e4b
Make StakeDelegations clone-on-write (#21542) 2021-12-03 08:54:38 -06:00
Alexander Meißner 1a4a039913
Adds feature reject_empty_instruction_without_program. (#21591) 2021-12-03 15:47:18 +01:00
Alexander Meißner a9d5ef2055
Cleanup: InvokeContext accessors (#21574)
* Removes blockhash accessors from InvokeContext.

* Removes lamports_per_signature accessors from InvokeContext.

* Removes return_data accessors from InvokeContext.

* Removes feature_set accessor from InvokeContext.

* Removes instruction_recorders and instruction_index accessors from InvokeContext.

* Moves get_sysvars() into InvokeContext.

* Removes compute_meter parameter from InvokeContext::new().

* Removes InvokeContext::new_mock_with_sysvars_and_features().

* Removes InvokeContext::update_timing().
2021-12-03 12:15:22 +01:00
Brooks Prumo bdc33ba0a1
Remove unnecessary Option (#21569) 2021-12-02 20:19:22 +00:00
Tao Zhu bf5b7f5d7f
report compute units without tx_wide_compute_cap feature enabled (#21421) 2021-12-02 12:14:57 -06:00
Jeff Washington (jwash) f0b32b75ab
AcctIdx: use ZeroLamport trait (#21552) 2021-12-02 12:10:11 -06:00
Jeff Washington (jwash) 314605e149
AcctIdx: test cleanup (#21550) 2021-12-02 12:09:37 -06:00
Jack May 976eb81d4f
Cleanup the bank's use of nonces (#21246) 2021-12-02 09:57:05 -08:00
Alexander Meißner bfdb775ffc
Unifies ThisInvokeContext and dyn trait InvokeContext. (#21563) 2021-12-02 18:47:16 +01:00
Jeff Washington (jwash) 8dfa83c579
AcctIdx: minor code cleanup (#21547) 2021-12-02 11:05:54 -06:00
Anton Lazarev 6bb884836c address review feedback - use u64 instead of usize 2021-12-02 08:12:30 -08:00
Anton Lazarev adbf31b98c avoid usize overflow for MAXIMUM_APPEND_VEC_FILE_SIZE on 32-bit platforms 2021-12-02 08:12:30 -08:00
Jeff Washington (jwash) c8a52337c8
AcctIdx: remove unused function (#21546) 2021-12-01 17:14:49 -06:00
Trent Nelson 1fae3d24de bank: reuse last hash/fees helper to instatiate `InvokeContext` 2021-12-01 12:59:20 -07:00
Trent Nelson 91c6a10cda bank: get hash and fee rate from the same source when storing accounts 2021-12-01 12:59:20 -07:00
Jeff Washington (jwash) 308d7d40d0
aggressively flush cache based on cache size (#21332)
* aggressively flush cache based on cache size

* find_older_frozen_slots -> cached_frozen_slots

* remove 'WRITE_CACHE_LIMIT_BYTES_DEFAULT'

* tweaks to stats

* fix tests
2021-12-01 13:10:48 -06:00
Jeff Washington (jwash) 1eefdeba85
AcctIdx: cleanup trait (#21522) 2021-12-01 12:19:06 -06:00
Brooks Prumo 4a1ef12bd9
Refactor fns that calculate stake and voter balances (#21529) 2021-12-01 10:29:24 -06:00
Jeff Washington (jwash) b108d7ddaa
AcctIdx: insert goes directly to disk to avoid unnecessary allocations (#21490)
* AcctIdx: upsert avoids unnecessary allocation (during startup)

* feedback
2021-12-01 10:21:01 -06:00
Brooks Prumo cb368e6554
Pass Epoch by value in StakeHistory::get() (#21523) 2021-12-01 08:57:29 -06:00
Alexander Meißner b78f5b6032
Refactor: Cleanup InstructionProcessor (#21404)
* Moves create_message(), native_invoke() and process_cross_program_instruction()
from the InstructionProcessor to the InvokeContext so that they can have a useful "self" parameter.

* Moves InstructionProcessor into InvokeContext and Bank.

* Moves ExecuteDetailsTimings into its own file.

* Moves Executor into invoke_context.rs

* Moves PreAccount into its own file.

* impl AbiExample for BuiltinPrograms
2021-12-01 08:54:42 +01:00
Michael Vines e922c2da9d Update to Rust 1.56.1 2021-11-30 23:28:07 -08:00
Michael Vines dd12d90eac Upgrade to Rust 2021 2021-11-30 20:43:46 -08:00
ryleung-solana 8cf36e5cb0
Add GPU sigverify for verify path (#20851)
Allows the use of GPU acceleration in verifying the signatures in Entry's after deserialization in the replay stage

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Ryan Leung <ryan.leung@solana.com>
2021-11-30 21:16:13 -05:00
Jeff Washington (jwash) d8fb7ce511
AcctIdx: upsert avoids unnecessary allocation (#21488) 2021-11-30 11:36:46 -06:00
Jeff Washington (jwash) 7ec88226ee
AcctIdx: stats are usize (#21493)
* AcctIdx: stats are usize

* rename test function
2021-11-30 11:35:21 -06:00
Jeff Washington (jwash) 2be859d304
add test for aborting index scan (#21487) 2021-11-29 16:13:39 -06:00
Michael Vines ba9dfa0d22 Remove frozen account support 2021-11-29 08:38:11 -08:00
Trent Nelson 41cc7af7cd bank: `TransactionLogCollector` - only push address mappings after pushing a log message 2021-11-25 04:50:51 +00:00
Trent Nelson e558ad4c71 bank: don't panic when getting logs from malformed TransactionLogCollector 2021-11-25 04:50:51 +00:00
Trent Nelson dafdc15dfe bank: Add failing test for `TransactionLogCollector::get_logs_for_address()` 2021-11-25 04:50:51 +00:00
Trent Nelson efedb55705 bank: factor tx log getter out to method on `TransactionLogCollector` 2021-11-25 04:50:51 +00:00
Tao Zhu 9edfc5936d
Refactor accounts.rs with Justin's comments to improve lock accounts (#21406)
with results code path.
- fix a bug that could unlock accounts that weren't locked
- add test to the refactored function
- skip enumerating transaction accounts if qos results is an error
- add #[must_use] annotation
- avoid clone error in results
- add qos error code to unlock_accounts match statement
- remove unnecessary AbiExample
2021-11-23 21:17:55 +00:00
Alexander Meißner 22a2537aac
Refactor: Cleanup program-runtime dyn Traits (#21395)
* Unifies dyn Trait ComputeMeter, ThisComputeMeter and TransactionComputeMeter.

* Unifies dyn Trait Logger and ThisLogger.

* Moves Logger to log_collector.rs

* Unifies Logger and LogCollector.

* Removes inner RefCell from LogCollector.

* Adds the log::debug!() message to ic_logger_msg!() again.
2021-11-23 13:23:40 +01:00
Jeff Washington (jwash) b87ebf9e58
AcctIdx: PreAllocatedAccountMapEntry does not make unnecessary Arc (#21364) 2021-11-22 15:52:45 -06:00
Jeff Washington (jwash) a762066310
AcctIdx: don't log disk stats when disk isn't enabled (#21371) 2021-11-22 15:26:25 -06:00
sakridge c4d68063c7
Add timing for accounts add_root (#21379) 2021-11-22 18:29:45 +01:00
sakridge cd6f931223
Add timing metrics to bank creation and rewards calculation (#21066) 2021-11-21 15:11:02 +01:00
Jon Cinque 02bc4e3fc1
spl-token: New program feature flag (#21354)
* spl-token: Add feature flag for new release

* Remove all spl token version declarations
2021-11-21 14:27:03 +01:00
Jeff Washington (jwash) ebea3297f9
AcctIdx: generate index inserts/updates directly to disk (#21363)
* when initially creating account index, write directly to disk

* AcctIdx: generate index inserts/updates directly to disk
2021-11-19 17:17:07 -06:00
Brooks Prumo 0bb059185c
Add lamports_in_rent_paying_accounts to TotalAccountsStats (#21368) 2021-11-19 17:02:23 -06:00
Alexander Meißner 8a50b6302f
Fix #21346 (#21362)
Fixes the empty transaction bug in ThisInvokeContext::push() and adds a test for it to the bank.
2021-11-19 20:43:42 +01:00
Jeff Washington (jwash) 79d21d6805
add --accounts-index-scan-results-limit-mb to allow scans to abort (#21327)
* ScanConfig -> &ScanConfig

* add --accounts-index-scan-results-limit-mb to allow scans to abort

* feedback
2021-11-19 09:00:19 -06:00
Brooks Prumo c24e30f689
Constrain memory ordering on AccountsDb::next_id (#21311) 2021-11-18 20:34:37 -06:00
Brooks Prumo f7152c889c
Constrain memory ordering on AppendVec::id (#21310) 2021-11-18 20:34:20 -06:00
Brooks Prumo 5788973153
Constrain memory ordering on AccountStorageEntry::alive_bytes (#21309) 2021-11-18 20:34:08 -06:00
Jack May 03c36d240a
Nonce naming cleanup (#21336) 2021-11-18 16:07:17 -08:00
carllin b30c94ce55
ClusterInfoVoteListener send only missing votes to BankingStage (#20873) 2021-11-18 15:20:41 -08:00
Jeff Washington (jwash) a272e19f8d
AcctIdx: report on partial progress during flushing (#21306) 2021-11-18 11:01:19 -06:00
Lijun Wang 89c45a57f8
Refactor slot status notification to decouple from accounts notifications (#21308)
Problem

Slot status can be used of in other scenarios in addition to account information such as transactions, blocks. The current implementation is too tightly coupled.

Summary of Changes

Decouple the slot status notification from accounts notification. Created a new slot status notification module.
2021-11-17 17:11:38 -08:00
Jeff Washington (jwash) 9cb74e8421
remove store_cached stall (#21326) 2021-11-17 18:30:40 -06:00
Jeff Washington (jwash) 83de2f7376
fix initial massive metrics spike (#21304) 2021-11-17 17:16:48 -06:00
Jeff Washington (jwash) 0f69a14247
Add ability to abort scan (#21314) 2021-11-17 13:10:29 -06:00
Alexander Meißner e540b1cf3c
Refactor: Move sdk::process_instruction in program-runtime-crate (#21180)
* Moves the Executor dyn Trait to instruction_processor.rs

* Moves the Logger dyn Trait as well as the ic_msg and ic_logger_msg macros to log_collector.rs,
and moves the stable_log to stable_log.rs

* Moves the ComputeMeter dyn Trait to invoke_context.rs

* Moves the InvokeContext dyn Trait and the ProcessInstructionWithContext type to invoke_context.rs

* Updates cargo files.

* Re-export InvokeContext in program-test

Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
2021-11-17 19:35:07 +01:00
Brooks Prumo c3e5927d16
Constrain memory ordering on AccountsDb::write_version (#21312) 2021-11-17 07:11:16 -06:00
Jack May cb0bb5bd1e
Nonce accounts must be writeable (#21260)
* Nonce accounts must be writeable

* feedback

* feedback
2021-11-16 23:01:00 +00:00
Jeff Washington (jwash) ae497715cc
HashCalc: Perf: use refs for hashes (#21280) 2021-11-16 10:30:55 -06:00
sakridge 398af132a5
More set_root metrics (#21286) 2021-11-15 16:28:18 -07:00
Ivan Mironov 9b1bf98aa2
Accept only decimal digits in file names inside snapshot (#21213)
This also should make snapshot validation a bit faster.
2021-11-15 17:02:15 +00:00
Jeff Washington (jwash) 39340ed25b
throttle store_cached when cache size is too large (#21188)
* throttle store_cached when cache size is too large

* reduce max delay

* 100ms max

* 10ms max delay
2021-11-13 14:00:37 -06:00
Jeff Washington (jwash) f8dcb2f38b
report mem stats (#21258) 2021-11-13 00:59:41 +00:00
Jack May 778b5d736c
Load nonce with fixed root (#21248) 2021-11-12 14:12:34 -08:00
Jeff Washington (jwash) 1509513339
hash calc: filler_accounts_enabled (#21243) 2021-11-12 09:53:21 -06:00
Tao Zhu 11153e1f87
refactor cost calculation (#21062)
* - cache calculated transaction cost to allow sharing;
- atomic cost tracking op;
- only lock accounts for transactions eligible for current block;
- moved qos service and stats reporting to its own model;
- add cost_weight default to neutral (as 1), vote has zero weight;

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Update core/src/qos_service.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Update core/src/qos_service.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-11-12 01:04:53 -06:00
Justin Starry ef29d2d172
Refactor vote state to remove double negative (#21244) 2021-11-12 00:26:43 +00:00
Jack May 131c0d78c4
Fix compute budget bump (#21238) 2021-11-11 14:09:28 -08:00
Jeff Washington (jwash) c0f9e351db
hashcalc eliminate zeros optimizations (#21209) 2021-11-11 09:44:32 +00:00
Brooks Prumo 8b2133b078
Refactor total stats calculation into shared fn (#21212) 2021-11-09 19:13:37 -06:00
dependabot[bot] 48265ada39
chore: bump blake3 from 1.1.0 to 1.2.0 (#21203)
* chore: bump blake3 from 1.1.0 to 1.2.0

Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases)
- [Commits](https://github.com/BLAKE3-team/BLAKE3/compare/1.1.0...1.2.0)

---
updated-dependencies:
- dependency-name: blake3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* Make versions consistent

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-11-08 15:27:37 -07:00
Jeff Washington (jwash) 5827cf4c51
hash calc eliminate zero refactor get_item (#21036) 2021-11-08 20:27:48 +00:00
carllin 706b60b5c8
Add `set_root` bank drop logging (#21144) 2021-11-06 01:05:36 +00:00
Jeff Washington (jwash) 1b46d1d54d
hashcalc eliminate zeros cleanup (#21175) 2021-11-05 10:54:49 -05:00
Ivan Mironov c78f474373 Add validator option to change niceness of snapshot packager thread 2021-11-04 17:16:46 -06:00
Alexander Meißner 7200c5106e
Replaces MockInvokeContext by ThisInvokeContext in tests (#20881)
* Replaces MockInvokeContext by ThisInvokeContext in BpfLoader, SystemInstructionProcessor, CLIs, ConfigProcessor, StakeProcessor and VoteProcessor.

* Finally, removes MockInvokeContext, MockComputeMeter and MockLogger.

* Adjusts assert_instruction_count test.

* Moves ThisInvokeContext to the program-runtime crate.
2021-11-04 21:47:32 +01:00
Justin Starry 0597594943
Fix rent debits test (#21177) 2021-11-04 20:03:19 +00:00
Jack May 4e527b84b0
Remove unused upgrade epoch handling (#21164) 2021-11-04 10:44:36 -07:00
Jeff Washington (jwash) 171243a370
add rent exempt count to get_total_accounts_stats (#21153) 2021-11-04 11:17:19 -05:00
Jeff Washington (jwash) e765542cd8
eliminate pubkey copy in eliminate_zeros (#21032)
* rework eliminate zero loop

* eliminate pubkey copy in eliminate_zeros
2021-11-04 11:16:36 -05:00
Jeff Washington (jwash) 76e533be46
LoadedAccountAccessor::Cached does not store Pubkey (#21030) 2021-11-04 10:28:04 -05:00
Justin Starry aaccba8377 Refactor `RentDebits` to use hashmap instead of vec 2021-11-04 15:02:24 +00:00
Jeff Washington (jwash) b1e3a82c11
refactor rent collector and logs (#21155) 2021-11-04 06:28:41 -05:00
Alexander Meißner 29ad081555
Stop caching sysvars, instead load them ahead of time. (#21108) 2021-11-04 09:48:34 +01:00
Brooks Prumo c326f41dfe
Add bank::get_total_accounts_stats() (#21159) 2021-11-03 19:23:45 -05:00
Jeff Washington (jwash) c59c56c0f8
log rent exempt accounts (#21137) 2021-11-03 14:44:52 -05:00
Jeff Washington (jwash) c0952831be
correctly clean filler accounts (#21139) 2021-11-03 11:27:15 -05:00
Justin Starry 140a5f633d
Simplify replay vote tracking by using packet metadata (#21112) 2021-11-03 09:02:48 +00:00
sakridge 1e5212e60d
Add set_root timing metrics (#21119) 2021-11-02 18:23:35 +01:00
Jack May bced07a099
Update fee api to use blockhash (#21054) 2021-10-29 13:52:59 -07:00
Jack May 4aa12a52b6
Feature cleanup (#21038) 2021-10-28 09:04:03 -07:00
Jeff Washington (jwash) 7a41b2c5d1
optimize get_accounts_delta_hash (#21027) 2021-10-28 10:40:32 -05:00
Jeff Washington (jwash) e9ab214237
filler accts: only add filler accts to slots in the current epoch (#21024) 2021-10-28 10:26:30 -05:00
Jeff Washington (jwash) 6a1ff60c61
percent stats 0-1 -> 0-100 (#21028) 2021-10-27 18:08:27 -05:00
Justin Starry 036d7fcc81
Clean up sanitized tx creation for tests (#21006) 2021-10-27 18:09:16 +01:00
Jeff Washington (jwash) 9d330fc638
FillerAccts: use variable cycle partitions (#20963) 2021-10-27 11:18:27 -05:00
Jack May 1e2bef76e3
Add compute budget program as a noop (#20992) 2021-10-26 20:08:59 -07:00
Jeff Washington (jwash) 4d6190a1ba
change bg waiting metrics to percent (#20955)
* Revert part of "AcctIdx: cleanup bg threads (#20731)"

This reverts part of commit 47de4f31b6.

* change bg waiting metrics to percent
2021-10-26 15:09:17 -05:00
Jack May 2515f6a04f
Update deprecation versions (#20959) 2021-10-26 09:06:41 -07:00
Lijun Wang bbe3ce3db5
Accountsdb plugin write ordering (#20948)
Use the write_version in the Accounts's meta data so that account write with lower write_version would not overwrite the higher ones.
2021-10-25 14:07:56 -07:00
Alexander Meißner cf0fd5b2ca
Moves NonceKeyedAccount from the SDK to the Runtime. (#20954) 2021-10-25 22:58:18 +02:00
Jeff Washington (jwash) 9e85499d4c
fix hash calc # storages, # slots (#20832) 2021-10-25 19:37:50 +00:00
Jeff Washington (jwash) 43ea579f63
add cli for --accounts-hash-num-passes (#20827) 2021-10-25 09:45:46 -05:00
Tao Zhu c2bfce90b3
- cost_tracker is data member of a bank, it can report metrics when bank is frozen (#20802)
- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze
2021-10-24 22:19:23 -05:00
Lijun Wang f14365f4b4
Accountsdb plugin postgres -- bulk insertion at startup (#20763)
Use bulk insertion to Postgres at startup to reduce time taken for initial snapshot restore for postgres plugin. Avoid duplicate writes of accounts at startup. Doing account plugin notification and indexing in parallel.

Improved error handling for postgres plugin to show the real db issues for debug purpose
Added more metrics for postgres plugin.
Refactored plugin centric code out to a sub module from accounts_db and added unit tests
2021-10-24 12:43:33 -07:00
Jack May bfbbc53dac
Divorce the runtime from FeeCalculator (#20737) 2021-10-22 14:32:40 -07:00
Justin Starry 735016661b
Report timing info for stakes cache updates from txs (#20856) 2021-10-22 12:49:02 -04:00
dependabot[bot] 1d9c1ccfd2
chore: bump blake3 from 1.0.0 to 1.1.0 (#20877)
* chore: bump blake3 from 1.0.0 to 1.1.0

Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 1.0.0 to 1.1.0.
- [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases)
- [Commits](https://github.com/BLAKE3-team/BLAKE3/compare/1.0.0...1.1.0)

---
updated-dependencies:
- dependency-name: blake3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-10-22 09:44:42 -06:00
Alexander Meißner 97c2732d02
Refactor: Cleanup InvokeContext (#20785)
* Move blockhash and fee_calculator in ThisInvokeContext instead of using a reference.

* Moves tx_wide_compute_cap into InvokeContext::push().

* Adds ThisInvokeContext::new_mock() constructor.

* Adds missing loader account in uses of MockInvokeContext.

* Use keyed_account_at_index() when accessing keyed_accounts.

* Makes sysvar interface consistent between ThisInvokeContext and MockInvokeContext,
in order to add InvokeContext::get_sysvars().

* Adds InvokeContext::set_blockhash() and InvokeContext ::set_fee_calculator().

* Adds new_mock_with_features.

* Makes ancestors optional in ThisInvokeContext.

* Adds prepare_mock_invoke_context() and mock_process_instruction().
2021-10-21 20:57:42 +02:00
Jack May a8098f37d0
add checked instructions sysvar api (#20790) 2021-10-19 21:01:58 -07:00
Jack May 58164517e4
Add program heap bump instruction (#20607) 2021-10-19 21:01:39 -07:00
Jon Cinque cb2bd65858
runtime: Add foundation stake pool withdraw authority (#20797) 2021-10-19 21:49:21 +00:00
Jeff Washington (jwash) 47a58a38c2
clean_accounts calls AcctIdx: get_many (#20715)
* AcctIdx: get_many

* keep read lock

* AcctIdx: get_many optionally adds to cache

* rename
2021-10-19 15:54:06 -05:00
Jeff Washington (jwash) 0acbfdfcb9
hash calc: sort time is redundantly counted as part of 'total time' (#20791) 2021-10-19 15:52:43 -05:00
Tao Zhu 7496b5784b
- make cost_tracker a member of bank, remove shared instance from TPU; (#20627)
- decouple cost_model from cost_tracker; allowing one cost_model
  instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
2021-10-19 14:37:33 -05:00
Michael Vines 2b76ea51b4 Reduce visibility of `Hash` struct contents 2021-10-19 09:30:47 -07:00
Jeff Washington (jwash) 46bf7d4a4a
AcctIdx: support 2^24 bins (#20739) 2021-10-18 14:05:16 -05:00
Brooks Prumo 751b45df64
Make pub get_incremental_snapshot_archives() (#20725) 2021-10-18 13:48:50 -05:00
sakridge c9b367c350
Minimum of 1 lamport for rent (#20650) 2021-10-18 19:09:53 +02:00
Jeff Washington (jwash) 6ec159a1ae
parallelize update_index (#20601) 2021-10-18 09:07:01 -05:00
Jeff Washington (jwash) 0d934d311e
sort shrink acct indexes (#20708) 2021-10-18 09:06:32 -05:00
Alexander Meißner e98cfbb570
Removes deprecated BpfComputeBudget. (#20728) 2021-10-18 09:55:41 +02:00
Jeff Washington (jwash) 70b2e5fef2
hash calc uses all threads during startup (#20735) 2021-10-15 17:41:23 -05:00
Jeff Washington (jwash) 47de4f31b6
AcctIdx: cleanup bg threads (#20731) 2021-10-15 16:15:11 -05:00
Jeff Washington (jwash) 254ad45878
sort pubkeys for clean (#20706) 2021-10-15 14:31:06 -05:00
Jeff Washington (jwash) 6d7da6dbee
clean:clone uncleaned_roots for loop (#20714) 2021-10-15 13:18:00 -05:00
Jeff Washington (jwash) 5a6d732d48
AcctIdx: separate account index by startup and !startup (#20711) 2021-10-15 13:04:32 -05:00
Jeff Washington (jwash) 9d62963240
AcctIdx: get_internal allows not adding to in_mem (#20716) 2021-10-15 13:04:05 -05:00
Jeff Washington (jwash) 1e064c32e2
pull append_vec_id() out of loop (#20713) 2021-10-15 10:41:23 -05:00
Jeff Washington (jwash) 00b9056004
clean metric: total_us (#20707) 2021-10-15 10:41:05 -05:00
Jeff Washington (jwash) b80fd7566a
add worker threads during startup (#20700) 2021-10-15 10:40:36 -05:00
Tao Zhu e225ed7ab0
update config by testnet (#20699) 2021-10-15 15:32:46 +00:00
Jeff Washington (jwash) 7eead4e25f
clean stats on acct idx lookup (#20632) 2021-10-14 08:37:42 -05:00
Jeff Washington (jwash) c43b6e0248
log metric for clean_accounts.delta_insert_us (#20634) 2021-10-14 08:37:14 -05:00
Jeff Washington (jwash) 9b042fed30
AcctIdx: support unref first class for perf (#20638) 2021-10-14 08:36:53 -05:00
Jack May da45be366a
Remove blockhash from fee calculation (#20641) 2021-10-13 13:10:58 -07:00
Tao Zhu 220fd41bbc
fix unstable test (#20645) 2021-10-13 11:43:18 -05:00
Jeff Washington (jwash) aec9d8bf2f
add clean stats (#20631) 2021-10-13 08:22:11 -05:00
Alexander Meißner 1d813ea078
Refactor: Make program_id always last in program chain (#20598)
* Replaces program_id field in InvokeContextStackFrame by index.

* Swaps order of program account and programdata account.

* Removes program_id parameter from InvokeContext::push().
2021-10-13 08:58:20 +02:00
Jeff Washington (jwash) 7631011d8c
AcctIdx: add flush remove stat (#20625) 2021-10-12 16:13:32 -05:00
Alexander Meißner 06cad19c91
Adds missing program_id in InvokeContext::push() calls. (#20621) 2021-10-12 19:26:55 +02:00
Tao Zhu 005d6863fd
- move cost tracker into bank, so each bank has its own cost tracker; (#20527)
- move related modules to runtime
2021-10-12 08:51:33 -05:00
Jack May b0ee5d6cfa
Add debug guard to catch invalid params (#20603) 2021-10-11 16:47:17 -07:00
Jeff Washington (jwash) 0da677e213
AcctIdx: write disk only when removing from mem (#20578) 2021-10-11 14:47:06 -05:00
dependabot[bot] 9d94e43839
chore: bump ouroboros from 0.12.0 to 0.13.0 (#20591)
* chore: bump ouroboros from 0.12.0 to 0.13.0

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.12.0 to 0.13.0.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-10-11 11:54:37 -06:00
Jeff Washington (jwash) a8e000a2a6
add filler accounts to bloat validator and predict failure (#20491)
* add filler accounts to bloat validator and predict failure

* assert no accounts match filler

* cleanup magic numbers

* panic if can't load from snapshot with filler accounts specified

* some renames

* renames

* into_par_iter

* clean filler accts, too
2021-10-11 12:46:27 -05:00
Alexander Meißner f30f3bddbb
Refactor: Remove program_id from process_instruction() (#20540)
* Replaces usage of program_id parameter by invoke_context.get_caller()?.

* Removes "pubkey: &Pubkey" parameter from "process_instruction()".
2021-10-10 22:29:18 +02:00
Jeff Washington (jwash) 8cf222b8bd
AcctIdx: combine if (#20545) 2021-10-10 13:48:18 -05:00
Jeff Washington (jwash) 33d8c07364
AcctIdx: combine scan and update loops (#20546) 2021-10-10 13:47:52 -05:00
Justin Starry db9336c99e
Remove feature switch handling for checking vote init (#20557) 2021-10-08 23:35:26 -04:00
Jeff Washington (jwash) 0545306f0c
AcctIdx: remove stats from write lock section (#20548) 2021-10-08 15:27:38 -05:00
Brooks Prumo 5440c1d2e1
SnapshotPackagerService pushes incremental snapshot hashes to CRDS (#20442)
Now that CRDS supports incremental snapshot hashes,
SnapshotPackagerService needs to push 'em!

This commit does two main things:

1. SnapshotPackagerService now knows about incremental snapshot hashes,
   and will push SnapshotPackage::IncrementalSnapshot hashes to CRDS.
2. At startup, when loading from a full + incremental snapshot, the
   hashes need to be passed all the way to SnapshotPackagerService so it
   can push these starting hashes to CRDS.  Those values have been piped
   through.

Fixes #20441 and #20423
2021-10-08 15:14:56 -05:00
Jeff Washington (jwash) 4f6a0b2650
AcctIdx: stats for buckets on disk, add median (#20528) 2021-10-08 13:58:38 -05:00
Jeff Washington (jwash) 2c3d52b4cc
AcctIdx: avoid overcounting inserts (#20544) 2021-10-08 13:57:47 -05:00
Alexander Meißner 4e65487d2f
Refactor: process_instruction() (#20448)
* Adds first_instruction_account parameter to process_instruction().

* Removes InvokeContext::remove_first_keyed_account() from all BPF loaders.

* Removes InvokeContext::remove_first_keyed_account() from all builtin programs.

* Removes InvokeContext::remove_first_keyed_account() from all mock ups.

* Deprecates InvokeContext::remove_first_keyed_account().

* Documents index base of keyed_account_at_index().

* Adds dynamic offset to call sites of "keyed_account_at_index()".
2021-10-08 11:41:07 +02:00
Jeff Washington (jwash) 069586c171
AcctIdx: set startup around clean and shrink at startup to flush cache (#20526) 2021-10-07 22:57:40 +00:00
Tao Zhu 177a375479
Tpu vote 1.7 (#20187) (#20494)
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>

Co-authored-by: sakridge <sakridge@gmail.com>
2021-10-07 09:38:23 +00:00
Jeff Washington (jwash) df39b37cb8
improve clean stats (#20469) 2021-10-06 20:04:26 -05:00
Trent Nelson 767f740305 Bump version to 1.9.0 2021-10-06 17:57:41 -07:00
Michael Vines 3b70cdb8d8 Upgrade to Rust 1.55.0 2021-10-06 17:48:58 -07:00
Jack May 785fcb63f5
Remove support for dynamically loaded native programs (#20444) 2021-10-06 14:53:23 -07:00
Michael Vines 7027d56064 Resolve nightly-2021-10-05 clippy complaints 2021-10-06 10:37:58 -07:00
Jeff Washington (jwash) 895b46e0c3
simplify AccountsDb::new_with_config (#20473) 2021-10-06 11:41:23 -05:00
Justin Starry 129716f3f0
Optimize stakes cache and rewards at epoch boundaries (#20432)
* Optimize stakes cache and rewards at epoch boundaries

* Fetch from accounts db

* Add cli flag for disabling epoch boundary optimization
2021-10-06 00:53:26 -04:00
Justin Starry 250a8503fe
Make rewards tracer async friendly (#20452) 2021-10-05 21:30:08 -04:00
Jeff Washington (jwash) d03bf2bbfe
AcctIdx: metrics fixes (#20449) 2021-10-05 22:26:29 +00:00
Jeff Washington (jwash) 4267419349
AcctIdx: sleep thread when enough threads are running to complete the remaining work (#20446) 2021-10-05 16:48:23 -05:00
Jeff Washington (jwash) 0d98a91511
AcctIdx: metric for bucket grow time (#20445) 2021-10-05 20:22:01 +00:00
Jeff Washington (jwash) 64cf354651
Add clean metrics (#20410) 2021-10-04 18:21:45 -05:00
Jeff Washington (jwash) 2d78f8ad2a
parallelize unref_from_storage from clean_dead_slots_from_accounts_index (#20411) 2021-10-04 18:20:10 -05:00
Justin Starry 0ddb34a0b4
Add struct and convenience methods to track stake activation status (#20392)
* Add struct and convenience methods to track stake activation status

* fix nits

* rename
2021-10-04 18:59:11 -04:00
Jeff Washington (jwash) b6a6d6986d
lift store.slot() from loop (#20409) 2021-10-04 16:07:01 -05:00
Jeff Washington (jwash) 8da2eb980a
AcctIdx: introduce BucketApi for access to a specific bucket (#20359) 2021-10-04 11:48:09 -05:00
Jeff Washington (jwash) e31c065544
AcctIdx: when disk index insert needs to grow, drop locks, grow, then restart flush (#20333) 2021-09-30 19:27:09 -04:00
Lijun Wang fe97cb2ddf
AccountsDb plugin framework (#20047)
Summary of Changes

Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows

Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.

The code comprises 4 major parts:

accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
dependabot[bot] 9ef1f06b3f
chore: bump ouroboros from 0.11.1 to 0.12.0 (#20344)
* chore: bump ouroboros from 0.11.1 to 0.12.0

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.11.1 to 0.12.0.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-30 13:20:05 -06:00
Jeff Washington (jwash) 8db9586599
AcctIdx: randomly evict items from in mem (#20319) 2021-09-30 02:51:54 +00:00
Jeff Washington (jwash) 5e05f12c48
AcctIdx: bucket perf improvements (#20328) 2021-09-30 01:53:26 +00:00
Jeff Washington (jwash) 59112adac1
AcctIdx: us stats were showing ms (#20324) 2021-09-29 22:18:07 +00:00
Jeff Washington (jwash) 2ec74474e9
AcctIdx: optimize should_remove_from_mem (#20327) 2021-09-29 16:41:54 -05:00
Alexander Meißner 4de5fff3ca
Simplifies return_data accessors in InvokeContext. (#20290) 2021-09-29 19:11:06 +02:00
Alexander Meißner 57c8abf499
Refactor: Merge message processor into invoke context (#20308)
* Inlines MessageProcessor::execute_instruction() in MessageProcessor::process_message().

* Moves MessageProcessor::create_pre_accounts() into ThisInvokeContext::push().

* Move instruction_recorders slice into InvokeContext.

* Makes account_indices optional in InvokeContext::push().

* Separates initial InvokeContext::push() from ThisInvokeContext::new().

* invoke_context.pop() the base invocation frame.

* Zip message.instructions.iter() and program_indices.iter().

* Moves ThisInvokeContext::new() to the beginning of the loop inside MessageProcessor::process_message().

* Hoists ThisInvokeContext::new() out of loop inside MessageProcessor::process_message().

* Removes unnecessary clone() from ThisInvokeContext::new() in MessageProcessor ::process_message().

* Stop ignoring errors from MessageProcessor::create_pre_accounts().

* Moves MessageProcessor::verify_account_references() and MessageProcessor::verify() into InvokeContext::verify().
2021-09-29 12:05:25 -05:00
Jeff Washington (jwash) 062d8e6c5a
AcctIdx: fix logic on should_remove_from_mem (#20302) 2021-09-29 13:07:40 +00:00
Jeff Washington (jwash) bd9b2f6f39
AcctIdx: throttling bg flushing to spread out work over interval (#20176)
* AcctIdx: throttling bg flushing to spread out work over interval

* prevent hang
2021-09-29 07:42:54 -05:00
Jack May 8fee9a2e1a
Dont call precompiled programs (#19930) 2021-09-29 06:25:08 +00:00
Jack May 491877de3d
Revert "Refactor: Merge MessageProcessor into InvokeContext (#20165)" (#20301)
This reverts commit df6905c3a6.
2021-09-28 16:59:01 -07:00
Jeff Washington (jwash) fa5b091b4c
prior to panicing with cap mismatch, try other calculation (#20292) 2021-09-28 16:09:36 -05:00
Jeff Washington (jwash) 320beb76f6
AcctIdx: implement wait_for_idle (#20256) 2021-09-28 11:07:19 -05:00
Tyera Eulberg 2cd9dc99b6
Restore ability for programs to upgrade themselves (#20265)
* Make helper associated fn

* Add feature definition

* Add handling to preserve program-id write lock when upgradeable loader is present; restore bpf upgrade-self test

* Use single feature
2021-09-28 09:59:08 -06:00
Jack May 4e27543415
Allow programs to realloc their accounts within limits (#19475) 2021-09-28 01:13:03 -07:00
Jeff Washington (jwash) e50a26e6cf
AcctIdx: add age_interval_ms() to remove constant use (#20264) 2021-09-28 01:22:07 +00:00
Jeff Washington (jwash) af309c126c
AcctIdx: refactor get to use more refs (#20268) 2021-09-28 00:18:36 +00:00
dependabot[bot] 34406d4815
chore: bump ouroboros from 0.10.1 to 0.11.1 (#20244)
* chore: bump ouroboros from 0.10.1 to 0.11.1

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.10.1 to 0.11.1.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-27 23:26:08 +00:00
Jeff Washington (jwash) f6812523cd
handle race condition with flush and delete (#20255) 2021-09-27 16:55:15 -05:00
Jeff Washington (jwash) bbd2f9672d
clone -> Arc::clone to make cost clear (#20262) 2021-09-27 16:49:53 -05:00
Jeff Washington (jwash) 4f83818fdb
get rid of WriteAccountMapEntry instance (#19093) 2021-09-27 16:24:19 -05:00
Jeff Washington (jwash) e4795ae7ed
AcctIdx: keep cached or multiple slot lists in cache (#20257) 2021-09-27 16:09:26 -05:00
Jeff Washington (jwash) 9d91a6f1b7
shrink slots at startup uses more threads (#20251) 2021-09-27 14:14:55 -05:00
Jeff Washington (jwash) 8266864adc
AcctIdx: maybe advance age when a bg thread wakes up (#20250)
* fix math error with stats calc

* AcctIdx: maybe report stats when a bg thread wakes up
2021-09-27 18:39:41 +00:00
Brooks Prumo 95fbc29500
Add stats when getting snapshot storages (#20234) 2021-09-27 13:07:00 -05:00
Jeff Washington (jwash) 46263d405e
fix math error with stats calc (#20253) 2021-09-27 11:55:19 -05:00
Jeff Washington (jwash) bc50fd4ad6
allocate threads for accounts idx based on cpus (#20118) 2021-09-27 11:25:06 -05:00
Jeff Washington (jwash) 025261c05e
AcctIdx: when no disk index, always wait for stats (#20248) 2021-09-27 11:24:36 -05:00
Jeff Washington (jwash) 833f7761d2
AcctIdx: upsert uses get from mem first (#20186) 2021-09-27 11:23:53 -05:00
Jeff Washington (jwash) d8f98b3dcb
AcctIdx: avoid crazy #s on ms_per_age stat (#20184) 2021-09-27 08:25:23 -05:00
Alexander Meißner df6905c3a6
Refactor: Merge MessageProcessor into InvokeContext (#20165)
* Inlines MessageProcessor::execute_instruction() in MessageProcessor::process_message().

* Moves MessageProcessor::create_pre_accounts() into ThisInvokeContext::push().

* Hoists ThisInvokeContext::new() out of loop inside MessageProcessor::process_message().

* Moves MessageProcessor::verify_account_references() and MessageProcessor::verify() into InvokeContext::verify().
2021-09-27 08:28:45 +02:00
Jeff Washington (jwash) c6d189dac2
bg threads wait until next expected event (#20126) 2021-09-24 16:00:41 -05:00
Jeff Washington (jwash) 122206dbb1
AcctIdx: bg loop sleeps when all buckets flushed at current age (#20168) 2021-09-24 13:19:06 -05:00
Jeff Washington (jwash) 1d13594c1c
AcctIdx: store # threads in BucketHolder. Used later. (#20151) 2021-09-23 19:56:44 -05:00
Jeff Washington (jwash) b3bb079d9f
AcctIdx: condense upsert to always use read lock (#20148) 2021-09-23 19:19:27 -05:00
Brooks Prumo 1347b507da
Handle duplicate AppendVec IDs (#20096)
When reconstructing the AccountsDb, if the storages came from full and
incremental snapshots generated on different nodes, it's possible that
the AppendVec IDs could overlap/have duplicates, which would cause the
reconstruction to fail.

This commit handles this issue by unconditionally remapping the
AppendVec ID for every AppendVec.

Fixes #17088
2021-09-23 17:34:32 -05:00
Jeff Washington (jwash) 2ae1e80013
AcctIdx: items needs to load from disk (#20138) 2021-09-23 17:33:54 -05:00
Jeff Washington (jwash) 334d38d200
AcctIdx: generation per thread starts at a random bucket (#20122) 2021-09-23 19:59:37 +00:00
Jeff Washington (jwash) 254d9c8903
range takes a ref (#20140) 2021-09-23 13:57:56 -05:00
Jeff Washington (jwash) 093dd68214
AcctIdx: stats for how much time bg threads wait (#20127) 2021-09-23 18:37:14 +00:00
Jeff Washington (jwash) fe28f17718
AcctIdx: startup causes us to flush remove everything asap (#20121) 2021-09-23 12:17:52 -05:00
Jeff Washington (jwash) 5dae615208
AcctIdx: set age of cache accesses to future (#20125) 2021-09-23 11:43:48 -05:00
Jeff Washington (jwash) fb55e9a9b6
AcctIdx: when stop_flush is disabled, prepare to age (#20113) 2021-09-23 11:42:32 -05:00
Jeff Washington (jwash) 5bbb0da7b8
AcctIdx: stats for in-mem (#20114) 2021-09-23 08:14:53 -05:00
Jeff Washington (jwash) cb9d93525b
AcctIdx: metrics for loading from disk (#20124) 2021-09-23 08:14:08 -05:00
Jeff Washington (jwash) 4d3e32803f
AcctIdx: PreAllocatedAccountMapEntry (#20117) 2021-09-23 08:11:32 -05:00
Trent Nelson 7b365c564f
runtime: remove inactive delegation from stakes cache 2021-09-22 20:50:58 -06:00
Jeff Washington (jwash) 476124de50
AcctIdx: report disk bucket stats (#20120) 2021-09-22 18:17:20 -05:00