[#765] Store blocks on disk instead of in SQLite

* Switch to FsBlockDb for caching CompactBlocks

* Add RustBackend.getLatestHeight() method

* Raise MSRV to 1.60

* Migrate to latest Rust crate API

* Add RustBackend.findBlockMetadata() method

* Add RustBackend.rewindBlockMetadataToHeight() method

* [#765] implementation of FileCompactBlockRepository

* writing block metadata to database

* split write function into smaller easier to test blocks

* testing for FileCompactBlockRepository

* fixed rewinding

* fixed tests

* fixed FileCompactBlockRepositoryTest and SynchronizerFactoryTest

* code review fixes

* updated proto files

* override all functions in FakeRustBackend

* code review fixes

* Fix function body formatting

* Improve clear function clarity

* Use length of string const

* Delete single file instead of directory

* Improve function clarity

* Refactor outputs counting

- Found a typo in intermediary model class JniBlockMeta parameter change of which does not impact encoding from rust to kotlin according to rust layer implementation.

* Check blocks mkdir result

* Remove unnecessary detekt warning suppression

* Refactor buffer size check

* Improve visibility annotations

* Make file finalise obvious and self documenting

* Remove prevHash logging

* Move instantiation to the object itself

* Enrich fixture with default values

* Extend eror message

* Rename benchmark blocks range fixture

* Fix rebase changes

* Improve FileCompactBlockRepositoryTest

- "De-integrated" the test suite - it now works with fixture blocks
- Created needed fixtures for a clear mocked blocks providing
- Enhanced getting of FileCompactBlockRepository in FileCompactBlockRepositoryTest to clarify that it works with mock components

* Fix ktlintFormat findings

* Bump actions/cache from 3.2.4 to 3.2.5 in /.github/actions/setup (#927)

Bumps [actions/cache](https://github.com/actions/cache) from 3.2.4 to 3.2.5.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](627f0f41f6...6998d139dd)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix incorrect dir checking

* More robust min/max checking

* Rewinding to more robust middle value

* Strengthen blocks counts after rewind

* Add check of temp finalized file

* Refactor DatabaseCoordinatorTest

* Rename cache files root directory

- To have the same unified variable name across our repositories - fsBlockDbRoot

* Refactor FileCompactBlockRepositoryTest

- To have these tests more clear
- Fixed FakeRustBackend instantiation bug

* Revert back JniBlockMeta param name

* Delete legacy Cache db files

- Deleted from both the older and the newer legacy locations
- All related db files deleted - rollback files included
- The deletion is run once we try to access the new store blocks on disk root directory
- The deletion check does not throw any exception in case of failure, we just log it in console and try it on the next time
- Related new test added too

* Test refactoring

- Made few changes to improve clarity of provided tests and fixtures
- Prepared few new "failure path" tests
- Enhanced existing tests

* Manual test case

* Simplify error printing

- As we had some commented out code there

* Reset manual tests steps numbering

* [#924] Remove alias from WalletCoordinator

Also make Compose UI the default.  The old UI is deprecated but is still used by the benchmarking tests

* Bump benchmark version

* Protect JniBlockMetadata agains minification

* Enable debuggable while benchmarking

* Enhance benchmark screen waiting

* Fix cache db files deletion

- With this construct we delete all blocks blob metadata files, as well as their sqlite file

* Add new benchmark results

* Remove benchmark operations receiver fix

- As it's not needed after the latest profiler dependency update

* Bump gradle/wrapper-validation-action from 1.0.5 to 1.0.6 (#928)

Bumps [gradle/wrapper-validation-action](https://github.com/gradle/wrapper-validation-action) from 1.0.5 to 1.0.6.
- [Release notes](https://github.com/gradle/wrapper-validation-action/releases)
- [Commits](55e685c48d...8d49e559aa)

---
updated-dependencies:
- dependency-name: gradle/wrapper-validation-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/cache from 3.2.5 to 3.2.6 in /.github/actions/setup (#929)

Bumps [actions/cache](https://github.com/actions/cache) from 3.2.5 to 3.2.6.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](6998d139dd...69d9d449ac)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update DatabaseCoordinator.kt

Move if/else out from log message body

* Update Switch cache to store blocks on disk.md

Reset documentation bullets numbering

* Hide internal constants from public package

* Update manual test user with new UI requirement

* Comment out cargo.toml test values

* Inline fixture blocks metadata with its creation

* Switch to FsBlockDb for caching CompactBlocks

* Add RustBackend.getLatestHeight() method

* Raise MSRV to 1.60

* Migrate to latest Rust crate API

* Add RustBackend.findBlockMetadata() method

* Add RustBackend.rewindBlockMetadataToHeight() method

* Update cargo.lock

* Switch to FsBlockDb for caching CompactBlocks

* Migrate to latest Rust crate API

* [#765] implementation of FileCompactBlockRepository

* writing block metadata to database

* split write function into smaller easier to test blocks

* testing for FileCompactBlockRepository

* fixed rewinding

* fixed tests

* fixed FileCompactBlockRepositoryTest and SynchronizerFactoryTest

* code review fixes

* updated proto files

* override all functions in FakeRustBackend

* code review fixes

* Fix function body formatting

* Improve clear function clarity

* Use length of string const

* Delete single file instead of directory

* Improve function clarity

* Refactor outputs counting

- Found a typo in intermediary model class JniBlockMeta parameter change of which does not impact encoding from rust to kotlin according to rust layer implementation.

* Check blocks mkdir result

* Remove unnecessary detekt warning suppression

* Refactor buffer size check

* Improve visibility annotations

* Make file finalise obvious and self documenting

* Remove prevHash logging

* Move instantiation to the object itself

* Enrich fixture with default values

* Extend eror message

* Rename benchmark blocks range fixture

* Fix rebase changes

* Improve FileCompactBlockRepositoryTest

- "De-integrated" the test suite - it now works with fixture blocks
- Created needed fixtures for a clear mocked blocks providing
- Enhanced getting of FileCompactBlockRepository in FileCompactBlockRepositoryTest to clarify that it works with mock components

* Fix ktlintFormat findings

* Fix incorrect dir checking

* More robust min/max checking

* Rewinding to more robust middle value

* Strengthen blocks counts after rewind

* Add check of temp finalized file

* Refactor DatabaseCoordinatorTest

* Rename cache files root directory

- To have the same unified variable name across our repositories - fsBlockDbRoot

* Refactor FileCompactBlockRepositoryTest

- To have these tests more clear
- Fixed FakeRustBackend instantiation bug

* Revert back JniBlockMeta param name

* Delete legacy Cache db files

- Deleted from both the older and the newer legacy locations
- All related db files deleted - rollback files included
- The deletion is run once we try to access the new store blocks on disk root directory
- The deletion check does not throw any exception in case of failure, we just log it in console and try it on the next time
- Related new test added too

* Test refactoring

- Made few changes to improve clarity of provided tests and fixtures
- Prepared few new "failure path" tests
- Enhanced existing tests

* Manual test case

* Simplify error printing

- As we had some commented out code there

* Reset manual tests steps numbering

* Bump benchmark version

* Protect JniBlockMetadata agains minification

* Enable debuggable while benchmarking

* Enhance benchmark screen waiting

* Fix cache db files deletion

- With this construct we delete all blocks blob metadata files, as well as their sqlite file

* Add new benchmark results

* Remove benchmark operations receiver fix

- As it's not needed after the latest profiler dependency update

* Update DatabaseCoordinator.kt

Move if/else out from log message body

* Update Switch cache to store blocks on disk.md

Reset documentation bullets numbering

* Hide internal constants from public package

* Update manual test user with new UI requirement

* Comment out cargo.toml test values

* Inline fixture blocks metadata with its creation

* Update cargo.lock

* Update Cargo.lock

* Check and document JniBlockMeta params ranges

* Add assert for gradle property

* Use UInt internally

uint

* Change array API to list

* Fix for using correct SDK Synchronizer alias

- Only the older Demo-app UI was impacted.
- Thus also our benchmarking was impacted.

* Apply documentation suggestions from code review

Co-authored-by: Kris Nuttycombe <kris@electriccoin.co>

* Final manual test instructions update

* Fixture block hash deterministically generated from block height

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jack Grigg <jack@electriccoin.co>
Co-authored-by: Jack Grigg <jack@z.cash>
Co-authored-by: Honza <rychnovsky.honza@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
Co-authored-by: Kris Nuttycombe <kris@electriccoin.co>
This commit is contained in:
Alex 2023-03-08 16:04:04 +01:00 committed by GitHub
parent 5366353e6b
commit 9ee5a4e568
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
47 changed files with 1425 additions and 370 deletions

View File

@ -17,7 +17,7 @@ The way the SDK is initialized has changed. The `Initializer` object has been r
SDK initialization also now requires access to the seed bytes at two times: 1. during new wallet creation and 2. during upgrade of an existing wallet to SDK 1.10 due to internal data migrations. To handle case #2, client should wrap `Synchronizer.new()` with a try-catch for `InitializerException.SeedRequired`. Clients can pass `null` to try to initialize the SDK without the seed, then try again if the exception is thrown to indicate the seed is needed. This pattern future-proofs initialization, as the seed may be required by future SDK updates.
`Synchronizer.stop()` has been removed. `Synchronizer.new()` now returns an instance that implements the `Closeable` interface. This effectively means that calls to `stop()` are replaced with `close()`. This change also enables greater safety within client applications, as the Closeable interface can be hidden from global synchronizer instances. For exmaple:
`Synchronizer.stop()` has been removed. `Synchronizer.new()` now returns an instance that implements the `Closeable` interface. This effectively means that calls to `stop()` are replaced with `close()`. This change also enables greater safety within client applications, as the Closeable interface can be hidden from global synchronizer instances. For example:
```
val synchronizerFlow: Flow<Synchronizer> = callbackFlow<Synchronizer> {
val closeableSynchronizer: CloseableSynchronizer = Synchronizer.new(...)

View File

@ -77,6 +77,8 @@ tasks {
"ZCASH_RELEASE_KEY_ALIAS_PASSWORD" to "",
"IS_SIGN_RELEASE_BUILD_WITH_DEBUG_KEY" to "false",
"IS_DEBUGGABLE_WHILE_BENCHMARKING" to "false"
)
val warnings = expectedPropertyValues.filter { (key, value) ->

View File

@ -11,8 +11,15 @@ android {
defaultConfig {
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
// To enable benchmarking for emulators, although only a physical device us gives real results
testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "EMULATOR"
var suppressOptions = "EMULATOR"
// To enable debugging while running benchmark tests, although it reduces their performance
if (project.property("IS_DEBUGGABLE_WHILE_BENCHMARKING").toString().toBoolean()) {
suppressOptions += ",DEBUGGABLE"
}
testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = suppressOptions
// To simplify module variants, we assume to run benchmarking against mainnet only
missingDimensionStrategy("network", "zcashmainnet")
}

View File

@ -82,7 +82,7 @@ class SyncBlockchainBenchmark : UiTestPrerequisites() {
// Open toolbar overflow menu
device.findObject(By.desc("More options")).clickAndWaitFor(Until.newWindow(), 2.seconds) // NON-NLS
// Click on the reset sdk menu item
device.findObject(By.text("Reset SDK")).click() // NON-NLS
device.findObject(By.text("Reset SDK")).clickAndWaitFor(Until.newWindow(), 2.seconds) // NON-NLS
device.waitForIdle()
}

View File

@ -90,6 +90,11 @@ android {
initWith(buildTypes.getByName("release"))
signingConfig = signingConfigs.getByName("debug")
matchingFallbacks += listOf("release")
// To enable debugging while running benchmark tests, although it reduces their performance
if (project.property("IS_DEBUGGABLE_WHILE_BENCHMARKING").toString().toBoolean()) {
isDebuggable = true
}
}
}

View File

@ -38,17 +38,6 @@
android:shell="true"
tools:targetApi="29" />
<!-- To bypass "The DROP_SHADER_CACHE broadcast was not received." error
see https://issuetracker.google.com/issues/258619948 -->
<receiver
android:name="androidx.profileinstaller.ProfileInstallReceiver"
android:exported="true"
android:permission="android.permission.DUMP">
<intent-filter>
<action android:name="androidx.profileinstaller.action.BENCHMARK_OPERATION" />
</intent-filter>
</receiver>
</application>
</manifest>

View File

@ -13,7 +13,7 @@ import cash.z.ecc.android.sdk.model.BlockHeight
import cash.z.ecc.android.sdk.model.ZcashNetwork
import cash.z.ecc.android.sdk.model.defaultForNetwork
import co.electriccoin.lightwallet.client.ext.BenchmarkingExt
import co.electriccoin.lightwallet.client.fixture.BlockRangeFixture
import co.electriccoin.lightwallet.client.fixture.BenchmarkingBlockRangeFixture
import co.electriccoin.lightwallet.client.model.LightWalletEndpoint
import kotlinx.coroutines.ExperimentalCoroutinesApi
import kotlinx.coroutines.channels.awaitClose
@ -78,7 +78,7 @@ class SharedViewModel(application: Application) : AndroidViewModel(application)
lightWalletEndpoint = LightWalletEndpoint.defaultForNetwork(network),
seed = seedBytes,
birthday = if (BenchmarkingExt.isBenchmarking()) {
BlockHeight.new(ZcashNetwork.Mainnet, BlockRangeFixture.new().start)
BlockHeight.new(ZcashNetwork.Mainnet, BenchmarkingBlockRangeFixture.new().start)
} else {
birthdayHeight.value
},
@ -127,7 +127,8 @@ class SharedViewModel(application: Application) : AndroidViewModel(application)
.onFirst {
val didDelete = Synchronizer.erase(
appContext = getApplication(),
network = ZcashNetwork.fromResources(getApplication())
network = ZcashNetwork.fromResources(getApplication()),
alias = OLD_UI_SYNCHRONIZER_ALIAS
)
Twig.debug { "SDK erase result: $didDelete" }
}

View File

@ -14,7 +14,7 @@ Start by making sure the command line with Gradle works first, because **all the
1. Install JVM 11 or greater on your system. Our setup has been tested with Java 11-17. Although a variety of JVM distributions are available and should work, we have settled on recommending [Adoptium/Temurin](https://adoptium.net), because this is the default distribution used by Gradle toolchains. For Windows or Linux, be sure that the `JAVA_HOME` environment variable points to the right Java version. Note: If you switch from a newer to an older JVM version, you may see an error like the following `> com.android.ide.common.signing.KeytoolException: Failed to read key AndroidDebugKey from store "~/.android/debug.keystore": Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available`. A solution is to delete the debug keystore and allow it to be re-generated.
1. Android Studio has an embedded JVM, although running Gradle tasks from the command line requires a separate JVM to be installed. Our Gradle scripts are configured to use toolchains to automatically install the correct JVM version.
1. Configure Rust
1. [Install Rust](https://www.rust-lang.org/learn/get-started). You will need Rust 1.59 or greater. If you install with `rustup` then you are guaranteed to get a compatible Rust version. If you use system packages, check the provided version.
1. [Install Rust](https://www.rust-lang.org/learn/get-started). You will need Rust 1.60 or greater. If you install with `rustup` then you are guaranteed to get a compatible Rust version. If you use system packages, check the provided version.
1. macOS with Homebrew
1. `brew install rustup`
1. `rustup-init`

View File

@ -102,6 +102,62 @@ commits of that date. Generate tests results with the Android Studio run configu
BUILD SUCCESSFUL in 6m 12s
```
#### Feb 18, 2023:
- SDK version: `1.14.0-beta01`
- Git branch: `765-Store_blocks_on_disk_instead_of_in_SQLite`
- Note: Switched to storing cache blocks blob files on disk instead of in SQLite database
- Device:
- Pixel 6 - Android 13:
```
Starting 3 tests on Pixel 6 - 13
StartupBenchmark_appStartup
timeToInitialDisplayMs min 248.6, median 287.0, max 385.7
Traces: Iteration 0 1 2 3 4
StartupBenchmark_tracesSdkStartup
ADDRESS_SCREENMs min 935.5, median 943.3, max 1,013.7
SAPLING_ADDRESSMs min 2.1, median 3.7, max 6.9
TRANSPARENT_ADDRESSMs min 2.2, median 3.3, max 7.5
UNIFIED_ADDRESSMs min 2.2, median 3.8, max 5.8
Traces: Iteration 0 1 2 3 4
SyncBlockchainBenchmark_tracesSyncBlockchain
BALANCE_SCREENMs min 67,376.0, median 67,614.0, max 73,651.2
BLOCKCHAIN_SYNCMs min 66,449.6, median 66,624.1, max 72,865.5
DOWNLOADMs min 52,245.6, median 52,442.0, max 56,489.1
SCANMs min 14,061.2, median 14,074.5, max 16,261.6
VALIDATIONMs min 114.6, median 120.7, max 129.1
Traces: Iteration 0 1 2
BUILD SUCCESSFUL in 5m 9s
```
- Pixel 3a - Android 12:
```
Starting 3 tests on Pixel 3a - 12
StartupBenchmark_appStartup
timeToInitialDisplayMs min 475.8, median 513.6, max 531.8
Traces: Iteration 0 1 2 3 4
StartupBenchmark_tracesSdkStartup
ADDRESS_SCREENMs min 978.6, median 1,094.6, max 1,156.1
SAPLING_ADDRESSMs min 4.7, median 6.9, max 7.3
TRANSPARENT_ADDRESSMs min 5.3, median 5.5, max 11.6
UNIFIED_ADDRESSMs min 3.9, median 7.5, max 10.8
Traces: Iteration 0 1 2 3 4
SyncBlockchainBenchmark_tracesSyncBlockchain
BALANCE_SCREENMs min 66,203.0, median 66,274.0, max 66,472.4
BLOCKCHAIN_SYNCMs min 65,279.5, median 65,417.6, max 65,570.8
DOWNLOADMs min 34,191.6, median 34,392.5, max 34,447.2
SCANMs min 30,870.4, median 30,944.6, max 30,981.0
VALIDATIONMs min 142.4, median 143.2, max 154.5
Traces: Iteration 0 1 2
BUILD SUCCESSFUL in 6m 58s
```

View File

@ -13,18 +13,18 @@ of automatic user data backup to user's cloud storage. Our sapling files are qui
# Download files steps
1. Remove a previous version of the demo-app from the emulator, if there is any
2. Install the latest version of the demo-app from the latest commit on the **Main** branch
3. Run the demo-app on selected emulator
4. Once it's opened on the Home screen, change the wallet seed phrase to your preferred one to have some funds
1. Install the latest version of the demo-app from the latest commit on the **Main** branch
1. Run the demo-app on selected emulator
1. Once it's opened on the Home screen, change the wallet seed phrase to your preferred one to have some funds
available, which can be spent for the purpose of this test
5. Go to the Send screen and wait for Downloading and Syncing processes to finish
6. Then type the ZEC amount you want to send and the Address to which you want the Zec amount sent
7. Wait for send confirmation
8. Sapling params files should be now downloaded in the preferred location. Open Device File Explorer from Android
1. Go to the Send screen and wait for Downloading and Syncing processes to finish
1. Then type the ZEC amount you want to send and the Address to which you want the Zec amount sent
1. Wait for send confirmation
1. Sapling params files should be now downloaded in the preferred location. Open Device File Explorer from Android
Studio bottom-left corner, select the same emulator device from the top. Go to
`/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/no_backup/co.electricoin.zcash`, which should be created
automatically
9. Now verify there both of our sapling params files (`sapling-spend.params`, `sapling-output.params`) placed in the
1. Now verify there both of our sapling params files (`sapling-spend.params`, `sapling-output.params`) placed in the
`no_backup/co.electricoin.zcash` folder
# Check result

View File

@ -1,55 +0,0 @@
# About
This manual test case provides information on how to manually test an implemented action of moving all of our databases
files from default `/databases/` to preferred `/no_backup/co.electricoin.zcash` directory. The benefit of this approach
is that the content `no_backup` folder is not part of automatic user data backup to user's cloud storage. Our databases
can contain potentially big and sensitive data.
The move feature takes all related files (database file itself as well as `journal` and `wal` rollback files) and moves
them only once on app start (before first database access) when a client app uses an updated version of this SDK.
# Prerequisite
- Installed [Android Studio](https://developer.android.com/studio)
- Ideally two emulators with min and max supported API level
- A working git client
- Cloned [Zcash Android SDK repository](https://github.com/zcash/zcash-android-wallet-sdk)
# Prepare steps
1. Install a previous version of the SDK and its demo-app to create database files in the original `database` folder
2. Switch back to commit **Bump version to 1.8.0-beta01 [3fda6da]** from Jul 11 2022 on the **Main** branch in your
git client, or with this git command `git checkout 3fda6da1cae5b83174e5b1e020c91dfe95d93458`
3. Update dependencies lock (if needed) and sync Gradle files
4. Run the demo-app on selected emulator
5. Once it's opened go through the app to let the SDK create all the database files. Visit these screens step by step
from the side menu:
1. Get Balance
2. List Transactions
3. List UTXOs
6. Open Device File Explorer from Android Studio bottom-left corner, select the same emulator device from the top
drop-down menu
7. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/databases`
8. Verify there are `data.db`, `cache.db` and `utxos.db` files (their names can vary, depends on the current build
variant). There can be several rollback files created.
# Move steps
1. Install the newer version of the SDK and its demo-app to the same device to check the database files move operation
result
1. Switch to the latest commit on the **Main** branch in your git client
2. Update dependencies lock (if needed) and sync Gradle files
3. Run the demo-app on the same emulator device as previously
2. Once the app is opened go through the same steps as previously to let the SDK apply the move mechanisms to all our
database files. Visit these screens step by step from the side menu:
1. Get Balance
2. List Transactions
3. List UTXOs
3. Go to the Device File Explorer from Android Studio bottom-left corner again
4. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/databases` again, now there shouldn't be any files placed
in the `database` folder
5. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/no_backup/co.electricoin.zcash`, which should be created
automatically
6. Now verify there are the same files placed in the `no_backup/co.electricoin.zcash` folder as in `databases` were
7. To be sure everything is alright, just visit several screens from the side-menu and see no unexpected behavior
# Check result
Ideally run this test (Prepare and Move steps) for both emulators (min and max supported API level) to ensure the
correct functionality on both Android version. There is a difference in implementation for these Android versions, but
the result should be the same.

View File

@ -14,32 +14,32 @@ the preferred location `/no_backup/co.electricoin.zcash/`. The benefit of this a
# Prepare steps
1. Install a previous version of the SDK and its demo-app to create sapling files in the original `cache/params` folder
2. Switch back to commit **Check sapling files size [12c23dd0]** from Aug 26 2022 on the **Main** branch in your
1. Switch back to commit **Check sapling files size [12c23dd0]** from Aug 26 2022 on the **Main** branch in your
git client, or with this git command `git checkout 12c23dd054c687431aaf51bfc5f67d5dbc08625b`
3. Update dependencies lock (if needed) and sync Gradle files
4. Run the demo-app on selected emulator
5. Once it's opened on the Home screen, change the wallet seed phrase to your preferred one to have some funds
1. Update dependencies lock (if needed) and sync Gradle files
1. Run the demo-app on selected emulator
1. Once it's opened on the Home screen, change the wallet seed phrase to your preferred one to have some funds
available, which can be spent for the purpose of this test
6. Go to the Send screen and wait for Downloading and Syncing processes to finish
7. Then type the ZEC amount you want to send and the Address to which you want the Zec amount sent
8. Wait for send confirmation
9. Sapling params files should be now moved to the original location. Open Device File
1. Go to the Send screen and wait for Downloading and Syncing processes to finish
1. Then type the ZEC amount you want to send and the Address to which you want the Zec amount sent
1. Wait for send confirmation
1. Sapling params files should be now moved to the original location. Open Device File
Explorer from Android Studio bottom-left corner, select the same emulator device from the top
drop-down menu. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/cache/params`
10. Verify there are `sapling-spend.params` and `sapling-output.params`
1. Verify there are `sapling-spend.params` and `sapling-output.params`
# Move steps
1. Install the newer version of the SDK and its demo-app to the same device to check the database files move operation
result
1. Switch to the latest commit on the **Main** branch in your git client
2. Update dependencies lock (if needed) and sync Gradle files
3. Run the demo-app on the same emulator device as previously
2. Once the app is opened, go to the Device File Explorer from Android Studio bottom-left corner again
3. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/cache/params` again, now there shouldn't be our sapling
1. Update dependencies lock (if needed) and sync Gradle files
1. Run the demo-app on the same emulator device as previously
1. Once the app is opened, go to the Device File Explorer from Android Studio bottom-left corner again
1. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/cache/params` again, now there shouldn't be our sapling
params files placed in the folder and the folder `/params/` should be missing
4. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/no_backup/co.electricoin.zcash`, which should be created
1. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/no_backup/co.electricoin.zcash`, which should be created
automatically
5. Now verify there are the same files placed in the `no_backup/co.electricoin.zcash` folder as in `cache/params` were
1. Now verify there are the same files placed in the `no_backup/co.electricoin.zcash` folder as in `cache/params` were
# Check result
Ideally run this test (Prepare and Move steps) for both emulators (min and max supported API level) to ensure the

View File

@ -0,0 +1,49 @@
# About
This manual test case provides information on how to manually test the implemented action of switching our
mechanism of persisting CompactBlocks cache from in database storage to on disk storage.
The benefit of this approach is that the storing blocks on-disk does not require storing of blobs into
the sqlite database, which can cause poor performance as the database grows large. Instead,
we store them on disk and insert only a limited portion of a block information (metadata) into the database.
Observed result of this manual test should be:
- there should be no Cache database in the older (`/databases/`) or the newer (`/no_backup/co.electricoin.zcash/`) legacy locations
- files containing protobuf-encoded `CompactBlock`s stored in `/no_backup/co.electricoin.zcash/blocks/` location
# Prerequisites
- Install [Android Studio](https://developer.android.com/studio)
- Install an [emulator](https://developer.android.com/studio/run/emulator) to Android Studio
- Install a `git` client
- Clone the [Zcash Android SDK repository](https://github.com/zcash/zcash-android-wallet-sdk)
# Prepare steps
1. Create some existing persistent wallet state using a version prior to the introduction of the disk-based cache. To do this, check out commit [zcash/zcash-android-wallet-sdk#910](https://github.com/zcash/zcash-android-wallet-sdk/pull/910) (currently commit `a67d287e5cc90fe3a774b02174dca1b32331058c`).
1. Update dependencies lock (if needed) and sync Gradle files
1. Select one of the **Mainnet** build variants from **Build Variant** window
1. Build and run the demo-app on selected emulator
1. Once the app is open, select e.g. _Alyssa P. Hacker_ secret phrase and then let the SDK create the Cache database
files with auto-start syncing on the home screen
1. Wait a moment to be sure that the sync mechanism has already been initialized and started to fill in the Cache
database with CompactBlocks entries.
1. Open Device File Explorer in Android Studio, select the same emulator device from the top drop-down menu
1. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/no_backup/co.electricoin.zcash/`
1. Verify there are `cache.sqlite3` and possibly some rollback files too (suffixed with `journal` or `wal`). The
file names can vary, depending on the current build variant.
# Check steps
1. Install the newer version of the SDK and its demo-app to the same device to check the switch to the new type of
the CompactBlocks cache storing
1. Switch to the latest commit on the **Main** branch in your git client
1. Update dependencies lock (if needed) and sync Gradle files
1. Run the demo-app on the same emulator device as previously
1. Once the app is opened go through the same steps as previously to let the SDK apply the new cache storing
mechanisms
1. Open the Device File Explorer in the Android Studio again
1. Go to `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/no_backup/co.electricoin.zcash/`
1. Verify there is a new `zcash_sdk_[network_name]_fs_cache` directory in place
1. Verify it contains a `blockmeta.sqlite` block metadata database file
1. Also verify that this directory contains a `blocks` directory with new block cache files in it
1. Verify there is no `cache.sqlite3` database file, and check that no rollback files (suffixed with `journal` or
`wal`) are present. The file names can vary, depending on the current build variant.
1. Inspect older legacy database folder `/data/data/cash.z.ecc.android.sdk.demoapp.mainnet/databases/`, which
also should not contain `cache.sqlite3` or rollback files.

View File

@ -64,6 +64,12 @@ ZCASH_RELEASE_KEY_ALIAS_PASSWORD=
# Switch this property to true only if you need to sign the release build with a debug key for local development.
IS_SIGN_RELEASE_BUILD_WITH_DEBUG_KEY=false
# Make the benchmarking target app debuggable. This is supposed to be used only for debugging while benchmarking.
# To get more realistic results, keep it turned off, please. The debuggable option drastically reduces runtime
# performance in order to support debugging features. Debuggable affects execution speed in ways that mean benchmark
# improvements might not carry over to a real user experience (or even regress release performance).
IS_DEBUGGABLE_WHILE_BENCHMARKING=false
# Versions
ANDROID_MIN_SDK_VERSION=24
ANDROID_TARGET_SDK_VERSION=33
@ -103,7 +109,7 @@ ANDROIDX_NAVIGATION_VERSION=2.5.3
ANDROIDX_NAVIGATION_COMPOSE_VERSION=2.5.3
ANDROIDX_NAVIGATION_FRAGMENT_VERSION=2.4.2
ANDROIDX_PAGING_VERSION=2.1.2
ANDROIDX_PROFILE_INSTALLER_VERSION=1.3.0-alpha03
ANDROIDX_PROFILE_INSTALLER_VERSION=1.3.0-beta01
ANDROIDX_ROOM_VERSION=2.5.0
ANDROIDX_SECURITY_CRYPTO_VERSION=1.1.0-alpha04
ANDROIDX_TEST_JUNIT_VERSION=1.1.5

View File

@ -9,9 +9,9 @@ class BlockRangeFixtureTest {
@Test
@SmallTest
fun compare_default_values() {
BlockRangeFixture.new().also {
assertEquals(BlockRangeFixture.BLOCK_HEIGHT_LOWER_BOUND, it.start)
assertEquals(BlockRangeFixture.BLOCK_HEIGHT_UPPER_BOUND, it.endInclusive)
BenchmarkingBlockRangeFixture.new().also {
assertEquals(BenchmarkingBlockRangeFixture.BLOCK_HEIGHT_LOWER_BOUND, it.start)
assertEquals(BenchmarkingBlockRangeFixture.BLOCK_HEIGHT_UPPER_BOUND, it.endInclusive)
}
}
}

View File

@ -2,7 +2,10 @@ package co.electriccoin.lightwallet.client.fixture
import androidx.annotation.VisibleForTesting
object BlockRangeFixture {
/**
* Used for getting mocked blocks range for benchmarking purposes.
*/
object BenchmarkingBlockRangeFixture {
// Be aware that changing these bounds values in a broader range may result in a timeout reached in
// SyncBlockchainBenchmark. So if changing these, don't forget to align also the test timeout in

View File

@ -0,0 +1,21 @@
package co.electriccoin.lightwallet.client.fixture
import co.electriccoin.lightwallet.client.model.BlockHeightUnsafe
/**
* Used for getting mocked blocks range for processing and persisting compact blocks purposes.
*/
internal object FileBlockRangeFixture {
@Suppress("MagicNumber")
private val BLOCK_HEIGHT_LOWER_BOUND = BlockHeightUnsafe(500_000L)
@Suppress("MagicNumber")
private val BLOCK_HEIGHT_UPPER_BOUND = BlockHeightUnsafe(500_009L)
fun new(
lowerBound: BlockHeightUnsafe = BLOCK_HEIGHT_LOWER_BOUND,
upperBound: BlockHeightUnsafe = BLOCK_HEIGHT_UPPER_BOUND
): ClosedRange<BlockHeightUnsafe> {
return lowerBound..upperBound
}
}

View File

@ -0,0 +1,28 @@
package co.electriccoin.lightwallet.client.fixture
import androidx.annotation.VisibleForTesting
import cash.z.wallet.sdk.internal.rpc.CompactFormats.CompactBlock
import co.electriccoin.lightwallet.client.model.BlockHeightUnsafe
/**
* Used for getting mocked blocks list for processing and persisting compact blocks purposes.
*/
@VisibleForTesting(otherwise = VisibleForTesting.PACKAGE_PRIVATE)
object ListOfCompactBlocksFixture {
val DEFAULT_FILE_BLOCK_RANGE = FileBlockRangeFixture.new()
fun new(
blocksHeightRange: ClosedRange<BlockHeightUnsafe> = DEFAULT_FILE_BLOCK_RANGE
): Sequence<CompactBlock> {
val blocks = mutableListOf<CompactBlock>()
for (blockHeight in blocksHeightRange.start.value..blocksHeightRange.endInclusive.value) {
blocks.add(
SingleCompactBlockFixture.new(blockHeight = blockHeight)
)
}
return blocks.asSequence()
}
}

View File

@ -0,0 +1,28 @@
package co.electriccoin.lightwallet.client.fixture
import cash.z.wallet.sdk.internal.rpc.CompactFormats.CompactBlock
import com.google.protobuf.ByteString
import com.google.protobuf.kotlin.toByteStringUtf8
/**
* Used for getting single mocked compact block for processing and persisting purposes.
*/
internal object SingleCompactBlockFixture {
const val DEFAULT_BLOCK_HEIGHT = 500_000L
// Keep this because it makes test assertions easier
const val DEFAULT_BLOCK_HASH = DEFAULT_BLOCK_HEIGHT
internal fun heightToFixtureHash(height: Long) = height.toString().toByteStringUtf8()
fun new(
blockHeight: Long = DEFAULT_BLOCK_HEIGHT,
blockHash: ByteString = heightToFixtureHash(blockHeight)
): CompactBlock {
return CompactBlock.newBuilder()
.setHeight(blockHeight)
.setHash(blockHash)
.build()
}
}

View File

@ -5,7 +5,7 @@ import cash.z.wallet.sdk.internal.rpc.CompactTxStreamerGrpc
import cash.z.wallet.sdk.internal.rpc.Service
import co.electriccoin.lightwallet.client.BlockingLightWalletClient
import co.electriccoin.lightwallet.client.ext.BenchmarkingExt
import co.electriccoin.lightwallet.client.fixture.BlockRangeFixture
import co.electriccoin.lightwallet.client.fixture.BenchmarkingBlockRangeFixture
import co.electriccoin.lightwallet.client.model.BlockHeightUnsafe
import co.electriccoin.lightwallet.client.model.LightWalletEndpoint
import co.electriccoin.lightwallet.client.model.LightWalletEndpointInfoUnsafe
@ -53,7 +53,7 @@ internal class BlockingLightWalletClientImpl private constructor(
if (BenchmarkingExt.isBenchmarking()) {
// We inject a benchmark test blocks range at this point to process only a restricted range of blocks
// for a more reliable benchmark results.
Response.Success(BlockHeightUnsafe(BlockRangeFixture.new().endInclusive))
Response.Success(BlockHeightUnsafe(BenchmarkingBlockRangeFixture.new().endInclusive))
} else {
val response = requireChannel().createStub(singleRequestTimeout)
.getLatestBlock(Service.ChainSpec.newBuilder().build())

View File

@ -5,7 +5,7 @@ import cash.z.wallet.sdk.internal.rpc.CompactTxStreamerGrpcKt
import cash.z.wallet.sdk.internal.rpc.Service
import co.electriccoin.lightwallet.client.CoroutineLightWalletClient
import co.electriccoin.lightwallet.client.ext.BenchmarkingExt
import co.electriccoin.lightwallet.client.fixture.BlockRangeFixture
import co.electriccoin.lightwallet.client.fixture.BenchmarkingBlockRangeFixture
import co.electriccoin.lightwallet.client.model.BlockHeightUnsafe
import co.electriccoin.lightwallet.client.model.LightWalletEndpoint
import co.electriccoin.lightwallet.client.model.LightWalletEndpointInfoUnsafe
@ -55,7 +55,7 @@ internal class CoroutineLightWalletClientImpl private constructor(
if (BenchmarkingExt.isBenchmarking()) {
// We inject a benchmark test blocks range at this point to process only a restricted range of blocks
// for a more reliable benchmark results.
Response.Success(BlockHeightUnsafe(BlockRangeFixture.new().endInclusive))
Response.Success(BlockHeightUnsafe(BenchmarkingBlockRangeFixture.new().endInclusive))
} else {
val response = requireChannel().createStub(singleRequestTimeout)
.getLatestBlock(Service.ChainSpec.newBuilder().build())

View File

@ -15,53 +15,53 @@ option swift_prefix = "";
// 2. Detect a spend of your shielded Sapling notes
// 3. Update your witnesses to generate new Sapling spend proofs.
message CompactBlock {
uint32 protoVersion = 1; // the version of this wire format, for storage
uint64 height = 2; // the height of this block
bytes hash = 3; // the ID (hash) of this block, same as in block explorers
bytes prevHash = 4; // the ID (hash) of this block's predecessor
uint32 time = 5; // Unix epoch time when the block was mined
bytes header = 6; // (hash, prevHash, and time) OR (full header)
repeated CompactTx vtx = 7; // zero or more compact transactions from this block
uint32 protoVersion = 1; // the version of this wire format, for storage
uint64 height = 2; // the height of this block
bytes hash = 3; // the ID (hash) of this block, same as in block explorers
bytes prevHash = 4; // the ID (hash) of this block's predecessor
uint32 time = 5; // Unix epoch time when the block was mined
bytes header = 6; // (hash, prevHash, and time) OR (full header)
repeated CompactTx vtx = 7; // zero or more compact transactions from this block
}
// CompactTx contains the minimum information for a wallet to know if this transaction
// is relevant to it (either pays to it or spends from it) via shielded elements
// only. This message will not encode a transparent-to-transparent transaction.
message CompactTx {
uint64 index = 1; // the index within the full block
bytes hash = 2; // the ID (hash) of this transaction, same as in block explorers
uint64 index = 1; // the index within the full block
bytes hash = 2; // the ID (hash) of this transaction, same as in block explorers
// The transaction fee: present if server can provide. In the case of a
// stateless server and a transaction with transparent inputs, this will be
// unset because the calculation requires reference to prior transactions.
// in a pure-Sapling context, the fee will be calculable as:
// valueBalance + (sum(vPubNew) - sum(vPubOld) - sum(tOut))
uint32 fee = 3;
// The transaction fee: present if server can provide. In the case of a
// stateless server and a transaction with transparent inputs, this will be
// unset because the calculation requires reference to prior transactions.
// in a pure-Sapling context, the fee will be calculable as:
// valueBalance + (sum(vPubNew) - sum(vPubOld) - sum(tOut))
uint32 fee = 3;
repeated CompactSaplingSpend spends = 4; // inputs
repeated CompactSaplingOutput outputs = 5; // outputs
repeated CompactOrchardAction actions = 6;
repeated CompactSaplingSpend spends = 4; // inputs
repeated CompactSaplingOutput outputs = 5; // outputs
repeated CompactOrchardAction actions = 6;
}
// CompactSaplingSpend is a Sapling Spend Description as described in 7.3 of the Zcash
// protocol specification.
message CompactSaplingSpend {
bytes nf = 1; // nullifier (see the Zcash protocol specification)
bytes nf = 1; // nullifier (see the Zcash protocol specification)
}
// output is a Sapling Output Description as described in section 7.4 of the
// Zcash protocol spec. Total size is 948.
message CompactSaplingOutput {
bytes cmu = 1; // note commitment u-coordinate
bytes epk = 2; // ephemeral public key
bytes ciphertext = 3; // first 52 bytes of ciphertext
bytes cmu = 1; // note commitment u-coordinate
bytes epk = 2; // ephemeral public key
bytes ciphertext = 3; // first 52 bytes of ciphertext
}
// https://github.com/zcash/zips/blob/main/zip-0225.rst#orchard-action-description-orchardaction
// (but not all fields are needed)
message CompactOrchardAction {
bytes nullifier = 1; // [32] The nullifier of the input note
bytes cmx = 2; // [32] The x-coordinate of the note commitment for the output note
bytes ephemeralKey = 3; // [32] An encoding of an ephemeral Pallas public key
bytes ciphertext = 4; // [52] The note plaintext component of the encCiphertext field
}
bytes nullifier = 1; // [32] The nullifier of the input note
bytes cmx = 2; // [32] The x-coordinate of the note commitment for the output note
bytes ephemeralKey = 3; // [32] An encoding of an ephemeral Pallas public key
bytes ciphertext = 4; // [52] The note plaintext component of the encCiphertext field
}

View File

@ -185,4 +185,4 @@ service CompactTxStreamer {
rpc GetLightdInfo(Empty) returns (LightdInfo) {}
// Testing-only, requires lightwalletd --ping-very-insecure (do not enable in production)
rpc Ping(Duration) returns (PingResponse) {}
}
}

16
sdk-lib/Cargo.lock generated
View File

@ -2208,9 +2208,9 @@ dependencies = [
[[package]]
name = "zcash_client_backend"
version = "0.6.1"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e90362fd61a41f694c072d8db0f8dd59a1fdc902cab3602c42e755d1b9882831"
checksum = "54c054a049b69506098b5fa44830d4196a8bbea7bee9762718f251f1c4d8277e"
dependencies = [
"base64",
"bech32",
@ -2240,9 +2240,9 @@ dependencies = [
[[package]]
name = "zcash_client_sqlite"
version = "0.4.2"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2b1bfd170ee7e73b390dac2799ffbc8bb83724aa3e3cb24f5e83089c97afefd"
checksum = "1df5fd0152fd7207100581918ce772348266f1173cfb0f0a3f3900ac824cacb5"
dependencies = [
"bs58",
"group",
@ -2285,9 +2285,9 @@ dependencies = [
[[package]]
name = "zcash_primitives"
version = "0.9.1"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0f9a45953c4ddd81d68f45920955707f45c8926800671f354dd13b97507edf28"
checksum = "b6879bd4026d9269a41ca91858f453b523f30824288248148211e1cab23b3e0d"
dependencies = [
"aes",
"bip0039",
@ -2321,9 +2321,9 @@ dependencies = [
[[package]]
name = "zcash_proofs"
version = "0.9.0"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77381adc72286874e563ee36ba99953946abcbd195ada45440a2754ca823d407"
checksum = "28ca180a8138ae6e2de2b88573ed19dd57798f42a79a00d992b4d727132c7081"
dependencies = [
"bellman",
"blake2b_simd",

View File

@ -8,7 +8,7 @@ authors = [
description = "JNI backend for the Android wallet SDK"
publish = false
edition = "2018"
rust-version = "1.59"
rust-version = "1.60"
[dependencies]
failure = "0.1"
@ -20,10 +20,10 @@ schemer = "0.2"
secp256k1 = "0.21"
secrecy = "0.8"
zcash_address = "0.2"
zcash_client_backend = { version = "0.6", features = ["transparent-inputs", "unstable"] }
zcash_client_sqlite = { version = "0.4", features = ["transparent-inputs", "unstable"] }
zcash_primitives = "0.9"
zcash_proofs = "0.9"
zcash_client_backend = { version = "0.7", features = ["transparent-inputs", "unstable"] }
zcash_client_sqlite = { version = "0.5", features = ["transparent-inputs", "unstable"] }
zcash_primitives = "0.10"
zcash_proofs = "0.10"
# Logging
log-panics = "2.0.0"
@ -45,11 +45,11 @@ libc = "0.2"
## Uncomment this to test someone else's librustzcash changes in a branch
#[patch.crates-io]
#zcash_address = { git = "https://github.com/zcash/librustzcash", branch = "branch-name" }
#zcash_client_backend = { git = "https://github.com/zcash/librustzcash", branch = "branch-name" }
#zcash_client_sqlite = { git = "https://github.com/zcash/librustzcash", branch = "branch-name" }
#zcash_primitives = { git = "https://github.com/zcash/librustzcash", branch = "branch-name" }
#zcash_proofs = { git = "https://github.com/zcash/librustzcash", branch = "branch-name" }
#zcash_address = { git = "https://github.com/zcash/librustzcash", branch = "main" }
#zcash_client_backend = { git = "https://github.com/zcash/librustzcash", branch = "main" }
#zcash_client_sqlite = { git = "https://github.com/zcash/librustzcash", branch = "main" }
#zcash_primitives = { git = "https://github.com/zcash/librustzcash", branch = "main" }
#zcash_proofs = { git = "https://github.com/zcash/librustzcash", branch = "main" }
[features]
mainnet = ["zcash_client_sqlite/mainnet"]

View File

@ -4,6 +4,7 @@ import androidx.test.filters.SmallTest
import cash.z.ecc.android.sdk.internal.db.DatabaseCoordinator
import cash.z.ecc.android.sdk.internal.ext.existsSuspend
import cash.z.ecc.android.sdk.test.getAppContext
import cash.z.ecc.fixture.DatabaseCacheFilesRootFixture
import cash.z.ecc.fixture.DatabaseNameFixture
import cash.z.ecc.fixture.DatabasePathFixture
import kotlinx.coroutines.ExperimentalCoroutinesApi
@ -30,16 +31,16 @@ class DatabaseCoordinatorTest {
@Test
@SmallTest
@OptIn(ExperimentalCoroutinesApi::class)
fun database_cache_file_creation_test() = runTest {
val directory = File(DatabasePathFixture.new())
val fileName = DatabaseNameFixture.newDb(name = DatabaseCoordinator.DB_CACHE_NAME)
val expectedFilePath = File(directory, fileName).path
fun database_cache_root_directory_creation_test() = runTest {
val parentDirectory = File(DatabasePathFixture.new())
val destinationDirectory = DatabaseCacheFilesRootFixture.newCacheRoot()
val expectedDirectoryPath = File(parentDirectory, destinationDirectory).path
dbCoordinator.cacheDbFile(
dbCoordinator.fsBlockDbRoot(
DatabaseNameFixture.TEST_DB_NETWORK,
DatabaseNameFixture.TEST_DB_ALIAS
).also { resultFile ->
assertEquals(expectedFilePath, resultFile.absolutePath)
assertEquals(expectedDirectoryPath, resultFile.absolutePath)
}
}
@ -78,7 +79,7 @@ class DatabaseCoordinatorTest {
@Test
@SmallTest
@OptIn(ExperimentalCoroutinesApi::class)
fun database_files_move_test() = runTest {
fun data_database_files_move_test() = runTest {
val parentFile = File(
DatabasePathFixture.new(
baseFolderPath = DatabasePathFixture.DATABASE_DIR_PATH,
@ -89,7 +90,7 @@ class DatabaseCoordinatorTest {
val originalDbFile = getEmptyFile(
parent = parentFile,
fileName = DatabaseNameFixture.newDb(
name = DatabaseCoordinator.DB_CACHE_NAME_LEGACY,
name = DatabaseCoordinator.DB_DATA_NAME_LEGACY,
alias = DatabaseCoordinator.ALIAS_LEGACY
)
)
@ -97,7 +98,7 @@ class DatabaseCoordinatorTest {
val originalDbJournalFile = getEmptyFile(
parent = parentFile,
fileName = DatabaseNameFixture.newDbJournal(
name = DatabaseCoordinator.DB_CACHE_NAME_LEGACY,
name = DatabaseCoordinator.DB_DATA_NAME_LEGACY,
alias = DatabaseCoordinator.ALIAS_LEGACY
)
)
@ -105,22 +106,22 @@ class DatabaseCoordinatorTest {
val originalDbWalFile = getEmptyFile(
parent = parentFile,
fileName = DatabaseNameFixture.newDbWal(
name = DatabaseCoordinator.DB_CACHE_NAME_LEGACY,
name = DatabaseCoordinator.DB_DATA_NAME_LEGACY,
alias = DatabaseCoordinator.ALIAS_LEGACY
)
)
val expectedDbFile = File(
DatabasePathFixture.new(),
DatabaseNameFixture.newDb(name = DatabaseCoordinator.DB_CACHE_NAME)
DatabaseNameFixture.newDb(name = DatabaseCoordinator.DB_DATA_NAME)
)
val expectedDbJournalFile = File(
DatabasePathFixture.new(),
DatabaseNameFixture.newDbJournal(name = DatabaseCoordinator.DB_CACHE_NAME)
DatabaseNameFixture.newDbJournal(name = DatabaseCoordinator.DB_DATA_NAME)
)
val expectedDbWalFile = File(
DatabasePathFixture.new(),
DatabaseNameFixture.newDbWal(name = DatabaseCoordinator.DB_CACHE_NAME)
DatabaseNameFixture.newDbWal(name = DatabaseCoordinator.DB_DATA_NAME)
)
assertTrue(originalDbFile.existsSuspend())
@ -131,7 +132,7 @@ class DatabaseCoordinatorTest {
assertFalse(expectedDbJournalFile.existsSuspend())
assertFalse(expectedDbWalFile.existsSuspend())
dbCoordinator.cacheDbFile(
dbCoordinator.dataDbFile(
DatabaseNameFixture.TEST_DB_NETWORK,
DatabaseNameFixture.TEST_DB_ALIAS
).also { resultFile ->
@ -162,7 +163,7 @@ class DatabaseCoordinatorTest {
@Test
@SmallTest
@OptIn(ExperimentalCoroutinesApi::class)
fun delete_database_files_test() = runTest {
fun delete_data_database_files_test() = runTest {
val parentFile = File(
DatabasePathFixture.new(
baseFolderPath = DatabasePathFixture.NO_BACKUP_DIR_PATH,
@ -172,17 +173,17 @@ class DatabaseCoordinatorTest {
val dbFile = getEmptyFile(
parent = parentFile,
fileName = DatabaseNameFixture.newDb(name = DatabaseCoordinator.DB_CACHE_NAME)
fileName = DatabaseNameFixture.newDb(name = DatabaseCoordinator.DB_DATA_NAME)
)
val dbJournalFile = getEmptyFile(
parent = parentFile,
fileName = DatabaseNameFixture.newDbJournal(name = DatabaseCoordinator.DB_CACHE_NAME)
fileName = DatabaseNameFixture.newDbJournal(name = DatabaseCoordinator.DB_DATA_NAME)
)
val dbWalFile = getEmptyFile(
parent = parentFile,
fileName = DatabaseNameFixture.newDbWal(name = DatabaseCoordinator.DB_CACHE_NAME)
fileName = DatabaseNameFixture.newDbWal(name = DatabaseCoordinator.DB_DATA_NAME)
)
assertTrue(dbFile.existsSuspend())
@ -195,4 +196,107 @@ class DatabaseCoordinatorTest {
assertFalse(dbWalFile.existsSuspend())
}
}
/**
* Note that this situation is just hypothetical, as the legacy database files should be placed only on one of
* the legacy locations, not both, but it is alright to test it together.
*/
@Test
@SmallTest
@OptIn(ExperimentalCoroutinesApi::class)
fun delete_all_legacy_database_files_test() = runTest {
// create older location legacy files
val olderLegacyParentFile = File(
DatabasePathFixture.new(
baseFolderPath = DatabasePathFixture.DATABASE_DIR_PATH,
internalPath = ""
)
)
val olderLegacyDbFile = getEmptyFile(
parent = olderLegacyParentFile,
fileName = DatabaseNameFixture.newDb(
name = DatabaseCoordinator.DB_CACHE_OLDER_NAME_LEGACY,
network = DatabaseNameFixture.TEST_DB_NETWORK.networkName,
alias = DatabaseCoordinator.ALIAS_LEGACY
)
)
val olderLegacyDbJournalFile = getEmptyFile(
parent = olderLegacyParentFile,
fileName = DatabaseNameFixture.newDbJournal(
name = DatabaseCoordinator.DB_CACHE_OLDER_NAME_LEGACY,
network = DatabaseNameFixture.TEST_DB_NETWORK.networkName,
alias = DatabaseCoordinator.ALIAS_LEGACY
)
)
val olderLegacyDbWalFile = getEmptyFile(
parent = olderLegacyParentFile,
fileName = DatabaseNameFixture.newDbWal(
name = DatabaseCoordinator.DB_CACHE_OLDER_NAME_LEGACY,
network = DatabaseNameFixture.TEST_DB_NETWORK.networkName,
alias = DatabaseCoordinator.ALIAS_LEGACY
)
)
// create newer location legacy files
val newerLegacyParentFile = File(
DatabasePathFixture.new(
baseFolderPath = DatabasePathFixture.NO_BACKUP_DIR_PATH,
internalPath = DatabasePathFixture.INTERNAL_DATABASE_PATH
)
)
val newerLegacyDbFile = getEmptyFile(
parent = newerLegacyParentFile,
fileName = DatabaseNameFixture.newDb(
name = DatabaseCoordinator.DB_CACHE_NEWER_NAME_LEGACY,
network = DatabaseNameFixture.TEST_DB_NETWORK.networkName,
alias = DatabaseNameFixture.TEST_DB_ALIAS
)
)
val newerLegacyDbJournalFile = getEmptyFile(
parent = newerLegacyParentFile,
fileName = DatabaseNameFixture.newDbJournal(
name = DatabaseCoordinator.DB_CACHE_NEWER_NAME_LEGACY,
network = DatabaseNameFixture.TEST_DB_NETWORK.networkName,
alias = DatabaseNameFixture.TEST_DB_ALIAS
)
)
val newerLegacyDbWalFile = getEmptyFile(
parent = newerLegacyParentFile,
fileName = DatabaseNameFixture.newDbWal(
name = DatabaseCoordinator.DB_CACHE_NEWER_NAME_LEGACY,
network = DatabaseNameFixture.TEST_DB_NETWORK.networkName,
alias = DatabaseNameFixture.TEST_DB_ALIAS
)
)
// check all files in place
assertTrue(olderLegacyDbFile.existsSuspend())
assertTrue(olderLegacyDbJournalFile.existsSuspend())
assertTrue(olderLegacyDbWalFile.existsSuspend())
assertTrue(newerLegacyDbFile.existsSuspend())
assertTrue(newerLegacyDbJournalFile.existsSuspend())
assertTrue(newerLegacyDbWalFile.existsSuspend())
// once we access the latest file system blocks storage root directory, all the legacy database files should
// be removed
dbCoordinator.fsBlockDbRoot(
network = DatabaseNameFixture.TEST_DB_NETWORK,
alias = DatabaseNameFixture.TEST_DB_ALIAS
).also {
assertFalse(olderLegacyDbFile.existsSuspend())
assertFalse(olderLegacyDbJournalFile.existsSuspend())
assertFalse(olderLegacyDbWalFile.existsSuspend())
assertFalse(newerLegacyDbFile.existsSuspend())
assertFalse(newerLegacyDbJournalFile.existsSuspend())
assertFalse(newerLegacyDbWalFile.existsSuspend())
}
}
}

View File

@ -33,8 +33,8 @@ class SynchronizerFactoryTest {
)
assertTrue(
"Invalid CacheDB file",
rustBackend.cacheDbFile.absolutePath.endsWith(
"no_backup/co.electricoin.zcash/TestWallet_testnet_${DatabaseCoordinator.DB_CACHE_NAME}"
rustBackend.fsBlockDbRoot.absolutePath.endsWith(
"no_backup/co.electricoin.zcash/TestWallet_testnet_${DatabaseCoordinator.DB_FS_BLOCK_DB_ROOT_NAME}"
)
)
assertTrue(

View File

@ -0,0 +1,219 @@
package cash.z.ecc.android.sdk.internal.storage.block
import cash.z.ecc.android.sdk.internal.ext.deleteRecursivelySuspend
import cash.z.ecc.android.sdk.internal.ext.existsSuspend
import cash.z.ecc.android.sdk.internal.ext.mkdirsSuspend
import cash.z.ecc.android.sdk.jni.RustBackendWelding
import cash.z.ecc.android.sdk.model.BlockHeight
import cash.z.ecc.android.sdk.model.ZcashNetwork
import cash.z.ecc.fixture.FakeRustBackendFixture
import cash.z.ecc.fixture.FilePathFixture
import co.electriccoin.lightwallet.client.fixture.ListOfCompactBlocksFixture
import kotlinx.coroutines.ExperimentalCoroutinesApi
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.test.runTest
import org.junit.After
import org.junit.Before
import org.junit.Test
import java.io.File
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertNotNull
import kotlin.test.assertNull
import kotlin.test.assertTrue
class FileCompactBlockRepositoryTest {
@OptIn(ExperimentalCoroutinesApi::class)
@Before
fun setup() = runTest {
val rootDirectory = FilePathFixture.newBlocksDir()
if (rootDirectory.existsSuspend()) {
rootDirectory.deleteRecursivelySuspend()
}
val blocksDir = FilePathFixture.newBlocksDir()
blocksDir.mkdirsSuspend()
}
@OptIn(ExperimentalCoroutinesApi::class)
@After
fun tearDown() = runTest {
FilePathFixture.newBlocksDir().deleteRecursivelySuspend()
}
private fun getMockedFileCompactBlockRepository(
rustBackend: RustBackendWelding,
rootBlocksDirectory: File
): FileCompactBlockRepository = runBlocking {
FileCompactBlockRepository(
rootBlocksDirectory,
rustBackend
)
}
@OptIn(ExperimentalCoroutinesApi::class)
@Test
fun getLatestHeightTest() = runTest {
val rustBackend = FakeRustBackendFixture().new()
val blockRepository = getMockedFileCompactBlockRepository(rustBackend, FilePathFixture.newBlocksDir())
val blocks = ListOfCompactBlocksFixture.new()
blockRepository.write(blocks)
assertEquals(blocks.last().height, blockRepository.getLatestHeight()?.value)
}
@OptIn(ExperimentalCoroutinesApi::class)
@Test
fun findCompactBlockTest() = runTest {
val network = ZcashNetwork.Testnet
val rustBackend = FakeRustBackendFixture().new()
val blockRepository = getMockedFileCompactBlockRepository(rustBackend, FilePathFixture.newBlocksDir())
val blocks = ListOfCompactBlocksFixture.new()
blockRepository.write(blocks)
val firstPersistedBlock = blockRepository.findCompactBlock(
BlockHeight.new(network, blocks.first().height)
)
val lastPersistedBlock = blockRepository.findCompactBlock(
BlockHeight.new(network, blocks.last().height)
)
val notPersistedBlockHeight = BlockHeight.new(
network,
blockHeight = blocks.last().height + 1
)
assertNotNull(firstPersistedBlock)
assertNotNull(lastPersistedBlock)
assertEquals(blocks.first().height, firstPersistedBlock.height)
assertEquals(blocks.last().height, blockRepository.getLatestHeight()?.value)
assertNull(blockRepository.findCompactBlock(notPersistedBlockHeight))
}
@OptIn(ExperimentalCoroutinesApi::class)
@Test
fun writeBlocksTest() = runTest {
val rustBackend = FakeRustBackendFixture().new()
val blockRepository = getMockedFileCompactBlockRepository(rustBackend, FilePathFixture.newBlocksDir())
assertTrue { rustBackend.metadata.isEmpty() }
val blocks = ListOfCompactBlocksFixture.new()
val persistedBlocksCount = blockRepository.write(blocks)
assertEquals(blocks.count(), persistedBlocksCount)
assertEquals(blocks.count(), rustBackend.metadata.size)
}
@OptIn(ExperimentalCoroutinesApi::class)
@Test
fun writeFewBlocksTest() = runTest {
val rustBackend = FakeRustBackendFixture().new()
val blockRepository = getMockedFileCompactBlockRepository(rustBackend, FilePathFixture.newBlocksDir())
assertTrue { rustBackend.metadata.isEmpty() }
// prepare a list of blocks to be persisted, which has smaller size than buffer size
val reducedBlocksList = ListOfCompactBlocksFixture.new().apply {
val reduced = drop(count() / 2)
assertTrue { reduced.count() < FileCompactBlockRepository.BLOCKS_METADATA_BUFFER_SIZE }
}
val persistedBlocksCount = blockRepository.write(reducedBlocksList)
assertEquals(reducedBlocksList.count(), persistedBlocksCount)
assertEquals(reducedBlocksList.count(), rustBackend.metadata.size)
}
@OptIn(ExperimentalCoroutinesApi::class)
@Test
fun writeBlocksAndCheckStorageTest() = runTest {
val rustBackend = FakeRustBackendFixture().new()
val rootBlocksDirectory = FilePathFixture.newBlocksDir()
val blockRepository = getMockedFileCompactBlockRepository(rustBackend, rootBlocksDirectory)
assertTrue { rootBlocksDirectory.exists() }
assertTrue { rootBlocksDirectory.list()!!.isEmpty() }
val blocks = ListOfCompactBlocksFixture.new()
val persistedBlocksCount = blockRepository.write(blocks)
assertTrue { rootBlocksDirectory.exists() }
assertEquals(blocks.count(), persistedBlocksCount)
assertEquals(blocks.count(), rootBlocksDirectory.list()!!.size)
}
@OptIn(ExperimentalCoroutinesApi::class)
@Test
fun rewindToTest() = runTest {
val rustBackend = FakeRustBackendFixture().new()
val blockRepository = getMockedFileCompactBlockRepository(rustBackend, FilePathFixture.newBlocksDir())
val testedBlocksRange = ListOfCompactBlocksFixture.DEFAULT_FILE_BLOCK_RANGE
val blocks = ListOfCompactBlocksFixture.new(testedBlocksRange)
blockRepository.write(blocks)
val blocksRangeMiddleValue = testedBlocksRange.run {
start.value.plus(endInclusive.value).div(2)
}
val rewindHeight: Long = blocksRangeMiddleValue
blockRepository.rewindTo(BlockHeight(rewindHeight))
// suppose to be 0
val keptMetadataAboveRewindHeight = rustBackend.metadata
.filter { it.height > rewindHeight }
assertTrue { keptMetadataAboveRewindHeight.isEmpty() }
val expectedRewoundMetadataCount =
(testedBlocksRange.endInclusive.value - blocksRangeMiddleValue).toInt()
assertEquals(expectedRewoundMetadataCount, blocks.count() - rustBackend.metadata.size)
val expectedKeptMetadataCount =
(blocks.count() - expectedRewoundMetadataCount)
assertEquals(expectedKeptMetadataCount, rustBackend.metadata.size)
val keptMetadataBelowRewindHeight = rustBackend.metadata
.filter { it.height <= rewindHeight }
assertEquals(expectedKeptMetadataCount, keptMetadataBelowRewindHeight.size)
}
@OptIn(ExperimentalCoroutinesApi::class)
@Test
fun createTemporaryFileTest() = runTest {
val blocksDir = FilePathFixture.newBlocksDir()
val blocks = ListOfCompactBlocksFixture.new()
val block = blocks.first()
val file = block.createTemporaryFile(blocksDir)
assertTrue { file.existsSuspend() }
}
@OptIn(ExperimentalCoroutinesApi::class)
@Test
fun finalizeFileTest() = runTest {
val blocksDir = FilePathFixture.newBlocksDir()
val blocks = ListOfCompactBlocksFixture.new()
val block = blocks.first()
val tempFile = block.createTemporaryFile(blocksDir)
val finalizedFile = File(tempFile.absolutePath.dropLast(FileCompactBlockRepository.TEMPORARY_FILENAME_SUFFIX.length))
assertFalse { finalizedFile.existsSuspend() }
tempFile.finalizeFile()
assertTrue { finalizedFile.existsSuspend() }
assertFalse { tempFile.existsSuspend() }
}
}

View File

@ -0,0 +1,19 @@
package cash.z.ecc.fixture
import cash.z.ecc.android.sdk.internal.db.DatabaseCoordinator
import cash.z.ecc.android.sdk.model.ZcashNetwork
/**
* Provides a unified way for getting a fixture root name for database cache files for test purposes.
*/
object DatabaseCacheFilesRootFixture {
const val TEST_CACHE_ROOT_NAME = DatabaseCoordinator.DB_FS_BLOCK_DB_ROOT_NAME
const val TEST_CACHE_ROOT_NAME_ALIAS = "zcash_sdk"
val TEST_NETWORK = ZcashNetwork.Testnet
internal fun newCacheRoot(
name: String = TEST_CACHE_ROOT_NAME,
alias: String = TEST_CACHE_ROOT_NAME_ALIAS,
network: String = TEST_NETWORK.networkName
) = "${alias}_${network}_$name"
}

View File

@ -3,6 +3,9 @@ package cash.z.ecc.fixture
import cash.z.ecc.android.sdk.internal.db.DatabaseCoordinator
import cash.z.ecc.android.sdk.model.ZcashNetwork
/**
* Provides a unified way for getting a fixture database files names for test purposes.
*/
object DatabaseNameFixture {
const val TEST_DB_NAME = "empty.db"
const val TEST_DB_JOURNAL_NAME_SUFFIX = DatabaseCoordinator.DATABASE_FILE_JOURNAL_SUFFIX

View File

@ -8,6 +8,9 @@ import cash.z.ecc.android.sdk.test.getAppContext
import kotlinx.coroutines.runBlocking
import java.io.File
/**
* Provides a unified way for getting fixture directories on the database root path for test purposes.
*/
object DatabasePathFixture {
val NO_BACKUP_DIR_PATH: String = runBlocking {
getAppContext().getNoBackupFilesDirSuspend().absolutePath

View File

@ -0,0 +1,117 @@
package cash.z.ecc.fixture
import cash.z.ecc.android.sdk.internal.model.Checkpoint
import cash.z.ecc.android.sdk.internal.model.JniBlockMeta
import cash.z.ecc.android.sdk.jni.RustBackendWelding
import cash.z.ecc.android.sdk.model.BlockHeight
import cash.z.ecc.android.sdk.model.UnifiedSpendingKey
import cash.z.ecc.android.sdk.model.WalletBalance
import cash.z.ecc.android.sdk.model.Zatoshi
import cash.z.ecc.android.sdk.model.ZcashNetwork
import cash.z.ecc.android.sdk.type.UnifiedFullViewingKey
import java.io.File
internal class FakeRustBackend(
override val network: ZcashNetwork,
override val saplingParamDir: File,
val metadata: MutableList<JniBlockMeta>
) : RustBackendWelding {
override suspend fun writeBlockMetadata(blockMetadata: List<JniBlockMeta>): Boolean =
metadata.addAll(blockMetadata)
override suspend fun rewindToHeight(height: BlockHeight): Boolean {
metadata.removeAll { it.height > height.value }
return true
}
override suspend fun getLatestHeight(): BlockHeight = BlockHeight(metadata.maxOf { it.height })
override suspend fun findBlockMetadata(height: BlockHeight): JniBlockMeta? {
return metadata.findLast { it.height == height.value }
}
override suspend fun rewindBlockMetadataToHeight(height: BlockHeight) {
metadata.removeAll { it.height > height.value }
}
override suspend fun initBlockMetaDb(): Int =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun createToAddress(usk: UnifiedSpendingKey, to: String, value: Long, memo: ByteArray?): Long =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun shieldToAddress(usk: UnifiedSpendingKey, memo: ByteArray?): Long =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun decryptAndStoreTransaction(tx: ByteArray) =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun initAccountsTable(seed: ByteArray, numberOfAccounts: Int): Array<UnifiedFullViewingKey> =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun initAccountsTable(vararg keys: UnifiedFullViewingKey): Boolean =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun initBlocksTable(checkpoint: Checkpoint): Boolean =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun initDataDb(seed: ByteArray?): Int =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun createAccount(seed: ByteArray): UnifiedSpendingKey =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override fun isValidShieldedAddr(addr: String): Boolean =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override fun isValidTransparentAddr(addr: String): Boolean =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override fun isValidUnifiedAddr(addr: String): Boolean =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun getCurrentAddress(account: Int): String =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override fun getTransparentReceiver(ua: String): String? =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override fun getSaplingReceiver(ua: String): String? =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun getBalance(account: Int): Zatoshi =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override fun getBranchIdForHeight(height: BlockHeight): Long =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun getReceivedMemoAsUtf8(idNote: Long): String? =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun getSentMemoAsUtf8(idNote: Long): String? =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun getVerifiedBalance(account: Int): Zatoshi =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun getNearestRewindHeight(height: BlockHeight): BlockHeight =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun scanBlocks(limit: Int): Boolean =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun validateCombinedChain(): BlockHeight? =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun putUtxo(
tAddress: String,
txId: ByteArray,
index: Int,
script: ByteArray,
value: Long,
height: BlockHeight
): Boolean = error("Intentionally not implemented in mocked FakeRustBackend implementation.")
override suspend fun getDownloadedUtxoBalance(address: String): WalletBalance =
error("Intentionally not implemented in mocked FakeRustBackend implementation.")
}

View File

@ -0,0 +1,21 @@
package cash.z.ecc.fixture
import cash.z.ecc.android.sdk.internal.model.JniBlockMeta
import cash.z.ecc.android.sdk.model.ZcashNetwork
import java.io.File
internal class FakeRustBackendFixture {
private val DEFAULT_SAPLING_PARAM_DIR = File(DatabasePathFixture.new())
private val DEFAULT_NETWORK = ZcashNetwork.Testnet
fun new(
saplingParamDir: File = DEFAULT_SAPLING_PARAM_DIR,
network: ZcashNetwork = DEFAULT_NETWORK,
metadata: MutableList<JniBlockMeta> = mutableListOf()
) = FakeRustBackend(
saplingParamDir = saplingParamDir,
network = network,
metadata = metadata
)
}

View File

@ -0,0 +1,14 @@
package cash.z.ecc.fixture
import cash.z.ecc.android.sdk.internal.storage.block.FileCompactBlockRepository
import java.io.File
object FilePathFixture {
private val DEFAULT_ROOT_DIR_PATH = DatabasePathFixture.new()
private const val DEFAULT_BLOCKS_DIR_NAME = FileCompactBlockRepository.BLOCKS_DOWNLOAD_DIRECTORY
internal fun newBlocksDir(
rootDirectoryPath: String = DEFAULT_ROOT_DIR_PATH,
blockDirectoryName: String = DEFAULT_BLOCKS_DIR_NAME
) = File(rootDirectoryPath, blockDirectoryName)
}

View File

@ -23,7 +23,6 @@ import cash.z.ecc.android.sdk.internal.SaplingParamTool
import cash.z.ecc.android.sdk.internal.Twig
import cash.z.ecc.android.sdk.internal.block.CompactBlockDownloader
import cash.z.ecc.android.sdk.internal.db.DatabaseCoordinator
import cash.z.ecc.android.sdk.internal.db.block.DbCompactBlockRepository
import cash.z.ecc.android.sdk.internal.db.derived.DbDerivedDataRepository
import cash.z.ecc.android.sdk.internal.db.derived.DerivedDataDb
import cash.z.ecc.android.sdk.internal.ext.toHexReversed
@ -32,6 +31,7 @@ import cash.z.ecc.android.sdk.internal.isEmpty
import cash.z.ecc.android.sdk.internal.model.Checkpoint
import cash.z.ecc.android.sdk.internal.repository.CompactBlockRepository
import cash.z.ecc.android.sdk.internal.repository.DerivedDataRepository
import cash.z.ecc.android.sdk.internal.storage.block.FileCompactBlockRepository
import cash.z.ecc.android.sdk.internal.transaction.OutboundTransactionManager
import cash.z.ecc.android.sdk.internal.transaction.PersistentTransactionManager
import cash.z.ecc.android.sdk.internal.transaction.TransactionEncoder
@ -80,7 +80,6 @@ import kotlinx.coroutines.flow.launchIn
import kotlinx.coroutines.flow.map
import kotlinx.coroutines.flow.onEach
import kotlinx.coroutines.launch
import java.io.File
import java.util.concurrent.ConcurrentHashMap
import kotlin.coroutines.CoroutineContext
@ -736,7 +735,7 @@ internal object DefaultSynchronizerFactory {
val coordinator = DatabaseCoordinator.getInstance(context)
return RustBackend.init(
coordinator.cacheDbFile(network, alias),
coordinator.fsBlockDbRoot(network, alias),
coordinator.dataDbFile(network, alias),
saplingParamTool.properties.paramsDirectory,
network,
@ -764,12 +763,10 @@ internal object DefaultSynchronizerFactory {
)
)
internal fun defaultCompactBlockRepository(context: Context, cacheDbFile: File, zcashNetwork: ZcashNetwork):
internal suspend fun defaultFileCompactBlockRepository(rustBackend: RustBackend):
CompactBlockRepository =
DbCompactBlockRepository.new(
context,
zcashNetwork,
cacheDbFile
FileCompactBlockRepository.new(
rustBackend
)
fun defaultService(context: Context, lightWalletEndpoint: LightWalletEndpoint): BlockingLightWalletClient =

View File

@ -4,7 +4,6 @@ import android.content.Context
import cash.z.ecc.android.sdk.block.CompactBlockProcessor
import cash.z.ecc.android.sdk.ext.ZcashSdk
import cash.z.ecc.android.sdk.internal.SaplingParamTool
import cash.z.ecc.android.sdk.internal.db.DatabaseCoordinator
import cash.z.ecc.android.sdk.model.Account
import cash.z.ecc.android.sdk.model.BlockHeight
import cash.z.ecc.android.sdk.model.PendingTransaction
@ -467,8 +466,6 @@ interface Synchronizer {
birthday ?: zcashNetwork.saplingActivationHeight
)
val coordinator = DatabaseCoordinator.getInstance(context)
val rustBackend = DefaultSynchronizerFactory.defaultRustBackend(
applicationContext,
zcashNetwork,
@ -477,11 +474,9 @@ interface Synchronizer {
saplingParamTool
)
val blockStore = DefaultSynchronizerFactory.defaultCompactBlockRepository(
applicationContext,
coordinator.cacheDbFile(zcashNetwork, alias),
zcashNetwork
)
val blockStore =
DefaultSynchronizerFactory
.defaultFileCompactBlockRepository(rustBackend)
val viewingKeys = seed?.let {
DerivationTool.deriveUnifiedFullViewingKeys(

View File

@ -45,7 +45,7 @@ import cash.z.ecc.android.sdk.model.WalletBalance
import cash.z.ecc.android.sdk.model.ZcashNetwork
import cash.z.wallet.sdk.internal.rpc.Service
import co.electriccoin.lightwallet.client.ext.BenchmarkingExt
import co.electriccoin.lightwallet.client.fixture.BlockRangeFixture
import co.electriccoin.lightwallet.client.fixture.BenchmarkingBlockRangeFixture
import co.electriccoin.lightwallet.client.model.BlockHeightUnsafe
import co.electriccoin.lightwallet.client.model.LightWalletEndpointInfoUnsafe
import co.electriccoin.lightwallet.client.model.Response
@ -296,7 +296,7 @@ class CompactBlockProcessor internal constructor(
if (BenchmarkingExt.isBenchmarking()) {
// We inject a benchmark test blocks range at this point to process only a restricted range of blocks
// for a more reliable benchmark results.
val benchmarkBlockRange = BlockRangeFixture.new().let {
val benchmarkBlockRange = BenchmarkingBlockRangeFixture.new().let {
// Convert range of Longs to range of BlockHeights
BlockHeight.new(ZcashNetwork.Mainnet, it.start)..(
BlockHeight.new(ZcashNetwork.Mainnet, it.endInclusive)
@ -704,10 +704,7 @@ class CompactBlockProcessor internal constructor(
return BlockProcessingResult.NoBlocksToProcess
}
Twig.debug {
"validating blocks in range $range in db: ${
(rustBackend as RustBackend).cacheDbFile
.absolutePath
}"
"validating blocks in range $range in db: ${(rustBackend as RustBackend).fsBlockDbRoot.absolutePath}"
}
val result = rustBackend.validateCombinedChain()
@ -944,44 +941,32 @@ class CompactBlockProcessor internal constructor(
private suspend fun printValidationErrorInfo(errorHeight: BlockHeight, count: Int = 11) {
// Note: blocks are public information so it's okay to print them but, still, let's not unless we're
// debugging something
if (!BuildConfig.DEBUG) return
if (!BuildConfig.DEBUG) {
return
}
var errorInfo = fetchValidationErrorInfo(errorHeight)
Twig.debug {
"validation failed at block ${errorInfo.errorHeight} which had hash " +
"${errorInfo.actualPrevHash} but the expected hash was ${errorInfo.expectedPrevHash}"
}
Twig.debug { "validation failed at block ${errorInfo.errorHeight} with hash: ${errorInfo.hash}" }
errorInfo = fetchValidationErrorInfo(errorHeight + 1)
Twig.debug {
"The next block block: ${errorInfo.errorHeight} which had hash ${errorInfo.actualPrevHash} but " +
"the expected hash was ${errorInfo.expectedPrevHash}"
}
Twig.debug { "the next block is ${errorInfo.errorHeight} with hash: ${errorInfo.hash}" }
Twig.debug { "=================== BLOCKS [$errorHeight..${errorHeight.value + count - 1}]: START ========" }
repeat(count) { i ->
val height = errorHeight + i
val block = downloader.compactBlockRepository.findCompactBlock(height)
// sometimes the initial block was inserted via checkpoint and will not appear in the cache. We can get
// the hash another way but prevHash is correctly null.
val hash = block?.hash?.toByteArray()
?: repository.findBlockHash(height)
Twig.debug {
"block: $height\thash=${hash?.toHexReversed()} \tprevHash=${
block?.prevHash?.toByteArray()?.toHexReversed()
}"
}
// the hash another way.
val checkedHash = block?.hash ?: repository.findBlockHash(height)
Twig.debug { "block: $height\thash=${checkedHash?.toHexReversed()}" }
}
Twig.debug { "=================== BLOCKS [$errorHeight..${errorHeight.value + count - 1}]: END ========" }
}
private suspend fun fetchValidationErrorInfo(errorHeight: BlockHeight): ValidationErrorInfo {
val hash = repository.findBlockHash(errorHeight + 1)
?.toHexReversed()
val prevHash = repository.findBlockHash(errorHeight)?.toHexReversed()
val hash = repository.findBlockHash(errorHeight + 1)?.toHexReversed()
val compactBlock = downloader.compactBlockRepository.findCompactBlock(errorHeight + 1)
val expectedPrevHash = compactBlock?.prevHash?.toByteArray()?.toHexReversed()
return ValidationErrorInfo(errorHeight, hash, expectedPrevHash, prevHash)
return ValidationErrorInfo(errorHeight, hash)
}
/**
@ -1279,9 +1264,7 @@ class CompactBlockProcessor internal constructor(
data class ValidationErrorInfo(
val errorHeight: BlockHeight,
val hash: String?,
val expectedPrevHash: String?,
val actualPrevHash: String?
val hash: String?
)
//

View File

@ -98,7 +98,6 @@ open class CompactBlockDownloader private constructor(val compactBlockRepository
withContext(Dispatchers.IO) {
lightWalletClient.shutdown()
}
compactBlockRepository.close()
}
/**

View File

@ -11,6 +11,7 @@ import cash.z.ecc.android.sdk.internal.Files
import cash.z.ecc.android.sdk.internal.LazyWithArgument
import cash.z.ecc.android.sdk.internal.NoBackupContextWrapper
import cash.z.ecc.android.sdk.internal.Twig
import cash.z.ecc.android.sdk.internal.ext.deleteRecursivelySuspend
import cash.z.ecc.android.sdk.internal.ext.deleteSuspend
import cash.z.ecc.android.sdk.internal.ext.existsSuspend
import cash.z.ecc.android.sdk.internal.ext.getDatabasePathSuspend
@ -48,8 +49,9 @@ internal class DatabaseCoordinator private constructor(context: Context) {
const val DB_DATA_NAME = "data.sqlite3" // $NON-NLS
@VisibleForTesting
internal const val DB_CACHE_NAME_LEGACY = "Cache.db" // $NON-NLS
const val DB_CACHE_NAME = "cache.sqlite3" // $NON-NLS
internal const val DB_CACHE_OLDER_NAME_LEGACY = "Cache.db" // $NON-NLS
internal const val DB_CACHE_NEWER_NAME_LEGACY = "cache.sqlite3" // $NON-NLS
const val DB_FS_BLOCK_DB_ROOT_NAME = "fs_cache" // $NON-NLS
@VisibleForTesting
internal const val DB_PENDING_TRANSACTIONS_NAME_LEGACY = "PendingTransactions.db" // $NON-NLS
@ -68,32 +70,34 @@ internal class DatabaseCoordinator private constructor(context: Context) {
}
/**
* Returns the file of the Cache database that would correspond to the given alias
* and network attributes.
* Returns the root folder of the cache database files that would correspond to the given
* alias and network attributes.
*
* @param network the network associated with the data in the cache database.
* @param alias the alias to convert into a database path
* @param network the network associated with the data in the cache
* @param alias the alias to convert into a cache path
*
* @return the Cache database file
* @return the cache database folder
*/
internal suspend fun cacheDbFile(
internal suspend fun fsBlockDbRoot(
network: ZcashNetwork,
alias: String
): File {
val dbLocationsPair = prepareDbFiles(
applicationContext,
// First we deal with the legacy Cache database files (rollback included) on both older and newer path. In
// case of deletion failure caused by any reason, we try it on the next time again.
val legacyDbFilesDeleted = deleteLegacyCacheDbFiles(network, alias)
val result = if (legacyDbFilesDeleted) {
"are successfully deleted"
} else {
"failed to be deleted. Will be retried it on the next time"
}
Twig.debug { "Legacy Cache database files $result." }
return newDatabaseFilePointer(
network,
alias,
DB_CACHE_NAME_LEGACY,
DB_CACHE_NAME
DB_FS_BLOCK_DB_ROOT_NAME,
Files.getZcashNoBackupSubdirectory(applicationContext)
)
createFileMutex.withLock {
return checkAndMoveDatabaseFiles(
dbLocationsPair.first,
dbLocationsPair.second
)
}
}
/**
@ -167,30 +171,63 @@ internal class DatabaseCoordinator private constructor(context: Context) {
*
* @param network the network associated with the data in the database.
* @param alias the alias to convert into a database path
*
* @return true only if any database deleted, false otherwise
*/
internal suspend fun deleteDatabases(
network: ZcashNetwork,
alias: String
): Boolean {
deleteFileMutex.withLock {
val dataDeleted = deleteDatabase(
dataDbFile(network, alias)
)
val cacheDeleted = deleteDatabase(
cacheDbFile(network, alias)
)
val dataDeleted = deleteDatabase(dataDbFile(network, alias))
val cacheDeleted = fsBlockDbRoot(network, alias).deleteRecursivelySuspend()
return dataDeleted || cacheDeleted
}
}
/**
* This checks and potentially deletes all the legacy Cache database files, which correspond to the given alias and
* network attributes, as we recently switched to the store blocks on disk mechanism instead of putting them into
* the Cache database.
*
* This function deals with database rollback files too.
*
* @param network the network associated with the data in the Cache database
* @param alias the alias to convert into a database path
*
* @return true in case of successful deletion of all the files, false otherwise
*/
private suspend fun deleteLegacyCacheDbFiles(
network: ZcashNetwork,
alias: String
): Boolean {
val legacyDatabaseLocationPair = prepareDbFiles(
applicationContext,
network,
alias,
DB_CACHE_OLDER_NAME_LEGACY,
DB_CACHE_NEWER_NAME_LEGACY
)
var olderLegacyCacheDbDeleted = true
var newerLegacyCacheDbDeleted = true
if (legacyDatabaseLocationPair.first.existsSuspend()) {
olderLegacyCacheDbDeleted = deleteDatabase(legacyDatabaseLocationPair.first)
}
if (legacyDatabaseLocationPair.second.existsSuspend()) {
newerLegacyCacheDbDeleted = deleteDatabase(legacyDatabaseLocationPair.second)
}
return olderLegacyCacheDbDeleted && newerLegacyCacheDbDeleted
}
/**
* This helper function prepares a legacy (i.e. previously created) database file, as well
* as the preferred (i.e. newly created) file for subsequent use (and eventually move).
*
* Note: the database file placed under the fake no_backup folder for devices with Android SDK
* level lower than 21.
*
* @param appContext the application context
* @param network the network associated with the data in the database.
* @param alias the alias to convert into a database path

View File

@ -1,77 +0,0 @@
package cash.z.ecc.android.sdk.internal.db.block
import android.content.Context
import androidx.room.RoomDatabase
import cash.z.ecc.android.sdk.internal.SdkDispatchers
import cash.z.ecc.android.sdk.internal.SdkExecutors
import cash.z.ecc.android.sdk.internal.db.commonDatabaseBuilder
import cash.z.ecc.android.sdk.internal.repository.CompactBlockRepository
import cash.z.ecc.android.sdk.model.BlockHeight
import cash.z.ecc.android.sdk.model.ZcashNetwork
import cash.z.wallet.sdk.internal.rpc.CompactFormats
import kotlinx.coroutines.withContext
import java.io.File
/**
* An implementation of CompactBlockStore that persists information to a database in the given
* path. This represents the "cache db" or local cache of compact blocks waiting to be scanned.
*/
class DbCompactBlockRepository private constructor(
private val network: ZcashNetwork,
private val cacheDb: CompactBlockDb
) : CompactBlockRepository {
private val cacheDao = cacheDb.compactBlockDao()
override suspend fun getLatestHeight(): BlockHeight? = runCatching {
BlockHeight.new(network, cacheDao.latestBlockHeight())
}.getOrNull()
override suspend fun findCompactBlock(height: BlockHeight): CompactFormats.CompactBlock? =
cacheDao.findCompactBlock(height.value)?.let { CompactFormats.CompactBlock.parseFrom(it) }
override suspend fun write(result: Sequence<CompactFormats.CompactBlock>) =
cacheDao.insert(result.map { CompactBlockEntity(it.height, it.toByteArray()) })
override suspend fun rewindTo(height: BlockHeight) =
cacheDao.rewindTo(height.value)
override suspend fun close() {
withContext(SdkDispatchers.DATABASE_IO) {
cacheDb.close()
}
}
companion object {
/**
* @param appContext the application context. This is used for creating the database.
* @property databaseFile the database file.
*/
fun new(
appContext: Context,
zcashNetwork: ZcashNetwork,
databaseFile: File
): DbCompactBlockRepository {
val cacheDb = createCompactBlockCacheDb(appContext.applicationContext, databaseFile)
return DbCompactBlockRepository(zcashNetwork, cacheDb)
}
private fun createCompactBlockCacheDb(
appContext: Context,
databaseFile: File
): CompactBlockDb {
return commonDatabaseBuilder(
appContext,
CompactBlockDb::class.java,
databaseFile
)
.setJournalMode(RoomDatabase.JournalMode.TRUNCATE)
// this is a simple cache of blocks. destroying the db should be benign
.fallbackToDestructiveMigration()
.setQueryExecutor(SdkExecutors.DATABASE_IO)
.setTransactionExecutor(SdkExecutors.DATABASE_IO)
.build()
}
}
}

View File

@ -1,4 +1,4 @@
@file:Suppress("ktlint:filename")
@file:Suppress("ktlint:filename", "TooManyFunctions")
package cash.z.ecc.android.sdk.internal.ext
@ -27,6 +27,10 @@ suspend fun File.inputStreamSuspend(): FileInputStream = withContext(Dispatchers
suspend fun File.createNewFileSuspend() = withContext(Dispatchers.IO) { createNewFile() }
suspend fun File.writeBytesSuspend(byteArray: ByteArray) = withContext(Dispatchers.IO) { writeBytes(byteArray) }
suspend fun File.readBytesSuspend() = withContext(Dispatchers.IO) { readBytes() }
/**
* Preferred buffer size. We use the same buffer size as BufferedInputStream does.
*/

View File

@ -0,0 +1,51 @@
package cash.z.ecc.android.sdk.internal.model
import androidx.annotation.Keep
import cash.z.ecc.android.sdk.internal.storage.block.CompactBlockOutputsCounts
import cash.z.wallet.sdk.internal.rpc.CompactFormats.CompactBlock
/**
* Serves as cross layer (Kotlin, Rust) communication class.
*
* @param height the block's height - although it's type Long, it needs to be in UInt range
* @param hash the block's hash (ID of the block)
* @param time the block's time. Unix epoch time when the block was mined.
* @param saplingOutputsCount the sapling outputs count - although its type is Long, it needs to be in UInt range
* @param orchardOutputsCount the orchard outputs count - although its type is Long, it needs to be in UInt range
*/
@Keep
class JniBlockMeta(
val height: Long,
val hash: ByteArray,
val time: Long,
val saplingOutputsCount: Long,
val orchardOutputsCount: Long
) {
init {
// We require some of the parameters below to be in the range of unsigned integer, because of the Rust layer
// implementation.
require(UINT_RANGE.contains(height)) {
"Height $height is outside of allowed range $UINT_RANGE"
}
require(UINT_RANGE.contains(saplingOutputsCount)) {
"SaplingOutputsCount $saplingOutputsCount is outside of allowed range $UINT_RANGE"
}
require(UINT_RANGE.contains(orchardOutputsCount)) {
"SaplingOutputsCount $orchardOutputsCount is outside of allowed range $UINT_RANGE"
}
}
companion object {
private val UINT_RANGE = 0.toLong()..UInt.MAX_VALUE.toLong()
internal fun new(block: CompactBlock, outputs: CompactBlockOutputsCounts): JniBlockMeta {
return JniBlockMeta(
height = block.height,
hash = block.hash.toByteArray(),
time = block.time.toLong(),
saplingOutputsCount = outputs.saplingOutputsCount.toLong(),
orchardOutputsCount = outputs.orchardActionsCount.toLong()
)
}
}
}

View File

@ -1,5 +1,6 @@
package cash.z.ecc.android.sdk.internal.repository
import cash.z.ecc.android.sdk.internal.model.JniBlockMeta
import cash.z.ecc.android.sdk.model.BlockHeight
import cash.z.wallet.sdk.internal.rpc.CompactFormats
@ -19,7 +20,7 @@ interface CompactBlockRepository {
*
* @return the compact block or null when it did not exist.
*/
suspend fun findCompactBlock(height: BlockHeight): CompactFormats.CompactBlock?
suspend fun findCompactBlock(height: BlockHeight): JniBlockMeta?
/**
* Write the given blocks to this store, which may be anything from an in-memory cache to a DB.
@ -35,9 +36,4 @@ interface CompactBlockRepository {
* @param height the target height to which to rewind.
*/
suspend fun rewindTo(height: BlockHeight)
/**
* Close any connections to the block store.
*/
suspend fun close()
}

View File

@ -0,0 +1,161 @@
package cash.z.ecc.android.sdk.internal.storage.block
import androidx.annotation.VisibleForTesting
import cash.z.ecc.android.sdk.internal.Twig
import cash.z.ecc.android.sdk.internal.ext.createNewFileSuspend
import cash.z.ecc.android.sdk.internal.ext.deleteSuspend
import cash.z.ecc.android.sdk.internal.ext.existsSuspend
import cash.z.ecc.android.sdk.internal.ext.mkdirsSuspend
import cash.z.ecc.android.sdk.internal.ext.renameToSuspend
import cash.z.ecc.android.sdk.internal.ext.toHexReversed
import cash.z.ecc.android.sdk.internal.ext.writeBytesSuspend
import cash.z.ecc.android.sdk.internal.model.JniBlockMeta
import cash.z.ecc.android.sdk.internal.repository.CompactBlockRepository
import cash.z.ecc.android.sdk.jni.RustBackend
import cash.z.ecc.android.sdk.jni.RustBackendWelding
import cash.z.ecc.android.sdk.model.BlockHeight
import cash.z.wallet.sdk.internal.rpc.CompactFormats.CompactBlock
import java.io.File
internal class FileCompactBlockRepository(
private val blocksDirectory: File,
private val rustBackend: RustBackendWelding
) : CompactBlockRepository {
override suspend fun getLatestHeight() = rustBackend.getLatestHeight()
override suspend fun findCompactBlock(height: BlockHeight) = rustBackend.findBlockMetadata(height)
override suspend fun write(result: Sequence<CompactBlock>): Int {
var count = 0
val metaDataBuffer = mutableListOf<JniBlockMeta>()
result.forEach { block ->
val tmpFile = block.createTemporaryFile(blocksDirectory)
// write compact block bytes
tmpFile.writeBytesSuspend(block.toByteArray())
// buffer metadata
metaDataBuffer.add(block.toJniMetaData())
val isFinalizeSuccessful = tmpFile.finalizeFile()
check(isFinalizeSuccessful) {
"Failed to finalize file: ${tmpFile.absolutePath}"
}
count++
if (metaDataBuffer.isBufferFull()) {
// write blocks metadata to storage when the buffer is full
rustBackend.writeBlockMetadata(metaDataBuffer)
metaDataBuffer.clear()
}
}
if (metaDataBuffer.isNotEmpty()) {
// write the rest of the blocks metadata to storage even though the buffer is not full
rustBackend.writeBlockMetadata(metaDataBuffer)
metaDataBuffer.clear()
}
return count
}
override suspend fun rewindTo(height: BlockHeight) = rustBackend.rewindBlockMetadataToHeight(height)
companion object {
/**
* The name of the directory for downloading blocks
*/
const val BLOCKS_DOWNLOAD_DIRECTORY = "blocks"
/**
* The suffix for temporary files
*/
const val TEMPORARY_FILENAME_SUFFIX = ".tmp"
/**
* The suffix for block file name
*/
const val BLOCK_FILENAME_SUFFIX = "-compactblock"
/**
* The size of block meta data buffer
*/
const val BLOCKS_METADATA_BUFFER_SIZE = 10
suspend fun new(
rustBackend: RustBackend
): FileCompactBlockRepository {
Twig.debug { "${rustBackend.fsBlockDbRoot.absolutePath} \n ${rustBackend.dataDbFile.absolutePath}" }
// create and check cache directories
val blocksDirectory = File(rustBackend.fsBlockDbRoot, BLOCKS_DOWNLOAD_DIRECTORY).also {
it.mkdirsSuspend()
}
if (!blocksDirectory.existsSuspend()) {
error("${blocksDirectory.path} directory does not exist and could not be created.")
}
rustBackend.initBlockMetaDb()
return FileCompactBlockRepository(blocksDirectory, rustBackend)
}
}
}
//
// Private helper functions
//
private fun List<JniBlockMeta>.isBufferFull(): Boolean {
return size % FileCompactBlockRepository.BLOCKS_METADATA_BUFFER_SIZE == 0
}
internal data class CompactBlockOutputsCounts(
val saplingOutputsCount: UInt,
val orchardActionsCount: UInt
)
private fun CompactBlock.getOutputsCounts(): CompactBlockOutputsCounts {
var outputsCount: UInt = 0u
var actionsCount: UInt = 0u
vtxList.forEach { compactTx ->
outputsCount += compactTx.outputsCount.toUInt()
actionsCount += compactTx.actionsCount.toUInt()
}
return CompactBlockOutputsCounts(outputsCount, actionsCount)
}
private fun CompactBlock.toJniMetaData(): JniBlockMeta {
val outputs = getOutputsCounts()
return JniBlockMeta.new(this, outputs)
}
private fun CompactBlock.createFilename(): String {
val hashHex = hash.toByteArray().toHexReversed()
return "$height-$hashHex${FileCompactBlockRepository.BLOCK_FILENAME_SUFFIX}"
}
@VisibleForTesting(otherwise = VisibleForTesting.PRIVATE)
internal suspend fun CompactBlock.createTemporaryFile(blocksDirectory: File): File {
val tempFileName = "${createFilename()}${FileCompactBlockRepository.TEMPORARY_FILENAME_SUFFIX}"
val tmpFile = File(blocksDirectory, tempFileName)
if (tmpFile.existsSuspend()) {
tmpFile.deleteSuspend()
}
tmpFile.createNewFileSuspend()
return tmpFile
}
@VisibleForTesting(otherwise = VisibleForTesting.PRIVATE)
internal suspend fun File.finalizeFile(): Boolean {
// rename the file
val newFile = File(absolutePath.dropLast(FileCompactBlockRepository.TEMPORARY_FILENAME_SUFFIX.length))
return renameToSuspend(newFile)
}

View File

@ -3,8 +3,10 @@ package cash.z.ecc.android.sdk.jni
import cash.z.ecc.android.sdk.internal.SaplingParamTool
import cash.z.ecc.android.sdk.internal.SdkDispatchers
import cash.z.ecc.android.sdk.internal.Twig
import cash.z.ecc.android.sdk.internal.ext.deleteRecursivelySuspend
import cash.z.ecc.android.sdk.internal.ext.deleteSuspend
import cash.z.ecc.android.sdk.internal.model.Checkpoint
import cash.z.ecc.android.sdk.internal.model.JniBlockMeta
import cash.z.ecc.android.sdk.model.BlockHeight
import cash.z.ecc.android.sdk.model.UnifiedSpendingKey
import cash.z.ecc.android.sdk.model.WalletBalance
@ -25,25 +27,49 @@ internal class RustBackend private constructor(
override val network: ZcashNetwork,
val birthdayHeight: BlockHeight,
val dataDbFile: File,
val cacheDbFile: File,
val fsBlockDbRoot: File,
override val saplingParamDir: File
) : RustBackendWelding {
suspend fun clear(clearCacheDb: Boolean = true, clearDataDb: Boolean = true) {
if (clearCacheDb) {
Twig.debug { "Deleting the cache database!" }
cacheDbFile.deleteSuspend()
/**
* This function deletes the data database file and the cache directory (compact blocks files) if set by input
* parameters.
*
* @param clearCache to request the cache directory and its content deletion
* @param clearDataDb to request the data database file deletion
*
* @return false in case of any required and failed deletion, true otherwise.
*/
suspend fun clear(clearCache: Boolean = true, clearDataDb: Boolean = true): Boolean {
var cacheClearResult = true
var dataClearResult = true
if (clearCache) {
Twig.debug { "Deleting the cache files..." }
fsBlockDbRoot.deleteRecursivelySuspend().also { result ->
Twig.debug { "Deletion of the cache files ${if (result) "succeeded" else "failed"}!" }
cacheClearResult = result
}
}
if (clearDataDb) {
Twig.debug { "Deleting the data database!" }
dataDbFile.deleteSuspend()
Twig.debug { "Deleting the data database..." }
dataDbFile.deleteSuspend().also { result ->
Twig.debug { "Deletion of the data database ${if (result) "succeeded" else "failed"}!" }
dataClearResult = result
}
}
return cacheClearResult && dataClearResult
}
//
// Wrapper Functions
//
override suspend fun initBlockMetaDb() = withContext(SdkDispatchers.DATABASE_IO) {
initBlockMetaDb(
fsBlockDbRoot.absolutePath,
)
}
override suspend fun initDataDb(seed: ByteArray?) = withContext(SdkDispatchers.DATABASE_IO) {
initDataDb(
dataDbFile.absolutePath,
@ -145,27 +171,64 @@ internal class RustBackend private constructor(
)
}
override suspend fun getSentMemoAsUtf8(idNote: Long) = withContext(SdkDispatchers.DATABASE_IO) {
getSentMemoAsUtf8(
dataDbFile.absolutePath,
idNote,
networkId = network.id
)
}
override suspend fun validateCombinedChain() = withContext(SdkDispatchers.DATABASE_IO) {
val validationResult = validateCombinedChain(
cacheDbFile.absolutePath,
dataDbFile.absolutePath,
networkId = network.id
)
if (-1L == validationResult) {
null
} else {
BlockHeight.new(network, validationResult)
override suspend fun getSentMemoAsUtf8(idNote: Long) =
withContext(SdkDispatchers.DATABASE_IO) {
getSentMemoAsUtf8(
dataDbFile.absolutePath,
idNote,
networkId = network.id
)
}
override suspend fun writeBlockMetadata(blockMetadata: List<JniBlockMeta>) =
withContext(SdkDispatchers.DATABASE_IO) {
writeBlockMetadata(
fsBlockDbRoot.absolutePath,
blockMetadata.toTypedArray()
)
}
override suspend fun getLatestHeight() =
withContext(SdkDispatchers.DATABASE_IO) {
val height = getLatestHeight(fsBlockDbRoot.absolutePath)
if (-1L == height) {
null
} else {
BlockHeight.new(network, height)
}
}
override suspend fun findBlockMetadata(height: BlockHeight) =
withContext(SdkDispatchers.DATABASE_IO) {
findBlockMetadata(
fsBlockDbRoot.absolutePath,
height.value
)
}
override suspend fun rewindBlockMetadataToHeight(height: BlockHeight) =
withContext(SdkDispatchers.DATABASE_IO) {
rewindBlockMetadataToHeight(
fsBlockDbRoot.absolutePath,
height.value
)
}
override suspend fun validateCombinedChain() =
withContext(SdkDispatchers.DATABASE_IO) {
val validationResult = validateCombinedChain(
fsBlockDbRoot.absolutePath,
dataDbFile.absolutePath,
networkId = network.id
)
if (-1L == validationResult) {
null
} else {
BlockHeight.new(network, validationResult)
}
}
}
override suspend fun getNearestRewindHeight(height: BlockHeight): BlockHeight =
withContext(SdkDispatchers.DATABASE_IO) {
@ -196,7 +259,7 @@ internal class RustBackend private constructor(
override suspend fun scanBlocks(limit: Int): Boolean {
return withContext(SdkDispatchers.DATABASE_IO) {
scanBlocks(
cacheDbFile.absolutePath,
fsBlockDbRoot.absolutePath,
dataDbFile.absolutePath,
limit,
networkId = network.id
@ -340,7 +403,7 @@ internal class RustBackend private constructor(
* function once, it is idempotent.
*/
suspend fun init(
cacheDbFile: File,
fsBlockDbRoot: File,
dataDbFile: File,
saplingParamsDir: File,
zcashNetwork: ZcashNetwork,
@ -352,7 +415,7 @@ internal class RustBackend private constructor(
zcashNetwork,
birthdayHeight,
dataDbFile = dataDbFile,
cacheDbFile = cacheDbFile,
fsBlockDbRoot = fsBlockDbRoot,
saplingParamDir = saplingParamsDir
)
}
@ -364,6 +427,9 @@ internal class RustBackend private constructor(
@JvmStatic
private external fun initOnLoad()
@JvmStatic
private external fun initBlockMetaDb(fsBlockDbRoot: String): Int
@JvmStatic
private external fun initDataDb(dbDataPath: String, seed: ByteArray?, networkId: Int): Int
@ -440,6 +506,27 @@ internal class RustBackend private constructor(
networkId: Int
): String?
@JvmStatic
private external fun writeBlockMetadata(
dbCachePath: String,
blockMeta: Array<JniBlockMeta>
): Boolean
@JvmStatic
private external fun getLatestHeight(dbCachePath: String): Long
@JvmStatic
private external fun findBlockMetadata(
dbCachePath: String,
height: Long
): JniBlockMeta?
@JvmStatic
private external fun rewindBlockMetadataToHeight(
dbCachePath: String,
height: Long
)
@JvmStatic
private external fun validateCombinedChain(
dbCachePath: String,

View File

@ -1,6 +1,7 @@
package cash.z.ecc.android.sdk.jni
import cash.z.ecc.android.sdk.internal.model.Checkpoint
import cash.z.ecc.android.sdk.internal.model.JniBlockMeta
import cash.z.ecc.android.sdk.model.Account
import cash.z.ecc.android.sdk.model.BlockHeight
import cash.z.ecc.android.sdk.model.UnifiedSpendingKey
@ -23,6 +24,8 @@ internal interface RustBackendWelding {
val saplingParamDir: File
suspend fun initBlockMetaDb(): Int
@Suppress("LongParameterList")
suspend fun createToAddress(
usk: UnifiedSpendingKey,
@ -78,6 +81,17 @@ internal interface RustBackendWelding {
suspend fun scanBlocks(limit: Int = -1): Boolean
suspend fun writeBlockMetadata(blockMetadata: List<JniBlockMeta>): Boolean
/**
* @return The latest height in the CompactBlock cache metadata DB, or Null if no blocks have been cached.
*/
suspend fun getLatestHeight(): BlockHeight?
suspend fun findBlockMetadata(height: BlockHeight): JniBlockMeta?
suspend fun rewindBlockMetadataToHeight(height: BlockHeight)
/**
* @return Null if successful. If an error occurs, the height will be the height where the error was detected.
*/

View File

@ -34,11 +34,13 @@ use zcash_client_backend::{
wallet::{OvkPolicy, WalletTransparentOutput},
zip321::{Payment, TransactionRequest},
};
use zcash_client_sqlite::chain::init::init_blockmeta_db;
#[allow(deprecated)]
use zcash_client_sqlite::wallet::get_rewind_height;
use zcash_client_sqlite::{
chain::BlockMeta,
wallet::init::{init_accounts_table, init_blocks_table, init_wallet_db, WalletMigrationError},
BlockDb, NoteId, WalletDb,
FsBlockDb, NoteId, WalletDb,
};
use zcash_primitives::consensus::Network::{MainNetwork, TestNetwork};
use zcash_primitives::{
@ -47,7 +49,7 @@ use zcash_primitives::{
legacy::{Script, TransparentAddress},
memo::{Memo, MemoBytes},
transaction::{
components::{Amount, OutPoint, TxOut},
components::{amount::NonNegativeAmount, Amount, OutPoint, TxOut},
Transaction,
},
zip32::{AccountId, DiversifierIndex},
@ -89,9 +91,9 @@ fn wallet_db<P: Parameters>(
.map_err(|e| format_err!("Error opening wallet database connection: {}", e))
}
fn block_db(env: &JNIEnv<'_>, db_data: JString<'_>) -> Result<BlockDb, failure::Error> {
BlockDb::for_path(utils::java_string_to_rust(&env, db_data))
.map_err(|e| format_err!("Error opening block source database connection: {}", e))
fn block_db(env: &JNIEnv<'_>, fsblockdb_root: JString<'_>) -> Result<FsBlockDb, failure::Error> {
FsBlockDb::for_path(utils::java_string_to_rust(&env, fsblockdb_root))
.map_err(|e| format_err!("Error opening block source database connection: {:?}", e))
}
#[no_mangle]
@ -130,6 +132,29 @@ pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_initOnLoad(
print_debug_state();
}
/// Sets up the internal structure of the blockmeta database.
///
/// Returns 0 if successful, or -1 otherwise.
#[no_mangle]
pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_initBlockMetaDb(
env: JNIEnv<'_>,
_: JClass<'_>,
fsblockdb_root: JString<'_>,
) -> jint {
let res = panic::catch_unwind(|| {
let mut db_meta = block_db(&env, fsblockdb_root)?;
match init_blockmeta_db(&mut db_meta) {
Ok(()) => Ok(0),
Err(e) => Err(format_err!(
"Error while initializing block metadata DB: {}",
e
)),
}
});
unwrap_exc_or(&env, res, -1)
}
/// Sets up the internal structure of the data database.
///
/// If `seed` is `null`, database migrations will be attempted without it.
@ -883,6 +908,145 @@ pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_getSentMemo
unwrap_exc_or(&env, res, ptr::null_mut())
}
fn encode_blockmeta(env: &JNIEnv<'_>, meta: BlockMeta) -> Result<jobject, failure::Error> {
let block_hash = env.byte_array_from_slice(&meta.block_hash.0)?;
let output = env.new_object(
"cash/z/ecc/android/sdk/internal/model/JniBlockMeta",
"(J[BJJJ)V",
&[
JValue::Long(i64::from(u32::from(meta.height))),
JValue::Object(unsafe { JObject::from_raw(block_hash) }),
JValue::Long(i64::from(meta.block_time)),
JValue::Long(i64::from(meta.sapling_outputs_count)),
JValue::Long(i64::from(meta.orchard_actions_count)),
],
)?;
Ok(output.into_raw())
}
fn decode_blockmeta(env: &JNIEnv<'_>, obj: JObject<'_>) -> Result<BlockMeta, failure::Error> {
let long_as_u32 = |name| -> Result<u32, failure::Error> {
Ok(u32::try_from(env.get_field(obj, name, "J")?.j()?)?)
};
fn byte_array<const N: usize>(
env: &JNIEnv<'_>,
obj: JObject<'_>,
name: &str,
) -> Result<[u8; N], failure::Error> {
let field = env.get_field(obj, name, "[B")?.l()?.into_raw();
Ok(env.convert_byte_array(field)?[..].try_into()?)
}
Ok(BlockMeta {
height: BlockHeight::from_u32(long_as_u32("height")?),
block_hash: BlockHash(byte_array(env, obj, "hash")?),
block_time: long_as_u32("time")?,
sapling_outputs_count: long_as_u32("saplingOutputsCount")?,
orchard_actions_count: long_as_u32("orchardOutputsCount")?,
})
}
#[no_mangle]
pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_writeBlockMetadata(
env: JNIEnv<'_>,
_: JClass<'_>,
db_cache: JString<'_>,
block_meta: jobjectArray,
) -> jboolean {
let res = panic::catch_unwind(|| {
let block_db = block_db(&env, db_cache)?;
let block_meta = {
let count = env.get_array_length(block_meta).unwrap();
(0..count)
.map(|i| {
env.get_object_array_element(block_meta, i)
.map_err(|e| e.into())
.and_then(|jobj| decode_blockmeta(&env, jobj))
})
.collect::<Result<Vec<_>, _>>()?
};
match block_db.write_block_metadata(&block_meta) {
Ok(()) => Ok(JNI_TRUE),
Err(e) => Err(format_err!(
"Failed to write block metadata to FsBlockDb: {:?}",
e
)),
}
});
unwrap_exc_or(&env, res, JNI_FALSE)
}
#[no_mangle]
pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_getLatestHeight(
env: JNIEnv<'_>,
_: JClass<'_>,
fsblockdb_root: JString<'_>,
) -> jlong {
let res = panic::catch_unwind(|| {
let block_db = block_db(&env, fsblockdb_root)?;
match block_db.get_max_cached_height() {
Ok(Some(block_height)) => Ok(i64::from(u32::from(block_height))),
// Use -1 to return null across the FFI.
Ok(None) => Ok(-1),
Err(e) => Err(format_err!(
"Failed to read block metadata from FsBlockDb: {:?}",
e
)),
}
});
unwrap_exc_or(&env, res, -1)
}
#[no_mangle]
pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_findBlockMetadata(
env: JNIEnv<'_>,
_: JClass<'_>,
fsblockdb_root: JString<'_>,
height: jlong,
) -> jobject {
let res = panic::catch_unwind(|| {
let block_db = block_db(&env, fsblockdb_root)?;
let height = BlockHeight::try_from(height)?;
match block_db.find_block(height) {
Ok(Some(meta)) => encode_blockmeta(&env, meta),
Ok(None) => Ok(ptr::null_mut()),
Err(e) => Err(format_err!(
"Failed to read block metadata from FsBlockDb: {:?}",
e
)),
}
});
unwrap_exc_or(&env, res, ptr::null_mut())
}
#[no_mangle]
pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_rewindBlockMetadataToHeight(
env: JNIEnv<'_>,
_: JClass<'_>,
fsblockdb_root: JString<'_>,
height: jlong,
) {
let res = panic::catch_unwind(|| {
let block_db = block_db(&env, fsblockdb_root)?;
let height = BlockHeight::try_from(height)?;
block_db.rewind_to_height(height).map_err(|e| {
format_err!(
"Error while rewinding block metadata DB to height {}: {}",
height,
e
)
})
});
unwrap_exc_or(&env, res, ())
}
#[no_mangle]
pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_validateCombinedChain(
env: JNIEnv<'_>,
@ -900,7 +1064,7 @@ pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_validateCom
.get_max_height_hash()
.map_err(|e| format_err!("Error while validating chain: {}", e))?;
let val_res = validate_chain(&network, &block_db, validate_from);
let val_res = validate_chain(&block_db, validate_from, None);
if let Err(e) = val_res {
match e {
@ -908,7 +1072,7 @@ pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_validateCom
let upper_bound_u32 = u32::from(e.at_height());
Ok(upper_bound_u32 as i64)
}
_ => Err(format_err!("Error while validating chain: {}", e)),
_ => Err(format_err!("Error while validating chain: {:?}", e)),
}
} else {
// All blocks are valid, so "highest invalid block height" is below genesis.
@ -997,7 +1161,7 @@ pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_scanBlocks(
match scan_cached_blocks(&network, &db_cache, &mut db_data, limit) {
Ok(()) => Ok(JNI_TRUE),
Err(e) => Err(format_err!(
"Rust error while scanning blocks (limit {:?}): {}",
"Rust error while scanning blocks (limit {:?}): {:?}",
limit,
e
)),
@ -1251,6 +1415,8 @@ pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_shieldToAdd
let prover = LocalTxProver::new(Path::new(&spend_params), Path::new(&output_params));
let shielding_threshold = NonNegativeAmount::from_u64(100000).unwrap();
zip317_helper(
(&mut db_data, prover),
use_zip317_fees,
@ -1260,6 +1426,7 @@ pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_shieldToAdd
&network,
prover,
&input_selector,
shielding_threshold,
&usk,
&from_addrs,
&MemoBytes::from(&memo),
@ -1273,6 +1440,7 @@ pub unsafe extern "C" fn Java_cash_z_ecc_android_sdk_jni_RustBackend_shieldToAdd
&network,
prover,
&input_selector,
shielding_threshold,
&usk,
&from_addrs,
&MemoBytes::from(&memo),