* Switch to FsBlockDb for caching CompactBlocks
* Add RustBackend.getLatestHeight() method
* Raise MSRV to 1.60
* Migrate to latest Rust crate API
* Add RustBackend.findBlockMetadata() method
* Add RustBackend.rewindBlockMetadataToHeight() method
* [#765] implementation of FileCompactBlockRepository
* writing block metadata to database
* split write function into smaller easier to test blocks
* testing for FileCompactBlockRepository
* fixed rewinding
* fixed tests
* fixed FileCompactBlockRepositoryTest and SynchronizerFactoryTest
* code review fixes
* updated proto files
* override all functions in FakeRustBackend
* code review fixes
* Fix function body formatting
* Improve clear function clarity
* Use length of string const
* Delete single file instead of directory
* Improve function clarity
* Refactor outputs counting
- Found a typo in intermediary model class JniBlockMeta parameter change of which does not impact encoding from rust to kotlin according to rust layer implementation.
* Check blocks mkdir result
* Remove unnecessary detekt warning suppression
* Refactor buffer size check
* Improve visibility annotations
* Make file finalise obvious and self documenting
* Remove prevHash logging
* Move instantiation to the object itself
* Enrich fixture with default values
* Extend eror message
* Rename benchmark blocks range fixture
* Fix rebase changes
* Improve FileCompactBlockRepositoryTest
- "De-integrated" the test suite - it now works with fixture blocks
- Created needed fixtures for a clear mocked blocks providing
- Enhanced getting of FileCompactBlockRepository in FileCompactBlockRepositoryTest to clarify that it works with mock components
* Fix ktlintFormat findings
* Bump actions/cache from 3.2.4 to 3.2.5 in /.github/actions/setup (#927)
Bumps [actions/cache](https://github.com/actions/cache) from 3.2.4 to 3.2.5.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](627f0f41f6...6998d139dd)
---
updated-dependencies:
- dependency-name: actions/cache
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Fix incorrect dir checking
* More robust min/max checking
* Rewinding to more robust middle value
* Strengthen blocks counts after rewind
* Add check of temp finalized file
* Refactor DatabaseCoordinatorTest
* Rename cache files root directory
- To have the same unified variable name across our repositories - fsBlockDbRoot
* Refactor FileCompactBlockRepositoryTest
- To have these tests more clear
- Fixed FakeRustBackend instantiation bug
* Revert back JniBlockMeta param name
* Delete legacy Cache db files
- Deleted from both the older and the newer legacy locations
- All related db files deleted - rollback files included
- The deletion is run once we try to access the new store blocks on disk root directory
- The deletion check does not throw any exception in case of failure, we just log it in console and try it on the next time
- Related new test added too
* Test refactoring
- Made few changes to improve clarity of provided tests and fixtures
- Prepared few new "failure path" tests
- Enhanced existing tests
* Manual test case
* Simplify error printing
- As we had some commented out code there
* Reset manual tests steps numbering
* [#924] Remove alias from WalletCoordinator
Also make Compose UI the default. The old UI is deprecated but is still used by the benchmarking tests
* Bump benchmark version
* Protect JniBlockMetadata agains minification
* Enable debuggable while benchmarking
* Enhance benchmark screen waiting
* Fix cache db files deletion
- With this construct we delete all blocks blob metadata files, as well as their sqlite file
* Add new benchmark results
* Remove benchmark operations receiver fix
- As it's not needed after the latest profiler dependency update
* Bump gradle/wrapper-validation-action from 1.0.5 to 1.0.6 (#928)
Bumps [gradle/wrapper-validation-action](https://github.com/gradle/wrapper-validation-action) from 1.0.5 to 1.0.6.
- [Release notes](https://github.com/gradle/wrapper-validation-action/releases)
- [Commits](55e685c48d...8d49e559aa)
---
updated-dependencies:
- dependency-name: gradle/wrapper-validation-action
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump actions/cache from 3.2.5 to 3.2.6 in /.github/actions/setup (#929)
Bumps [actions/cache](https://github.com/actions/cache) from 3.2.5 to 3.2.6.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](6998d139dd...69d9d449ac)
---
updated-dependencies:
- dependency-name: actions/cache
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update DatabaseCoordinator.kt
Move if/else out from log message body
* Update Switch cache to store blocks on disk.md
Reset documentation bullets numbering
* Hide internal constants from public package
* Update manual test user with new UI requirement
* Comment out cargo.toml test values
* Inline fixture blocks metadata with its creation
* Switch to FsBlockDb for caching CompactBlocks
* Add RustBackend.getLatestHeight() method
* Raise MSRV to 1.60
* Migrate to latest Rust crate API
* Add RustBackend.findBlockMetadata() method
* Add RustBackend.rewindBlockMetadataToHeight() method
* Update cargo.lock
* Switch to FsBlockDb for caching CompactBlocks
* Migrate to latest Rust crate API
* [#765] implementation of FileCompactBlockRepository
* writing block metadata to database
* split write function into smaller easier to test blocks
* testing for FileCompactBlockRepository
* fixed rewinding
* fixed tests
* fixed FileCompactBlockRepositoryTest and SynchronizerFactoryTest
* code review fixes
* updated proto files
* override all functions in FakeRustBackend
* code review fixes
* Fix function body formatting
* Improve clear function clarity
* Use length of string const
* Delete single file instead of directory
* Improve function clarity
* Refactor outputs counting
- Found a typo in intermediary model class JniBlockMeta parameter change of which does not impact encoding from rust to kotlin according to rust layer implementation.
* Check blocks mkdir result
* Remove unnecessary detekt warning suppression
* Refactor buffer size check
* Improve visibility annotations
* Make file finalise obvious and self documenting
* Remove prevHash logging
* Move instantiation to the object itself
* Enrich fixture with default values
* Extend eror message
* Rename benchmark blocks range fixture
* Fix rebase changes
* Improve FileCompactBlockRepositoryTest
- "De-integrated" the test suite - it now works with fixture blocks
- Created needed fixtures for a clear mocked blocks providing
- Enhanced getting of FileCompactBlockRepository in FileCompactBlockRepositoryTest to clarify that it works with mock components
* Fix ktlintFormat findings
* Fix incorrect dir checking
* More robust min/max checking
* Rewinding to more robust middle value
* Strengthen blocks counts after rewind
* Add check of temp finalized file
* Refactor DatabaseCoordinatorTest
* Rename cache files root directory
- To have the same unified variable name across our repositories - fsBlockDbRoot
* Refactor FileCompactBlockRepositoryTest
- To have these tests more clear
- Fixed FakeRustBackend instantiation bug
* Revert back JniBlockMeta param name
* Delete legacy Cache db files
- Deleted from both the older and the newer legacy locations
- All related db files deleted - rollback files included
- The deletion is run once we try to access the new store blocks on disk root directory
- The deletion check does not throw any exception in case of failure, we just log it in console and try it on the next time
- Related new test added too
* Test refactoring
- Made few changes to improve clarity of provided tests and fixtures
- Prepared few new "failure path" tests
- Enhanced existing tests
* Manual test case
* Simplify error printing
- As we had some commented out code there
* Reset manual tests steps numbering
* Bump benchmark version
* Protect JniBlockMetadata agains minification
* Enable debuggable while benchmarking
* Enhance benchmark screen waiting
* Fix cache db files deletion
- With this construct we delete all blocks blob metadata files, as well as their sqlite file
* Add new benchmark results
* Remove benchmark operations receiver fix
- As it's not needed after the latest profiler dependency update
* Update DatabaseCoordinator.kt
Move if/else out from log message body
* Update Switch cache to store blocks on disk.md
Reset documentation bullets numbering
* Hide internal constants from public package
* Update manual test user with new UI requirement
* Comment out cargo.toml test values
* Inline fixture blocks metadata with its creation
* Update cargo.lock
* Update Cargo.lock
* Check and document JniBlockMeta params ranges
* Add assert for gradle property
* Use UInt internally
uint
* Change array API to list
* Fix for using correct SDK Synchronizer alias
- Only the older Demo-app UI was impacted.
- Thus also our benchmarking was impacted.
* Apply documentation suggestions from code review
Co-authored-by: Kris Nuttycombe <kris@electriccoin.co>
* Final manual test instructions update
* Fixture block hash deterministically generated from block height
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jack Grigg <jack@electriccoin.co>
Co-authored-by: Jack Grigg <jack@z.cash>
Co-authored-by: Honza <rychnovsky.honza@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
Co-authored-by: Kris Nuttycombe <kris@electriccoin.co>
* [#615] Refactor lightwalletd client
This moves the lightwalletd client code to a separate Gradle module, but does not yet do much clean up the public API (e.g. hiding generated GRPC objects). That will be a followon change via #673
I’ve prototyped a safer version of the API that would be implemented for #673 for two specific calls: obtaining the current block height and obtaining the light wallet info. These were the easiest endpoints to update and they provide a useful exploration for the future changes needed.
* Fix benchmarking for networking module
- Moved to fixture and build type check components to the new networking module, so it's accessible from all needed parts of the projects
- Changed fixture return type to fit to all needed usages
* Align with previous review comment
* Fix wrong merge
* Add basic sanity test
- To satisfy tests requirements on emulators.wtf service
* Remove grpc dependency from sdk-lib module
* Repackage cash.z.wallet.sdk.rpc to cash.z.wallet.sdk.internal.rpc
* Fix BuildConfig path
* Update demo-app paths to rpc
* Fix broken grpc services locations
- Our aim here is to change only the local services location (package name), the server location can't be changed due to backward compatibility.
* Enhance GRPC Response model
* Adopt enhanced GRPC response model
- Adopted in a few endpoints
* Enhance Connection failure type
* Add simple fixture test
* Refactor fetchTransaction() to work with Response
- Refactored this server call to adopt our new Response mechanism
- GrpsStatusResolver.resolveFailureFromStatus() now accepts Throwable input
- Added Response.Failure.toThrowable() to satisfy subsequent server calls results processing
- A few other minor refactoring changes
* Remove commented out log
* Unify return type of collection returning methods
- Switched fetchUtxos() to return Sequence instead of List
- Added a check on empty tAddress input
* fetchUtxos returns flow
- Switched fetchUtxos() to return Flow of Service.GetAddressUtxosReply
- Internally it calls getAddressUtxosStream() instead of getAddressUtxos() from GRPC layer
* Update unsafe suffix documentation
* Address several minor change requests
* Remove code parameter
- Removed from the locally triggered failures with default codes.
* Rename local variable
* Switch from empty response to exception
- Our server calls now rather throw IllegalArgumentException than return an empty response
- Removed commented out log message
- Updated javadocs
* Update proto files
- Plus related api changes adoption
* Unify new clients instances name
* AGP 7.4.0 changes
- packagingOptions -> androidComponents in sdk-lib and lightwallet-client-lib modules
* SDK version bump
* Response resolver tests
* Release build upload timeout increase
* Release build upload timeout increase fix
* Revert timeout
- As Github has some infrastructure troubles and we need to wait
* Add migrations documentation
* Sort packaging filters
* Remove private field from public documentation
* Hide private variables
* Remove package from Android Manifest
* Throw exception instead of custom error
- This step unify our approach to validation on client side across all server calls
* Replace setAddresses index const with number
* Fix indentation in proto file
---------
Co-authored-by: Honza <rychnovsky.honza@gmail.com>
* [#779] Resolve flaky DatabaseCoordinatorTest
- We agreed on just removing this flaky test, in which we hardly prepare the ideal conditions for it
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
* [#789] Add Benchmark module to Demo-app
* Code cleanup
* Opti-in experimental coroutines api in tests
* Add Testing documentation
* Documentation update + ktlint
* Check screen on and keyguard unlocked in each test
* Introduce UiAutomator extensions
* Enhance BenchmarkTrace events definition
* Remove unnecessary mutex
* Change blocks range
* Increase sync blockchain timeout
- To always fit in the Balances screen timeout for the selected blocks range
* Remove unnecessary fun suspend modifiers
* Macrobenchmark lib bump to 1.2.0-alpha8
* Remove duplicate label attr in app/manifest
* File and link benchmark on CI task
* Add proguard keep rules
* Documentation update
The Android and Rust code have previously managed joint custody of the derived data database. With more complex migrations now required, we need to make the Android side read-only. To achieve that, the Android side will remove Room and instead rely on more primitive SQLite APIs for read-only access.
As part of implementing this change, database management throughout the SDK is being refactored.
There will be multiple representations of the data:
- Database representation (and some Entity representations in the places that Room hasn’t been removed yet). These representations are not as type safe and don’t match Kotlin best practices in all cases.
- Once #615 is implemented there will also be network representations
- Type safe models, which often appear in the public API The Database and Network representations will be converted to and from the type safe model representation.
* [#665] Recover from corrupted sapling-spend.params and sapling-output.params
* Make parameters on exception internal
* Make SaplingParameters internal
* Fix for changed parameters visibility in tests
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
- Enhanced implementation of SaplingParamTool component. It got sapling files move from legacy folder to the preferred one functionality. It downloads the sapling files atomically now (through the temporary file names). It contains all related constants now. It works with SaplingParamToolProperties, which allows us to easily test the SaplingParamTool functionalities. It also now checks file hashes. Removed unnecessary clear function from the component. Changed valid function.
- Moved related constants from ZcashSdk class to SaplingParamTool class
- Changed Initializer, WalletTransactionEncoder and RustBackend classes to work with File instead of path String
- Minor changes in comments in other classes
- Added getSha1Hash() extension function to the FileExt class
* Related tests
- Two new test fixtures to simplify our tests
- Test for getSha1Hash() extension function
- Minor changes in existing tests
- Created new SaplingParamToolBasicTest, which covers non-integration functionality of SaplingParamTool
- Moved integration tests of the SaplingParamTool to the new SaplingParamToolIntegrationTest and added some new
* Related manual tests
- Created Download sapling files manual test
- Created Move sapling files to no_backup manual test
- Update existing Move database files to no_bakcup manual test
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
* [#664] Transfer maximum bytes from remote file
* Tests for limiting sapling files max size
* Address comments from review
* Optimization of sapling files download
* Add units to data class
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
* [#509] Remove OkHttp and Okio dependencies
- Okio -> Java NIO anf OkHttp -> HttpURLConnection in SaplingParamTool class
- Both replaced also in the test packages
- Removed dependencies to the replaced components
- Again switched a few assertions from JUnit to kotlin.test
- Created readLinesInFlow() function which is now reused in few places and runs without OkHttp
- Added subsequent TODOs to code base
- Wrapped to withContext to move it to an IO thread.
- Added use function on the stream and close() at the end of the channels use.
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
* Disable baseline file. Too many functions.
* CurrencyFormatter.kt suppress too many functions
* PersistentTransactionManager.kt suppress too many functions
* OutboundTransactionManager suppress too many functions
* Suppress long parameter list
* Too many functions
* Add log to avoid empty block warning
* Fix several magic number warnings
* Solve max line length warnings
* Solve max line length warnings
* Suppress too long method warnings
* Suppress too complex method warnings
* Suppress large class warning
* Fixed empty catch block
* Changed directory path to the file
* Fix too generic and swallowed exception
* Fix print stack trace warning
* Suppressed single top level file name declaration
* Change parameters name
* Suppress Spread operator warning
* Remove unused private code
* Add Locale to suppress default locale used warning
* Solve several forbidden TODOs warnings
* Fixed another max line length warning
* Simplify return statement
* Suppress class to object change
* Make DemoConstants variables const
* Use error() instead of throwing an IllegalStateException
* Solve too complex condition
* Suppress intentionally generic and swallowed exception
* Suppress TooGenericExceptionCaught
* Solve or suppress several TooGenericExceptionCaught
* Fix swallowed exception
* Suppress warning TooGenericExceptionCaught of PersistentTransactionManager
* Suppress warning TooGenericExceptionCaught of WalletTransactionEncoder
* Suppress TooGenericExceptionCaught of SdkSynchronizer
* Suppress TooGenericExceptionCaught in SaplingParamTool
* Suppress TooGenericExceptionCaught in CompactBlockDownloader
* Suppress TooGenericExceptionCaught in CheckpointTool
* Fix TooGenericExceptionCaught in WalletService
* Suppress TooGenericExceptionCaught in DerivedDataDb
* Suppress TooGenericExceptionCaught in CompactBlockProcessor
* Apply ktlint format after all the previous changes
* Remove detekt baseline file
* Set Android studio right margin
* Address comments from review
* Suppress failing tests on CI
We need the transparent account privkey in order to be able to derive
the secret key for any leaf transparent address under the ZIP 316
diversified address tree.
UnifiedViewingKey had a bug that made it incompatible with ZIP 316.
For compatibility with the current `zcash/librustzcash` revision we use
a temporary fake UFVK encoding that stores sufficient information to use
the current APIs, and a superset of the actual ZIP 316 FVK information.
This change leverages the gRPC streaming API along with the elimination of bulk inserts to SQLite to reduce the amount of memory required to write blocks.
This includes a few different fixes that prevented the demo app and other SDK consumers from working.
First, initialization of the demo app was fixed to use the sapling activation height rather than the current latest checkpoint.
Second, CompactBlockProcessor wasn’t downloading blocks.
Third, a few potential thread safety issues were resolved by making fields that are accessed by multiple threads volatile.
- Replace int with type safe BlockHeight(Long) object
- Fix down casting bug, as BlockHeight is uint32 but Java only supports int32 or int64
- Rename WalletBirthday to Checkpoint and hide from the public API
This resolves several issues including:
- Updated checkpoints
- #550 Support for new checkpoint format with NU5
- #377 Removed explicit dependency on GSON
While there are a lot of changes, the most significant are:
- Update the main entrypoint of the SDK to use a suspending function and eliminate lazy initialization. This also resolves#310
- Changing Dao methods to `suspend`, allowing Room to implicitly manage the threading model under the hood
- Explicitly set the executor thread for the `CompactBlockDb` and `DerivedDataDb`, which is the same thread used when calling into the Rust native library. Room will then use this thread for `suspend` fun calls into it
- With the above set, remove unnecessary hops to Dispatchers.IO which is a thread pool
- Eliminate lazy initialization in `PagedTransactionRepository` to make it easier to reason about the threading model and resolve#355 where the database was being touched from the UI thread
Note that `PendingTransactionsDb` is not opted into this single-threaded approach, because that database is not shared with the Rust code
To quickly get this implemented, some calls in the demo-app have been wrapped in `runBlocking {}`. This is not ideal and will be addressed in followup issues.
Simplifies running the SDK test suite versus darkside tests (which require special environment setup).
Note that the darkside tests are still broken. This is not a regression, as they were broken before. This is an intermediate step towards fixing those tests.
Alex
Carter Jernigan
Honza Rychnovsky
Jack Grigg
Jack Grigg