* [#1170] Regular transaction flow emitting
Although zcash/librustzcash#918 will provide information on when to trigger transaction polling, this partly solves the issue as well
* Update changelog
* [#1180] Frequent SbS synchronization restarting
- Processing blocks with SbS split into preparation and processing functions, which can be called repeatedly.
- Refactored other parts of the synchronization mechanism
- Closes#1180
- This also partly solves #1137
* Update .gitignore
* Update LINCENSE documentation
* [#1177] Checkpoints update
* Fix Ktlint warning
* Jni objects attribute constraint tests
* Extend continuity error test
* ScanRange model tests
* [#1174] Move model classes out of the CompactBlockProcessor
* [#1174] SbS: Move model classes out of the CompactBlockProcessor
* Move SyncingResult out of the processor
* Add issue link
* [#1168] Checkpoints update
* [#1159] Updated sync progress reporting
* Increase test_robo_demo_app timeout
* Migrate to ClosedEndRange
It’s better to transform suggested ranges from OpenEndRange to ClosedRange as soon as possible to avoid its handling in the rest of the logic.
* Improve all batch count calculating
* Subsequent SbS sync algorithm renaming
* Add rewind button to Demo app
To be able to test refactored CompactBlockProcessor rewind feature.
* Rename non-linear to spend-before-sync
* [#1129] Refactor lastSyncRange lastSyncedHeight
* Fix updateRange function
As the overallSyncRange needs to be reset in case of all ranges are processed.
* [#1166] Remove alsoClearBlockCache parameter
* [#1133] Set the correct getSubtreeRoots inputs
- It also better handles setState calls
- It fixes OpenEndRange -> ClosedRange
- And it improves logging in syncing mechanism
* Increase build CI timeout
* [#1139][#1138] Process Priority.Verify blocks
This makes the new subtask functions more testable.
* Increase CI test timeout
* [#1047][#1140] Improve CBD internal state handling
* [#1047][1140] Improve CBD internal state handling
This improves internal state handling in CompactBlockProcessor. Error handling included. And it prepares us for ContinuityError handling too.
* [#1143] Handle ContinuityError
This handles the state in which is a transaction after it’s created with mined height null. It then doesn’t try to convert such a height into a BlockHeight object which is doing validation on input height. It rather returns null, similarly as for expiry height.
* [#1083] Detekt Compose 0.1.10
* [#1066] Detekt - MaxLineLength rule turn on
* Remove default values
As we keep only those that differ from the default in the latest Detekt update.
* [#1086] Improve TODO inline occurrence pattern
* [#920] Tweak RustBackend public APIs to have void return values
- Contains changes for putUtxo() function
* Changes for scanBlocks() function
* Changes for decryptAndStoreTransaction()
* Changes for rewindToHeight()
* Changes for writeBlockMetadata()
* Changes for initBlocksTable and initAccountsTable
Enhancing transaction details mechanism was refactored to provide more clarity about its inner phases (fetching and decrypting with storing) and to be able to retry when the transaction fetching fails due to flaky network condition
* [#1026] Consider Enhancing Every 1000 Synced Blocks
* Enhancing refactoring code cleaning
* Query un-enhanced transactions from repository
- And change enhancement range in case of any transaction found
- Changed enhancing trigger appropriately
- Covered also the case when the sync stage ended before the enhancement limit reached
- Cleaned up the code
* File and link sub-stages error handling follow-up
* File and link Duration instead of Long refactoring issue
---------
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
* rust: cargo update
* rust: Migrate to `zcash_client_sqlite 0.7`
* rust: cargo fmt
* Adopt zcash_client_sqlite 0.7 database changes
* [#1018] add limit parameter to validateCombinedChain
* Adopt limit parameter in Android layer
* Switch limit argument to Long
- Switched from Int to Long type
- Done for both validate and scan blocks functions
- Changed the default parameter for nullable one in the public interface to let callers be explicit about the argument
- Changed in the Rust layer as well
---------
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
Co-authored-by: Jack Grigg <jack@z.cash>
Co-authored-by: Honza <rychnovsky.honza@gmail.com>
* [#1009] Change CompactBlockProcessor methods signature
* Remove withContext {}
Since our networking, RustBackend, and CompactBlockRepository all use coroutines internally by implementing `suspend` for their APIs, we don’t need withContext() anymore. (If we discover something does do blocking IO on the thread, we should deal with it at the point the IO is occurring rather than at the top level)
* Sort FileExt functions
* Add isDirectorySuspend
- To avoid strict mode disc IO violation warning
* Omit try/catch on block files delete action
* Refactor scan batch function
- Removed retry mechanisms, as it’s not needed now after the previous switch from the database-based blocks persisting to file-based blocks persisting
- scan function now works with the current processing range of blocks size instead of hardcoded constant value
* Eliminate duplicate processorInfo object
* Change log text
* MutableList instead of ArrayList
* Eliminate sync range nullability
* Rewind not scanned blocks
If we find out that we previously downloaded, but not validated and scanned persisted blocks, we need to rewind the blocks above the last scanned height first.
* Fix ktlint warning
---------
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
* [#895] Adopt new Response result
* Change catch flow part
* Correct emit from flow catch
* Additional changes after Block store on disk merged
* Simplify networking model structure
* Enhance flow filter
* Switch to coroutine client in a few tests too
* Use StatusException
* Switch client in a few tests
* Switch client for download block case
* Link fetch UTXOs issue
* Refactor refresh and process UTXOs
* Improve downloadBlockRange logging
* Switch getTAddressTransactions to the Response result
* Move some parts of requests out of try-catch
* Remove link to issue
* Refactor fetchUtxos to return Response
* Throws custom exception from ResponseFailure
- To trigger existing re-try mechanisms in case of any particular trouble
* [#674] Use suspending functions for networking
- Initial commit to set up how we transform to the coroutine version of networking client
* Enhance flow filter
* Switch to coroutine client in a few tests too
* Use StatusException
* Comment out old UI demo app cases
* Switch client in a few tests
* Switch client for download block case
* Remove the blocking networking client entirely
* Link fetch UTXOs issue
* Refactor refresh and process UTXOs
* Improve downloadBlockRange logging
* Remove CompactBlockRepository.write(sequence)
* Remove “Coroutine” prefix
Since we only have a single implementation, the designation shouldn’t matter anymore
* Fix collection bug
* Finish network client renaming
* Collect write results internally
* Write method simplification
* Blocks sync logging reduce
* Unused unsafe models remove
---------
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
This is an initial step before moving the Rust backend into a separate Gradle module
- Use primitive values for the Rust backend interface. This will allow the backend to be moved into a new Gradle module in the future, without needing to access the model objects of the SDK
- Add extension functions to the Backend to replace the existing usage of more strongly typed objects. The extension functions can continue to live in the sdk-lib even after refactoring the Rust backend
- Add new decorator that can eventually replace using the Rust backend within the SDK
* Switch to FsBlockDb for caching CompactBlocks
* Add RustBackend.getLatestHeight() method
* Raise MSRV to 1.60
* Migrate to latest Rust crate API
* Add RustBackend.findBlockMetadata() method
* Add RustBackend.rewindBlockMetadataToHeight() method
* [#765] implementation of FileCompactBlockRepository
* writing block metadata to database
* split write function into smaller easier to test blocks
* testing for FileCompactBlockRepository
* fixed rewinding
* fixed tests
* fixed FileCompactBlockRepositoryTest and SynchronizerFactoryTest
* code review fixes
* updated proto files
* override all functions in FakeRustBackend
* code review fixes
* Fix function body formatting
* Improve clear function clarity
* Use length of string const
* Delete single file instead of directory
* Improve function clarity
* Refactor outputs counting
- Found a typo in intermediary model class JniBlockMeta parameter change of which does not impact encoding from rust to kotlin according to rust layer implementation.
* Check blocks mkdir result
* Remove unnecessary detekt warning suppression
* Refactor buffer size check
* Improve visibility annotations
* Make file finalise obvious and self documenting
* Remove prevHash logging
* Move instantiation to the object itself
* Enrich fixture with default values
* Extend eror message
* Rename benchmark blocks range fixture
* Fix rebase changes
* Improve FileCompactBlockRepositoryTest
- "De-integrated" the test suite - it now works with fixture blocks
- Created needed fixtures for a clear mocked blocks providing
- Enhanced getting of FileCompactBlockRepository in FileCompactBlockRepositoryTest to clarify that it works with mock components
* Fix ktlintFormat findings
* Bump actions/cache from 3.2.4 to 3.2.5 in /.github/actions/setup (#927)
Bumps [actions/cache](https://github.com/actions/cache) from 3.2.4 to 3.2.5.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](627f0f41f6...6998d139dd)
---
updated-dependencies:
- dependency-name: actions/cache
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Fix incorrect dir checking
* More robust min/max checking
* Rewinding to more robust middle value
* Strengthen blocks counts after rewind
* Add check of temp finalized file
* Refactor DatabaseCoordinatorTest
* Rename cache files root directory
- To have the same unified variable name across our repositories - fsBlockDbRoot
* Refactor FileCompactBlockRepositoryTest
- To have these tests more clear
- Fixed FakeRustBackend instantiation bug
* Revert back JniBlockMeta param name
* Delete legacy Cache db files
- Deleted from both the older and the newer legacy locations
- All related db files deleted - rollback files included
- The deletion is run once we try to access the new store blocks on disk root directory
- The deletion check does not throw any exception in case of failure, we just log it in console and try it on the next time
- Related new test added too
* Test refactoring
- Made few changes to improve clarity of provided tests and fixtures
- Prepared few new "failure path" tests
- Enhanced existing tests
* Manual test case
* Simplify error printing
- As we had some commented out code there
* Reset manual tests steps numbering
* [#924] Remove alias from WalletCoordinator
Also make Compose UI the default. The old UI is deprecated but is still used by the benchmarking tests
* Bump benchmark version
* Protect JniBlockMetadata agains minification
* Enable debuggable while benchmarking
* Enhance benchmark screen waiting
* Fix cache db files deletion
- With this construct we delete all blocks blob metadata files, as well as their sqlite file
* Add new benchmark results
* Remove benchmark operations receiver fix
- As it's not needed after the latest profiler dependency update
* Bump gradle/wrapper-validation-action from 1.0.5 to 1.0.6 (#928)
Bumps [gradle/wrapper-validation-action](https://github.com/gradle/wrapper-validation-action) from 1.0.5 to 1.0.6.
- [Release notes](https://github.com/gradle/wrapper-validation-action/releases)
- [Commits](55e685c48d...8d49e559aa)
---
updated-dependencies:
- dependency-name: gradle/wrapper-validation-action
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump actions/cache from 3.2.5 to 3.2.6 in /.github/actions/setup (#929)
Bumps [actions/cache](https://github.com/actions/cache) from 3.2.5 to 3.2.6.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](6998d139dd...69d9d449ac)
---
updated-dependencies:
- dependency-name: actions/cache
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update DatabaseCoordinator.kt
Move if/else out from log message body
* Update Switch cache to store blocks on disk.md
Reset documentation bullets numbering
* Hide internal constants from public package
* Update manual test user with new UI requirement
* Comment out cargo.toml test values
* Inline fixture blocks metadata with its creation
* Switch to FsBlockDb for caching CompactBlocks
* Add RustBackend.getLatestHeight() method
* Raise MSRV to 1.60
* Migrate to latest Rust crate API
* Add RustBackend.findBlockMetadata() method
* Add RustBackend.rewindBlockMetadataToHeight() method
* Update cargo.lock
* Switch to FsBlockDb for caching CompactBlocks
* Migrate to latest Rust crate API
* [#765] implementation of FileCompactBlockRepository
* writing block metadata to database
* split write function into smaller easier to test blocks
* testing for FileCompactBlockRepository
* fixed rewinding
* fixed tests
* fixed FileCompactBlockRepositoryTest and SynchronizerFactoryTest
* code review fixes
* updated proto files
* override all functions in FakeRustBackend
* code review fixes
* Fix function body formatting
* Improve clear function clarity
* Use length of string const
* Delete single file instead of directory
* Improve function clarity
* Refactor outputs counting
- Found a typo in intermediary model class JniBlockMeta parameter change of which does not impact encoding from rust to kotlin according to rust layer implementation.
* Check blocks mkdir result
* Remove unnecessary detekt warning suppression
* Refactor buffer size check
* Improve visibility annotations
* Make file finalise obvious and self documenting
* Remove prevHash logging
* Move instantiation to the object itself
* Enrich fixture with default values
* Extend eror message
* Rename benchmark blocks range fixture
* Fix rebase changes
* Improve FileCompactBlockRepositoryTest
- "De-integrated" the test suite - it now works with fixture blocks
- Created needed fixtures for a clear mocked blocks providing
- Enhanced getting of FileCompactBlockRepository in FileCompactBlockRepositoryTest to clarify that it works with mock components
* Fix ktlintFormat findings
* Fix incorrect dir checking
* More robust min/max checking
* Rewinding to more robust middle value
* Strengthen blocks counts after rewind
* Add check of temp finalized file
* Refactor DatabaseCoordinatorTest
* Rename cache files root directory
- To have the same unified variable name across our repositories - fsBlockDbRoot
* Refactor FileCompactBlockRepositoryTest
- To have these tests more clear
- Fixed FakeRustBackend instantiation bug
* Revert back JniBlockMeta param name
* Delete legacy Cache db files
- Deleted from both the older and the newer legacy locations
- All related db files deleted - rollback files included
- The deletion is run once we try to access the new store blocks on disk root directory
- The deletion check does not throw any exception in case of failure, we just log it in console and try it on the next time
- Related new test added too
* Test refactoring
- Made few changes to improve clarity of provided tests and fixtures
- Prepared few new "failure path" tests
- Enhanced existing tests
* Manual test case
* Simplify error printing
- As we had some commented out code there
* Reset manual tests steps numbering
* Bump benchmark version
* Protect JniBlockMetadata agains minification
* Enable debuggable while benchmarking
* Enhance benchmark screen waiting
* Fix cache db files deletion
- With this construct we delete all blocks blob metadata files, as well as their sqlite file
* Add new benchmark results
* Remove benchmark operations receiver fix
- As it's not needed after the latest profiler dependency update
* Update DatabaseCoordinator.kt
Move if/else out from log message body
* Update Switch cache to store blocks on disk.md
Reset documentation bullets numbering
* Hide internal constants from public package
* Update manual test user with new UI requirement
* Comment out cargo.toml test values
* Inline fixture blocks metadata with its creation
* Update cargo.lock
* Update Cargo.lock
* Check and document JniBlockMeta params ranges
* Add assert for gradle property
* Use UInt internally
uint
* Change array API to list
* Fix for using correct SDK Synchronizer alias
- Only the older Demo-app UI was impacted.
- Thus also our benchmarking was impacted.
* Apply documentation suggestions from code review
Co-authored-by: Kris Nuttycombe <kris@electriccoin.co>
* Final manual test instructions update
* Fixture block hash deterministically generated from block height
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jack Grigg <jack@electriccoin.co>
Co-authored-by: Jack Grigg <jack@z.cash>
Co-authored-by: Honza <rychnovsky.honza@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carter Jernigan <git@carterjernigan.com>
Co-authored-by: Kris Nuttycombe <kris@electriccoin.co>
* [#615] Refactor lightwalletd client
This moves the lightwalletd client code to a separate Gradle module, but does not yet do much clean up the public API (e.g. hiding generated GRPC objects). That will be a followon change via #673
I’ve prototyped a safer version of the API that would be implemented for #673 for two specific calls: obtaining the current block height and obtaining the light wallet info. These were the easiest endpoints to update and they provide a useful exploration for the future changes needed.
* Fix benchmarking for networking module
- Moved to fixture and build type check components to the new networking module, so it's accessible from all needed parts of the projects
- Changed fixture return type to fit to all needed usages
* Align with previous review comment
* Fix wrong merge
* Add basic sanity test
- To satisfy tests requirements on emulators.wtf service
* Remove grpc dependency from sdk-lib module
* Repackage cash.z.wallet.sdk.rpc to cash.z.wallet.sdk.internal.rpc
* Fix BuildConfig path
* Update demo-app paths to rpc
* Fix broken grpc services locations
- Our aim here is to change only the local services location (package name), the server location can't be changed due to backward compatibility.
* Enhance GRPC Response model
* Adopt enhanced GRPC response model
- Adopted in a few endpoints
* Enhance Connection failure type
* Add simple fixture test
* Refactor fetchTransaction() to work with Response
- Refactored this server call to adopt our new Response mechanism
- GrpsStatusResolver.resolveFailureFromStatus() now accepts Throwable input
- Added Response.Failure.toThrowable() to satisfy subsequent server calls results processing
- A few other minor refactoring changes
* Remove commented out log
* Unify return type of collection returning methods
- Switched fetchUtxos() to return Sequence instead of List
- Added a check on empty tAddress input
* fetchUtxos returns flow
- Switched fetchUtxos() to return Flow of Service.GetAddressUtxosReply
- Internally it calls getAddressUtxosStream() instead of getAddressUtxos() from GRPC layer
* Update unsafe suffix documentation
* Address several minor change requests
* Remove code parameter
- Removed from the locally triggered failures with default codes.
* Rename local variable
* Switch from empty response to exception
- Our server calls now rather throw IllegalArgumentException than return an empty response
- Removed commented out log message
- Updated javadocs
* Update proto files
- Plus related api changes adoption
* Unify new clients instances name
* AGP 7.4.0 changes
- packagingOptions -> androidComponents in sdk-lib and lightwallet-client-lib modules
* SDK version bump
* Response resolver tests
* Release build upload timeout increase
* Release build upload timeout increase fix
* Revert timeout
- As Github has some infrastructure troubles and we need to wait
* Add migrations documentation
* Sort packaging filters
* Remove private field from public documentation
* Hide private variables
* Remove package from Android Manifest
* Throw exception instead of custom error
- This step unify our approach to validation on client side across all server calls
* Replace setAddresses index const with number
* Fix indentation in proto file
---------
Co-authored-by: Honza <rychnovsky.honza@gmail.com>
* Add testnet faucet to Compose UI
* Show status on home view
* Add BlockHeight for fixture wallets
* Add balance screen
* Keep screen on while syncing
* Fix conversion of seed phrase to bytes
Previously if `initLogs()` was called more than once, the application
would crash. `NativeLibraryLoader` already implements a singleton for
loading the Rust library, so we now hook into that for initialization.
We now always initialize the Rust logging system, along with the tracing
event hooks and anything else we add in future.
* [#789] Add Benchmark module to Demo-app
* Code cleanup
* Opti-in experimental coroutines api in tests
* Add Testing documentation
* Documentation update + ktlint
* Check screen on and keyguard unlocked in each test
* Introduce UiAutomator extensions
* Enhance BenchmarkTrace events definition
* Remove unnecessary mutex
* Change blocks range
* Increase sync blockchain timeout
- To always fit in the Balances screen timeout for the selected blocks range
* Remove unnecessary fun suspend modifiers
* Macrobenchmark lib bump to 1.2.0-alpha8
* Remove duplicate label attr in app/manifest
* File and link benchmark on CI task
* Add proguard keep rules
* Documentation update
The Android and Rust code have previously managed joint custody of the derived data database. With more complex migrations now required, we need to make the Android side read-only. To achieve that, the Android side will remove Room and instead rely on more primitive SQLite APIs for read-only access.
As part of implementing this change, database management throughout the SDK is being refactored.
There will be multiple representations of the data:
- Database representation (and some Entity representations in the places that Room hasn’t been removed yet). These representations are not as type safe and don’t match Kotlin best practices in all cases.
- Once #615 is implemented there will also be network representations
- Type safe models, which often appear in the public API The Database and Network representations will be converted to and from the type safe model representation.
Honza Rychnovský
Jack Grigg
Jack Grigg
Carter Jernigan
Francisco Gindre
Alex
Jack Grigg