- For now I created `CompactBlockProcessorNG` where I started with
implementation of the state machine. I did it to not break the rest of
the SDK. This change will be merged to the feature branch. And before
it is merged to `main` branch code from `CompactBlockProcessorNG` will
be moved to `CompactBlockProcessor`.
- The new code is not used. It just shows and explains how it is done.
It is proof of concept.
- I did put either commented current code or comment to some places to
explain what should be done there.
- New important data types:
- `ActionContext` is context that can hold any data that needs to be
shared between actions. For example sync ranges or current state.
- `CBPState` is state of the `CompactBlockProcessor`. Each state is
handled by one action. This doesn't apply to terminal states like
`finished` or `failed`.
- `ActionProgress` is very similar to `CompactBlockProgress`.
Different actions reports progress differently and `ActionProgress`
represents this.
- `Action` is protocol that defines API of an action. It has one run
method that executes the code of the action
- CBP first creates actions for (almost) each state in `makeActions()`
method. Then the "magic" is done in `CompactBlockProcessorNG.run()` method.
Here is main loop which takes action for current state and execute it.
It's expected that action does it's work and then updates the context
with new state. And this happens until some terminal state
(`finished`, `failed`, `stopped`) is reached.
- After the transition to state machine API of the
`CompactBlockProcessor` should stay the same. No changes should be
required in `SDKSynchronizer`.
[#700] Add documentation for CompactBlockProcessor state machine
- plantuml tool is used to generate diagram.
[#1054] Add config to state machine CBP
Closes#1054
[#1043] Implement DownloadAction
Closes#1043
[#1049] Implement ValidateAction
Closes#1049
[#1050] Implement ValidateServerAction
Closes#1050
[#1056] Add constructors to state machine CBP
Closes#1056
[#1061] Add failure methods for state machine CBP
Closes#1061
[#1055] Implement retry timer to state machine CBP
Closes#1055
[#1057] Implement start for state machine CBP
Closes#1057
[#1058] Implement stop for state machine CBP
Closes#1058
[#1052] Implement AfterSyncHooksManager when using state machine CBP
Closes#1052
[#1060] Implement wipe for state machine CBP
Closes#1060
[#1059] Implement rewind for state machine CBP
Closes#1059
[#700] Add idle state to CBP state machine
This is required so the CBP can detect start of the sync process.
[#700] Implement sending of some events from CompactBlockProcessorNG
[#700] Implement progress reporting in state machine CBP
[#1045] Implement FetchUTXOsAction
- draft of the fetching
[#1045] Implement FetchUTXOsAction
- updated the way Actions communicate data back to the CBP
- used this mechanism to pass result of utxos fetch so it's passed to the SDKSynchronizer as an Event
[#1042] Implement ComputeSyncRangesAction
Closes#1042
[#700] Implement cache clearing when some actions fail
[#1043] Fix batch range computation in DownloadAction
[#1046] Implement SaplingParamsAction
- action for sapling param files finished
[#1048] Implement ScanDownloadedButUnscannedAction
- scan downloaded but unscanned blocks
[#1047] Implement ScanAction
- scan action with the proper ranges computed
[#1047] Implement ScanAction (#1085)
- fixed logger message
[#1044] Implement EnhanceAction
Closes#1044
[#1041] Implement ClearCacheAction
Closes#1041
[#1040] Implement ClearAlreadyScannedBlocksAction
Closes#1040
[#1039] Implement ChecksBeforeSyncAction
Closes#1039
[#700] Make CBP state machine work
[#1050] Implement ValidateServerAction
- broken tests commented out and tracked in the tickets
- new test for ValidateServerAction
[#1051] Update how progress is computed after switch to state machine
Closes#1051
- new proposal for the progress computation
- OverallProgress value is passed to the Synchronizer as a Float
- OverallProgress is a result of fetch, scan and enhance operations
- Order of actions no longer influences the computation
- Actions report partial updates, CompactBlockProgress actor holds current state and computes the OverallProgress
[#1049] Implement ValidateAction
- synchronizer offline tests updated so it compiles, review is requested in a different ticket
- ValidateAction tests added
- BlockValidator mock generated
[#1047] Implement ScanAction
- ScanAction tests
- refactor of validateAction -> validateServerAction
- generated few more mocks for the DI
[#1045] Implement FetchUTXOsAction
- FetchUTXOsAction tests
- UTXOFetcher mocks
[#1045] Implement FetchUTXOsAction
- enhanced with mocked values and more checks
[#1046] Implement SaplingParamsAction
- SaplingParamsAction tests
- added TODO for TestCoordinator reset()
- SaplingParametersHandler mock added
[#1046] Implement SaplingParamsAction
- SaplingParamsAction tests
- added TODO for TestCoordinator reset()
- SaplingParametersHandler mock added
[#1046] Implement SaplingParamsAction
- rebased so I get functionality of improved mock checks
- enhanced SaplingParamsAction tests
- enhanced ValidateAction tests
- enhanced ScanAction tests
[#1046] Implement SaplingParamsAction
- scanAction tests more checks added
[#1044] Implement EnhanceAction
- EnhanceAction tests focused on 2 different methods:
- decideWhatToDoNext covered separately, decisions where the state machine goes next
- run tests for different cases
- new mocks generated for enhacer
- some typos fixed
[#1044] Implement EnhanceAction (#1107)
- empty assert messages fixed
[#700] Get rid of ScanDownloadedButUnscannedAction
Before the state machine download and scan was called in one loop. And
processing range for one batch was same for both of them. Therefore
there was code which scanned downloaded but not scanned blocks.
But now download and scan are independent. So it is possible to remove
`ScanDownloadedButUnscannedAction`.
[#700] Make NetworkTests compilable
Some tests are disabled for now (list is in #1115). And `NetworkTests`
can be compiled and all the enabled tests work.
[#1043] Implement DownloadAction
- DownloadAction tests
- BlockDownloader mock
[#1043] Implement DownloadAction (#1110)
- support functions set to private
[#1039] Implement ChecksBeforeSyncAction
- ChecksBeforeSyncAction tests
- all support functions in Action tests are set to private
- let _ = -> _ = refactor
- CompactBlockRepository mock added
[#1040] Implement ClearAlreadyScannedBlocksAction
- Implement ClearAlreadyScannedBlocksAction tests
- CompactBlockRepository mock added
[#1041] Implement ClearCacheAction
- ClearCacheAction tests
- CompactBlockRepository mock added
[#1042] Implement ComputeSyncRangesAction
- ComputeSyncRangesAction tests
- fixed all tests after merge of latest SDK changes related InternalSyncProgress
- all actions marked as final class
[#1042] Implement ComputeSyncRangesAction (#1120)
- Custom LatestBlocksDataProviderMock removed from the project
[#1122] Implement FileManager protocol and dependency
- ZcashFileManager implemented
- MigrateLegacyCacheDBAction refactored to be dependent on ZcashFileManager
- ZcashFileManager mock added
[#1122] Implement FileManager protocol and dependency (#1124)
- code cleanup
[#1121] Implement MigrateLegacyCacheDBAction
- MigrateLegacyCacheDBAction tests WIP
- tests naming cleanup
[#1121] Implement MigrateLegacyCacheDBAction
- MigrateLegacyCacheDBAction tests finished
[#700] Fix DarksideTests
Closes#1102
Some tests that can't be compiled are disabled for now. List is in #1126.
This PR contains multiple fixes. Most of the fixes is done in the code.
Not in the tests. That is good news.
Fixes:
- `config` inside `CompactBlockProcessor` can be updated during the
tests. And it must be also updated inside the actions. So
`ConfigProvider` is added and it is wrapper for config that is passed
to any instance of `Action` and provides updated config.
- Fixed `EnhanceAction`. Now it should update all the blocks in the
enhance range even when the remaining count of blocks is lower than
1000.
- Fixed `fail()` and `validationFailed()`. These two were canceling
`syncTask`. But that stopped run loop in a bad way.
[#1129] Final check of all State Machine Action tests
- XTCAsset messages checked
- test naming checked and fixed
[#1126] Fix DarksideTests in state machine branch
Closes#1126
Fix offline tests
Closes#1098Closes#1095Closes#1094
Most of the tests is removed. Either the code that was tested doesn't
exists. Or now tests for state machine actions do this work.
[#1115] Fix NetworkTests in state machine branch
Closes#1115
[#700] Fix progress reporting
Some actions in the sync process may not run. For example there are no
transactions to enhance and therefore there is no enhance progress. And in
cases like this computation of final progress won't work properly.
So let's fake 100% progress at the end of the sync process.
Closes#1012
- This is the first change for the dependency injection. This change
adds `DIContainer` which is used to register and resolve dependencies.
- Then this change shows how `DIContainer` can be used to resolve
dependencies. And it also shows how to mock dependencies in tests.
- Constructors of `Initializer` are changed to support mocking of
depedendencies. But it doesn't affect public API in any way.
- Depedencies are registered in `Dependencies.setup()`. Then in the code
`container.resolve(SomeType.self)` is called to get instance of the
dependency.
- `ZcashTestCase` class is added. It inherits from `XCTestCase` and
should be used for base class in tests. `ZcashTestCase` takes care of
creation of `DIContainer` for dependencies mocking. In future we can
maybe move there more things that are used in each test.
- Lot is missing here. Not all the code is using dependency injection.
Tests for `DIContainer` are missing. Only `OfflineTests` work now
(other tests can't be even compiled). It will be added in future
changes.
Removes `PendingTransactionEntity` and all of its related components.
Pending items are still tracked and visualized by the existing APIs
but they are retrieved from the `TransactionRepository` instead by
returning `ZcashTransaction.Overview` instead.
`pendingDbURL` is removed from every place it was required. Its
deletion is responsibility of wallet developers.
`ClearedTransactions` are now just `transactions`.
`MigrationManager` is deleted. Now all migrations are in charge of
the rust welding layer.
`PendingTransactionDao.swift` is removed.
Implementation of `AccountEntity` called `Account` is now `DbAccount`
`ZcashTransaction.Overview` can be checked for "pending-ness" by calling
`.isPending(latestHeight:)` latest height must be provided so that minedHeight
can be compared with the lastest and the `defaultStaleTolerance` constant.
`TransactionRecipient` is now a public type.
protocol `PendingTransactionRepository` is removed.
`TransactionManagerError` and `PersistentTransactionManager` are deleted.
`OutboundTransactionManager` is deleted and replaced by `TransactionEncoder`
which now incorporates `submit(encoded:)` functionality
`WalletTransactionEncoder` now uses a `LightWalletService` to submit the
encoded transactions.
Add changelog changes
Delete references to PendingDb from tests and documentation.
Fixes some typos. Adds the ability to trace transaction repository
SQL queries from test
Fix rebase conflicts and generate code
[#837] Memo tests regarding transparent address
Closes#837
Add model for transaction output
Point to FFI branch
Fix issue where sync wouldn't resume after wipe. Becasue GRPC
channel would be closed
Fix Tests
Fix testPendingTransactionMinedHeightUpdated
Fix testLastStates
[#921] Fix broken SynchronizerDarksideTests
Add ZcashTransaction.Output API to Synchronizer
Changelog + comment fix
Add Assertions for transaction outputs and recipients
Point to FFI 0.3.1
Fix Demo App Compiler errors
Fix Demo App Compiler errors
fix cacheDb warnings
Fix Tests and compiler errors of rebase
build demo app
Remove `ZcashTransaction.Sent` and `.Received`. Add `.State` and tests
Fix SPM warning
PR Suggestions
Removes errors that are not used anymore
fix warnings
- `BlockDownloaderImpl` is changed. It now spawns detached Task to
download blocks. So blocks can be downloaded in parallel with scanning
process.
- `Synchronizer.stop()` is no longer async.
- Blocks cache cleaning is changed. It's not possible to drop the whole
cache now. There are some blocks pre-downloaded which weren't scanned
yet.
This change introduces two new protocols: `ClosureSynchronizer` and
`CombineSynchronizer`. These two protocols define API that doesn't use
`async`. So the client can choose exactly which API it wants to use.
This change also introduces two new objects: `ClosureSDKSynchronizer`
and `CombineSDKSynchronizer`. These two implement the respective protocols
mentioned above. Both are structures. Neither of these two keeps any state.
Thanks to this each is very cheap to create. And usage of these two isn't
mutually exclusive. So devs can really choose the best SDK API for each
part of the client app.
[#831] Use async inside of the SDKSynchronizer
- In general lot of methods inside the `SDKSynchronizer` and
`CompactBlockProcessoer` which weren't async are now async. And other
changes are made because of this change.
- `CompactBlockProcessor` no longer uses Combine to communicate with
`SDKSynchronizer`. Reason for this is that Combine doesn't play great
with async. Closure passed to `sink` isn't async.
- Because of this and because of how our tests work (receiving signals
from CBP directly) `CompactBlockProcessor` must be able to handle more
event closures. Not just one. So it now has `eventClosures`
dictionary. It's little bit strange but it works fine.
- `SyncStatus` inside the `SDKSynchronizer` was previously protected by
lock. Now it's protected by simple actor wrapper.
- Changes in tests are minimal. Changes were mady only because
`CompactBlockProcessor` changes from Combine to closures.
[#831] Add tests for ClosureSDKSynchronizer
- Added tests are testing in general if the `ClosuresSDKSynchronizer` is correctly
calling `Synchronizer` and if the values are correctly returned.
- `ClosuresSDKSynchronizer` doesn't contain any logic but it is public
API and we should be sure that it works correctly.
[#831] Add tests for CombineSDKSynchronizer
[#831] Add changelog
Closes#826
- `viewingKeys` and `walletBirthday` are removed from `Initializer`
constuctor. These parameters are moved to `SDKSynchronizer.prepare`
function.
- Constructor of the `SDKSynchronizer` no longer throws exception.
- Any value emitted from `lastState` stream before `SDKSynchronizer.prepare`
is called has `latestScannedHeight` set to 0.
- `Initializer.initialize` function isn't public anymore. To initialize
SDK call `SDKSynchronizer.prepare` instead.
- Real birthday is now known after the `SDKSynchronizer.prepare` is
called. But this function isn't async and therefore it's hard to
pass birthday from here to `CompactBlockProcessor` in synchronous
manner. We must be sure that it's safe to call start() after prepare
function finishes.
For this reason `CompactBlockProcessor` has `walletBirthdayProvider`
instead of `walletBirthday`. And this new parameter is closure which
reads birthday from `Initiliazer` on each call. Thanks to this
`CompactBlockProcessor` has always the right birthday.
- Some cleanup is done in `TestCoordinator`.
Closes#778
- .swiftlint.yml is now used to lint only the code of the SDK.
- .swiftlint_tests.yml is now used to lint only the code of the tests.
This config disables rules that were previously disabled in the code
in lot of tests.
Closes#764
- All notification that were previously sent from CompactBlockProcessor are now gone.
- `CompactBlockProcessor` now provides `eventStream` to communicate with `SDKSynchronizer`.
- Added `synchronizerStoredUTXOs` notification.
- Tests are updated to not use notifications anymore. Some tests there
weren't working before are now fixed.
- Added `CompactBlockProcessorEventHandler` utility class. Previously
expectations were subscribing to notifications. This class replaces
this functionality. It subscribes to events from
`CompactBlockProcessor` and fullfill expectations.
Closes#746
- `LightWalletGRPCService` is no longer public. `LightWalletService` is no longer public.
- `LightWalletGRPCService` shouldn't be used dicrectly. Use `LightWalletServiceFactory` to create instance of the service.
- Moved sending of `blockProcessorConnectivityStateChanged` notification to `Initilizer`.
- All the errors from GRPC are mapped to `LightWalletServiceError`. Handling of `GRPCStatus` in the SDK is no longer required.
- `Service` directory (that contains GRPC code) is moved to `Modules/Service`. We can put more code to `Modules/` if we decide
to modularize more code.
Closes https://github.com/zcash/ZcashLightClientKit/issues/697
Closes https://github.com/zcash/ZcashLightClientKit/issues/720
Closes https://github.com/zcash/ZcashLightClientKit/issues/587
Closes https://github.com/zcash/ZcashLightClientKit/issues/667
Closes https://github.com/zcash/ZcashLightClientKit/issues/443
Closes https://github.com/zcash/ZcashLightClientKit/issues/754
- [#790] Fix ShieldFundsTests
Closes#790
Removes comments on `ShieldFundsTests` since those issues have been fixed
Depends on zcash-light-client-ffi changes that adopt newer versions of
librustzcash crates `zcash_primitives 0.10`, `zcash_client_backend 0.7`,
`zcash_proofs 0.10`, `zcash_client_sqlite 0.5.0`.
Also allows wallets to define a shielding_threshold and will set
foundations to customize minimum confirmations for balances, spends
and shielding operations.
**Test Bootstrapping**
- `ZcashCompactBlockDescriptor`: struct that holds functions to
describe blocks as filenames and compare those filenames
`ZcashCompactBlockDescriptor.live` has the actual implementation
but it can be replaced by mocks if needed on Tests
main implementations are held under `FSCompactBlockRepository.filenameDescription` and `FSCompactBlockRepository.filenameComparison` on a separate extention
`DirectoryListingProviders` provide two default implementations of listing a
directory deterministically. `FileManager` does not define a sorting and needs to be done in-memory by calling `.sorted()` on the resulting collection. If this
is a big toll on performance it can be changed to a POSIX implementation but
this is good for now.
`ZcashCompactBlockDescriptor` adds a `height` helper function to
turn a filename into the height of the block stored.
Implemented `func latestHeight() throws -> BlockHeight ` that
returns the blockheight by querying the cache directory in a sorted
fashion and getting the last value and turning the filename into a
`BlockHeight`
Added `Meta` struct to ZcashCompactBlock.
Tests implemented:
- `filterBlockFiles`
- `testClearTheCache`
- `testLatestHeightEmptyCacheThrows`
- `testLatestHeightEmptyCacheThrowsAsync`
- `testRewindEmptyCacheDoesNothing`
- `testRewindEmptyCacheDoesNothingAsync`
- `testWhenBlockIsStoredItFollowsTheDescribedFormat`
- `testWhenBlockIsStoredItFollowsTheFilenameConvention`
- `testGetLatestHeight`
- `testRewindDeletesTheRightBlocks` test
- `testPerformanceExample` test. This isn't a real performance test because the API doesn't work with async/await yet
adopts `shield_funds` shielding threshold parameter
Implements `initBlockMetadataDb` and fix tests
Renames dbCache parameter to `fsBlockDbRoot`. Builds but tests don't pass.
Removes cacheDb uses from code. Testing utilities still persist.
Added needed information in MIGRATING and CHANGELOG.
Added helper to perform deletion of legacy db and creation a the
new file system backed cache.
Renames parameters and changes code where needed.
Network Constants turned into `enum` with static methods.
DeletelastDownloadedBlock helper from initializer
Removes CompactBlockStorage and CompactBlockEntity.
Implements `latestCachedBlockHeight` on rustbackend.
*Replaces dependencies on ZcashRustWelding with `FSMetadataStore`*
This allows the tests to not depend in a particular implementation
of either the MockRustBackend of or ZcashRustBackend. Also provides
a way to test errors properly and switch implementations of critical
areas like `writeBlocks`.
Closes#740
`CompactBlockDownloading` protocol is renamed to `BlockDownloaderService`. `CompactBlockDownloader` is renamed to `BlockDownloaderServiceImpl`.
What is the motivation for this rename?
Goal of this change is to extract code from `CompactBlockProcessor` which is related to downloading. So naturaly it should be called "downloader".
But `CompactBlockDownloader` was already taken. After analysis of `CompactBlockDownloading` and `CompactBlockDownloader` I found out that
`CompactBlockDownloader` doesn't download blocks exactly. It's more like abstraction over `LightWalletService`. And it does very similar job to
`LightWalletService`. That is why I decided to rename it to "service".
What is the motivation for protoco/impl naming scheme?
First `BlockDownloaderService` protocol doesn't describe capability in my eyes. It describe what something is. Also `BlockDownloaderServicing` sounds
strange. I agree that add suffix "Impl" to class name isn't super nice. But it is acceptable. And it makes sense to use nicer type name
(e.g. `BlockDownloaderService`) for protocol because it's used on much more places in the code.
- Added `BlockDownloader` protocol and `BlockDownloaderImpl` class. The code related to downloading was exctracted to this class.
- Previously we had one range for each sync which was used for each
phase of sync process. Newly there is separate range for each phase of
the sync process.
- I added `InternalSyncProgress` utility. This utility tracks progress
of some phases. And it is able to compute right ranges which should be
used for syncing.
- Some unused download code from `CompactBlockProcessor` is removed.
Fix tests
Address review comments
- Rebase to master
- Update how range for `processBatchFinished()` is computed.
- Refactor `InternalSyncProgress`
- Add tests for `InternalSyncProgress`
Address review comments
Change how latest downloaded block is tracked
- Cache DB is no longer used to track which block was downloaded as
latest. `InternalSyncProgress` is used to do that.
- Thanks to that #660 is fixed. And cache DB can be completely removed
after sync process.
- Added sleep(1) to some darkside tests when latest block height is set.
Otherwise lightwalletd in darkside mode doesn't have time to setup and
tests are flaky.
Fix TransactionEnhancementTests.testBasicEnhancement test
`XCAsyncTestCase` does not work as intended but `wait {}` does what
we need.
restored `needsToStartScanningWhenStopped` as a Bool var on actor
fixed Enhancement test
This commit makes Renames `WalletBirthday` to `Checkpoint` and
makes Checkpoint an internal type. Public ocurrences of this
type is replaced by `BlockHeight` (Int) and then retrieval of the
Checkpoint is deferred to the appropiate place in the code
Add an extension method to `BlockHeight` to get latest checkpoint
height present in the SDK's bundle
PR Suggestions:
Add test for integer overflow on JSON
File renamed from WalletBirthday to Checkpoint
Michal Fousek
Francisco Gindre
Lukas Korba
Francisco Gindre
Daniel Haight