wormhole-explorer/blockchain-watcher/README.md

132 lines
4.6 KiB
Markdown
Raw Permalink Normal View History

[Blockchain Watcher] Emit LogMessagePublished for Solana (#809) * adding gen-relayer watcher code to branch * folder rename * adding some new abstract types for processor redesign * added first cut of new process manager logic * large refactoring to environment configuration * Add basic default config * prettier run * modifying event handler interface * abstract handler typing changes * Blockchain watcher: adding domain and infra layers (#786) * reorg domain-infra * watch evm blocks action * adding evm log parser * wider prettier * renaming watch action * adding doc * persist latest metadata * gh action for blockchain-watcher * adding log-message-published mapper * deps: remove peers and nodemon * adding handler for LogMessagePublished * added parser for log message published --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> * Blockchain Watcher: ethereum -> sns implementation (#790) * evm block repo implementation * adding sns publisher * adding external config and initial runner * fix: start from latest if no fromBlock configured * feat: add dryRun option * fix: handler filtering and eth_getLogs filter payload * local e2e * actual sns arns for testnet * smaller docker image * deployment changes * [Blockchain Watcher] Apply missing formatting for log message published (#791) * Apply missing formatting for log message published * keep extracting * emitterAddress -> emitter * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * [Blockchain watcher] Adding light observability (#793) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * better error logs * wait when no block available * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * group actions per platform * [Blockchain Watcher] Add StartJob action + rate limit handling (#799) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * adding poll solana txs action * adding solana repo implementation * solana: mapping to logmsgpublished * fixing solana mapper test * updating readme * adding some metrics * adding solana worker * adding finality param to get solana block * minor changes - using confirmed for solana log extraction * updating version * add solana env var * [Blockchain Watcher] Feature-813/fix-coverage-report (#815) * Fix coverage report * Run prettier * Rollback global coverage value * Set global coverage value to 55 * Add github PR template * Run prettier * Rename github template file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * private solana urls as secret * adding prod cfg files * Change repository name and cretae error foulder (#819) * Change repository name and cretae error foulder * Rename SolanaFailure file * Rename SolanaFailure file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * passing node_options for solana * adding missing envs templates * cru * ratelimited solana client + increase coverage * [Blockchain Watcher] Feature 817/add hexagonal prefix (#824) * Put hexadecimal prefix * Create id variable * Add error log * Add error log * Remove bigint in getBlock params --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * fix never ending solana jobs * adding job counter metric * solana: add retry handling for 429s * cleanup * remove duplicated tests * cru + item count * minor readme update + do not use private solana rpc * fix items counter * fix solana job programID for devnet * log basic repo info * solana: passing finality everywhere * delete unnecesary files * adjusting solana cfgs * [Blockchain Watcher] Fix/re order some files (#828) * fix/re-order-some-files * run prettier * Run prettier * Resolve comment in PR * Remove import * Resolve comment * Remove prefix in sns log --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> Co-authored-by: Julian <52217955+julianmerlo95@users.noreply.github.com> Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local>
2023-11-30 07:05:43 -08:00
# Explorer Blockchain Watcher
The purpose of this process is to watch all Wormhole connected blockchains for events in Wormhole ecosystem contracts, and then produce database records for these events, and other important associated data.
## Installation
run npm ci in the root of the project
run npm dev in the root of the project
## Deployment
This process is meant to be deployed as a docker container. The dockerfile is located in the root of the project.
[Blockchain Watcher] Emit LogMessagePublished for Solana (#809) * adding gen-relayer watcher code to branch * folder rename * adding some new abstract types for processor redesign * added first cut of new process manager logic * large refactoring to environment configuration * Add basic default config * prettier run * modifying event handler interface * abstract handler typing changes * Blockchain watcher: adding domain and infra layers (#786) * reorg domain-infra * watch evm blocks action * adding evm log parser * wider prettier * renaming watch action * adding doc * persist latest metadata * gh action for blockchain-watcher * adding log-message-published mapper * deps: remove peers and nodemon * adding handler for LogMessagePublished * added parser for log message published --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> * Blockchain Watcher: ethereum -> sns implementation (#790) * evm block repo implementation * adding sns publisher * adding external config and initial runner * fix: start from latest if no fromBlock configured * feat: add dryRun option * fix: handler filtering and eth_getLogs filter payload * local e2e * actual sns arns for testnet * smaller docker image * deployment changes * [Blockchain Watcher] Apply missing formatting for log message published (#791) * Apply missing formatting for log message published * keep extracting * emitterAddress -> emitter * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * [Blockchain watcher] Adding light observability (#793) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * better error logs * wait when no block available * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * group actions per platform * [Blockchain Watcher] Add StartJob action + rate limit handling (#799) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * adding poll solana txs action * adding solana repo implementation * solana: mapping to logmsgpublished * fixing solana mapper test * updating readme * adding some metrics * adding solana worker * adding finality param to get solana block * minor changes - using confirmed for solana log extraction * updating version * add solana env var * [Blockchain Watcher] Feature-813/fix-coverage-report (#815) * Fix coverage report * Run prettier * Rollback global coverage value * Set global coverage value to 55 * Add github PR template * Run prettier * Rename github template file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * private solana urls as secret * adding prod cfg files * Change repository name and cretae error foulder (#819) * Change repository name and cretae error foulder * Rename SolanaFailure file * Rename SolanaFailure file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * passing node_options for solana * adding missing envs templates * cru * ratelimited solana client + increase coverage * [Blockchain Watcher] Feature 817/add hexagonal prefix (#824) * Put hexadecimal prefix * Create id variable * Add error log * Add error log * Remove bigint in getBlock params --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * fix never ending solana jobs * adding job counter metric * solana: add retry handling for 429s * cleanup * remove duplicated tests * cru + item count * minor readme update + do not use private solana rpc * fix items counter * fix solana job programID for devnet * log basic repo info * solana: passing finality everywhere * delete unnecesary files * adjusting solana cfgs * [Blockchain Watcher] Fix/re order some files (#828) * fix/re-order-some-files * run prettier * Run prettier * Resolve comment in PR * Remove import * Resolve comment * Remove prefix in sns log --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> Co-authored-by: Julian <52217955+julianmerlo95@users.noreply.github.com> Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local>
2023-11-30 07:05:43 -08:00
As of today, we have a one to one relationship from job to pod. So we statically define jobs as a config map that each pod then loads.
See `../deploy/blockchain-watcher/workers` for an example of how deployment works.
To deploy all workers for a given environment, run:
```bash
$ kubetpl render workers/* -i env/staging-testnet.env --syntax=go-template | kubectl apply -f -
```
## Configuration
Configuration is loaded from files in `config` directory.
There is a default file, and then a file for each environment. The environment is set by the NODE_ENV environment variable.
If NODE_ENV is not set, the default file is used.
2023-12-06 06:02:26 -08:00
Some values may be overridden by using environment variables. See `config/custom-environment-variables.json` for a list of these variables.
```bash
$ NODE_ENV=staging LOG_LEVEL=debug npm run dev
```
[Blockchain Watcher] Emit LogMessagePublished for Solana (#809) * adding gen-relayer watcher code to branch * folder rename * adding some new abstract types for processor redesign * added first cut of new process manager logic * large refactoring to environment configuration * Add basic default config * prettier run * modifying event handler interface * abstract handler typing changes * Blockchain watcher: adding domain and infra layers (#786) * reorg domain-infra * watch evm blocks action * adding evm log parser * wider prettier * renaming watch action * adding doc * persist latest metadata * gh action for blockchain-watcher * adding log-message-published mapper * deps: remove peers and nodemon * adding handler for LogMessagePublished * added parser for log message published --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> * Blockchain Watcher: ethereum -> sns implementation (#790) * evm block repo implementation * adding sns publisher * adding external config and initial runner * fix: start from latest if no fromBlock configured * feat: add dryRun option * fix: handler filtering and eth_getLogs filter payload * local e2e * actual sns arns for testnet * smaller docker image * deployment changes * [Blockchain Watcher] Apply missing formatting for log message published (#791) * Apply missing formatting for log message published * keep extracting * emitterAddress -> emitter * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * [Blockchain watcher] Adding light observability (#793) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * better error logs * wait when no block available * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * group actions per platform * [Blockchain Watcher] Add StartJob action + rate limit handling (#799) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * adding poll solana txs action * adding solana repo implementation * solana: mapping to logmsgpublished * fixing solana mapper test * updating readme * adding some metrics * adding solana worker * adding finality param to get solana block * minor changes - using confirmed for solana log extraction * updating version * add solana env var * [Blockchain Watcher] Feature-813/fix-coverage-report (#815) * Fix coverage report * Run prettier * Rollback global coverage value * Set global coverage value to 55 * Add github PR template * Run prettier * Rename github template file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * private solana urls as secret * adding prod cfg files * Change repository name and cretae error foulder (#819) * Change repository name and cretae error foulder * Rename SolanaFailure file * Rename SolanaFailure file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * passing node_options for solana * adding missing envs templates * cru * ratelimited solana client + increase coverage * [Blockchain Watcher] Feature 817/add hexagonal prefix (#824) * Put hexadecimal prefix * Create id variable * Add error log * Add error log * Remove bigint in getBlock params --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * fix never ending solana jobs * adding job counter metric * solana: add retry handling for 429s * cleanup * remove duplicated tests * cru + item count * minor readme update + do not use private solana rpc * fix items counter * fix solana job programID for devnet * log basic repo info * solana: passing finality everywhere * delete unnecesary files * adjusting solana cfgs * [Blockchain Watcher] Fix/re order some files (#828) * fix/re-order-some-files * run prettier * Run prettier * Resolve comment in PR * Remove import * Resolve comment * Remove prefix in sns log --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> Co-authored-by: Julian <52217955+julianmerlo95@users.noreply.github.com> Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local>
2023-11-30 07:05:43 -08:00
When running locally, you need to configure one or more jobs.
By default, jobs are read from `metadata-repo/jobs/jobs.json`.
Example:
```
{
"jobs": [
{
"id": "poll-log-message-published-ethereum",
"chain": "ethereum",
"source": {
"action": "PollEvm",
"config": {
"fromBlock": "10012499",
"blockBatchSize": 100,
"commitment": "latest",
"interval": 15000,
"addresses": ["0x706abc4E45D419950511e474C7B9Ed348A4a716c"],
"chain": "ethereum",
"topics": []
}
},
"handlers": [
{
"action": "HandleEvmLogs",
"target": "sns",
"mapper": "evmLogMessagePublishedMapper",
"config": {
"abi": "event LogMessagePublished(address indexed sender, uint64 sequence, uint32 nonce, bytes payload, uint8 consistencyLevel)",
"filter": {
"addresses": ["0x706abc4E45D419950511e474C7B9Ed348A4a716c"],
"topics": ["0x6eb224fb001ed210e379b335e35efe88672a8ce935d981a6896b27ffdf52a3b2"]
}
}
},
{
"action": "HandleEvmLogs",
"target": "sns",
"mapper": "evmTransactionFoundMapper",
"config": {
"abi": "event TransferRedeemed(uint16 indexed emitterChainId, bytes32 indexed emitterAddress, uint64 indexed sequence)",
"filter": {
"addresses": ["0x3ee18b2214aff97000d974cf647e7c347e8fa585"],
"topics": ["0xcaf280c8cfeba144da67230d9b009c8f868a75bac9a528fa0474be1ba317c169"]
}
}
},
{
"action": "HandleEvmLogs",
"target": "sns",
"mapper": "evmTransactionFoundMapper",
"config": {
"abi": "event Delivery(address indexed recipientContract, uint16 indexed sourceChain, uint64 indexed sequence, bytes32 deliveryVaaHash, uint8 status, uint256 gasUsed, uint8 refundStatus, bytes additionalStatusInfo, bytes overridesInfo)",
"filter": {
"addresses": ["0x27428dd2d3dd32a4d7f7c497eaaa23130d894911"],
"topics": ["0xbccc00b713f54173962e7de6098f643d8ebf53d488d71f4b2a5171496d038f9e"]
}
}
}
]
[Blockchain Watcher] Emit LogMessagePublished for Solana (#809) * adding gen-relayer watcher code to branch * folder rename * adding some new abstract types for processor redesign * added first cut of new process manager logic * large refactoring to environment configuration * Add basic default config * prettier run * modifying event handler interface * abstract handler typing changes * Blockchain watcher: adding domain and infra layers (#786) * reorg domain-infra * watch evm blocks action * adding evm log parser * wider prettier * renaming watch action * adding doc * persist latest metadata * gh action for blockchain-watcher * adding log-message-published mapper * deps: remove peers and nodemon * adding handler for LogMessagePublished * added parser for log message published --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> * Blockchain Watcher: ethereum -> sns implementation (#790) * evm block repo implementation * adding sns publisher * adding external config and initial runner * fix: start from latest if no fromBlock configured * feat: add dryRun option * fix: handler filtering and eth_getLogs filter payload * local e2e * actual sns arns for testnet * smaller docker image * deployment changes * [Blockchain Watcher] Apply missing formatting for log message published (#791) * Apply missing formatting for log message published * keep extracting * emitterAddress -> emitter * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * [Blockchain watcher] Adding light observability (#793) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * better error logs * wait when no block available * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * group actions per platform * [Blockchain Watcher] Add StartJob action + rate limit handling (#799) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * adding poll solana txs action * adding solana repo implementation * solana: mapping to logmsgpublished * fixing solana mapper test * updating readme * adding some metrics * adding solana worker * adding finality param to get solana block * minor changes - using confirmed for solana log extraction * updating version * add solana env var * [Blockchain Watcher] Feature-813/fix-coverage-report (#815) * Fix coverage report * Run prettier * Rollback global coverage value * Set global coverage value to 55 * Add github PR template * Run prettier * Rename github template file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * private solana urls as secret * adding prod cfg files * Change repository name and cretae error foulder (#819) * Change repository name and cretae error foulder * Rename SolanaFailure file * Rename SolanaFailure file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * passing node_options for solana * adding missing envs templates * cru * ratelimited solana client + increase coverage * [Blockchain Watcher] Feature 817/add hexagonal prefix (#824) * Put hexadecimal prefix * Create id variable * Add error log * Add error log * Remove bigint in getBlock params --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * fix never ending solana jobs * adding job counter metric * solana: add retry handling for 429s * cleanup * remove duplicated tests * cru + item count * minor readme update + do not use private solana rpc * fix items counter * fix solana job programID for devnet * log basic repo info * solana: passing finality everywhere * delete unnecesary files * adjusting solana cfgs * [Blockchain Watcher] Fix/re order some files (#828) * fix/re-order-some-files * run prettier * Run prettier * Resolve comment in PR * Remove import * Resolve comment * Remove prefix in sns log --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> Co-authored-by: Julian <52217955+julianmerlo95@users.noreply.github.com> Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local>
2023-11-30 07:05:43 -08:00
},
{
"id": "poll-transfer-redeemed-solana",
"chain": "solana",
"source": {
"action": "PollSolanaTransactions",
[Blockchain Watcher] Emit LogMessagePublished for Solana (#809) * adding gen-relayer watcher code to branch * folder rename * adding some new abstract types for processor redesign * added first cut of new process manager logic * large refactoring to environment configuration * Add basic default config * prettier run * modifying event handler interface * abstract handler typing changes * Blockchain watcher: adding domain and infra layers (#786) * reorg domain-infra * watch evm blocks action * adding evm log parser * wider prettier * renaming watch action * adding doc * persist latest metadata * gh action for blockchain-watcher * adding log-message-published mapper * deps: remove peers and nodemon * adding handler for LogMessagePublished * added parser for log message published --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> * Blockchain Watcher: ethereum -> sns implementation (#790) * evm block repo implementation * adding sns publisher * adding external config and initial runner * fix: start from latest if no fromBlock configured * feat: add dryRun option * fix: handler filtering and eth_getLogs filter payload * local e2e * actual sns arns for testnet * smaller docker image * deployment changes * [Blockchain Watcher] Apply missing formatting for log message published (#791) * Apply missing formatting for log message published * keep extracting * emitterAddress -> emitter * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * [Blockchain watcher] Adding light observability (#793) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * better error logs * wait when no block available * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * group actions per platform * [Blockchain Watcher] Add StartJob action + rate limit handling (#799) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * adding poll solana txs action * adding solana repo implementation * solana: mapping to logmsgpublished * fixing solana mapper test * updating readme * adding some metrics * adding solana worker * adding finality param to get solana block * minor changes - using confirmed for solana log extraction * updating version * add solana env var * [Blockchain Watcher] Feature-813/fix-coverage-report (#815) * Fix coverage report * Run prettier * Rollback global coverage value * Set global coverage value to 55 * Add github PR template * Run prettier * Rename github template file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * private solana urls as secret * adding prod cfg files * Change repository name and cretae error foulder (#819) * Change repository name and cretae error foulder * Rename SolanaFailure file * Rename SolanaFailure file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * passing node_options for solana * adding missing envs templates * cru * ratelimited solana client + increase coverage * [Blockchain Watcher] Feature 817/add hexagonal prefix (#824) * Put hexadecimal prefix * Create id variable * Add error log * Add error log * Remove bigint in getBlock params --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * fix never ending solana jobs * adding job counter metric * solana: add retry handling for 429s * cleanup * remove duplicated tests * cru + item count * minor readme update + do not use private solana rpc * fix items counter * fix solana job programID for devnet * log basic repo info * solana: passing finality everywhere * delete unnecesary files * adjusting solana cfgs * [Blockchain Watcher] Fix/re order some files (#828) * fix/re-order-some-files * run prettier * Run prettier * Resolve comment in PR * Remove import * Resolve comment * Remove prefix in sns log --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> Co-authored-by: Julian <52217955+julianmerlo95@users.noreply.github.com> Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local>
2023-11-30 07:05:43 -08:00
"config": {
"slotBatchSize": 1000,
"commitment": "finalized",
"interval": 1500,
"signaturesLimit": 200,
"programId": "wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb",
"chain": "solana"
}
},
"handlers": [
{
"action": "HandleSolanaTransactions",
"target": "sns",
2024-01-06 15:35:07 -08:00
"mapper": "solanaTransactionFoundMapper",
"config": {
"programId": "wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb"
[Blockchain Watcher] Emit LogMessagePublished for Solana (#809) * adding gen-relayer watcher code to branch * folder rename * adding some new abstract types for processor redesign * added first cut of new process manager logic * large refactoring to environment configuration * Add basic default config * prettier run * modifying event handler interface * abstract handler typing changes * Blockchain watcher: adding domain and infra layers (#786) * reorg domain-infra * watch evm blocks action * adding evm log parser * wider prettier * renaming watch action * adding doc * persist latest metadata * gh action for blockchain-watcher * adding log-message-published mapper * deps: remove peers and nodemon * adding handler for LogMessagePublished * added parser for log message published --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> * Blockchain Watcher: ethereum -> sns implementation (#790) * evm block repo implementation * adding sns publisher * adding external config and initial runner * fix: start from latest if no fromBlock configured * feat: add dryRun option * fix: handler filtering and eth_getLogs filter payload * local e2e * actual sns arns for testnet * smaller docker image * deployment changes * [Blockchain Watcher] Apply missing formatting for log message published (#791) * Apply missing formatting for log message published * keep extracting * emitterAddress -> emitter * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * [Blockchain watcher] Adding light observability (#793) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * better error logs * wait when no block available * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * group actions per platform * [Blockchain Watcher] Add StartJob action + rate limit handling (#799) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * adding poll solana txs action * adding solana repo implementation * solana: mapping to logmsgpublished * fixing solana mapper test * updating readme * adding some metrics * adding solana worker * adding finality param to get solana block * minor changes - using confirmed for solana log extraction * updating version * add solana env var * [Blockchain Watcher] Feature-813/fix-coverage-report (#815) * Fix coverage report * Run prettier * Rollback global coverage value * Set global coverage value to 55 * Add github PR template * Run prettier * Rename github template file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * private solana urls as secret * adding prod cfg files * Change repository name and cretae error foulder (#819) * Change repository name and cretae error foulder * Rename SolanaFailure file * Rename SolanaFailure file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * passing node_options for solana * adding missing envs templates * cru * ratelimited solana client + increase coverage * [Blockchain Watcher] Feature 817/add hexagonal prefix (#824) * Put hexadecimal prefix * Create id variable * Add error log * Add error log * Remove bigint in getBlock params --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * fix never ending solana jobs * adding job counter metric * solana: add retry handling for 429s * cleanup * remove duplicated tests * cru + item count * minor readme update + do not use private solana rpc * fix items counter * fix solana job programID for devnet * log basic repo info * solana: passing finality everywhere * delete unnecesary files * adjusting solana cfgs * [Blockchain Watcher] Fix/re order some files (#828) * fix/re-order-some-files * run prettier * Run prettier * Resolve comment in PR * Remove import * Resolve comment * Remove prefix in sns log --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> Co-authored-by: Julian <52217955+julianmerlo95@users.noreply.github.com> Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local>
2023-11-30 07:05:43 -08:00
}
}
]
}
]
}
[Blockchain Watcher] Emit LogMessagePublished for Solana (#809) * adding gen-relayer watcher code to branch * folder rename * adding some new abstract types for processor redesign * added first cut of new process manager logic * large refactoring to environment configuration * Add basic default config * prettier run * modifying event handler interface * abstract handler typing changes * Blockchain watcher: adding domain and infra layers (#786) * reorg domain-infra * watch evm blocks action * adding evm log parser * wider prettier * renaming watch action * adding doc * persist latest metadata * gh action for blockchain-watcher * adding log-message-published mapper * deps: remove peers and nodemon * adding handler for LogMessagePublished * added parser for log message published --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> * Blockchain Watcher: ethereum -> sns implementation (#790) * evm block repo implementation * adding sns publisher * adding external config and initial runner * fix: start from latest if no fromBlock configured * feat: add dryRun option * fix: handler filtering and eth_getLogs filter payload * local e2e * actual sns arns for testnet * smaller docker image * deployment changes * [Blockchain Watcher] Apply missing formatting for log message published (#791) * Apply missing formatting for log message published * keep extracting * emitterAddress -> emitter * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * [Blockchain watcher] Adding light observability (#793) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * better error logs * wait when no block available * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * group actions per platform * [Blockchain Watcher] Add StartJob action + rate limit handling (#799) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * adding poll solana txs action * adding solana repo implementation * solana: mapping to logmsgpublished * fixing solana mapper test * updating readme * adding some metrics * adding solana worker * adding finality param to get solana block * minor changes - using confirmed for solana log extraction * updating version * add solana env var * [Blockchain Watcher] Feature-813/fix-coverage-report (#815) * Fix coverage report * Run prettier * Rollback global coverage value * Set global coverage value to 55 * Add github PR template * Run prettier * Rename github template file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * private solana urls as secret * adding prod cfg files * Change repository name and cretae error foulder (#819) * Change repository name and cretae error foulder * Rename SolanaFailure file * Rename SolanaFailure file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * passing node_options for solana * adding missing envs templates * cru * ratelimited solana client + increase coverage * [Blockchain Watcher] Feature 817/add hexagonal prefix (#824) * Put hexadecimal prefix * Create id variable * Add error log * Add error log * Remove bigint in getBlock params --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * fix never ending solana jobs * adding job counter metric * solana: add retry handling for 429s * cleanup * remove duplicated tests * cru + item count * minor readme update + do not use private solana rpc * fix items counter * fix solana job programID for devnet * log basic repo info * solana: passing finality everywhere * delete unnecesary files * adjusting solana cfgs * [Blockchain Watcher] Fix/re order some files (#828) * fix/re-order-some-files * run prettier * Run prettier * Resolve comment in PR * Remove import * Resolve comment * Remove prefix in sns log --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> Co-authored-by: Julian <52217955+julianmerlo95@users.noreply.github.com> Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local>
2023-11-30 07:05:43 -08:00
```
## Usage & Modification
[Blockchain Watcher] Emit LogMessagePublished for Solana (#809) * adding gen-relayer watcher code to branch * folder rename * adding some new abstract types for processor redesign * added first cut of new process manager logic * large refactoring to environment configuration * Add basic default config * prettier run * modifying event handler interface * abstract handler typing changes * Blockchain watcher: adding domain and infra layers (#786) * reorg domain-infra * watch evm blocks action * adding evm log parser * wider prettier * renaming watch action * adding doc * persist latest metadata * gh action for blockchain-watcher * adding log-message-published mapper * deps: remove peers and nodemon * adding handler for LogMessagePublished * added parser for log message published --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> * Blockchain Watcher: ethereum -> sns implementation (#790) * evm block repo implementation * adding sns publisher * adding external config and initial runner * fix: start from latest if no fromBlock configured * feat: add dryRun option * fix: handler filtering and eth_getLogs filter payload * local e2e * actual sns arns for testnet * smaller docker image * deployment changes * [Blockchain Watcher] Apply missing formatting for log message published (#791) * Apply missing formatting for log message published * keep extracting * emitterAddress -> emitter * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * [Blockchain watcher] Adding light observability (#793) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * better error logs * wait when no block available * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * group actions per platform * [Blockchain Watcher] Add StartJob action + rate limit handling (#799) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * adding poll solana txs action * adding solana repo implementation * solana: mapping to logmsgpublished * fixing solana mapper test * updating readme * adding some metrics * adding solana worker * adding finality param to get solana block * minor changes - using confirmed for solana log extraction * updating version * add solana env var * [Blockchain Watcher] Feature-813/fix-coverage-report (#815) * Fix coverage report * Run prettier * Rollback global coverage value * Set global coverage value to 55 * Add github PR template * Run prettier * Rename github template file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * private solana urls as secret * adding prod cfg files * Change repository name and cretae error foulder (#819) * Change repository name and cretae error foulder * Rename SolanaFailure file * Rename SolanaFailure file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * passing node_options for solana * adding missing envs templates * cru * ratelimited solana client + increase coverage * [Blockchain Watcher] Feature 817/add hexagonal prefix (#824) * Put hexadecimal prefix * Create id variable * Add error log * Add error log * Remove bigint in getBlock params --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * fix never ending solana jobs * adding job counter metric * solana: add retry handling for 429s * cleanup * remove duplicated tests * cru + item count * minor readme update + do not use private solana rpc * fix items counter * fix solana job programID for devnet * log basic repo info * solana: passing finality everywhere * delete unnecesary files * adjusting solana cfgs * [Blockchain Watcher] Fix/re order some files (#828) * fix/re-order-some-files * run prettier * Run prettier * Resolve comment in PR * Remove import * Resolve comment * Remove prefix in sns log --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> Co-authored-by: Julian <52217955+julianmerlo95@users.noreply.github.com> Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local>
2023-11-30 07:05:43 -08:00
Currently, jobs are read and loaded based on a JSON file.
Each job has a source, and one or more handlers.
Each handler has an action, a mapper and a target. For example, you can choose to use PollEvm (GetEvmLogs) as an action and HandleEvmLogs as a handler. For this handler you need to set a mapper like evmLogMessagePublishedMapper.
[Blockchain Watcher] Emit LogMessagePublished for Solana (#809) * adding gen-relayer watcher code to branch * folder rename * adding some new abstract types for processor redesign * added first cut of new process manager logic * large refactoring to environment configuration * Add basic default config * prettier run * modifying event handler interface * abstract handler typing changes * Blockchain watcher: adding domain and infra layers (#786) * reorg domain-infra * watch evm blocks action * adding evm log parser * wider prettier * renaming watch action * adding doc * persist latest metadata * gh action for blockchain-watcher * adding log-message-published mapper * deps: remove peers and nodemon * adding handler for LogMessagePublished * added parser for log message published --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> * Blockchain Watcher: ethereum -> sns implementation (#790) * evm block repo implementation * adding sns publisher * adding external config and initial runner * fix: start from latest if no fromBlock configured * feat: add dryRun option * fix: handler filtering and eth_getLogs filter payload * local e2e * actual sns arns for testnet * smaller docker image * deployment changes * [Blockchain Watcher] Apply missing formatting for log message published (#791) * Apply missing formatting for log message published * keep extracting * emitterAddress -> emitter * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * [Blockchain watcher] Adding light observability (#793) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * better error logs * wait when no block available * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * group actions per platform * [Blockchain Watcher] Add StartJob action + rate limit handling (#799) * using configured default winston logger * wait for new block * adding basic metrics * staging mainnet config * send hexa block number * simpler log format * group entities in different files * grouping watchers * adding 429 handling http client * abstracting polling iteration * Load jobs dinamically * changing deployment strategy to jobs per pod * adding poll solana txs action * adding solana repo implementation * solana: mapping to logmsgpublished * fixing solana mapper test * updating readme * adding some metrics * adding solana worker * adding finality param to get solana block * minor changes - using confirmed for solana log extraction * updating version * add solana env var * [Blockchain Watcher] Feature-813/fix-coverage-report (#815) * Fix coverage report * Run prettier * Rollback global coverage value * Set global coverage value to 55 * Add github PR template * Run prettier * Rename github template file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * private solana urls as secret * adding prod cfg files * Change repository name and cretae error foulder (#819) * Change repository name and cretae error foulder * Rename SolanaFailure file * Rename SolanaFailure file --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * passing node_options for solana * adding missing envs templates * cru * ratelimited solana client + increase coverage * [Blockchain Watcher] Feature 817/add hexagonal prefix (#824) * Put hexadecimal prefix * Create id variable * Add error log * Add error log * Remove bigint in getBlock params --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> * fix never ending solana jobs * adding job counter metric * solana: add retry handling for 429s * cleanup * remove duplicated tests * cru + item count * minor readme update + do not use private solana rpc * fix items counter * fix solana job programID for devnet * log basic repo info * solana: passing finality everywhere * delete unnecesary files * adjusting solana cfgs * [Blockchain Watcher] Fix/re order some files (#828) * fix/re-order-some-files * run prettier * Run prettier * Resolve comment in PR * Remove import * Resolve comment * Remove prefix in sns log --------- Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local> --------- Co-authored-by: chase-45 <chasemoran45@gmail.com> Co-authored-by: Julian <52217955+julianmerlo95@users.noreply.github.com> Co-authored-by: Julian Merlo <julianmerlo@MacBook-Pro-de-Julian.local>
2023-11-30 07:05:43 -08:00
The target can be sns, or a fake one if dryRun is enabled.