When the solana watcher is restarted (due to network errors, for
example) then the `lastSlot` state is lost. This means that any
transactions in between the last processed slot and the most recent slot
will be lost and require manual re-observation. Fix this by making the
`lastSlot` state persistent across watcher restarts.
* node: poller timeout
Change-Id: Ia324f1ac482fa9c5bea2b501970f0b22b16e67ce
* Add a comment explaining readiness change
* Add comment explaining why we are using a timeout
* Retry if polling fails
* governor listen to quorum gossip
* governor listen to quorum gossip
* Fix build error in tests
* See p2p from other guardians in devnet
* Fix test broke during merge
* Change delete function being used
* Don't reload duplicates on startup
* node: eth publish immediately
Change-Id: I242f02d0ed5bcced5ed366a512c07d5757b300e8
* testing: immediate publish test
* Move magic number to the sdk
Change-Id: I806f73d4442af4736aa6e5fc1c8e48e434c4a6d4
Co-authored-by: Evan Gray <battledingo@gmail.com>
Commit "02cf08531 node: move watchers to dedicated package (#1647)"
removed a lot of packages from the node dependency list. Run `go mod
tidy` to clean up the go.mod and go.sum files.
* node: reorganize watchers into own package
Change-Id: Idda82def1c8e1e07376bdc46a50fc02bd6f2386d
* node: rename terra to cosmwasm
Change-Id: Iebe5ccc7c66b772962425a42997bd2cfb66c6908
* Governor publish gossip
Change-Id: I2b8b1ea84a0c411101a7027acd3a27a6d6464d59
* Update the config publish time
Change-Id: Ic6abf84befb1c20756da2ff66b15a8325dc46067
* Not setting value on enqueued VAAs correctly
Change-Id: I9fd3a5d8fc574f8382125445fa688efdae45b88c
* Publish at most 20 VAAs, not 20 per chain
Change-Id: Ic9dff99c59ee89d57fd79158844a1fe1a0003112
* Switch to using signed messages
Change-Id: I66cddc7477cd477aa77bdadfc346b588f2ae645b
* Publish status only once per minute
Change-Id: I972fb0cf868e89c6f74ae4441471a55df389f4dd
* Minor comment change
Change-Id: I0d3e443cbec7edd282f89c1a5cce5d5ec8776d55
* Add command to purge pythnet VAAs
* Add test for purging a single emitter address
* Fix lint error
* Using the wrong delete primative
Change-Id: I80d5294c17279d4e49220d81807e5964a5591721
The per-watcher channels for reobservation requests don't have a
buffer, which means that if sending on any one channel blocks then _all_
reobservation requests get blocked. Make the send fallible and log if
it blocks instead.
We currently run the cleanup loop every 30 seconds, which means that
once 5 minutes have passed for an observation without quorum we will
send out re-observation requests to the p2p network every 30 seconds.
This is a bit excessive so limit sending these requests out to once
every 5 minutes.
When processing pending observations, the `handleCleanup` function
checks if we already have a stored quorum VAA and deletes the in-memory
observation if one is found. If a stored quorum VAA is not found, then
we're supposed to continue evaluating the other conditions and take
the appropriate action. This was implemented using a `fallthrough`
statement.
Unfortunately, this is not how `fallthrough` works. `fallthrough`
simply tells the compiler to execute the body of the next branch in
the switch block, *without evaluating the condition*. It also doesn't
evaluate the conditions for any of the other branches in the switch.
In practice what this meant is that for local observations that didn't
have quorum we would always take the first branch, fall through to the
second branch, and then exit the switch. Only once we had a quorum
(`s.submitted == true`) would we actually consider any of the other
branches in the switch. It also meant that there was no case where we
would take the branch for re-observing messages that hadn't reached
quorum.
Fix this by moving the stored quorum VAA check into an if statement and
then falling through to the switch statement if one is not found.
The wormhole sdk is a new go module in the sdk/ directory. This
initially contains the *_consts.go files from the common package in the
top-level sdk package and the entire vaa package as a sub-package.
For go reasons this needs to be in the sdk directory itself (rather than
a sdk/go subdir). To prevent the go tooling from looking into the other
non-go subdirs, add an empty go.mod file in each one. See
golang issue 42965 for more details on why we can't have nice
things (I'm deliberately not linking to stop github from spamming that
issue).
Needed to build with newer go toolchains (up to 1.19). Many of the
libp2p subpackages have been merged as sub-directories of the main
libp2p package so update the imports to use the newer versions.
* Initial code
* Add test cases
* Get rid of excessive logging of price updates
* Add big tx size to available notional query
* Add tool to verify price query still works
* Fix lint error
* Rename check_query.go
* Set big transaction sizes
* More rework
* Add unit-test for big transfer config
* Fix unit tests after updates for Near
* Add a couple more test cases
Co-authored-by: claudijd <jclaudius@jumptrading.com>
* node/cmd/guardiand: pythnet for mainnet
Pythnet is a read only chain that will only use the core bridge to
transfer messages to other chains. No tokens will be transferred
to pythnet, so it has no token or nft bridge.
* node/cmd/guardiand: fix help for solana flags
Commit `1753bb3: Send re-observation request when re-broadcasting local
observations` changed the processor code to automatically send
re-observation requests. If multiple guardians do this around the same
time that would lead to multiple re-observation requests for the same
transaction.
Deduplicate re-observation requests by keeping a cache of the requests
that were received in the last 11 minutes. Any request that's already
in the cache will not be forwarded to the chain-specific observation
code. The 11 minute value was chosen because the guardians will send
these requests approximately every 5 minutes.
Also move the re-observation code into a standalone function so that it
can be more easily tested.
Configure the bootstrapping nodes that we will connect to to seed and refresh
the Routing Table if it becomes empty.
Fixes an issue where the spy stops receiving gossiped messages after the node
is restarted, because the Routing Table becomes empty and is never refreshed.
Currently if an observation hasn't reached quorum within 5 minutes, the
processor will re-broadcast the signed local observation to the other
guardians in the network. However if not enough guardians actually
observed the original tx, then no amount of re-broadcasting will help
the network reach quorum.
Fix this issue by sending a re-observation request whenever we
re-broadcast a signed local observation. This ensures that any
guardians that missed the tx the first time it happened have a chance to
re-observe it and help the network reach quorum.
Any goroutine can push into a channel so the current implementation has
a race condition where the channel can become full immediately after the
length check, causing the subsequent send on the channel to block.
Fix this by wrapping the send on the channel with a select block.
Control will fall through to the default case only if the actual send
operation blocks, avoiding the potential race with other goroutines.
* Non-critical tweaks for the next scheduled deploy
* Make tests fail if mainnet config is invalid
* Make the test pass
* Add test to verify testnet config
* Add support for included token list to script
* Initial include list and algorand token addresses
* Fix typo in algorand token
* Lowering min notional to zero
* Rebase
* Reload from db on start up
Change-Id: I1deac9db28ad1157ea7e0c84af41c35b38497f4e
* Console commands
Change-Id: Ic242038312b7c837443a2df8823f100b3cdffd77
* Query prices from CoinGecko
Change-Id: I9a8c282ba374d32ef7045d11979a27ede3c52827
* Move chain config to separate file
Change-Id: I6a790eca765bce1f2caf48134e225df5c4daff15
* More code cleanup
Change-Id: Id12affa78cdc2d394d6dab0c53bb7ad06df8007e
* Few minor tweaks
Change-Id: I6cb511599d669e0b3d716d9f314ac0f26935ee92
* Create separate tests for different packages
Change-Id: Idb4da6817c9daad2a7420abc11bdaa702ae056dc
* Fix lint errors
Change-Id: I137c9e7e4574aee9c9fec22e91e19eee0e86a349
* Simplify chainlock message tests
* Add more governor db test coverage
* Next batch of review rework
Change-Id: Ife54852fca6c6990d1ffb3d60a8dd7f49d526f0a
* Still more rework
Change-Id: I43a8aa7fa4e1a7cea4d7fde68c963123c1ca8d53
* More rework
Change-Id: I9382412af4ffeda74967a834a6b0195a9d28b720
* Fix lint errors
Change-Id: Idaafce9b0314192557b7207911375d000bac5ae2
* Add rest and prometheus support
Change-Id: Ib870ed7eb305ef1ebbf6a7cedabc665c37c19171
* Add separate configs for testnet and devnet
Change-Id: I76b11d8940a8dc9935b3f276a008ed20ef60b850
* Update mainnet tokens to fix decimals
Change-Id: Iab018827766bc7748038b7be2f51342afb79b87c
* Let small enqueued VAAs go out when big ones can't
Change-Id: I7d3ef88d4579702d0c6ff4eaf5a8524799610ff6
* Tweak testnet config parameters
Change-Id: Id2c54151a7183ab3fb4af8060929198f6021ba4e
* Rework / enhancements from testnet testing
Change-Id: I1387b1d22667fa6ffe0bb1832dbc0b31196505d3
* Use known emitter maps
Change-Id: If330ee9d30ac3c2d1c6dca674f7777dc759de230
* Fix typo and out of date comments
Change-Id: I54a19792104ccc6ca023020303a710ef3ba18f74
Co-authored-by: claudijd <jclaudius@jumptrading.com>
We need to reuse almost all of the gossip infrastructure for accounting
transactions, with the only difference being that accounting will use a
`Transfer` message rather than a `VAA`.
Make the observation stored in the processor state generic so that it
can be either a VAA or a Transfer. The rest of the code is shared.
Create separate variables for the known tokenbridge and nft bridge
emitters and dynamically build KnownEmitters from those lists. Having
separate variables for the tokenbridge and nft emitters will make it
easier to look them up without having to iterate over the whole emitter
list every time.
Also add a devnet_consts file to list the known emitters in the
development network.
* node/pkg/terra: remove setEvents
The guardian set should only be authoritative from ethereum, so having
this code in the terra watcher is unnecessary. Getting this information
on injective requires a few more hoops to jump through but it is never
used. It is easier to maintain less code, so remove it.
* node/cmd/guardiand: remove setEvents from cosmwasm
This removes the set events from:
* terra 2
* injective
* terra classic
* node/pkg/terra: don't query guardianset from cosmwasm
* node/pkg/terra: support injective
* node/pkg/terra: update queryLatency metric
To show the query latency of fetching the latest block height from rpc
* Add unit tests for db
* Fix error checks and clean up tempdirs
* Add string explainer in the absence of a panic
* Remove unnecessary comment
* Fix NoError suggestion
* Add linter exception for panic test
* node: Shrink final docker image size
* Tiltfile: guardiand should run the build stage
* node: allow building without -race
* node: Support development builds
A development build must use the --unsafeDevMode flag.
* CI: build docker image
Co-authored-by: Csongor Kiss <ckiss@jumptrading.com>
* Add unit-test coverage for marshalling
* Add test to see if Unmarshal will panic on a > 1000 payload
* Error instead of truncate on payloads over 1000 bytes
* Clarify intents and tests for vaa.Unmarshal
* Implement message posting with account reuse
Change-Id: I195f493f6816048f5f8f76e1f0f6e561fa0fe692
* Use different magic for unreliable messages
* guardiand: Ignore solana instructions with empty data
Co-authored-by: Csongor Kiss <ckiss@jumptrading.com>
The current guardian set index is 2, so the `guardiand` template command
should generate prototxt files accordingly.
Co-authored-by: Csongor Kiss <ckiss@jumptrading.com>
* Added the `User-Agent` header to all requests to the block explorer
so cloudflare's browser check is happier.
* Add an option to sleep between loops hitting the block explorer. More
than 1 / second on polygonscan gets you blocked temporarily.
These changes are required to repair missed messages on polygon.
* Initial Celo support
Change-Id: Iee98ee29a8b09af9ca76945792b750d96a170702
* Watcher support
* Fix high level make file
* Remove node binary committed by mistake
* Port change from EthWatcher to CeloWatcher
* Use docker to build ABIs
* Refactor watcher to eliminate massive redunancy
* Fix lint-and-tests to handle multiple defs on link
* Minor code cleanup
* Celo should use ganache in devnet
* Prep for mainnet deploy
* Code review rework
* Remove celo from sdk package.json
* node/hack/repair_eth: use a cookiejar for http
Otherwise, explorers like polygonscan, which implement a cloudflare WAF,
will block all requests.
* node/hack/repair_eth: add -showError flag
When an explorer is having problems or blocks a repair http request, it returns an error like:
Failed to get current height: failed to decode response: invalid character '<' looking for beginning of value
This allows showing the actual response body to better troubleshoot the problem.
The 'timestamp' field is now allowed in a governance prototxt, which
applies to all the messages in the struct. If the timestamp is not
specified, then it defaults to 0, which is identical to the existing
behaviour.
* Add more comprehensive structs unit-tests
* Make VerifySignatures fail on duplicate addresses
* Adjust duplicate detection to guard on signatures instead of addresses
* Add monotonic check in VerifySignatures
* Move logic into VerifySignatures and add more test cases
* Add a paranoid check for duplicate signers
* Make VerifySignatures unit-tests less contrived
* Add more verify signature test cases
* Refactor VerifySignatures tests
* Add VerifySignature fuzz tests
* Add tc.result checking instead of hardcoded true
* Change comparison so it throws debug on failure for fuzz tests
* Add unit-tests for observation signature logic
* Fix comment typos
* Refactor observation tests
* Add missing test case
* Fix VAAInvalidSignatures test case label
* Clean up unit-tests for observation and structs
* Change errorString convention in test
* Format Signature Verification Test Cases
* Remove unnecessary casting
* Add multi-signer same key cases
* Fix err usage in test cases
* Remove duplicate getVAA