Commit Graph

578 Commits

Author SHA1 Message Date
Braydon Fuller 962e7b87f8 test: update regtest 2016-04-08 12:00:10 -04:00
Braydon Fuller 7c344b5f24 build: updates for npm install 2016-04-08 12:00:10 -04:00
Braydon Fuller c4649c9b13 test: mark last zmq block before rpc calls 2016-04-08 12:00:10 -04:00
Braydon Fuller 31da32ecfd test: updated p2p integration test 2016-04-08 12:00:09 -04:00
Braydon Fuller 7c6e5cf7b1 bitcoind: only cache transaction with height if confirmations >= 6 2016-04-08 11:59:26 -04:00
Braydon Fuller 18310268a5 node: log intro with config path 2016-04-08 11:59:26 -04:00
Braydon Fuller 0f24dd5f49 config: update configuration options for exec path
- config options for bitcoind to specify exec path of bitcoind
- config options to connect to multiple bitcoind processes
- systemd and upstart preferred methods to daemonize
2016-04-08 11:59:26 -04:00
Braydon Fuller 5932b34a1f bitcoind: set height when starting 2016-04-08 11:59:26 -04:00
Braydon Fuller 9409374fbe bitcoind: fix multiple addresses for address history 2016-04-08 11:59:26 -04:00
Braydon Fuller b473b65207 bitcoind: fix tx event to include tx buffer 2016-04-08 11:59:26 -04:00
Braydon Fuller 60333bcb0e bitcoind: add mempool to address txid results 2016-04-08 11:59:26 -04:00
Braydon Fuller ab70aa666e bitcoind: add address utxos 2016-04-08 11:59:26 -04:00
Braydon Fuller 7d7dfe329d bitcoind: variable name fixes 2016-04-08 11:59:26 -04:00
Braydon Fuller af573b765b bitcoind: fix txid pagination 2016-04-08 11:59:26 -04:00
Braydon Fuller b69d848352 bitcoind: add lru caching for results 2016-04-08 11:59:26 -04:00
Braydon Fuller 7e70bbfa7d bitcoind: bitcoind service using rpc and zmq with address index 2016-04-08 11:59:26 -04:00
Jan Pochyla b55ecf3044 clamp tx pagination to 0 2016-04-01 18:11:49 +02:00
Braydon Fuller 462e4e3cdd Merge pull request #407 from bitpay/opt/multiple-address-history
Opt/multiple address history
2016-02-11 12:15:15 -05:00
Matias Alejo Garcia e36cdb717a rm empty keys 2016-02-11 10:42:30 -05:00
Matias Alejo Garcia 1a68ca4fae add tests to _updateAddressIndex 2016-02-10 15:38:02 -05:00
Matias Alejo Garcia 3bb3d82aac add counter for address mempool index 2016-02-10 15:03:34 -05:00
Matias Alejo Garcia e7895b4b34 use key as binary 2016-02-09 15:30:40 -05:00
Matias Alejo Garcia d0c2fa61d8 fix tests 2016-02-09 10:57:40 -05:00
Matias Alejo Garcia 9f87156adc fix format 2016-02-09 10:26:09 -05:00
Matias Alejo Garcia dae5c9d3d5 fix regtests 2016-02-08 15:07:32 -05:00
Matias Alejo Garcia c1d3f351f2 add address index to mempool + noBalance options 2016-02-08 13:21:53 -05:00
Matias Alejo Garcia e7e33313cf add noBalance options + mempoolAddressIndex 2016-02-08 12:50:32 -05:00
Braydon Fuller 6e8f3ee917 Add regtest from bitcore-lib 2016-02-03 18:29:33 -05:00
Braydon Fuller f473ddeddd Lower and include new concurrency limits 2016-02-03 12:28:32 -05:00
Braydon Fuller 93e5dbfc34 Address Service: Limit the number of simultaneous requests 2016-02-02 13:27:45 -05:00
Braydon Fuller 995b4b57d4 DB: Include docs on how to recreate the database 2016-01-28 13:47:26 -05:00
Braydon Fuller 98bd8ee560 DB Service: Include a version number for upgrading purposes 2016-01-28 12:08:05 -05:00
Braydon Fuller 3d7fb6f234 Address Service: End stream without pausing first
There was an issue where streams would still be held open if "pause" was
called before "end", this would lead to http requests from the insight-api
not being returned with an error status as soon as possible but would
instead stay open.
2016-01-26 13:25:53 -05:00
Braydon Fuller 39f8355cd9 Address Service: Bump maximum number of addresses default 2016-01-19 14:07:28 -05:00
Braydon Fuller 45029030f1 Address Service: Sort after unconfirmed and confirmed 2016-01-18 16:16:53 -05:00
Braydon Fuller e498e0fac2 Address Service: Include default callback earlier 2016-01-18 16:03:37 -05:00
Braydon Fuller d4f2df5c51 Address Service: Sort mempool txids 2016-01-18 15:56:05 -05:00
Braydon Fuller a166b6af23 Address Service: Removed nolonger used constant for cache 2016-01-18 15:06:18 -05:00
Braydon Fuller 62934b4b66 Address Service: Removed event listeners prior to stopping 2016-01-18 15:00:55 -05:00
Braydon Fuller 3d9b6d5532 Address Service: More tests for history 2016-01-18 12:59:49 -05:00
Braydon Fuller e79c00db10 Address Service: Updated tests and fixed various bugs 2016-01-15 18:04:32 -05:00
Braydon Fuller ead6c2f45f Address Service: Removed caching and added max query limits
Querying addresses that have millions of transactions is supported however
takes hundreds of seconds to fully calculate the balance. Creating a cache of
previous results wasn't currently working because the `isSpent` query is always
based on the current bitcoind tip. Thus the balance of the outputs would be included
however wouldn't be removed when spent as the output wouldn't be checked again
when querying for blocks past the last checkpoint. Including the satoshis in the
inputs address index would make it possible to subtract the spent amount,
however this degrades optimizations elsewhere. The syncing times or querying
for addresses with 10,000 transactions per address.

It may preferrable to have an additional address service that handles high-volume
addresses be on an opt-in basis so that a custom running client could select
high volume addresses to create optimizations for querying balances and history.
The strategies for creating indexes differs on these use cases.
2016-01-14 17:17:04 -05:00
Braydon Fuller 4fcec8755c Address Service: Fixed many bugs from tests
- Refactored getAddressSummary and added several tests
- Fixed bugs revealed from the integration regtests
- Updated many unit tests
2016-01-13 17:15:14 -05:00
Braydon Fuller 188ff28ec7 Address Service: Fixed HASH_TYPES_MAP naming issue 2016-01-11 16:51:00 -05:00
Braydon Fuller 8d2f69c5fd Address Service: Restored multi-address history queries
- Restored functionality to be able to query the history of multiple addresses in one query
- Sorted mempool transactions by timestamp in txid lists
2016-01-11 16:17:55 -05:00
Braydon Fuller 5c4f3c4453 Address Service: Use address summary cache for pagination 2016-01-11 16:17:55 -05:00
Braydon Fuller 8298e380ed Address Service: Use streams to combine inputs and outputs 2016-01-11 16:17:55 -05:00
Braydon Fuller cef2f7686d Address Service: Limit the length of outputs that can be queried at a time 2016-01-11 16:17:55 -05:00
Braydon Fuller 40eb4f50ae Address Service: Start to cache `getAddressSummary` based on range of block heights 2016-01-11 16:17:55 -05:00
Braydon Fuller cab25cf397 Address Service: Start to use streams for memory optimization with large queries 2016-01-11 16:17:55 -05:00
Samuel Reed 3214390d4c
Fix #377; db must contain hash type, not just hash.
Prevents erroneous crediting of all transactions to both the
p2pkh and the corresponding p2sh address.
2016-01-11 09:47:00 -06:00
Patrick Nagurny 02620a5b47 Merge pull request #352 from braydonf/mem
Optimized address service mempool index size
2015-11-03 17:23:28 -05:00
Braydon Fuller 0ea035c4f0 Address: Fixed race condition with transaction event handlers 2015-11-03 17:12:13 -05:00
Braydon Fuller 378edd4186 Web: Changed to pass express subapp to services instead of the router. 2015-11-03 10:32:01 -05:00
Braydon Fuller c5c8e21c6c Address: Fixed bug with isSpent confusion with prevTxId and txid 2015-11-03 10:26:32 -05:00
Braydon Fuller a1bae366b3 Database: Removed `runAllMempoolIndexes` method
Replaced with using `tx` and `txleave` to manage the state of the mempool indexes.
2015-11-02 17:18:37 -05:00
Braydon Fuller 89ef28f0b7 Optimize mempool address index memory footprint
- Adds default to store a large portion of the mempool index in leveldb
- Includes an option to use memdown to have the mempool index in-memory
2015-10-30 18:36:16 -04:00
Braydon Fuller 086ba5fcfc Switch to remove items from mempool index as they leave asynchronously. 2015-10-29 19:16:56 -04:00
Braydon Fuller f0ec424161 Added bindings to be able to listen to tx leaving mempool. 2015-10-29 16:15:00 -04:00
Rob Riddle 824e624929 Fix typos and some examples in service doc 2015-10-22 18:47:19 +03:00
Patrick Nagurny 9bdd9e7995 unconfirmed balance shouldn't include confirmed balance 2015-10-21 18:25:57 -04:00
Gabe Gattis 0c88a540fd Merge pull request #339 from braydonf/create-testnet
Add --testnet option to create command.
2015-10-21 15:44:22 -04:00
Braydon Fuller 92382d69cc Add --testnet option to create command. 2015-10-21 15:28:28 -04:00
Braydon Fuller 17b669c98f Add date to logs and formatting changes. 2015-10-21 14:47:35 -04:00
Chris Kleeschulte 0b761511a0 Merge pull request #335 from braydonf/aliases
Remove aliases for commands
2015-10-20 18:21:42 -04:00
Braydon Fuller 9ab989a68a Remove aliases for commands 2015-10-20 17:09:28 -04:00
Braydon Fuller 95bd5913b8 Clear warnings with bitcore create command. 2015-10-20 16:48:51 -04:00
Braydon Fuller 83058eae26 Only use local install if config file exists. 2015-10-20 14:56:09 -04:00
Braydon Fuller e046f7294e Changes to be able to run with only a configuration file.
- Adds parameter to cli methods to be able to specify the location of services modules. This is useful for packages that wrap bitcore-node to be able to pass along a node_modules directory with services.
- Adds another parameter for including additional services in the default settings.
- Will use the `process.env.HOME + '/.bitcore` as the default configuration location.
- There are now two `getDefaultConfig`, one that will instatiate a `~/.bitcore` directory with a default if it doesn't exist, and `getBaseDefaultConfig` that will return a basic configuration without additional services enabled.
- Changes logic to use the global install if a local node_modules version is not available, this would previously assume that it was a local install because of the existence of a configuration file.
2015-10-20 12:33:53 -04:00
Stephen Pair 05eb5a03f2 remove debug output from bitcore cli utility 2015-10-20 15:49:01 +00:00
Rob Riddle e9290d3cdc Use bitcore-lib in node package creation 2015-10-19 15:43:51 +03:00
Braydon Fuller 95b5bfbfcb Fix library reference. 2015-10-18 16:28:24 -04:00
Braydon Fuller 58894bc4b2 Expose cli methods. 2015-10-18 16:20:51 -04:00
Braydon Fuller 1dec93edd4 bitcore -> bitcore-lib 2015-10-17 00:56:29 -04:00
Patrick Nagurny 236d44441e don't crash when sendTransaction is called with bad data 2015-10-14 10:50:50 -04:00
Braydon Fuller 2e3d58c07a Fix bug where the outputIndex would be removed.
The address history `combineTransactionInfo` method removes the
outputIndex when creating the outputIndexes property. When these are
from the mempool the original reference is also modified.

We can easily solve this by returning new instances in `getOutputs`
and `getInputs` instead of a reference to the actual mempool instance.

This will also have the additional benefit that height and other
properties that will be the same for every mempool entry will
not be stored in memory longer than what is necessary to fulfill
a request.
2015-10-12 18:35:37 -04:00
Braydon Fuller d9b2a0969b Fixed bug with balance
There was a bug when getting unspent outputs that would include an output
that was spent in the mempool in addition to the new output with the change
address. This lead to a balance having an output counted twice towards the
end balance. The solution is to have the isSpent method for the address service
to also include if the output was spent in the mempool, as the isSpent
method exposed from bitcoind only includes if the output was spent in a block.
2015-10-12 09:29:49 -04:00
Patrick Nagurny 3e38d84d0b emit address/balance events for mempool transactions 2015-10-02 17:10:12 -04:00
Patrick Nagurny 9acb896f10 Wait for bitcoind to catch up if tip is not found 2015-10-02 13:35:42 -04:00
Braydon Fuller da9d856da3 Add comments to describe each prefix. 2015-10-02 10:56:28 -04:00
Braydon Fuller 27e90ef41a Give false if spent information not available. 2015-10-01 23:50:06 -04:00
Braydon Fuller 539b263c67 Add spentTxId Index
- To be able to query an inputTxId and inputIndex that spends an outputTxId and outputIndex
- Extends the mempoolSpentIndex to include the inputTxId and inputIndex
2015-10-01 20:03:31 -04:00
Patrick Nagurny caefe5dbd1 Merge pull request #265 from braydonf/doc-update
Updates to documentation for bitcore.io website.
2015-10-01 12:26:09 -04:00
Patrick Nagurny 3fa1340ef3 save tip as part of block db operations 2015-09-29 15:20:05 -04:00
Braydon Fuller f337622790 Updates to documentation for bitcore.io website. 2015-09-29 14:15:49 -04:00
Chris Kleeschulte af7966f94a Removed unneeded getTxOutSetInfo 2015-09-25 11:33:51 -04:00
Braydon Fuller 1183e0cae7 Service Documentation 2015-09-23 18:14:37 -04:00
Braydon Fuller 80fadc7ec3 Fix bug with address summary transaction count. 2015-09-21 17:02:11 -04:00
Chris Kleeschulte d474d03cf2 Merge pull request #248 from braydonf/pubkeyout
Address compatibility with pay-to-publickey output scripts.
2015-09-21 14:39:42 -04:00
Braydon Fuller 7e82c933c0 Merge pull request #249 from kleetus/feature/getNextBlockHash
Next Block Hash Bindings
2015-09-21 14:30:30 -04:00
Chris Kleeschulte 402d01d7ac Next Block Hash Bindings
- Added bindings for next block hash
- Added the reg/unit tests
2015-09-21 14:07:14 -04:00
Braydon Fuller e3096d3646 Address compatibility with pay-to-publickey output scripts. 2015-09-21 13:49:48 -04:00
Patrick Nagurny 4e763d189f fixes 2015-09-18 14:06:26 -04:00
Patrick Nagurny b11ab97739 add tests 2015-09-18 14:06:26 -04:00
Patrick Nagurny bc39a5e480 use mempool spent index 2015-09-18 14:06:26 -04:00
Patrick Nagurny 8b1a2aa001 get address summary 2015-09-18 14:06:26 -04:00
Patrick Nagurny 4a5031a917 Merge pull request #240 from braydonf/gettxoutsetinfo
Add binding for getting transaction output set information.
2015-09-18 13:20:46 -04:00
Patrick Nagurny 85f636f154 fix db stopping bug 2015-09-18 11:30:37 -04:00
Patrick Nagurny e5a58f986c Merge pull request #239 from kleetus/mempool-index
Mempool index
2015-09-18 10:19:18 -04:00
Braydon Fuller 68368397d3 Add binding to get the best block hash. 2015-09-17 18:11:47 -04:00
Braydon Fuller 4c674a8fbb Add binding for getting transaction output set information. 2015-09-17 17:52:23 -04:00
Chris Kleeschulte 0ba168e2a2 Fixed typo and added a test to prove that the inputs from the mempoolInputIndex are added on stream close. 2015-09-17 17:20:35 -04:00
Chris Kleeschulte c3654120ee MempoolInputIndex and MempoolOutputIndex
- Added in memory indexes for mempool input and 
output by address.
2015-09-17 14:29:11 -04:00
Braydon Fuller 62faecb9ae Address Mempool Index: Fix bugs with reset mempool. 2015-09-17 14:26:53 -04:00
Braydon Fuller ad120213e1 Start of mempool address index. 2015-09-17 14:26:53 -04:00
Braydon Fuller 37550166dd Database: Warn about inconsistent database state. 2015-09-16 20:02:09 -04:00
Braydon Fuller 4343b90de2 Merge pull request #219 from pnagurny/feature/timestamp-blocks
Get block hashes by timestamp range
2015-09-16 13:47:00 -04:00
Patrick Nagurny aa9504a402 Merge pull request #227 from braydonf/add-cmd-remote
Bitcore Node Add Command with Remote URLS
2015-09-16 13:34:24 -04:00
Patrick Nagurny a0be38f074 check for timestamp out of bounds 2015-09-16 12:04:44 -04:00
Chris Kleeschulte a105c0a35e Removed errors memoization. 2015-09-16 10:38:08 -04:00
Chris Kleeschulte c9d4dc276f Crash on reindex
- Removed unneeded cancellation error and handlers for it.
2015-09-16 10:22:42 -04:00
Chris Kleeschulte 4ee11ed73b Crash on reindex
- Introduced the concept of a Cancellation error so that services can choose to watch for a cancellation flag. 
- Services can then send this error back and it will be forwarded to the node.
- The node will then know to call shutdown appropriately.
2015-09-16 10:22:42 -04:00
Chris Kleeschulte dc6d0e681c Crash on reindex
- Added the concept of loadServices on the node so that the node can conditionally call stop on loadingServices
- This serves the case where services might be loading versus fully loaded (which is not always the cases for heavy services like bitcoind)
2015-09-16 10:22:42 -04:00
Patrick Nagurny 7e1d433781 go from highest timestamp to lowest timestamp 2015-09-15 18:23:06 -04:00
Patrick Nagurny e6b850124c fixes 2015-09-15 17:25:41 -04:00
Patrick Nagurny 00d3a0ba67 binary encode key and value 2015-09-15 16:55:14 -04:00
Braydon Fuller bfba4a1b7f Address History: Include count in response results. 2015-09-15 13:17:15 -04:00
Patrick Nagurny eaee098cf0 add getBlockHashesByTimestamp 2015-09-15 12:56:44 -04:00
Patrick Nagurny 696759849f Merge pull request #223 from braydonf/history
Pagination for Address History
2015-09-15 12:34:26 -04:00
Braydon Fuller 25e801126f Bitcore Node Add Command with Remote URLS 2015-09-15 11:08:55 -04:00
Braydon Fuller 5fbd8991ed Address History: Combine addresses so that history is txid unique. 2015-09-14 21:04:54 -04:00
Braydon Fuller e7587564a7 Address History: Add getAddressHistoryCount to available API methods. 2015-09-14 17:29:43 -04:00
Braydon Fuller b1b40c892e Address History: Add getAddressHistoryCount method 2015-09-14 16:48:28 -04:00
Braydon Fuller 87a9163743 Address History: Docs for "start" and "end" arguments 2015-09-14 16:29:27 -04:00
Braydon Fuller a80b58e004 Address History: Fix sorting bug 2015-09-14 16:11:31 -04:00
Chris Kleeschulte 7b50f5ff3d Switched to using a flag instead of using a replacement. 2015-09-14 13:15:55 -04:00
Braydon Fuller f88eee5a1c Improve db storage efficiency by ~2 times
Encodes and decodes keys and values for leveldb storage more efficiently.
2015-09-14 00:30:21 -04:00
Braydon Fuller d3641f3b0a Pagination Optimization
- Sort and paginate before getting full transaction details.
- Only get detailed transaction information for items within the current page.
- Improves the performance with large sets of transactions.
2015-09-14 00:30:14 -04:00
Braydon Fuller 1cf34f2dd8 Address Block Handling Optimizations
- Changes to use ripemd160 hash directly instead of the base58check encoded values
- Speeds block handling performance by ~4 times
2015-09-12 19:56:13 -04:00
Braydon Fuller c205f781a5 Pagination for Address History
- Reindexing the bitcore-node database is required with this change.
- Address indexes are updated to include address and height in spent index so that both inputs and outputs can be queried by address and height using "start" and "stop" to limit the range of the query.
- Address history also now supports paginated results using "from" and "to" values that indicate an index in the array.
2015-09-12 19:44:10 -04:00
Chris Kleeschulte 51355c51f1 Multiple sigints sent to the node process
- Added a simple replacement function after the first sigint is handled.
- Added unit tests for the exitHandler, requiring it to be exposed from the module.
2015-09-11 16:27:24 -04:00
Chris Kleeschulte 60a133f36c Multiple SIGINT signals sent to node process while in shutdown cycle.
- Overrode the sigint listener if in the shutdown cycle. This prevents control being returned to the pty before the node process is fully halted.
2015-09-11 12:39:33 -04:00
Chris Kleeschulte 65e09ef281 Added integration for the bus
- Added a new file that exercises the subscribe/unsubscribe/close methods with a test service
- Removed a duplicate call to inherit
2015-09-10 13:19:12 -04:00
Patrick Nagurny c4d28c3947 wait for db operations to complete before closing leveldb 2015-09-10 12:45:00 -04:00
Patrick Nagurny 8b0b401d52 inherit https options from node 2015-09-10 11:08:48 -04:00
Patrick Nagurny 5677964651 add https to web service 2015-09-09 16:49:36 -04:00
Patrick Nagurny 4682714c69 check for blockHandler on services 2015-09-09 15:04:07 -04:00
Braydon Fuller 2f6fc6e3e9 Merge pull request #202 from pnagurny/feature/main-chain
Add isMainChain method
2015-09-09 10:08:21 -04:00
Patrick Nagurny 0ff742c1b6 add isMainChain method 2015-09-08 17:38:59 -04:00
Braydon Fuller 60af86777f Start/Stop Improvements
- A Node will shutdown if there is an error starting a service as it would lead to cascading errors.
- `node.start()` needs to be called, and nolonger is called automatically when the instance is created.
- A service will only be added to node.services after it's started
- Stopping services that are not started will gracefully continue.
- Logging sync status of db will only apply if the service is started.
- Debug log about a service without a route will always include the service name
2015-09-08 10:17:57 -04:00
Braydon Fuller b4ed29eabe Merge pull request #193 from kleetus/reindex
Reindex logic
2015-09-08 10:15:34 -04:00
Chris Kleeschulte 94c345134e Daemon
- Added the daemon option to the cli start command
- Added the initialization routine to spawn the child process
- Added unit tests
2015-09-04 17:36:53 -04:00
Chris Kleeschulte 18aff3de0c Fixed the warning message to the user on a reindex. 2015-09-04 16:59:04 -04:00
Braydon Fuller 9aec734122 Merge pull request #195 from pnagurny/bug/transaction-fee
Handle getFee() error
2015-09-04 14:28:37 -04:00
Patrick Nagurny 79ae3bb412 handle getFee() error 2015-09-04 14:28:04 -04:00
Braydon Fuller 50925d1e0f Add tests for reorg improvement, and remove nolonger need code. 2015-09-04 13:52:59 -04:00
Chris Kleeschulte 730dd37318 Reindex logic
- If the reindex option is set in bitcoin.conf, then when start is called and onBlocksReady's callback is fired:
	- start's callback will not be fired until the reindex takes place.
	- along the way the sync percentage is display once per second
2015-09-04 11:39:38 -04:00
Patrick Nagurny 60a7f5ea29 optimizations for handling reorgs better 2015-09-04 11:32:04 -04:00
Patrick Nagurny a0e40ffd15 check for duplicate events 2015-09-03 17:29:28 -04:00
Patrick Nagurny 7ac429fbd2 remove name field from node 2015-09-03 16:59:43 -04:00