Commit Graph

137 Commits

Author SHA1 Message Date
Ethan Buchman f5824bc837
Update abci and app docs (#2470)
* mempool: update some comments

* make build_c

* docs: notes about databases and WAL files

* docs: determinism. closes #1279

* docs: small note about query paths. closes #2090

* docs: gas

* docs: abci consensus params
2018-09-23 01:14:05 -04:00
Dev Ojha 111e627037 mempool: Filter new txs if they have insufficient gas (#2385)
This also refactors the prior mempool to filter to be known as
"precheck filter" and this new filter is called "postcheck filter"

This PR also fixes a bug where the precheck filter previously didn't
account for the amino overhead, which could a maliciously sized tx to
halt blocks from getting any txs in them.

* Move maxGas outside of function definition to avoid race condition
* Type filter funcs and make public
* Use helper method for post check
* Remove superfluous Filter suffix
* Move default pre/post checks into package
* Fix broken references
* Fix typos
* Expand on examples for checks
2018-09-22 02:50:06 +02:00
Anton Kaliaev 8d50bb9dad conesnsu: follow up to removing some consensus params (#2427)
* follow up to removing some consensus params Refs #2382
* change args type to int64 in state#makeParams
* make valsCount and evidenceCount ints again
* MaxEvidenceBytesPerBlock: include magic number in godoc
* [spec] creating a proposal
* test state#TxFilter
* panic if MaxDataBytes is less than 0
* fixes after review
* use amino#UvarintSize to calculate overhead
0c74291f3b/encoder.go (L85-L90)
* avoid cyclic imports
* you can do better Go, come on
* remove testdouble package
2018-09-21 11:00:36 +02:00
bradyjoestar 26aa978456 Make mempool cache a proper LRU (#2407)
Closes #2399
2018-09-19 21:40:22 +04:00
Dev Ojha 1ea64fc27f Make mempool aware of MaxGas requirement (#2360)
* Make mempool aware of MaxGas requirement

* update spec

* Add tests

* Switch GasWanted from kv store to persistent kv store

* Fix typo in test name

* switch back to using kvstore, not persistent kv store
2018-09-12 16:41:19 -04:00
Anton Kaliaev 166fd82b70
max-bytes PR follow-up (#2318)
* ReapMaxTxs: return all txs if max is negative

this mirrors ReapMaxBytes behavior

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214439950

* increase MaxAminoOverheadForBlock

tested with:

```
func TestMaxAminoOverheadForBlock(t *testing.T) {

        maxChainID := ""
        for i := 0; i < MaxChainIDLen; i++ {
                maxChainID += "𠜎"
        }

        h := Header{
                ChainID:            maxChainID,
                Height:             10,
                Time:               time.Now().UTC(),
                NumTxs:             100,
                TotalTxs:           200,
                LastBlockID:        makeBlockID(make([]byte, 20), 300, make([]byte, 20)),
                LastCommitHash:     tmhash.Sum([]byte("last_commit_hash")),
                DataHash:           tmhash.Sum([]byte("data_hash")),
                ValidatorsHash:     tmhash.Sum([]byte("validators_hash")),
                NextValidatorsHash: tmhash.Sum([]byte("next_validators_hash")),
                ConsensusHash:      tmhash.Sum([]byte("consensus_hash")),
                AppHash:            tmhash.Sum([]byte("app_hash")),
                LastResultsHash:    tmhash.Sum([]byte("last_results_hash")),
                EvidenceHash:       tmhash.Sum([]byte("evidence_hash")),
                ProposerAddress:    tmhash.Sum([]byte("proposer_address")),
        }
        b := Block{
                Header:     h,
                Data:       Data{Txs: makeTxs(10000, 100)},
                Evidence:   EvidenceData{},
                LastCommit: &Commit{},
        }

        bz, err := cdc.MarshalBinary(b)
        require.NoError(t, err)

        assert.Equal(t, MaxHeaderBytes+MaxAminoOverheadForBlock-2, len(bz)-1000000-20000-1)
}
```

* fix MaxYYY constants calculation

by using math.MaxInt64

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214444244

* pass mempool filter as an option

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214445869

* fixes after Dev's comments
2018-09-04 11:46:34 +04:00
Anton Kaliaev e873fed815
calculate amino overhead on the fly 2018-08-31 16:01:22 +04:00
Anton Kaliaev d73c5cbdb1
reap max bytes from the mempool & check transaction size
See ADR 020: Limiting txs size inside a block docs/architecture/adr-020-block-size.md

Refs #2035
2018-08-31 16:01:21 +04:00
Dev Ojha bd531401a0 mempool: Store txs by hash inside of cache (#2234)
* mempool: Store txs by hash inside of cache

This allows for large cachesizes, without fear of the memory
consumption growing rapidly.

* (squash this) rename hashedTx -> txHash
2018-08-29 08:10:06 +04:00
Dev Ojha 8a1a79257e mempool: Keep cache hashmap and linked list in sync (#2188)
* mempool: Keep cache hashmap and linked list in sync

This removes bugs with the linked list being full, but hashmap empty

* address PR comments

* switch clist back to list
2018-08-13 14:42:33 +04:00
Dev Ojha 2756be5a59 libs: Remove usage of custom Fmt, in favor of fmt.Sprintf (#2199)
* libs: Remove usage of custom Fmt, in favor of fmt.Sprintf

Closes #2193

* Fix bug that was masked by custom Fmt!
2018-08-10 09:25:57 +04:00
Jun Kimura e1b9bf7c81 set capacity of txsmap (#2166) 2018-08-07 20:50:10 +04:00
Ethan Buchman 15b112e669 mempool: chan bool -> chan struct{} 2018-07-23 21:06:47 -04:00
srmo e36ce6f893 fix race condition on proposal height for published txs (#2021)
* #1920 try to fix race condition on proposal height for published txs

- related to create_empty_blocks=false
- published height for accepted tx can be wrong (too low)
- use the actual mempool height + 1 for the proposal
- expose Height() on mempool

* #1920 add initial test for mempool.Height()

- not sure how to test the lock
- can the mutex reference be of type Locker?
-- this way, we can use a "mock" of the mutex to test triggering

* #1920 use the ConsensusState height in favor of mempool

- gets rid of indirections
- doesn't need any "+1" magic

* #1920 cosmetic

- if we use cs.Height, it's enough to evaluate right before propose

* #1920 cleanup TODO and non-needed code

* #1920 add changelog entry
2018-07-23 15:34:45 +04:00
Ethan Buchman d55243f0e6 fix import paths 2018-07-01 22:36:49 -04:00
Anton Kaliaev aa20c45ae9 log total when adding new txs to the mempool 2018-06-27 18:14:45 +04:00
Ethan Buchman fa3bd05d44 mempool: log hashes, not whole tx 2018-06-27 18:14:45 +04:00
Alexander Simmerl a605b66c5a
Move abci imports 2018-06-22 07:28:07 +02:00
Ethan Buchman 9cd9f3338b Merge branch 'master' into develop 2018-06-21 12:57:09 -07:00
Anton Kaliaev 205d8b8062
fixes after @xla review
- move prometheus metrics into internal packages
- *Option structs
- misc. format changes
2018-06-20 12:40:25 +04:00
Anton Kaliaev 84812145cb
friendly apis for constructors 2018-06-20 12:40:25 +04:00
Anton Kaliaev 7efb73aa18
mempool size metric 2018-06-20 12:38:45 +04:00
Anton Kaliaev 4f5492c831
add nopTxCache (Nil Object Pattern)
to better handle zero cache size
2018-06-19 11:59:07 +04:00
Ethan Buchman 4b2348f697 mempool: fix cache_size==0. closes #1761 2018-06-18 18:21:19 -07:00
Ethan Buchman a519825bf8 consensus: fixes #1754
* updateToState exits early if the state isn't new, which happens after
* fast syncing. This results in not sending a NewRoundStep message. The mempool
* reactor depends on PeerState, which is updated by NewRoundStep
* messages. If the peer never sends a NewRoundStep, the mempool reactor
* will think they're behind, and never forward transactions. Note this
* only happens when `create_empty_blocks = false`, because otherwise
* peers will move through the consensus state and send a NewRoundStep
* for a new step soon anyways. Simple fix is just to send the
* NewRoundStep message during updateToState even if exit early
2018-06-18 17:08:09 -07:00
Anton Kaliaev 3fa734ef5a
recheck only if there are txs left in the mempool (#1645) 2018-06-04 14:28:47 +04:00
Ethan Buchman 98b0c51b5f fix possible mempool deadlock 2018-06-03 16:11:52 -04:00
Anton Kaliaev 202a43a5af
remove TODO
no longer relevant, I guess. since ABCI only defines 0 (success) code.
2018-05-18 10:48:13 +04:00
Anton Kaliaev c9001d5a11
bound the mempool
Refs #345
2018-05-18 10:45:51 +04:00
Anton Kaliaev ae572b9038
remove extra call to Exists
Refs #345

https://github.com/tendermint/tendermint/issues/1539#issuecomment-387240606
2018-05-18 10:17:09 +04:00
Anton Kaliaev bbe1355957
log only hash, not tx itself 2018-05-17 10:00:59 +04:00
Anton Kaliaev 7c14fa820d
do not log txs at info level
BEFORE:

```
./tm-bench -c 5 -r 1000 127.0.0.1:46657
Stats          Avg       StdDev     Max
Txs/sec        1826      843        2744
Blocks/sec     1.100     0.300      2
```

AFTER:

```
./tm-bench -T 30 -c 5 -r 1000 127.0.0.1:46657
Stats          Avg       StdDev     Max
Txs/sec        6120      1970       9776
Blocks/sec     1.000     0.000      1
```
2018-05-17 10:00:59 +04:00
Anton Kaliaev 11b68f1934
rewrite broadcastTxRoutine to use channels
https://play.golang.org/p/gN21yO9IRs3

```
func waitWithCancel(f func() *clist.CElement, ctx context.Context) *clist.CElement {
	el := make(chan *clist.CElement, 1)
	select {
	case el <- f():
```
will just run f() blockingly, so this doesn't change much in terms of behavior.
2018-02-05 16:36:26 +04:00
Anton Kaliaev 5f3048bd09
call FlushSync before calling CommitSync
if we call it after, we might receive a "fresh" transaction from
  `broadcast_tx_sync` before old transactions (which were not
  committed).

Refs #1091

```
Commit is called with a lock on the mempool, meaning no calls to CheckTx
can start. However, since CheckTx is called async in the mempool
connection, some CheckTx might have already "sailed", when the lock is
released in the mempool and Commit proceeds.

Then, that spurious CheckTx has not yet "begun" in the ABCI app (stuck
in transport?). Instead, ABCI app manages to start to process the
Commit. Next, the spurious, "sailed" CheckTx happens in the wrong place.
```
2018-01-23 16:56:14 +04:00
Ethan Buchman f06cc6630b mempool: cfg.CacheSize and expose InitWAL 2018-01-19 01:03:03 -05:00
Ethan Buchman c9be2b89f9 mempool: return error on cached txs 2017-12-02 01:15:11 -05:00
Anton Kaliaev 922af7c405
int64 height
uint64 is considered dangerous. the details will follow in a blog post.
2017-12-01 19:04:53 -06:00
Anton Kaliaev 86af889dfb remove unnecessary casts (Refs #911) 2017-12-01 17:17:22 -05:00
Anton Kaliaev b3492356e6 uint64 height (Refs #911) 2017-12-01 17:17:22 -05:00
Emmanuel Odeke 3595b5931a
mempool: implement Mempool.CloseWAL
Fixes https://github.com/tendermint/tendermint/issues/890

Add a CloseWAL method to Mempool to close the underlying WAL file
and then discard it so that further writes to it will have no effect.
2017-11-27 21:37:25 -07:00
Zach Ramsay 8f0237610e linting errors: clean it all up 2017-11-27 22:39:11 +00:00
Zach Ramsay b75d4f73e7 errcheck: PR comment fixes 2017-11-27 22:39:11 +00:00
Zach Ramsay 331857c9e6 linting: apply errcheck part2 2017-11-27 22:39:11 +00:00
Ethan Buchman ac2ef9e0ea Merge pull request #750 from tendermint/feature/cleanup
Cleanup of code and code docs
2017-10-23 11:14:15 -04:00
Adrian Brink 819f81f702 Change NOTE to CONTRACT 2017-10-23 11:04:45 -04:00
Adrian Brink 782a836db0 Cleanup of code and code docs
This cleans up some of the code in the state package
2017-10-23 11:04:45 -04:00
Anton Kaliaev f908dd0e55 only notify when there are some txs (Refs #753) 2017-10-23 10:31:00 -04:00
Anton Kaliaev 54f2cc9709
[docs] add how to read logs guide [ci skip] 2017-10-04 18:35:22 +04:00
Ethan Buchman 88138c38cf mempool: reactor test 2017-09-05 16:25:02 -04:00
Ethan Buchman c5a657f540 consensus: test proposal heartbeat 2017-08-10 01:24:23 -04:00