Commit Graph

230 Commits

Author SHA1 Message Date
Ethan Buchman f5824bc837
Update abci and app docs (#2470)
* mempool: update some comments

* make build_c

* docs: notes about databases and WAL files

* docs: determinism. closes #1279

* docs: small note about query paths. closes #2090

* docs: gas

* docs: abci consensus params
2018-09-23 01:14:05 -04:00
Dev Ojha 111e627037 mempool: Filter new txs if they have insufficient gas (#2385)
This also refactors the prior mempool to filter to be known as
"precheck filter" and this new filter is called "postcheck filter"

This PR also fixes a bug where the precheck filter previously didn't
account for the amino overhead, which could a maliciously sized tx to
halt blocks from getting any txs in them.

* Move maxGas outside of function definition to avoid race condition
* Type filter funcs and make public
* Use helper method for post check
* Remove superfluous Filter suffix
* Move default pre/post checks into package
* Fix broken references
* Fix typos
* Expand on examples for checks
2018-09-22 02:50:06 +02:00
Anton Kaliaev 8d50bb9dad conesnsu: follow up to removing some consensus params (#2427)
* follow up to removing some consensus params Refs #2382
* change args type to int64 in state#makeParams
* make valsCount and evidenceCount ints again
* MaxEvidenceBytesPerBlock: include magic number in godoc
* [spec] creating a proposal
* test state#TxFilter
* panic if MaxDataBytes is less than 0
* fixes after review
* use amino#UvarintSize to calculate overhead
0c74291f3b/encoder.go (L85-L90)
* avoid cyclic imports
* you can do better Go, come on
* remove testdouble package
2018-09-21 11:00:36 +02:00
bradyjoestar 26aa978456 Make mempool cache a proper LRU (#2407)
Closes #2399
2018-09-19 21:40:22 +04:00
Dev Ojha 1ea64fc27f Make mempool aware of MaxGas requirement (#2360)
* Make mempool aware of MaxGas requirement

* update spec

* Add tests

* Switch GasWanted from kv store to persistent kv store

* Fix typo in test name

* switch back to using kvstore, not persistent kv store
2018-09-12 16:41:19 -04:00
Anton Kaliaev 166fd82b70
max-bytes PR follow-up (#2318)
* ReapMaxTxs: return all txs if max is negative

this mirrors ReapMaxBytes behavior

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214439950

* increase MaxAminoOverheadForBlock

tested with:

```
func TestMaxAminoOverheadForBlock(t *testing.T) {

        maxChainID := ""
        for i := 0; i < MaxChainIDLen; i++ {
                maxChainID += "𠜎"
        }

        h := Header{
                ChainID:            maxChainID,
                Height:             10,
                Time:               time.Now().UTC(),
                NumTxs:             100,
                TotalTxs:           200,
                LastBlockID:        makeBlockID(make([]byte, 20), 300, make([]byte, 20)),
                LastCommitHash:     tmhash.Sum([]byte("last_commit_hash")),
                DataHash:           tmhash.Sum([]byte("data_hash")),
                ValidatorsHash:     tmhash.Sum([]byte("validators_hash")),
                NextValidatorsHash: tmhash.Sum([]byte("next_validators_hash")),
                ConsensusHash:      tmhash.Sum([]byte("consensus_hash")),
                AppHash:            tmhash.Sum([]byte("app_hash")),
                LastResultsHash:    tmhash.Sum([]byte("last_results_hash")),
                EvidenceHash:       tmhash.Sum([]byte("evidence_hash")),
                ProposerAddress:    tmhash.Sum([]byte("proposer_address")),
        }
        b := Block{
                Header:     h,
                Data:       Data{Txs: makeTxs(10000, 100)},
                Evidence:   EvidenceData{},
                LastCommit: &Commit{},
        }

        bz, err := cdc.MarshalBinary(b)
        require.NoError(t, err)

        assert.Equal(t, MaxHeaderBytes+MaxAminoOverheadForBlock-2, len(bz)-1000000-20000-1)
}
```

* fix MaxYYY constants calculation

by using math.MaxInt64

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214444244

* pass mempool filter as an option

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214445869

* fixes after Dev's comments
2018-09-04 11:46:34 +04:00
Anton Kaliaev e873fed815
calculate amino overhead on the fly 2018-08-31 16:01:22 +04:00
Anton Kaliaev d73c5cbdb1
reap max bytes from the mempool & check transaction size
See ADR 020: Limiting txs size inside a block docs/architecture/adr-020-block-size.md

Refs #2035
2018-08-31 16:01:21 +04:00
Dev Ojha bd531401a0 mempool: Store txs by hash inside of cache (#2234)
* mempool: Store txs by hash inside of cache

This allows for large cachesizes, without fear of the memory
consumption growing rapidly.

* (squash this) rename hashedTx -> txHash
2018-08-29 08:10:06 +04:00
Dev Ojha 8a1a79257e mempool: Keep cache hashmap and linked list in sync (#2188)
* mempool: Keep cache hashmap and linked list in sync

This removes bugs with the linked list being full, but hashmap empty

* address PR comments

* switch clist back to list
2018-08-13 14:42:33 +04:00
Dev Ojha 2756be5a59 libs: Remove usage of custom Fmt, in favor of fmt.Sprintf (#2199)
* libs: Remove usage of custom Fmt, in favor of fmt.Sprintf

Closes #2193

* Fix bug that was masked by custom Fmt!
2018-08-10 09:25:57 +04:00
Jun Kimura e1b9bf7c81 set capacity of txsmap (#2166) 2018-08-07 20:50:10 +04:00
Anton Kaliaev 96ae535fb8 proto3 timestamp (#2064)
This PR changes ABCI time format from int64 (Unix seconds) to WKT (WellKnownType) google.protobuf.Timestamp.

Refs #1857

Reasons:

better precision
standard DT for proto

* update Gopkg.lock
* [makefile] remove extra grep
    - go list excludes vendor by default now
* proto3 timestamp
* [docs/abci-spec] note about serialisation format
* make time non-nullable
2018-07-27 04:23:19 +02:00
Ethan Buchman 15b112e669 mempool: chan bool -> chan struct{} 2018-07-23 21:06:47 -04:00
srmo e36ce6f893 fix race condition on proposal height for published txs (#2021)
* #1920 try to fix race condition on proposal height for published txs

- related to create_empty_blocks=false
- published height for accepted tx can be wrong (too low)
- use the actual mempool height + 1 for the proposal
- expose Height() on mempool

* #1920 add initial test for mempool.Height()

- not sure how to test the lock
- can the mutex reference be of type Locker?
-- this way, we can use a "mock" of the mutex to test triggering

* #1920 use the ConsensusState height in favor of mempool

- gets rid of indirections
- doesn't need any "+1" magic

* #1920 cosmetic

- if we use cs.Height, it's enough to evaluate right before propose

* #1920 cleanup TODO and non-needed code

* #1920 add changelog entry
2018-07-23 15:34:45 +04:00
Anton Kaliaev 9120fd5d14
unexport DecodeMessage functions
Refs #646
2018-07-09 13:01:23 +04:00
Ethan Buchman d55243f0e6 fix import paths 2018-07-01 22:36:49 -04:00
Anton Kaliaev ab04201c3d
fix empty packet size calculation
Fixes #1762#issuecomment-398940107
2018-06-29 12:40:32 +04:00
Anton Kaliaev aa20c45ae9 log total when adding new txs to the mempool 2018-06-27 18:14:45 +04:00
Ethan Buchman fa3bd05d44 mempool: log hashes, not whole tx 2018-06-27 18:14:45 +04:00
Alexander Simmerl a605b66c5a
Move abci imports 2018-06-22 07:28:07 +02:00
Ethan Buchman 9cd9f3338b Merge branch 'master' into develop 2018-06-21 12:57:09 -07:00
Anton Kaliaev 205d8b8062
fixes after @xla review
- move prometheus metrics into internal packages
- *Option structs
- misc. format changes
2018-06-20 12:40:25 +04:00
Anton Kaliaev e4bb3566a0
move metrics constructors to a separate package 2018-06-20 12:40:25 +04:00
Anton Kaliaev 84812145cb
friendly apis for constructors 2018-06-20 12:40:25 +04:00
Anton Kaliaev b10b0da3fd
bundle imports 2018-06-20 12:40:11 +04:00
Anton Kaliaev 7efb73aa18
mempool size metric 2018-06-20 12:38:45 +04:00
Anton Kaliaev 4f5492c831
add nopTxCache (Nil Object Pattern)
to better handle zero cache size
2018-06-19 11:59:07 +04:00
Anton Kaliaev 70d973016e
output msg only once during start 2018-06-19 11:40:40 +04:00
Ethan Buchman 4b2348f697 mempool: fix cache_size==0. closes #1761 2018-06-18 18:21:19 -07:00
Ethan Buchman a519825bf8 consensus: fixes #1754
* updateToState exits early if the state isn't new, which happens after
* fast syncing. This results in not sending a NewRoundStep message. The mempool
* reactor depends on PeerState, which is updated by NewRoundStep
* messages. If the peer never sends a NewRoundStep, the mempool reactor
* will think they're behind, and never forward transactions. Note this
* only happens when `create_empty_blocks = false`, because otherwise
* peers will move through the consensus state and send a NewRoundStep
* for a new step soon anyways. Simple fix is just to send the
* NewRoundStep message during updateToState even if exit early
2018-06-18 17:08:09 -07:00
Anton Kaliaev 3fa734ef5a
recheck only if there are txs left in the mempool (#1645) 2018-06-04 14:28:47 +04:00
Ethan Buchman 98b0c51b5f fix possible mempool deadlock 2018-06-03 16:11:52 -04:00
Anton Kaliaev 202a43a5af
remove TODO
no longer relevant, I guess. since ABCI only defines 0 (success) code.
2018-05-18 10:48:13 +04:00
Anton Kaliaev c9001d5a11
bound the mempool
Refs #345
2018-05-18 10:45:51 +04:00
Anton Kaliaev ae572b9038
remove extra call to Exists
Refs #345

https://github.com/tendermint/tendermint/issues/1539#issuecomment-387240606
2018-05-18 10:17:09 +04:00
Anton Kaliaev bbe1355957
log only hash, not tx itself 2018-05-17 10:00:59 +04:00
Anton Kaliaev 7c14fa820d
do not log txs at info level
BEFORE:

```
./tm-bench -c 5 -r 1000 127.0.0.1:46657
Stats          Avg       StdDev     Max
Txs/sec        1826      843        2744
Blocks/sec     1.100     0.300      2
```

AFTER:

```
./tm-bench -T 30 -c 5 -r 1000 127.0.0.1:46657
Stats          Avg       StdDev     Max
Txs/sec        6120      1970       9776
Blocks/sec     1.000     0.000      1
```
2018-05-17 10:00:59 +04:00
Ethan Buchman 1a1e4e767b check max msg size in DecodeMessage 2018-04-09 15:18:47 +03:00
Jae Kwon 35a1d747b0 Fix mempool 2018-03-31 11:51:32 +02:00
Anton Kaliaev 4242352852
stop peer on decoding error 2018-03-15 11:58:19 +04:00
Zach 2cc63069c6 rename dummy to kvstore (#1223)
* remove accidental binary

* docs: s/Dummy&dummy/KVStore&kvstore/g

* glide update to abci

* update abci import paths

* dummy begone, hello kvstore

* RequestInitChain needs genesisBytes

* glide update
2018-02-27 18:01:10 +04:00
Anton Kaliaev 0ae0155cba restore mempool memory leak tests 2018-02-19 15:34:33 -05:00
Anton Kaliaev 2a24ae90c1
fixes from Jae's review
1. remove pointer
2. add Quit() method to Service interface
2018-02-12 14:32:09 +04:00
Anton Kaliaev 84a0a1987c
comment out tests for now
https://github.com/tendermint/tendermint/pull/1173#issuecomment-363173047
2018-02-05 22:26:14 +04:00
Anton Kaliaev 11b68f1934
rewrite broadcastTxRoutine to use channels
https://play.golang.org/p/gN21yO9IRs3

```
func waitWithCancel(f func() *clist.CElement, ctx context.Context) *clist.CElement {
	el := make(chan *clist.CElement, 1)
	select {
	case el <- f():
```
will just run f() blockingly, so this doesn't change much in terms of behavior.
2018-02-05 16:36:26 +04:00
Anton Kaliaev 202d9a2c0c
fix memory leak in mempool reactor
Leaking goroutine:
```
114 @ 0x42f2bc 0x42f3ae 0x440794 0x4403b9 0x468002 0x9fe32d 0x9ff78f 0xa025ed 0x45e571
```

Explanation:
it blocks on an empty clist forever. so unless theres txs coming in,
this go routine will just sit there, holding onto the peer too.
if we're constantly reconnecting to some peer, old instances are not
garbage collected, leading to memory leak.

Fixes https://github.com/cosmos/gaia/issues/108
Previous attempt https://github.com/tendermint/tendermint/pull/1156
2018-02-05 13:52:18 +04:00
Ethan Buchman cd0fd06b0d update for sdk2 libs. need to fix kv test
NOTE we only updating for tmlibs and abci
2018-02-03 03:35:02 -05:00
Ethan Buchman 57cc8ab977
Merge pull request #1143 from tendermint/1091-race-condition
call FlushSync before calling CommitSync
2018-01-24 14:22:43 -05:00
Ethan Buchman 260affd037 docs consolidation 2018-01-23 23:46:28 -05:00