Commit Graph

234 Commits

Author SHA1 Message Date
Matthew Slipper 92343ef484 Add additional metrics (#2500)
* Add additional metrics
Continues addressing https://github.com/cosmos/cosmos-sdk/issues/2169.
* Add nop metrics to fix NPE
* Tweak buckets, code review
* Update buckets
* Update docs with new metrics
* Code review updates
2018-10-10 18:27:43 +02:00
Ethan Buchman 52e21cebcf
remove some xxx comments and the config.mempool.recheck_empty (#2505)
* remove some XXX

* config: remove Mempool.RecheckEmpty

* docs: remove recheck_empty
2018-09-30 13:28:34 -04:00
Dev Ojha 69c7aa77bc clist: speedup Next by removing defers (#2511)
This change doubles the speed of the mempool's reaping.
Before:

BenchmarkReap-8   	    5000	    365390 ns/op	  122887 B/op

After:

BenchmarkReap-8   	   10000	    158274 ns/op	  122882 B/op
2018-09-30 13:26:14 -04:00
Matthew Slipper 587116dae1 metrics: Add additional metrics to p2p and consensus (#2425)
* Add additional metrics to p2p and consensus
Partially addresses https://github.com/cosmos/cosmos-sdk/issues/2169.
* WIP
* Updates from code review
* Updates from code review
* Add instrumentation namespace to configuration
* Fix test failure
* Updates from code review
* Add quotes
* Add atomic load
* Use storeint64
* Use addInt64 in writePacketMsgTo
2018-09-25 13:14:38 +02:00
Ethan Buchman f5824bc837
Update abci and app docs (#2470)
* mempool: update some comments

* make build_c

* docs: notes about databases and WAL files

* docs: determinism. closes #1279

* docs: small note about query paths. closes #2090

* docs: gas

* docs: abci consensus params
2018-09-23 01:14:05 -04:00
Dev Ojha 111e627037 mempool: Filter new txs if they have insufficient gas (#2385)
This also refactors the prior mempool to filter to be known as
"precheck filter" and this new filter is called "postcheck filter"

This PR also fixes a bug where the precheck filter previously didn't
account for the amino overhead, which could a maliciously sized tx to
halt blocks from getting any txs in them.

* Move maxGas outside of function definition to avoid race condition
* Type filter funcs and make public
* Use helper method for post check
* Remove superfluous Filter suffix
* Move default pre/post checks into package
* Fix broken references
* Fix typos
* Expand on examples for checks
2018-09-22 02:50:06 +02:00
Anton Kaliaev 8d50bb9dad conesnsu: follow up to removing some consensus params (#2427)
* follow up to removing some consensus params Refs #2382
* change args type to int64 in state#makeParams
* make valsCount and evidenceCount ints again
* MaxEvidenceBytesPerBlock: include magic number in godoc
* [spec] creating a proposal
* test state#TxFilter
* panic if MaxDataBytes is less than 0
* fixes after review
* use amino#UvarintSize to calculate overhead
0c74291f3b/encoder.go (L85-L90)
* avoid cyclic imports
* you can do better Go, come on
* remove testdouble package
2018-09-21 11:00:36 +02:00
bradyjoestar 26aa978456 Make mempool cache a proper LRU (#2407)
Closes #2399
2018-09-19 21:40:22 +04:00
Dev Ojha 1ea64fc27f Make mempool aware of MaxGas requirement (#2360)
* Make mempool aware of MaxGas requirement

* update spec

* Add tests

* Switch GasWanted from kv store to persistent kv store

* Fix typo in test name

* switch back to using kvstore, not persistent kv store
2018-09-12 16:41:19 -04:00
Anton Kaliaev 166fd82b70
max-bytes PR follow-up (#2318)
* ReapMaxTxs: return all txs if max is negative

this mirrors ReapMaxBytes behavior

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214439950

* increase MaxAminoOverheadForBlock

tested with:

```
func TestMaxAminoOverheadForBlock(t *testing.T) {

        maxChainID := ""
        for i := 0; i < MaxChainIDLen; i++ {
                maxChainID += "𠜎"
        }

        h := Header{
                ChainID:            maxChainID,
                Height:             10,
                Time:               time.Now().UTC(),
                NumTxs:             100,
                TotalTxs:           200,
                LastBlockID:        makeBlockID(make([]byte, 20), 300, make([]byte, 20)),
                LastCommitHash:     tmhash.Sum([]byte("last_commit_hash")),
                DataHash:           tmhash.Sum([]byte("data_hash")),
                ValidatorsHash:     tmhash.Sum([]byte("validators_hash")),
                NextValidatorsHash: tmhash.Sum([]byte("next_validators_hash")),
                ConsensusHash:      tmhash.Sum([]byte("consensus_hash")),
                AppHash:            tmhash.Sum([]byte("app_hash")),
                LastResultsHash:    tmhash.Sum([]byte("last_results_hash")),
                EvidenceHash:       tmhash.Sum([]byte("evidence_hash")),
                ProposerAddress:    tmhash.Sum([]byte("proposer_address")),
        }
        b := Block{
                Header:     h,
                Data:       Data{Txs: makeTxs(10000, 100)},
                Evidence:   EvidenceData{},
                LastCommit: &Commit{},
        }

        bz, err := cdc.MarshalBinary(b)
        require.NoError(t, err)

        assert.Equal(t, MaxHeaderBytes+MaxAminoOverheadForBlock-2, len(bz)-1000000-20000-1)
}
```

* fix MaxYYY constants calculation

by using math.MaxInt64

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214444244

* pass mempool filter as an option

See https://github.com/tendermint/tendermint/pull/2184#discussion_r214445869

* fixes after Dev's comments
2018-09-04 11:46:34 +04:00
Anton Kaliaev e873fed815
calculate amino overhead on the fly 2018-08-31 16:01:22 +04:00
Anton Kaliaev d73c5cbdb1
reap max bytes from the mempool & check transaction size
See ADR 020: Limiting txs size inside a block docs/architecture/adr-020-block-size.md

Refs #2035
2018-08-31 16:01:21 +04:00
Dev Ojha bd531401a0 mempool: Store txs by hash inside of cache (#2234)
* mempool: Store txs by hash inside of cache

This allows for large cachesizes, without fear of the memory
consumption growing rapidly.

* (squash this) rename hashedTx -> txHash
2018-08-29 08:10:06 +04:00
Dev Ojha 8a1a79257e mempool: Keep cache hashmap and linked list in sync (#2188)
* mempool: Keep cache hashmap and linked list in sync

This removes bugs with the linked list being full, but hashmap empty

* address PR comments

* switch clist back to list
2018-08-13 14:42:33 +04:00
Dev Ojha 2756be5a59 libs: Remove usage of custom Fmt, in favor of fmt.Sprintf (#2199)
* libs: Remove usage of custom Fmt, in favor of fmt.Sprintf

Closes #2193

* Fix bug that was masked by custom Fmt!
2018-08-10 09:25:57 +04:00
Jun Kimura e1b9bf7c81 set capacity of txsmap (#2166) 2018-08-07 20:50:10 +04:00
Anton Kaliaev 96ae535fb8 proto3 timestamp (#2064)
This PR changes ABCI time format from int64 (Unix seconds) to WKT (WellKnownType) google.protobuf.Timestamp.

Refs #1857

Reasons:

better precision
standard DT for proto

* update Gopkg.lock
* [makefile] remove extra grep
    - go list excludes vendor by default now
* proto3 timestamp
* [docs/abci-spec] note about serialisation format
* make time non-nullable
2018-07-27 04:23:19 +02:00
Ethan Buchman 15b112e669 mempool: chan bool -> chan struct{} 2018-07-23 21:06:47 -04:00
srmo e36ce6f893 fix race condition on proposal height for published txs (#2021)
* #1920 try to fix race condition on proposal height for published txs

- related to create_empty_blocks=false
- published height for accepted tx can be wrong (too low)
- use the actual mempool height + 1 for the proposal
- expose Height() on mempool

* #1920 add initial test for mempool.Height()

- not sure how to test the lock
- can the mutex reference be of type Locker?
-- this way, we can use a "mock" of the mutex to test triggering

* #1920 use the ConsensusState height in favor of mempool

- gets rid of indirections
- doesn't need any "+1" magic

* #1920 cosmetic

- if we use cs.Height, it's enough to evaluate right before propose

* #1920 cleanup TODO and non-needed code

* #1920 add changelog entry
2018-07-23 15:34:45 +04:00
Anton Kaliaev 9120fd5d14
unexport DecodeMessage functions
Refs #646
2018-07-09 13:01:23 +04:00
Ethan Buchman d55243f0e6 fix import paths 2018-07-01 22:36:49 -04:00
Anton Kaliaev ab04201c3d
fix empty packet size calculation
Fixes #1762#issuecomment-398940107
2018-06-29 12:40:32 +04:00
Anton Kaliaev aa20c45ae9 log total when adding new txs to the mempool 2018-06-27 18:14:45 +04:00
Ethan Buchman fa3bd05d44 mempool: log hashes, not whole tx 2018-06-27 18:14:45 +04:00
Alexander Simmerl a605b66c5a
Move abci imports 2018-06-22 07:28:07 +02:00
Ethan Buchman 9cd9f3338b Merge branch 'master' into develop 2018-06-21 12:57:09 -07:00
Anton Kaliaev 205d8b8062
fixes after @xla review
- move prometheus metrics into internal packages
- *Option structs
- misc. format changes
2018-06-20 12:40:25 +04:00
Anton Kaliaev e4bb3566a0
move metrics constructors to a separate package 2018-06-20 12:40:25 +04:00
Anton Kaliaev 84812145cb
friendly apis for constructors 2018-06-20 12:40:25 +04:00
Anton Kaliaev b10b0da3fd
bundle imports 2018-06-20 12:40:11 +04:00
Anton Kaliaev 7efb73aa18
mempool size metric 2018-06-20 12:38:45 +04:00
Anton Kaliaev 4f5492c831
add nopTxCache (Nil Object Pattern)
to better handle zero cache size
2018-06-19 11:59:07 +04:00
Anton Kaliaev 70d973016e
output msg only once during start 2018-06-19 11:40:40 +04:00
Ethan Buchman 4b2348f697 mempool: fix cache_size==0. closes #1761 2018-06-18 18:21:19 -07:00
Ethan Buchman a519825bf8 consensus: fixes #1754
* updateToState exits early if the state isn't new, which happens after
* fast syncing. This results in not sending a NewRoundStep message. The mempool
* reactor depends on PeerState, which is updated by NewRoundStep
* messages. If the peer never sends a NewRoundStep, the mempool reactor
* will think they're behind, and never forward transactions. Note this
* only happens when `create_empty_blocks = false`, because otherwise
* peers will move through the consensus state and send a NewRoundStep
* for a new step soon anyways. Simple fix is just to send the
* NewRoundStep message during updateToState even if exit early
2018-06-18 17:08:09 -07:00
Anton Kaliaev 3fa734ef5a
recheck only if there are txs left in the mempool (#1645) 2018-06-04 14:28:47 +04:00
Ethan Buchman 98b0c51b5f fix possible mempool deadlock 2018-06-03 16:11:52 -04:00
Anton Kaliaev 202a43a5af
remove TODO
no longer relevant, I guess. since ABCI only defines 0 (success) code.
2018-05-18 10:48:13 +04:00
Anton Kaliaev c9001d5a11
bound the mempool
Refs #345
2018-05-18 10:45:51 +04:00
Anton Kaliaev ae572b9038
remove extra call to Exists
Refs #345

https://github.com/tendermint/tendermint/issues/1539#issuecomment-387240606
2018-05-18 10:17:09 +04:00
Anton Kaliaev bbe1355957
log only hash, not tx itself 2018-05-17 10:00:59 +04:00
Anton Kaliaev 7c14fa820d
do not log txs at info level
BEFORE:

```
./tm-bench -c 5 -r 1000 127.0.0.1:46657
Stats          Avg       StdDev     Max
Txs/sec        1826      843        2744
Blocks/sec     1.100     0.300      2
```

AFTER:

```
./tm-bench -T 30 -c 5 -r 1000 127.0.0.1:46657
Stats          Avg       StdDev     Max
Txs/sec        6120      1970       9776
Blocks/sec     1.000     0.000      1
```
2018-05-17 10:00:59 +04:00
Ethan Buchman 1a1e4e767b check max msg size in DecodeMessage 2018-04-09 15:18:47 +03:00
Jae Kwon 35a1d747b0 Fix mempool 2018-03-31 11:51:32 +02:00
Anton Kaliaev 4242352852
stop peer on decoding error 2018-03-15 11:58:19 +04:00
Zach 2cc63069c6 rename dummy to kvstore (#1223)
* remove accidental binary

* docs: s/Dummy&dummy/KVStore&kvstore/g

* glide update to abci

* update abci import paths

* dummy begone, hello kvstore

* RequestInitChain needs genesisBytes

* glide update
2018-02-27 18:01:10 +04:00
Anton Kaliaev 0ae0155cba restore mempool memory leak tests 2018-02-19 15:34:33 -05:00
Anton Kaliaev 2a24ae90c1
fixes from Jae's review
1. remove pointer
2. add Quit() method to Service interface
2018-02-12 14:32:09 +04:00
Anton Kaliaev 84a0a1987c
comment out tests for now
https://github.com/tendermint/tendermint/pull/1173#issuecomment-363173047
2018-02-05 22:26:14 +04:00
Anton Kaliaev 11b68f1934
rewrite broadcastTxRoutine to use channels
https://play.golang.org/p/gN21yO9IRs3

```
func waitWithCancel(f func() *clist.CElement, ctx context.Context) *clist.CElement {
	el := make(chan *clist.CElement, 1)
	select {
	case el <- f():
```
will just run f() blockingly, so this doesn't change much in terms of behavior.
2018-02-05 16:36:26 +04:00