From 97397e63da63e6effaeb4542187e8fb3ef022b0b Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Wed, 8 Aug 2018 13:09:51 -0400 Subject: [PATCH 01/49] simple distribution overview --- docs/spec/simple-distribution/end_block.md | 36 ++ .../future_improvements.md | 16 + docs/spec/simple-distribution/overview.md | 64 +++ docs/spec/simple-distribution/state.md | 100 +++++ docs/spec/simple-distribution/transactions.md | 399 ++++++++++++++++++ docs/spec/simple-distribution/triggers.md | 31 ++ 6 files changed, 646 insertions(+) create mode 100644 docs/spec/simple-distribution/end_block.md create mode 100644 docs/spec/simple-distribution/future_improvements.md create mode 100644 docs/spec/simple-distribution/overview.md create mode 100644 docs/spec/simple-distribution/state.md create mode 100644 docs/spec/simple-distribution/transactions.md create mode 100644 docs/spec/simple-distribution/triggers.md diff --git a/docs/spec/simple-distribution/end_block.md b/docs/spec/simple-distribution/end_block.md new file mode 100644 index 000000000..bc6847ef4 --- /dev/null +++ b/docs/spec/simple-distribution/end_block.md @@ -0,0 +1,36 @@ +# End Block + +At each endblock, the fees received are sorted to the proposer, community fund, +and global pool. When the validator is the proposer of the round, that +validator (and their delegators) receives between 1% and 5% of fee rewards, the +reserve tax is then charged, then the remainder is distributed proportionally +by voting power to all bonded validators independent of whether they voted +(social distribution). Note the social distribution is applied to proposer +validator in addition to the proposer reward. + +The amount of proposer reward is calculated from pre-commits Tendermint +messages in order to incentivize validators to wait and include additional +pre-commits in the block. All provision rewards are added to a provision reward +pool which validator holds individually +(`ValidatorDistribution.ProvisionsRewardPool`). + +``` +func SortFees(feesCollected sdk.Coins, global Global, proposer ValidatorDistribution, + sumPowerPrecommitValidators, totalBondedTokens, communityTax sdk.Dec) + + feesCollectedDec = MakeDecCoins(feesCollected) + proposerReward = feesCollectedDec * (0.01 + 0.04 + * sumPowerPrecommitValidators / totalBondedTokens) + proposer.ProposerPool += proposerReward + + communityFunding = feesCollectedDec * communityTax + global.CommunityFund += communityFunding + + poolReceived = feesCollectedDec - proposerReward - communityFunding + global.Pool += poolReceived + global.EverReceivedPool += poolReceived + global.LastReceivedPool = poolReceived + + SetValidatorDistribution(proposer) + SetGlobal(global) +``` diff --git a/docs/spec/simple-distribution/future_improvements.md b/docs/spec/simple-distribution/future_improvements.md new file mode 100644 index 000000000..954fb4d62 --- /dev/null +++ b/docs/spec/simple-distribution/future_improvements.md @@ -0,0 +1,16 @@ +## Future Improvements + +### Power Change + +Within the current implementation all power changes ever made are indefinitely stored +within the current state. In the future this state should be trimmed on an epoch basis. Delegators +which will have not withdrawn their fees will be penalized in some way, depending on what is +computationally feasible this may include: + - burning non-withdrawn fees + - requiring more expensive withdrawal costs which include proofs from archive nodes of historical state + +In addition or as an alternative it may make sense to implement a "rolling" epoch which cycles through +all the delegators in small groups (for example 5 delegators per block) and just runs the withdrawal transaction +at standard rates and takes transaction fees from the withdrawal amount. + + diff --git a/docs/spec/simple-distribution/overview.md b/docs/spec/simple-distribution/overview.md new file mode 100644 index 000000000..62cec2e76 --- /dev/null +++ b/docs/spec/simple-distribution/overview.md @@ -0,0 +1,64 @@ +# Distribution + +## Overview + +This _simple_ distribution mechanism describes a functional way to passively +distribute rewards between validator and delegators. Note that this mechanism does +not distribute funds in as precisely as active reward distribution and will therefor +be upgraded in the future. + +The mechanism operates as follows. Collected rewards are pooled globally and +divided out passively to validators and delegators. Each validator has the +opportunity to charge commission to the delegators on the rewards collected on +behalf of the delegators by the validators. Fees are paid directly into a +global reward pool, and validator proposer-reward pool. Due to the nature of +passive accounting whenever changes to parameters which affect the rate of reward +distribution occurs, withdrawal of rewards must also occur when: + + - withdrawing one must withdrawal the maximum amount they are entitled + too, leaving nothing in the pool, + - bonding, unbonding, or re-delegating tokens to an existing account a + full withdrawal of the rewards must occur (as the rules for lazy accounting + change), + - a validator chooses to change the commission on rewards, all accumulated + commission rewards must be simultaneously withdrawn. + +The above scenarios are covered in `triggers.md`. + +The distribution mechanism outlines herein is used to lazily distribute the +following rewards between validators and associated delegators: + - multi-token fees to be socially distributed, + - proposer reward pool, + - inflated atom provisions, and + - validator commission on all rewards earned by their delegators stake + +Fees are pooled within a global pool, as well as validator specific +proposer-reward pools. The mechanisms used allow for validators and delegators +to independently and lazily withdrawn their rewards. + +Within this spec + +As a part of the lazy computations, each validator and delegator holds an +accumulation term which is used to estimate what their approximate fair portion +of tokens held in the global pool is owed to them. This approximation of owed +rewards would be equivalent to the active distribution under the situation that +there was a constant flow of incoming reward tokens every block. Because this +is not the case, the approximation of owed rewards will deviate from the active +distribution based on fluctuations of incoming reward tokens as well as timing +of reward withdrawal by other delegators and validators from the reward pool. + + +## Affect on Staking + +Charging commission on Atom provisions while also allowing for Atom-provisions +to be auto-bonded (distributed directly to the validators bonded stake) is +problematic within DPoS. Fundamentally these two mechnisms are mutually +exclusive. If there are atoms commissions and auto-bonding Atoms, the portion +of Atoms the reward distribution calculation would become very large as the Atom +portion for each delegator would change each block making a withdrawal of rewards +for a delegator require a calculation for every single block since the last +withdrawal. In conclusion we can only have atom commission and unbonded atoms +provisions, or bonded atom provisions with no Atom commission, and we elect to +implement the former. Stakeholders wishing to rebond their provisions may elect +to set up a script to periodically withdraw and rebond rewards. + diff --git a/docs/spec/simple-distribution/state.md b/docs/spec/simple-distribution/state.md new file mode 100644 index 000000000..7865c085e --- /dev/null +++ b/docs/spec/simple-distribution/state.md @@ -0,0 +1,100 @@ +## State + +### Global + +All globally tracked parameters for distribution are stored within +`Global`. Rewards are collected and added to the reward pool and +distributed to validators/delegators from here. + +Note that the reward pool holds decimal coins (`DecCoins`) to allow +for fractions of coins to be received from operations like inflation. +When coins are distributed from the pool they are truncated back to +`sdk.Coins` which are non-decimal. + + - Global: `0x00 -> amino(global)` + +```golang +// coins with decimal +type DecCoins []DecCoin + +type DecCoin struct { + Amount sdk.Dec + Denom string +} + +type Global struct { + PrevBondedTokens sdk.Dec // bonded token amount for the global pool on the previous block + Adjustment sdk.Dec // global adjustment factor for lazy calculations + Pool DecCoins // funds for all validators which have yet to be withdrawn + PrevReceivedPool DecCoins // funds added to the pool on the previous block + EverReceivedPool DecCoins // total funds ever added to the pool + CommunityFund DecCoins // pool for community funds yet to be spent +} +``` + +### Validator Distribution + +Validator distribution information for the relevant validator is updated each time: + 1. delegation amount to a validator are updated, + 2. a validator successfully proposes a block and receives a reward, + 3. any delegator withdraws from a validator, or + 4. the validator withdraws it's commission. + + - ValidatorDistribution: `0x02 | ValOwnerAddr -> amino(validatorDistribution)` + +```golang +type ValidatorDistribution struct { + CommissionWithdrawalHeight int64 // last time this validator withdrew commission + Adjustment sdk.Dec // global pool adjustment factor + ProposerAdjustment DecCoins // proposer pool adjustment factor + ProposerPool DecCoins // reward pool collected from being the proposer + EverReceivedProposerReward DecCoins // all rewards ever collected from being the proposer + PrevReceivedProposerReward DecCoins // previous rewards collected from being the proposer + PrevBondedTokens sdk.Dec // bonded token amount on the previous block + PrevDelegatorShares sdk.Dec // amount of delegator shares for the validator on the previous block +} +``` + +### Delegation Distribution + +Each delegation holds multiple adjustment factors to specify its entitlement to +the rewards from a validator. `AdjustmentPool` is used to passively calculate +each bonds entitled fees from the `RewardPool`. `AdjustmentPool` is used to +passively calculate each bonds entitled fees from +`ValidatorDistribution.ProposerRewardPool` + + - DelegatorDistribution: ` 0x02 | DelegatorAddr | ValOwnerAddr -> amino(delegatorDist)` + +```golang +type DelegatorDist struct { + WithdrawalHeight int64 // last time this delegation withdrew rewards + Adjustment sdk.Dec // fee provisioning adjustment factor + AdjustmentProposer DecCoins // proposers pool adjustment factor + PrevTokens sdk.Dec // bonded tokens held by the delegation on the previous block + PrevShares sdk.Dec // delegator shares held by the delegation on the previous block +} +``` + +### Power Change + +Every instance that the voting power changes, information about the state of +the validator set during the change must be recorded as a `PowerChange` for +other validators to run through. Each power change is indexed by its block +height. + + - PowerChange: `0x03 | amino(Height) -> amino(validatorDist)` + +```golang +type PowerChange struct { + Height int64 // block height at change + ValidatorBondedTokens sdk.Dec // following used to create distribution scenarios + ValidatorDelegatorShares sdk.Dec + ValidatorDelegatorShareExRate sdk.Dec + ValidatorCommission sdk.Dec + PoolBondedTokens sdk.Dec + Global Global + ValDistr ValidatorDistribution + DelegationShares sdk.Dec + DelDistr DelegatorDistribution +} +``` diff --git a/docs/spec/simple-distribution/transactions.md b/docs/spec/simple-distribution/transactions.md new file mode 100644 index 000000000..1401c3b85 --- /dev/null +++ b/docs/spec/simple-distribution/transactions.md @@ -0,0 +1,399 @@ +# Transactions + +## TxWithdrawDelegation + +When a delegator wishes to withdraw their transaction fees it must send +`TxWithdrawDelegation`. Note that parts of this transaction logic are also +triggered each with any change in individual delegations, such as an unbond, +redelegation, or delegation of additional tokens to a specific validator. + +Each time a withdrawal is made by a recipient the adjustment term must be +modified for each block with a change in distributors shares since the time of +last withdrawal. This is accomplished by iterating over all relevant +`PowerChange`'s stored in distribution state. + + +```golang +type TxWithdrawDelegation struct { + delegatorAddr sdk.AccAddress + withdrawAddr sdk.AccAddress // address to make the withdrawal to +} + +func WithdrawDelegator(delegatorAddr, withdrawAddr sdk.AccAddress) + entitlement = GetDelegatorEntitlement(delegatorAddr) + AddCoins(withdrawAddr, totalEntitlment.TruncateDecimal()) + +func GetDelegatorEntitlement(delegatorAddr sdk.AccAddress) DecCoins + + // compile all the distribution scenarios + delegations = GetDelegations(delegatorAddr) + DelDistr = GetDelegationDistribution(delegation.DelegatorAddr, + delegation.ValidatorAddr) + pcs = GetPowerChanges(DelDistr.WithdrawalHeight) + + // update all adjustment factors for each delegation since last withdrawal + for pc = range pcs + for delegation = range delegations + DelDistr = GetDelegationDistribution(delegation.DelegatorAddr, + delegation.ValidatorAddr) + pc.ProcessPowerChangeDelegation(delegation, DelDistr) + + // collect all entitled fees + entitlement = 0 + for delegation = range delegations + global = GetGlobal() + pool = GetPool() + DelDistr = GetDelegationDistribution(delegation.DelegatorAddr, + delegation.ValidatorAddr) + ValDistr = GetValidatorDistribution(delegation.ValidatorAddr) + validator = GetValidator(delegation.ValidatorAddr) + + scenerio1 = NewDelegationFromGlobalPool(delegation, validator, + pool, global, ValDistr, DelDistr) + scenerio2 = NewDelegationFromProvisionPool(delegation, validator, + ValDistr, DelDistr) + entitlement += scenerio1.WithdrawalEntitlement() + entitlement += scenerio2.WithdrawalEntitlement() + + return entitlement + +func (pc PowerChange) ProcessPowerChangeDelegation(delegation sdk.Delegation, + DelDistr DelegationDistribution) + + // get the historical scenarios + scenario1 = pc.DelegationFromGlobalPool(delegation, DelDistr) + scenario2 = pc.DelegationFromProvisionPool(delegation, DelDistr) + + // process the adjustment factors + scenario1.UpdateAdjustmentForPowerChange(pc.Height) + scenario2.UpdateAdjustmentForPowerChange(pc.Height) +``` + +## TxWithdrawValidator + +When a validator wishes to withdraw their transaction fees it must send +`TxWithdrawDelegation`. Note that parts of this transaction logic is also +triggered each with any change in individual delegations, such as an unbond, +redelegation, or delegation of additional tokens to a specific validator. This +transaction withdraws the validators commission rewards, as well as any rewards +earning on their self-delegation. + +```golang +type TxWithdrawValidator struct { + ownerAddr sdk.AccAddress // validator address to withdraw from + withdrawAddr sdk.AccAddress // address to make the withdrawal to +} + +func WithdrawalValidator(ownerAddr, withdrawAddr sdk.AccAddress) + + // update the delegator adjustment factors and also withdrawal delegation fees + entitlement = GetDelegatorEntitlement(ownerAddr) + + // update the validator adjustment factors for commission + ValDistr = GetValidatorDistribution(ownerAddr.ValidatorAddr) + pcs = GetPowerChanges(ValDistr.CommissionWithdrawalHeight) + for pc = range pcs + pc.ProcessPowerChangeCommission() + + // withdrawal validator commission rewards + global = GetGlobal() + pool = GetPool() + ValDistr = GetValidatorDistribution(delegation.ValidatorAddr) + validator = GetValidator(delegation.ValidatorAddr) + + scenerio1 = NewCommissionFromGlobalPool(validator, + pool, global, ValDistr) + scenerio2 = CommissionFromProposerPool(validator, ValDistr) + entitlement += scenerio1.WithdrawalEntitlement() + entitlement += scenerio2.WithdrawalEntitlement() + + AddCoins(withdrawAddr, totalEntitlment.TruncateDecimal()) + +func (pc PowerChange) ProcessPowerChangeCommission() + + // get the historical scenarios + scenario1 = pc.CommissionFromGlobalPool() + scenario2 = pc.CommissionFromProposerPool() + + // process the adjustment factors + scenario1.UpdateAdjustmentForPowerChange(pc.Height) + scenario2.UpdateAdjustmentForPowerChange(pc.Height) +``` + +## Common Calculations + +### Distribution scenario + +A common form of abstracted calculations exists between validators and +delegations attempting to withdrawal their rewards, either from `Global.Pool` +or from `ValidatorDistribution.ProposerPool`. With the following interface +fulfilled the entitled fees for the various scenarios can be calculated. + +```golang +type DistributionScenario interface { + DistributorTokens() DecCoins // current tokens from distributor + DistributorCumulativeTokens() DecCoins // total tokens ever received + DistributorPrevReceivedTokens() DecCoins // last value of tokens received + DistributorShares() sdk.Dec // current shares + DistributorPrevShares() sdk.Dec // shares last block + + RecipientAdjustment() sdk.Dec + RecipientShares() sdk.Dec // current shares + RecipientPrevShares() sdk.Dec // shares last block + + ModifyAdjustments(withdrawal sdk.Dec) // proceedure to modify adjustment factors +} +``` + +#### Entitled reward from distribution scenario + +The entitlement to the distributor's tokens held can be accounted for lazily. +To begin this calculation we must determine the recipient's _simple pool_ and +_projected pool_. The simple pool represents a lazy accounting of what a +recipient's entitlement to the distributor's tokens would be if all recipients +for that distributor had static shares (equal to the current shares), and no +recipients had ever withdrawn their entitled rewards. The projected pool +represents the anticipated recipient's entitlement to the distributors tokens +based on the current blocks token input (for example fees reward received) to +the distributor, and the distributor's tokens and shares of the previous block +assuming that neither had changed in the current block. Using the simple and +projected pools we can determine all cumulative changes which have taken place +outside of the recipient and adjust the recipient's _adjustment factor_ to +account for these changes and ultimately keep track of the correct entitlement +to the distributors tokens. + +``` +func (d DistributionScenario) RecipientCount(height int64) sdk.Dec + return v.RecipientShares() * height + +func (d DistributionScenario) GlobalCount(height int64) sdk.Dec + return d.DistributorShares() * height + +func (d DistributionScenario) SimplePool() DecCoins + return d.RecipientCount() / d.GlobalCount() * d.DistributorCumulativeTokens + +func (d DistributionScenario) ProjectedPool(height int64) DecCoins + return d.RecipientPrevShares() * (height-1) + / (d.DistributorPrevShares() * (height-1)) + * d.DistributorCumulativeTokens + + d.RecipientShares() / d.DistributorShares() + * d.DistributorPrevReceivedTokens() +``` + +The `DistributionScenario` _adjustment_ terms account for changes in +recipient/distributor shares and recipient withdrawals. The adjustment factor +must be modified whenever the recipient withdraws from the distributor or the +distributor's/recipient's shares are changed. + - When the shares of the recipient is changed the adjustment factor is + increased/decreased by the difference between the _simple_ and _projected_ + pools. In other words, the cumulative difference in the shares if the shares + has been the new shares as opposed to the old shares for the entire duration of + the blockchain up the previous block. + - When a recipient makes a withdrawal the adjustment factor is increased by the + withdrawal amount. + +``` +func (d DistributionScenario) UpdateAdjustmentForPowerChange(height int64) + simplePool = d.SimplePool() + projectedPool = d.ProjectedPool(height) + AdjustmentChange = simplePool - projectedPool + if AdjustmentChange > 0 + d.ModifyAdjustments(AdjustmentChange) + +func (d DistributionScenario) WithdrawalEntitlement() DecCoins + entitlement = d.SimplePool() - d.RecipientAdjustment() + d.ModifyAdjustments(entitlement) + return entitlement +``` + +### Distribution scenarios + +Note that the distribution scenario structures are found in `state.md`. + +#### Delegation's entitlement to Global.Pool + +For delegations (including validator's self-delegation) all fees from fee pool +are subject to commission rate from the owner of the validator. The global +shares should be taken as true number of global bonded shares. The recipients +shares should be taken as the bonded tokens less the validator's commission. + +``` +type DelegationFromGlobalPool struct { + DelegationShares sdk.Dec + ValidatorCommission sdk.Dec + ValidatorBondedTokens sdk.Dec + ValidatorDelegatorShareExRate sdk.Dec + PoolBondedTokens sdk.Dec + Global Global + ValDistr ValidatorDistribution + DelDistr DelegatorDistribution +} + +func (d DelegationFromGlobalPool) DistributorTokens() DecCoins + return d.Global.Pool + +func (d DelegationFromGlobalPool) DistributorCumulativeTokens() DecCoins + return d.Global.EverReceivedPool + +func (d DelegationFromGlobalPool) DistributorPrevReceivedTokens() DecCoins + return d.Global.PrevReceivedPool + +func (d DelegationFromGlobalPool) DistributorShares() sdk.Dec + return d.PoolBondedTokens + +func (d DelegationFromGlobalPool) DistributorPrevShares() sdk.Dec + return d.Global.PrevBondedTokens + +func (d DelegationFromGlobalPool) RecipientShares() sdk.Dec + return d.DelegationShares * d.ValidatorDelegatorShareExRate() * + d.ValidatorBondedTokens() * (1 - d.ValidatorCommission) + +func (d DelegationFromGlobalPool) RecipientPrevShares() sdk.Dec + return d.DelDistr.PrevTokens + +func (d DelegationFromGlobalPool) RecipientAdjustment() sdk.Dec + return d.DelDistr.Adjustment + +func (d DelegationFromGlobalPool) ModifyAdjustments(withdrawal sdk.Dec) + d.ValDistr.Adjustment += withdrawal + d.DelDistr.Adjustment += withdrawal + d.global.Adjustment += withdrawal + SetValidatorDistribution(d.ValDistr) + SetDelegatorDistribution(d.DelDistr) + SetGlobal(d.Global) +``` + +#### Delegation's entitlement to ValidatorDistribution.ProposerPool + +Delegations (including validator's self-delegation) are still subject +commission on the rewards gained from the proposer pool. Global shares in this +context is actually the validators total delegations shares. The recipient's +shares is taken as the effective delegation shares less the validator's +commission. + +``` +type DelegationFromProposerPool struct { + DelegationShares sdk.Dec + ValidatorCommission sdk.Dec + ValidatorDelegatorShares sdk.Dec + ValDistr ValidatorDistribution + DelDistr DelegatorDistribution +} + +func (d DelegationFromProposerPool) DistributorTokens() DecCoins + return d.ValDistr.ProposerPool + +func (d DelegationFromProposerPool) DistributorCumulativeTokens() DecCoins + return d.ValDistr.EverReceivedProposerReward + +func (d DelegationFromProposerPool) DistributorPrevReceivedTokens() DecCoins + return d.ValDistr.PrevReceivedProposerReward + +func (d DelegationFromProposerPool) DistributorShares() sdk.Dec + return d.ValidatorDelegatorShares + +func (d DelegationFromProposerPool) DistributorPrevShares() sdk.Dec + return d.ValDistr.PrevDelegatorShares + +func (d DelegationFromProposerPool) RecipientShares() sdk.Dec + return d.DelegationShares * (1 - d.ValidatorCommission) + +func (d DelegationFromProposerPool) RecipientPrevShares() sdk.Dec + return d.DelDistr.PrevShares + +func (d DelegationFromProposerPool) RecipientAdjustment() sdk.Dec + return d.DelDistr.AdjustmentProposer + +func (d DelegationFromProposerPool) ModifyAdjustments(withdrawal sdk.Dec) + d.ValDistr.AdjustmentProposer += withdrawal + d.DelDistr.AdjustmentProposer += withdrawal + SetValidatorDistribution(d.ValDistr) + SetDelegatorDistribution(d.DelDistr) +``` + +#### Validators's commission entitlement to Global.Pool + +Similar to a delegator's entitlement, but with recipient shares based on the +commission portion of bonded tokens. + +``` +type CommissionFromGlobalPool struct { + ValidatorBondedTokens sdk.Dec + ValidatorCommission sdk.Dec + PoolBondedTokens sdk.Dec + Global Global + ValDistr ValidatorDistribution +} + +func (c CommissionFromGlobalPool) DistributorTokens() DecCoins + return c.Global.Pool + +func (c CommissionFromGlobalPool) DistributorCumulativeTokens() DecCoins + return c.Global.EverReceivedPool + +func (c CommissionFromGlobalPool) DistributorPrevReceivedTokens() DecCoins + return c.Global.PrevReceivedPool + +func (c CommissionFromGlobalPool) DistributorShares() sdk.Dec + return c.PoolBondedTokens + +func (c CommissionFromGlobalPool) DistributorPrevShares() sdk.Dec + return c.Global.PrevBondedTokens + +func (c CommissionFromGlobalPool) RecipientShares() sdk.Dec + return c.ValidatorBondedTokens() * c.ValidatorCommission + +func (c CommissionFromGlobalPool) RecipientPrevShares() sdk.Dec + return c.ValDistr.PrevBondedTokens * c.ValidatorCommission + +func (c CommissionFromGlobalPool) RecipientAdjustment() sdk.Dec + return c.ValDistr.Adjustment + +func (c CommissionFromGlobalPool) ModifyAdjustments(withdrawal sdk.Dec) + c.ValDistr.Adjustment += withdrawal + c.Global.Adjustment += withdrawal + SetValidatorDistribution(c.ValDistr) + SetGlobal(c.Global) +``` + +#### Validators's commission entitlement to ValidatorDistribution.ProposerPool + +Similar to a delegators entitlement to the proposer pool, but with recipient +shares based on the commission portion of the total delegator shares. + +``` +type CommissionFromProposerPool struct { + ValidatorDelegatorShares sdk.Dec + ValidatorCommission sdk.Dec + ValDistr ValidatorDistribution +} + +func (c CommissionFromProposerPool) DistributorTokens() DecCoins + return c.ValDistr.ProposerPool + +func (c CommissionFromProposerPool) DistributorCumulativeTokens() DecCoins + return c.ValDistr.EverReceivedProposerReward + +func (c CommissionFromProposerPool) DistributorPrevReceivedTokens() DecCoins + return c.ValDistr.PrevReceivedProposerReward + +func (c CommissionFromProposerPool) DistributorShares() sdk.Dec + return c.ValidatorDelegatorShares + +func (c CommissionFromProposerPool) DistributorPrevShares() sdk.Dec + return c.ValDistr.PrevDelegatorShares + +func (c CommissionFromProposerPool) RecipientShares() sdk.Dec + return c.ValidatorDelegatorShares * (c.ValidatorCommission) + +func (c CommissionFromProposerPool) RecipientPrevShares() sdk.Dec + return c.ValDistr.PrevDelegatorShares * (c.ValidatorCommission) + +func (c CommissionFromProposerPool) RecipientAdjustment() sdk.Dec + return c.ValDistr.AdjustmentProposer + +func (c CommissionFromProposerPool) ModifyAdjustments(withdrawal sdk.Dec) + c.ValDistr.AdjustmentProposer += withdrawal + SetValidatorDistribution(c.ValDistr) +``` + diff --git a/docs/spec/simple-distribution/triggers.md b/docs/spec/simple-distribution/triggers.md new file mode 100644 index 000000000..8800609a4 --- /dev/null +++ b/docs/spec/simple-distribution/triggers.md @@ -0,0 +1,31 @@ +# Triggers + +## Create validator distribution + + - triggered-by: validator entering bonded validator group (`stake.bondValidator()`) + +Whenever a new validator is added to the Tendermint validator set they are +entitled to begin earning rewards of atom provisions and fees. At this point +`ValidatorDistribution.Pool()` must be zero (as the validator has not yet +earned any rewards) meaning that the initial value for `validator.Adjustment` +must be set to the value of `validator.SimplePool()` for the height which the +validator is added on the validator set. + +## Create or modify delegation distribution + + - triggered-by: `stake.TxDelegate`, `stake.TxBeginRedelegate`, `stake.TxBeginUnbonding` + +The pool of a new delegator bond will be 0 for the height at which the bond was +added. This is achieved by setting `DelegationDistribution.WithdrawalHeight` to +the height which the bond was added. Additionally the `AdjustmentPool` and +`AdjustmentProposerPool` must be set to the equivalent values of +`DelegationDistribution.SimplePool()` and +`DelegationDistribution.SimpleProposerPool()` for the height of delegation. + +## Commission rate change + + - triggered-by: `stake.TxEditValidator` + +If a validator changes its commission rate, all commission on fees must be +simultaneously withdrawn using the transaction `TxWithdrawValidator` + From eb6f51ed459ad49073f907199b118ca23e0acffc Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Wed, 8 Aug 2018 13:28:30 -0400 Subject: [PATCH 02/49] piggy bank state --- docs/spec/simple-distribution/state.md | 56 ++++++-------------------- 1 file changed, 12 insertions(+), 44 deletions(-) diff --git a/docs/spec/simple-distribution/state.md b/docs/spec/simple-distribution/state.md index 7865c085e..757287004 100644 --- a/docs/spec/simple-distribution/state.md +++ b/docs/spec/simple-distribution/state.md @@ -23,11 +23,8 @@ type DecCoin struct { } type Global struct { - PrevBondedTokens sdk.Dec // bonded token amount for the global pool on the previous block - Adjustment sdk.Dec // global adjustment factor for lazy calculations + Accum sdk.Dec // global accumulation factor for lazy calculations Pool DecCoins // funds for all validators which have yet to be withdrawn - PrevReceivedPool DecCoins // funds added to the pool on the previous block - EverReceivedPool DecCoins // total funds ever added to the pool CommunityFund DecCoins // pool for community funds yet to be spent } ``` @@ -44,57 +41,28 @@ Validator distribution information for the relevant validator is updated each ti ```golang type ValidatorDistribution struct { - CommissionWithdrawalHeight int64 // last time this validator withdrew commission - Adjustment sdk.Dec // global pool adjustment factor - ProposerAdjustment DecCoins // proposer pool adjustment factor - ProposerPool DecCoins // reward pool collected from being the proposer - EverReceivedProposerReward DecCoins // all rewards ever collected from being the proposer - PrevReceivedProposerReward DecCoins // previous rewards collected from being the proposer - PrevBondedTokens sdk.Dec // bonded token amount on the previous block - PrevDelegatorShares sdk.Dec // amount of delegator shares for the validator on the previous block + CommissionWithdrawalHeight int64 // last time this validator withdrew commission + Accum sdk.Dec // global pool accumulation factor + ProposerAccum sdk.Dec // proposer pool accumulation factor + ProposerPool DecCoins // reward pool collected from being the proposer } ``` ### Delegation Distribution -Each delegation holds multiple adjustment factors to specify its entitlement to -the rewards from a validator. `AdjustmentPool` is used to passively calculate -each bonds entitled fees from the `RewardPool`. `AdjustmentPool` is used to -passively calculate each bonds entitled fees from +Each delegation holds multiple accumulation factors to specify its entitlement to +the rewards from a validator. `Accum` is used to passively calculate +each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to +passively calculate each bonds entitled rewards from `ValidatorDistribution.ProposerRewardPool` - DelegatorDistribution: ` 0x02 | DelegatorAddr | ValOwnerAddr -> amino(delegatorDist)` ```golang type DelegatorDist struct { - WithdrawalHeight int64 // last time this delegation withdrew rewards - Adjustment sdk.Dec // fee provisioning adjustment factor - AdjustmentProposer DecCoins // proposers pool adjustment factor - PrevTokens sdk.Dec // bonded tokens held by the delegation on the previous block - PrevShares sdk.Dec // delegator shares held by the delegation on the previous block + WithdrawalHeight int64 // last time this delegation withdrew rewards + Accum sdk.Dec // reward provisioning accumulation factor + AccumProposer sdk.Dec // proposers pool accumulation factor } ``` -### Power Change - -Every instance that the voting power changes, information about the state of -the validator set during the change must be recorded as a `PowerChange` for -other validators to run through. Each power change is indexed by its block -height. - - - PowerChange: `0x03 | amino(Height) -> amino(validatorDist)` - -```golang -type PowerChange struct { - Height int64 // block height at change - ValidatorBondedTokens sdk.Dec // following used to create distribution scenarios - ValidatorDelegatorShares sdk.Dec - ValidatorDelegatorShareExRate sdk.Dec - ValidatorCommission sdk.Dec - PoolBondedTokens sdk.Dec - Global Global - ValDistr ValidatorDistribution - DelegationShares sdk.Dec - DelDistr DelegatorDistribution -} -``` From 2d613cefe7da94d9e9bdd30314e914c1447c499f Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Wed, 8 Aug 2018 14:24:38 -0400 Subject: [PATCH 03/49] move lamborghini distribution to subfolder --- .../WIP-lamborghini-distribution/README.md | 2 + .../end_block.md | 0 .../example_sheet/distribution.xlsx | Bin .../future_improvements.md | 0 .../WIP-lamborghini-distribution/overview.md | 54 ++++++++++ .../WIP-lamborghini-distribution/state.md | 100 ++++++++++++++++++ .../transactions.md | 0 .../WIP-lamborghini-distribution}/triggers.md | 0 docs/spec/distribution/overview.md | 54 ++++++---- docs/spec/distribution/state.md | 56 +++------- docs/spec/simple-distribution/overview.md | 64 ----------- docs/spec/simple-distribution/state.md | 68 ------------ 12 files changed, 200 insertions(+), 198 deletions(-) create mode 100644 docs/spec/distribution/WIP-lamborghini-distribution/README.md rename docs/spec/{simple-distribution => distribution/WIP-lamborghini-distribution}/end_block.md (100%) rename docs/spec/distribution/{ => WIP-lamborghini-distribution}/example_sheet/distribution.xlsx (100%) rename docs/spec/{simple-distribution => distribution/WIP-lamborghini-distribution}/future_improvements.md (100%) create mode 100644 docs/spec/distribution/WIP-lamborghini-distribution/overview.md create mode 100644 docs/spec/distribution/WIP-lamborghini-distribution/state.md rename docs/spec/{simple-distribution => distribution/WIP-lamborghini-distribution}/transactions.md (100%) rename docs/spec/{simple-distribution => distribution/WIP-lamborghini-distribution}/triggers.md (100%) delete mode 100644 docs/spec/simple-distribution/overview.md delete mode 100644 docs/spec/simple-distribution/state.md diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/README.md b/docs/spec/distribution/WIP-lamborghini-distribution/README.md new file mode 100644 index 000000000..c09042f7f --- /dev/null +++ b/docs/spec/distribution/WIP-lamborghini-distribution/README.md @@ -0,0 +1,2 @@ +Please note that this folder is a WIP specification for an advanced fee distribution +mechanism which is not set to be implemented. diff --git a/docs/spec/simple-distribution/end_block.md b/docs/spec/distribution/WIP-lamborghini-distribution/end_block.md similarity index 100% rename from docs/spec/simple-distribution/end_block.md rename to docs/spec/distribution/WIP-lamborghini-distribution/end_block.md diff --git a/docs/spec/distribution/example_sheet/distribution.xlsx b/docs/spec/distribution/WIP-lamborghini-distribution/example_sheet/distribution.xlsx similarity index 100% rename from docs/spec/distribution/example_sheet/distribution.xlsx rename to docs/spec/distribution/WIP-lamborghini-distribution/example_sheet/distribution.xlsx diff --git a/docs/spec/simple-distribution/future_improvements.md b/docs/spec/distribution/WIP-lamborghini-distribution/future_improvements.md similarity index 100% rename from docs/spec/simple-distribution/future_improvements.md rename to docs/spec/distribution/WIP-lamborghini-distribution/future_improvements.md diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/overview.md b/docs/spec/distribution/WIP-lamborghini-distribution/overview.md new file mode 100644 index 000000000..5e28e8b3b --- /dev/null +++ b/docs/spec/distribution/WIP-lamborghini-distribution/overview.md @@ -0,0 +1,54 @@ +# Distribution + +## Overview + +Collected fees are pooled globally and divided out passively to validators and +delegators. Each validator has the opportunity to charge commission to the +delegators on the fees collected on behalf of the delegators by the validators. +Fees are paid directly into a global fee pool, and validator proposer-reward +pool. Due to the nature of passive accounting whenever changes to parameters +which affect the rate of fee distribution occurs, withdrawal of fees must also +occur when: + + - withdrawing one must withdrawal the maximum amount they are entitled + too, leaving nothing in the pool, + - bonding, unbonding, or re-delegating tokens to an existing account a + full withdrawal of the fees must occur (as the rules for lazy accounting + change), + - a validator chooses to change the commission on fees, all accumulated + commission fees must be simultaneously withdrawn. + +The above scenarios are covered in `triggers.md`. + +The distribution mechanism outlines herein is used to lazily distribute the +following between validators and associated delegators: + - multi-token fees to be socially distributed, + - proposer reward pool, + - inflated atom provisions, and + - validator commission on all rewards earned by their delegators stake + +Fees are pooled within a global pool, as well as validator specific +proposer-reward pools. The mechanisms used allow for validators and delegators +to independently and lazily withdrawn their rewards. As a part of the lazy +computations adjustment factors must be maintained for each validator and +delegator to determine the true proportion of fees in each pool which they are +entitled too. Adjustment factors are updated every time a validator or +delegator's voting power changes. Validators and delegators must withdraw all +fees they are entitled too before they can change their portion of bonded +Atoms. + +## Affect on Staking + + +Charging commission on Atom provisions while also allowing for Atom-provisions +to be auto-bonded (distributed directly to the validators bonded stake) is +problematic within DPoS. Fundamentally these two mechnisms are mutually +exclusive. If there are atoms commissions and auto-bonding Atoms, the portion +of Atoms the fee distribution calculation would become very large as the Atom +portion for each delegator would change each block making a withdrawal of fees +for a delegator require a calculation for every single block since the last +withdrawal. In conclusion we can only have atom commission and unbonded atoms +provisions, or bonded atom provisions with no Atom commission, and we elect to +implement the former. Stakeholders wishing to rebond their provisions may elect +to set up a script to periodically withdraw and rebond fees. + diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/state.md b/docs/spec/distribution/WIP-lamborghini-distribution/state.md new file mode 100644 index 000000000..7865c085e --- /dev/null +++ b/docs/spec/distribution/WIP-lamborghini-distribution/state.md @@ -0,0 +1,100 @@ +## State + +### Global + +All globally tracked parameters for distribution are stored within +`Global`. Rewards are collected and added to the reward pool and +distributed to validators/delegators from here. + +Note that the reward pool holds decimal coins (`DecCoins`) to allow +for fractions of coins to be received from operations like inflation. +When coins are distributed from the pool they are truncated back to +`sdk.Coins` which are non-decimal. + + - Global: `0x00 -> amino(global)` + +```golang +// coins with decimal +type DecCoins []DecCoin + +type DecCoin struct { + Amount sdk.Dec + Denom string +} + +type Global struct { + PrevBondedTokens sdk.Dec // bonded token amount for the global pool on the previous block + Adjustment sdk.Dec // global adjustment factor for lazy calculations + Pool DecCoins // funds for all validators which have yet to be withdrawn + PrevReceivedPool DecCoins // funds added to the pool on the previous block + EverReceivedPool DecCoins // total funds ever added to the pool + CommunityFund DecCoins // pool for community funds yet to be spent +} +``` + +### Validator Distribution + +Validator distribution information for the relevant validator is updated each time: + 1. delegation amount to a validator are updated, + 2. a validator successfully proposes a block and receives a reward, + 3. any delegator withdraws from a validator, or + 4. the validator withdraws it's commission. + + - ValidatorDistribution: `0x02 | ValOwnerAddr -> amino(validatorDistribution)` + +```golang +type ValidatorDistribution struct { + CommissionWithdrawalHeight int64 // last time this validator withdrew commission + Adjustment sdk.Dec // global pool adjustment factor + ProposerAdjustment DecCoins // proposer pool adjustment factor + ProposerPool DecCoins // reward pool collected from being the proposer + EverReceivedProposerReward DecCoins // all rewards ever collected from being the proposer + PrevReceivedProposerReward DecCoins // previous rewards collected from being the proposer + PrevBondedTokens sdk.Dec // bonded token amount on the previous block + PrevDelegatorShares sdk.Dec // amount of delegator shares for the validator on the previous block +} +``` + +### Delegation Distribution + +Each delegation holds multiple adjustment factors to specify its entitlement to +the rewards from a validator. `AdjustmentPool` is used to passively calculate +each bonds entitled fees from the `RewardPool`. `AdjustmentPool` is used to +passively calculate each bonds entitled fees from +`ValidatorDistribution.ProposerRewardPool` + + - DelegatorDistribution: ` 0x02 | DelegatorAddr | ValOwnerAddr -> amino(delegatorDist)` + +```golang +type DelegatorDist struct { + WithdrawalHeight int64 // last time this delegation withdrew rewards + Adjustment sdk.Dec // fee provisioning adjustment factor + AdjustmentProposer DecCoins // proposers pool adjustment factor + PrevTokens sdk.Dec // bonded tokens held by the delegation on the previous block + PrevShares sdk.Dec // delegator shares held by the delegation on the previous block +} +``` + +### Power Change + +Every instance that the voting power changes, information about the state of +the validator set during the change must be recorded as a `PowerChange` for +other validators to run through. Each power change is indexed by its block +height. + + - PowerChange: `0x03 | amino(Height) -> amino(validatorDist)` + +```golang +type PowerChange struct { + Height int64 // block height at change + ValidatorBondedTokens sdk.Dec // following used to create distribution scenarios + ValidatorDelegatorShares sdk.Dec + ValidatorDelegatorShareExRate sdk.Dec + ValidatorCommission sdk.Dec + PoolBondedTokens sdk.Dec + Global Global + ValDistr ValidatorDistribution + DelegationShares sdk.Dec + DelDistr DelegatorDistribution +} +``` diff --git a/docs/spec/simple-distribution/transactions.md b/docs/spec/distribution/WIP-lamborghini-distribution/transactions.md similarity index 100% rename from docs/spec/simple-distribution/transactions.md rename to docs/spec/distribution/WIP-lamborghini-distribution/transactions.md diff --git a/docs/spec/simple-distribution/triggers.md b/docs/spec/distribution/WIP-lamborghini-distribution/triggers.md similarity index 100% rename from docs/spec/simple-distribution/triggers.md rename to docs/spec/distribution/WIP-lamborghini-distribution/triggers.md diff --git a/docs/spec/distribution/overview.md b/docs/spec/distribution/overview.md index 5e28e8b3b..62cec2e76 100644 --- a/docs/spec/distribution/overview.md +++ b/docs/spec/distribution/overview.md @@ -2,26 +2,31 @@ ## Overview -Collected fees are pooled globally and divided out passively to validators and -delegators. Each validator has the opportunity to charge commission to the -delegators on the fees collected on behalf of the delegators by the validators. -Fees are paid directly into a global fee pool, and validator proposer-reward -pool. Due to the nature of passive accounting whenever changes to parameters -which affect the rate of fee distribution occurs, withdrawal of fees must also -occur when: +This _simple_ distribution mechanism describes a functional way to passively +distribute rewards between validator and delegators. Note that this mechanism does +not distribute funds in as precisely as active reward distribution and will therefor +be upgraded in the future. + +The mechanism operates as follows. Collected rewards are pooled globally and +divided out passively to validators and delegators. Each validator has the +opportunity to charge commission to the delegators on the rewards collected on +behalf of the delegators by the validators. Fees are paid directly into a +global reward pool, and validator proposer-reward pool. Due to the nature of +passive accounting whenever changes to parameters which affect the rate of reward +distribution occurs, withdrawal of rewards must also occur when: - withdrawing one must withdrawal the maximum amount they are entitled too, leaving nothing in the pool, - bonding, unbonding, or re-delegating tokens to an existing account a - full withdrawal of the fees must occur (as the rules for lazy accounting + full withdrawal of the rewards must occur (as the rules for lazy accounting change), - - a validator chooses to change the commission on fees, all accumulated - commission fees must be simultaneously withdrawn. + - a validator chooses to change the commission on rewards, all accumulated + commission rewards must be simultaneously withdrawn. The above scenarios are covered in `triggers.md`. The distribution mechanism outlines herein is used to lazily distribute the -following between validators and associated delegators: +following rewards between validators and associated delegators: - multi-token fees to be socially distributed, - proposer reward pool, - inflated atom provisions, and @@ -29,26 +34,31 @@ following between validators and associated delegators: Fees are pooled within a global pool, as well as validator specific proposer-reward pools. The mechanisms used allow for validators and delegators -to independently and lazily withdrawn their rewards. As a part of the lazy -computations adjustment factors must be maintained for each validator and -delegator to determine the true proportion of fees in each pool which they are -entitled too. Adjustment factors are updated every time a validator or -delegator's voting power changes. Validators and delegators must withdraw all -fees they are entitled too before they can change their portion of bonded -Atoms. +to independently and lazily withdrawn their rewards. + +Within this spec + +As a part of the lazy computations, each validator and delegator holds an +accumulation term which is used to estimate what their approximate fair portion +of tokens held in the global pool is owed to them. This approximation of owed +rewards would be equivalent to the active distribution under the situation that +there was a constant flow of incoming reward tokens every block. Because this +is not the case, the approximation of owed rewards will deviate from the active +distribution based on fluctuations of incoming reward tokens as well as timing +of reward withdrawal by other delegators and validators from the reward pool. + ## Affect on Staking - Charging commission on Atom provisions while also allowing for Atom-provisions to be auto-bonded (distributed directly to the validators bonded stake) is problematic within DPoS. Fundamentally these two mechnisms are mutually exclusive. If there are atoms commissions and auto-bonding Atoms, the portion -of Atoms the fee distribution calculation would become very large as the Atom -portion for each delegator would change each block making a withdrawal of fees +of Atoms the reward distribution calculation would become very large as the Atom +portion for each delegator would change each block making a withdrawal of rewards for a delegator require a calculation for every single block since the last withdrawal. In conclusion we can only have atom commission and unbonded atoms provisions, or bonded atom provisions with no Atom commission, and we elect to implement the former. Stakeholders wishing to rebond their provisions may elect -to set up a script to periodically withdraw and rebond fees. +to set up a script to periodically withdraw and rebond rewards. diff --git a/docs/spec/distribution/state.md b/docs/spec/distribution/state.md index 7865c085e..757287004 100644 --- a/docs/spec/distribution/state.md +++ b/docs/spec/distribution/state.md @@ -23,11 +23,8 @@ type DecCoin struct { } type Global struct { - PrevBondedTokens sdk.Dec // bonded token amount for the global pool on the previous block - Adjustment sdk.Dec // global adjustment factor for lazy calculations + Accum sdk.Dec // global accumulation factor for lazy calculations Pool DecCoins // funds for all validators which have yet to be withdrawn - PrevReceivedPool DecCoins // funds added to the pool on the previous block - EverReceivedPool DecCoins // total funds ever added to the pool CommunityFund DecCoins // pool for community funds yet to be spent } ``` @@ -44,57 +41,28 @@ Validator distribution information for the relevant validator is updated each ti ```golang type ValidatorDistribution struct { - CommissionWithdrawalHeight int64 // last time this validator withdrew commission - Adjustment sdk.Dec // global pool adjustment factor - ProposerAdjustment DecCoins // proposer pool adjustment factor - ProposerPool DecCoins // reward pool collected from being the proposer - EverReceivedProposerReward DecCoins // all rewards ever collected from being the proposer - PrevReceivedProposerReward DecCoins // previous rewards collected from being the proposer - PrevBondedTokens sdk.Dec // bonded token amount on the previous block - PrevDelegatorShares sdk.Dec // amount of delegator shares for the validator on the previous block + CommissionWithdrawalHeight int64 // last time this validator withdrew commission + Accum sdk.Dec // global pool accumulation factor + ProposerAccum sdk.Dec // proposer pool accumulation factor + ProposerPool DecCoins // reward pool collected from being the proposer } ``` ### Delegation Distribution -Each delegation holds multiple adjustment factors to specify its entitlement to -the rewards from a validator. `AdjustmentPool` is used to passively calculate -each bonds entitled fees from the `RewardPool`. `AdjustmentPool` is used to -passively calculate each bonds entitled fees from +Each delegation holds multiple accumulation factors to specify its entitlement to +the rewards from a validator. `Accum` is used to passively calculate +each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to +passively calculate each bonds entitled rewards from `ValidatorDistribution.ProposerRewardPool` - DelegatorDistribution: ` 0x02 | DelegatorAddr | ValOwnerAddr -> amino(delegatorDist)` ```golang type DelegatorDist struct { - WithdrawalHeight int64 // last time this delegation withdrew rewards - Adjustment sdk.Dec // fee provisioning adjustment factor - AdjustmentProposer DecCoins // proposers pool adjustment factor - PrevTokens sdk.Dec // bonded tokens held by the delegation on the previous block - PrevShares sdk.Dec // delegator shares held by the delegation on the previous block + WithdrawalHeight int64 // last time this delegation withdrew rewards + Accum sdk.Dec // reward provisioning accumulation factor + AccumProposer sdk.Dec // proposers pool accumulation factor } ``` -### Power Change - -Every instance that the voting power changes, information about the state of -the validator set during the change must be recorded as a `PowerChange` for -other validators to run through. Each power change is indexed by its block -height. - - - PowerChange: `0x03 | amino(Height) -> amino(validatorDist)` - -```golang -type PowerChange struct { - Height int64 // block height at change - ValidatorBondedTokens sdk.Dec // following used to create distribution scenarios - ValidatorDelegatorShares sdk.Dec - ValidatorDelegatorShareExRate sdk.Dec - ValidatorCommission sdk.Dec - PoolBondedTokens sdk.Dec - Global Global - ValDistr ValidatorDistribution - DelegationShares sdk.Dec - DelDistr DelegatorDistribution -} -``` diff --git a/docs/spec/simple-distribution/overview.md b/docs/spec/simple-distribution/overview.md deleted file mode 100644 index 62cec2e76..000000000 --- a/docs/spec/simple-distribution/overview.md +++ /dev/null @@ -1,64 +0,0 @@ -# Distribution - -## Overview - -This _simple_ distribution mechanism describes a functional way to passively -distribute rewards between validator and delegators. Note that this mechanism does -not distribute funds in as precisely as active reward distribution and will therefor -be upgraded in the future. - -The mechanism operates as follows. Collected rewards are pooled globally and -divided out passively to validators and delegators. Each validator has the -opportunity to charge commission to the delegators on the rewards collected on -behalf of the delegators by the validators. Fees are paid directly into a -global reward pool, and validator proposer-reward pool. Due to the nature of -passive accounting whenever changes to parameters which affect the rate of reward -distribution occurs, withdrawal of rewards must also occur when: - - - withdrawing one must withdrawal the maximum amount they are entitled - too, leaving nothing in the pool, - - bonding, unbonding, or re-delegating tokens to an existing account a - full withdrawal of the rewards must occur (as the rules for lazy accounting - change), - - a validator chooses to change the commission on rewards, all accumulated - commission rewards must be simultaneously withdrawn. - -The above scenarios are covered in `triggers.md`. - -The distribution mechanism outlines herein is used to lazily distribute the -following rewards between validators and associated delegators: - - multi-token fees to be socially distributed, - - proposer reward pool, - - inflated atom provisions, and - - validator commission on all rewards earned by their delegators stake - -Fees are pooled within a global pool, as well as validator specific -proposer-reward pools. The mechanisms used allow for validators and delegators -to independently and lazily withdrawn their rewards. - -Within this spec - -As a part of the lazy computations, each validator and delegator holds an -accumulation term which is used to estimate what their approximate fair portion -of tokens held in the global pool is owed to them. This approximation of owed -rewards would be equivalent to the active distribution under the situation that -there was a constant flow of incoming reward tokens every block. Because this -is not the case, the approximation of owed rewards will deviate from the active -distribution based on fluctuations of incoming reward tokens as well as timing -of reward withdrawal by other delegators and validators from the reward pool. - - -## Affect on Staking - -Charging commission on Atom provisions while also allowing for Atom-provisions -to be auto-bonded (distributed directly to the validators bonded stake) is -problematic within DPoS. Fundamentally these two mechnisms are mutually -exclusive. If there are atoms commissions and auto-bonding Atoms, the portion -of Atoms the reward distribution calculation would become very large as the Atom -portion for each delegator would change each block making a withdrawal of rewards -for a delegator require a calculation for every single block since the last -withdrawal. In conclusion we can only have atom commission and unbonded atoms -provisions, or bonded atom provisions with no Atom commission, and we elect to -implement the former. Stakeholders wishing to rebond their provisions may elect -to set up a script to periodically withdraw and rebond rewards. - diff --git a/docs/spec/simple-distribution/state.md b/docs/spec/simple-distribution/state.md deleted file mode 100644 index 757287004..000000000 --- a/docs/spec/simple-distribution/state.md +++ /dev/null @@ -1,68 +0,0 @@ -## State - -### Global - -All globally tracked parameters for distribution are stored within -`Global`. Rewards are collected and added to the reward pool and -distributed to validators/delegators from here. - -Note that the reward pool holds decimal coins (`DecCoins`) to allow -for fractions of coins to be received from operations like inflation. -When coins are distributed from the pool they are truncated back to -`sdk.Coins` which are non-decimal. - - - Global: `0x00 -> amino(global)` - -```golang -// coins with decimal -type DecCoins []DecCoin - -type DecCoin struct { - Amount sdk.Dec - Denom string -} - -type Global struct { - Accum sdk.Dec // global accumulation factor for lazy calculations - Pool DecCoins // funds for all validators which have yet to be withdrawn - CommunityFund DecCoins // pool for community funds yet to be spent -} -``` - -### Validator Distribution - -Validator distribution information for the relevant validator is updated each time: - 1. delegation amount to a validator are updated, - 2. a validator successfully proposes a block and receives a reward, - 3. any delegator withdraws from a validator, or - 4. the validator withdraws it's commission. - - - ValidatorDistribution: `0x02 | ValOwnerAddr -> amino(validatorDistribution)` - -```golang -type ValidatorDistribution struct { - CommissionWithdrawalHeight int64 // last time this validator withdrew commission - Accum sdk.Dec // global pool accumulation factor - ProposerAccum sdk.Dec // proposer pool accumulation factor - ProposerPool DecCoins // reward pool collected from being the proposer -} -``` - -### Delegation Distribution - -Each delegation holds multiple accumulation factors to specify its entitlement to -the rewards from a validator. `Accum` is used to passively calculate -each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to -passively calculate each bonds entitled rewards from -`ValidatorDistribution.ProposerRewardPool` - - - DelegatorDistribution: ` 0x02 | DelegatorAddr | ValOwnerAddr -> amino(delegatorDist)` - -```golang -type DelegatorDist struct { - WithdrawalHeight int64 // last time this delegation withdrew rewards - Accum sdk.Dec // reward provisioning accumulation factor - AccumProposer sdk.Dec // proposers pool accumulation factor -} -``` - From a0e05a8f0373d4b123200c98c60f9517484f22dd Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Wed, 8 Aug 2018 15:03:43 -0400 Subject: [PATCH 04/49] triggers, endblock, transactions --- docs/spec/distribution/end_block.md | 8 +- docs/spec/distribution/future_improvements.md | 16 -- docs/spec/distribution/transactions.md | 176 +----------------- docs/spec/distribution/triggers.md | 27 ++- 4 files changed, 21 insertions(+), 206 deletions(-) delete mode 100644 docs/spec/distribution/future_improvements.md diff --git a/docs/spec/distribution/end_block.md b/docs/spec/distribution/end_block.md index bc6847ef4..19c9e8302 100644 --- a/docs/spec/distribution/end_block.md +++ b/docs/spec/distribution/end_block.md @@ -3,10 +3,10 @@ At each endblock, the fees received are sorted to the proposer, community fund, and global pool. When the validator is the proposer of the round, that validator (and their delegators) receives between 1% and 5% of fee rewards, the -reserve tax is then charged, then the remainder is distributed proportionally -by voting power to all bonded validators independent of whether they voted -(social distribution). Note the social distribution is applied to proposer -validator in addition to the proposer reward. +reserve community tax is then charged, then the remainder is distributed +proportionally by voting power to all bonded validators independent of whether +they voted (social distribution). Note the social distribution is applied to +proposer validator in addition to the proposer reward. The amount of proposer reward is calculated from pre-commits Tendermint messages in order to incentivize validators to wait and include additional diff --git a/docs/spec/distribution/future_improvements.md b/docs/spec/distribution/future_improvements.md deleted file mode 100644 index 954fb4d62..000000000 --- a/docs/spec/distribution/future_improvements.md +++ /dev/null @@ -1,16 +0,0 @@ -## Future Improvements - -### Power Change - -Within the current implementation all power changes ever made are indefinitely stored -within the current state. In the future this state should be trimmed on an epoch basis. Delegators -which will have not withdrawn their fees will be penalized in some way, depending on what is -computationally feasible this may include: - - burning non-withdrawn fees - - requiring more expensive withdrawal costs which include proofs from archive nodes of historical state - -In addition or as an alternative it may make sense to implement a "rolling" epoch which cycles through -all the delegators in small groups (for example 5 delegators per block) and just runs the withdrawal transaction -at standard rates and takes transaction fees from the withdrawal amount. - - diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index 1401c3b85..7816757e5 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -131,17 +131,12 @@ fulfilled the entitled fees for the various scenarios can be calculated. ```golang type DistributionScenario interface { - DistributorTokens() DecCoins // current tokens from distributor - DistributorCumulativeTokens() DecCoins // total tokens ever received - DistributorPrevReceivedTokens() DecCoins // last value of tokens received - DistributorShares() sdk.Dec // current shares - DistributorPrevShares() sdk.Dec // shares last block + DistributorTokens() DecCoins // current tokens from distributor + DistributorShares() sdk.Dec // current shares + RecipientAccum() sdk.Dec + RecipientShares() sdk.Dec // current shares - RecipientAdjustment() sdk.Dec - RecipientShares() sdk.Dec // current shares - RecipientPrevShares() sdk.Dec // shares last block - - ModifyAdjustments(withdrawal sdk.Dec) // proceedure to modify adjustment factors + ModifyAccums(withdrawal sdk.Dec) // proceedure to modify adjustment factors } ``` @@ -169,18 +164,9 @@ func (d DistributionScenario) RecipientCount(height int64) sdk.Dec func (d DistributionScenario) GlobalCount(height int64) sdk.Dec return d.DistributorShares() * height -func (d DistributionScenario) SimplePool() DecCoins - return d.RecipientCount() / d.GlobalCount() * d.DistributorCumulativeTokens - -func (d DistributionScenario) ProjectedPool(height int64) DecCoins - return d.RecipientPrevShares() * (height-1) - / (d.DistributorPrevShares() * (height-1)) - * d.DistributorCumulativeTokens - + d.RecipientShares() / d.DistributorShares() - * d.DistributorPrevReceivedTokens() ``` -The `DistributionScenario` _adjustment_ terms account for changes in +The `DistributionScenario` _accum_ terms account for changes in recipient/distributor shares and recipient withdrawals. The adjustment factor must be modified whenever the recipient withdraws from the distributor or the distributor's/recipient's shares are changed. @@ -218,49 +204,6 @@ shares should be taken as true number of global bonded shares. The recipients shares should be taken as the bonded tokens less the validator's commission. ``` -type DelegationFromGlobalPool struct { - DelegationShares sdk.Dec - ValidatorCommission sdk.Dec - ValidatorBondedTokens sdk.Dec - ValidatorDelegatorShareExRate sdk.Dec - PoolBondedTokens sdk.Dec - Global Global - ValDistr ValidatorDistribution - DelDistr DelegatorDistribution -} - -func (d DelegationFromGlobalPool) DistributorTokens() DecCoins - return d.Global.Pool - -func (d DelegationFromGlobalPool) DistributorCumulativeTokens() DecCoins - return d.Global.EverReceivedPool - -func (d DelegationFromGlobalPool) DistributorPrevReceivedTokens() DecCoins - return d.Global.PrevReceivedPool - -func (d DelegationFromGlobalPool) DistributorShares() sdk.Dec - return d.PoolBondedTokens - -func (d DelegationFromGlobalPool) DistributorPrevShares() sdk.Dec - return d.Global.PrevBondedTokens - -func (d DelegationFromGlobalPool) RecipientShares() sdk.Dec - return d.DelegationShares * d.ValidatorDelegatorShareExRate() * - d.ValidatorBondedTokens() * (1 - d.ValidatorCommission) - -func (d DelegationFromGlobalPool) RecipientPrevShares() sdk.Dec - return d.DelDistr.PrevTokens - -func (d DelegationFromGlobalPool) RecipientAdjustment() sdk.Dec - return d.DelDistr.Adjustment - -func (d DelegationFromGlobalPool) ModifyAdjustments(withdrawal sdk.Dec) - d.ValDistr.Adjustment += withdrawal - d.DelDistr.Adjustment += withdrawal - d.global.Adjustment += withdrawal - SetValidatorDistribution(d.ValDistr) - SetDelegatorDistribution(d.DelDistr) - SetGlobal(d.Global) ``` #### Delegation's entitlement to ValidatorDistribution.ProposerPool @@ -272,43 +215,6 @@ shares is taken as the effective delegation shares less the validator's commission. ``` -type DelegationFromProposerPool struct { - DelegationShares sdk.Dec - ValidatorCommission sdk.Dec - ValidatorDelegatorShares sdk.Dec - ValDistr ValidatorDistribution - DelDistr DelegatorDistribution -} - -func (d DelegationFromProposerPool) DistributorTokens() DecCoins - return d.ValDistr.ProposerPool - -func (d DelegationFromProposerPool) DistributorCumulativeTokens() DecCoins - return d.ValDistr.EverReceivedProposerReward - -func (d DelegationFromProposerPool) DistributorPrevReceivedTokens() DecCoins - return d.ValDistr.PrevReceivedProposerReward - -func (d DelegationFromProposerPool) DistributorShares() sdk.Dec - return d.ValidatorDelegatorShares - -func (d DelegationFromProposerPool) DistributorPrevShares() sdk.Dec - return d.ValDistr.PrevDelegatorShares - -func (d DelegationFromProposerPool) RecipientShares() sdk.Dec - return d.DelegationShares * (1 - d.ValidatorCommission) - -func (d DelegationFromProposerPool) RecipientPrevShares() sdk.Dec - return d.DelDistr.PrevShares - -func (d DelegationFromProposerPool) RecipientAdjustment() sdk.Dec - return d.DelDistr.AdjustmentProposer - -func (d DelegationFromProposerPool) ModifyAdjustments(withdrawal sdk.Dec) - d.ValDistr.AdjustmentProposer += withdrawal - d.DelDistr.AdjustmentProposer += withdrawal - SetValidatorDistribution(d.ValDistr) - SetDelegatorDistribution(d.DelDistr) ``` #### Validators's commission entitlement to Global.Pool @@ -317,43 +223,6 @@ Similar to a delegator's entitlement, but with recipient shares based on the commission portion of bonded tokens. ``` -type CommissionFromGlobalPool struct { - ValidatorBondedTokens sdk.Dec - ValidatorCommission sdk.Dec - PoolBondedTokens sdk.Dec - Global Global - ValDistr ValidatorDistribution -} - -func (c CommissionFromGlobalPool) DistributorTokens() DecCoins - return c.Global.Pool - -func (c CommissionFromGlobalPool) DistributorCumulativeTokens() DecCoins - return c.Global.EverReceivedPool - -func (c CommissionFromGlobalPool) DistributorPrevReceivedTokens() DecCoins - return c.Global.PrevReceivedPool - -func (c CommissionFromGlobalPool) DistributorShares() sdk.Dec - return c.PoolBondedTokens - -func (c CommissionFromGlobalPool) DistributorPrevShares() sdk.Dec - return c.Global.PrevBondedTokens - -func (c CommissionFromGlobalPool) RecipientShares() sdk.Dec - return c.ValidatorBondedTokens() * c.ValidatorCommission - -func (c CommissionFromGlobalPool) RecipientPrevShares() sdk.Dec - return c.ValDistr.PrevBondedTokens * c.ValidatorCommission - -func (c CommissionFromGlobalPool) RecipientAdjustment() sdk.Dec - return c.ValDistr.Adjustment - -func (c CommissionFromGlobalPool) ModifyAdjustments(withdrawal sdk.Dec) - c.ValDistr.Adjustment += withdrawal - c.Global.Adjustment += withdrawal - SetValidatorDistribution(c.ValDistr) - SetGlobal(c.Global) ``` #### Validators's commission entitlement to ValidatorDistribution.ProposerPool @@ -362,38 +231,5 @@ Similar to a delegators entitlement to the proposer pool, but with recipient shares based on the commission portion of the total delegator shares. ``` -type CommissionFromProposerPool struct { - ValidatorDelegatorShares sdk.Dec - ValidatorCommission sdk.Dec - ValDistr ValidatorDistribution -} - -func (c CommissionFromProposerPool) DistributorTokens() DecCoins - return c.ValDistr.ProposerPool - -func (c CommissionFromProposerPool) DistributorCumulativeTokens() DecCoins - return c.ValDistr.EverReceivedProposerReward - -func (c CommissionFromProposerPool) DistributorPrevReceivedTokens() DecCoins - return c.ValDistr.PrevReceivedProposerReward - -func (c CommissionFromProposerPool) DistributorShares() sdk.Dec - return c.ValidatorDelegatorShares - -func (c CommissionFromProposerPool) DistributorPrevShares() sdk.Dec - return c.ValDistr.PrevDelegatorShares - -func (c CommissionFromProposerPool) RecipientShares() sdk.Dec - return c.ValidatorDelegatorShares * (c.ValidatorCommission) - -func (c CommissionFromProposerPool) RecipientPrevShares() sdk.Dec - return c.ValDistr.PrevDelegatorShares * (c.ValidatorCommission) - -func (c CommissionFromProposerPool) RecipientAdjustment() sdk.Dec - return c.ValDistr.AdjustmentProposer - -func (c CommissionFromProposerPool) ModifyAdjustments(withdrawal sdk.Dec) - c.ValDistr.AdjustmentProposer += withdrawal - SetValidatorDistribution(c.ValDistr) ``` diff --git a/docs/spec/distribution/triggers.md b/docs/spec/distribution/triggers.md index 8800609a4..93358fe2e 100644 --- a/docs/spec/distribution/triggers.md +++ b/docs/spec/distribution/triggers.md @@ -1,26 +1,14 @@ # Triggers -## Create validator distribution - - - triggered-by: validator entering bonded validator group (`stake.bondValidator()`) - -Whenever a new validator is added to the Tendermint validator set they are -entitled to begin earning rewards of atom provisions and fees. At this point -`ValidatorDistribution.Pool()` must be zero (as the validator has not yet -earned any rewards) meaning that the initial value for `validator.Adjustment` -must be set to the value of `validator.SimplePool()` for the height which the -validator is added on the validator set. - ## Create or modify delegation distribution - triggered-by: `stake.TxDelegate`, `stake.TxBeginRedelegate`, `stake.TxBeginUnbonding` The pool of a new delegator bond will be 0 for the height at which the bond was -added. This is achieved by setting `DelegationDistribution.WithdrawalHeight` to -the height which the bond was added. Additionally the `AdjustmentPool` and -`AdjustmentProposerPool` must be set to the equivalent values of -`DelegationDistribution.SimplePool()` and -`DelegationDistribution.SimpleProposerPool()` for the height of delegation. +added, or the withdrawal has taken place. This is achieved by setting +`DelegatorDist.WithdrawalHeight` to the relevant height, withdrawing any +remaining fees, and setting `DelegatorDist.Accum` and +`DelegatorDist.ProposerAccum` to 0. ## Commission rate change @@ -29,3 +17,10 @@ the height which the bond was added. Additionally the `AdjustmentPool` and If a validator changes its commission rate, all commission on fees must be simultaneously withdrawn using the transaction `TxWithdrawValidator` +## Change in Validator State + + - triggered-by: `stake.Slash`, `stake.UpdateValidator` + +Whenever a validator is slashed or enters/leaves the validator group +`ValidatorUpdate` information must be recorded in order to properly calculate +the accum factors. From 5285489977f5126b1f9a954d0e7c80cb75229da5 Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Wed, 8 Aug 2018 23:43:40 -0400 Subject: [PATCH 05/49] general updates --- docs/spec/distribution/state.md | 49 +++++++---- docs/spec/distribution/transactions.md | 108 ++++--------------------- docs/spec/distribution/triggers.md | 14 ++-- 3 files changed, 58 insertions(+), 113 deletions(-) diff --git a/docs/spec/distribution/state.md b/docs/spec/distribution/state.md index 757287004..115782594 100644 --- a/docs/spec/distribution/state.md +++ b/docs/spec/distribution/state.md @@ -23,9 +23,10 @@ type DecCoin struct { } type Global struct { - Accum sdk.Dec // global accumulation factor for lazy calculations - Pool DecCoins // funds for all validators which have yet to be withdrawn - CommunityFund DecCoins // pool for community funds yet to be spent + TotalValAccumUpdateHeight int64 // last height which the total validator accum was updated + TotalValAccum sdk.Dec // total valdator accum held by validators + Pool DecCoins // funds for all validators which have yet to be withdrawn + CommunityPool DecCoins // pool for community funds yet to be spent } ``` @@ -41,28 +42,48 @@ Validator distribution information for the relevant validator is updated each ti ```golang type ValidatorDistribution struct { - CommissionWithdrawalHeight int64 // last time this validator withdrew commission - Accum sdk.Dec // global pool accumulation factor - ProposerAccum sdk.Dec // proposer pool accumulation factor - ProposerPool DecCoins // reward pool collected from being the proposer + CommissionWithdrawalHeight int64 // last height this validator withdrew commission + + GlobalWithdrawalHeight int64 // last height this validator withdrew from the global pool + Pool DecCoins // reward pool collected held within this validator (includes proposer rewards) + + TotalDelAccumUpdateHeight int64 // last height which the total delegator accum was updated + TotalDelAccum sdk.Dec // total proposer pool accumulation factor held by delegators } ``` ### Delegation Distribution -Each delegation holds multiple accumulation factors to specify its entitlement to -the rewards from a validator. `Accum` is used to passively calculate -each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to -passively calculate each bonds entitled rewards from -`ValidatorDistribution.ProposerRewardPool` +Each delegation distribution only needs to record the height at which it last +withdrew fees. Because a delegation must withdraw fees each time it's +properties change (aka bonded tokens etc.) its properties will remain constant +and the delegator's _accumulation_ factor can be calculated passively knowing +only the height of the last withdrawal and its current properties. - DelegatorDistribution: ` 0x02 | DelegatorAddr | ValOwnerAddr -> amino(delegatorDist)` ```golang type DelegatorDist struct { WithdrawalHeight int64 // last time this delegation withdrew rewards - Accum sdk.Dec // reward provisioning accumulation factor - AccumProposer sdk.Dec // proposers pool accumulation factor } ``` +### Validator Update + +Every instance that a validator: + - enters into the bonded state, + - leaves the bonded state, + - is slashed, or + - changes its commission rate, + +information about the state change must be recorded as a `ValidatorUpdate`. +Each power change is indexed by validator and its block height. + + - ValidatorUpdate: `0x03 | ValOwnerAddr | amino(Height) -> amino(ValidatorUpdate)` + +```golang +type ValidatorUpdate struct { + Height int64 // block height of update + NewCommissionRate sdk.Dec // commission rate at this height +} +``` diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index 7816757e5..7d6bcc752 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -1,3 +1,16 @@ + + + +Each delegation holds multiple accumulation factors to specify its entitlement to +the rewards from a validator. `Accum` is used to passively calculate +each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to +passively calculate each bonds entitled rewards from +`ValidatorDistribution.ProposerRewardPool` + + + + + # Transactions ## TxWithdrawDelegation @@ -10,7 +23,7 @@ redelegation, or delegation of additional tokens to a specific validator. Each time a withdrawal is made by a recipient the adjustment term must be modified for each block with a change in distributors shares since the time of last withdrawal. This is accomplished by iterating over all relevant -`PowerChange`'s stored in distribution state. +`ValidatorUpdate`'s stored in distribution state. ```golang @@ -57,7 +70,7 @@ func GetDelegatorEntitlement(delegatorAddr sdk.AccAddress) DecCoins return entitlement -func (pc PowerChange) ProcessPowerChangeDelegation(delegation sdk.Delegation, +func (pc ValidatorUpdate) ProcessPowerChangeDelegation(delegation sdk.Delegation, DelDistr DelegationDistribution) // get the historical scenarios @@ -109,7 +122,7 @@ func WithdrawalValidator(ownerAddr, withdrawAddr sdk.AccAddress) AddCoins(withdrawAddr, totalEntitlment.TruncateDecimal()) -func (pc PowerChange) ProcessPowerChangeCommission() +func (pc ValidatorUpdate) ProcessPowerChangeCommission() // get the historical scenarios scenario1 = pc.CommissionFromGlobalPool() @@ -122,75 +135,6 @@ func (pc PowerChange) ProcessPowerChangeCommission() ## Common Calculations -### Distribution scenario - -A common form of abstracted calculations exists between validators and -delegations attempting to withdrawal their rewards, either from `Global.Pool` -or from `ValidatorDistribution.ProposerPool`. With the following interface -fulfilled the entitled fees for the various scenarios can be calculated. - -```golang -type DistributionScenario interface { - DistributorTokens() DecCoins // current tokens from distributor - DistributorShares() sdk.Dec // current shares - RecipientAccum() sdk.Dec - RecipientShares() sdk.Dec // current shares - - ModifyAccums(withdrawal sdk.Dec) // proceedure to modify adjustment factors -} -``` - -#### Entitled reward from distribution scenario - -The entitlement to the distributor's tokens held can be accounted for lazily. -To begin this calculation we must determine the recipient's _simple pool_ and -_projected pool_. The simple pool represents a lazy accounting of what a -recipient's entitlement to the distributor's tokens would be if all recipients -for that distributor had static shares (equal to the current shares), and no -recipients had ever withdrawn their entitled rewards. The projected pool -represents the anticipated recipient's entitlement to the distributors tokens -based on the current blocks token input (for example fees reward received) to -the distributor, and the distributor's tokens and shares of the previous block -assuming that neither had changed in the current block. Using the simple and -projected pools we can determine all cumulative changes which have taken place -outside of the recipient and adjust the recipient's _adjustment factor_ to -account for these changes and ultimately keep track of the correct entitlement -to the distributors tokens. - -``` -func (d DistributionScenario) RecipientCount(height int64) sdk.Dec - return v.RecipientShares() * height - -func (d DistributionScenario) GlobalCount(height int64) sdk.Dec - return d.DistributorShares() * height - -``` - -The `DistributionScenario` _accum_ terms account for changes in -recipient/distributor shares and recipient withdrawals. The adjustment factor -must be modified whenever the recipient withdraws from the distributor or the -distributor's/recipient's shares are changed. - - When the shares of the recipient is changed the adjustment factor is - increased/decreased by the difference between the _simple_ and _projected_ - pools. In other words, the cumulative difference in the shares if the shares - has been the new shares as opposed to the old shares for the entire duration of - the blockchain up the previous block. - - When a recipient makes a withdrawal the adjustment factor is increased by the - withdrawal amount. - -``` -func (d DistributionScenario) UpdateAdjustmentForPowerChange(height int64) - simplePool = d.SimplePool() - projectedPool = d.ProjectedPool(height) - AdjustmentChange = simplePool - projectedPool - if AdjustmentChange > 0 - d.ModifyAdjustments(AdjustmentChange) - -func (d DistributionScenario) WithdrawalEntitlement() DecCoins - entitlement = d.SimplePool() - d.RecipientAdjustment() - d.ModifyAdjustments(entitlement) - return entitlement -``` ### Distribution scenarios @@ -206,17 +150,6 @@ shares should be taken as the bonded tokens less the validator's commission. ``` ``` -#### Delegation's entitlement to ValidatorDistribution.ProposerPool - -Delegations (including validator's self-delegation) are still subject -commission on the rewards gained from the proposer pool. Global shares in this -context is actually the validators total delegations shares. The recipient's -shares is taken as the effective delegation shares less the validator's -commission. - -``` -``` - #### Validators's commission entitlement to Global.Pool Similar to a delegator's entitlement, but with recipient shares based on the @@ -224,12 +157,3 @@ commission portion of bonded tokens. ``` ``` - -#### Validators's commission entitlement to ValidatorDistribution.ProposerPool - -Similar to a delegators entitlement to the proposer pool, but with recipient -shares based on the commission portion of the total delegator shares. - -``` -``` - diff --git a/docs/spec/distribution/triggers.md b/docs/spec/distribution/triggers.md index 93358fe2e..0e02f0431 100644 --- a/docs/spec/distribution/triggers.md +++ b/docs/spec/distribution/triggers.md @@ -6,21 +6,21 @@ The pool of a new delegator bond will be 0 for the height at which the bond was added, or the withdrawal has taken place. This is achieved by setting -`DelegatorDist.WithdrawalHeight` to the relevant height, withdrawing any -remaining fees, and setting `DelegatorDist.Accum` and -`DelegatorDist.ProposerAccum` to 0. +`DelegatorDist.WithdrawalHeight` to the height of the triggering transaction. ## Commission rate change - triggered-by: `stake.TxEditValidator` If a validator changes its commission rate, all commission on fees must be -simultaneously withdrawn using the transaction `TxWithdrawValidator` +simultaneously withdrawn using the transaction `TxWithdrawValidator`. +Additionally the change and associated height must be recorded in a +`ValidatorUpdate` state record. ## Change in Validator State - triggered-by: `stake.Slash`, `stake.UpdateValidator` -Whenever a validator is slashed or enters/leaves the validator group -`ValidatorUpdate` information must be recorded in order to properly calculate -the accum factors. +Whenever a validator is slashed or enters/leaves the validator group all of the +validator entitled reward tokens must be simultaniously withdrawn from +`Global.Pool` and added to `ValidatorDistribution.Pool` From e7ebe35b0a530bf470c4c0351cc369821e123541 Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Thu, 9 Aug 2018 00:34:19 -0400 Subject: [PATCH 06/49] updating transactions --- docs/spec/distribution/transactions.md | 95 +++++++++++++++++--------- 1 file changed, 61 insertions(+), 34 deletions(-) diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index 7d6bcc752..30ea6ab46 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -1,16 +1,3 @@ - - - -Each delegation holds multiple accumulation factors to specify its entitlement to -the rewards from a validator. `Accum` is used to passively calculate -each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to -passively calculate each bonds entitled rewards from -`ValidatorDistribution.ProposerRewardPool` - - - - - # Transactions ## TxWithdrawDelegation @@ -42,14 +29,14 @@ func GetDelegatorEntitlement(delegatorAddr sdk.AccAddress) DecCoins delegations = GetDelegations(delegatorAddr) DelDistr = GetDelegationDistribution(delegation.DelegatorAddr, delegation.ValidatorAddr) - pcs = GetPowerChanges(DelDistr.WithdrawalHeight) + vus = GetValidatorUpdates(DelDistr.WithdrawalHeight) // update all adjustment factors for each delegation since last withdrawal - for pc = range pcs + for vu = range vus for delegation = range delegations DelDistr = GetDelegationDistribution(delegation.DelegatorAddr, delegation.ValidatorAddr) - pc.ProcessPowerChangeDelegation(delegation, DelDistr) + vu.ProcessPowerChangeDelegation(delegation, DelDistr) // collect all entitled fees entitlement = 0 @@ -70,16 +57,16 @@ func GetDelegatorEntitlement(delegatorAddr sdk.AccAddress) DecCoins return entitlement -func (pc ValidatorUpdate) ProcessPowerChangeDelegation(delegation sdk.Delegation, +func (vu ValidatorUpdate) ProcessPowerChangeDelegation(delegation sdk.Delegation, DelDistr DelegationDistribution) // get the historical scenarios - scenario1 = pc.DelegationFromGlobalPool(delegation, DelDistr) - scenario2 = pc.DelegationFromProvisionPool(delegation, DelDistr) + scenario1 = vu.DelegationFromGlobalPool(delegation, DelDistr) + scenario2 = vu.DelegationFromProvisionPool(delegation, DelDistr) // process the adjustment factors - scenario1.UpdateAdjustmentForPowerChange(pc.Height) - scenario2.UpdateAdjustmentForPowerChange(pc.Height) + scenario1.UpdateAdjustmentForPowerChange(vu.Height) + scenario2.UpdateAdjustmentForPowerChange(vu.Height) ``` ## TxWithdrawValidator @@ -91,7 +78,7 @@ redelegation, or delegation of additional tokens to a specific validator. This transaction withdraws the validators commission rewards, as well as any rewards earning on their self-delegation. -```golang +``` type TxWithdrawValidator struct { ownerAddr sdk.AccAddress // validator address to withdraw from withdrawAddr sdk.AccAddress // address to make the withdrawal to @@ -104,9 +91,9 @@ func WithdrawalValidator(ownerAddr, withdrawAddr sdk.AccAddress) // update the validator adjustment factors for commission ValDistr = GetValidatorDistribution(ownerAddr.ValidatorAddr) - pcs = GetPowerChanges(ValDistr.CommissionWithdrawalHeight) - for pc = range pcs - pc.ProcessPowerChangeCommission() + vus = GetValidatorUpdates(ValDistr.CommissionWithdrawalHeight) + for vu = range vus + vu.ProcessPowerChangeCommission() // withdrawal validator commission rewards global = GetGlobal() @@ -122,38 +109,78 @@ func WithdrawalValidator(ownerAddr, withdrawAddr sdk.AccAddress) AddCoins(withdrawAddr, totalEntitlment.TruncateDecimal()) -func (pc ValidatorUpdate) ProcessPowerChangeCommission() +func (vu ValidatorUpdate) ProcessPowerChangeCommission() // get the historical scenarios - scenario1 = pc.CommissionFromGlobalPool() - scenario2 = pc.CommissionFromProposerPool() + scenario1 = vu.CommissionFromGlobalPool() + scenario2 = vu.CommissionFromProposerPool() // process the adjustment factors - scenario1.UpdateAdjustmentForPowerChange(pc.Height) - scenario2.UpdateAdjustmentForPowerChange(pc.Height) + scenario1.UpdateAdjustmentForPowerChange(vu.Height) + scenario2.UpdateAdjustmentForPowerChange(vu.Height) ``` ## Common Calculations -### Distribution scenarios +### Update total validator accum -Note that the distribution scenario structures are found in `state.md`. +The total amount of validator accum must be calculated in order to determine +the amount of pool tokens which a validator is entitled to at a particular block. +This term is to be updated during a validator withdrawal. -#### Delegation's entitlement to Global.Pool +``` +func (g Global) UpdateTotalValAccum() + +TODO + +``` + +### Update total delegator accum + +Each delegation holds multiple accumulation factors to specify its entitlement to +the rewards from a validator. `Accum` is used to passively calculate +each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to +passively calculate each bonds entitled rewards from +`ValidatorDistribution.ProposerRewardPool` + +``` +TODO +``` + +### Global Pool to Validator Pool + +Everytime a validator or delegator make a withdraw or the validator is the +proposer and receives new tokens - the relavent validator must move tokens from +the passive global pool to thier own pool. + +``` +TODO +``` + + +### Delegation's entitlement to ValidatorDistribution.Pool For delegations (including validator's self-delegation) all fees from fee pool are subject to commission rate from the owner of the validator. The global shares should be taken as true number of global bonded shares. The recipients shares should be taken as the bonded tokens less the validator's commission. +Each delegation holds multiple accumulation factors to specify its entitlement to +the rewards from a validator. `Accum` is used to passively calculate +each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to +passively calculate each bonds entitled rewards from +`ValidatorDistribution.ProposerRewardPool` + ``` +TODO ``` -#### Validators's commission entitlement to Global.Pool +### Validators's commission entitlement to ValidatorDistribution.Pool Similar to a delegator's entitlement, but with recipient shares based on the commission portion of bonded tokens. ``` +TODO ``` From 5e5fad454849131246c30796f22e6de0d0a4e96a Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Tue, 14 Aug 2018 13:45:13 -0400 Subject: [PATCH 07/49] txs --- docs/spec/distribution/transactions.md | 43 +++++++++++++++++--------- 1 file changed, 29 insertions(+), 14 deletions(-) diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index 30ea6ab46..b8de54ca5 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -122,30 +122,43 @@ func (vu ValidatorUpdate) ProcessPowerChangeCommission() ## Common Calculations - ### Update total validator accum The total amount of validator accum must be calculated in order to determine -the amount of pool tokens which a validator is entitled to at a particular block. -This term is to be updated during a validator withdrawal. +the amount of pool tokens which a validator is entitled to at a particular +block. The accum is always additive to the existing accum. This term is to be +updates each time rewards are withdrawn from the system. ``` -func (g Global) UpdateTotalValAccum() - -TODO - +func (g Global) UpdateTotalValAccum(height int64, totalBondedTokens Dec) + blocks = height - g.TotalValAccumUpdateHeight + g.TotalValAccum += totalDelShares * blocks ``` ### Update total delegator accum -Each delegation holds multiple accumulation factors to specify its entitlement to -the rewards from a validator. `Accum` is used to passively calculate -each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to -passively calculate each bonds entitled rewards from -`ValidatorDistribution.ProposerRewardPool` +The total amount of delegator accum must be updated in order to determine the +amount of pool tokens which each delegator is entitled to, relative to the +other delegators for that validator. The accum is always additive to +the existing accum. This term is to be updated each time a +withdrawal is made from a validator. ``` -TODO +func (vd ValidatorDistribution) UpdateTotalDelAccum(height int64, totalDelShares Dec) + blocks = height - vd.TotalDelAccumUpdateHeight + vd.TotalDelAccum += totalDelShares * blocks +``` + +### Delegator accum + +Each delegation has a passively calculated accumulation factor to specify its +entitlement to the rewards from a validator. `Accum` is used to passively +calculate each bonds entitled rewards from the `Validator.Pool`. + +``` +func (dd DelegatorDist) Accum(height int64, delegatorShares Dec) Dec + blocks = height - dd.WithdrawalHeight + return delegatorShares * blocks ``` ### Global Pool to Validator Pool @@ -155,7 +168,9 @@ proposer and receives new tokens - the relavent validator must move tokens from the passive global pool to thier own pool. ``` -TODO +func (dd DelegatorDist) ValidatorUpdate(g Global, totalBondedShares, height int64, Tokens Dec) Dec + + g.UpdateTotalValAccum(height, totalBondedShares) ``` From 360eb75c8e6cbc11ac7b7ae885a8bb3f7d11db20 Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Tue, 14 Aug 2018 23:33:40 -0400 Subject: [PATCH 08/49] working --- docs/spec/distribution/transactions.md | 45 +++++++++++++++----------- 1 file changed, 26 insertions(+), 19 deletions(-) diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index b8de54ca5..22e4e4077 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -120,14 +120,14 @@ func (vu ValidatorUpdate) ProcessPowerChangeCommission() scenario2.UpdateAdjustmentForPowerChange(vu.Height) ``` -## Common Calculations +## Common calculations ### Update total validator accum The total amount of validator accum must be calculated in order to determine the amount of pool tokens which a validator is entitled to at a particular block. The accum is always additive to the existing accum. This term is to be -updates each time rewards are withdrawn from the system. +updates each time rewards are withdrawn from the system. ``` func (g Global) UpdateTotalValAccum(height int64, totalBondedTokens Dec) @@ -135,7 +135,7 @@ func (g Global) UpdateTotalValAccum(height int64, totalBondedTokens Dec) g.TotalValAccum += totalDelShares * blocks ``` -### Update total delegator accum +### Update validator's accums The total amount of delegator accum must be updated in order to determine the amount of pool tokens which each delegator is entitled to, relative to the @@ -149,7 +149,7 @@ func (vd ValidatorDistribution) UpdateTotalDelAccum(height int64, totalDelShares vd.TotalDelAccum += totalDelShares * blocks ``` -### Delegator accum +### Get delegator accum Each delegation has a passively calculated accumulation factor to specify its entitlement to the rewards from a validator. `Accum` is used to passively @@ -161,37 +161,44 @@ func (dd DelegatorDist) Accum(height int64, delegatorShares Dec) Dec return delegatorShares * blocks ``` -### Global Pool to Validator Pool +### Global pool to validator pool Everytime a validator or delegator make a withdraw or the validator is the proposer and receives new tokens - the relavent validator must move tokens from the passive global pool to thier own pool. ``` -func (dd DelegatorDist) ValidatorUpdate(g Global, totalBondedShares, height int64, Tokens Dec) Dec - +func (vd ValidatorDistribution) TakeAccum(g Global, height int64, totalBonded, vdTokens Dec) g Global g.UpdateTotalValAccum(height, totalBondedShares) + g.UpdateValAccum(height, totalBondedShares) + + // update the validators pool + blocks = height - vd.GlobalWithdrawalHeight + accum = blocks * vdTokens + withdrawalTokens := g.Pool * accum / g.TotalValAccum + + vd.Pool += withdrawalTokens + g.Pool -= withdrawalTokens + + return g ``` -### Delegation's entitlement to ValidatorDistribution.Pool +### Delegation's withdrawal For delegations (including validator's self-delegation) all fees from fee pool -are subject to commission rate from the owner of the validator. The global -shares should be taken as true number of global bonded shares. The recipients -shares should be taken as the bonded tokens less the validator's commission. - -Each delegation holds multiple accumulation factors to specify its entitlement to -the rewards from a validator. `Accum` is used to passively calculate -each bonds entitled rewards from the `RewardPool`. `AccumProposer` is used to -passively calculate each bonds entitled rewards from -`ValidatorDistribution.ProposerRewardPool` +are subject to commission rate from the owner of the validator. ``` -TODO +func (dd DelegatorDist) WithdrawalRewards(g Global, vd ValidatorDistribution, + height int64, totalBonded, vdTokens Dec) g Global + + vd.TakeAccum(g, height, totalBonded, vdTokens) + + ``` -### Validators's commission entitlement to ValidatorDistribution.Pool +### Validators's commission withdrawal Similar to a delegator's entitlement, but with recipient shares based on the commission portion of bonded tokens. From 37ed3b1a6eae69e2f8a361849c01417c6fa598ce Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Wed, 15 Aug 2018 04:12:44 -0400 Subject: [PATCH 09/49] agasg --- docs/spec/distribution/transactions.md | 40 ++++++++++++++++---------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index 22e4e4077..cf10d8e2e 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -149,18 +149,6 @@ func (vd ValidatorDistribution) UpdateTotalDelAccum(height int64, totalDelShares vd.TotalDelAccum += totalDelShares * blocks ``` -### Get delegator accum - -Each delegation has a passively calculated accumulation factor to specify its -entitlement to the rewards from a validator. `Accum` is used to passively -calculate each bonds entitled rewards from the `Validator.Pool`. - -``` -func (dd DelegatorDist) Accum(height int64, delegatorShares Dec) Dec - blocks = height - dd.WithdrawalHeight - return delegatorShares * blocks -``` - ### Global pool to validator pool Everytime a validator or delegator make a withdraw or the validator is the @@ -177,6 +165,7 @@ func (vd ValidatorDistribution) TakeAccum(g Global, height int64, totalBonded, v accum = blocks * vdTokens withdrawalTokens := g.Pool * accum / g.TotalValAccum + g.TotalValAccum -= accumm vd.Pool += withdrawalTokens g.Pool -= withdrawalTokens @@ -191,10 +180,18 @@ are subject to commission rate from the owner of the validator. ``` func (dd DelegatorDist) WithdrawalRewards(g Global, vd ValidatorDistribution, - height int64, totalBonded, vdTokens Dec) g Global + height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn Dec) - vd.TakeAccum(g, height, totalBonded, vdTokens) + vd.UpdateTotalDelAccum(height, totalDelShares) + g = vd.TakeAccum(g, height, totalBonded, vdTokens) + blocks = height - dd.WithdrawalHeight + accum = delegatorShares * blocks * (1 - commissionRate) + + withdrawalTokens := vd.Pool * accum / cd.TotalDelAccum + vd.Pool -= withdrawalTokens + vd.TotalDelAccum -= accum + return withdrawalTokens ``` @@ -204,5 +201,18 @@ Similar to a delegator's entitlement, but with recipient shares based on the commission portion of bonded tokens. ``` -TODO +func (dd DelegatorDist) WithdrawalCommission(g Global, vd ValidatorDistribution, + height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) g Global + + vd.UpdateTotalDelAccum(height, totalDelShares) + g = vd.TakeAccum(g, height, totalBonded, vdTokens) + + blocks = height - vd.CommissionWithdrawalHeight + accum = delegatorShares * blocks * (commissionRate) + + withdrawalTokens := vd.Pool * accum / cd.TotalDelAccum + + vd.Pool -= withdrawalTokens + vd.TotalDelAccum -= accum + ``` From 31d5348c4ae164a187f3670ddf4535045d758444 Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Wed, 15 Aug 2018 20:03:39 -0400 Subject: [PATCH 10/49] txs pretty much ready --- docs/spec/distribution/transactions.md | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index cf10d8e2e..4e8ad92e4 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -133,6 +133,7 @@ updates each time rewards are withdrawn from the system. func (g Global) UpdateTotalValAccum(height int64, totalBondedTokens Dec) blocks = height - g.TotalValAccumUpdateHeight g.TotalValAccum += totalDelShares * blocks + g.TotalValAccumUpdateHeight = height ``` ### Update validator's accums @@ -147,6 +148,7 @@ withdrawal is made from a validator. func (vd ValidatorDistribution) UpdateTotalDelAccum(height int64, totalDelShares Dec) blocks = height - vd.TotalDelAccumUpdateHeight vd.TotalDelAccum += totalDelShares * blocks + vd.TotalDelAccumUpdateHeight = height ``` ### Global pool to validator pool @@ -162,6 +164,7 @@ func (vd ValidatorDistribution) TakeAccum(g Global, height int64, totalBonded, v // update the validators pool blocks = height - vd.GlobalWithdrawalHeight + vd.GlobalWithdrawalHeight = height accum = blocks * vdTokens withdrawalTokens := g.Pool * accum / g.TotalValAccum @@ -179,19 +182,22 @@ For delegations (including validator's self-delegation) all fees from fee pool are subject to commission rate from the owner of the validator. ``` -func (dd DelegatorDist) WithdrawalRewards(g Global, vd ValidatorDistribution, +func (dd DelegatorDist) WithdrawRewards(g Global, vd ValidatorDistribution, height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn Dec) vd.UpdateTotalDelAccum(height, totalDelShares) g = vd.TakeAccum(g, height, totalBonded, vdTokens) blocks = height - dd.WithdrawalHeight + dd.WithdrawalHeight = height accum = delegatorShares * blocks * (1 - commissionRate) - withdrawalTokens := vd.Pool * accum / cd.TotalDelAccum + withdrawalTokens := vd.Pool * accum / vd.TotalDelAccum + vd.TotalDelAccum -= accum + vd.Pool -= withdrawalTokens vd.TotalDelAccum -= accum - return withdrawalTokens + return g, withdrawalTokens ``` @@ -201,18 +207,21 @@ Similar to a delegator's entitlement, but with recipient shares based on the commission portion of bonded tokens. ``` -func (dd DelegatorDist) WithdrawalCommission(g Global, vd ValidatorDistribution, - height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) g Global +func (vd ValidatorDist) WithdrawCommission(g Global, vd ValidatorDistribution, + height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn Dec) vd.UpdateTotalDelAccum(height, totalDelShares) g = vd.TakeAccum(g, height, totalBonded, vdTokens) blocks = height - vd.CommissionWithdrawalHeight + vd.CommissionWithdrawalHeight = height accum = delegatorShares * blocks * (commissionRate) - withdrawalTokens := vd.Pool * accum / cd.TotalDelAccum + withdrawalTokens := vd.Pool * accum / vd.TotalDelAccum + vd.TotalDelAccum -= accum vd.Pool -= withdrawalTokens vd.TotalDelAccum -= accum + return g, withdrawalTokens ``` From 5ef0f5d70ea2cc62c28d3ddfd8ba06f7e0e1b547 Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Wed, 15 Aug 2018 23:33:26 -0400 Subject: [PATCH 11/49] finalize txs --- docs/spec/distribution/transactions.md | 122 ++++++++----------------- 1 file changed, 39 insertions(+), 83 deletions(-) diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index 4e8ad92e4..fbae152eb 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -2,124 +2,80 @@ ## TxWithdrawDelegation -When a delegator wishes to withdraw their transaction fees it must send +When a delegator wishes to withdraw their rewards it must send `TxWithdrawDelegation`. Note that parts of this transaction logic are also triggered each with any change in individual delegations, such as an unbond, redelegation, or delegation of additional tokens to a specific validator. -Each time a withdrawal is made by a recipient the adjustment term must be -modified for each block with a change in distributors shares since the time of -last withdrawal. This is accomplished by iterating over all relevant -`ValidatorUpdate`'s stored in distribution state. - - ```golang type TxWithdrawDelegation struct { delegatorAddr sdk.AccAddress withdrawAddr sdk.AccAddress // address to make the withdrawal to } -func WithdrawDelegator(delegatorAddr, withdrawAddr sdk.AccAddress) - entitlement = GetDelegatorEntitlement(delegatorAddr) +func WithdrawFromDelegator(delegatorAddr, withdrawAddr sdk.AccAddress) + height = GetHeight() + withdraw = GetDelegatorAllWithdraws(delegatorAddr, height) AddCoins(withdrawAddr, totalEntitlment.TruncateDecimal()) -func GetDelegatorEntitlement(delegatorAddr sdk.AccAddress) DecCoins +func GetDelegatorAllWithdraws(delegatorAddr sdk.AccAddress, height int64) DecCoins - // compile all the distribution scenarios + // get all distribution scenarios delegations = GetDelegations(delegatorAddr) - DelDistr = GetDelegationDistribution(delegation.DelegatorAddr, - delegation.ValidatorAddr) - vus = GetValidatorUpdates(DelDistr.WithdrawalHeight) - // update all adjustment factors for each delegation since last withdrawal - for vu = range vus - for delegation = range delegations - DelDistr = GetDelegationDistribution(delegation.DelegatorAddr, - delegation.ValidatorAddr) - vu.ProcessPowerChangeDelegation(delegation, DelDistr) - - // collect all entitled fees - entitlement = 0 + // collect all entitled rewards + withdraw = 0 + pool = stake.GetPool() + global = GetGlobal() for delegation = range delegations - global = GetGlobal() - pool = GetPool() - DelDistr = GetDelegationDistribution(delegation.DelegatorAddr, + delDistr = GetDelegationDistribution(delegation.DelegatorAddr, delegation.ValidatorAddr) - ValDistr = GetValidatorDistribution(delegation.ValidatorAddr) + valDistr = GetValidatorDistribution(delegation.ValidatorAddr) validator = GetValidator(delegation.ValidatorAddr) - scenerio1 = NewDelegationFromGlobalPool(delegation, validator, - pool, global, ValDistr, DelDistr) - scenerio2 = NewDelegationFromProvisionPool(delegation, validator, - ValDistr, DelDistr) - entitlement += scenerio1.WithdrawalEntitlement() - entitlement += scenerio2.WithdrawalEntitlement() - - return entitlement + global, ddWithdraw = delDistr.WithdrawRewards(global, valDistr, height, pool.BondedTokens, + validator.Tokens, validator.DelegatorShares, validator.Commission) + withdraw += ddWithdraw -func (vu ValidatorUpdate) ProcessPowerChangeDelegation(delegation sdk.Delegation, - DelDistr DelegationDistribution) - - // get the historical scenarios - scenario1 = vu.DelegationFromGlobalPool(delegation, DelDistr) - scenario2 = vu.DelegationFromProvisionPool(delegation, DelDistr) - - // process the adjustment factors - scenario1.UpdateAdjustmentForPowerChange(vu.Height) - scenario2.UpdateAdjustmentForPowerChange(vu.Height) + SetGlobal(global) + return withdraw ``` ## TxWithdrawValidator -When a validator wishes to withdraw their transaction fees it must send +When a validator wishes to withdraw their rewards it must send `TxWithdrawDelegation`. Note that parts of this transaction logic is also triggered each with any change in individual delegations, such as an unbond, redelegation, or delegation of additional tokens to a specific validator. This -transaction withdraws the validators commission rewards, as well as any rewards +transaction withdraws the validators commission fee, as well as any rewards earning on their self-delegation. ``` type TxWithdrawValidator struct { - ownerAddr sdk.AccAddress // validator address to withdraw from + operatorAddr sdk.AccAddress // validator address to withdraw from withdrawAddr sdk.AccAddress // address to make the withdrawal to } -func WithdrawalValidator(ownerAddr, withdrawAddr sdk.AccAddress) +func WithdrawFromValidator(operatorAddr, withdrawAddr sdk.AccAddress) - // update the delegator adjustment factors and also withdrawal delegation fees - entitlement = GetDelegatorEntitlement(ownerAddr) - - // update the validator adjustment factors for commission - ValDistr = GetValidatorDistribution(ownerAddr.ValidatorAddr) - vus = GetValidatorUpdates(ValDistr.CommissionWithdrawalHeight) - for vu = range vus - vu.ProcessPowerChangeCommission() - - // withdrawal validator commission rewards + height = GetHeight() global = GetGlobal() pool = GetPool() ValDistr = GetValidatorDistribution(delegation.ValidatorAddr) validator = GetValidator(delegation.ValidatorAddr) - scenerio1 = NewCommissionFromGlobalPool(validator, - pool, global, ValDistr) - scenerio2 = CommissionFromProposerPool(validator, ValDistr) - entitlement += scenerio1.WithdrawalEntitlement() - entitlement += scenerio2.WithdrawalEntitlement() - + // withdraw self-delegation + withdraw = GetDelegatorAllWithdraws(validator.OperatorAddr, height) + + // withdrawal validator commission rewards + global, commission = valDistr.WithdrawCommission(global, valDistr, height, pool.BondedTokens, + validator.Tokens, validator.DelegatorShares, validator.Commission) + withdraw += commission + SetGlobal(global) + AddCoins(withdrawAddr, totalEntitlment.TruncateDecimal()) - -func (vu ValidatorUpdate) ProcessPowerChangeCommission() - - // get the historical scenarios - scenario1 = vu.CommissionFromGlobalPool() - scenario2 = vu.CommissionFromProposerPool() - - // process the adjustment factors - scenario1.UpdateAdjustmentForPowerChange(vu.Height) - scenario2.UpdateAdjustmentForPowerChange(vu.Height) ``` - + ## Common calculations ### Update total validator accum @@ -153,9 +109,9 @@ func (vd ValidatorDistribution) UpdateTotalDelAccum(height int64, totalDelShares ### Global pool to validator pool -Everytime a validator or delegator make a withdraw or the validator is the -proposer and receives new tokens - the relavent validator must move tokens from -the passive global pool to thier own pool. +Every time a validator or delegator make a withdraw or the validator is the +proposer and receives new tokens - the relevant validator must move tokens from +the passive global pool to their own pool. ``` func (vd ValidatorDistribution) TakeAccum(g Global, height int64, totalBonded, vdTokens Dec) g Global @@ -178,12 +134,12 @@ func (vd ValidatorDistribution) TakeAccum(g Global, height int64, totalBonded, v ### Delegation's withdrawal -For delegations (including validator's self-delegation) all fees from fee pool -are subject to commission rate from the owner of the validator. +For delegations (including validator's self-delegation) all rewards from reward pool +are subject to commission rate from the operator of the validator. ``` func (dd DelegatorDist) WithdrawRewards(g Global, vd ValidatorDistribution, - height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn Dec) + height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn DecCoins) vd.UpdateTotalDelAccum(height, totalDelShares) g = vd.TakeAccum(g, height, totalBonded, vdTokens) @@ -208,7 +164,7 @@ commission portion of bonded tokens. ``` func (vd ValidatorDist) WithdrawCommission(g Global, vd ValidatorDistribution, - height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn Dec) + height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn DecCoins) vd.UpdateTotalDelAccum(height, totalDelShares) g = vd.TakeAccum(g, height, totalBonded, vdTokens) From 8fc9b95929a48a538c2f320d3eedeebf94082cc0 Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Wed, 15 Aug 2018 23:49:07 -0400 Subject: [PATCH 12/49] pending --- PENDING.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/PENDING.md b/PENDING.md index 6cc6ce10c..c0971cf80 100644 --- a/PENDING.md +++ b/PENDING.md @@ -63,6 +63,7 @@ IMPROVEMENTS * [tests] \#1806 CLI tests are now behind the build flag 'cli_test', so go test works on a new repo * [x/gov] Initial governance parameters can now be set in the genesis file * [x/stake] \#1815 Sped up the processing of `EditValidator` txs. +* [spec] Added simple piggy bank distribution spec BUG FIXES * \#1666 Add intra-tx counter to the genesis validators @@ -85,4 +86,4 @@ BUG FIXES * \#1666 Add intra-tx counter to the genesis validators * [tests] \#1551: Fixed invalid LCD test JSON payload in `doIBCTransfer` * \#1787 Fixed bug where Tally fails due to revoked/unbonding validator -* [basecoin] Fixes coin transaction failure and account query [discussion](https://forum.cosmos.network/t/unmarshalbinarybare-expected-to-read-prefix-bytes-75fbfab8-since-it-is-registered-concrete-but-got-0a141dfa/664/6) \ No newline at end of file +* [basecoin] Fixes coin transaction failure and account query [discussion](https://forum.cosmos.network/t/unmarshalbinarybare-expected-to-read-prefix-bytes-75fbfab8-since-it-is-registered-concrete-but-got-0a141dfa/664/6) From 3c564c01b911e51f2428452b3489b3303ac14b9c Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Thu, 16 Aug 2018 16:41:32 -0400 Subject: [PATCH 13/49] clearer names, missing commission logic --- docs/spec/distribution/state.md | 14 ++--- docs/spec/distribution/transactions.md | 83 ++++++++++++++------------ docs/spec/distribution/triggers.md | 4 +- 3 files changed, 55 insertions(+), 46 deletions(-) diff --git a/docs/spec/distribution/state.md b/docs/spec/distribution/state.md index 115782594..83e82f494 100644 --- a/docs/spec/distribution/state.md +++ b/docs/spec/distribution/state.md @@ -38,10 +38,10 @@ Validator distribution information for the relevant validator is updated each ti 3. any delegator withdraws from a validator, or 4. the validator withdraws it's commission. - - ValidatorDistribution: `0x02 | ValOwnerAddr -> amino(validatorDistribution)` + - ValidatorDistInfo: `0x02 | ValOperatorAddr -> amino(validatorDistribution)` ```golang -type ValidatorDistribution struct { +type ValidatorDistInfo struct { CommissionWithdrawalHeight int64 // last height this validator withdrew commission GlobalWithdrawalHeight int64 // last height this validator withdrew from the global pool @@ -60,10 +60,10 @@ properties change (aka bonded tokens etc.) its properties will remain constant and the delegator's _accumulation_ factor can be calculated passively knowing only the height of the last withdrawal and its current properties. - - DelegatorDistribution: ` 0x02 | DelegatorAddr | ValOwnerAddr -> amino(delegatorDist)` + - DelegatorDistInfo: ` 0x02 | DelegatorAddr | ValOperatorAddr -> amino(delegatorDist)` ```golang -type DelegatorDist struct { +type DelegatorDistInfo struct { WithdrawalHeight int64 // last time this delegation withdrew rewards } ``` @@ -76,14 +76,14 @@ Every instance that a validator: - is slashed, or - changes its commission rate, -information about the state change must be recorded as a `ValidatorUpdate`. +information about the state change to each validator must be recorded as a `ValidatorUpdate`. Each power change is indexed by validator and its block height. - - ValidatorUpdate: `0x03 | ValOwnerAddr | amino(Height) -> amino(ValidatorUpdate)` + - ValidatorUpdate: `0x03 | ValOperatorAddr | amino(Height) -> amino(ValidatorUpdate)` ```golang type ValidatorUpdate struct { Height int64 // block height of update - NewCommissionRate sdk.Dec // commission rate at this height + OldCommissionRate sdk.Dec // commission rate at this height } ``` diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index fbae152eb..364346ff4 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -8,12 +8,12 @@ triggered each with any change in individual delegations, such as an unbond, redelegation, or delegation of additional tokens to a specific validator. ```golang -type TxWithdrawDelegation struct { +type TxWithdrawDelegationRewards struct { delegatorAddr sdk.AccAddress withdrawAddr sdk.AccAddress // address to make the withdrawal to } -func WithdrawFromDelegator(delegatorAddr, withdrawAddr sdk.AccAddress) +func WithdrawDelegationRewards(delegatorAddr, withdrawAddr sdk.AccAddress) height = GetHeight() withdraw = GetDelegatorAllWithdraws(delegatorAddr, height) AddCoins(withdrawAddr, totalEntitlment.TruncateDecimal()) @@ -28,14 +28,23 @@ func GetDelegatorAllWithdraws(delegatorAddr sdk.AccAddress, height int64) DecCoi pool = stake.GetPool() global = GetGlobal() for delegation = range delegations - delDistr = GetDelegationDistribution(delegation.DelegatorAddr, + delInfo = GetDelegationDistInfo(delegation.DelegatorAddr, delegation.ValidatorAddr) - valDistr = GetValidatorDistribution(delegation.ValidatorAddr) + valInfo = GetValidatorDistInfo(delegation.ValidatorAddr) validator = GetValidator(delegation.ValidatorAddr) - global, ddWithdraw = delDistr.WithdrawRewards(global, valDistr, height, pool.BondedTokens, + // get all commission rate changes since last withdraw + vus = GetValidatorUpdates(delegation.ValidatorAddr, delInfo.WithdrawalHeight) + + for vu = range vus { + global, diWithdraw = delInfo.WithdrawRewards(global, valInfo, vu.Height, pool.BondedTokens, + validator.Tokens, validator.DelegatorShares, vu.OldCommissionRate) + withdraw += diWithdraw + } + + global, diWithdraw = delInfo.WithdrawRewards(global, valInfo, height, pool.BondedTokens, validator.Tokens, validator.DelegatorShares, validator.Commission) - withdraw += ddWithdraw + withdraw += diWithdraw SetGlobal(global) return withdraw @@ -51,24 +60,24 @@ transaction withdraws the validators commission fee, as well as any rewards earning on their self-delegation. ``` -type TxWithdrawValidator struct { +type TxWithdrawValidatorRewards struct { operatorAddr sdk.AccAddress // validator address to withdraw from withdrawAddr sdk.AccAddress // address to make the withdrawal to } -func WithdrawFromValidator(operatorAddr, withdrawAddr sdk.AccAddress) +func WithdrawValidatorRewards(operatorAddr, withdrawAddr sdk.AccAddress) height = GetHeight() global = GetGlobal() pool = GetPool() - ValDistr = GetValidatorDistribution(delegation.ValidatorAddr) + ValInfo = GetValidatorDistInfo(delegation.ValidatorAddr) validator = GetValidator(delegation.ValidatorAddr) // withdraw self-delegation withdraw = GetDelegatorAllWithdraws(validator.OperatorAddr, height) // withdrawal validator commission rewards - global, commission = valDistr.WithdrawCommission(global, valDistr, height, pool.BondedTokens, + global, commission = valInfo.WithdrawCommission(global, valInfo, height, pool.BondedTokens, validator.Tokens, validator.DelegatorShares, validator.Commission) withdraw += commission SetGlobal(global) @@ -101,10 +110,10 @@ the existing accum. This term is to be updated each time a withdrawal is made from a validator. ``` -func (vd ValidatorDistribution) UpdateTotalDelAccum(height int64, totalDelShares Dec) - blocks = height - vd.TotalDelAccumUpdateHeight - vd.TotalDelAccum += totalDelShares * blocks - vd.TotalDelAccumUpdateHeight = height +func (vi ValidatorDistInfo) UpdateTotalDelAccum(height int64, totalDelShares Dec) + blocks = height - vi.TotalDelAccumUpdateHeight + vi.TotalDelAccum += totalDelShares * blocks + vi.TotalDelAccumUpdateHeight = height ``` ### Global pool to validator pool @@ -114,18 +123,18 @@ proposer and receives new tokens - the relevant validator must move tokens from the passive global pool to their own pool. ``` -func (vd ValidatorDistribution) TakeAccum(g Global, height int64, totalBonded, vdTokens Dec) g Global +func (vi ValidatorDistInfo) TakeAccum(g Global, height int64, totalBonded, vdTokens Dec) g Global g.UpdateTotalValAccum(height, totalBondedShares) g.UpdateValAccum(height, totalBondedShares) // update the validators pool - blocks = height - vd.GlobalWithdrawalHeight - vd.GlobalWithdrawalHeight = height + blocks = height - vi.GlobalWithdrawalHeight + vi.GlobalWithdrawalHeight = height accum = blocks * vdTokens withdrawalTokens := g.Pool * accum / g.TotalValAccum g.TotalValAccum -= accumm - vd.Pool += withdrawalTokens + vi.Pool += withdrawalTokens g.Pool -= withdrawalTokens return g @@ -138,21 +147,21 @@ For delegations (including validator's self-delegation) all rewards from reward are subject to commission rate from the operator of the validator. ``` -func (dd DelegatorDist) WithdrawRewards(g Global, vd ValidatorDistribution, +func (di DelegatorDistInfo) WithdrawRewards(g Global, vi ValidatorDistInfo, height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn DecCoins) - vd.UpdateTotalDelAccum(height, totalDelShares) - g = vd.TakeAccum(g, height, totalBonded, vdTokens) + vi.UpdateTotalDelAccum(height, totalDelShares) + g = vi.TakeAccum(g, height, totalBonded, vdTokens) - blocks = height - dd.WithdrawalHeight - dd.WithdrawalHeight = height + blocks = height - di.WithdrawalHeight + di.WithdrawalHeight = height accum = delegatorShares * blocks * (1 - commissionRate) - withdrawalTokens := vd.Pool * accum / vd.TotalDelAccum - vd.TotalDelAccum -= accum + withdrawalTokens := vi.Pool * accum / vi.TotalDelAccum + vi.TotalDelAccum -= accum - vd.Pool -= withdrawalTokens - vd.TotalDelAccum -= accum + vi.Pool -= withdrawalTokens + vi.TotalDelAccum -= accum return g, withdrawalTokens ``` @@ -163,21 +172,21 @@ Similar to a delegator's entitlement, but with recipient shares based on the commission portion of bonded tokens. ``` -func (vd ValidatorDist) WithdrawCommission(g Global, vd ValidatorDistribution, - height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn DecCoins) +func (vi ValidatorDistInfo) WithdrawCommission(g Global, height int64, + totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn DecCoins) - vd.UpdateTotalDelAccum(height, totalDelShares) - g = vd.TakeAccum(g, height, totalBonded, vdTokens) + vi.UpdateTotalDelAccum(height, totalDelShares) + g = vi.TakeAccum(g, height, totalBonded, vdTokens) - blocks = height - vd.CommissionWithdrawalHeight - vd.CommissionWithdrawalHeight = height + blocks = height - vi.CommissionWithdrawalHeight + vi.CommissionWithdrawalHeight = height accum = delegatorShares * blocks * (commissionRate) - withdrawalTokens := vd.Pool * accum / vd.TotalDelAccum - vd.TotalDelAccum -= accum + withdrawalTokens := vi.Pool * accum / vi.TotalDelAccum + vi.TotalDelAccum -= accum - vd.Pool -= withdrawalTokens - vd.TotalDelAccum -= accum + vi.Pool -= withdrawalTokens + vi.TotalDelAccum -= accum return g, withdrawalTokens ``` diff --git a/docs/spec/distribution/triggers.md b/docs/spec/distribution/triggers.md index 0e02f0431..2938e674f 100644 --- a/docs/spec/distribution/triggers.md +++ b/docs/spec/distribution/triggers.md @@ -6,7 +6,7 @@ The pool of a new delegator bond will be 0 for the height at which the bond was added, or the withdrawal has taken place. This is achieved by setting -`DelegatorDist.WithdrawalHeight` to the height of the triggering transaction. +`DelegatorDistInfo.WithdrawalHeight` to the height of the triggering transaction. ## Commission rate change @@ -23,4 +23,4 @@ Additionally the change and associated height must be recorded in a Whenever a validator is slashed or enters/leaves the validator group all of the validator entitled reward tokens must be simultaniously withdrawn from -`Global.Pool` and added to `ValidatorDistribution.Pool` +`Global.Pool` and added to `ValidatorDistInfo.Pool` From 6e3e57e23e695680d8414509072c51abe1dccde9 Mon Sep 17 00:00:00 2001 From: Christopher Goes Date: Mon, 20 Aug 2018 17:50:13 +0200 Subject: [PATCH 14/49] Typos; tiny wording changes --- docs/spec/distribution/overview.md | 36 ++++++++++++-------------- docs/spec/distribution/state.md | 2 +- docs/spec/distribution/transactions.md | 14 +++++----- docs/spec/distribution/triggers.md | 4 +-- 4 files changed, 26 insertions(+), 30 deletions(-) diff --git a/docs/spec/distribution/overview.md b/docs/spec/distribution/overview.md index 62cec2e76..7b912f282 100644 --- a/docs/spec/distribution/overview.md +++ b/docs/spec/distribution/overview.md @@ -4,23 +4,23 @@ This _simple_ distribution mechanism describes a functional way to passively distribute rewards between validator and delegators. Note that this mechanism does -not distribute funds in as precisely as active reward distribution and will therefor +not distribute funds in as precisely as active reward distribution and will therefore be upgraded in the future. The mechanism operates as follows. Collected rewards are pooled globally and divided out passively to validators and delegators. Each validator has the opportunity to charge commission to the delegators on the rewards collected on -behalf of the delegators by the validators. Fees are paid directly into a +behalf of the delegators by the validators. Fees are paid directly into a global reward pool, and validator proposer-reward pool. Due to the nature of -passive accounting whenever changes to parameters which affect the rate of reward -distribution occurs, withdrawal of rewards must also occur when: - - - withdrawing one must withdrawal the maximum amount they are entitled - too, leaving nothing in the pool, - - bonding, unbonding, or re-delegating tokens to an existing account a +passive accounting, whenever changes to parameters which affect the rate of reward +distribution occurs, withdrawal of rewards must also occur. + + - Whenever withdrawing, one must withdraw the maximum amount they are entitled + too, leaving nothing in the pool. + - Whenever bonding, unbonding, or re-delegating tokens to an existing account, a full withdrawal of the rewards must occur (as the rules for lazy accounting - change), - - a validator chooses to change the commission on rewards, all accumulated + change). + - Whenever a validator chooses to change the commission on rewards, all accumulated commission rewards must be simultaneously withdrawn. The above scenarios are covered in `triggers.md`. @@ -33,10 +33,8 @@ following rewards between validators and associated delegators: - validator commission on all rewards earned by their delegators stake Fees are pooled within a global pool, as well as validator specific -proposer-reward pools. The mechanisms used allow for validators and delegators -to independently and lazily withdrawn their rewards. - -Within this spec +proposer-reward pools. The mechanisms used allow for validators and delegators +to independently and lazily withdraw their rewards. As a part of the lazy computations, each validator and delegator holds an accumulation term which is used to estimate what their approximate fair portion @@ -47,18 +45,16 @@ is not the case, the approximation of owed rewards will deviate from the active distribution based on fluctuations of incoming reward tokens as well as timing of reward withdrawal by other delegators and validators from the reward pool. - ## Affect on Staking Charging commission on Atom provisions while also allowing for Atom-provisions to be auto-bonded (distributed directly to the validators bonded stake) is -problematic within DPoS. Fundamentally these two mechnisms are mutually -exclusive. If there are atoms commissions and auto-bonding Atoms, the portion +problematic within DPoS. Fundamentally these two mechnisms are mutually +exclusive. If there are Atom commissions and auto-bonding Atoms, the portion of Atoms the reward distribution calculation would become very large as the Atom portion for each delegator would change each block making a withdrawal of rewards for a delegator require a calculation for every single block since the last -withdrawal. In conclusion we can only have atom commission and unbonded atoms -provisions, or bonded atom provisions with no Atom commission, and we elect to +withdrawal. In conclusion, we can only have Atom commission and unbonded atoms +provisions or bonded atom provisions with no Atom commission, and we elect to implement the former. Stakeholders wishing to rebond their provisions may elect to set up a script to periodically withdraw and rebond rewards. - diff --git a/docs/spec/distribution/state.md b/docs/spec/distribution/state.md index 83e82f494..7dc011e95 100644 --- a/docs/spec/distribution/state.md +++ b/docs/spec/distribution/state.md @@ -33,7 +33,7 @@ type Global struct { ### Validator Distribution Validator distribution information for the relevant validator is updated each time: - 1. delegation amount to a validator are updated, + 1. delegation amount to a validator is updated, 2. a validator successfully proposes a block and receives a reward, 3. any delegator withdraws from a validator, or 4. the validator withdraws it's commission. diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index 364346ff4..82f9e84f7 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -53,7 +53,7 @@ func GetDelegatorAllWithdraws(delegatorAddr sdk.AccAddress, height int64) DecCoi ## TxWithdrawValidator When a validator wishes to withdraw their rewards it must send -`TxWithdrawDelegation`. Note that parts of this transaction logic is also +`TxWithdrawValidatorRewards`. Note that parts of this transaction logic are also triggered each with any change in individual delegations, such as an unbond, redelegation, or delegation of additional tokens to a specific validator. This transaction withdraws the validators commission fee, as well as any rewards @@ -61,7 +61,7 @@ earning on their self-delegation. ``` type TxWithdrawValidatorRewards struct { - operatorAddr sdk.AccAddress // validator address to withdraw from + operatorAddr sdk.AccAddress // validator address to withdraw from withdrawAddr sdk.AccAddress // address to make the withdrawal to } @@ -92,7 +92,7 @@ func WithdrawValidatorRewards(operatorAddr, withdrawAddr sdk.AccAddress) The total amount of validator accum must be calculated in order to determine the amount of pool tokens which a validator is entitled to at a particular block. The accum is always additive to the existing accum. This term is to be -updates each time rewards are withdrawn from the system. +updated each time rewards are withdrawn from the system. ``` func (g Global) UpdateTotalValAccum(height int64, totalBondedTokens Dec) @@ -118,8 +118,8 @@ func (vi ValidatorDistInfo) UpdateTotalDelAccum(height int64, totalDelShares Dec ### Global pool to validator pool -Every time a validator or delegator make a withdraw or the validator is the -proposer and receives new tokens - the relevant validator must move tokens from +Every time a validator or delegator executes a withdrawal or the validator is the +proposer and receives new tokens, the relevant validator must move tokens from the passive global pool to their own pool. ``` @@ -141,7 +141,7 @@ func (vi ValidatorDistInfo) TakeAccum(g Global, height int64, totalBonded, vdTok ``` -### Delegation's withdrawal +### Delegation reward withdrawal For delegations (including validator's self-delegation) all rewards from reward pool are subject to commission rate from the operator of the validator. @@ -166,7 +166,7 @@ func (di DelegatorDistInfo) WithdrawRewards(g Global, vi ValidatorDistInfo, ``` -### Validators's commission withdrawal +### Validator commission withdrawal Similar to a delegator's entitlement, but with recipient shares based on the commission portion of bonded tokens. diff --git a/docs/spec/distribution/triggers.md b/docs/spec/distribution/triggers.md index 2938e674f..e3971ea99 100644 --- a/docs/spec/distribution/triggers.md +++ b/docs/spec/distribution/triggers.md @@ -22,5 +22,5 @@ Additionally the change and associated height must be recorded in a - triggered-by: `stake.Slash`, `stake.UpdateValidator` Whenever a validator is slashed or enters/leaves the validator group all of the -validator entitled reward tokens must be simultaniously withdrawn from -`Global.Pool` and added to `ValidatorDistInfo.Pool` +validator entitled reward tokens must be simultaneously withdrawn from +`Global.Pool` and added to `ValidatorDistInfo.Pool`. From b32285ec3b11aea53beb79d5fcd4c2488881d038 Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Thu, 23 Aug 2018 03:19:21 -0400 Subject: [PATCH 15/49] commission actively calculated, withdrawDelegationReward Tx --- docs/spec/distribution/end_block.md | 12 ++-- docs/spec/distribution/state.md | 25 +------ docs/spec/distribution/transactions.md | 92 +++++++++++++++----------- 3 files changed, 64 insertions(+), 65 deletions(-) diff --git a/docs/spec/distribution/end_block.md b/docs/spec/distribution/end_block.md index 19c9e8302..952d1832a 100644 --- a/docs/spec/distribution/end_block.md +++ b/docs/spec/distribution/end_block.md @@ -1,6 +1,6 @@ # End Block -At each endblock, the fees received are sorted to the proposer, community fund, +At each endblock, the fees received are allocated to the proposer, community fund, and global pool. When the validator is the proposer of the round, that validator (and their delegators) receives between 1% and 5% of fee rewards, the reserve community tax is then charged, then the remainder is distributed @@ -15,13 +15,17 @@ pool which validator holds individually (`ValidatorDistribution.ProvisionsRewardPool`). ``` -func SortFees(feesCollected sdk.Coins, global Global, proposer ValidatorDistribution, - sumPowerPrecommitValidators, totalBondedTokens, communityTax sdk.Dec) +func AllocateFees(feesCollected sdk.Coins, global Global, proposer ValidatorDistribution, + sumPowerPrecommitValidators, totalBondedTokens, communityTax, + proposerCommissionRate sdk.Dec) feesCollectedDec = MakeDecCoins(feesCollected) proposerReward = feesCollectedDec * (0.01 + 0.04 * sumPowerPrecommitValidators / totalBondedTokens) - proposer.ProposerPool += proposerReward + + commission = proposerReward * proposerCommissionRate + proposer.PoolCommission += commission + proposer.Pool += proposerReward - commission communityFunding = feesCollectedDec * communityTax global.CommunityFund += communityFunding diff --git a/docs/spec/distribution/state.md b/docs/spec/distribution/state.md index 7dc011e95..3e3669789 100644 --- a/docs/spec/distribution/state.md +++ b/docs/spec/distribution/state.md @@ -42,10 +42,9 @@ Validator distribution information for the relevant validator is updated each ti ```golang type ValidatorDistInfo struct { - CommissionWithdrawalHeight int64 // last height this validator withdrew commission - GlobalWithdrawalHeight int64 // last height this validator withdrew from the global pool - Pool DecCoins // reward pool collected held within this validator (includes proposer rewards) + Pool DecCoins // rewards owed to delegators, commission has already been charged (includes proposer reward) + PoolCommission DecCoins // commission collected by this validator (pending withdrawal) TotalDelAccumUpdateHeight int64 // last height which the total delegator accum was updated TotalDelAccum sdk.Dec // total proposer pool accumulation factor held by delegators @@ -67,23 +66,3 @@ type DelegatorDistInfo struct { WithdrawalHeight int64 // last time this delegation withdrew rewards } ``` - -### Validator Update - -Every instance that a validator: - - enters into the bonded state, - - leaves the bonded state, - - is slashed, or - - changes its commission rate, - -information about the state change to each validator must be recorded as a `ValidatorUpdate`. -Each power change is indexed by validator and its block height. - - - ValidatorUpdate: `0x03 | ValOperatorAddr | amino(Height) -> amino(ValidatorUpdate)` - -```golang -type ValidatorUpdate struct { - Height int64 // block height of update - OldCommissionRate sdk.Dec // commission rate at this height -} -``` diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index 82f9e84f7..2ed12d2c3 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -1,9 +1,9 @@ # Transactions -## TxWithdrawDelegation +## TxWithdrawDelegationRewards When a delegator wishes to withdraw their rewards it must send -`TxWithdrawDelegation`. Note that parts of this transaction logic are also +`TxWithdrawDelegationRewards`. Note that parts of this transaction logic are also triggered each with any change in individual delegations, such as an unbond, redelegation, or delegation of additional tokens to a specific validator. @@ -16,7 +16,7 @@ type TxWithdrawDelegationRewards struct { func WithdrawDelegationRewards(delegatorAddr, withdrawAddr sdk.AccAddress) height = GetHeight() withdraw = GetDelegatorAllWithdraws(delegatorAddr, height) - AddCoins(withdrawAddr, totalEntitlment.TruncateDecimal()) + AddCoins(withdrawAddr, withdraw.TruncateDecimal()) func GetDelegatorAllWithdraws(delegatorAddr sdk.AccAddress, height int64) DecCoins @@ -33,15 +33,6 @@ func GetDelegatorAllWithdraws(delegatorAddr sdk.AccAddress, height int64) DecCoi valInfo = GetValidatorDistInfo(delegation.ValidatorAddr) validator = GetValidator(delegation.ValidatorAddr) - // get all commission rate changes since last withdraw - vus = GetValidatorUpdates(delegation.ValidatorAddr, delInfo.WithdrawalHeight) - - for vu = range vus { - global, diWithdraw = delInfo.WithdrawRewards(global, valInfo, vu.Height, pool.BondedTokens, - validator.Tokens, validator.DelegatorShares, vu.OldCommissionRate) - withdraw += diWithdraw - } - global, diWithdraw = delInfo.WithdrawRewards(global, valInfo, height, pool.BondedTokens, validator.Tokens, validator.DelegatorShares, validator.Commission) withdraw += diWithdraw @@ -50,7 +41,38 @@ func GetDelegatorAllWithdraws(delegatorAddr sdk.AccAddress, height int64) DecCoi return withdraw ``` -## TxWithdrawValidator +## TxWithdrawDelegationReward + +under special circumstances a delegator may wish to withdraw rewards from only +a single validator. + +```golang +type TxWithdrawDelegationReward struct { + delegatorAddr sdk.AccAddress + validatorAddr sdk.AccAddress + withdrawAddr sdk.AccAddress // address to make the withdrawal to +} + +func WithdrawDelegationReward(delegatorAddr, validatorAddr, withdrawAddr sdk.AccAddress) + height = GetHeight() + + // get all distribution scenarios + pool = stake.GetPool() + global = GetGlobal() + delInfo = GetDelegationDistInfo(delegatorAddr, + validatorAddr) + valInfo = GetValidatorDistInfo(validatorAddr) + validator = GetValidator(validatorAddr) + + global, withdraw = delInfo.WithdrawRewards(global, valInfo, height, pool.BondedTokens, + validator.Tokens, validator.DelegatorShares, validator.Commission) + + SetGlobal(global) + AddCoins(withdrawAddr, withdraw.TruncateDecimal()) +``` + + +## TxWithdrawValidatorRewards When a validator wishes to withdraw their rewards it must send `TxWithdrawValidatorRewards`. Note that parts of this transaction logic are also @@ -78,11 +100,11 @@ func WithdrawValidatorRewards(operatorAddr, withdrawAddr sdk.AccAddress) // withdrawal validator commission rewards global, commission = valInfo.WithdrawCommission(global, valInfo, height, pool.BondedTokens, - validator.Tokens, validator.DelegatorShares, validator.Commission) + validator.Tokens, validator.Commission) withdraw += commission SetGlobal(global) - AddCoins(withdrawAddr, totalEntitlment.TruncateDecimal()) + AddCoins(withdrawAddr, withdraw.TruncateDecimal()) ``` ## Common calculations @@ -118,12 +140,13 @@ func (vi ValidatorDistInfo) UpdateTotalDelAccum(height int64, totalDelShares Dec ### Global pool to validator pool -Every time a validator or delegator executes a withdrawal or the validator is the -proposer and receives new tokens, the relevant validator must move tokens from -the passive global pool to their own pool. +Every time a validator or delegator executes a withdrawal or the validator is +the proposer and receives new tokens, the relevant validator must move tokens +from the passive global pool to their own pool. It is at this point that the +commission is withdrawn ``` -func (vi ValidatorDistInfo) TakeAccum(g Global, height int64, totalBonded, vdTokens Dec) g Global +func (vi ValidatorDistInfo) TakeAccum(g Global, height int64, totalBonded, vdTokens, commissionRate Dec) g Global g.UpdateTotalValAccum(height, totalBondedShares) g.UpdateValAccum(height, totalBondedShares) @@ -132,9 +155,11 @@ func (vi ValidatorDistInfo) TakeAccum(g Global, height int64, totalBonded, vdTok vi.GlobalWithdrawalHeight = height accum = blocks * vdTokens withdrawalTokens := g.Pool * accum / g.TotalValAccum + commission := withdrawalTokens * commissionRate g.TotalValAccum -= accumm - vi.Pool += withdrawalTokens + vi.PoolCommission += commission + vi.PoolCommissionFree += withdrawalTokens - commission g.Pool -= withdrawalTokens return g @@ -143,19 +168,19 @@ func (vi ValidatorDistInfo) TakeAccum(g Global, height int64, totalBonded, vdTok ### Delegation reward withdrawal -For delegations (including validator's self-delegation) all rewards from reward pool -are subject to commission rate from the operator of the validator. +For delegations (including validator's self-delegation) all rewards from reward +pool have already had the validator's commission taken away. ``` func (di DelegatorDistInfo) WithdrawRewards(g Global, vi ValidatorDistInfo, height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn DecCoins) vi.UpdateTotalDelAccum(height, totalDelShares) - g = vi.TakeAccum(g, height, totalBonded, vdTokens) + g = vi.TakeAccum(g, height, totalBonded, vdTokens, commissionRate) blocks = height - di.WithdrawalHeight di.WithdrawalHeight = height - accum = delegatorShares * blocks * (1 - commissionRate) + accum = delegatorShares * blocks withdrawalTokens := vi.Pool * accum / vi.TotalDelAccum vi.TotalDelAccum -= accum @@ -168,25 +193,16 @@ func (di DelegatorDistInfo) WithdrawRewards(g Global, vi ValidatorDistInfo, ### Validator commission withdrawal -Similar to a delegator's entitlement, but with recipient shares based on the -commission portion of bonded tokens. +Commission is calculated each time rewards enter into the validator. ``` func (vi ValidatorDistInfo) WithdrawCommission(g Global, height int64, - totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn DecCoins) + totalBonded, vdTokens, commissionRate Dec) (g Global, withdrawn DecCoins) - vi.UpdateTotalDelAccum(height, totalDelShares) - g = vi.TakeAccum(g, height, totalBonded, vdTokens) + g = vi.TakeAccum(g, height, totalBonded, vdTokens, commissionRate) - blocks = height - vi.CommissionWithdrawalHeight - vi.CommissionWithdrawalHeight = height - accum = delegatorShares * blocks * (commissionRate) - - withdrawalTokens := vi.Pool * accum / vi.TotalDelAccum - vi.TotalDelAccum -= accum - - vi.Pool -= withdrawalTokens - vi.TotalDelAccum -= accum + withdrawalTokens := vi.PoolCommission + vi.PoolCommission = 0 return g, withdrawalTokens ``` From cce4cbc9ec9bac54dc4d8b214b7efee585034ffc Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Thu, 23 Aug 2018 04:02:26 -0400 Subject: [PATCH 16/49] expand on shortcomings of the system --- docs/spec/distribution/overview.md | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/spec/distribution/overview.md b/docs/spec/distribution/overview.md index 7b912f282..e4308a21d 100644 --- a/docs/spec/distribution/overview.md +++ b/docs/spec/distribution/overview.md @@ -36,20 +36,32 @@ Fees are pooled within a global pool, as well as validator specific proposer-reward pools. The mechanisms used allow for validators and delegators to independently and lazily withdraw their rewards. -As a part of the lazy computations, each validator and delegator holds an -accumulation term which is used to estimate what their approximate fair portion -of tokens held in the global pool is owed to them. This approximation of owed -rewards would be equivalent to the active distribution under the situation that -there was a constant flow of incoming reward tokens every block. Because this -is not the case, the approximation of owed rewards will deviate from the active -distribution based on fluctuations of incoming reward tokens as well as timing -of reward withdrawal by other delegators and validators from the reward pool. +## Shortcomings + +As a part of the lazy computations, each delegator holds an accumulation term +specific to each validator which is used to estimate what their approximate +fair portion of tokens held in the global pool is owed to them. + +``` +entitlement = delegator-accumulation / all-delegators-accumulation +``` + +Under the circumstance that there were constant and equal flow of incoming +reward tokens every block, this distribution mechanism would be equal to the +active distribution (distribute individually to all delegators each block). +However this is unrealistic so deviations from the active distribution will +occur based on fluctuations of incoming reward tokens as well as timing of +reward withdrawal by other delegators. + +If you happen to know that incoming rewards are about significantly move up, +you are incentivized to not withdraw until after this event, increasing the +worth of your existing _accum_. ## Affect on Staking Charging commission on Atom provisions while also allowing for Atom-provisions to be auto-bonded (distributed directly to the validators bonded stake) is -problematic within DPoS. Fundamentally these two mechnisms are mutually +problematic within DPoS. Fundamentally these two mechanisms are mutually exclusive. If there are Atom commissions and auto-bonding Atoms, the portion of Atoms the reward distribution calculation would become very large as the Atom portion for each delegator would change each block making a withdrawal of rewards From 222597165e5ae84936da4d3393a7a39d7845173d Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Tue, 28 Aug 2018 18:23:57 -0400 Subject: [PATCH 17/49] trigger -> hooks --- docs/spec/distribution/{triggers.md => hooks.md} | 2 +- docs/spec/distribution/overview.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) rename docs/spec/distribution/{triggers.md => hooks.md} (98%) diff --git a/docs/spec/distribution/triggers.md b/docs/spec/distribution/hooks.md similarity index 98% rename from docs/spec/distribution/triggers.md rename to docs/spec/distribution/hooks.md index e3971ea99..e1a18ef59 100644 --- a/docs/spec/distribution/triggers.md +++ b/docs/spec/distribution/hooks.md @@ -1,4 +1,4 @@ -# Triggers +# Hooks ## Create or modify delegation distribution diff --git a/docs/spec/distribution/overview.md b/docs/spec/distribution/overview.md index e4308a21d..13a4ea3f1 100644 --- a/docs/spec/distribution/overview.md +++ b/docs/spec/distribution/overview.md @@ -23,7 +23,7 @@ distribution occurs, withdrawal of rewards must also occur. - Whenever a validator chooses to change the commission on rewards, all accumulated commission rewards must be simultaneously withdrawn. -The above scenarios are covered in `triggers.md`. +The above scenarios are covered in `hooks.md`. The distribution mechanism outlines herein is used to lazily distribute the following rewards between validators and associated delegators: From 46bbada4eea507426a9909491279ceb5564522a5 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Wed, 29 Aug 2018 23:02:15 -0700 Subject: [PATCH 18/49] simulation: Add benchmarking --- PENDING.md | 1 + cmd/gaia/app/sim_test.go | 21 +++ x/bank/simulation/msgs.go | 18 +-- x/gov/simulation/msgs.go | 50 +++--- x/mock/simulation/random_simulate_blocks.go | 166 +++++++++++++++----- x/mock/simulation/types.go | 2 +- x/slashing/simulation/msgs.go | 8 +- x/stake/simulation/msgs.go | 44 ++++-- 8 files changed, 220 insertions(+), 90 deletions(-) diff --git a/PENDING.md b/PENDING.md index f45541ccd..2e862ead6 100644 --- a/PENDING.md +++ b/PENDING.md @@ -45,6 +45,7 @@ FEATURES * SDK * [querier] added custom querier functionality, so ABCI query requests can be handled by keepers * [simulation] \#1924 allow operations to specify future operations + * [simulation] Add benchmarking capabilities * Tendermint diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go index 0ca936bdb..2bad6aba2 100644 --- a/cmd/gaia/app/sim_test.go +++ b/cmd/gaia/app/sim_test.go @@ -112,6 +112,27 @@ func invariants(app *GaiaApp) []simulation.Invariant { } } +// Profile with: +// /usr/local/go/bin/go test -benchmem -run=^$ github.com/cosmos/cosmos-sdk/cmd/gaia/app -bench ^BenchmarkFullGaiaSimulation$ -cpuprofile cpu.out +func BenchmarkFullGaiaSimulation(b *testing.B) { + // Setup Gaia application + var logger log.Logger + logger = log.NewNopLogger() + db := dbm.NewMemDB() + app := NewGaiaApp(logger, db, nil) + + // Run randomized simulation + // TODO parameterize numbers, save for a later PR + simulation.BenchmarkSimulationFromSeed( + b, app.BaseApp, appStateFn, seed, + testAndRunTxs(app), + []simulation.RandSetup{}, + 10, + 100, + false, + ) +} + func TestFullGaiaSimulation(t *testing.T) { if !enabled { t.Skip("Skipping Gaia simulation") diff --git a/x/bank/simulation/msgs.go b/x/bank/simulation/msgs.go index e3f3ab77b..c53916a61 100644 --- a/x/bank/simulation/msgs.go +++ b/x/bank/simulation/msgs.go @@ -21,7 +21,7 @@ import ( // SimulateSingleInputMsgSend tests and runs a single msg send, with one input and one output, where both // accounts already exist. func SimulateSingleInputMsgSend(mapper auth.AccountMapper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOps []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOps []simulation.FutureOperation, err sdk.Error) { fromKey := simulation.RandomKey(r, keys) fromAddr := sdk.AccAddress(fromKey.PubKey().Address()) toKey := simulation.RandomKey(r, keys) @@ -58,7 +58,7 @@ func SimulateSingleInputMsgSend(mapper auth.AccountMapper) simulation.Operation Inputs: []bank.Input{bank.NewInput(fromAddr, coins)}, Outputs: []bank.Output{bank.NewOutput(toAddr, coins)}, } - sendAndVerifyMsgSend(t, app, mapper, msg, ctx, log, []crypto.PrivKey{fromKey}) + sendAndVerifyMsgSend(tb, app, mapper, msg, ctx, log, []crypto.PrivKey{fromKey}) event("bank/sendAndVerifyMsgSend/ok") return action, nil, nil @@ -66,7 +66,7 @@ func SimulateSingleInputMsgSend(mapper auth.AccountMapper) simulation.Operation } // Sends and verifies the transition of a msg send. This fails if there are repeated inputs or outputs -func sendAndVerifyMsgSend(t *testing.T, app *baseapp.BaseApp, mapper auth.AccountMapper, msg bank.MsgSend, ctx sdk.Context, log string, privkeys []crypto.PrivKey) { +func sendAndVerifyMsgSend(tb testing.TB, app *baseapp.BaseApp, mapper auth.AccountMapper, msg bank.MsgSend, ctx sdk.Context, log string, privkeys []crypto.PrivKey) { initialInputAddrCoins := make([]sdk.Coins, len(msg.Inputs)) initialOutputAddrCoins := make([]sdk.Coins, len(msg.Outputs)) AccountNumbers := make([]int64, len(msg.Inputs)) @@ -91,12 +91,12 @@ func sendAndVerifyMsgSend(t *testing.T, app *baseapp.BaseApp, mapper auth.Accoun // TODO: Do this in a more 'canonical' way fmt.Println(res) fmt.Println(log) - t.FailNow() + tb.FailNow() } for i := 0; i < len(msg.Inputs); i++ { terminalInputCoins := mapper.GetAccount(ctx, msg.Inputs[i].Address).GetCoins() - require.Equal(t, + require.Equal(tb, initialInputAddrCoins[i].Minus(msg.Inputs[i].Coins), terminalInputCoins, fmt.Sprintf("Input #%d had an incorrect amount of coins\n%s", i, log), @@ -104,11 +104,9 @@ func sendAndVerifyMsgSend(t *testing.T, app *baseapp.BaseApp, mapper auth.Accoun } for i := 0; i < len(msg.Outputs); i++ { terminalOutputCoins := mapper.GetAccount(ctx, msg.Outputs[i].Address).GetCoins() - require.Equal(t, - initialOutputAddrCoins[i].Plus(msg.Outputs[i].Coins), - terminalOutputCoins, - fmt.Sprintf("Output #%d had an incorrect amount of coins\n%s", i, log), - ) + if !terminalOutputCoins.IsEqual(initialOutputAddrCoins[i].Plus(msg.Outputs[i].Coins)) { + tb.Fatalf("Output #%d had an incorrect amount of coins\n%s", i, log) + } } } diff --git a/x/gov/simulation/msgs.go b/x/gov/simulation/msgs.go index 0b1530138..eca8accae 100644 --- a/x/gov/simulation/msgs.go +++ b/x/gov/simulation/msgs.go @@ -5,8 +5,6 @@ import ( "math/rand" "testing" - "github.com/stretchr/testify/require" - "github.com/tendermint/tendermint/crypto" "github.com/cosmos/cosmos-sdk/baseapp" @@ -23,24 +21,15 @@ const ( // SimulateMsgSubmitProposal simulates a msg Submit Proposal // Note: Currently doesn't ensure that the proposal txt is in JSON form func SimulateMsgSubmitProposal(k gov.Keeper, sk stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOps []simulation.FutureOperation, err sdk.Error) { - key := simulation.RandomKey(r, keys) - addr := sdk.AccAddress(key.PubKey().Address()) - deposit := randomDeposit(r) - msg := gov.NewMsgSubmitProposal( - simulation.RandStringOfLength(r, 5), - simulation.RandStringOfLength(r, 5), - gov.ProposalTypeText, - addr, - deposit, - ) - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + handler := gov.NewHandler(k) + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOps []simulation.FutureOperation, err sdk.Error) { + msg := simulationCreateMsgSubmitProposal(tb, r, keys, log) ctx, write := ctx.CacheContext() - result := gov.NewHandler(k)(ctx, msg) + result := handler(ctx, msg) if result.IsOK() { // Update pool to keep invariants pool := sk.GetPool(ctx) - pool.LooseTokens = pool.LooseTokens.Sub(sdk.NewDecFromInt(deposit.AmountOf(denom))) + pool.LooseTokens = pool.LooseTokens.Sub(sdk.NewDecFromInt(msg.InitialDeposit.AmountOf(denom))) sk.SetPool(ctx, pool) write() } @@ -50,9 +39,26 @@ func SimulateMsgSubmitProposal(k gov.Keeper, sk stake.Keeper) simulation.Operati } } +func simulationCreateMsgSubmitProposal(tb testing.TB, r *rand.Rand, keys []crypto.PrivKey, log string) gov.MsgSubmitProposal { + key := simulation.RandomKey(r, keys) + addr := sdk.AccAddress(key.PubKey().Address()) + deposit := randomDeposit(r) + msg := gov.NewMsgSubmitProposal( + simulation.RandStringOfLength(r, 5), + simulation.RandStringOfLength(r, 5), + gov.ProposalTypeText, + addr, + deposit, + ) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } + return msg +} + // SimulateMsgDeposit func SimulateMsgDeposit(k gov.Keeper, sk stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { key := simulation.RandomKey(r, keys) addr := sdk.AccAddress(key.PubKey().Address()) proposalID, ok := randomProposalID(r, k, ctx) @@ -61,7 +67,9 @@ func SimulateMsgDeposit(k gov.Keeper, sk stake.Keeper) simulation.Operation { } deposit := randomDeposit(r) msg := gov.NewMsgDeposit(addr, proposalID, deposit) - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := gov.NewHandler(k)(ctx, msg) if result.IsOK() { @@ -79,7 +87,7 @@ func SimulateMsgDeposit(k gov.Keeper, sk stake.Keeper) simulation.Operation { // SimulateMsgVote func SimulateMsgVote(k gov.Keeper, sk stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { key := simulation.RandomKey(r, keys) addr := sdk.AccAddress(key.PubKey().Address()) proposalID, ok := randomProposalID(r, k, ctx) @@ -88,7 +96,9 @@ func SimulateMsgVote(k gov.Keeper, sk stake.Keeper) simulation.Operation { } option := randomVotingOption(r) msg := gov.NewMsgVote(addr, proposalID, option) - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := gov.NewHandler(k)(ctx, msg) if result.IsOK() { diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index 995013ef8..aa4df613d 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -5,6 +5,7 @@ import ( "fmt" "math" "math/rand" + "os" "sort" "testing" "time" @@ -28,6 +29,26 @@ func Simulate( SimulateFromSeed(t, app, appStateFn, time, ops, setups, invariants, numBlocks, blockSize, commit) } +func initChain(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress, setups []RandSetup, app *baseapp.BaseApp, + appStateFn func(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json.RawMessage) (validators map[string]mockValidator) { + res := app.InitChain(abci.RequestInitChain{AppStateBytes: appStateFn(r, keys, accs)}) + validators = make(map[string]mockValidator) + for _, validator := range res.Validators { + validators[string(validator.Address)] = mockValidator{validator, GetMemberOfInitialState(r, initialLivenessWeightings)} + } + + for i := 0; i < len(setups); i++ { + setups[i](r, keys) + } + + return +} + +func randTimestamp(r *rand.Rand) time.Time { + unixTime := r.Int63n(int64(math.Pow(2, 40))) + return time.Unix(unixTime, 0) +} + // SimulateFromSeed tests an application by running the provided // operations, testing the provided invariants, but using the provided seed. func SimulateFromSeed( @@ -36,11 +57,7 @@ func SimulateFromSeed( ) { log := fmt.Sprintf("Starting SimulateFromSeed with randomness created with seed %d", int(seed)) r := rand.New(rand.NewSource(seed)) - - unixTime := r.Int63n(int64(math.Pow(2, 40))) - - // Set the timestamp for simulation - timestamp := time.Unix(unixTime, 0) + timestamp := randTimestamp(r) log = fmt.Sprintf("%s\nStarting the simulation from time %v, unixtime %v", log, timestamp.UTC().Format(time.UnixDate), timestamp.Unix()) fmt.Printf("%s\n", log) timeDiff := maxTimePerBlock - minTimePerBlock @@ -54,30 +71,18 @@ func SimulateFromSeed( events[what]++ } - res := app.InitChain(abci.RequestInitChain{AppStateBytes: appStateFn(r, keys, accs)}) - validators := make(map[string]mockValidator) - for _, validator := range res.Validators { - validators[string(validator.Address)] = mockValidator{validator, GetMemberOfInitialState(r, initialLivenessWeightings)} - } - - for i := 0; i < len(setups); i++ { - setups[i](r, keys) - } + validators := initChain(r, keys, accs, setups, app, appStateFn) header := abci.Header{Height: 0, Time: timestamp} opCount := 0 request := abci.RequestBeginBlock{Header: header} - var pastTimes []time.Time + var lastHeaderTime time.Time // These are operations which have been queued by previous operations operationQueue := make(map[int][]Operation) for i := 0; i < numBlocks; i++ { - - // Log the header time for future lookup - pastTimes = append(pastTimes, header.Time) - // Run the BeginBlock handler app.BeginBlock(request) @@ -88,16 +93,8 @@ func SimulateFromSeed( ctx := app.NewContext(false, header) - var thisBlockSize int - load := r.Float64() - switch { - case load < 0.33: - thisBlockSize = 0 - case load < 0.66: - thisBlockSize = r.Intn(blockSize * 2) - default: - thisBlockSize = r.Intn(blockSize * 4) - } + thisBlockSize := getBlockSize(r, blockSize) + // Run queued operations. Ignores blocksize if blocksize is too small log, numQueuedOpsRan := runQueuedOperations(operationQueue, int(header.Height), t, r, app, ctx, keys, log, event) opCount += numQueuedOpsRan @@ -119,6 +116,7 @@ func SimulateFromSeed( res := app.EndBlock(abci.RequestEndBlock{}) header.Height++ + lastHeaderTime = header.Time header.Time = header.Time.Add(time.Duration(minTimePerBlock) * time.Second).Add(time.Duration(int64(r.Intn(int(timeDiff)))) * time.Second) log += "\nEndBlock" @@ -131,7 +129,7 @@ func SimulateFromSeed( } // Generate a random RequestBeginBlock with the current validator set for the next block - request = RandomRequestBeginBlock(t, r, validators, livenessTransitionMatrix, evidenceFraction, pastTimes, event, header, log) + request = RandomRequestBeginBlock(r, validators, livenessTransitionMatrix, evidenceFraction, lastHeaderTime, event, header, log) // Update the validator set validators = updateValidators(t, r, validators, res.ValidatorUpdates, event) @@ -141,6 +139,84 @@ func SimulateFromSeed( DisplayEvents(events) } +func getBlockSize(r *rand.Rand, blockSize int) int { + load := r.Float64() + switch { + case load < 0.33: + return 0 + case load < 0.66: + return r.Intn(blockSize * 2) + default: + return r.Intn(blockSize * 4) + } +} + +// Simulate from seed, benchmarks +func BenchmarkSimulationFromSeed(b *testing.B, app *baseapp.BaseApp, appStateFn func(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json.RawMessage, + seed int64, ops []Operation, setups []RandSetup, numBlocks int, blockSize int, commit bool) { + r := rand.New(rand.NewSource(seed)) + timestamp := randTimestamp(r) + timeDiff := maxTimePerBlock - minTimePerBlock + keys, accs := mock.GeneratePrivKeyAddressPairsFromRand(r, numKeys) + + // Setup event stats + events := make(map[string]uint) + event := func(what string) { + events[what]++ + } + + validators := initChain(r, keys, accs, setups, app, appStateFn) + + header := abci.Header{Height: 0, Time: timestamp} + opCount := 0 + + request := abci.RequestBeginBlock{Header: header} + + var lastHeaderTime time.Time + // These are operations which have been queued by previous operations + operationQueue := make(map[int][]Operation) + b.ResetTimer() + + for i := 0; i < numBlocks; i++ { + + // Run the BeginBlock handler + app.BeginBlock(request) + + ctx := app.NewContext(false, header) + + thisBlockSize := getBlockSize(r, blockSize) + + // Run queued operations. Ignores blocksize if blocksize is too small + log, numQueuedOpsRan := runQueuedOperations(operationQueue, int(header.Height), b, r, app, ctx, keys, "", event) + opCount += numQueuedOpsRan + thisBlockSize -= numQueuedOpsRan + for j := 0; j < thisBlockSize; j++ { + _, futureOps, err := ops[r.Intn(len(ops))](b, r, app, ctx, keys, "", event) + queueOperations(operationQueue, futureOps) + if err != nil { + b.Fatalf("error on operation %d, %v", opCount, err) + } + opCount++ + } + + res := app.EndBlock(abci.RequestEndBlock{}) + header.Height++ + lastHeaderTime = header.Time + header.Time = header.Time.Add(time.Duration(minTimePerBlock) * time.Second).Add(time.Duration(int64(r.Intn(int(timeDiff)))) * time.Second) + if commit { + app.Commit() + } + + // Generate a random RequestBeginBlock with the current validator set for the next block + request = RandomRequestBeginBlock(r, validators, livenessTransitionMatrix, evidenceFraction, lastHeaderTime, event, header, log) + + // Update the validator set + validators = updateValidators(b, r, validators, res.ValidatorUpdates, event) + } + DisplayEvents(events) + fmt.Printf("Benchmark simulation ran %d operations\n", opCount) +} + // adds all future operations into the operation queue. func queueOperations(queuedOperations map[int][]Operation, futureOperations []FutureOperation) { if futureOperations == nil { @@ -155,7 +231,7 @@ func queueOperations(queuedOperations map[int][]Operation, futureOperations []Fu } } -func runQueuedOperations(queueOperations map[int][]Operation, height int, t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, +func runQueuedOperations(queueOperations map[int][]Operation, height int, tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, privKeys []crypto.PrivKey, log string, event func(string)) (updatedLog string, numOpsRan int) { updatedLog = log if queuedOps, ok := queueOperations[height]; ok { @@ -164,9 +240,12 @@ func runQueuedOperations(queueOperations map[int][]Operation, height int, t *tes // For now, queued operations cannot queue more operations. // If a need arises for us to support queued messages to queue more messages, this can // be changed. - logUpdate, _, err := queuedOps[i](t, r, app, ctx, privKeys, updatedLog, event) - updatedLog += "\n" + logUpdate - require.Nil(t, err, updatedLog) + logUpdate, _, err := queuedOps[i](tb, r, app, ctx, privKeys, updatedLog, event) + updatedLog = fmt.Sprintf("%s\n%s", updatedLog, logUpdate) + if err != nil { + fmt.Fprint(os.Stderr, updatedLog) + tb.FailNow() + } } delete(queueOperations, height) return updatedLog, numOps @@ -186,8 +265,8 @@ func getKeys(validators map[string]mockValidator) []string { } // RandomRequestBeginBlock generates a list of signing validators according to the provided list of validators, signing fraction, and evidence fraction -func RandomRequestBeginBlock(t *testing.T, r *rand.Rand, validators map[string]mockValidator, livenessTransitions TransitionMatrix, evidenceFraction float64, - pastTimes []time.Time, event func(string), header abci.Header, log string) abci.RequestBeginBlock { +func RandomRequestBeginBlock(r *rand.Rand, validators map[string]mockValidator, livenessTransitions TransitionMatrix, evidenceFraction float64, + lastHeaderTime time.Time, event func(string), header abci.Header, log string) abci.RequestBeginBlock { if len(validators) == 0 { return abci.RequestBeginBlock{Header: header} } @@ -219,13 +298,14 @@ func RandomRequestBeginBlock(t *testing.T, r *rand.Rand, validators map[string]m } i++ } + // TODO: Determine capacity before allocation evidence := make([]abci.Evidence, 0) for r.Float64() < evidenceFraction { height := header.Height time := header.Time if r.Float64() < pastEvidenceFraction { height = int64(r.Intn(int(header.Height))) - time = pastTimes[height] + time = lastHeaderTime } validator := signingValidators[r.Intn(len(signingValidators))].Validator var currentTotalVotingPower int64 @@ -258,11 +338,19 @@ func AssertAllInvariants(t *testing.T, app *baseapp.BaseApp, tests []Invariant, } // updateValidators mimicks Tendermint's update logic -func updateValidators(t *testing.T, r *rand.Rand, current map[string]mockValidator, updates []abci.Validator, event func(string)) map[string]mockValidator { +func updateValidators(tb testing.TB, r *rand.Rand, current map[string]mockValidator, updates []abci.Validator, event func(string)) map[string]mockValidator { for _, update := range updates { switch { case update.Power == 0: - require.NotNil(t, current[string(update.PubKey.Data)], "tried to delete a nonexistent validator") + // // TEMPORARY DEBUG CODE TO PROVE THAT THE OLD METHOD WAS BROKEN + // // (i.e. didn't catch in the event of problem) + // if val, ok := tb.(*testing.T); ok { + // require.NotNil(val, current[string(update.PubKey.Data)]) + // } + // // CORRECT CHECK + // if _, ok := current[string(update.PubKey.Data)]; !ok { + // tb.Fatalf("tried to delete a nonexistent validator") + // } event("endblock/validatorupdates/kicked") delete(current, string(update.PubKey.Data)) default: diff --git a/x/mock/simulation/types.go b/x/mock/simulation/types.go index 2516e07ae..2f91a4f26 100644 --- a/x/mock/simulation/types.go +++ b/x/mock/simulation/types.go @@ -23,7 +23,7 @@ type ( // Operations can optionally provide a list of "FutureOperations" to run later // These will be ran at the beginning of the corresponding block. Operation func( - t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, + tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, privKeys []crypto.PrivKey, log string, event func(string), ) (action string, futureOperations []FutureOperation, err sdk.Error) diff --git a/x/slashing/simulation/msgs.go b/x/slashing/simulation/msgs.go index e4900889e..7b5490e42 100644 --- a/x/slashing/simulation/msgs.go +++ b/x/slashing/simulation/msgs.go @@ -5,8 +5,6 @@ import ( "math/rand" "testing" - "github.com/stretchr/testify/require" - "github.com/tendermint/tendermint/crypto" "github.com/cosmos/cosmos-sdk/baseapp" @@ -17,11 +15,13 @@ import ( // SimulateMsgUnjail func SimulateMsgUnjail(k slashing.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { key := simulation.RandomKey(r, keys) address := sdk.AccAddress(key.PubKey().Address()) msg := slashing.NewMsgUnjail(address) - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := slashing.NewHandler(k)(ctx, msg) if result.IsOK() { diff --git a/x/stake/simulation/msgs.go b/x/stake/simulation/msgs.go index 4280e70c1..bc1063494 100644 --- a/x/stake/simulation/msgs.go +++ b/x/stake/simulation/msgs.go @@ -5,8 +5,6 @@ import ( "math/rand" "testing" - "github.com/stretchr/testify/require" - "github.com/cosmos/cosmos-sdk/baseapp" sdk "github.com/cosmos/cosmos-sdk/types" "github.com/cosmos/cosmos-sdk/x/auth" @@ -19,7 +17,7 @@ import ( // SimulateMsgCreateValidator func SimulateMsgCreateValidator(m auth.AccountMapper, k stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { denom := k.GetParams(ctx).BondDenom description := stake.Description{ Moniker: simulation.RandStringOfLength(r, 10), @@ -41,7 +39,9 @@ func SimulateMsgCreateValidator(m auth.AccountMapper, k stake.Keeper) simulation PubKey: pubkey, Delegation: sdk.NewCoin(denom, amount), } - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := stake.NewHandler(k)(ctx, msg) if result.IsOK() { @@ -56,7 +56,7 @@ func SimulateMsgCreateValidator(m auth.AccountMapper, k stake.Keeper) simulation // SimulateMsgEditValidator func SimulateMsgEditValidator(k stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { description := stake.Description{ Moniker: simulation.RandStringOfLength(r, 10), Identity: simulation.RandStringOfLength(r, 10), @@ -70,7 +70,9 @@ func SimulateMsgEditValidator(k stake.Keeper) simulation.Operation { Description: description, ValidatorAddr: address, } - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := stake.NewHandler(k)(ctx, msg) if result.IsOK() { @@ -84,7 +86,7 @@ func SimulateMsgEditValidator(k stake.Keeper) simulation.Operation { // SimulateMsgDelegate func SimulateMsgDelegate(m auth.AccountMapper, k stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { denom := k.GetParams(ctx).BondDenom validatorKey := simulation.RandomKey(r, keys) validatorAddress := sdk.AccAddress(validatorKey.PubKey().Address()) @@ -102,7 +104,9 @@ func SimulateMsgDelegate(m auth.AccountMapper, k stake.Keeper) simulation.Operat ValidatorAddr: validatorAddress, Delegation: sdk.NewCoin(denom, amount), } - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := stake.NewHandler(k)(ctx, msg) if result.IsOK() { @@ -116,7 +120,7 @@ func SimulateMsgDelegate(m auth.AccountMapper, k stake.Keeper) simulation.Operat // SimulateMsgBeginUnbonding func SimulateMsgBeginUnbonding(m auth.AccountMapper, k stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { denom := k.GetParams(ctx).BondDenom validatorKey := simulation.RandomKey(r, keys) validatorAddress := sdk.AccAddress(validatorKey.PubKey().Address()) @@ -134,7 +138,9 @@ func SimulateMsgBeginUnbonding(m auth.AccountMapper, k stake.Keeper) simulation. ValidatorAddr: validatorAddress, SharesAmount: sdk.NewDecFromInt(amount), } - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := stake.NewHandler(k)(ctx, msg) if result.IsOK() { @@ -148,7 +154,7 @@ func SimulateMsgBeginUnbonding(m auth.AccountMapper, k stake.Keeper) simulation. // SimulateMsgCompleteUnbonding func SimulateMsgCompleteUnbonding(k stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { validatorKey := simulation.RandomKey(r, keys) validatorAddress := sdk.AccAddress(validatorKey.PubKey().Address()) delegatorKey := simulation.RandomKey(r, keys) @@ -157,7 +163,9 @@ func SimulateMsgCompleteUnbonding(k stake.Keeper) simulation.Operation { DelegatorAddr: delegatorAddress, ValidatorAddr: validatorAddress, } - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := stake.NewHandler(k)(ctx, msg) if result.IsOK() { @@ -171,7 +179,7 @@ func SimulateMsgCompleteUnbonding(k stake.Keeper) simulation.Operation { // SimulateMsgBeginRedelegate func SimulateMsgBeginRedelegate(m auth.AccountMapper, k stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { denom := k.GetParams(ctx).BondDenom sourceValidatorKey := simulation.RandomKey(r, keys) sourceValidatorAddress := sdk.AccAddress(sourceValidatorKey.PubKey().Address()) @@ -193,7 +201,9 @@ func SimulateMsgBeginRedelegate(m auth.AccountMapper, k stake.Keeper) simulation ValidatorDstAddr: destValidatorAddress, SharesAmount: sdk.NewDecFromInt(amount), } - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := stake.NewHandler(k)(ctx, msg) if result.IsOK() { @@ -207,7 +217,7 @@ func SimulateMsgBeginRedelegate(m auth.AccountMapper, k stake.Keeper) simulation // SimulateMsgCompleteRedelegate func SimulateMsgCompleteRedelegate(k stake.Keeper) simulation.Operation { - return func(t *testing.T, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { + return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { validatorSrcKey := simulation.RandomKey(r, keys) validatorSrcAddress := sdk.AccAddress(validatorSrcKey.PubKey().Address()) validatorDstKey := simulation.RandomKey(r, keys) @@ -219,7 +229,9 @@ func SimulateMsgCompleteRedelegate(k stake.Keeper) simulation.Operation { ValidatorSrcAddr: validatorSrcAddress, ValidatorDstAddr: validatorDstAddress, } - require.Nil(t, msg.ValidateBasic(), "expected msg to pass ValidateBasic: %s", msg.GetSignBytes()) + if msg.ValidateBasic() != nil { + tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) + } ctx, write := ctx.CacheContext() result := stake.NewHandler(k)(ctx, msg) if result.IsOK() { From 31ca2e058b9d666ac3ea60ad5b399f3f9da28da2 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Wed, 29 Aug 2018 23:11:14 -0700 Subject: [PATCH 19/49] Update PENDING --- PENDING.md | 2 +- x/mock/simulation/random_simulate_blocks.go | 10 ++-------- 2 files changed, 3 insertions(+), 9 deletions(-) diff --git a/PENDING.md b/PENDING.md index 2e862ead6..53c7a6b34 100644 --- a/PENDING.md +++ b/PENDING.md @@ -45,7 +45,7 @@ FEATURES * SDK * [querier] added custom querier functionality, so ABCI query requests can be handled by keepers * [simulation] \#1924 allow operations to specify future operations - * [simulation] Add benchmarking capabilities + * [simulation] \#1924 Add benchmarking capabilities * Tendermint diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index aa4df613d..ea0078853 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -85,14 +85,12 @@ func SimulateFromSeed( for i := 0; i < numBlocks; i++ { // Run the BeginBlock handler app.BeginBlock(request) - log += "\nBeginBlock" // Make sure invariants hold at beginning of block AssertAllInvariants(t, app, invariants, log) ctx := app.NewContext(false, header) - thisBlockSize := getBlockSize(r, blockSize) // Run queued operations. Ignores blocksize if blocksize is too small @@ -102,9 +100,10 @@ func SimulateFromSeed( for j := 0; j < thisBlockSize; j++ { logUpdate, futureOps, err := ops[r.Intn(len(ops))](t, r, app, ctx, keys, log, event) log += "\n" + logUpdate + require.Nil(t, err, log) + queueOperations(operationQueue, futureOps) - require.Nil(t, err, log) if onOperation { AssertAllInvariants(t, app, invariants, log) } @@ -118,12 +117,10 @@ func SimulateFromSeed( header.Height++ lastHeaderTime = header.Time header.Time = header.Time.Add(time.Duration(minTimePerBlock) * time.Second).Add(time.Duration(int64(r.Intn(int(timeDiff)))) * time.Second) - log += "\nEndBlock" // Make sure invariants hold at end of block AssertAllInvariants(t, app, invariants, log) - if commit { app.Commit() } @@ -178,12 +175,9 @@ func BenchmarkSimulationFromSeed(b *testing.B, app *baseapp.BaseApp, appStateFn b.ResetTimer() for i := 0; i < numBlocks; i++ { - // Run the BeginBlock handler app.BeginBlock(request) - ctx := app.NewContext(false, header) - thisBlockSize := getBlockSize(r, blockSize) // Run queued operations. Ignores blocksize if blocksize is too small From 03d2f7331b418d68f0b3e4f5448754e4c27771f5 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Thu, 30 Aug 2018 00:13:31 -0700 Subject: [PATCH 20/49] Remove code duplication --- cmd/gaia/app/sim_test.go | 3 +- x/mock/simulation/random_simulate_blocks.go | 125 +++++++------------- x/stake/simulation/msgs.go | 21 ++-- 3 files changed, 61 insertions(+), 88 deletions(-) diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go index 2bad6aba2..1fd07b330 100644 --- a/cmd/gaia/app/sim_test.go +++ b/cmd/gaia/app/sim_test.go @@ -123,10 +123,11 @@ func BenchmarkFullGaiaSimulation(b *testing.B) { // Run randomized simulation // TODO parameterize numbers, save for a later PR - simulation.BenchmarkSimulationFromSeed( + simulation.SimulateFromSeed( b, app.BaseApp, appStateFn, seed, testAndRunTxs(app), []simulation.RandSetup{}, + invariants(app), // these shouldn't get ran 10, 100, false, diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index ea0078853..6376d53dc 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -17,7 +17,6 @@ import ( "github.com/cosmos/cosmos-sdk/baseapp" sdk "github.com/cosmos/cosmos-sdk/types" "github.com/cosmos/cosmos-sdk/x/mock" - "github.com/stretchr/testify/require" ) // Simulate tests application by sending random messages. @@ -52,13 +51,22 @@ func randTimestamp(r *rand.Rand) time.Time { // SimulateFromSeed tests an application by running the provided // operations, testing the provided invariants, but using the provided seed. func SimulateFromSeed( - t *testing.T, app *baseapp.BaseApp, appStateFn func(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json.RawMessage, seed int64, ops []Operation, setups []RandSetup, + tb testing.TB, app *baseapp.BaseApp, appStateFn func(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json.RawMessage, seed int64, ops []Operation, setups []RandSetup, invariants []Invariant, numBlocks int, blockSize int, commit bool, ) { + var t *testing.T + var b *testing.B + testingmode := false + if _t, ok := tb.(*testing.T); ok { + t = _t + testingmode = true + } else { + b = tb.(*testing.B) + } log := fmt.Sprintf("Starting SimulateFromSeed with randomness created with seed %d", int(seed)) r := rand.New(rand.NewSource(seed)) timestamp := randTimestamp(r) - log = fmt.Sprintf("%s\nStarting the simulation from time %v, unixtime %v", log, timestamp.UTC().Format(time.UnixDate), timestamp.Unix()) + log = updateLog(testingmode, log, "Starting the simulation from time %v, unixtime %v", timestamp.UTC().Format(time.UnixDate), timestamp.Unix()) fmt.Printf("%s\n", log) timeDiff := maxTimePerBlock - minTimePerBlock @@ -67,7 +75,7 @@ func SimulateFromSeed( // Setup event stats events := make(map[string]uint) event := func(what string) { - log += "\nevent - " + what + log = updateLog(testingmode, log, "event - "+what) events[what]++ } @@ -82,32 +90,40 @@ func SimulateFromSeed( // These are operations which have been queued by previous operations operationQueue := make(map[int][]Operation) + if !testingmode { + b.ResetTimer() + } + for i := 0; i < numBlocks; i++ { // Run the BeginBlock handler app.BeginBlock(request) - log += "\nBeginBlock" + log = updateLog(testingmode, log, "BeginBlock") - // Make sure invariants hold at beginning of block - AssertAllInvariants(t, app, invariants, log) + if testingmode { + // Make sure invariants hold at beginning of block + AssertAllInvariants(t, app, invariants, log) + } ctx := app.NewContext(false, header) thisBlockSize := getBlockSize(r, blockSize) // Run queued operations. Ignores blocksize if blocksize is too small - log, numQueuedOpsRan := runQueuedOperations(operationQueue, int(header.Height), t, r, app, ctx, keys, log, event) + log, numQueuedOpsRan := runQueuedOperations(operationQueue, int(header.Height), tb, r, app, ctx, keys, log, event) opCount += numQueuedOpsRan thisBlockSize -= numQueuedOpsRan for j := 0; j < thisBlockSize; j++ { - logUpdate, futureOps, err := ops[r.Intn(len(ops))](t, r, app, ctx, keys, log, event) - log += "\n" + logUpdate - require.Nil(t, err, log) + logUpdate, futureOps, err := ops[r.Intn(len(ops))](tb, r, app, ctx, keys, log, event) + log = updateLog(testingmode, log, logUpdate) + if err != nil { + tb.Fatalf("error on operation %d, %v", opCount, err) + } queueOperations(operationQueue, futureOps) - if onOperation { + if onOperation && testingmode { AssertAllInvariants(t, app, invariants, log) } - if opCount%200 == 0 { + if testingmode && opCount%200 == 0 { fmt.Printf("\rSimulating... block %d/%d, operation %d.", header.Height, numBlocks, opCount) } opCount++ @@ -119,8 +135,10 @@ func SimulateFromSeed( header.Time = header.Time.Add(time.Duration(minTimePerBlock) * time.Second).Add(time.Duration(int64(r.Intn(int(timeDiff)))) * time.Second) log += "\nEndBlock" - // Make sure invariants hold at end of block - AssertAllInvariants(t, app, invariants, log) + if testingmode { + // Make sure invariants hold at end of block + AssertAllInvariants(t, app, invariants, log) + } if commit { app.Commit() } @@ -129,13 +147,23 @@ func SimulateFromSeed( request = RandomRequestBeginBlock(r, validators, livenessTransitionMatrix, evidenceFraction, lastHeaderTime, event, header, log) // Update the validator set - validators = updateValidators(t, r, validators, res.ValidatorUpdates, event) + validators = updateValidators(tb, r, validators, res.ValidatorUpdates, event) } - fmt.Printf("\nSimulation complete. Final height (blocks): %d, final time (seconds): %v\n", header.Height, header.Time) + if testingmode { + fmt.Printf("\nSimulation complete. Final height (blocks): %d, final time (seconds): %v\n", header.Height, header.Time) + } DisplayEvents(events) } +func updateLog(testingmode bool, log string, update string, args ...interface{}) (updatedLog string) { + if testingmode == true { + update = fmt.Sprintf(update, args) + return fmt.Sprintf("%s\n%s", log, update) + } + return "" +} + func getBlockSize(r *rand.Rand, blockSize int) int { load := r.Float64() switch { @@ -148,69 +176,6 @@ func getBlockSize(r *rand.Rand, blockSize int) int { } } -// Simulate from seed, benchmarks -func BenchmarkSimulationFromSeed(b *testing.B, app *baseapp.BaseApp, appStateFn func(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json.RawMessage, - seed int64, ops []Operation, setups []RandSetup, numBlocks int, blockSize int, commit bool) { - r := rand.New(rand.NewSource(seed)) - timestamp := randTimestamp(r) - timeDiff := maxTimePerBlock - minTimePerBlock - keys, accs := mock.GeneratePrivKeyAddressPairsFromRand(r, numKeys) - - // Setup event stats - events := make(map[string]uint) - event := func(what string) { - events[what]++ - } - - validators := initChain(r, keys, accs, setups, app, appStateFn) - - header := abci.Header{Height: 0, Time: timestamp} - opCount := 0 - - request := abci.RequestBeginBlock{Header: header} - - var lastHeaderTime time.Time - // These are operations which have been queued by previous operations - operationQueue := make(map[int][]Operation) - b.ResetTimer() - - for i := 0; i < numBlocks; i++ { - // Run the BeginBlock handler - app.BeginBlock(request) - ctx := app.NewContext(false, header) - thisBlockSize := getBlockSize(r, blockSize) - - // Run queued operations. Ignores blocksize if blocksize is too small - log, numQueuedOpsRan := runQueuedOperations(operationQueue, int(header.Height), b, r, app, ctx, keys, "", event) - opCount += numQueuedOpsRan - thisBlockSize -= numQueuedOpsRan - for j := 0; j < thisBlockSize; j++ { - _, futureOps, err := ops[r.Intn(len(ops))](b, r, app, ctx, keys, "", event) - queueOperations(operationQueue, futureOps) - if err != nil { - b.Fatalf("error on operation %d, %v", opCount, err) - } - opCount++ - } - - res := app.EndBlock(abci.RequestEndBlock{}) - header.Height++ - lastHeaderTime = header.Time - header.Time = header.Time.Add(time.Duration(minTimePerBlock) * time.Second).Add(time.Duration(int64(r.Intn(int(timeDiff)))) * time.Second) - if commit { - app.Commit() - } - - // Generate a random RequestBeginBlock with the current validator set for the next block - request = RandomRequestBeginBlock(r, validators, livenessTransitionMatrix, evidenceFraction, lastHeaderTime, event, header, log) - - // Update the validator set - validators = updateValidators(b, r, validators, res.ValidatorUpdates, event) - } - DisplayEvents(events) - fmt.Printf("Benchmark simulation ran %d operations\n", opCount) -} - // adds all future operations into the operation queue. func queueOperations(queuedOperations map[int][]Operation, futureOperations []FutureOperation) { if futureOperations == nil { diff --git a/x/stake/simulation/msgs.go b/x/stake/simulation/msgs.go index bc1063494..033487254 100644 --- a/x/stake/simulation/msgs.go +++ b/x/stake/simulation/msgs.go @@ -17,6 +17,7 @@ import ( // SimulateMsgCreateValidator func SimulateMsgCreateValidator(m auth.AccountMapper, k stake.Keeper) simulation.Operation { + handler := stake.NewHandler(k) return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { denom := k.GetParams(ctx).BondDenom description := stake.Description{ @@ -43,7 +44,7 @@ func SimulateMsgCreateValidator(m auth.AccountMapper, k stake.Keeper) simulation tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) } ctx, write := ctx.CacheContext() - result := stake.NewHandler(k)(ctx, msg) + result := handler(ctx, msg) if result.IsOK() { write() } @@ -56,6 +57,7 @@ func SimulateMsgCreateValidator(m auth.AccountMapper, k stake.Keeper) simulation // SimulateMsgEditValidator func SimulateMsgEditValidator(k stake.Keeper) simulation.Operation { + handler := stake.NewHandler(k) return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { description := stake.Description{ Moniker: simulation.RandStringOfLength(r, 10), @@ -74,7 +76,7 @@ func SimulateMsgEditValidator(k stake.Keeper) simulation.Operation { tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) } ctx, write := ctx.CacheContext() - result := stake.NewHandler(k)(ctx, msg) + result := handler(ctx, msg) if result.IsOK() { write() } @@ -86,6 +88,7 @@ func SimulateMsgEditValidator(k stake.Keeper) simulation.Operation { // SimulateMsgDelegate func SimulateMsgDelegate(m auth.AccountMapper, k stake.Keeper) simulation.Operation { + handler := stake.NewHandler(k) return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { denom := k.GetParams(ctx).BondDenom validatorKey := simulation.RandomKey(r, keys) @@ -108,7 +111,7 @@ func SimulateMsgDelegate(m auth.AccountMapper, k stake.Keeper) simulation.Operat tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) } ctx, write := ctx.CacheContext() - result := stake.NewHandler(k)(ctx, msg) + result := handler(ctx, msg) if result.IsOK() { write() } @@ -120,6 +123,7 @@ func SimulateMsgDelegate(m auth.AccountMapper, k stake.Keeper) simulation.Operat // SimulateMsgBeginUnbonding func SimulateMsgBeginUnbonding(m auth.AccountMapper, k stake.Keeper) simulation.Operation { + handler := stake.NewHandler(k) return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { denom := k.GetParams(ctx).BondDenom validatorKey := simulation.RandomKey(r, keys) @@ -142,7 +146,7 @@ func SimulateMsgBeginUnbonding(m auth.AccountMapper, k stake.Keeper) simulation. tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) } ctx, write := ctx.CacheContext() - result := stake.NewHandler(k)(ctx, msg) + result := handler(ctx, msg) if result.IsOK() { write() } @@ -154,6 +158,7 @@ func SimulateMsgBeginUnbonding(m auth.AccountMapper, k stake.Keeper) simulation. // SimulateMsgCompleteUnbonding func SimulateMsgCompleteUnbonding(k stake.Keeper) simulation.Operation { + handler := stake.NewHandler(k) return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { validatorKey := simulation.RandomKey(r, keys) validatorAddress := sdk.AccAddress(validatorKey.PubKey().Address()) @@ -167,7 +172,7 @@ func SimulateMsgCompleteUnbonding(k stake.Keeper) simulation.Operation { tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) } ctx, write := ctx.CacheContext() - result := stake.NewHandler(k)(ctx, msg) + result := handler(ctx, msg) if result.IsOK() { write() } @@ -179,6 +184,7 @@ func SimulateMsgCompleteUnbonding(k stake.Keeper) simulation.Operation { // SimulateMsgBeginRedelegate func SimulateMsgBeginRedelegate(m auth.AccountMapper, k stake.Keeper) simulation.Operation { + handler := stake.NewHandler(k) return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { denom := k.GetParams(ctx).BondDenom sourceValidatorKey := simulation.RandomKey(r, keys) @@ -205,7 +211,7 @@ func SimulateMsgBeginRedelegate(m auth.AccountMapper, k stake.Keeper) simulation tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) } ctx, write := ctx.CacheContext() - result := stake.NewHandler(k)(ctx, msg) + result := handler(ctx, msg) if result.IsOK() { write() } @@ -217,6 +223,7 @@ func SimulateMsgBeginRedelegate(m auth.AccountMapper, k stake.Keeper) simulation // SimulateMsgCompleteRedelegate func SimulateMsgCompleteRedelegate(k stake.Keeper) simulation.Operation { + handler := stake.NewHandler(k) return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { validatorSrcKey := simulation.RandomKey(r, keys) validatorSrcAddress := sdk.AccAddress(validatorSrcKey.PubKey().Address()) @@ -233,7 +240,7 @@ func SimulateMsgCompleteRedelegate(k stake.Keeper) simulation.Operation { tb.Fatalf("expected msg to pass ValidateBasic: %s, log %s", msg.GetSignBytes(), log) } ctx, write := ctx.CacheContext() - result := stake.NewHandler(k)(ctx, msg) + result := handler(ctx, msg) if result.IsOK() { write() } From b3d08bcb239198709191895ee15c86a09434ed08 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Thu, 30 Aug 2018 00:35:02 -0700 Subject: [PATCH 21/49] minor cleanup --- x/mock/simulation/random_simulate_blocks.go | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index 6376d53dc..e609d3c5f 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -75,7 +75,7 @@ func SimulateFromSeed( // Setup event stats events := make(map[string]uint) event := func(what string) { - log = updateLog(testingmode, log, "event - "+what) + log = updateLog(testingmode, log, "event - %s", what) events[what]++ } @@ -119,12 +119,13 @@ func SimulateFromSeed( } queueOperations(operationQueue, futureOps) - - if onOperation && testingmode { - AssertAllInvariants(t, app, invariants, log) - } - if testingmode && opCount%200 == 0 { - fmt.Printf("\rSimulating... block %d/%d, operation %d.", header.Height, numBlocks, opCount) + if testingmode { + if onOperation { + AssertAllInvariants(t, app, invariants, log) + } + if opCount%200 == 0 { + fmt.Printf("\rSimulating... block %d/%d, operation %d.", header.Height, numBlocks, opCount) + } } opCount++ } @@ -133,7 +134,7 @@ func SimulateFromSeed( header.Height++ lastHeaderTime = header.Time header.Time = header.Time.Add(time.Duration(minTimePerBlock) * time.Second).Add(time.Duration(int64(r.Intn(int(timeDiff)))) * time.Second) - log += "\nEndBlock" + log = updateLog(testingmode, log, "EndBlock") if testingmode { // Make sure invariants hold at end of block @@ -152,13 +153,15 @@ func SimulateFromSeed( if testingmode { fmt.Printf("\nSimulation complete. Final height (blocks): %d, final time (seconds): %v\n", header.Height, header.Time) + } else { + fmt.Printf("%d operations ran\n", opCount) } DisplayEvents(events) } func updateLog(testingmode bool, log string, update string, args ...interface{}) (updatedLog string) { if testingmode == true { - update = fmt.Sprintf(update, args) + update = fmt.Sprintf(update, args...) return fmt.Sprintf("%s\n%s", log, update) } return "" From d1a5808a7518f1244a1716877580c4df9c5ceb12 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Thu, 30 Aug 2018 09:32:15 -0700 Subject: [PATCH 22/49] Address Anton's comment --- x/mock/simulation/random_simulate_blocks.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index e609d3c5f..e0ab9ba58 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -115,7 +115,7 @@ func SimulateFromSeed( logUpdate, futureOps, err := ops[r.Intn(len(ops))](tb, r, app, ctx, keys, log, event) log = updateLog(testingmode, log, logUpdate) if err != nil { - tb.Fatalf("error on operation %d, %v", opCount, err) + tb.Fatalf("error on operation %d, %v, log %s", opCount, err, log) } queueOperations(operationQueue, futureOps) @@ -160,7 +160,7 @@ func SimulateFromSeed( } func updateLog(testingmode bool, log string, update string, args ...interface{}) (updatedLog string) { - if testingmode == true { + if testingmode { update = fmt.Sprintf(update, args...) return fmt.Sprintf("%s\n%s", log, update) } From 4be69077fe262524527a1e265d4c1032877946ff Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Thu, 30 Aug 2018 11:43:56 -0700 Subject: [PATCH 23/49] Fix cyclomatic complexity, add ability to use GoLevelDB --- cmd/gaia/app/sim_test.go | 19 ++++-- server/export_test.go | 16 ++--- server/mock/app.go | 2 +- x/mock/simulation/random_simulate_blocks.go | 76 ++++++++++++--------- 4 files changed, 66 insertions(+), 47 deletions(-) diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go index 1fd07b330..6568f124b 100644 --- a/cmd/gaia/app/sim_test.go +++ b/cmd/gaia/app/sim_test.go @@ -4,6 +4,7 @@ import ( "encoding/json" "flag" "math/rand" + "os" "testing" "github.com/stretchr/testify/require" @@ -23,11 +24,12 @@ import ( ) var ( - seed int64 - numBlocks int - blockSize int - enabled bool - verbose bool + seed int64 + numBlocks int + blockSize int + enabled bool + verbose bool + usegoleveldb bool ) func init() { @@ -36,6 +38,7 @@ func init() { flag.IntVar(&blockSize, "SimulationBlockSize", 200, "Operations per block") flag.BoolVar(&enabled, "SimulationEnabled", false, "Enable the simulation") flag.BoolVar(&verbose, "SimulationVerbose", false, "Verbose log output") + flag.BoolVar(&usegoleveldb, "SimulationGoLevelDB", false, "Use GoLevelDB instead of memdb") } func appStateFn(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json.RawMessage { @@ -118,7 +121,11 @@ func BenchmarkFullGaiaSimulation(b *testing.B) { // Setup Gaia application var logger log.Logger logger = log.NewNopLogger() - db := dbm.NewMemDB() + var db dbm.DB + db = dbm.NewMemDB() + if usegoleveldb { + db, _ = dbm.NewGoLevelDB("Simulation", os.TempDir()) + } app := NewGaiaApp(logger, db, nil) // Run randomized simulation diff --git a/server/export_test.go b/server/export_test.go index 358f72cf6..488c55bbf 100644 --- a/server/export_test.go +++ b/server/export_test.go @@ -1,16 +1,16 @@ package server import ( - "testing" - "github.com/stretchr/testify/require" - "github.com/cosmos/cosmos-sdk/wire" - "github.com/tendermint/tendermint/libs/log" - tcmd "github.com/tendermint/tendermint/cmd/tendermint/commands" - "os" "bytes" + "github.com/cosmos/cosmos-sdk/server/mock" + "github.com/cosmos/cosmos-sdk/wire" + "github.com/stretchr/testify/require" + tcmd "github.com/tendermint/tendermint/cmd/tendermint/commands" + "github.com/tendermint/tendermint/libs/log" "io" - "github.com/cosmos/cosmos-sdk/server/mock" - ) + "os" + "testing" +) func TestEmptyState(t *testing.T) { defer setupViper(t)() diff --git a/server/mock/app.go b/server/mock/app.go index eb2dfc3cc..3c6ad3ec2 100644 --- a/server/mock/app.go +++ b/server/mock/app.go @@ -129,7 +129,7 @@ func AppGenStateEmpty(_ *wire.Codec, _ []json.RawMessage) (appState json.RawMess // Return a validator, not much else func AppGenTx(_ *wire.Codec, pk crypto.PubKey, genTxConfig gc.GenTx) ( - appGenTx, cliPrint json.RawMessage, validator tmtypes.GenesisValidator, err error) { + appGenTx, cliPrint json.RawMessage, validator tmtypes.GenesisValidator, err error) { validator = tmtypes.GenesisValidator{ PubKey: pk, diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index e0ab9ba58..42a616331 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -54,15 +54,7 @@ func SimulateFromSeed( tb testing.TB, app *baseapp.BaseApp, appStateFn func(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json.RawMessage, seed int64, ops []Operation, setups []RandSetup, invariants []Invariant, numBlocks int, blockSize int, commit bool, ) { - var t *testing.T - var b *testing.B - testingmode := false - if _t, ok := tb.(*testing.T); ok { - t = _t - testingmode = true - } else { - b = tb.(*testing.B) - } + testingmode, t, b := getTestingMode(tb) log := fmt.Sprintf("Starting SimulateFromSeed with randomness created with seed %d", int(seed)) r := rand.New(rand.NewSource(seed)) timestamp := randTimestamp(r) @@ -93,6 +85,7 @@ func SimulateFromSeed( if !testingmode { b.ResetTimer() } + blockSimulator := createBlockSimulator(testingmode, tb, t, event, invariants, ops, operationQueue, numBlocks) for i := 0; i < numBlocks; i++ { // Run the BeginBlock handler @@ -111,24 +104,8 @@ func SimulateFromSeed( log, numQueuedOpsRan := runQueuedOperations(operationQueue, int(header.Height), tb, r, app, ctx, keys, log, event) opCount += numQueuedOpsRan thisBlockSize -= numQueuedOpsRan - for j := 0; j < thisBlockSize; j++ { - logUpdate, futureOps, err := ops[r.Intn(len(ops))](tb, r, app, ctx, keys, log, event) - log = updateLog(testingmode, log, logUpdate) - if err != nil { - tb.Fatalf("error on operation %d, %v, log %s", opCount, err, log) - } - - queueOperations(operationQueue, futureOps) - if testingmode { - if onOperation { - AssertAllInvariants(t, app, invariants, log) - } - if opCount%200 == 0 { - fmt.Printf("\rSimulating... block %d/%d, operation %d.", header.Height, numBlocks, opCount) - } - } - opCount++ - } + log, operations := blockSimulator(thisBlockSize, r, app, ctx, keys, log, header) + opCount += operations res := app.EndBlock(abci.RequestEndBlock{}) header.Height++ @@ -151,14 +128,49 @@ func SimulateFromSeed( validators = updateValidators(tb, r, validators, res.ValidatorUpdates, event) } - if testingmode { - fmt.Printf("\nSimulation complete. Final height (blocks): %d, final time (seconds): %v\n", header.Height, header.Time) - } else { - fmt.Printf("%d operations ran\n", opCount) - } + fmt.Printf("\nSimulation complete. Final height (blocks): %d, final time (seconds), : %v, operations ran %d\n", header.Height, header.Time, opCount) DisplayEvents(events) } +// Returns a function to simulate blocks. Written like this to avoid constant parameters being passed everytime, to minimize +// memory overhead +func createBlockSimulator(testingmode bool, tb testing.TB, t *testing.T, event func(string), invariants []Invariant, ops []Operation, operationQueue map[int][]Operation, totalNumBlocks int) func( + blocksize int, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, privKeys []crypto.PrivKey, log string, header abci.Header) (updatedLog string, opCount int) { + return func(blocksize int, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, + keys []crypto.PrivKey, log string, header abci.Header) (updatedLog string, opCount int) { + for j := 0; j < blocksize; j++ { + logUpdate, futureOps, err := ops[r.Intn(len(ops))](tb, r, app, ctx, keys, log, event) + log = updateLog(testingmode, log, logUpdate) + if err != nil { + tb.Fatalf("error on operation %d within block %d, %v, log %s", header.Height, opCount, err, log) + } + + queueOperations(operationQueue, futureOps) + if testingmode { + if onOperation { + AssertAllInvariants(t, app, invariants, log) + } + if opCount%50 == 0 { + fmt.Printf("\rSimulating... block %d/%d, operation %d/%d.", header.Height, totalNumBlocks, opCount, blocksize) + } + } + opCount++ + } + return log, opCount + } +} + +func getTestingMode(tb testing.TB) (testingmode bool, t *testing.T, b *testing.B) { + testingmode = false + if _t, ok := tb.(*testing.T); ok { + t = _t + testingmode = true + } else { + b = tb.(*testing.B) + } + return +} + func updateLog(testingmode bool, log string, update string, args ...interface{}) (updatedLog string) { if testingmode { update = fmt.Sprintf(update, args...) From a69725d40d5ceaf2b27c0f5fba8e4e3c47029703 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Thu, 30 Aug 2018 11:47:32 -0700 Subject: [PATCH 24/49] Add method to benchmark with commits --- cmd/gaia/app/sim_test.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go index 6568f124b..c7452ae53 100644 --- a/cmd/gaia/app/sim_test.go +++ b/cmd/gaia/app/sim_test.go @@ -30,6 +30,7 @@ var ( enabled bool verbose bool usegoleveldb bool + commit bool ) func init() { @@ -39,6 +40,7 @@ func init() { flag.BoolVar(&enabled, "SimulationEnabled", false, "Enable the simulation") flag.BoolVar(&verbose, "SimulationVerbose", false, "Verbose log output") flag.BoolVar(&usegoleveldb, "SimulationGoLevelDB", false, "Use GoLevelDB instead of memdb") + flag.BoolVar(&commit, "SimulationCommit", false, "Have the simulation commit") } func appStateFn(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json.RawMessage { @@ -137,7 +139,7 @@ func BenchmarkFullGaiaSimulation(b *testing.B) { invariants(app), // these shouldn't get ran 10, 100, - false, + commit, ) } @@ -165,7 +167,7 @@ func TestFullGaiaSimulation(t *testing.T) { invariants(app), numBlocks, blockSize, - false, + commit, ) } From e64c6da6f223019a5db6fbcb51c8d51745e42e44 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Thu, 30 Aug 2018 23:42:12 -0700 Subject: [PATCH 25/49] cleanup goleveldb dirs --- cmd/gaia/app/sim_test.go | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go index c7452ae53..38dd352bd 100644 --- a/cmd/gaia/app/sim_test.go +++ b/cmd/gaia/app/sim_test.go @@ -124,10 +124,14 @@ func BenchmarkFullGaiaSimulation(b *testing.B) { var logger log.Logger logger = log.NewNopLogger() var db dbm.DB - db = dbm.NewMemDB() if usegoleveldb { - db, _ = dbm.NewGoLevelDB("Simulation", os.TempDir()) + dir := os.TempDir() + db, _ = dbm.NewGoLevelDB("Simulation", dir) + defer os.RemoveAll(dir) + } else { + db = dbm.NewMemDB() } + defer db.Close() app := NewGaiaApp(logger, db, nil) // Run randomized simulation From 75025720a3f63ae162ebe9ae48aa789b81287958 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Fri, 31 Aug 2018 21:57:33 -0700 Subject: [PATCH 26/49] simulation: Initialize governance properly --- cmd/gaia/app/sim_test.go | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go index 38dd352bd..be42511d0 100644 --- a/cmd/gaia/app/sim_test.go +++ b/cmd/gaia/app/sim_test.go @@ -16,6 +16,7 @@ import ( "github.com/cosmos/cosmos-sdk/baseapp" sdk "github.com/cosmos/cosmos-sdk/types" banksim "github.com/cosmos/cosmos-sdk/x/bank/simulation" + "github.com/cosmos/cosmos-sdk/x/gov" govsim "github.com/cosmos/cosmos-sdk/x/gov/simulation" "github.com/cosmos/cosmos-sdk/x/mock/simulation" slashingsim "github.com/cosmos/cosmos-sdk/x/slashing/simulation" @@ -54,7 +55,7 @@ func appStateFn(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json Coins: coins, }) } - + govGenesis := gov.DefaultGenesisState() // Default genesis state stakeGenesis := stake.DefaultGenesisState() var validators []stake.Validator @@ -78,6 +79,7 @@ func appStateFn(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json genesis := GenesisState{ Accounts: genesisAccounts, StakeData: stakeGenesis, + GovData: govGenesis, } // Marshal genesis @@ -141,8 +143,8 @@ func BenchmarkFullGaiaSimulation(b *testing.B) { testAndRunTxs(app), []simulation.RandSetup{}, invariants(app), // these shouldn't get ran - 10, - 100, + 210, + blockSize, commit, ) } From 3c2100793eb8ba584a5dff16706acee39463b76b Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Sat, 1 Sep 2018 12:32:24 -0700 Subject: [PATCH 27/49] Address @cwgoes comments --- cmd/gaia/app/sim_test.go | 28 ++++++------- x/mock/simulation/random_simulate_blocks.go | 46 +++++++++++---------- 2 files changed, 36 insertions(+), 38 deletions(-) diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go index be42511d0..68f67709a 100644 --- a/cmd/gaia/app/sim_test.go +++ b/cmd/gaia/app/sim_test.go @@ -25,13 +25,12 @@ import ( ) var ( - seed int64 - numBlocks int - blockSize int - enabled bool - verbose bool - usegoleveldb bool - commit bool + seed int64 + numBlocks int + blockSize int + enabled bool + verbose bool + commit bool ) func init() { @@ -40,7 +39,6 @@ func init() { flag.IntVar(&blockSize, "SimulationBlockSize", 200, "Operations per block") flag.BoolVar(&enabled, "SimulationEnabled", false, "Enable the simulation") flag.BoolVar(&verbose, "SimulationVerbose", false, "Verbose log output") - flag.BoolVar(&usegoleveldb, "SimulationGoLevelDB", false, "Use GoLevelDB instead of memdb") flag.BoolVar(&commit, "SimulationCommit", false, "Have the simulation commit") } @@ -126,14 +124,12 @@ func BenchmarkFullGaiaSimulation(b *testing.B) { var logger log.Logger logger = log.NewNopLogger() var db dbm.DB - if usegoleveldb { - dir := os.TempDir() - db, _ = dbm.NewGoLevelDB("Simulation", dir) - defer os.RemoveAll(dir) - } else { - db = dbm.NewMemDB() - } - defer db.Close() + dir := os.TempDir() + db, _ = dbm.NewGoLevelDB("Simulation", dir) + defer func() { + db.Close() + os.RemoveAll(dir) + }() app := NewGaiaApp(logger, db, nil) // Run randomized simulation diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index 42a616331..be1963a9e 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -54,11 +54,11 @@ func SimulateFromSeed( tb testing.TB, app *baseapp.BaseApp, appStateFn func(r *rand.Rand, keys []crypto.PrivKey, accs []sdk.AccAddress) json.RawMessage, seed int64, ops []Operation, setups []RandSetup, invariants []Invariant, numBlocks int, blockSize int, commit bool, ) { - testingmode, t, b := getTestingMode(tb) + testingMode, t, b := getTestingMode(tb) log := fmt.Sprintf("Starting SimulateFromSeed with randomness created with seed %d", int(seed)) r := rand.New(rand.NewSource(seed)) timestamp := randTimestamp(r) - log = updateLog(testingmode, log, "Starting the simulation from time %v, unixtime %v", timestamp.UTC().Format(time.UnixDate), timestamp.Unix()) + log = updateLog(testingMode, log, "Starting the simulation from time %v, unixtime %v", timestamp.UTC().Format(time.UnixDate), timestamp.Unix()) fmt.Printf("%s\n", log) timeDiff := maxTimePerBlock - minTimePerBlock @@ -67,7 +67,7 @@ func SimulateFromSeed( // Setup event stats events := make(map[string]uint) event := func(what string) { - log = updateLog(testingmode, log, "event - %s", what) + log = updateLog(testingMode, log, "event - %s", what) events[what]++ } @@ -78,21 +78,24 @@ func SimulateFromSeed( request := abci.RequestBeginBlock{Header: header} - var lastHeaderTime time.Time + var pastTimes []time.Time // These are operations which have been queued by previous operations operationQueue := make(map[int][]Operation) - if !testingmode { + if !testingMode { b.ResetTimer() } - blockSimulator := createBlockSimulator(testingmode, tb, t, event, invariants, ops, operationQueue, numBlocks) + blockSimulator := createBlockSimulator(testingMode, tb, t, event, invariants, ops, operationQueue, numBlocks) for i := 0; i < numBlocks; i++ { + // Log the header time for future lookup + pastTimes = append(pastTimes, header.Time) + // Run the BeginBlock handler app.BeginBlock(request) - log = updateLog(testingmode, log, "BeginBlock") + log = updateLog(testingMode, log, "BeginBlock") - if testingmode { + if testingMode { // Make sure invariants hold at beginning of block AssertAllInvariants(t, app, invariants, log) } @@ -109,11 +112,10 @@ func SimulateFromSeed( res := app.EndBlock(abci.RequestEndBlock{}) header.Height++ - lastHeaderTime = header.Time header.Time = header.Time.Add(time.Duration(minTimePerBlock) * time.Second).Add(time.Duration(int64(r.Intn(int(timeDiff)))) * time.Second) - log = updateLog(testingmode, log, "EndBlock") + log = updateLog(testingMode, log, "EndBlock") - if testingmode { + if testingMode { // Make sure invariants hold at end of block AssertAllInvariants(t, app, invariants, log) } @@ -122,7 +124,7 @@ func SimulateFromSeed( } // Generate a random RequestBeginBlock with the current validator set for the next block - request = RandomRequestBeginBlock(r, validators, livenessTransitionMatrix, evidenceFraction, lastHeaderTime, event, header, log) + request = RandomRequestBeginBlock(r, validators, livenessTransitionMatrix, evidenceFraction, pastTimes, event, header, log) // Update the validator set validators = updateValidators(tb, r, validators, res.ValidatorUpdates, event) @@ -134,19 +136,19 @@ func SimulateFromSeed( // Returns a function to simulate blocks. Written like this to avoid constant parameters being passed everytime, to minimize // memory overhead -func createBlockSimulator(testingmode bool, tb testing.TB, t *testing.T, event func(string), invariants []Invariant, ops []Operation, operationQueue map[int][]Operation, totalNumBlocks int) func( +func createBlockSimulator(testingMode bool, tb testing.TB, t *testing.T, event func(string), invariants []Invariant, ops []Operation, operationQueue map[int][]Operation, totalNumBlocks int) func( blocksize int, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, privKeys []crypto.PrivKey, log string, header abci.Header) (updatedLog string, opCount int) { return func(blocksize int, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, header abci.Header) (updatedLog string, opCount int) { for j := 0; j < blocksize; j++ { logUpdate, futureOps, err := ops[r.Intn(len(ops))](tb, r, app, ctx, keys, log, event) - log = updateLog(testingmode, log, logUpdate) + log = updateLog(testingMode, log, logUpdate) if err != nil { tb.Fatalf("error on operation %d within block %d, %v, log %s", header.Height, opCount, err, log) } queueOperations(operationQueue, futureOps) - if testingmode { + if testingMode { if onOperation { AssertAllInvariants(t, app, invariants, log) } @@ -160,19 +162,19 @@ func createBlockSimulator(testingmode bool, tb testing.TB, t *testing.T, event f } } -func getTestingMode(tb testing.TB) (testingmode bool, t *testing.T, b *testing.B) { - testingmode = false +func getTestingMode(tb testing.TB) (testingMode bool, t *testing.T, b *testing.B) { + testingMode = false if _t, ok := tb.(*testing.T); ok { t = _t - testingmode = true + testingMode = true } else { b = tb.(*testing.B) } return } -func updateLog(testingmode bool, log string, update string, args ...interface{}) (updatedLog string) { - if testingmode { +func updateLog(testingMode bool, log string, update string, args ...interface{}) (updatedLog string) { + if testingMode { update = fmt.Sprintf(update, args...) return fmt.Sprintf("%s\n%s", log, update) } @@ -240,7 +242,7 @@ func getKeys(validators map[string]mockValidator) []string { // RandomRequestBeginBlock generates a list of signing validators according to the provided list of validators, signing fraction, and evidence fraction func RandomRequestBeginBlock(r *rand.Rand, validators map[string]mockValidator, livenessTransitions TransitionMatrix, evidenceFraction float64, - lastHeaderTime time.Time, event func(string), header abci.Header, log string) abci.RequestBeginBlock { + pastTimes []time.Time, event func(string), header abci.Header, log string) abci.RequestBeginBlock { if len(validators) == 0 { return abci.RequestBeginBlock{Header: header} } @@ -279,7 +281,7 @@ func RandomRequestBeginBlock(r *rand.Rand, validators map[string]mockValidator, time := header.Time if r.Float64() < pastEvidenceFraction { height = int64(r.Intn(int(header.Height))) - time = lastHeaderTime + time = pastTimes[height] } validator := signingValidators[r.Intn(len(signingValidators))].Validator var currentTotalVotingPower int64 From a991a2e1c4b328cb70af86fb5b794a1967813c5a Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Fri, 31 Aug 2018 00:37:25 -0700 Subject: [PATCH 28/49] Improve GetValidator speed In simulation, this was shown to cause a 4x speedup. There are no safety concerns here, as amino encoding is deterministic. --- PENDING.md | 2 +- x/stake/keeper/validator.go | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/PENDING.md b/PENDING.md index 0aef9fd19..5557a8771 100644 --- a/PENDING.md +++ b/PENDING.md @@ -78,7 +78,7 @@ IMPROVEMENTS * [x/stake] [#2023](https://github.com/cosmos/cosmos-sdk/pull/2023) Terminate iteration loop in `UpdateBondedValidators` and `UpdateBondedValidatorsFull` when the first revoked validator is encountered and perform a sanity check. * [x/auth] Signature verification's gas cost now accounts for pubkey type. [#2046](https://github.com/tendermint/tendermint/pull/2046) * [x/stake] [x/slashing] Ensure delegation invariants to jailed validators [#1883](https://github.com/cosmos/cosmos-sdk/issues/1883). - + * [x/stake] Improve speed of GetValidator, which was shown to be a performance bottleneck. [#2046](https://github.com/tendermint/tendermint/pull/2200) * SDK * [tools] Make get_vendor_deps deletes `.vendor-new` directories, in case scratch files are present. * [cli] \#1632 Add integration tests to ensure `basecoind init && basecoind` start sequences run successfully for both `democoin` and `basecoin` examples. diff --git a/x/stake/keeper/validator.go b/x/stake/keeper/validator.go index 0ea24e639..5dc075742 100644 --- a/x/stake/keeper/validator.go +++ b/x/stake/keeper/validator.go @@ -2,6 +2,7 @@ package keeper import ( "bytes" + "container/list" "fmt" abci "github.com/tendermint/tendermint/abci/types" @@ -11,6 +12,14 @@ import ( "github.com/cosmos/cosmos-sdk/x/stake/types" ) +type cachedValidator struct { + val types.Validator + marshalled string +} + +var validatorCache = make(map[string]cachedValidator, 1000) +var validatorCacheList = list.New() + // get a single validator func (k Keeper) GetValidator(ctx sdk.Context, addr sdk.ValAddress) (validator types.Validator, found bool) { store := ctx.KVStore(k.storeKey) @@ -18,6 +27,23 @@ func (k Keeper) GetValidator(ctx sdk.Context, addr sdk.ValAddress) (validator ty if value == nil { return validator, false } + // return cached validator + strValue := string(value) + if val, ok := validatorCache[strValue]; ok { + valToReturn := val.val + // Doesn't mutate the cache's value + valToReturn.Operator = addr + return valToReturn, true + } else { // get validator from cache + validator = types.MustUnmarshalValidator(k.cdc, addr, value) + cachedVal := cachedValidator{validator, strValue} + validatorCache[strValue] = cachedValidator{validator, strValue} + validatorCacheList.PushBack(cachedVal) + if validatorCacheList.Len() > 500 { + valToRemove := validatorCacheList.Remove(validatorCacheList.Front()).(cachedValidator) + delete(validatorCache, valToRemove.marshalled) + } + } validator = types.MustUnmarshalValidator(k.cdc, addr, value) return validator, true } From 3b4caa5dd2bf218f24fd53616a41aefea79c9f20 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Fri, 31 Aug 2018 08:29:32 -0700 Subject: [PATCH 29/49] fix lint --- x/stake/keeper/validator.go | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/x/stake/keeper/validator.go b/x/stake/keeper/validator.go index 5dc075742..a12801dc9 100644 --- a/x/stake/keeper/validator.go +++ b/x/stake/keeper/validator.go @@ -34,16 +34,17 @@ func (k Keeper) GetValidator(ctx sdk.Context, addr sdk.ValAddress) (validator ty // Doesn't mutate the cache's value valToReturn.Operator = addr return valToReturn, true - } else { // get validator from cache - validator = types.MustUnmarshalValidator(k.cdc, addr, value) - cachedVal := cachedValidator{validator, strValue} - validatorCache[strValue] = cachedValidator{validator, strValue} - validatorCacheList.PushBack(cachedVal) - if validatorCacheList.Len() > 500 { - valToRemove := validatorCacheList.Remove(validatorCacheList.Front()).(cachedValidator) - delete(validatorCache, valToRemove.marshalled) - } } + // get validator from cache + validator = types.MustUnmarshalValidator(k.cdc, addr, value) + cachedVal := cachedValidator{validator, strValue} + validatorCache[strValue] = cachedValidator{validator, strValue} + validatorCacheList.PushBack(cachedVal) + if validatorCacheList.Len() > 500 { + valToRemove := validatorCacheList.Remove(validatorCacheList.Front()).(cachedValidator) + delete(validatorCache, valToRemove.marshalled) + } + validator = types.MustUnmarshalValidator(k.cdc, addr, value) return validator, true } From f29fdcafddd84e1faf290305b840aca1cddd7ec2 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Sat, 1 Sep 2018 13:21:42 -0700 Subject: [PATCH 30/49] Add comments --- x/stake/keeper/validator.go | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/x/stake/keeper/validator.go b/x/stake/keeper/validator.go index a12801dc9..d5fba1433 100644 --- a/x/stake/keeper/validator.go +++ b/x/stake/keeper/validator.go @@ -12,12 +12,16 @@ import ( "github.com/cosmos/cosmos-sdk/x/stake/types" ) +// Cache the amino decoding of validators, as it can be the case that repeated slashing calls +// cause many calls to GetValidator, which were shown to throttle the state machine in our +// simulation. Note this is quite biased though, as the simulator does more slashes than a +// live chain should, however we require the slashing to be fast as noone pays gas for it. type cachedValidator struct { val types.Validator marshalled string } -var validatorCache = make(map[string]cachedValidator, 1000) +var validatorCache = make(map[string]cachedValidator, 500) var validatorCacheList = list.New() // get a single validator @@ -27,7 +31,7 @@ func (k Keeper) GetValidator(ctx sdk.Context, addr sdk.ValAddress) (validator ty if value == nil { return validator, false } - // return cached validator + // If these amino encoded bytes are in the cache, return the cached validator strValue := string(value) if val, ok := validatorCache[strValue]; ok { valToReturn := val.val @@ -35,11 +39,12 @@ func (k Keeper) GetValidator(ctx sdk.Context, addr sdk.ValAddress) (validator ty valToReturn.Operator = addr return valToReturn, true } - // get validator from cache + // amino bytes weren't found in cache, so amino unmarshal and add it to the cache validator = types.MustUnmarshalValidator(k.cdc, addr, value) cachedVal := cachedValidator{validator, strValue} validatorCache[strValue] = cachedValidator{validator, strValue} validatorCacheList.PushBack(cachedVal) + // if the cache is too big, pop off the last element from it if validatorCacheList.Len() > 500 { valToRemove := validatorCacheList.Remove(validatorCacheList.Front()).(cachedValidator) delete(validatorCache, valToRemove.marshalled) From 8a452b940a38c799ebb432361a0daeeec833fe63 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Sat, 1 Sep 2018 19:04:44 -0700 Subject: [PATCH 31/49] simulation: display db size at end of simulation, add makefile entries --- Makefile | 11 +++++++++++ PENDING.md | 2 +- cmd/gaia/app/sim_test.go | 14 +++++++++++--- 3 files changed, 23 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 983234dbc..e363891e2 100644 --- a/Makefile +++ b/Makefile @@ -163,6 +163,17 @@ test_sim_gaia_slow: @echo "Running full Gaia simulation. This may take awhile!" @go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=1000 -SimulationVerbose=true -v -timeout 24h +SIM_NUM_BLOCKS ?= 210 +SIM_BLOCK_SIZE ?= 200 +SIM_COMMIT ?= true +test_sim_gaia_benchmark: + @echo "Running Gaia benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!" + @go test -benchmem -run=^$$ github.com/cosmos/cosmos-sdk/cmd/gaia/app -bench ^BenchmarkFullGaiaSimulation$$ -SimulationEnabled=true -SimulationNumBlocks=$(SIM_NUM_BLOCKS) -SimulationBlockSize=$(SIM_BLOCK_SIZE) -SimulationCommit=$(SIM_COMMIT) -timeout 24h + +test_sim_gaia_profile: + @echo "Running Gaia benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!" + @go test -benchmem -run=^$$ github.com/cosmos/cosmos-sdk/cmd/gaia/app -bench ^BenchmarkFullGaiaSimulation$$ -SimulationEnabled=true -SimulationNumBlocks=$(SIM_NUM_BLOCKS) -SimulationBlockSize=$(SIM_BLOCK_SIZE) -SimulationCommit=$(SIM_COMMIT) -timeout 24h -cpuprofile cpu.out -memprofile mem.out + test_cover: @bash tests/test_cover.sh diff --git a/PENDING.md b/PENDING.md index 5557a8771..34e202999 100644 --- a/PENDING.md +++ b/PENDING.md @@ -58,7 +58,7 @@ FEATURES * SDK * [querier] added custom querier functionality, so ABCI query requests can be handled by keepers * [simulation] \#1924 allow operations to specify future operations - * [simulation] \#1924 Add benchmarking capabilities + * [simulation] \#1924 Add benchmarking capabilities, with makefile commands "test_sim_gaia_benchmark, test_sim_gaia_profile" * Tendermint diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go index 58d7ca18d..3878166b3 100644 --- a/cmd/gaia/app/sim_test.go +++ b/cmd/gaia/app/sim_test.go @@ -3,6 +3,7 @@ package app import ( "encoding/json" "flag" + "fmt" "math/rand" "os" "testing" @@ -118,7 +119,7 @@ func invariants(app *GaiaApp) []simulation.Invariant { } // Profile with: -// /usr/local/go/bin/go test -benchmem -run=^$ github.com/cosmos/cosmos-sdk/cmd/gaia/app -bench ^BenchmarkFullGaiaSimulation$ -cpuprofile cpu.out +// /usr/local/go/bin/go test -benchmem -run=^$ github.com/cosmos/cosmos-sdk/cmd/gaia/app -bench ^BenchmarkFullGaiaSimulation$ -SimulationCommit=true -cpuprofile cpu.out func BenchmarkFullGaiaSimulation(b *testing.B) { // Setup Gaia application var logger log.Logger @@ -139,10 +140,15 @@ func BenchmarkFullGaiaSimulation(b *testing.B) { testAndRunTxs(app), []simulation.RandSetup{}, invariants(app), // these shouldn't get ran - 210, + numBlocks, blockSize, commit, ) + if commit { + fmt.Println("GoLevelDB Stats") + fmt.Println(db.Stats()["leveldb.stats"]) + fmt.Println("GoLevelDB cached block size", db.Stats()["leveldb.cachedblock"]) + } } func TestFullGaiaSimulation(t *testing.T) { @@ -171,7 +177,9 @@ func TestFullGaiaSimulation(t *testing.T) { blockSize, commit, ) - + if commit { + fmt.Println("Database Size", db.Stats()["database.size"]) + } } // TODO: Make another test for the fuzzer itself, which just has noOp txs From 2c66ba0bd4e47a4692f55105b0c405a63b004875 Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Sun, 2 Sep 2018 15:42:25 -0400 Subject: [PATCH 32/49] extra comment on cache key usage --- x/stake/keeper/validator.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/x/stake/keeper/validator.go b/x/stake/keeper/validator.go index d5fba1433..2d8f947fc 100644 --- a/x/stake/keeper/validator.go +++ b/x/stake/keeper/validator.go @@ -18,9 +18,10 @@ import ( // live chain should, however we require the slashing to be fast as noone pays gas for it. type cachedValidator struct { val types.Validator - marshalled string + marshalled string // marshalled amino bytes for the validator object (not operator address) } +// validatorCache-key: validator amino bytes var validatorCache = make(map[string]cachedValidator, 500) var validatorCacheList = list.New() @@ -31,6 +32,7 @@ func (k Keeper) GetValidator(ctx sdk.Context, addr sdk.ValAddress) (validator ty if value == nil { return validator, false } + // If these amino encoded bytes are in the cache, return the cached validator strValue := string(value) if val, ok := validatorCache[strValue]; ok { @@ -39,11 +41,13 @@ func (k Keeper) GetValidator(ctx sdk.Context, addr sdk.ValAddress) (validator ty valToReturn.Operator = addr return valToReturn, true } + // amino bytes weren't found in cache, so amino unmarshal and add it to the cache validator = types.MustUnmarshalValidator(k.cdc, addr, value) cachedVal := cachedValidator{validator, strValue} validatorCache[strValue] = cachedValidator{validator, strValue} validatorCacheList.PushBack(cachedVal) + // if the cache is too big, pop off the last element from it if validatorCacheList.Len() > 500 { valToRemove := validatorCacheList.Remove(validatorCacheList.Front()).(cachedValidator) From b72f87a0270cf6b234f5e4d15855c032cf5a1d6e Mon Sep 17 00:00:00 2001 From: Jeremiah Andrews Date: Fri, 3 Aug 2018 16:24:03 -0700 Subject: [PATCH 33/49] Update IAVL dependency for v0.10.0 --- Gopkg.lock | 8 ++++---- Gopkg.toml | 2 +- store/iavlstore.go | 16 ++++++++-------- store/iavlstore_test.go | 12 ++++++------ store/prefixstore_test.go | 2 +- 5 files changed, 20 insertions(+), 20 deletions(-) diff --git a/Gopkg.lock b/Gopkg.lock index 5e40c17e6..5edc19816 100644 --- a/Gopkg.lock +++ b/Gopkg.lock @@ -415,12 +415,12 @@ version = "v0.12.0-rc0" [[projects]] - digest = "1:d4a15d404afbf591e8be16fcda7f5ac87948d5c7531f9d909fd84cc730ab16e2" + digest = "1:e99ef92d64f2391efbbfb15310df635f96247532bbac2676ea43e466d706401d" name = "github.com/tendermint/iavl" packages = ["."] pruneopts = "UT" - revision = "35f66e53d9b01e83b30de68b931f54b2477a94c9" - version = "v0.9.2" + revision = "e5726c0066ccdd299a2ec9262f93c7896cdfcd87" + version = "v0.10.0" [[projects]] digest = "1:4f15e95fe3888cc75dd34f407d6394cbc7fd3ff24920851b92b295f6a8b556e6" @@ -579,7 +579,7 @@ [[projects]] branch = "master" - digest = "1:077c1c599507b3b3e9156d17d36e1e61928ee9b53a5b420f10f28ebd4a0b275c" + digest = "1:601e63e7d4577f907118bec825902505291918859d223bce015539e79f1160e3" name = "google.golang.org/genproto" packages = ["googleapis/rpc/status"] pruneopts = "UT" diff --git a/Gopkg.toml b/Gopkg.toml index 4368699b6..f8194ccd0 100644 --- a/Gopkg.toml +++ b/Gopkg.toml @@ -53,7 +53,7 @@ [[override]] name = "github.com/tendermint/iavl" - version = "=v0.9.2" + version = "=v0.10.0" [[override]] name = "github.com/tendermint/tendermint" diff --git a/store/iavlstore.go b/store/iavlstore.go index daffa7dd5..57c103238 100644 --- a/store/iavlstore.go +++ b/store/iavlstore.go @@ -20,7 +20,7 @@ const ( // load the iavl store func LoadIAVLStore(db dbm.DB, id CommitID, pruning sdk.PruningStrategy) (CommitStore, error) { - tree := iavl.NewVersionedTree(db, defaultIAVLCacheSize) + tree := iavl.NewMutableTree(db, defaultIAVLCacheSize) _, err := tree.LoadVersion(id.Version) if err != nil { return nil, err @@ -40,7 +40,7 @@ var _ Queryable = (*iavlStore)(nil) type iavlStore struct { // The underlying tree. - tree *iavl.VersionedTree + tree *iavl.MutableTree // How many old versions we hold onto. // A value of 0 means keep no recent states. @@ -56,7 +56,7 @@ type iavlStore struct { } // CONTRACT: tree should be fully loaded. -func newIAVLStore(tree *iavl.VersionedTree, numRecent int64, storeEvery int64) *iavlStore { +func newIAVLStore(tree *iavl.MutableTree, numRecent int64, storeEvery int64) *iavlStore { st := &iavlStore{ tree: tree, numRecent: numRecent, @@ -167,16 +167,16 @@ func (st *iavlStore) Gas(meter GasMeter, config GasConfig) KVStore { // Implements KVStore. func (st *iavlStore) Iterator(start, end []byte) Iterator { - return newIAVLIterator(st.tree.Tree(), start, end, true) + return newIAVLIterator(st.tree.ImmutableTree, start, end, true) } // Implements KVStore. func (st *iavlStore) ReverseIterator(start, end []byte) Iterator { - return newIAVLIterator(st.tree.Tree(), start, end, false) + return newIAVLIterator(st.tree.ImmutableTree, start, end, false) } // Handle gatest the latest height, if height is 0 -func getHeight(tree *iavl.VersionedTree, req abci.RequestQuery) int64 { +func getHeight(tree *iavl.MutableTree, req abci.RequestQuery) int64 { height := req.Height if height == 0 { latest := tree.Version64() @@ -255,7 +255,7 @@ func (st *iavlStore) Query(req abci.RequestQuery) (res abci.ResponseQuery) { // Implements Iterator. type iavlIterator struct { // Underlying store - tree *iavl.Tree + tree *iavl.ImmutableTree // Domain start, end []byte @@ -286,7 +286,7 @@ var _ Iterator = (*iavlIterator)(nil) // newIAVLIterator will create a new iavlIterator. // CONTRACT: Caller must release the iavlIterator, as each one creates a new // goroutine. -func newIAVLIterator(tree *iavl.Tree, start, end []byte, ascending bool) *iavlIterator { +func newIAVLIterator(tree *iavl.ImmutableTree, start, end []byte, ascending bool) *iavlIterator { iter := &iavlIterator{ tree: tree, start: cp(start), diff --git a/store/iavlstore_test.go b/store/iavlstore_test.go index 793089a26..c200a63d4 100644 --- a/store/iavlstore_test.go +++ b/store/iavlstore_test.go @@ -29,8 +29,8 @@ var ( ) // make a tree and save it -func newTree(t *testing.T, db dbm.DB) (*iavl.VersionedTree, CommitID) { - tree := iavl.NewVersionedTree(db, cacheSize) +func newTree(t *testing.T, db dbm.DB) (*iavl.MutableTree, CommitID) { + tree := iavl.NewMutableTree(db, cacheSize) for k, v := range treeData { tree.Set([]byte(k), []byte(v)) } @@ -325,7 +325,7 @@ type pruneState struct { func testPruning(t *testing.T, numRecent int64, storeEvery int64, states []pruneState) { db := dbm.NewMemDB() - tree := iavl.NewVersionedTree(db, cacheSize) + tree := iavl.NewMutableTree(db, cacheSize) iavlStore := newIAVLStore(tree, numRecent, storeEvery) for step, state := range states { for _, ver := range state.stored { @@ -344,7 +344,7 @@ func testPruning(t *testing.T, numRecent int64, storeEvery int64, states []prune func TestIAVLNoPrune(t *testing.T) { db := dbm.NewMemDB() - tree := iavl.NewVersionedTree(db, cacheSize) + tree := iavl.NewMutableTree(db, cacheSize) iavlStore := newIAVLStore(tree, numRecent, int64(1)) nextVersion(iavlStore) for i := 1; i < 100; i++ { @@ -359,7 +359,7 @@ func TestIAVLNoPrune(t *testing.T) { func TestIAVLPruneEverything(t *testing.T) { db := dbm.NewMemDB() - tree := iavl.NewVersionedTree(db, cacheSize) + tree := iavl.NewMutableTree(db, cacheSize) iavlStore := newIAVLStore(tree, int64(0), int64(0)) nextVersion(iavlStore) for i := 1; i < 100; i++ { @@ -377,7 +377,7 @@ func TestIAVLPruneEverything(t *testing.T) { func TestIAVLStoreQuery(t *testing.T) { db := dbm.NewMemDB() - tree := iavl.NewVersionedTree(db, cacheSize) + tree := iavl.NewMutableTree(db, cacheSize) iavlStore := newIAVLStore(tree, numRecent, storeEvery) k1, v1 := []byte("key1"), []byte("val1") diff --git a/store/prefixstore_test.go b/store/prefixstore_test.go index ff37b27d4..49bc68037 100644 --- a/store/prefixstore_test.go +++ b/store/prefixstore_test.go @@ -66,7 +66,7 @@ func testPrefixStore(t *testing.T, baseStore KVStore, prefix []byte) { func TestIAVLStorePrefix(t *testing.T) { db := dbm.NewMemDB() - tree := iavl.NewVersionedTree(db, cacheSize) + tree := iavl.NewMutableTree(db, cacheSize) iavlStore := newIAVLStore(tree, numRecent, storeEvery) testPrefixStore(t, iavlStore, []byte("test")) From b83e6e8b1aa21422a27533da949bed6b587809e9 Mon Sep 17 00:00:00 2001 From: Jeremiah Andrews Date: Mon, 20 Aug 2018 22:15:58 -0700 Subject: [PATCH 34/49] Add to pending changelog --- PENDING.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/PENDING.md b/PENDING.md index 17616ced1..00e609039 100644 --- a/PENDING.md +++ b/PENDING.md @@ -1,4 +1,17 @@ -## PENDING +## v0.24.1 PENDING + +BREAKING CHANGES + +FEATURES + +IMPROVEMENTS + +* [store] \#1952 Update IAVL dependency to v0.10.0 + +BUG FIXES + +## v0.24.0 PENDING +^--- PENDING wasn't purged on sdk v0.23.0 release. BREAKING CHANGES From bd227ec6dbca3623a72f382ffe9053b80ab325cb Mon Sep 17 00:00:00 2001 From: Christopher Goes Date: Tue, 21 Aug 2018 15:53:44 +0200 Subject: [PATCH 35/49] dep ensure -v -update --- Gopkg.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Gopkg.lock b/Gopkg.lock index 5edc19816..224522341 100644 --- a/Gopkg.lock +++ b/Gopkg.lock @@ -552,7 +552,7 @@ "unix", ] pruneopts = "UT" - revision = "11551d06cbcc94edc80a0facaccbda56473c19c1" + revision = "4ea2f632f6e912459fe60b26b1749377f0d889d5" [[projects]] digest = "1:a2ab62866c75542dd18d2b069fec854577a20211d7c0ea6ae746072a1dccdd18" @@ -579,7 +579,7 @@ [[projects]] branch = "master" - digest = "1:601e63e7d4577f907118bec825902505291918859d223bce015539e79f1160e3" + digest = "1:077c1c599507b3b3e9156d17d36e1e61928ee9b53a5b420f10f28ebd4a0b275c" name = "google.golang.org/genproto" packages = ["googleapis/rpc/status"] pruneopts = "UT" From 3ac55b589dfd7b5b22135451c41b2fa84017f016 Mon Sep 17 00:00:00 2001 From: Christopher Goes Date: Wed, 29 Aug 2018 15:08:04 +0200 Subject: [PATCH 36/49] Update testcase from merge --- store/iavlstore_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/store/iavlstore_test.go b/store/iavlstore_test.go index c200a63d4..49793d376 100644 --- a/store/iavlstore_test.go +++ b/store/iavlstore_test.go @@ -468,7 +468,7 @@ func TestIAVLStoreQuery(t *testing.T) { func BenchmarkIAVLIteratorNext(b *testing.B) { db := dbm.NewMemDB() treeSize := 1000 - tree := iavl.NewVersionedTree(db, cacheSize) + tree := iavl.NewMutableTree(db, cacheSize) for i := 0; i < treeSize; i++ { key := cmn.RandBytes(4) value := cmn.RandBytes(50) From 2378e3431c050d4763a7d362a71f07b98ce1d532 Mon Sep 17 00:00:00 2001 From: Jeremiah Andrews Date: Fri, 31 Aug 2018 15:22:37 -0700 Subject: [PATCH 37/49] Fix linter errors --- baseapp/baseapp.go | 1 + baseapp/baseapp_test.go | 4 ++-- client/context/query.go | 4 +++- cmd/cosmos-sdk-cli/cmd/init.go | 3 +++ examples/democoin/x/assoc/validator_set.go | 3 +++ server/init.go | 1 + server/start.go | 1 + store/iavlstore.go | 1 + store/multistoreproof_test.go | 8 +++++--- store/tracekvstore.go | 1 + store/tracekvstore_test.go | 8 ++++---- tests/gobash.go | 1 + types/account.go | 3 +++ types/context.go | 1 + types/errors.go | 1 + x/auth/client/cli/account.go | 1 + x/gov/depositsvotes.go | 1 + x/gov/keeper.go | 6 ++++++ x/gov/proposals.go | 2 ++ x/gov/queryable.go | 7 +++++++ x/gov/simulation/msgs.go | 1 + x/ibc/client/cli/relay.go | 1 + x/mock/app.go | 2 ++ x/mock/simulation/random_simulate_blocks.go | 1 + x/params/keeper.go | 1 + x/params/msg_status.go | 1 + x/slashing/client/rest/query.go | 1 + x/slashing/keeper_test.go | 10 +++++----- x/stake/client/rest/utils.go | 1 + x/stake/handler_test.go | 2 +- x/stake/keeper/key.go | 1 + x/stake/keeper/slash.go | 1 + x/stake/keeper/validator.go | 2 ++ x/stake/simulation/invariants.go | 1 + x/stake/simulation/msgs.go | 1 + x/stake/types/delegation_test.go | 4 ++-- 36 files changed, 71 insertions(+), 18 deletions(-) diff --git a/baseapp/baseapp.go b/baseapp/baseapp.go index 60c694ff7..5552ff784 100644 --- a/baseapp/baseapp.go +++ b/baseapp/baseapp.go @@ -345,6 +345,7 @@ func handleQueryStore(app *BaseApp, path []string, req abci.RequestQuery) (res a return queryable.Query(req) } +// nolint: unparam func handleQueryP2P(app *BaseApp, path []string, req abci.RequestQuery) (res abci.ResponseQuery) { // "/p2p" prefix for p2p queries if len(path) >= 4 { diff --git a/baseapp/baseapp_test.go b/baseapp/baseapp_test.go index 04e47214d..b87162786 100644 --- a/baseapp/baseapp_test.go +++ b/baseapp/baseapp_test.go @@ -626,12 +626,12 @@ func TestSimulateTx(t *testing.T) { // simulate a message, check gas reported result := app.Simulate(tx) require.True(t, result.IsOK(), result.Log) - require.Equal(t, int64(gasConsumed), result.GasUsed) + require.Equal(t, gasConsumed, result.GasUsed) // simulate again, same result result = app.Simulate(tx) require.True(t, result.IsOK(), result.Log) - require.Equal(t, int64(gasConsumed), result.GasUsed) + require.Equal(t, gasConsumed, result.GasUsed) // simulate by calling Query with encoded tx txBytes, err := codec.MarshalBinary(tx) diff --git a/client/context/query.go b/client/context/query.go index 4c1cad877..33610bc15 100644 --- a/client/context/query.go +++ b/client/context/query.go @@ -10,6 +10,8 @@ import ( "github.com/pkg/errors" + "strings" + "github.com/cosmos/cosmos-sdk/store" "github.com/cosmos/cosmos-sdk/wire" abci "github.com/tendermint/tendermint/abci/types" @@ -17,7 +19,6 @@ import ( tmliteProxy "github.com/tendermint/tendermint/lite/proxy" rpcclient "github.com/tendermint/tendermint/rpc/client" ctypes "github.com/tendermint/tendermint/rpc/core/types" - "strings" ) // GetNode returns an RPC client. If the context's client is not defined, an @@ -323,6 +324,7 @@ func (ctx CLIContext) query(path string, key cmn.HexBytes) (res []byte, err erro } // verifyProof perform response proof verification +// nolint: unparam func (ctx CLIContext) verifyProof(path string, resp abci.ResponseQuery) error { if ctx.Certifier == nil { diff --git a/cmd/cosmos-sdk-cli/cmd/init.go b/cmd/cosmos-sdk-cli/cmd/init.go index 8cebc48f2..e9e9cd8a9 100644 --- a/cmd/cosmos-sdk-cli/cmd/init.go +++ b/cmd/cosmos-sdk-cli/cmd/init.go @@ -60,6 +60,7 @@ func resolveProjectPath(remoteProjectPath string) string { return gopath + string(os.PathSeparator) + "src" + string(os.PathSeparator) + remoteProjectPath } +// nolint: unparam, errcheck func copyBasecoinTemplate(projectName string, projectPath string, remoteProjectPath string) { basecoinProjectPath := resolveProjectPath(remoteBasecoinPath) filepath.Walk(basecoinProjectPath, func(path string, f os.FileInfo, err error) error { @@ -88,6 +89,7 @@ func copyBasecoinTemplate(projectName string, projectPath string, remoteProjectP }) } +// nolint: errcheck func createGopkg(projectPath string) { // Create gopkg.toml file dependencies := map[string]string{ @@ -111,6 +113,7 @@ func createGopkg(projectPath string) { ioutil.WriteFile(projectPath+"/Gopkg.toml", []byte(contents), os.ModePerm) } +// nolint: errcheck func createMakefile(projectPath string) { // Create makefile // TODO: Should we use tools/ directory as in Cosmos-SDK to get tools for linting etc. diff --git a/examples/democoin/x/assoc/validator_set.go b/examples/democoin/x/assoc/validator_set.go index 8a954c720..7515e1ad6 100644 --- a/examples/democoin/x/assoc/validator_set.go +++ b/examples/democoin/x/assoc/validator_set.go @@ -62,6 +62,7 @@ func GetAssocKey(base sdk.ValAddress, assoc sdk.ValAddress) []byte { } // Associate associates new address with validator address +// nolint: unparam func (valset ValidatorSet) Associate(ctx sdk.Context, base sdk.ValAddress, assoc sdk.ValAddress) bool { if len(base) != valset.addrLen || len(assoc) != valset.addrLen { return false @@ -76,6 +77,7 @@ func (valset ValidatorSet) Associate(ctx sdk.Context, base sdk.ValAddress, assoc } // Dissociate removes association between addresses +// nolint: unparam func (valset ValidatorSet) Dissociate(ctx sdk.Context, base sdk.ValAddress, assoc sdk.ValAddress) bool { if len(base) != valset.addrLen || len(assoc) != valset.addrLen { return false @@ -90,6 +92,7 @@ func (valset ValidatorSet) Dissociate(ctx sdk.Context, base sdk.ValAddress, asso } // Associations returns all associated addresses with a validator +// nolint: unparam func (valset ValidatorSet) Associations(ctx sdk.Context, base sdk.ValAddress) (res []sdk.ValAddress) { res = make([]sdk.ValAddress, valset.maxAssoc) iter := sdk.KVStorePrefixIterator(valset.store, GetAssocPrefix(base)) diff --git a/server/init.go b/server/init.go index 105325e0d..ffe2b6edd 100644 --- a/server/init.go +++ b/server/init.go @@ -344,6 +344,7 @@ func readOrCreatePrivValidator(tmConfig *cfg.Config) crypto.PubKey { // writeGenesisFile creates and writes the genesis configuration to disk. An // error is returned if building or writing the configuration to file fails. +// nolint: unparam func writeGenesisFile(cdc *wire.Codec, genesisFile, chainID string, validators []tmtypes.GenesisValidator, appState json.RawMessage) error { genDoc := tmtypes.GenesisDoc{ ChainID: chainID, diff --git a/server/start.go b/server/start.go index 8f369d517..829e39363 100644 --- a/server/start.go +++ b/server/start.go @@ -84,6 +84,7 @@ func startStandAlone(ctx *Context, appCreator AppCreator) error { return nil } +// nolint: unparam func startInProcess(ctx *Context, appCreator AppCreator) (*node.Node, error) { cfg := ctx.Config home := cfg.RootDir diff --git a/store/iavlstore.go b/store/iavlstore.go index 57c103238..61bdc5630 100644 --- a/store/iavlstore.go +++ b/store/iavlstore.go @@ -56,6 +56,7 @@ type iavlStore struct { } // CONTRACT: tree should be fully loaded. +// nolint: unparam func newIAVLStore(tree *iavl.MutableTree, numRecent int64, storeEvery int64) *iavlStore { st := &iavlStore{ tree: tree, diff --git a/store/multistoreproof_test.go b/store/multistoreproof_test.go index b4e8a84b1..790588fb2 100644 --- a/store/multistoreproof_test.go +++ b/store/multistoreproof_test.go @@ -2,10 +2,12 @@ package store import ( "encoding/hex" + "testing" + "github.com/stretchr/testify/assert" "github.com/tendermint/iavl" cmn "github.com/tendermint/tendermint/libs/common" - "testing" + "github.com/tendermint/tendermint/libs/db" ) func TestVerifyMultiStoreCommitInfo(t *testing.T) { @@ -91,7 +93,7 @@ func TestVerifyMultiStoreCommitInfo(t *testing.T) { } func TestVerifyRangeProof(t *testing.T) { - tree := iavl.NewTree(nil, 0) + tree := iavl.NewMutableTree(db.NewMemDB(), 0) rand := cmn.NewRand() rand.Seed(0) // for determinism @@ -100,7 +102,7 @@ func TestVerifyRangeProof(t *testing.T) { tree.Set(key, []byte(rand.Str(8))) } - root := tree.Hash() + root := tree.WorkingHash() key := []byte{0x32} val, proof, err := tree.GetWithProof(key) diff --git a/store/tracekvstore.go b/store/tracekvstore.go index 0224e8c12..d8c34c354 100644 --- a/store/tracekvstore.go +++ b/store/tracekvstore.go @@ -179,6 +179,7 @@ func (tkv *TraceKVStore) CacheWrapWithTrace(_ io.Writer, _ TraceContext) CacheWr // writeOperation writes a KVStore operation to the underlying io.Writer as // JSON-encoded data where the key/value pair is base64 encoded. +// nolint: errcheck func writeOperation(w io.Writer, op operation, tc TraceContext, key, value []byte) { traceOp := traceOperation{ Operation: op, diff --git a/store/tracekvstore_test.go b/store/tracekvstore_test.go index 2182c5288..887fcf96e 100644 --- a/store/tracekvstore_test.go +++ b/store/tracekvstore_test.go @@ -156,8 +156,8 @@ func TestTestTraceKVStoreIterator(t *testing.T) { iterator := store.Iterator(nil, nil) s, e := iterator.Domain() - require.Equal(t, []uint8([]byte(nil)), s) - require.Equal(t, []uint8([]byte(nil)), e) + require.Equal(t, []byte(nil), s) + require.Equal(t, []byte(nil), e) testCases := []struct { expectedKey []byte @@ -212,8 +212,8 @@ func TestTestTraceKVStoreReverseIterator(t *testing.T) { iterator := store.ReverseIterator(nil, nil) s, e := iterator.Domain() - require.Equal(t, []uint8([]byte(nil)), s) - require.Equal(t, []uint8([]byte(nil)), e) + require.Equal(t, []byte(nil), s) + require.Equal(t, []byte(nil), e) testCases := []struct { expectedKey []byte diff --git a/tests/gobash.go b/tests/gobash.go index 11f4407f2..6282f2fda 100644 --- a/tests/gobash.go +++ b/tests/gobash.go @@ -13,6 +13,7 @@ import ( // ExecuteT executes the command, pipes any input to STDIN and return STDOUT, // logging STDOUT/STDERR to t. +// nolint: errcheck func ExecuteT(t *testing.T, cmd, input string) (out string) { t.Log("Running", cmn.Cyan(cmd)) diff --git a/types/account.go b/types/account.go index 429cbf285..943867d79 100644 --- a/types/account.go +++ b/types/account.go @@ -133,6 +133,7 @@ func (aa AccAddress) String() string { } // Format implements the fmt.Formatter interface. +// nolint: errcheck func (aa AccAddress) Format(s fmt.State, verb rune) { switch verb { case 's': @@ -247,6 +248,7 @@ func (va ValAddress) String() string { } // Format implements the fmt.Formatter interface. +// nolint: errcheck func (va ValAddress) Format(s fmt.State, verb rune) { switch verb { case 's': @@ -361,6 +363,7 @@ func (ca ConsAddress) String() string { } // Format implements the fmt.Formatter interface. +// nolint: errcheck func (ca ConsAddress) Format(s fmt.State, verb rune) { switch verb { case 's': diff --git a/types/context.go b/types/context.go index 44b0474e8..85fb16a7f 100644 --- a/types/context.go +++ b/types/context.go @@ -30,6 +30,7 @@ type Context struct { } // create a new context +// nolint: unparam func NewContext(ms MultiStore, header abci.Header, isCheckTx bool, logger log.Logger) Context { c := Context{ Context: context.Background(), diff --git a/types/errors.go b/types/errors.go index 45ef79f56..c72933d60 100644 --- a/types/errors.go +++ b/types/errors.go @@ -219,6 +219,7 @@ func (err *sdkError) WithDefaultCodespace(cs CodespaceType) Error { } // Implements ABCIError. +// nolint: errcheck func (err *sdkError) TraceSDK(format string, args ...interface{}) Error { err.Trace(1, format, args...) return err diff --git a/x/auth/client/cli/account.go b/x/auth/client/cli/account.go index 3b93798a0..6f6359288 100644 --- a/x/auth/client/cli/account.go +++ b/x/auth/client/cli/account.go @@ -30,6 +30,7 @@ func GetAccountDecoder(cdc *wire.Codec) auth.AccountDecoder { // GetAccountCmd returns a query account that will display the state of the // account at a given address. +// nolint: unparam func GetAccountCmd(storeName string, cdc *wire.Codec, decoder auth.AccountDecoder) *cobra.Command { return &cobra.Command{ Use: "account [address]", diff --git a/x/gov/depositsvotes.go b/x/gov/depositsvotes.go index b5c65931f..d1179023f 100644 --- a/x/gov/depositsvotes.go +++ b/x/gov/depositsvotes.go @@ -132,6 +132,7 @@ func (vo VoteOption) String() string { } // For Printf / Sprintf, returns bech32 when using %s +// nolint: errcheck func (vo VoteOption) Format(s fmt.State, verb rune) { switch verb { case 's': diff --git a/x/gov/keeper.go b/x/gov/keeper.go index e8dbbc4a7..576d2cc22 100644 --- a/x/gov/keeper.go +++ b/x/gov/keeper.go @@ -210,6 +210,7 @@ func (keeper Keeper) activateVotingPeriod(ctx sdk.Context, proposal Proposal) { // Procedures // Returns the current Deposit Procedure from the global param store +// nolint: errcheck func (keeper Keeper) GetDepositProcedure(ctx sdk.Context) DepositProcedure { var depositProcedure DepositProcedure keeper.ps.Get(ctx, ParamStoreKeyDepositProcedure, &depositProcedure) @@ -217,6 +218,7 @@ func (keeper Keeper) GetDepositProcedure(ctx sdk.Context) DepositProcedure { } // Returns the current Voting Procedure from the global param store +// nolint: errcheck func (keeper Keeper) GetVotingProcedure(ctx sdk.Context) VotingProcedure { var votingProcedure VotingProcedure keeper.ps.Get(ctx, ParamStoreKeyVotingProcedure, &votingProcedure) @@ -224,20 +226,24 @@ func (keeper Keeper) GetVotingProcedure(ctx sdk.Context) VotingProcedure { } // Returns the current Tallying Procedure from the global param store +// nolint: errcheck func (keeper Keeper) GetTallyingProcedure(ctx sdk.Context) TallyingProcedure { var tallyingProcedure TallyingProcedure keeper.ps.Get(ctx, ParamStoreKeyTallyingProcedure, &tallyingProcedure) return tallyingProcedure } +// nolint: errcheck func (keeper Keeper) setDepositProcedure(ctx sdk.Context, depositProcedure DepositProcedure) { keeper.ps.Set(ctx, ParamStoreKeyDepositProcedure, &depositProcedure) } +// nolint: errcheck func (keeper Keeper) setVotingProcedure(ctx sdk.Context, votingProcedure VotingProcedure) { keeper.ps.Set(ctx, ParamStoreKeyVotingProcedure, &votingProcedure) } +// nolint: errcheck func (keeper Keeper) setTallyingProcedure(ctx sdk.Context, tallyingProcedure TallyingProcedure) { keeper.ps.Set(ctx, ParamStoreKeyTallyingProcedure, &tallyingProcedure) } diff --git a/x/gov/proposals.go b/x/gov/proposals.go index c52ab1e87..b4c678367 100644 --- a/x/gov/proposals.go +++ b/x/gov/proposals.go @@ -187,6 +187,7 @@ func (pt ProposalKind) String() string { } // For Printf / Sprintf, returns bech32 when using %s +// nolint: errcheck func (pt ProposalKind) Format(s fmt.State, verb rune) { switch verb { case 's': @@ -289,6 +290,7 @@ func (status ProposalStatus) String() string { } // For Printf / Sprintf, returns bech32 when using %s +// nolint: errcheck func (status ProposalStatus) Format(s fmt.State, verb rune) { switch verb { case 's': diff --git a/x/gov/queryable.go b/x/gov/queryable.go index e64d506d1..090b9a914 100644 --- a/x/gov/queryable.go +++ b/x/gov/queryable.go @@ -36,6 +36,7 @@ type QueryProposalParams struct { ProposalID int64 } +// nolint: unparam func queryProposal(ctx sdk.Context, path []string, req abci.RequestQuery, keeper Keeper) (res []byte, err sdk.Error) { var params QueryProposalParams err2 := keeper.cdc.UnmarshalJSON(req.Data, ¶ms) @@ -61,6 +62,7 @@ type QueryDepositParams struct { Depositer sdk.AccAddress } +// nolint: unparam func queryDeposit(ctx sdk.Context, path []string, req abci.RequestQuery, keeper Keeper) (res []byte, err sdk.Error) { var params QueryDepositParams err2 := keeper.cdc.UnmarshalJSON(req.Data, ¶ms) @@ -82,6 +84,7 @@ type QueryVoteParams struct { Voter sdk.AccAddress } +// nolint: unparam func queryVote(ctx sdk.Context, path []string, req abci.RequestQuery, keeper Keeper) (res []byte, err sdk.Error) { var params QueryVoteParams err2 := keeper.cdc.UnmarshalJSON(req.Data, ¶ms) @@ -102,6 +105,7 @@ type QueryDepositsParams struct { ProposalID int64 } +// nolint: unparam func queryDeposits(ctx sdk.Context, path []string, req abci.RequestQuery, keeper Keeper) (res []byte, err sdk.Error) { var params QueryDepositParams err2 := keeper.cdc.UnmarshalJSON(req.Data, ¶ms) @@ -129,6 +133,7 @@ type QueryVotesParams struct { ProposalID int64 } +// nolint: unparam func queryVotes(ctx sdk.Context, path []string, req abci.RequestQuery, keeper Keeper) (res []byte, err sdk.Error) { var params QueryVotesParams err2 := keeper.cdc.UnmarshalJSON(req.Data, ¶ms) @@ -160,6 +165,7 @@ type QueryProposalsParams struct { NumLatestProposals int64 } +// nolint: unparam func queryProposals(ctx sdk.Context, path []string, req abci.RequestQuery, keeper Keeper) (res []byte, err sdk.Error) { var params QueryProposalsParams err2 := keeper.cdc.UnmarshalJSON(req.Data, ¶ms) @@ -181,6 +187,7 @@ type QueryTallyParams struct { ProposalID int64 } +// nolint: unparam func queryTally(ctx sdk.Context, path []string, req abci.RequestQuery, keeper Keeper) (res []byte, err sdk.Error) { // TODO: Dependant on #1914 diff --git a/x/gov/simulation/msgs.go b/x/gov/simulation/msgs.go index eca8accae..13e537efa 100644 --- a/x/gov/simulation/msgs.go +++ b/x/gov/simulation/msgs.go @@ -86,6 +86,7 @@ func SimulateMsgDeposit(k gov.Keeper, sk stake.Keeper) simulation.Operation { } // SimulateMsgVote +// nolint: unparam func SimulateMsgVote(k gov.Keeper, sk stake.Keeper) simulation.Operation { return func(tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, keys []crypto.PrivKey, log string, event func(string)) (action string, fOp []simulation.FutureOperation, err sdk.Error) { key := simulation.RandomKey(r, keys) diff --git a/x/ibc/client/cli/relay.go b/x/ibc/client/cli/relay.go index 92b03a66f..37479980c 100644 --- a/x/ibc/client/cli/relay.go +++ b/x/ibc/client/cli/relay.go @@ -163,6 +163,7 @@ func query(node string, key []byte, storeName string) (res []byte, err error) { return context.NewCLIContext().WithNodeURI(node).QueryStore(key, storeName) } +// nolint: unparam func (c relayCommander) broadcastTx(seq int64, node string, tx []byte) error { _, err := context.NewCLIContext().WithNodeURI(node).BroadcastTx(tx) return err diff --git a/x/mock/app.go b/x/mock/app.go index f472c5531..97068a3a5 100644 --- a/x/mock/app.go +++ b/x/mock/app.go @@ -86,6 +86,7 @@ func (app *App) CompleteSetup(newKeys []*sdk.KVStoreKey) error { } // InitChainer performs custom logic for initialization. +// nolint: errcheck func (app *App) InitChainer(ctx sdk.Context, _ abci.RequestInitChain) abci.ResponseInitChain { // Load the genesis accounts for _, genacc := range app.GenesisAccounts { @@ -207,6 +208,7 @@ func GeneratePrivKeyAddressPairsFromRand(rand *rand.Rand, n int) (keys []crypto. // RandomSetGenesis set genesis accounts with random coin values using the // provided addresses and coin denominations. +// nolint: errcheck func RandomSetGenesis(r *rand.Rand, app *App, addrs []sdk.AccAddress, denoms []string) { accts := make([]auth.Account, len(addrs), len(addrs)) randCoinIntervals := []BigInterval{ diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index fc7543280..11c3dbd6a 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -243,6 +243,7 @@ func getKeys(validators map[string]mockValidator) []string { } // RandomRequestBeginBlock generates a list of signing validators according to the provided list of validators, signing fraction, and evidence fraction +// nolint: unparam func RandomRequestBeginBlock(r *rand.Rand, validators map[string]mockValidator, livenessTransitions TransitionMatrix, evidenceFraction float64, pastTimes []time.Time, pastSigningValidators [][]abci.SigningValidator, event func(string), header abci.Header, log string) abci.RequestBeginBlock { if len(validators) == 0 { diff --git a/x/params/keeper.go b/x/params/keeper.go index 69bcc05ea..7fd9bb3c9 100644 --- a/x/params/keeper.go +++ b/x/params/keeper.go @@ -23,6 +23,7 @@ func NewKeeper(cdc *wire.Codec, key sdk.StoreKey) Keeper { } // InitKeeper constructs a new Keeper with initial parameters +// nolint: errcheck func InitKeeper(ctx sdk.Context, cdc *wire.Codec, key sdk.StoreKey, params ...interface{}) Keeper { if len(params)%2 != 0 { panic("Odd params list length for InitKeeper") diff --git a/x/params/msg_status.go b/x/params/msg_status.go index 7f9197c5c..a8516a85b 100644 --- a/x/params/msg_status.go +++ b/x/params/msg_status.go @@ -15,6 +15,7 @@ func ActivatedParamKey(ty string) string { } // InitGenesis stores activated type to param store +// nolint: errcheck func InitGenesis(ctx sdk.Context, k Keeper, data GenesisState) { for _, ty := range data.ActivatedTypes { k.set(ctx, ActivatedParamKey(ty), true) diff --git a/x/slashing/client/rest/query.go b/x/slashing/client/rest/query.go index 78c4a2d2f..0ebe13250 100644 --- a/x/slashing/client/rest/query.go +++ b/x/slashing/client/rest/query.go @@ -19,6 +19,7 @@ func registerQueryRoutes(cliCtx context.CLIContext, r *mux.Router, cdc *wire.Cod } // http request handler to query signing info +// nolint: unparam func signingInfoHandlerFn(cliCtx context.CLIContext, storeName string, cdc *wire.Codec) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { vars := mux.Vars(r) diff --git a/x/slashing/keeper_test.go b/x/slashing/keeper_test.go index c0932acf4..0eb67bfb6 100644 --- a/x/slashing/keeper_test.go +++ b/x/slashing/keeper_test.go @@ -165,7 +165,7 @@ func TestHandleAbsentValidator(t *testing.T) { validator, _ := sk.GetValidatorByPubKey(ctx, val) require.Equal(t, sdk.Bonded, validator.GetStatus()) pool := sk.GetPool(ctx) - require.Equal(t, int64(amtInt), pool.BondedTokens.RoundInt64()) + require.Equal(t, amtInt, pool.BondedTokens.RoundInt64()) // 501st block missed ctx = ctx.WithBlockHeight(height) @@ -195,7 +195,7 @@ func TestHandleAbsentValidator(t *testing.T) { // validator should have been slashed pool = sk.GetPool(ctx) slashAmt := sdk.NewDec(amtInt).Mul(keeper.SlashFractionDowntime(ctx)).RoundInt64() - require.Equal(t, int64(amtInt)-slashAmt, pool.BondedTokens.RoundInt64()) + require.Equal(t, amtInt-slashAmt, pool.BondedTokens.RoundInt64()) // validator start height should have been changed info, found = keeper.getValidatorSigningInfo(ctx, sdk.ConsAddress(val.Address())) @@ -252,7 +252,7 @@ func TestHandleNewValidator(t *testing.T) { info, found := keeper.getValidatorSigningInfo(ctx, sdk.ConsAddress(val.Address())) require.True(t, found) - require.Equal(t, int64(keeper.SignedBlocksWindow(ctx)+1), info.StartHeight) + require.Equal(t, keeper.SignedBlocksWindow(ctx)+1, info.StartHeight) require.Equal(t, int64(2), info.IndexOffset) require.Equal(t, int64(1), info.SignedBlocksCounter) require.Equal(t, time.Unix(0, 0).UTC(), info.JailedUntil) @@ -296,7 +296,7 @@ func TestHandleAlreadyJailed(t *testing.T) { require.Equal(t, sdk.Unbonding, validator.GetStatus()) // validator should have been slashed - require.Equal(t, int64(amtInt-1), validator.GetTokens().RoundInt64()) + require.Equal(t, amtInt-1, validator.GetTokens().RoundInt64()) // another block missed ctx = ctx.WithBlockHeight(height) @@ -304,6 +304,6 @@ func TestHandleAlreadyJailed(t *testing.T) { // validator should not have been slashed twice validator, _ = sk.GetValidatorByPubKey(ctx, val) - require.Equal(t, int64(amtInt-1), validator.GetTokens().RoundInt64()) + require.Equal(t, amtInt-1, validator.GetTokens().RoundInt64()) } diff --git a/x/stake/client/rest/utils.go b/x/stake/client/rest/utils.go index 171ded2b3..d8ad5fec5 100644 --- a/x/stake/client/rest/utils.go +++ b/x/stake/client/rest/utils.go @@ -163,6 +163,7 @@ func getValidators(validatorKVs []sdk.KVPair, cdc *wire.Codec) ([]types.BechVali } // gets all Bech32 validators from a key +// nolint: unparam func getBech32Validators(storeName string, cliCtx context.CLIContext, cdc *wire.Codec) ( validators []types.BechValidator, httpStatusCode int, errMsg string, err error) { diff --git a/x/stake/handler_test.go b/x/stake/handler_test.go index 68c342fd8..a733b7f15 100644 --- a/x/stake/handler_test.go +++ b/x/stake/handler_test.go @@ -430,7 +430,7 @@ func TestIncrementsMsgUnbond(t *testing.T) { initBond, } for _, c := range errorCases { - unbondShares := sdk.NewDec(int64(c)) + unbondShares := sdk.NewDec(c) msgBeginUnbonding := NewMsgBeginUnbonding(delegatorAddr, validatorAddr, unbondShares) got = handleMsgBeginUnbonding(ctx, msgBeginUnbonding, keeper) require.False(t, got.IsOK(), "expected unbond msg to fail") diff --git a/x/stake/keeper/key.go b/x/stake/keeper/key.go index d539c35d0..f3e2967c8 100644 --- a/x/stake/keeper/key.go +++ b/x/stake/keeper/key.go @@ -68,6 +68,7 @@ func GetValidatorsByPowerIndexKey(validator types.Validator, pool types.Pool) [] // get the power ranking of a validator // NOTE the larger values are of higher value +// nolint: unparam func getValidatorPowerRank(validator types.Validator, pool types.Pool) []byte { potentialPower := validator.Tokens diff --git a/x/stake/keeper/slash.go b/x/stake/keeper/slash.go index be26d1a23..e077accb7 100644 --- a/x/stake/keeper/slash.go +++ b/x/stake/keeper/slash.go @@ -203,6 +203,7 @@ func (k Keeper) slashUnbondingDelegation(ctx sdk.Context, unbondingDelegation ty // the unbonding delegation had enough stake to slash // (the amount actually slashed may be less if there's // insufficient stake remaining) +// nolint: unparam func (k Keeper) slashRedelegation(ctx sdk.Context, validator types.Validator, redelegation types.Redelegation, infractionHeight int64, slashFactor sdk.Dec) (slashAmount sdk.Dec) { diff --git a/x/stake/keeper/validator.go b/x/stake/keeper/validator.go index 47a5d5f66..96719d860 100644 --- a/x/stake/keeper/validator.go +++ b/x/stake/keeper/validator.go @@ -372,6 +372,7 @@ func (k Keeper) updateForJailing(ctx sdk.Context, oldFound bool, oldValidator, n return newValidator } +// nolint: unparam func (k Keeper) getPowerIncreasing(ctx sdk.Context, oldFound bool, oldValidator, newValidator types.Validator) bool { if oldFound && oldValidator.BondedTokens().LT(newValidator.BondedTokens()) { return true @@ -380,6 +381,7 @@ func (k Keeper) getPowerIncreasing(ctx sdk.Context, oldFound bool, oldValidator, } // get the bond height and incremented intra-tx counter +// nolint: unparam func (k Keeper) bondIncrement(ctx sdk.Context, oldFound bool, oldValidator, newValidator types.Validator) (height int64, intraTxCounter int16) { diff --git a/x/stake/simulation/invariants.go b/x/stake/simulation/invariants.go index 859bb0591..8f218f45b 100644 --- a/x/stake/simulation/invariants.go +++ b/x/stake/simulation/invariants.go @@ -25,6 +25,7 @@ func AllInvariants(ck bank.Keeper, k stake.Keeper, am auth.AccountMapper) simula } // SupplyInvariants checks that the total supply reflects all held loose tokens, bonded tokens, and unbonding delegations +// nolint: unparam func SupplyInvariants(ck bank.Keeper, k stake.Keeper, am auth.AccountMapper) simulation.Invariant { return func(t *testing.T, app *baseapp.BaseApp, log string) { ctx := app.NewContext(false, abci.Header{}) diff --git a/x/stake/simulation/msgs.go b/x/stake/simulation/msgs.go index af42dba49..264460132 100644 --- a/x/stake/simulation/msgs.go +++ b/x/stake/simulation/msgs.go @@ -258,6 +258,7 @@ func SimulateMsgCompleteRedelegate(k stake.Keeper) simulation.Operation { } // Setup +// nolint: errcheck func Setup(mapp *mock.App, k stake.Keeper) simulation.RandSetup { return func(r *rand.Rand, privKeys []crypto.PrivKey) { ctx := mapp.NewContext(false, abci.Header{}) diff --git a/x/stake/types/delegation_test.go b/x/stake/types/delegation_test.go index 8e0dda7e2..e75212257 100644 --- a/x/stake/types/delegation_test.go +++ b/x/stake/types/delegation_test.go @@ -23,7 +23,7 @@ func TestDelegationEqual(t *testing.T) { ok := d1.Equal(d2) require.True(t, ok) - d2.ValidatorAddr = sdk.ValAddress(addr3) + d2.ValidatorAddr = addr3 d2.Shares = sdk.NewDec(200) ok = d1.Equal(d2) @@ -57,7 +57,7 @@ func TestUnbondingDelegationEqual(t *testing.T) { ok := ud1.Equal(ud2) require.True(t, ok) - ud2.ValidatorAddr = sdk.ValAddress(addr3) + ud2.ValidatorAddr = addr3 ud2.MinTime = time.Unix(20*20*2, 0) ok = ud1.Equal(ud2) From 01da56a020dd091c9af8599e52777120989c468b Mon Sep 17 00:00:00 2001 From: Jeremiah Andrews Date: Mon, 3 Sep 2018 08:00:41 -0700 Subject: [PATCH 38/49] Fix pending changelog --- PENDING.md | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/PENDING.md b/PENDING.md index 00e609039..5e253df30 100644 --- a/PENDING.md +++ b/PENDING.md @@ -1,17 +1,4 @@ -## v0.24.1 PENDING - -BREAKING CHANGES - -FEATURES - -IMPROVEMENTS - -* [store] \#1952 Update IAVL dependency to v0.10.0 - -BUG FIXES - -## v0.24.0 PENDING -^--- PENDING wasn't purged on sdk v0.23.0 release. +## PENDING BREAKING CHANGES @@ -98,6 +85,7 @@ IMPROVEMENTS * [tools] Make get_vendor_deps deletes `.vendor-new` directories, in case scratch files are present. * [cli] \#1632 Add integration tests to ensure `basecoind init && basecoind` start sequences run successfully for both `democoin` and `basecoin` examples. * [store] Speedup IAVL iteration, and consequently everything that requires IAVL iteration. [#2143](https://github.com/cosmos/cosmos-sdk/issues/2143) + * [store] \#1952 Update IAVL dependency to v0.10.0 * [simulation] Make timestamps randomized [#2153](https://github.com/cosmos/cosmos-sdk/pull/2153) * Tendermint From 20f5325f479ca513538dec1ec240379b6f89fbe3 Mon Sep 17 00:00:00 2001 From: Jeremiah Andrews Date: Mon, 3 Sep 2018 09:47:24 -0700 Subject: [PATCH 39/49] Fix more linter errors --- x/mock/simulation/random_simulate_blocks.go | 2 ++ x/slashing/handler_test.go | 8 +++---- x/slashing/keeper_test.go | 24 ++++++++++----------- x/slashing/tick_test.go | 4 ++-- 4 files changed, 20 insertions(+), 18 deletions(-) diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index 11c3dbd6a..00b7a7365 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -209,6 +209,7 @@ func queueOperations(queuedOperations map[int][]Operation, futureOperations []Fu } } +// nolint: errcheck func runQueuedOperations(queueOperations map[int][]Operation, height int, tb testing.TB, r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, privKeys []crypto.PrivKey, log string, event func(string)) (updatedLog string, numOpsRan int) { updatedLog = log @@ -321,6 +322,7 @@ func AssertAllInvariants(t *testing.T, app *baseapp.BaseApp, tests []Invariant, } // updateValidators mimicks Tendermint's update logic +// nolint: unparam func updateValidators(tb testing.TB, r *rand.Rand, current map[string]mockValidator, updates []abci.Validator, event func(string)) map[string]mockValidator { for _, update := range updates { switch { diff --git a/x/slashing/handler_test.go b/x/slashing/handler_test.go index c5afb8738..9cdcd0533 100644 --- a/x/slashing/handler_test.go +++ b/x/slashing/handler_test.go @@ -16,15 +16,15 @@ func TestCannotUnjailUnlessJailed(t *testing.T) { slh := NewHandler(keeper) amtInt := int64(100) addr, val, amt := addrs[0], pks[0], sdk.NewInt(amtInt) - msg := newTestMsgCreateValidator(sdk.ValAddress(addr), val, amt) + msg := newTestMsgCreateValidator(addr, val, amt) got := stake.NewHandler(sk)(ctx, msg) require.True(t, got.IsOK()) stake.EndBlocker(ctx, sk) require.Equal(t, ck.GetCoins(ctx, sdk.AccAddress(addr)), sdk.Coins{{sk.GetParams(ctx).BondDenom, initCoins.Sub(amt)}}) - require.True(t, sdk.NewDecFromInt(amt).Equal(sk.Validator(ctx, sdk.ValAddress(addr)).GetPower())) + require.True(t, sdk.NewDecFromInt(amt).Equal(sk.Validator(ctx, addr).GetPower())) // assert non-jailed validator can't be unjailed - got = slh(ctx, NewMsgUnjail(sdk.ValAddress(addr))) + got = slh(ctx, NewMsgUnjail(addr)) require.False(t, got.IsOK(), "allowed unjail of non-jailed validator") require.Equal(t, sdk.ToABCICode(DefaultCodespace, CodeValidatorNotJailed), got.Code) } @@ -39,7 +39,7 @@ func TestJailedValidatorDelegations(t *testing.T) { // create a validator amount := int64(10) valPubKey, bondAmount := pks[0], sdk.NewInt(amount) - valAddr, consAddr := sdk.ValAddress(addrs[1]), sdk.ConsAddress(addrs[0]) + valAddr, consAddr := addrs[1], sdk.ConsAddress(addrs[0]) msgCreateVal := newTestMsgCreateValidator(valAddr, valPubKey, bondAmount) got := stake.NewHandler(stakeKeeper)(ctx, msgCreateVal) diff --git a/x/slashing/keeper_test.go b/x/slashing/keeper_test.go index 0eb67bfb6..ff50a594e 100644 --- a/x/slashing/keeper_test.go +++ b/x/slashing/keeper_test.go @@ -27,12 +27,12 @@ func TestHandleDoubleSign(t *testing.T) { sk = sk.WithValidatorHooks(keeper.ValidatorHooks()) amtInt := int64(100) addr, val, amt := addrs[0], pks[0], sdk.NewInt(amtInt) - got := stake.NewHandler(sk)(ctx, newTestMsgCreateValidator(sdk.ValAddress(addr), val, amt)) + got := stake.NewHandler(sk)(ctx, newTestMsgCreateValidator(addr, val, amt)) require.True(t, got.IsOK()) validatorUpdates := stake.EndBlocker(ctx, sk) keeper.AddValidators(ctx, validatorUpdates) require.Equal(t, ck.GetCoins(ctx, sdk.AccAddress(addr)), sdk.Coins{{sk.GetParams(ctx).BondDenom, initCoins.Sub(amt)}}) - require.True(t, sdk.NewDecFromInt(amt).Equal(sk.Validator(ctx, sdk.ValAddress(addr)).GetPower())) + require.True(t, sdk.NewDecFromInt(amt).Equal(sk.Validator(ctx, addr).GetPower())) // handle a signature to set signing info keeper.handleValidatorSignature(ctx, val.Address(), amtInt, true) @@ -41,13 +41,13 @@ func TestHandleDoubleSign(t *testing.T) { keeper.handleDoubleSign(ctx, val.Address(), 0, time.Unix(0, 0), amtInt) // should be jailed - require.True(t, sk.Validator(ctx, sdk.ValAddress(addr)).GetJailed()) + require.True(t, sk.Validator(ctx, addr).GetJailed()) // unjail to measure power sk.Unjail(ctx, val) // power should be reduced require.Equal( t, sdk.NewDecFromInt(amt).Mul(sdk.NewDec(19).Quo(sdk.NewDec(20))), - sk.Validator(ctx, sdk.ValAddress(addr)).GetPower(), + sk.Validator(ctx, addr).GetPower(), ) ctx = ctx.WithBlockHeader(abci.Header{Time: time.Unix(1, 0).Add(keeper.MaxEvidenceAge(ctx))}) @@ -55,7 +55,7 @@ func TestHandleDoubleSign(t *testing.T) { keeper.handleDoubleSign(ctx, val.Address(), 0, time.Unix(0, 0), amtInt) require.Equal( t, sdk.NewDecFromInt(amt).Mul(sdk.NewDec(19).Quo(sdk.NewDec(20))), - sk.Validator(ctx, sdk.ValAddress(addr)).GetPower(), + sk.Validator(ctx, addr).GetPower(), ) } @@ -125,12 +125,12 @@ func TestHandleAbsentValidator(t *testing.T) { addr, val, amt := addrs[0], pks[0], sdk.NewInt(amtInt) sh := stake.NewHandler(sk) slh := NewHandler(keeper) - got := sh(ctx, newTestMsgCreateValidator(sdk.ValAddress(addr), val, amt)) + got := sh(ctx, newTestMsgCreateValidator(addr, val, amt)) require.True(t, got.IsOK()) validatorUpdates := stake.EndBlocker(ctx, sk) keeper.AddValidators(ctx, validatorUpdates) require.Equal(t, ck.GetCoins(ctx, sdk.AccAddress(addr)), sdk.Coins{{sk.GetParams(ctx).BondDenom, initCoins.Sub(amt)}}) - require.True(t, sdk.NewDecFromInt(amt).Equal(sk.Validator(ctx, sdk.ValAddress(addr)).GetPower())) + require.True(t, sdk.NewDecFromInt(amt).Equal(sk.Validator(ctx, addr).GetPower())) info, found := keeper.getValidatorSigningInfo(ctx, sdk.ConsAddress(val.Address())) require.False(t, found) require.Equal(t, int64(0), info.StartHeight) @@ -180,12 +180,12 @@ func TestHandleAbsentValidator(t *testing.T) { require.Equal(t, sdk.Unbonding, validator.GetStatus()) // unrevocation should fail prior to jail expiration - got = slh(ctx, NewMsgUnjail(sdk.ValAddress(addr))) + got = slh(ctx, NewMsgUnjail(addr)) require.False(t, got.IsOK()) // unrevocation should succeed after jail expiration ctx = ctx.WithBlockHeader(abci.Header{Time: time.Unix(1, 0).Add(keeper.DowntimeUnbondDuration(ctx))}) - got = slh(ctx, NewMsgUnjail(sdk.ValAddress(addr))) + got = slh(ctx, NewMsgUnjail(addr)) require.True(t, got.IsOK()) // validator should be rebonded now @@ -235,12 +235,12 @@ func TestHandleNewValidator(t *testing.T) { ctx, ck, sk, _, keeper := createTestInput(t) addr, val, amt := addrs[0], pks[0], int64(100) sh := stake.NewHandler(sk) - got := sh(ctx, newTestMsgCreateValidator(sdk.ValAddress(addr), val, sdk.NewInt(amt))) + got := sh(ctx, newTestMsgCreateValidator(addr, val, sdk.NewInt(amt))) require.True(t, got.IsOK()) validatorUpdates := stake.EndBlocker(ctx, sk) keeper.AddValidators(ctx, validatorUpdates) require.Equal(t, ck.GetCoins(ctx, sdk.AccAddress(addr)), sdk.Coins{{sk.GetParams(ctx).BondDenom, initCoins.SubRaw(amt)}}) - require.Equal(t, sdk.NewDec(amt), sk.Validator(ctx, sdk.ValAddress(addr)).GetPower()) + require.Equal(t, sdk.NewDec(amt), sk.Validator(ctx, addr).GetPower()) // 1000 first blocks not a validator ctx = ctx.WithBlockHeight(keeper.SignedBlocksWindow(ctx) + 1) @@ -273,7 +273,7 @@ func TestHandleAlreadyJailed(t *testing.T) { amtInt := int64(100) addr, val, amt := addrs[0], pks[0], sdk.NewInt(amtInt) sh := stake.NewHandler(sk) - got := sh(ctx, newTestMsgCreateValidator(sdk.ValAddress(addr), val, amt)) + got := sh(ctx, newTestMsgCreateValidator(addr, val, amt)) require.True(t, got.IsOK()) validatorUpdates := stake.EndBlocker(ctx, sk) keeper.AddValidators(ctx, validatorUpdates) diff --git a/x/slashing/tick_test.go b/x/slashing/tick_test.go index 25167578d..81003a968 100644 --- a/x/slashing/tick_test.go +++ b/x/slashing/tick_test.go @@ -17,12 +17,12 @@ func TestBeginBlocker(t *testing.T) { addr, pk, amt := addrs[2], pks[2], sdk.NewInt(100) // bond the validator - got := stake.NewHandler(sk)(ctx, newTestMsgCreateValidator(sdk.ValAddress(addr), pk, amt)) + got := stake.NewHandler(sk)(ctx, newTestMsgCreateValidator(addr, pk, amt)) require.True(t, got.IsOK()) validatorUpdates := stake.EndBlocker(ctx, sk) keeper.AddValidators(ctx, validatorUpdates) require.Equal(t, ck.GetCoins(ctx, sdk.AccAddress(addr)), sdk.Coins{{sk.GetParams(ctx).BondDenom, initCoins.Sub(amt)}}) - require.True(t, sdk.NewDecFromInt(amt).Equal(sk.Validator(ctx, sdk.ValAddress(addr)).GetPower())) + require.True(t, sdk.NewDecFromInt(amt).Equal(sk.Validator(ctx, addr).GetPower())) val := abci.Validator{ Address: pk.Address(), From 5c600f2e2b66884723892ba53a0d6f780e2364fe Mon Sep 17 00:00:00 2001 From: rigelrozanski Date: Mon, 3 Sep 2018 15:49:36 -0400 Subject: [PATCH 40/49] jae comments --- docs/spec/distribution/transactions.md | 45 ++++++++++++++------------ 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/docs/spec/distribution/transactions.md b/docs/spec/distribution/transactions.md index 2ed12d2c3..0b89ae44e 100644 --- a/docs/spec/distribution/transactions.md +++ b/docs/spec/distribution/transactions.md @@ -1,24 +1,24 @@ # Transactions -## TxWithdrawDelegationRewards +## TxWithdrawDelegationRewardsAll When a delegator wishes to withdraw their rewards it must send -`TxWithdrawDelegationRewards`. Note that parts of this transaction logic are also +`TxWithdrawDelegationRewardsAll`. Note that parts of this transaction logic are also triggered each with any change in individual delegations, such as an unbond, redelegation, or delegation of additional tokens to a specific validator. ```golang -type TxWithdrawDelegationRewards struct { +type TxWithdrawDelegationRewardsAll struct { delegatorAddr sdk.AccAddress withdrawAddr sdk.AccAddress // address to make the withdrawal to } -func WithdrawDelegationRewards(delegatorAddr, withdrawAddr sdk.AccAddress) +func WithdrawDelegationRewardsAll(delegatorAddr, withdrawAddr sdk.AccAddress) height = GetHeight() - withdraw = GetDelegatorAllWithdraws(delegatorAddr, height) + withdraw = GetDelegatorRewardsAll(delegatorAddr, height) AddCoins(withdrawAddr, withdraw.TruncateDecimal()) -func GetDelegatorAllWithdraws(delegatorAddr sdk.AccAddress, height int64) DecCoins +func GetDelegatorRewardsAll(delegatorAddr sdk.AccAddress, height int64) DecCoins // get all distribution scenarios delegations = GetDelegations(delegatorAddr) @@ -72,22 +72,22 @@ func WithdrawDelegationReward(delegatorAddr, validatorAddr, withdrawAddr sdk.Acc ``` -## TxWithdrawValidatorRewards +## TxWithdrawValidatorRewardsAll When a validator wishes to withdraw their rewards it must send -`TxWithdrawValidatorRewards`. Note that parts of this transaction logic are also +`TxWithdrawValidatorRewardsAll`. Note that parts of this transaction logic are also triggered each with any change in individual delegations, such as an unbond, redelegation, or delegation of additional tokens to a specific validator. This transaction withdraws the validators commission fee, as well as any rewards earning on their self-delegation. ``` -type TxWithdrawValidatorRewards struct { +type TxWithdrawValidatorRewardsAll struct { operatorAddr sdk.AccAddress // validator address to withdraw from withdrawAddr sdk.AccAddress // address to make the withdrawal to } -func WithdrawValidatorRewards(operatorAddr, withdrawAddr sdk.AccAddress) +func WithdrawValidatorRewardsAll(operatorAddr, withdrawAddr sdk.AccAddress) height = GetHeight() global = GetGlobal() @@ -96,7 +96,7 @@ func WithdrawValidatorRewards(operatorAddr, withdrawAddr sdk.AccAddress) validator = GetValidator(delegation.ValidatorAddr) // withdraw self-delegation - withdraw = GetDelegatorAllWithdraws(validator.OperatorAddr, height) + withdraw = GetDelegatorRewardsAll(validator.OperatorAddr, height) // withdrawal validator commission rewards global, commission = valInfo.WithdrawCommission(global, valInfo, height, pool.BondedTokens, @@ -117,10 +117,11 @@ block. The accum is always additive to the existing accum. This term is to be updated each time rewards are withdrawn from the system. ``` -func (g Global) UpdateTotalValAccum(height int64, totalBondedTokens Dec) +func (g Global) UpdateTotalValAccum(height int64, totalBondedTokens Dec) Global blocks = height - g.TotalValAccumUpdateHeight g.TotalValAccum += totalDelShares * blocks g.TotalValAccumUpdateHeight = height + return g ``` ### Update validator's accums @@ -132,10 +133,11 @@ the existing accum. This term is to be updated each time a withdrawal is made from a validator. ``` -func (vi ValidatorDistInfo) UpdateTotalDelAccum(height int64, totalDelShares Dec) +func (vi ValidatorDistInfo) UpdateTotalDelAccum(height int64, totalDelShares Dec) ValidatorDistInfo blocks = height - vi.TotalDelAccumUpdateHeight vi.TotalDelAccum += totalDelShares * blocks vi.TotalDelAccumUpdateHeight = height + return vi ``` ### Global pool to validator pool @@ -146,9 +148,10 @@ from the passive global pool to their own pool. It is at this point that the commission is withdrawn ``` -func (vi ValidatorDistInfo) TakeAccum(g Global, height int64, totalBonded, vdTokens, commissionRate Dec) g Global +func (vi ValidatorDistInfo) TakeAccum(g Global, height int64, totalBonded, vdTokens, commissionRate Dec) ( + vi ValidatorDistInfo, g Global) + g.UpdateTotalValAccum(height, totalBondedShares) - g.UpdateValAccum(height, totalBondedShares) // update the validators pool blocks = height - vi.GlobalWithdrawalHeight @@ -162,7 +165,7 @@ func (vi ValidatorDistInfo) TakeAccum(g Global, height int64, totalBonded, vdTok vi.PoolCommissionFree += withdrawalTokens - commission g.Pool -= withdrawalTokens - return g + return vi, g ``` @@ -173,7 +176,8 @@ pool have already had the validator's commission taken away. ``` func (di DelegatorDistInfo) WithdrawRewards(g Global, vi ValidatorDistInfo, - height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) (g Global, withdrawn DecCoins) + height int64, totalBonded, vdTokens, totalDelShares, commissionRate Dec) ( + di DelegatorDistInfo, g Global, withdrawn DecCoins) vi.UpdateTotalDelAccum(height, totalDelShares) g = vi.TakeAccum(g, height, totalBonded, vdTokens, commissionRate) @@ -187,7 +191,7 @@ func (di DelegatorDistInfo) WithdrawRewards(g Global, vi ValidatorDistInfo, vi.Pool -= withdrawalTokens vi.TotalDelAccum -= accum - return g, withdrawalTokens + return di, g, withdrawalTokens ``` @@ -197,12 +201,13 @@ Commission is calculated each time rewards enter into the validator. ``` func (vi ValidatorDistInfo) WithdrawCommission(g Global, height int64, - totalBonded, vdTokens, commissionRate Dec) (g Global, withdrawn DecCoins) + totalBonded, vdTokens, commissionRate Dec) ( + vi ValidatorDistInfo, g Global, withdrawn DecCoins) g = vi.TakeAccum(g, height, totalBonded, vdTokens, commissionRate) withdrawalTokens := vi.PoolCommission vi.PoolCommission = 0 - return g, withdrawalTokens + return vi, g, withdrawalTokens ``` From 86395809cbcbe99ba7da9a6f4b3103eee8c0cd05 Mon Sep 17 00:00:00 2001 From: Alessio Treglia Date: Sun, 2 Sep 2018 20:20:14 +0200 Subject: [PATCH 41/49] Implement generate-only option for commands that create txs The new CLI flag builds an unsigned transaction and writes it to STDOUT. Likewise, REST clients can now append generate_only=true to a request's query arguments list and expect a JSON response carrying the unsigned transaction. Closes: #966 --- PENDING.md | 1 + client/context/context.go | 5 +- client/flags.go | 2 + client/lcd/lcd_test.go | 15 ++++++ client/utils/rest.go | 33 +++++++++++-- client/utils/utils.go | 37 +++++++++++++- cmd/gaia/cli_test/cli_test.go | 91 ++++++++++++++++++++++++++++++++++- crypto/keys/keybase.go | 4 +- docs/sdk/clients.md | 11 +++++ x/auth/stdtx.go | 12 ++--- x/bank/client/cli/sendtx.go | 3 ++ x/bank/client/rest/sendtx.go | 5 ++ x/gov/client/cli/tx.go | 11 ++++- x/gov/client/rest/util.go | 6 +++ x/gov/msgs.go | 10 ++-- x/ibc/client/cli/ibctx.go | 3 ++ x/ibc/client/rest/transfer.go | 5 ++ x/ibc/types.go | 10 ++-- x/slashing/client/cli/tx.go | 4 +- x/slashing/client/rest/tx.go | 5 ++ x/stake/client/cli/tx.go | 22 ++++++++- x/stake/client/rest/tx.go | 5 ++ 22 files changed, 270 insertions(+), 30 deletions(-) diff --git a/PENDING.md b/PENDING.md index 17616ced1..7289cd23f 100644 --- a/PENDING.md +++ b/PENDING.md @@ -53,6 +53,7 @@ FEATURES * [cli] \#2047 Setting the --gas flag value to 0 triggers a simulation of the tx before the actual execution. The gas estimate obtained via the simulation will be used as gas limit in the actual execution. * [cli] \#2047 The --gas-adjustment flag can be used to adjust the estimate obtained via the simulation triggered by --gas=0. * [cli] \#2110 Add --dry-run flag to perform a simulation of a transaction without broadcasting it. The --gas flag is ignored as gas would be automatically estimated. + * [cli] \#966 Add --generate-only flag to build an unsigned transaction and write it to STDOUT. * Gaia * [cli] #2170 added ability to show the node's address via `gaiad tendermint show-address` diff --git a/client/context/context.go b/client/context/context.go index 8e56cfa1a..a4d4afd3b 100644 --- a/client/context/context.go +++ b/client/context/context.go @@ -3,10 +3,11 @@ package context import ( "bytes" "fmt" + "io" + "github.com/cosmos/cosmos-sdk/client" "github.com/cosmos/cosmos-sdk/wire" "github.com/cosmos/cosmos-sdk/x/auth" - "io" "github.com/spf13/viper" @@ -38,6 +39,7 @@ type CLIContext struct { PrintResponse bool Certifier tmlite.Certifier DryRun bool + GenerateOnly bool } // NewCLIContext returns a new initialized CLIContext with parameters from the @@ -65,6 +67,7 @@ func NewCLIContext() CLIContext { PrintResponse: viper.GetBool(client.FlagPrintResponse), Certifier: createCertifier(), DryRun: viper.GetBool(client.FlagDryRun), + GenerateOnly: viper.GetBool(client.FlagGenerateOnly), } } diff --git a/client/flags.go b/client/flags.go index a1d3c6e17..97fce42a5 100644 --- a/client/flags.go +++ b/client/flags.go @@ -27,6 +27,7 @@ const ( FlagJson = "json" FlagPrintResponse = "print-response" FlagDryRun = "dry-run" + FlagGenerateOnly = "generate-only" ) // LineBreak can be included in a command list to provide a blank line @@ -64,6 +65,7 @@ func PostCommands(cmds ...*cobra.Command) []*cobra.Command { c.Flags().Bool(FlagPrintResponse, true, "return tx response (only works with async = false)") c.Flags().Bool(FlagTrustNode, true, "Don't verify proofs for query responses") c.Flags().Bool(FlagDryRun, false, "ignore the --gas flag and perform a simulation of a transaction, but don't broadcast it") + c.Flags().Bool(FlagGenerateOnly, false, "build an unsigned transaction and write it to STDOUT") } return cmds } diff --git a/client/lcd/lcd_test.go b/client/lcd/lcd_test.go index 707cc21f7..7eb592a70 100644 --- a/client/lcd/lcd_test.go +++ b/client/lcd/lcd_test.go @@ -313,6 +313,21 @@ func TestIBCTransfer(t *testing.T) { // TODO: query ibc egress packet state } +func TestCoinSendGenerateOnly(t *testing.T) { + name, password := "test", "1234567890" + addr, seed := CreateAddr(t, "test", password, GetKeyBase(t)) + cleanup, _, port := InitializeTestLCD(t, 1, []sdk.AccAddress{addr}) + defer cleanup() + // create TX + res, body, _ := doSendWithGas(t, port, seed, name, password, addr, 0, 0, "?generate_only=true") + require.Equal(t, http.StatusOK, res.StatusCode, body) + var msg auth.StdTx + require.Nil(t, cdc.UnmarshalJSON([]byte(body), &msg)) + require.Equal(t, len(msg.Msgs), 1) + require.Equal(t, msg.Msgs[0].Type(), "bank") + require.Equal(t, msg.Msgs[0].GetSigners(), []sdk.AccAddress{addr}) +} + func TestTxs(t *testing.T) { name, password := "test", "1234567890" addr, seed := CreateAddr(t, "test", password, GetKeyBase(t)) diff --git a/client/utils/rest.go b/client/utils/rest.go index 25e1c23d8..b6cee6c1f 100644 --- a/client/utils/rest.go +++ b/client/utils/rest.go @@ -3,11 +3,17 @@ package utils import ( "fmt" "net/http" + "net/url" "strconv" + + sdk "github.com/cosmos/cosmos-sdk/types" + auth "github.com/cosmos/cosmos-sdk/x/auth" + authctx "github.com/cosmos/cosmos-sdk/x/auth/client/context" ) const ( - queryArgDryRun = "simulate" + queryArgDryRun = "simulate" + queryArgGenerateOnly = "generate_only" ) // WriteErrorResponse prepares and writes a HTTP error @@ -26,9 +32,10 @@ func WriteSimulationResponse(w http.ResponseWriter, gas int64) { // HasDryRunArg returns true if the request's URL query contains // the dry run argument and its value is set to "true". -func HasDryRunArg(r *http.Request) bool { - return r.URL.Query().Get(queryArgDryRun) == "true" -} +func HasDryRunArg(r *http.Request) bool { return urlQueryHasArg(r.URL, queryArgDryRun) } + +// HasGenerateOnlyArg returns whether a URL's query "generate-only" parameter is set to "true". +func HasGenerateOnlyArg(r *http.Request) bool { return urlQueryHasArg(r.URL, queryArgGenerateOnly) } // ParseFloat64OrReturnBadRequest converts s to a float64 value. It returns a default // value if the string is empty. Write @@ -43,3 +50,21 @@ func ParseFloat64OrReturnBadRequest(w http.ResponseWriter, s string, defaultIfEm } return n, true } + +// WriteGenerateStdTxResponse writes response for the generate_only mode. +func WriteGenerateStdTxResponse(w http.ResponseWriter, txCtx authctx.TxContext, msgs []sdk.Msg) { + stdMsg, err := txCtx.Build(msgs) + if err != nil { + WriteErrorResponse(w, http.StatusBadRequest, err.Error()) + return + } + output, err := txCtx.Codec.MarshalJSON(auth.NewStdTx(stdMsg.Msgs, stdMsg.Fee, nil, stdMsg.Memo)) + if err != nil { + WriteErrorResponse(w, http.StatusInternalServerError, err.Error()) + return + } + w.Write(output) + return +} + +func urlQueryHasArg(url *url.URL, arg string) bool { return url.Query().Get(arg) == "true" } diff --git a/client/utils/utils.go b/client/utils/utils.go index 20b1c6bc5..fa5bcf817 100644 --- a/client/utils/utils.go +++ b/client/utils/utils.go @@ -7,6 +7,7 @@ import ( "github.com/cosmos/cosmos-sdk/client/context" "github.com/cosmos/cosmos-sdk/client/keys" sdk "github.com/cosmos/cosmos-sdk/types" + auth "github.com/cosmos/cosmos-sdk/x/auth" authctx "github.com/cosmos/cosmos-sdk/x/auth/client/context" amino "github.com/tendermint/go-amino" "github.com/tendermint/tendermint/libs/common" @@ -28,7 +29,7 @@ func SendTx(txCtx authctx.TxContext, cliCtx context.CLIContext, msgs []sdk.Msg) if err != nil { return err } - fmt.Fprintf(os.Stdout, "estimated gas = %v\n", txCtx.Gas) + fmt.Fprintf(os.Stderr, "estimated gas = %v\n", txCtx.Gas) } if cliCtx.DryRun { return nil @@ -85,6 +86,19 @@ func CalculateGas(queryFunc func(string, common.HexBytes) ([]byte, error), cdc * return } +// PrintUnsignedStdTx builds an unsigned StdTx and prints it to os.Stdout. +func PrintUnsignedStdTx(txCtx authctx.TxContext, cliCtx context.CLIContext, msgs []sdk.Msg) (err error) { + stdTx, err := buildUnsignedStdTx(txCtx, cliCtx, msgs) + if err != nil { + return + } + json, err := txCtx.Codec.MarshalJSON(stdTx) + if err == nil { + fmt.Printf("%s\n", json) + } + return +} + func adjustGasEstimate(estimate int64, adjustment float64) int64 { return int64(adjustment * float64(estimate)) } @@ -128,3 +142,24 @@ func prepareTxContext(txCtx authctx.TxContext, cliCtx context.CLIContext) (authc } return txCtx, nil } + +// buildUnsignedStdTx builds a StdTx as per the parameters passed in the +// contexts. Gas is automatically estimated if gas wanted is set to 0. +func buildUnsignedStdTx(txCtx authctx.TxContext, cliCtx context.CLIContext, msgs []sdk.Msg) (stdTx auth.StdTx, err error) { + txCtx, err = prepareTxContext(txCtx, cliCtx) + if err != nil { + return + } + if txCtx.Gas == 0 { + txCtx, err = EnrichCtxWithGas(txCtx, cliCtx, cliCtx.FromAddressName, msgs) + if err != nil { + return + } + fmt.Fprintf(os.Stderr, "estimated gas = %v\n", txCtx.Gas) + } + stdSignMsg, err := txCtx.Build(msgs) + if err != nil { + return + } + return auth.NewStdTx(stdSignMsg.Msgs, stdSignMsg.Fee, nil, stdSignMsg.Memo), nil +} diff --git a/cmd/gaia/cli_test/cli_test.go b/cmd/gaia/cli_test/cli_test.go index 696c0fc6a..473a3bb3b 100644 --- a/cmd/gaia/cli_test/cli_test.go +++ b/cmd/gaia/cli_test/cli_test.go @@ -13,6 +13,7 @@ import ( "github.com/tendermint/tendermint/crypto" cmn "github.com/tendermint/tendermint/libs/common" + "github.com/cosmos/cosmos-sdk/client" "github.com/cosmos/cosmos-sdk/client/keys" "github.com/cosmos/cosmos-sdk/cmd/gaia/app" "github.com/cosmos/cosmos-sdk/server" @@ -155,8 +156,17 @@ func TestGaiaCLICreateValidator(t *testing.T) { initialPool.BondedTokens = initialPool.BondedTokens.Add(sdk.NewDec(1)) + // Test --generate-only + success, stdout, stderr := executeWriteRetStdStreams(t, cvStr+" --generate-only", app.DefaultKeyPass) + require.True(t, success) + require.True(t, success) + require.Empty(t, stderr) + msg := unmarshalStdTx(t, stdout) + require.NotZero(t, msg.Fee.Gas) + require.Equal(t, len(msg.Msgs), 1) + // Test --dry-run - success := executeWrite(t, cvStr+" --dry-run", app.DefaultKeyPass) + success = executeWrite(t, cvStr+" --dry-run", app.DefaultKeyPass) require.True(t, success) executeWrite(t, cvStr, app.DefaultKeyPass) @@ -222,8 +232,17 @@ func TestGaiaCLISubmitProposal(t *testing.T) { spStr += fmt.Sprintf(" --title=%s", "Test") spStr += fmt.Sprintf(" --description=%s", "test") + // Test generate only + success, stdout, stderr := executeWriteRetStdStreams(t, spStr+" --generate-only", app.DefaultKeyPass) + require.True(t, success) + require.True(t, success) + require.Empty(t, stderr) + msg := unmarshalStdTx(t, stdout) + require.NotZero(t, msg.Fee.Gas) + require.Equal(t, len(msg.Msgs), 1) + // Test --dry-run - success := executeWrite(t, spStr+" --dry-run", app.DefaultKeyPass) + success = executeWrite(t, spStr+" --dry-run", app.DefaultKeyPass) require.True(t, success) executeWrite(t, spStr, app.DefaultKeyPass) @@ -244,6 +263,15 @@ func TestGaiaCLISubmitProposal(t *testing.T) { depositStr += fmt.Sprintf(" --deposit=%s", "10steak") depositStr += fmt.Sprintf(" --proposal-id=%s", "1") + // Test generate only + success, stdout, stderr = executeWriteRetStdStreams(t, depositStr+" --generate-only", app.DefaultKeyPass) + require.True(t, success) + require.True(t, success) + require.Empty(t, stderr) + msg = unmarshalStdTx(t, stdout) + require.NotZero(t, msg.Fee.Gas) + require.Equal(t, len(msg.Msgs), 1) + executeWrite(t, depositStr, app.DefaultKeyPass) tests.WaitForNextNBlocksTM(2, port) @@ -258,6 +286,15 @@ func TestGaiaCLISubmitProposal(t *testing.T) { voteStr += fmt.Sprintf(" --proposal-id=%s", "1") voteStr += fmt.Sprintf(" --option=%s", "Yes") + // Test generate only + success, stdout, stderr = executeWriteRetStdStreams(t, voteStr+" --generate-only", app.DefaultKeyPass) + require.True(t, success) + require.True(t, success) + require.Empty(t, stderr) + msg = unmarshalStdTx(t, stdout) + require.NotZero(t, msg.Fee.Gas) + require.Equal(t, len(msg.Msgs), 1) + executeWrite(t, voteStr, app.DefaultKeyPass) tests.WaitForNextNBlocksTM(2, port) @@ -291,6 +328,50 @@ func TestGaiaCLISubmitProposal(t *testing.T) { require.Equal(t, " 2 - Apples", proposalsQuery) } +func TestGaiaCLISendGenerateOnly(t *testing.T) { + chainID, servAddr, port := initializeFixtures(t) + flags := fmt.Sprintf("--home=%s --node=%v --chain-id=%v", gaiacliHome, servAddr, chainID) + + // start gaiad server + proc := tests.GoExecuteTWithStdout(t, fmt.Sprintf("gaiad start --home=%s --rpc.laddr=%v", gaiadHome, servAddr)) + + defer proc.Stop(false) + tests.WaitForTMStart(port) + tests.WaitForNextNBlocksTM(2, port) + + barAddr, _ := executeGetAddrPK(t, fmt.Sprintf("gaiacli keys show bar --output=json --home=%s", gaiacliHome)) + + // Test generate sendTx with default gas + success, stdout, stderr := executeWriteRetStdStreams(t, fmt.Sprintf( + "gaiacli send %v --amount=10steak --to=%s --from=foo --generate-only", + flags, barAddr), []string{}...) + require.True(t, success) + require.Empty(t, stderr) + msg := unmarshalStdTx(t, stdout) + require.Equal(t, msg.Fee.Gas, int64(client.DefaultGasLimit)) + require.Equal(t, len(msg.Msgs), 1) + + // Test generate sendTx, estimate gas + success, stdout, stderr = executeWriteRetStdStreams(t, fmt.Sprintf( + "gaiacli send %v --amount=10steak --to=%s --from=foo --gas=0 --generate-only", + flags, barAddr), []string{}...) + require.True(t, success) + require.NotEmpty(t, stderr) + msg = unmarshalStdTx(t, stdout) + require.NotZero(t, msg.Fee.Gas) + require.Equal(t, len(msg.Msgs), 1) + + // Test generate sendTx with --gas=$amount + success, stdout, stderr = executeWriteRetStdStreams(t, fmt.Sprintf( + "gaiacli send %v --amount=10steak --to=%s --from=foo --gas=100 --generate-only", + flags, barAddr), []string{}...) + require.True(t, success) + require.Empty(t, stderr) + msg = unmarshalStdTx(t, stdout) + require.Equal(t, msg.Fee.Gas, int64(100)) + require.Equal(t, len(msg.Msgs), 1) +} + //___________________________________________________________________________________ // helper methods @@ -315,6 +396,12 @@ func initializeFixtures(t *testing.T) (chainID, servAddr, port string) { return } +func unmarshalStdTx(t *testing.T, s string) (stdTx auth.StdTx) { + cdc := app.MakeCodec() + require.Nil(t, cdc.UnmarshalJSON([]byte(s), &stdTx)) + return +} + //___________________________________________________________________________________ // executors diff --git a/crypto/keys/keybase.go b/crypto/keys/keybase.go index ae036362e..ec5b7b0fd 100644 --- a/crypto/keys/keybase.go +++ b/crypto/keys/keybase.go @@ -224,9 +224,9 @@ func (kb dbKeybase) Sign(name, passphrase string, msg []byte) (sig []byte, pub t } case offlineInfo: linfo := info.(offlineInfo) - fmt.Printf("Bytes to sign:\n%s", msg) + fmt.Fprintf(os.Stderr, "Bytes to sign:\n%s", msg) buf := bufio.NewReader(os.Stdin) - fmt.Printf("\nEnter Amino-encoded signature:\n") + fmt.Fprintf(os.Stderr, "\nEnter Amino-encoded signature:\n") // Will block until user inputs the signature signed, err := buf.ReadString('\n') if err != nil { diff --git a/docs/sdk/clients.md b/docs/sdk/clients.md index 62d79bbac..fdfbca7bd 100644 --- a/docs/sdk/clients.md +++ b/docs/sdk/clients.md @@ -139,6 +139,17 @@ gaiacli send \ --dry-run ``` +Furthermore, you can build a transaction and print its JSON format to STDOUT by appending `--generate-only` to the list of the command line arguments: + +```bash +gaiacli send \ + --amount=10faucetToken \ + --chain-id= \ + --name= \ + --to= \ + --generate-only +``` + ### Staking #### Set up a Validator diff --git a/x/auth/stdtx.go b/x/auth/stdtx.go index c6e280157..69627b31a 100644 --- a/x/auth/stdtx.go +++ b/x/auth/stdtx.go @@ -143,12 +143,12 @@ func StdSignBytes(chainID string, accnum int64, sequence int64, fee StdFee, msgs // a Msg with the other requirements for a StdSignDoc before // it is signed. For use in the CLI. type StdSignMsg struct { - ChainID string - AccountNumber int64 - Sequence int64 - Fee StdFee - Msgs []sdk.Msg - Memo string + ChainID string `json:"chain_id"` + AccountNumber int64 `json:"account_number"` + Sequence int64 `json:"sequence"` + Fee StdFee `json:"fee"` + Msgs []sdk.Msg `json:"msgs"` + Memo string `json:"memo"` } // get message bytes diff --git a/x/bank/client/cli/sendtx.go b/x/bank/client/cli/sendtx.go index 92ac37c1e..08210569e 100644 --- a/x/bank/client/cli/sendtx.go +++ b/x/bank/client/cli/sendtx.go @@ -68,6 +68,9 @@ func SendTxCmd(cdc *wire.Codec) *cobra.Command { // build and sign the transaction, then broadcast to Tendermint msg := client.BuildMsg(from, to, coins) + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, diff --git a/x/bank/client/rest/sendtx.go b/x/bank/client/rest/sendtx.go index 23506e0fe..1c09943d0 100644 --- a/x/bank/client/rest/sendtx.go +++ b/x/bank/client/rest/sendtx.go @@ -107,6 +107,11 @@ func SendRequestHandlerFn(cdc *wire.Codec, kb keys.Keybase, cliCtx context.CLICo txCtx = newCtx } + if utils.HasGenerateOnlyArg(r) { + utils.WriteGenerateStdTxResponse(w, txCtx, []sdk.Msg{msg}) + return + } + txBytes, err := txCtx.BuildAndSign(m.LocalAccountName, m.Password, []sdk.Msg{msg}) if err != nil { utils.WriteErrorResponse(w, http.StatusUnauthorized, err.Error()) diff --git a/x/gov/client/cli/tx.go b/x/gov/client/cli/tx.go index 65c922c6c..b0fa4d6be 100644 --- a/x/gov/client/cli/tx.go +++ b/x/gov/client/cli/tx.go @@ -99,6 +99,9 @@ $ gaiacli gov submit-proposal --title="Test Proposal" --description="My awesome } msg := gov.NewMsgSubmitProposal(proposal.Title, proposal.Description, proposalType, fromAddr, amount) + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } err = msg.ValidateBasic() if err != nil { @@ -177,7 +180,9 @@ func GetCmdDeposit(cdc *wire.Codec) *cobra.Command { } msg := gov.NewMsgDeposit(depositerAddr, proposalID, amount) - + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } err = msg.ValidateBasic() if err != nil { return err @@ -221,7 +226,9 @@ func GetCmdVote(cdc *wire.Codec) *cobra.Command { } msg := gov.NewMsgVote(voterAddr, proposalID, byteVoteOption) - + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } err = msg.ValidateBasic() if err != nil { return err diff --git a/x/gov/client/rest/util.go b/x/gov/client/rest/util.go index a5fae3c3f..7ba23befa 100644 --- a/x/gov/client/rest/util.go +++ b/x/gov/client/rest/util.go @@ -96,6 +96,12 @@ func signAndBuild(w http.ResponseWriter, r *http.Request, cliCtx context.CLICont } txCtx = newCtx } + + if utils.HasGenerateOnlyArg(r) { + utils.WriteGenerateStdTxResponse(w, txCtx, []sdk.Msg{msg}) + return + } + txBytes, err := txCtx.BuildAndSign(baseReq.Name, baseReq.Password, []sdk.Msg{msg}) if err != nil { utils.WriteErrorResponse(w, http.StatusUnauthorized, err.Error()) diff --git a/x/gov/msgs.go b/x/gov/msgs.go index 5d85f689e..dcd7112aa 100644 --- a/x/gov/msgs.go +++ b/x/gov/msgs.go @@ -12,11 +12,11 @@ const MsgType = "gov" //----------------------------------------------------------- // MsgSubmitProposal type MsgSubmitProposal struct { - Title string // Title of the proposal - Description string // Description of the proposal - ProposalType ProposalKind // Type of proposal. Initial set {PlainTextProposal, SoftwareUpgradeProposal} - Proposer sdk.AccAddress // Address of the proposer - InitialDeposit sdk.Coins // Initial deposit paid by sender. Must be strictly positive. + Title string `json:"title"` // Title of the proposal + Description string `json:"description"` // Description of the proposal + ProposalType ProposalKind `json:"proposal_type"` // Type of proposal. Initial set {PlainTextProposal, SoftwareUpgradeProposal} + Proposer sdk.AccAddress `json:"proposer"` // Address of the proposer + InitialDeposit sdk.Coins `json:"initial_deposit"` // Initial deposit paid by sender. Must be strictly positive. } func NewMsgSubmitProposal(title string, description string, proposalType ProposalKind, proposer sdk.AccAddress, initialDeposit sdk.Coins) MsgSubmitProposal { diff --git a/x/ibc/client/cli/ibctx.go b/x/ibc/client/cli/ibctx.go index f44c736d8..37cf7ce91 100644 --- a/x/ibc/client/cli/ibctx.go +++ b/x/ibc/client/cli/ibctx.go @@ -43,6 +43,9 @@ func IBCTransferCmd(cdc *wire.Codec) *cobra.Command { if err != nil { return err } + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, diff --git a/x/ibc/client/rest/transfer.go b/x/ibc/client/rest/transfer.go index 0580ff406..7f5595dd7 100644 --- a/x/ibc/client/rest/transfer.go +++ b/x/ibc/client/rest/transfer.go @@ -98,6 +98,11 @@ func TransferRequestHandlerFn(cdc *wire.Codec, kb keys.Keybase, cliCtx context.C txCtx = newCtx } + if utils.HasGenerateOnlyArg(r) { + utils.WriteGenerateStdTxResponse(w, txCtx, []sdk.Msg{msg}) + return + } + txBytes, err := txCtx.BuildAndSign(m.LocalAccountName, m.Password, []sdk.Msg{msg}) if err != nil { utils.WriteErrorResponse(w, http.StatusUnauthorized, err.Error()) diff --git a/x/ibc/types.go b/x/ibc/types.go index 5f9ee611b..b3fe6fc39 100644 --- a/x/ibc/types.go +++ b/x/ibc/types.go @@ -22,11 +22,11 @@ func init() { // IBCPacket defines a piece of data that can be send between two separate // blockchains. type IBCPacket struct { - SrcAddr sdk.AccAddress - DestAddr sdk.AccAddress - Coins sdk.Coins - SrcChain string - DestChain string + SrcAddr sdk.AccAddress `json:"src_addr"` + DestAddr sdk.AccAddress `json:"dest_addr"` + Coins sdk.Coins `json:"coins"` + SrcChain string `json:"src_chain"` + DestChain string `json:"dest_chain"` } func NewIBCPacket(srcAddr sdk.AccAddress, destAddr sdk.AccAddress, coins sdk.Coins, diff --git a/x/slashing/client/cli/tx.go b/x/slashing/client/cli/tx.go index 4234e52ce..d69a9e158 100644 --- a/x/slashing/client/cli/tx.go +++ b/x/slashing/client/cli/tx.go @@ -33,7 +33,9 @@ func GetCmdUnjail(cdc *wire.Codec) *cobra.Command { } msg := slashing.NewMsgUnjail(sdk.ValAddress(valAddr)) - + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, } diff --git a/x/slashing/client/rest/tx.go b/x/slashing/client/rest/tx.go index faf0341ce..bbed6ed9a 100644 --- a/x/slashing/client/rest/tx.go +++ b/x/slashing/client/rest/tx.go @@ -99,6 +99,11 @@ func unjailRequestHandlerFn(cdc *wire.Codec, kb keys.Keybase, cliCtx context.CLI txCtx = newCtx } + if utils.HasGenerateOnlyArg(r) { + utils.WriteGenerateStdTxResponse(w, txCtx, []sdk.Msg{msg}) + return + } + txBytes, err := txCtx.BuildAndSign(m.LocalAccountName, m.Password, []sdk.Msg{msg}) if err != nil { utils.WriteErrorResponse(w, http.StatusUnauthorized, "Must use own validator address") diff --git a/x/stake/client/cli/tx.go b/x/stake/client/cli/tx.go index defc2b1e2..bd38e7b5b 100644 --- a/x/stake/client/cli/tx.go +++ b/x/stake/client/cli/tx.go @@ -77,7 +77,9 @@ func GetCmdCreateValidator(cdc *wire.Codec) *cobra.Command { } else { msg = stake.NewMsgCreateValidator(sdk.ValAddress(valAddr), pk, amount, description) } - + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } // build and sign the transaction, then broadcast to Tendermint return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, @@ -117,6 +119,9 @@ func GetCmdEditValidator(cdc *wire.Codec) *cobra.Command { msg := stake.NewMsgEditValidator(sdk.ValAddress(valAddr), description) + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } // build and sign the transaction, then broadcast to Tendermint return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, @@ -156,6 +161,9 @@ func GetCmdDelegate(cdc *wire.Codec) *cobra.Command { msg := stake.NewMsgDelegate(delAddr, valAddr, amount) + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } // build and sign the transaction, then broadcast to Tendermint return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, @@ -225,6 +233,9 @@ func GetCmdBeginRedelegate(storeName string, cdc *wire.Codec) *cobra.Command { msg := stake.NewMsgBeginRedelegate(delAddr, valSrcAddr, valDstAddr, sharesAmount) + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } // build and sign the transaction, then broadcast to Tendermint return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, @@ -313,6 +324,9 @@ func GetCmdCompleteRedelegate(cdc *wire.Codec) *cobra.Command { msg := stake.NewMsgCompleteRedelegate(delAddr, valSrcAddr, valDstAddr) + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } // build and sign the transaction, then broadcast to Tendermint return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, @@ -374,6 +388,9 @@ func GetCmdBeginUnbonding(storeName string, cdc *wire.Codec) *cobra.Command { msg := stake.NewMsgBeginUnbonding(delAddr, valAddr, sharesAmount) + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } // build and sign the transaction, then broadcast to Tendermint return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, @@ -409,6 +426,9 @@ func GetCmdCompleteUnbonding(cdc *wire.Codec) *cobra.Command { msg := stake.NewMsgCompleteUnbonding(delAddr, valAddr) + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } // build and sign the transaction, then broadcast to Tendermint return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) }, diff --git a/x/stake/client/rest/tx.go b/x/stake/client/rest/tx.go index 9925b6b1c..4bb6d0f7e 100644 --- a/x/stake/client/rest/tx.go +++ b/x/stake/client/rest/tx.go @@ -297,6 +297,11 @@ func delegationsRequestHandlerFn(cdc *wire.Codec, kb keys.Keybase, cliCtx contex txCtx = newCtx } + if utils.HasGenerateOnlyArg(r) { + utils.WriteGenerateStdTxResponse(w, txCtx, []sdk.Msg{msg}) + return + } + txBytes, err := txCtx.BuildAndSign(m.LocalAccountName, m.Password, []sdk.Msg{msg}) if err != nil { utils.WriteErrorResponse(w, http.StatusUnauthorized, err.Error()) From 5ed24e44d2b84c2d433bbcc1be3f02f4345cac43 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Mon, 3 Sep 2018 19:13:08 -0700 Subject: [PATCH 42/49] simulation: Minor changes Now that we properly initialize governance, a ton of governance slashing doesn't happen in the first few blocks. Because of this, we can run through blocks in the range (0,200) quite rapidly. This PR acknowledges that and increases many of the default block heights. --- Makefile | 2 +- cmd/gaia/app/sim_test.go | 9 +++++---- x/mock/simulation/random_simulate_blocks.go | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index e363891e2..2f23fd54f 100644 --- a/Makefile +++ b/Makefile @@ -157,7 +157,7 @@ test_sim_gaia_nondeterminism: test_sim_gaia_fast: @echo "Running quick Gaia simulation. This may take several minutes..." - @go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=50 -v -timeout 24h + @go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=150 -v -timeout 24h test_sim_gaia_slow: @echo "Running full Gaia simulation. This may take awhile!" diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go index 3878166b3..61affb261 100644 --- a/cmd/gaia/app/sim_test.go +++ b/cmd/gaia/app/sim_test.go @@ -189,7 +189,7 @@ func TestAppStateDeterminism(t *testing.T) { t.Skip("Skipping Gaia simulation") } - numSeeds := 5 + numSeeds := 3 numTimesToRunPerSeed := 5 appHashList := make([]json.RawMessage, numTimesToRunPerSeed) @@ -206,10 +206,11 @@ func TestAppStateDeterminism(t *testing.T) { testAndRunTxs(app), []simulation.RandSetup{}, []simulation.Invariant{}, - 20, - 20, - true, + 50, + 100, + false, ) + app.Commit() appHash := app.LastCommitID().Hash appHashList[j] = appHash } diff --git a/x/mock/simulation/random_simulate_blocks.go b/x/mock/simulation/random_simulate_blocks.go index fc7543280..14ca6c216 100644 --- a/x/mock/simulation/random_simulate_blocks.go +++ b/x/mock/simulation/random_simulate_blocks.go @@ -155,7 +155,7 @@ func createBlockSimulator(testingMode bool, tb testing.TB, t *testing.T, event f AssertAllInvariants(t, app, invariants, log) } if opCount%50 == 0 { - fmt.Printf("\rSimulating... block %d/%d, operation %d/%d.", header.Height, totalNumBlocks, opCount, blocksize) + fmt.Printf("\rSimulating... block %d/%d, operation %d/%d. ", header.Height, totalNumBlocks, opCount, blocksize) } } opCount++ From 9b26ccfa76e1a100ba5cb12f46ed48ae03a78881 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Mon, 3 Sep 2018 21:29:59 -0700 Subject: [PATCH 43/49] tools: Remove gocyclo We can investigate re-introducing our own fork #postlaunch. Closes #2211 --- PENDING.md | 1 + tools/Makefile | 18 +++++++++--------- tools/gometalinter.json | 2 +- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/PENDING.md b/PENDING.md index 17616ced1..dd663564a 100644 --- a/PENDING.md +++ b/PENDING.md @@ -35,6 +35,7 @@ BREAKING CHANGES * [x/slashing] [#2122](https://github.com/cosmos/cosmos-sdk/pull/2122) - Implement slashing period * [types] \#2119 Parsed error messages and ABCI log errors to make them more human readable. * [simulation] Rename TestAndRunTx to Operation [#2153](https://github.com/cosmos/cosmos-sdk/pull/2153) + * [tools] Removed golint [#2211](https://github.com/cosmos/cosmos-sdk/issues/2211) * Tendermint diff --git a/tools/Makefile b/tools/Makefile index a11f2ec70..87544107c 100644 --- a/tools/Makefile +++ b/tools/Makefile @@ -22,7 +22,7 @@ INEFFASSIGN_CHECK := $(shell command -v ineffassign 2> /dev/null) MISSPELL_CHECK := $(shell command -v misspell 2> /dev/null) ERRCHECK_CHECK := $(shell command -v errcheck 2> /dev/null) UNPARAM_CHECK := $(shell command -v unparam 2> /dev/null) -GOCYCLO_CHECK := $(shell command -v gocyclo 2> /dev/null) +# GOCYCLO_CHECK := $(shell command -v gocyclo 2> /dev/null) check_tools: ifndef DEP_CHECK @@ -126,12 +126,12 @@ else @echo "Installing unparam" go get -v $(UNPARAM) endif -ifdef GOCYCLO_CHECK - @echo "gocyclo is already installed. Run 'make update_tools' to update." -else - @echo "Installing gocyclo" - go get -v $(GOCYCLO) -endif +# ifdef GOCYCLO_CHECK +# @echo "gocyclo is already installed. Run 'make update_tools' to update." +# else +# @echo "Installing gocyclo" +# go get -v $(GOCYCLO) +# endif update_tools: @echo "Updating dep" @@ -153,8 +153,8 @@ update_dev_tools: go get -u -v $(ERRCHECK) @echo "Updating unparam" go get -u -v $(UNPARAM) - @echo "Updating goyclo" - go get -u -v $(GOCYCLO) + # @echo "Updating goyclo" + # go get -u -v $(GOCYCLO) # To avoid unintended conflicts with file names, always add to .PHONY # unless there is a reason not to. diff --git a/tools/gometalinter.json b/tools/gometalinter.json index 124e28c14..42788714b 100644 --- a/tools/gometalinter.json +++ b/tools/gometalinter.json @@ -2,7 +2,7 @@ "Linters": { "vet": "go tool vet -composites=false :PATH:LINE:MESSAGE" }, - "Enable": ["golint", "vet", "ineffassign", "unparam", "unconvert", "misspell", "gocyclo"], + "Enable": ["golint", "vet", "ineffassign", "unparam", "unconvert", "misspell"], "Deadline": "500s", "Vendor": true, "Cyclo": 11 From ea01b91958766075b5461a5017a53ab681f33e42 Mon Sep 17 00:00:00 2001 From: ValarDragon Date: Mon, 3 Sep 2018 22:53:07 -0700 Subject: [PATCH 44/49] baseapp: Remove baseapp.SetTxDecoder() Closes #1441 --- PENDING.md | 1 + baseapp/setters.go | 6 ------ 2 files changed, 1 insertion(+), 6 deletions(-) diff --git a/PENDING.md b/PENDING.md index 17616ced1..3fcf0b3b3 100644 --- a/PENDING.md +++ b/PENDING.md @@ -35,6 +35,7 @@ BREAKING CHANGES * [x/slashing] [#2122](https://github.com/cosmos/cosmos-sdk/pull/2122) - Implement slashing period * [types] \#2119 Parsed error messages and ABCI log errors to make them more human readable. * [simulation] Rename TestAndRunTx to Operation [#2153](https://github.com/cosmos/cosmos-sdk/pull/2153) + * [baseapp] Remove `SetTxDecoder` in favor of requiring the decoder be set in baseapp initialization. [#1441](https://github.com/cosmos/cosmos-sdk/issues/1441) * Tendermint diff --git a/baseapp/setters.go b/baseapp/setters.go index a8b1591a7..28782c96d 100644 --- a/baseapp/setters.go +++ b/baseapp/setters.go @@ -26,12 +26,6 @@ func (app *BaseApp) SetCMS(cms store.CommitMultiStore) { } app.cms = cms } -func (app *BaseApp) SetTxDecoder(txDecoder sdk.TxDecoder) { - if app.sealed { - panic("SetTxDecoder() on sealed BaseApp") - } - app.txDecoder = txDecoder -} func (app *BaseApp) SetInitChainer(initChainer sdk.InitChainer) { if app.sealed { panic("SetInitChainer() on sealed BaseApp") From 8c32a8f2295dd9ce93b7dea6194b4910a37dbe62 Mon Sep 17 00:00:00 2001 From: Alessio Treglia Date: Tue, 4 Sep 2018 14:57:40 +0200 Subject: [PATCH 45/49] Check generate_only's output is actually unsigned --- client/lcd/lcd_test.go | 1 + cmd/gaia/cli_test/cli_test.go | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/client/lcd/lcd_test.go b/client/lcd/lcd_test.go index 7eb592a70..656362bcd 100644 --- a/client/lcd/lcd_test.go +++ b/client/lcd/lcd_test.go @@ -326,6 +326,7 @@ func TestCoinSendGenerateOnly(t *testing.T) { require.Equal(t, len(msg.Msgs), 1) require.Equal(t, msg.Msgs[0].Type(), "bank") require.Equal(t, msg.Msgs[0].GetSigners(), []sdk.AccAddress{addr}) + require.Equal(t, 0, len(msg.Signatures)) } func TestTxs(t *testing.T) { diff --git a/cmd/gaia/cli_test/cli_test.go b/cmd/gaia/cli_test/cli_test.go index 473a3bb3b..8b66ab4a1 100644 --- a/cmd/gaia/cli_test/cli_test.go +++ b/cmd/gaia/cli_test/cli_test.go @@ -164,6 +164,7 @@ func TestGaiaCLICreateValidator(t *testing.T) { msg := unmarshalStdTx(t, stdout) require.NotZero(t, msg.Fee.Gas) require.Equal(t, len(msg.Msgs), 1) + require.Equal(t, 0, len(msg.GetSignatures())) // Test --dry-run success = executeWrite(t, cvStr+" --dry-run", app.DefaultKeyPass) @@ -240,6 +241,7 @@ func TestGaiaCLISubmitProposal(t *testing.T) { msg := unmarshalStdTx(t, stdout) require.NotZero(t, msg.Fee.Gas) require.Equal(t, len(msg.Msgs), 1) + require.Equal(t, 0, len(msg.GetSignatures())) // Test --dry-run success = executeWrite(t, spStr+" --dry-run", app.DefaultKeyPass) @@ -271,6 +273,7 @@ func TestGaiaCLISubmitProposal(t *testing.T) { msg = unmarshalStdTx(t, stdout) require.NotZero(t, msg.Fee.Gas) require.Equal(t, len(msg.Msgs), 1) + require.Equal(t, 0, len(msg.GetSignatures())) executeWrite(t, depositStr, app.DefaultKeyPass) tests.WaitForNextNBlocksTM(2, port) @@ -294,6 +297,7 @@ func TestGaiaCLISubmitProposal(t *testing.T) { msg = unmarshalStdTx(t, stdout) require.NotZero(t, msg.Fee.Gas) require.Equal(t, len(msg.Msgs), 1) + require.Equal(t, 0, len(msg.GetSignatures())) executeWrite(t, voteStr, app.DefaultKeyPass) tests.WaitForNextNBlocksTM(2, port) @@ -350,6 +354,7 @@ func TestGaiaCLISendGenerateOnly(t *testing.T) { msg := unmarshalStdTx(t, stdout) require.Equal(t, msg.Fee.Gas, int64(client.DefaultGasLimit)) require.Equal(t, len(msg.Msgs), 1) + require.Equal(t, 0, len(msg.GetSignatures())) // Test generate sendTx, estimate gas success, stdout, stderr = executeWriteRetStdStreams(t, fmt.Sprintf( @@ -370,6 +375,7 @@ func TestGaiaCLISendGenerateOnly(t *testing.T) { msg = unmarshalStdTx(t, stdout) require.Equal(t, msg.Fee.Gas, int64(100)) require.Equal(t, len(msg.Msgs), 1) + require.Equal(t, 0, len(msg.GetSignatures())) } //___________________________________________________________________________________ From 89b3ab9e515c8992a210a7462930905d80465ed1 Mon Sep 17 00:00:00 2001 From: Alessio Treglia Date: Tue, 4 Sep 2018 17:29:33 +0200 Subject: [PATCH 46/49] Don't skip basic validation --- x/gov/client/cli/tx.go | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/x/gov/client/cli/tx.go b/x/gov/client/cli/tx.go index b0fa4d6be..e9a55c557 100644 --- a/x/gov/client/cli/tx.go +++ b/x/gov/client/cli/tx.go @@ -99,15 +99,15 @@ $ gaiacli gov submit-proposal --title="Test Proposal" --description="My awesome } msg := gov.NewMsgSubmitProposal(proposal.Title, proposal.Description, proposalType, fromAddr, amount) - if cliCtx.GenerateOnly { - return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) - } - err = msg.ValidateBasic() if err != nil { return err } + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } + // Build and sign the transaction, then broadcast to Tendermint // proposalID must be returned, and it is a part of response. cliCtx.PrintResponse = true @@ -180,14 +180,15 @@ func GetCmdDeposit(cdc *wire.Codec) *cobra.Command { } msg := gov.NewMsgDeposit(depositerAddr, proposalID, amount) - if cliCtx.GenerateOnly { - return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) - } err = msg.ValidateBasic() if err != nil { return err } + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } + // Build and sign the transaction, then broadcast to a Tendermint // node. return utils.SendTx(txCtx, cliCtx, []sdk.Msg{msg}) @@ -226,14 +227,15 @@ func GetCmdVote(cdc *wire.Codec) *cobra.Command { } msg := gov.NewMsgVote(voterAddr, proposalID, byteVoteOption) - if cliCtx.GenerateOnly { - return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) - } err = msg.ValidateBasic() if err != nil { return err } + if cliCtx.GenerateOnly { + return utils.PrintUnsignedStdTx(txCtx, cliCtx, []sdk.Msg{msg}) + } + fmt.Printf("Vote[Voter:%s,ProposalID:%d,Option:%s]", voterAddr.String(), msg.ProposalID, msg.Option.String(), ) From 8378f3ea86cc1db946987e3856be4f2bede943e2 Mon Sep 17 00:00:00 2001 From: Alessio Treglia Date: Tue, 4 Sep 2018 17:35:35 +0200 Subject: [PATCH 47/49] Drop note in PENDING.md about generate_only=true query arg --- PENDING.md | 1 + 1 file changed, 1 insertion(+) diff --git a/PENDING.md b/PENDING.md index 7289cd23f..e3be5b6b1 100644 --- a/PENDING.md +++ b/PENDING.md @@ -44,6 +44,7 @@ FEATURES * Gaia REST API (`gaiacli advanced rest-server`) * [lcd] Endpoints to query staking pool and params * [lcd] \#2110 Add support for `simulate=true` requests query argument to endpoints that send txs to run simulations of transactions + * [lcd] \#966 Add support for `generate_only=true` query argument to generate offline unsigned transactions * Gaia CLI (`gaiacli`) * [cli] Cmds to query staking pool and params From e9e83f0b6df672f7fe8e286e538ee51ee89eaf08 Mon Sep 17 00:00:00 2001 From: Rigel Date: Wed, 5 Sep 2018 13:59:37 -0400 Subject: [PATCH 48/49] Merge PR #2237: move WIP-lamborghini-distribution --- .../WIP-lamborghini-distribution/README.md | 0 .../WIP-lamborghini-distribution/end_block.md | 0 .../example_sheet/distribution.xlsx | Bin .../future_improvements.md | 0 .../WIP-lamborghini-distribution/overview.md | 0 .../WIP-lamborghini-distribution/state.md | 0 .../WIP-lamborghini-distribution/transactions.md | 0 .../WIP-lamborghini-distribution/triggers.md | 0 8 files changed, 0 insertions(+), 0 deletions(-) rename docs/{spec/distribution => _attic}/WIP-lamborghini-distribution/README.md (100%) rename docs/{spec/distribution => _attic}/WIP-lamborghini-distribution/end_block.md (100%) rename docs/{spec/distribution => _attic}/WIP-lamborghini-distribution/example_sheet/distribution.xlsx (100%) rename docs/{spec/distribution => _attic}/WIP-lamborghini-distribution/future_improvements.md (100%) rename docs/{spec/distribution => _attic}/WIP-lamborghini-distribution/overview.md (100%) rename docs/{spec/distribution => _attic}/WIP-lamborghini-distribution/state.md (100%) rename docs/{spec/distribution => _attic}/WIP-lamborghini-distribution/transactions.md (100%) rename docs/{spec/distribution => _attic}/WIP-lamborghini-distribution/triggers.md (100%) diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/README.md b/docs/_attic/WIP-lamborghini-distribution/README.md similarity index 100% rename from docs/spec/distribution/WIP-lamborghini-distribution/README.md rename to docs/_attic/WIP-lamborghini-distribution/README.md diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/end_block.md b/docs/_attic/WIP-lamborghini-distribution/end_block.md similarity index 100% rename from docs/spec/distribution/WIP-lamborghini-distribution/end_block.md rename to docs/_attic/WIP-lamborghini-distribution/end_block.md diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/example_sheet/distribution.xlsx b/docs/_attic/WIP-lamborghini-distribution/example_sheet/distribution.xlsx similarity index 100% rename from docs/spec/distribution/WIP-lamborghini-distribution/example_sheet/distribution.xlsx rename to docs/_attic/WIP-lamborghini-distribution/example_sheet/distribution.xlsx diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/future_improvements.md b/docs/_attic/WIP-lamborghini-distribution/future_improvements.md similarity index 100% rename from docs/spec/distribution/WIP-lamborghini-distribution/future_improvements.md rename to docs/_attic/WIP-lamborghini-distribution/future_improvements.md diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/overview.md b/docs/_attic/WIP-lamborghini-distribution/overview.md similarity index 100% rename from docs/spec/distribution/WIP-lamborghini-distribution/overview.md rename to docs/_attic/WIP-lamborghini-distribution/overview.md diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/state.md b/docs/_attic/WIP-lamborghini-distribution/state.md similarity index 100% rename from docs/spec/distribution/WIP-lamborghini-distribution/state.md rename to docs/_attic/WIP-lamborghini-distribution/state.md diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/transactions.md b/docs/_attic/WIP-lamborghini-distribution/transactions.md similarity index 100% rename from docs/spec/distribution/WIP-lamborghini-distribution/transactions.md rename to docs/_attic/WIP-lamborghini-distribution/transactions.md diff --git a/docs/spec/distribution/WIP-lamborghini-distribution/triggers.md b/docs/_attic/WIP-lamborghini-distribution/triggers.md similarity index 100% rename from docs/spec/distribution/WIP-lamborghini-distribution/triggers.md rename to docs/_attic/WIP-lamborghini-distribution/triggers.md From a94219e062d585d396ed8cf242675d58bbd6f737 Mon Sep 17 00:00:00 2001 From: Dev Ojha Date: Thu, 6 Sep 2018 21:54:59 -0700 Subject: [PATCH 49/49] Merge PR #2246: add script to make changelog links actual links --- PENDING.md | 56 ++++++++++++++++++------------------ scripts/linkify_changelog.py | 13 +++++++++ 2 files changed, 41 insertions(+), 28 deletions(-) create mode 100644 scripts/linkify_changelog.py diff --git a/PENDING.md b/PENDING.md index 6ccd96be3..e5ea9884a 100644 --- a/PENDING.md +++ b/PENDING.md @@ -8,32 +8,32 @@ BREAKING CHANGES * Gaia CLI (`gaiacli`) * [x/stake] Validator.Owner renamed to Validator.Operator * [cli] unsafe_reset_all, show_validator, and show_node_id have been renamed to unsafe-reset-all, show-validator, and show-node-id - * [cli] \#1983 --print-response now defaults to true in commands that create and send a transaction - * [cli] \#1983 you can now pass --pubkey or --address to gaiacli keys show to return a plaintext representation of the key's address or public key for use with other commands - * [cli] \#2061 changed proposalID in governance REST endpoints to proposal-id - * [cli] \#2014 `gaiacli advanced` no longer exists - to access `ibc`, `rest-server`, and `validator-set` commands use `gaiacli ibc`, `gaiacli rest-server`, and `gaiacli tendermint`, respectively + * [cli] [\#1983](https://github.com/cosmos/cosmos-sdk/issues/1983) --print-response now defaults to true in commands that create and send a transaction + * [cli] [\#1983](https://github.com/cosmos/cosmos-sdk/issues/1983) you can now pass --pubkey or --address to gaiacli keys show to return a plaintext representation of the key's address or public key for use with other commands + * [cli] [\#2061](https://github.com/cosmos/cosmos-sdk/issues/2061) changed proposalID in governance REST endpoints to proposal-id + * [cli] [\#2014](https://github.com/cosmos/cosmos-sdk/issues/2014) `gaiacli advanced` no longer exists - to access `ibc`, `rest-server`, and `validator-set` commands use `gaiacli ibc`, `gaiacli rest-server`, and `gaiacli tendermint`, respectively * [makefile] `get_vendor_deps` no longer updates lock file it just updates vendor directory. Use `update_vendor_deps` to update the lock file. [#2152](https://github.com/cosmos/cosmos-sdk/pull/2152) - * [cli] \#2190 `gaiacli init --gen-txs` is now `gaiacli init --with-txs` to reduce confusion - * \#2040 All commands that utilize a validator's address must now use the new + * [cli] [\#2190](https://github.com/cosmos/cosmos-sdk/issues/2190) `gaiacli init --gen-txs` is now `gaiacli init --with-txs` to reduce confusion + * [\#2040](https://github.com/cosmos/cosmos-sdk/issues/2040) All commands that utilize a validator's address must now use the new bech32 prefix, `cosmosval`. A validator's Tendermint signing key and address now use a new bech32 prefix, `cosmoscons`. * Gaia * Make the transient store key use a distinct store key. [#2013](https://github.com/cosmos/cosmos-sdk/pull/2013) - * [x/stake] \#1901 Validator type's Owner field renamed to Operator; Validator's GetOwner() renamed accordingly to comply with the SDK's Validator interface. + * [x/stake] [\#1901](https://github.com/cosmos/cosmos-sdk/issues/1901) Validator type's Owner field renamed to Operator; Validator's GetOwner() renamed accordingly to comply with the SDK's Validator interface. * [docs] [#2001](https://github.com/cosmos/cosmos-sdk/pull/2001) Update slashing spec for slashing period * [x/stake, x/slashing] [#1305](https://github.com/cosmos/cosmos-sdk/issues/1305) - Rename "revoked" to "jailed" * [x/stake] [#1676] Revoked and jailed validators put into the unbonding state * [x/stake] [#1877] Redelegations/unbonding-delegation from unbonding validator have reduced time - * [x/stake] \#2040 Validator operator type has now changed to `sdk.ValAddress` + * [x/stake] [\#2040](https://github.com/cosmos/cosmos-sdk/issues/2040) Validator operator type has now changed to `sdk.ValAddress` * A new bech32 prefix has been introduced for Tendermint signing keys and addresses, `cosmosconspub` and `cosmoscons` respectively. - + * SDK - * [core] \#1807 Switch from use of rational to decimal - * [types] \#1901 Validator interface's GetOwner() renamed to GetOperator() + * [core] [\#1807](https://github.com/cosmos/cosmos-sdk/issues/1807) Switch from use of rational to decimal + * [types] [\#1901](https://github.com/cosmos/cosmos-sdk/issues/1901) Validator interface's GetOwner() renamed to GetOperator() * [x/slashing] [#2122](https://github.com/cosmos/cosmos-sdk/pull/2122) - Implement slashing period - * [types] \#2119 Parsed error messages and ABCI log errors to make them more human readable. + * [types] [\#2119](https://github.com/cosmos/cosmos-sdk/issues/2119) Parsed error messages and ABCI log errors to make them more human readable. * [simulation] Rename TestAndRunTx to Operation [#2153](https://github.com/cosmos/cosmos-sdk/pull/2153) * [tools] Removed gocyclo [#2211](https://github.com/cosmos/cosmos-sdk/issues/2211) * [baseapp] Remove `SetTxDecoder` in favor of requiring the decoder be set in baseapp initialization. [#1441](https://github.com/cosmos/cosmos-sdk/issues/1441) @@ -45,26 +45,26 @@ FEATURES * Gaia REST API (`gaiacli advanced rest-server`) * [lcd] Endpoints to query staking pool and params - * [lcd] \#2110 Add support for `simulate=true` requests query argument to endpoints that send txs to run simulations of transactions - * [lcd] \#966 Add support for `generate_only=true` query argument to generate offline unsigned transactions + * [lcd] [\#2110](https://github.com/cosmos/cosmos-sdk/issues/2110) Add support for `simulate=true` requests query argument to endpoints that send txs to run simulations of transactions + * [lcd] [\#966](https://github.com/cosmos/cosmos-sdk/issues/966) Add support for `generate_only=true` query argument to generate offline unsigned transactions * Gaia CLI (`gaiacli`) * [cli] Cmds to query staking pool and params * [gov][cli] #2062 added `--proposal` flag to `submit-proposal` that allows a JSON file containing a proposal to be passed in - * \#2040 Add `--bech` to `gaiacli keys show` and respective REST endpoint to + * [\#2040](https://github.com/cosmos/cosmos-sdk/issues/2040) Add `--bech` to `gaiacli keys show` and respective REST endpoint to provide desired Bech32 prefix encoding - * [cli] \#2047 Setting the --gas flag value to 0 triggers a simulation of the tx before the actual execution. The gas estimate obtained via the simulation will be used as gas limit in the actual execution. - * [cli] \#2047 The --gas-adjustment flag can be used to adjust the estimate obtained via the simulation triggered by --gas=0. - * [cli] \#2110 Add --dry-run flag to perform a simulation of a transaction without broadcasting it. The --gas flag is ignored as gas would be automatically estimated. - * [cli] \#966 Add --generate-only flag to build an unsigned transaction and write it to STDOUT. + * [cli] [\#2047](https://github.com/cosmos/cosmos-sdk/issues/2047) Setting the --gas flag value to 0 triggers a simulation of the tx before the actual execution. The gas estimate obtained via the simulation will be used as gas limit in the actual execution. + * [cli] [\#2047](https://github.com/cosmos/cosmos-sdk/issues/2047) The --gas-adjustment flag can be used to adjust the estimate obtained via the simulation triggered by --gas=0. + * [cli] [\#2110](https://github.com/cosmos/cosmos-sdk/issues/2110) Add --dry-run flag to perform a simulation of a transaction without broadcasting it. The --gas flag is ignored as gas would be automatically estimated. + * [cli] [\#966](https://github.com/cosmos/cosmos-sdk/issues/966) Add --generate-only flag to build an unsigned transaction and write it to STDOUT. * Gaia * [cli] #2170 added ability to show the node's address via `gaiad tendermint show-address` * SDK * [querier] added custom querier functionality, so ABCI query requests can be handled by keepers - * [simulation] \#1924 allow operations to specify future operations - * [simulation] \#1924 Add benchmarking capabilities, with makefile commands "test_sim_gaia_benchmark, test_sim_gaia_profile" + * [simulation] [\#1924](https://github.com/cosmos/cosmos-sdk/issues/1924) allow operations to specify future operations + * [simulation] [\#1924](https://github.com/cosmos/cosmos-sdk/issues/1924) Add benchmarking capabilities, with makefile commands "test_sim_gaia_benchmark, test_sim_gaia_profile" * Tendermint @@ -74,7 +74,7 @@ IMPROVEMENTS * [tools] Added ansible script to enable process core dumps * Gaia REST API (`gaiacli advanced rest-server`) - * [x/stake] \#2000 Added tests for new staking endpoints + * [x/stake] [\#2000](https://github.com/cosmos/cosmos-sdk/issues/2000) Added tests for new staking endpoints * Gaia CLI (`gaiacli`) * [cli] #2060 removed `--select` from `block` command @@ -88,7 +88,7 @@ IMPROVEMENTS * SDK * [tools] Make get_vendor_deps deletes `.vendor-new` directories, in case scratch files are present. * [spec] Added simple piggy bank distribution spec - * [cli] \#1632 Add integration tests to ensure `basecoind init && basecoind` start sequences run successfully for both `democoin` and `basecoin` examples. + * [cli] [\#1632](https://github.com/cosmos/cosmos-sdk/issues/1632) Add integration tests to ensure `basecoind init && basecoind` start sequences run successfully for both `democoin` and `basecoin` examples. * [store] Speedup IAVL iteration, and consequently everything that requires IAVL iteration. [#2143](https://github.com/cosmos/cosmos-sdk/issues/2143) * [store] \#1952 Update IAVL dependency to v0.10.0 * [simulation] Make timestamps randomized [#2153](https://github.com/cosmos/cosmos-sdk/pull/2153) @@ -100,15 +100,15 @@ BUG FIXES * Gaia REST API (`gaiacli advanced rest-server`) * Gaia CLI (`gaiacli`) - * [cli] \#1997 Handle panics gracefully when `gaiacli stake {delegation,unbond}` fail to unmarshal delegation. + * [cli] [\#1997](https://github.com/cosmos/cosmos-sdk/issues/1997) Handle panics gracefully when `gaiacli stake {delegation,unbond}` fail to unmarshal delegation. * Gaia * SDK - * \#1988 Make us compile on OpenBSD (disable ledger) [#1988] (https://github.com/cosmos/cosmos-sdk/issues/1988) - * \#2105 Fix DB Iterator leak, which may leak a go routine. - * [ledger] \#2064 Fix inability to sign and send transactions via the LCD by + * [\#1988](https://github.com/cosmos/cosmos-sdk/issues/1988) Make us compile on OpenBSD (disable ledger) [#1988] (https://github.com/cosmos/cosmos-sdk/issues/1988) + * [\#2105](https://github.com/cosmos/cosmos-sdk/issues/2105) Fix DB Iterator leak, which may leak a go routine. + * [ledger] [\#2064](https://github.com/cosmos/cosmos-sdk/issues/2064) Fix inability to sign and send transactions via the LCD by loading a Ledger device at runtime. - * \#2158 Fix non-deterministic ordering of validator iteration when slashing in `gov EndBlocker` + * [\#2158](https://github.com/cosmos/cosmos-sdk/issues/2158) Fix non-deterministic ordering of validator iteration when slashing in `gov EndBlocker` * Tendermint diff --git a/scripts/linkify_changelog.py b/scripts/linkify_changelog.py new file mode 100644 index 000000000..3423506e7 --- /dev/null +++ b/scripts/linkify_changelog.py @@ -0,0 +1,13 @@ +import fileinput +import re + +# This script goes through the provided file, and replaces any " \#", +# with the valid mark down formatted link to it. e.g. +# " [\#number](https://github.com/cosmos/cosmos-sdk/issues/) +# Note that if the number is for a PR, github will auto-redirect you when you click the link. +# It is safe to run the script multiple times in succession. +# +# Example usage $ python3 linkify_changelog.py ../PENDING.md +for line in fileinput.input(inplace=1): + line = re.sub(r"\s\\#([0-9]*)", r" [\\#\1](https://github.com/cosmos/cosmos-sdk/issues/\1)", line.rstrip()) + print(line) \ No newline at end of file