cleanup/optimization

This commit is contained in:
Matthew Pilkey 2020-04-21 11:47:26 -05:00
parent 3e9e784b3b
commit 173fb76bac
24 changed files with 116 additions and 301 deletions

View File

@ -47,7 +47,7 @@ projects
Feel free to add additional files and folders within your project adapter directly as needed to help organize your code, but most projects shouldn't be too large. Please keep all code within your project adapter directory - PR's with modifications or additions outside the scope of your own project usually won't be allowed, and in cases where they are needed will need to be discussed with the DeFi Pulse team in advance.
## The Run Function
## The TVL Function
The main tvl function of a project adater is where token balances are fetched. On DeFi Pulse, these functions are run every hour, with a unix timestamp and block number passed to the function. Please note that project adapters need to be able to run successfully for any point back to a projects starting time, not just for recent points. This is necessary both to allow collection if historical data that may exist prior to the release of a newly added project, and for repairing or catching up a projects data history in the event of any errors.
@ -58,19 +58,15 @@ async function tvl(timestamp, block) {
'0x6B175474E89094C44Da98b954EedeAC495271d0F': 2000000000000000000 // DAI
};
let symbolBalances = await sdk.api.util.toSymbols(balances);
return symbolBalances.output;
return balances;
}
```
In the case of the `_template' adapter, we're just using some hard coded values as token balances to illustrate a minimal implementation.
For consistency, we treat balances associated with token addresses as raw/wei values (before decimal conversion) and balances associated with token symbols as decimal converted values. Due to the methods most commonly available in core contracts for most projects already on Defi Pulse (and a lack of broad standardization), we've found the most effective solution is for project adapters to work internally with token addresses where possible, and translate those raw addresses/wei to tokens/decimal converted values as late as possible.
For consistency, we treat balances associated with token addresses as raw/wei values (before decimal conversion) and balances associated with token symbols as decimal converted values. Due to the methods most commonly available in core contracts for most projects already on Defi Pulse (and a lack of broad standardization), we've found the most effective solution is for project adapters to work internally with token addresses; symbol conversions are done automatically after the adapter runs;
the `_template` project illustrates a helper function available to assist with this conversion, the `toSymbols` method accepts an object containing these raw values, and outputs the decimal converted values with proper token symbols. This is completely optional however, and depending on how your project works and the methods available it may make more sense to deal directly with symbols from the start.
The important thing is that the adapter is expected to output an object with token symbols and decimal converted values. For the `_template` adapter, the output looks like this:
The important thing is that the adapter is expected to output an object with token addresses and raw non-decimal converted balances. For the `_template` adapter, the output looks like this:
```js
{
@ -99,22 +95,9 @@ async function run(timestamp, block) {
abi: 'erc20:balanceOf'
});
_.each(balanceOfResults.output, (balanceOf) => {
if(balanceOf.success) {
let balance = balanceOf.output;
let address = balanceOf.input.target;
await sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
if (BigNumber(balance).toNumber() <= 0) {
return;
}
balances[address] = BigNumber(balances[address] || 0).plus(balance).toFixed();
}
});
let symbolBalances = await sdk.api.util.toSymbols(balances);
return symbolBalances.output;
return balances;
}
```
@ -154,9 +137,9 @@ Each project adapter needs to export the main run function, in addition to some
module.exports = {
name: 'Template Project', // project name
token: null, // null, or token symbol if project has a custom token
category: 'Assets', // allowed values as shown on DefiPulse: 'Derivatives', 'DEXes', 'Lending', 'Payments', 'Assets'
category: 'assets', // allowed values as shown on DefiPulse: 'derivatives', 'dexes', 'lending', 'payments', 'assets'
start: 1514764800, // unix timestamp (utc 0) specifying when the project began, or where live data begins
tvl // adapter
tvl // tvl adapter
}
```
@ -166,13 +149,13 @@ Here's a look at the `bancor` adapter for a practical example of this:
module.exports = {
name: 'Bancor',
token: 'BNT',
category: 'DEXes',
category: 'dexes',
start: 1501632000, // 08/02/2017 @ 12:00am (UTC)
tvl
}
```
You can see the affect of these settings on [DeFi Pulse](https://defipulse.com/).
You can see the effect of these settings on [DeFi Pulse](https://defipulse.com/).
The project's name and protocol token are simple values shown in the UI and API data. Category influences how the project is classified and what section it's tvl contributes to for aggregate results on the main page. Finally the start time defines the limit for how far back data needs to be fetched to retrieve the projects full historical data.
@ -187,21 +170,28 @@ npm run test -- --project=_template
The `test` command will run a project adapter once for the latest hourly point, perform some basic checks on it's output, and log the result.
```
_template project running & output format
runs for specified time: latest
✓ returns valid data at point hour 0 (1600ms)
✔ Output: {
"ethCallCount": 0,
"timestamp": 1581026400,
"block": 9431690,
"output": {
"ETH": "1",
"DAI": "2"
}
}
_template project running & output format
runs for specified time: latest hour
✓ returns valid tvl data at hour 0 (1172ms)
1 passing (2s)
1 passing (1s)
```
Output of tests are stored in json files under `output/**project_name**/**adapter_type**` and are named based on the time they were run to fetch data for.
In the above example, the output is saved to `output/_template/tvl/2020-04-16T17:00:00Z.json` since the project is `_template`, and the adapter provided is for tvl. It's output is shown below:
```json
{
"ethCallCount": 0,
"timestamp": 1587481200,
"block": 9916481,
"output": {
"ETH": "1",
"DAI": "2"
}
}
```
The test will output how many Ethereum network calls were used for the run, confirm the timestamp and block the test was run on, and provide the output generated by the adapter.

View File

@ -47,48 +47,6 @@ Example:
# Methods
## Utility - *sdk.api.util.___*
### toSymbols(*object*)
Convert a key/value list of token addresses/balances to decimal converted key/value list of symbols/balances.
###### Arguments
| Type | Description |
| ------------------------------ | ------------------------------------------------------------ |
| *object* | *(address)*: *number/string* |
###### Return Object
| Name | Type | Description |
| ------------------------------ | ------------------------------ | ------------------------------------------------------------ |
| ethCallCount | *number* | Number of Ethereum network calls used |
| output | *object* | *(symbol)*: *number/string* |
For consistency, we treat balances associated with token addresses as raw/wei values (before decimal conversion) and balances associated with token symbols as decimal converted values. **toSymbols** accepts key/value pairs of token addresses and balances, and will return the resulting key/value pairs of symbols and decimal converted balances.
the SDK server maintains an extensive list of symbol and decimal values for popular token addresses, but if no stored information is available a fallback is used to call erc20 contract methods to retrieve symbol and decimal values. In most cases this means no Ethereum node calls will need to be made to convert addresses to symbols - If you need to work with an address/symbol that doesn't have data listed, please advise the DeFi Pulse team so we can add it.
###### Example Call
```js
let result = await sdk.api.util.toSymbols({
'0x0000000000000000000000000000000000000000': '1000000000000000000', // ETH
'0x6B175474E89094C44Da98b954EedeAC495271d0F': '2000000000000000000' // DAI
});
```
###### Result
```js
{
ethCallCount: 0,
output: {
ETH: '1',
DAI: '2'
}
}
```
## ERC20 - *sdk.api.erc20.___*
### symbol(*string*)

View File

@ -14,9 +14,7 @@
'0x6B175474E89094C44Da98b954EedeAC495271d0F': 2000000000000000000 // DAI
};
let symbolBalances = await sdk.api.util.toSymbols(balances);
return symbolBalances.output;
return balances;
}
/*==================================================
@ -26,7 +24,7 @@
module.exports = {
name: 'Template Project', // project name
token: null, // null, or token symbol if project has a custom token
category: 'Assets', // allowed values as shown on DefiPulse: 'Derivatives', 'DEXes', 'Lending', 'Payments', 'Assets'
category: 'assets', // allowed values as shown on DefiPulse: 'Derivatives', 'DEXes', 'Lending', 'Payments', 'Assets'
start: 1514764800, // unix timestamp (utc 0) specifying when the project began, or where live data begins
tvl // tvl adapter
}

View File

@ -48,7 +48,7 @@
abi: 'erc20:balanceOf'
});
sdk.util.sumMultiCall(balances, balanceOfResults);
sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
return balances;
}
@ -60,7 +60,7 @@
module.exports = {
name: 'Aave',
token: 'LEND',
category: 'Lending',
category: 'lending',
start: 1578355200, // 01/07/2020 @ 12:00am (UTC)
tvl
}

View File

@ -23,7 +23,7 @@
module.exports = {
name: 'Augur',
token: 'REP',
category: 'Derivatives',
category: 'derivatives',
start: 1531008000, // 07/08/2018 @ 12:00am (UTC)
tvl
}

View File

@ -84,36 +84,21 @@
==================================================*/
async function tvl(timestamp, block) {
let getBalance = await sdk.api.eth.getBalance({target: '0xc0829421C1d260BD3cB3E0F06cfE2D52db2cE315', block});
let balances = {
'0x0000000000000000000000000000000000000000': getBalance.output
'0x0000000000000000000000000000000000000000': (await sdk.api.eth.getBalance({target: '0xc0829421C1d260BD3cB3E0F06cfE2D52db2cE315', block})).output
};
let calls = await GenerateCallList(timestamp);
const calls = await GenerateCallList(timestamp);
let balanceOfResults = await sdk.api.abi.multiCall({
const balanceOfResults = await sdk.api.abi.multiCall({
block,
calls,
abi: 'erc20:balanceOf'
});
_.each(balanceOfResults.output, (balanceOf) => {
if(balanceOf.success) {
let balance = balanceOf.output;
let address = balanceOf.input.target;
sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
if (BigNumber(balance).toNumber() <= 0) {
return;
}
balances[address] = BigNumber(balances[address] || 0).plus(balance).toFixed();
}
});
let symbolBalances = await sdk.api.util.toSymbols(balances);
return symbolBalances.output;
return balances;
}
/*==================================================
@ -123,7 +108,7 @@
module.exports = {
name: 'Bancor',
token: 'BNT',
category: 'DEXes',
category: 'dexes',
start: 1501632000, // 08/02/2017 @ 12:00am (UTC)
tvl
}

View File

@ -61,10 +61,14 @@
});
_.each(iTokens, (iToken) => {
const totalSupply = _.find(supplyResult.output, (result) => (result.input.target === iToken.iTokenAddress)).output;
const totalBorrow = _.find(borrowResult.output, (result) => (result.input.target === iToken.iTokenAddress)).output;
const supply = _.find(supplyResult.output, (result) => (result.input.target === iToken.iTokenAddress));
const borrow = _.find(borrowResult.output, (result) => (result.input.target === iToken.iTokenAddress));
balances[iToken.underlyingAddress] = BigNumber(totalSupply).minus(totalBorrow).toFixed();
if(supply.success && borrow.success) {
const totalSupply = supply.output;
const totalBorrow = borrow.output;
balances[iToken.underlyingAddress] = BigNumber(totalSupply).minus(totalBorrow).toFixed();
}
});
const kyberTokens = (await sdk.api.util.kyberTokens()).output;
@ -82,7 +86,7 @@
abi: 'erc20:balanceOf',
});
sdk.util.sumMultiCall(balances, balanceOfResult);;
sdk.util.sumMultiBalanceOf(balances, balanceOfResult);;
return balances;
}
@ -94,7 +98,7 @@
module.exports = {
name: 'bZx',
token: 'BZRX',
category: 'Lending',
category: 'lending',
start: 1558742400, // 05/25/2019(UTC)
tvl
};

View File

@ -46,7 +46,7 @@ async function tvl(timestamp, block) {
abi: 'erc20:balanceOf'
});
await sdk.util.sumMultiCall(balances, balanceOfResults);
await sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
return balances;
}
@ -57,7 +57,7 @@ async function tvl(timestamp, block) {
module.exports = {
name: 'DDEX',
category: 'Lending',
category: 'lending',
start: 1566470505, // 2019-08-22T18:41:45+08:00
tvl
}

View File

@ -42,7 +42,7 @@
abi: 'erc20:balanceOf'
});
sdk.util.sumMultiCall(balances, balanceOfResults);
sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
return balances;
}
@ -54,7 +54,7 @@
module.exports = {
name: 'dForce',
token: 'DF',
category: 'Lending',
category: 'lending',
start: 1565043417, // Aug-06-2019 06:16:57 AM +UTC
tvl
}

View File

@ -278,9 +278,7 @@
}
}
symbolBalances = await sdk.api.util.toSymbols(balances);
return symbolBalances.output;
return balances;
}
/*==================================================
@ -290,7 +288,7 @@
module.exports = {
name: 'Dharma',
token: null,
category: 'Lending',
category: 'lending',
contributesTo: ['Compound'],
start: 1526947200, // 05/22/2018 @ 12:00am (UTC)
tvl

View File

@ -118,9 +118,7 @@
}
});
let symbolBalances = await sdk.api.util.toSymbols(balances);
return symbolBalances.output;
return balances;
}
/*==================================================
@ -130,7 +128,7 @@
module.exports = {
name: "Erasure",
token: "NMR",
category: "Derivatives",
category: "derivatives",
start: 1566518400, // 08/23/2019 @ 12:00am (UTC)
tvl
};

View File

@ -31,7 +31,7 @@
targets: wallets
})).output;
return (await sdk.api.util.toSymbols(balances)).output;
return balances;
}
/*==================================================
@ -41,7 +41,7 @@
module.exports = {
name: 'InstaDApp',
token: null,
category: 'Lending',
category: 'lending',
contributesTo: ['Maker', 'Compound'],
start: 1543622400, // 12/01/2018 @ 12:00am (UTC)
tvl

View File

@ -9,8 +9,10 @@
/*==================================================
Vars
==================================================*/
const loopringExchangeAddr = '0x944644Ea989Ec64c2Ab9eF341D383cEf586A5777';
const wedexExchangeAddr = '0xD97D09f3bd931a14382ac60f156C1285a56Bb51B';
const listedTokens = [
"0xBBbbCA6A901c926F240b89EacB641d8Aec7AEafD", // LRC
"0xdac17f958d2ee523a2206206994597c13d831ec7", // USDT
@ -49,22 +51,9 @@
abi: 'erc20:balanceOf'
});
_.each(balanceOfResults.output, (balanceOf) => {
if(balanceOf.success) {
let balance = balanceOf.output;
let address = balanceOf.input.target;
sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
if (BigNumber(balance).toNumber() <= 0) {
return;
}
balances[address] = BigNumber(balances[address] || 0).plus(balance).toFixed();
}
});
let symbolBalances = await sdk.api.util.toSymbols(balances);
return symbolBalances.output;
return balances;
}
/*==================================================
@ -74,7 +63,7 @@
module.exports = {
name: 'Loopring',
token: 'LRC',
category: 'DEXes',
category: 'dexes',
start: 1574241665, // 11/20/2019 @ 09:21AM (UTC)
tvl
}

View File

@ -178,7 +178,7 @@ async function tvl(timestamp, block) {
}
});
return (await sdk.api.util.toSymbols(balances)).output;
return balances;
}
/*==================================================
@ -188,7 +188,7 @@ async function tvl(timestamp, block) {
module.exports = {
name: "Melon",
token: "MLN",
category: "Assets",
category: "assets",
start: 1551398400, // 03/01/2019 @ 12:00am (UTC)
tvl
};

View File

@ -44,19 +44,14 @@
abi: 'erc20:balanceOf'
});
_.each(balanceOfResults.output, (balanceOf) => {
if(balanceOf.success) {
let address = balanceOf.input.target
balances[address] = BigNumber(balances[address] || 0).plus(balanceOf.output).toFixed();
}
});
sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
for(let pool of pools) {
let balance = (await sdk.api.eth.getBalance({target: pool, block})).output;
balances['0x0000000000000000000000000000000000000000'] = BigNumber(balances['0x0000000000000000000000000000000000000000'] || 0).plus(balance).toFixed();
}
return (await sdk.api.util.toSymbols(balances)).output;
return balances;
}
/*==================================================
@ -66,7 +61,7 @@
module.exports = {
name: 'Nexus Mutual',
token: 'NXM',
category: 'Derivatives',
category: 'derivatives',
start: 1558569600, // 05/23/2019 @ 12:00am (UTC)
tvl
}

View File

@ -114,20 +114,6 @@
});
});
let balanceOfResults = await sdk.api.abi.multiCall({
block,
calls,
abi: 'erc20:balanceOf'
});
_.each(balanceOfResults.output, (balanceOf) => {
if(balanceOf.success) {
let address = balanceOf.input.target
balances[address] = BigNumber(balances[address] || 0).plus(balanceOf.output).toFixed();
}
});
let lastReserveRunsResults = await sdk.api.abi.multiCall({
block,
calls: _.map(tokenAddresses, (tokenAddress) => {
@ -175,22 +161,15 @@
});
});
let actualBalances = await sdk.api.abi.multiCall({
let balanceOfResults = await sdk.api.abi.multiCall({
block: block,
calls: aCalls,
calls: [...calls, ...aCalls],
abi: 'erc20:balanceOf'
});
_.each(actualBalances.output, (actualBalance) => {
if(actualBalance.success) {
let address = actualBalance.input.target
balances[address] = BigNumber(balances[address] || 0).plus(actualBalance.output).toFixed();
}
});
sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
let symbolBalances = await sdk.api.util.toSymbols(balances);
return symbolBalances.output;
return balances;
}
/*==================================================
@ -225,7 +204,7 @@
module.exports = {
name: 'Nuo Network',
token: null,
category: 'Lending',
category: 'lending',
start: 1548115200, // 01/22/2019 @ 12:00am (UTC)
tvl,
rates,

View File

@ -29,6 +29,9 @@
async function tvl(timestamp, block) {
let balances = {};
let balanceOfCalls = [];
let optionsAddresses = []
for(let i = 0; i < factoriesAddresses.length; i++) {
// number of created oTokens
let numberOfOptionsContracts = (
@ -55,9 +58,6 @@
})
).output;
// list of options addresses
let optionsAddresses = []
_.each(optionsContracts, async (contracts) => {
optionsAddresses = [
...optionsAddresses,
@ -70,35 +70,27 @@
let balance = (await sdk.api.eth.getBalance({target: optionAddress, block})).output;
balances["0x0000000000000000000000000000000000000000"] = BigNumber(balances["0x0000000000000000000000000000000000000000"] || 0).plus(BigNumber(balance)).toFixed();
})
// batch balanceOf calls
let balanceOfCalls = [];
_.each(optionsAddresses, (optionsAddress) => {
_.each(tokenAddresses, (tokenAddress) => {
balanceOfCalls.push({
target: tokenAddress,
params: [optionsAddress]
});
});
});
// get tokens balances
const balanceOfResults = await sdk.api.abi.multiCall({
block,
calls: balanceOfCalls,
abi: "erc20:balanceOf"
});
_.each(balanceOfResults.output, (balanceOf) => {
if(balanceOf.success) {
let address = balanceOf.input.target
balances[address] = BigNumber(balances[address] || 0).plus(balanceOf.output).toFixed();
}
});
}
return (await sdk.api.util.toSymbols(balances)).output;
_.each(optionsAddresses, (optionsAddress) => {
_.each(tokenAddresses, (tokenAddress) => {
balanceOfCalls.push({
target: tokenAddress,
params: [optionsAddress]
});
});
});
// get tokens balances
const balanceOfResults = await sdk.api.abi.multiCall({
block,
calls: balanceOfCalls,
abi: "erc20:balanceOf"
});
sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
return balances;
}
/*==================================================
@ -108,7 +100,7 @@
module.exports = {
name: 'Opyn',
token: null,
category: 'Derivatives',
category: 'derivatives',
start: 1581542700, // 02/12/2020 @ 09:25PM (UTC)
tvl
}

View File

@ -157,24 +157,13 @@
})
}));
let pmBalances = (await sdk.api.abi.multiCall({
let balanceOfResults = (await sdk.api.abi.multiCall({
block,
calls: pmCalls,
abi: 'erc20:balanceOf'
})).output;
_.each(pmBalances, (result) => {
if (result.success) {
let balance = result.output;
let address = result.input.target;
if (BigNumber(balance).toNumber() <= 0) {
return;
}
balances[address] = BigNumber(balances[address] || 0).plus(balance).toFixed();
}
});
await sdk.util.sumMultiBalanceOf(balances, balanceOfResults);
let opportunityBalances = (await sdk.api.abi.multiCall({
block,
@ -195,7 +184,7 @@
}
});
return (await sdk.api.util.toSymbols(balances)).output;
return balances;
}
/*==================================================
@ -206,7 +195,7 @@
name: 'Robo-Advisor for Yield',
shortName: 'RAY',
token: 'RAY',
category: 'Lending',
category: 'lending',
start: 1568274392, // 09/12/2019 @ 7:46am (UTC)
tvl
}

View File

@ -88,9 +88,7 @@
}
});
let symbolBalances = await sdk.api.util.toSymbols(balances);
return symbolBalances.output;
return balances;
}
/*==================================================
@ -100,7 +98,7 @@
module.exports = {
name: 'Set Protocol',
token: null,
category: 'Assets',
category: 'assets',
start: 1554848955, // 04/09/2019 @ 10:29pm (UTC)
tvl
}

View File

@ -1,60 +0,0 @@
/*==================================================
Modules
==================================================*/
const sdk = require('../../sdk');
const _ = require('underscore');
const BigNumber = require('bignumber.js');
/*==================================================
Settings
==================================================*/
const reserves = [
'0x8E870D67F660D95d5be530380D0eC0bd388289E1', // PAX
'0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48', // USDC
'0x0000000000085d4780B73119b644AE5ecd22b376' // TUSD
]
const poolCore = "0x786bF554473f9aB733Fd683C528212492A23D895" // dForce: Stablecoin Pool
/*==================================================
TVL
==================================================*/
async function tvl(timestamp, block) {
let balances = {};
const calls = _.reduce(reserves, (accum, reserve) => [...accum, {
target: reserve,
params: poolCore
}] , []);
let balanceOfResults = await sdk.api.abi.multiCall({
block,
calls,
abi: 'erc20:balanceOf'
});
_.each(balanceOfResults.output, (balanceOf) => {
if(balanceOf.success) {
let address = balanceOf.input.target
balances[address] = BigNumber(balances[address] || 0).plus(balanceOf.output).toFixed();
}
});
return (await sdk.api.util.toSymbols(balances)).output;
}
/*==================================================
Exports
==================================================*/
module.exports = {
name: 'USDx',
token: 'USDx',
category: 'Assets',
start: 1563991581, // Jul-25-2019 02:06:21 AM +UTC
tvl
}

View File

@ -22,8 +22,10 @@
let point = await sdk.api.util.lookupBlock(timestamp);
await sdk.api.util.resetEthCallCount();
let balances = await project[runFunction](point.timestamp, point.block);
let output = (await sdk.api.util.toSymbols(balances)).output;
let output = await project[runFunction](point.timestamp, point.block);
if(runFunction == 'tvl') {
output = (await sdk.api.util.toSymbols(output)).output;
}
let ethCallCount = await sdk.api.util.getEthCallCount();
return {

View File

@ -20,7 +20,7 @@
return balances;
}
function SumMultiCall(balances, results) {
function SumMultiBalanceOf(balances, results) {
_.each(results.output, (result) => {
if(result.success) {
let address = result.input.target;
@ -41,5 +41,5 @@
module.exports = {
sum: Sum,
sumMultiCall: SumMultiCall
sumMultiBalanceOf: SumMultiBalanceOf
}

View File

@ -19,7 +19,7 @@
const tvlTimeLimit = 1000 * 60 * 5;
const ratesTimeLimit = 30 * 1000;
const symbolLengthLimit = 6;
const ethCallCountLimit = 1000;
const ethCallCountLimit = 2000;
/*==================================================
TestRun

View File

@ -9,11 +9,11 @@
const args = require('../args');
const categories = [
'Derivatives',
'DEXes',
'Lending',
'Payments',
'Assets'
'derivatives',
'dexes',
'lending',
'payments',
'assets'
]
/*==================================================