From baa2e047b4c81223ac5703127038abcc4aa6889b Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Mon, 14 Aug 2023 11:59:14 +0800 Subject: [PATCH 01/24] prices.usd description and sample --- docs/resources/sample-queries.md | 61 ++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 docs/resources/sample-queries.md diff --git a/docs/resources/sample-queries.md b/docs/resources/sample-queries.md new file mode 100644 index 00000000..dc0c627e --- /dev/null +++ b/docs/resources/sample-queries.md @@ -0,0 +1,61 @@ +--- +title: Sample Queries +description: Learn how to write queries using popular datasets +--- + +# Writing Queries Using Popular Datasets + +There are plenty of dataset created that you can use to query what you need. Here are some commonly requested queries using popular datasets + +## prices.usd + +The prices.usd table contains price feed data derived from [Coinpaprika API](https://coinpaprika.com/). +You can also do a pull request in the [here](https://github.com/duneanalytics/spellbook/tree/main/models/prices) to have your token added in prices.usd (Make sure that the token is active on Coinpaprika!) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Column NamesData TypeDescription
contract_addressvarbinarycontract_address of the token
blockchainvarcharblockchain that the token is on
decimalsintnumber of decimals
minutetimestamptimestamp of price feed
pricedoubletoken price
symbolvarchartoken symbol
+ + +```sql +-- getting the token price for WETH for the past 30 days +select * from prices.usd +where contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 +and blockchain = 'ethereum' +and minute >= NOW() - interval '30' days +``` \ No newline at end of file From f0bcc815cb7b73754814468832c89f2223491b8f Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Mon, 14 Aug 2023 21:15:53 +0800 Subject: [PATCH 02/24] add table for erc20 evt transfer --- docs/resources/sample-queries.md | 77 ++++++++++++++++++++++++++++++-- 1 file changed, 74 insertions(+), 3 deletions(-) diff --git a/docs/resources/sample-queries.md b/docs/resources/sample-queries.md index dc0c627e..cd3581b2 100644 --- a/docs/resources/sample-queries.md +++ b/docs/resources/sample-queries.md @@ -12,6 +12,7 @@ There are plenty of dataset created that you can use to query what you need. Her The prices.usd table contains price feed data derived from [Coinpaprika API](https://coinpaprika.com/). You can also do a pull request in the [here](https://github.com/duneanalytics/spellbook/tree/main/models/prices) to have your token added in prices.usd (Make sure that the token is active on Coinpaprika!) +
@@ -50,12 +51,82 @@ You can also do a pull request in the [here](https://github.com/duneanalytics/sp
Column Namestoken symbol
- +
```sql -- getting the token price for WETH for the past 30 days select * from prices.usd where contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 and blockchain = 'ethereum' -and minute >= NOW() - interval '30' days -``` \ No newline at end of file +and minute >= NOW() - interval '30' day +``` + +## prices.usd_latest + +Just as the table name suggest, this table contains the latest price available, derived from prices.usd + +```sql +-- getting the latest price for 4 tokens +select * from prices.usd_latest +where blockchain = 'ethereum' +and symbol IN ('WETH','WBTC','USDC','USDT') +``` + +## erc20_ethereum.evt_transfer + +The erc20_ethereum.evt_transfer contains all erc20 token transfer events that occured. With this table, you will be able to get information such as number of token holders,token balances and many more! This table is available for other EVM chain too, just change the suffix to the chain desired(eg.erc20_arbitrum.evt_transfer,erc20_bnb.evt_transfer) + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Column NamesData TypeDescription
contract_addressvarbinaryerc20 token contract address
fromvarbinaryAddress tokens are transferred from
tovarbinaryAddress tokens are transferred to
valueuint256Number of tokens transferred
evt_block_numberbigintBlock number in which the event occurred
evt_block_timetimestampTimestamp of the event occurrence
evt_indexbigintIndex of the event within the block
evt_tx_hashvarbinaryTransaction hash of the event
+
+ + From a50765256f72a226da395ee0cc4c83660ebf40b6 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 01:29:39 +0800 Subject: [PATCH 03/24] add balances example --- docs/resources/sample-queries.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/resources/sample-queries.md b/docs/resources/sample-queries.md index cd3581b2..4599b3a2 100644 --- a/docs/resources/sample-queries.md +++ b/docs/resources/sample-queries.md @@ -129,4 +129,28 @@ The erc20_ethereum.evt_transfer contains all erc20 token transfer events that oc +```sql +-- get total amount of weth sent daily in the past 3 days +SELECT DATE_TRUNC('day', evt_block_time) AS date, + SUM(value / 1e18) AS daily_amount +FROM erc20_ethereum.evt_transfer +WHERE contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 +AND evt_block_time >= NOW() - INTERVAL '3' day +GROUP BY 1 +``` + +You can also get your wallet erc20 token balances! There is a evms.erc20_transfers table that contains all evms erc20 token transfer event + +```sql +-- getting erc20 token balances for 0xbe0eb53f46cd790cd13851d5eff43d12404d33e8 +SELECT * FROM ( +select blockchain,symbol,SUM(CASE WHEN "from" = 0xbe0eb53f46cd790cd13851d5eff43d12404d33e8 THEN -(value/POW(10,decimals)) ELSE (value/POW(10,decimals)) END) AS token_balance +from evms.erc20_transfers JOIN tokens.erc20 USING (contract_address,blockchain) +where ("from" = 0xbe0eb53f46cd790cd13851d5eff43d12404d33e8 OR "to" = 0xbe0eb53f46cd790cd13851d5eff43d12404d33e8) +GROUP BY 1,2 +) +WHERE token_balance > 0 +``` + + From 9591f45c3032c262d27d478e7324b37bf034a08a Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 01:50:56 +0800 Subject: [PATCH 04/24] update prices --- .../spellbook/top-tables/prices.md | 51 +++++-- docs/resources/sample-queries.md | 128 ++++++++++++++++++ 2 files changed, 169 insertions(+), 10 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index 1c85bc51..11646972 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -2,17 +2,14 @@ title: prices description: These tables allow you to get the price of almost all relevant erc20 tokens. --- -We pull price data from the [coinpaprika](https://coinpaprika.com) API. +The price feeds are derived from [Coinpaprika](https://coinpaprika.com/)'s API. The price is the volume-weighted price based on real-time market data, translated to USD. -The Price is the volume-weighted price based on real-time market data, translated to USD. ## prices.usd -This table supports a range of erc20.tokens. - ### adding a token to price tracking -If the token you desire is not listed in here, please make a pull request to our [GitHub repository](https://github.com/duneanalytics/spellbook/blob/main/models/prices/prices_tokens.sql). (For V1 Engine, you can also use the decentralized price feed **dex.view_token_prices.**) +If the token has yet to be listed in here, please make a pull request to our [GitHub repository](https://github.com/duneanalytics/spellbook/blob/main/models/prices/prices_tokens.sql)(Make sure that the token is active on Coinpaprika!). | Column name | Data type | Description | | - | :-: | - | @@ -23,6 +20,25 @@ If the token you desire is not listed in here, please make a pull request to our Note that `WETH` can be used for ETH price as it trades at virtually the same price. +```sql +-- getting the token price for WETH for the past 30 days +select * from prices.usd +where contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 +and blockchain = 'ethereum' +and minute >= NOW() - interval '30' day +``` + +## prices.usd_latest + +Just as the table name suggest, this table contains the latest price available, derived from prices.usd + +```sql +-- getting the latest price for 4 tokens +select * from prices.usd_latest +where blockchain = 'ethereum' +and symbol IN ('WETH','WBTC','USDC','USDT') +``` + ## How we get prices from DEXs We created a table that creates price feeds based on decentralized exchange trading data. This table covers much more assets than `prices.usd`, since it covers all assets that are traded on any of the decentralized exchanges that are indexed in `dex.trades`. @@ -33,7 +49,7 @@ This table is very resource intensive and can therefore only be updated every fe This table currently only exists for Ethereum on our old database architecture. -The logic of how this table works can be accessed in our [public github](https://github.com/duneanalytics/spellbook/tree/master/ethereum/prices) repo. +The logic of how this table works can be accessed in our [public github](https://github.com/duneanalytics/spellbook/blob/main/models/dex/dex_prices.sql) repo. This script generates median hourly prices based on data from decentralized exchanges found in `dex.trades`. It will assign asset prices based on a trading pair which has a pricefeed in `prices.usd`. @@ -60,16 +76,31 @@ We now have successfully calculated the price of 1 $SPELL. In order to correct for extreme outliers and in order for this table to be performant the script then aggregates all recorded data into one `median_price` per hour. +```sql +-- get daily average price of weth from dex.prices +select date_trunc('day',hour) as date, + contract_address, + AVG(median_price) as avg_price +from dex.prices +WHERE blockchain = 'ethereum' +AND contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 +AND hour >= NOW() - interval '3' day +GROUP BY 1,2 +``` + ### Known issues In rare cases this script will generate price feeds that are based on illiquid pairs and therefore report wrong data. This happens when all liquid trading pools of this token do not have a price feed in `prices.usd`. -An example of this would be $PLAY, a metaverse index from PieDAO. The liquid trading pair for this asset is $PLAY/$DOUGH. The "correct" price of $PLAY is represented in this pool, but the combination of `dex.trades` and `prices.prices_from_dex_data` are not able to pick up this price. +An example of this would be $PLAY, a metaverse index from PieDAO. The liquid trading pair for this asset is $PLAY/$DOUGH. The "correct" price of $PLAY is represented in this pool, but the combination of `dex.trades` and `dex.prices` are not able to pick up this price. -Instead, `dex.trades` will only have a `usd_amount` for illiquid pairs of this asset. In this case, the $PLAY/$ETH pool has trades once in a while and these will have a `usd_amount` in `dex.trades`. The liquidity of the $PLAY/$ETH pool is very low and it pretty much only consists of arbitrage trades. Therefore, the resulting pricefeed in `prices.prices_from_dex_data` is faulty since it depends on the `usd_amount` in `dex.trades`. +Instead, `dex.trades` will only have a `usd_amount` for illiquid pairs of this asset. In this case, the $PLAY/$ETH pool has trades once in a while and these will have a `usd_amount` in `dex.trades`. The liquidity of the $PLAY/$ETH pool is very low and it pretty much only consists of arbitrage trades. Therefore, the resulting pricefeed in `dex.prices` is faulty since it depends on the `usd_amount` in `dex.trades`. -In order to check for this, you should manually verify the results of `prices.prices_from_dex_data` in order to make sure arbitrage trades do not disturb the price feed constructed. A simple way of validating that the script is working with the right pools is checking the `sample_size` column. If the number seems suspiciously low, the script probably doesn't pick up the right price. +In order to check for this, you should manually verify the results of `dex.prices` in order to make sure arbitrage trades do not disturb the price feed constructed. A simple way of validating that the script is working with the right pools is checking the `sample_size` column. If the number seems suspiciously low, the script probably doesn't pick up the right price. In cases like this, you have to manually construct a price feed. -We are always looking to improve this table, if you have any ideas or comments don't hesitate to open a PR or contact us in our Discord. \ No newline at end of file +We are always looking to improve this table, if you have any ideas or comments don't hesitate to open a PR or contact us in our Discord. + + + diff --git a/docs/resources/sample-queries.md b/docs/resources/sample-queries.md index 4599b3a2..b8bda6ac 100644 --- a/docs/resources/sample-queries.md +++ b/docs/resources/sample-queries.md @@ -152,5 +152,133 @@ GROUP BY 1,2 WHERE token_balance > 0 ``` +## dex.trades + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Column NamesData TypeDescription
amount_usddoubleEquivalent amount in USD
block_datetimestampDate of the block
block_timetimestampTime of the block
blockchainvarcharBlockchain identifier
evt_indexintIndex of the event within the block
makervarbinaryAddress of the maker
projectvarcharProject name
project_contract_addressvarbinaryContract address of the project
takervarbinaryAddress of the taker
token_bought_addressvarbinaryAddress of the token bought
token_bought_amountdoubleAmount of token bought
token_bought_amount_rawdecimal(38,0)Raw amount of token bought
token_bought_symbolvarcharSymbol of token bought
token_pairvarcharToken pair identifier
token_sold_addressvarbinaryAddress of the token sold
token_sold_amountdoubleAmount of token sold
token_sold_amount_rawdecimal(38,0)Raw amount of token sold
token_sold_symbolvarcharSymbol of token sold
trace_addressvarcharTrace address
tx_fromvarbinaryTransaction sender address
tx_hashvarbinaryTransaction hash
tx_tovarbinaryTransaction receiver address
versionvarcharVersion of the trade
+
+ + From e8833d3f10c761ea4f1d4e363b4dc1286910c161 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 14:59:19 +0800 Subject: [PATCH 05/24] update dex.trades column data --- .../spellbook/top-tables/dex.trades.md | 81 +++++++++++++------ docs/resources/sample-queries.md | 2 + 2 files changed, 59 insertions(+), 24 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/dex.trades.md b/docs/data-tables/spellbook/top-tables/dex.trades.md index d6dbc002..eef8d6bb 100644 --- a/docs/data-tables/spellbook/top-tables/dex.trades.md +++ b/docs/data-tables/spellbook/top-tables/dex.trades.md @@ -15,27 +15,60 @@ The scripts that generate the table dex.trades can be found in this [public gith ## Column Data -| Column name | Data type | Description | -| - | :-: | - | -| `block_time` | _timestamptz_ | The timestamp of the block that included this transaction | -| `token_a_symbol` | _varchar_ | The symbol of one of the two tokens that got traded | -| `token_b_symbol` | _varchar_ | The symbol of one of the two tokens that got traded | -| `token_a_amount` | _numeric_ | The amount of token A that got traded | -| `token_b_amount` | _numeric_ | The amount of token B that got traded | -| `project` | _varchar_ | The dex on which this trade was executed | -| `version` | _varchar_ | Which version of the dex got used? | -| `blockchain` | _varchar_ | Which blockchain did this occur on | -| `taker` | _varbinary_ | Which contract called the dex contract? | -| `maker` | _varbinary_ | In some special cases there actually is a counter party to transactions, this party will get displayed here if applicable | -| `token_a_amount_raw` | _numeric_ | The raw amount of token A that got traded | -| `token_b_amount_raw` | _numeric_ | The raw amount of token B that got traded | -| `amount_usd` | _numeric_ | The USD value of this trade | -| `token_a_address` | _varbinary_ | The ERC-20 token contract address of token A | -| `token_b_address` | _varbinary_ | The ERC-20 token contract address of token B | -| `exchange_contract_address` | _varbinary_ | The address of the decentralized exchange contract that made this trade possible | -| `tx_hash` | _varbinary_ | The hash of the transaction that contained this trade | -| `tx_from` | _varbinary_ | Which address initiated this transaction? | -| `tx_to` | _varbinary_ | What was the first smart contract that got called during this tx? | -| `trace_address` | _ARRAY_ | Which position in the graph tree does the execution of the trade have? | -| `evt_index` | _integer_ | This logs index position in the block (cumulative amount of logs ordered by execution) | -| `trade_id` | _integer_ | Just for database magic | + +Certainly! Here's the markup code with properly aligned columns: + +markdown +Copy code +| Column Name | Data Type | Description | +|-------------------------|-------------------|------------------------------------------------------------------------| +| `amount_usd` | _double_ | The USD value of this trade. | +| `block_date` | _timestamp_ | The truncated timestamp of the block including this transaction. | +| `block_time` | _timestamp_ | The timestamp of the block including this transaction. | +| `blockchain` | _varchar_ | The blockchain where this transaction was broadcasted. | +| `evt_index` | _int_ | This log's index position in the block (cumulative amount of logs ordered by execution). | +| `maker` | _varbinary_ | In some special cases, the counterparty to transactions (if applicable). | +| `project` | _varchar_ | The decentralized exchange (DEX) on which this trade was executed. | +| `project_contract_address` | _varbinary_ | The contract address of the project or DEX. | +| `taker` | _varbinary_ | The contract that called the DEX contract. | +| `token_bought_address` | _varbinary_ | The address of the token bought during this trade. | +| `token_bought_amount` | _double_ | The amount of the token bought during this trade. | +| `token_bought_amount_raw` | _decimal(38,0)_ | The raw amount of the token bought during this trade. | +| `token_bought_symbol` | _varchar_ | The symbol of the token bought. | +| `token_pair` | _varchar_ | The trading pair of tokens involved in this trade. | +| `token_sold_address` | _varbinary_ | The address of the token sold during this trade. | +| `token_sold_amount` | _double_ | The amount of the token sold during this trade. | +| `token_sold_amount_raw` | _decimal(38,0)_ | The raw amount of the token sold during this trade. | +| `token_sold_symbol` | _varchar_ | The symbol of the token sold. | +| `trace_address` | _varchar_ | The position in the execution graph tree for this trade. | +| `tx_from` | _varbinary_ | The address that initiated this transaction. | +| `tx_hash` | _varbinary_ | The hash of the transaction containing this trade. | +| `tx_to` | _varbinary_ | The address of the first smart contract called during this transaction. | +| `version` | _varchar_ | The version of the DEX used for this trade. | + +#### Get all transactions of USDC swaps in the past 24 hours + +```sql +select * from dex.trades +where (token_bought_address = 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 +OR token_sold_address = 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48) +AND blockchain = 'ethereum' +AND block_time >= NOW() - interval '24' hour +``` + +#### Get top 100 uniswap pairs' volume from uniswap in the past 3 days + +```sql +select token_pair, + SUM(amount_usd) as total_volume +from dex.trades +WHERE blockchain = 'ethereum' +AND project = 'uniswap' +AND block_time >= NOW() - interval '3' day +AND token_pair IS NOT NULL +GROUP BY 1 +ORDER BY 2 DESC -- order by total_volume +limit 100 -- 100 rows +``` + + diff --git a/docs/resources/sample-queries.md b/docs/resources/sample-queries.md index b8bda6ac..323b742e 100644 --- a/docs/resources/sample-queries.md +++ b/docs/resources/sample-queries.md @@ -154,6 +154,8 @@ WHERE token_balance > 0 ## dex.trades +The dex.trades table contains transfer events that occured. With this table, you will be able to get information such as number of token holders,token balances and many more! This table is available for other EVM chain too, just change the suffix to the chain desired(eg.erc20_arbitrum.evt_transfer,erc20_bnb.evt_transfer) +
From 76f7301aa82f9c39f59cc5fe4b2f24368674c9bb Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 15:03:02 +0800 Subject: [PATCH 06/24] update prices usd column --- .../spellbook/top-tables/dex.trades.md | 5 ----- docs/data-tables/spellbook/top-tables/prices.md | 15 +++++++++------ 2 files changed, 9 insertions(+), 11 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/dex.trades.md b/docs/data-tables/spellbook/top-tables/dex.trades.md index eef8d6bb..f3437514 100644 --- a/docs/data-tables/spellbook/top-tables/dex.trades.md +++ b/docs/data-tables/spellbook/top-tables/dex.trades.md @@ -15,11 +15,6 @@ The scripts that generate the table dex.trades can be found in this [public gith ## Column Data - -Certainly! Here's the markup code with properly aligned columns: - -markdown -Copy code | Column Name | Data Type | Description | |-------------------------|-------------------|------------------------------------------------------------------------| | `amount_usd` | _double_ | The USD value of this trade. | diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index 11646972..23787721 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -11,12 +11,15 @@ The price feeds are derived from [Coinpaprika](https://coinpaprika.com/)'s API. If the token has yet to be listed in here, please make a pull request to our [GitHub repository](https://github.com/duneanalytics/spellbook/blob/main/models/prices/prices_tokens.sql)(Make sure that the token is active on Coinpaprika!). -| Column name | Data type | Description | -| - | :-: | - | -| `contract_address`| _varbinary_ |string the contract address of the erc20 token | -| `symbol` | _varchar_ |the identifier of the asset (ticker, cashtag) | -| `price` | _bigint_ | The price of the asset in any given minute | -| `minute` | _timestampz_ | The resolution for this table is by minute | +| Column Name | Data Type | Description | +|-------------------|---------------|-------------------------------------------------------------| +| `contract_address`| `varbinary` | The contract address of the ERC20 token. | +| `blockchain` | `varchar` | The blockchain associated with the ERC20 token. | +| `decimals` | `int` | The number of decimal places for the token's value. | +| `minute` | `timestamp` | The resolution of this data table in minutes. | +| `price` | `double` | The price of the asset in any given minute. | +| `symbol` | `varchar` | The identifier of the asset (ticker) | + Note that `WETH` can be used for ETH price as it trades at virtually the same price. From 791341ef978b9959077c9fcedb0d7118326a60ba Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 15:48:13 +0800 Subject: [PATCH 07/24] update data table --- .../spellbook/top-tables/dex.trades.md | 50 +++++++++---------- .../spellbook/top-tables/prices.md | 10 ++-- .../spellbook/top-tables/tokens.md | 7 +++ 3 files changed, 37 insertions(+), 30 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/dex.trades.md b/docs/data-tables/spellbook/top-tables/dex.trades.md index f3437514..ee1b394a 100644 --- a/docs/data-tables/spellbook/top-tables/dex.trades.md +++ b/docs/data-tables/spellbook/top-tables/dex.trades.md @@ -15,31 +15,31 @@ The scripts that generate the table dex.trades can be found in this [public gith ## Column Data -| Column Name | Data Type | Description | -|-------------------------|-------------------|------------------------------------------------------------------------| -| `amount_usd` | _double_ | The USD value of this trade. | -| `block_date` | _timestamp_ | The truncated timestamp of the block including this transaction. | -| `block_time` | _timestamp_ | The timestamp of the block including this transaction. | -| `blockchain` | _varchar_ | The blockchain where this transaction was broadcasted. | -| `evt_index` | _int_ | This log's index position in the block (cumulative amount of logs ordered by execution). | -| `maker` | _varbinary_ | In some special cases, the counterparty to transactions (if applicable). | -| `project` | _varchar_ | The decentralized exchange (DEX) on which this trade was executed. | -| `project_contract_address` | _varbinary_ | The contract address of the project or DEX. | -| `taker` | _varbinary_ | The contract that called the DEX contract. | -| `token_bought_address` | _varbinary_ | The address of the token bought during this trade. | -| `token_bought_amount` | _double_ | The amount of the token bought during this trade. | -| `token_bought_amount_raw` | _decimal(38,0)_ | The raw amount of the token bought during this trade. | -| `token_bought_symbol` | _varchar_ | The symbol of the token bought. | -| `token_pair` | _varchar_ | The trading pair of tokens involved in this trade. | -| `token_sold_address` | _varbinary_ | The address of the token sold during this trade. | -| `token_sold_amount` | _double_ | The amount of the token sold during this trade. | -| `token_sold_amount_raw` | _decimal(38,0)_ | The raw amount of the token sold during this trade. | -| `token_sold_symbol` | _varchar_ | The symbol of the token sold. | -| `trace_address` | _varchar_ | The position in the execution graph tree for this trade. | -| `tx_from` | _varbinary_ | The address that initiated this transaction. | -| `tx_hash` | _varbinary_ | The hash of the transaction containing this trade. | -| `tx_to` | _varbinary_ | The address of the first smart contract called during this transaction. | -| `version` | _varchar_ | The version of the DEX used for this trade. | +| Column Name | Data Type | Description | +|-------------------------|-------------------|----------------------------------------------------------------------- | +| `amount_usd` | _double_ | The USD value of this trade | +| `block_date` | _timestamp_ | The truncated timestamp of the block including this transaction | +| `block_time` | _timestamp_ | The timestamp of the block including this transaction | +| `blockchain` | _varchar_ | The blockchain where this transaction was broadcasted | +| `evt_index` | _int_ | This log's index position in the block (cumulative amount of logs ordered by execution) | +| `maker` | _varbinary_ | In some special cases, the counterparty to transactions (if applicable) | +| `project` | _varchar_ | The decentralized exchange (DEX) on which this trade was executed | +| `project_contract_address` | _varbinary_ | The contract address of the project or DEX | +| `taker` | _varbinary_ | The contract that called the DEX contract | +| `token_bought_address` | _varbinary_ | The address of the token bought during this trade | +| `token_bought_amount` | _double_ | The amount of the token bought during this trade | +| `token_bought_amount_raw` | _decimal(38,0)_ | The raw amount of the token bought during this trade | +| `token_bought_symbol` | _varchar_ | The symbol of the token bought | +| `token_pair` | _varchar_ | The trading pair of tokens involved in this trade | +| `token_sold_address` | _varbinary_ | The address of the token sold during this trade | +| `token_sold_amount` | _double_ | The amount of the token sold during this trade | +| `token_sold_amount_raw` | _decimal(38,0)_ | The raw amount of the token sold during this trade | +| `token_sold_symbol` | _varchar_ | The symbol of the token sold | +| `trace_address` | _varchar_ | The position in the execution graph tree for this trade | +| `tx_from` | _varbinary_ | The address that initiated this transaction | +| `tx_hash` | _varbinary_ | The hash of the transaction containing this trade | +| `tx_to` | _varbinary_ | The address of the first smart contract called during this transaction | +| `version` | _varchar_ | The version of the DEX used for this trade | #### Get all transactions of USDC swaps in the past 24 hours diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index 23787721..e567e1f8 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -13,11 +13,11 @@ If the token has yet to be listed in here, please make a pull request to our [Gi | Column Name | Data Type | Description | |-------------------|---------------|-------------------------------------------------------------| -| `contract_address`| `varbinary` | The contract address of the ERC20 token. | -| `blockchain` | `varchar` | The blockchain associated with the ERC20 token. | -| `decimals` | `int` | The number of decimal places for the token's value. | -| `minute` | `timestamp` | The resolution of this data table in minutes. | -| `price` | `double` | The price of the asset in any given minute. | +| `contract_address`| `varbinary` | The contract address of the ERC20 token | +| `blockchain` | `varchar` | The blockchain associated with the ERC20 token | +| `decimals` | `int` | The number of decimal places for the token's value | +| `minute` | `timestamp` | The resolution of this data table in minutes | +| `price` | `double` | The price of the asset in any given minute | | `symbol` | `varchar` | The identifier of the asset (ticker) | diff --git a/docs/data-tables/spellbook/top-tables/tokens.md b/docs/data-tables/spellbook/top-tables/tokens.md index b9bf45d3..aeba5648 100644 --- a/docs/data-tables/spellbook/top-tables/tokens.md +++ b/docs/data-tables/spellbook/top-tables/tokens.md @@ -9,6 +9,13 @@ You'll likely be working with tokens that are fungible (erc20) and nonfungible ( 1. [**`tokens.erc20`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.tokens_erc20): contains useful information such as the token `symbol` and the `decimals` for any given `contract_address`, the latter of which is needed to get the actual amount from raw amounts in onchain data. +| Column Name | Data Type | Description | +|-------------------|-------------|--------------------------------------------------------| +| `blockchain` | _varchar_ | The blockchain associated with the ERC20 token | +| `contract_address`| _varbinary_ | The contract address of the ERC20 token | +| `decimals` | _int_ | The number of decimal places for the token's value | +| `symbol` | _varchar_ | The identifier of the asset (ticker) | + 2. [**`tokens.nft`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.tokens_nft): contains the collection `name` and `symbol` for any given `contract_address`. These tables are usually joined on `contract_address` at the end of a query to make everything more human readable. From b4a0ba7000917ffc4b227ca0661f3cfad650bd31 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 17:31:18 +0800 Subject: [PATCH 08/24] update --- .../spellbook/top-tables/tokens.md | 87 ++++++++++++++++++- 1 file changed, 85 insertions(+), 2 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/tokens.md b/docs/data-tables/spellbook/top-tables/tokens.md index aeba5648..e4cc706c 100644 --- a/docs/data-tables/spellbook/top-tables/tokens.md +++ b/docs/data-tables/spellbook/top-tables/tokens.md @@ -16,15 +16,98 @@ You'll likely be working with tokens that are fungible (erc20) and nonfungible ( | `decimals` | _int_ | The number of decimal places for the token's value | | `symbol` | _varchar_ | The identifier of the asset (ticker) | + +```sql +-- querying using ethereum's weth contract_address +select * from tokens.erc20 +WHERE contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 +``` + 2. [**`tokens.nft`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.tokens_nft): contains the collection `name` and `symbol` for any given `contract_address`. These tables are usually joined on `contract_address` at the end of a query to make everything more human readable. +```sql +-- querying using bayc contract +select * from tokens.nft +where contract_address = 0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d +``` + ## Transfer tables: -1. [**`erc20_ethereum.evt_Transfer`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.transfers_ethereum_erc20): all transfer events for every erc20 token. You can find how to get erc20 balances, mints, and burns using [this guide](https://www.youtube.com/watch?v=LT_PB-Fso3M). +1. [**`erc20_ethereum.evt_Transfer`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.transfers_ethereum_erc20): all transfer events for every erc20 token. + +| Column Name | Data Type | Description | +|--------------------|------------------|---------------------------------------------------| +| `contract_address` | _varbinary_ | The contract address. | +| `from` | _varbinary_ | The sender's address. | +| `value` | _uint256_ | The value of the transaction. | +| `evt_block_number` | _bigint_ | The block number of the event. | +| `evt_block_time` | _timestamp_ | The timestamp of the event's block. | +| `evt_index` | _bigint_ | The index of the event within the block. | +| `evt_tx_hash` | _varbinary_ | The transaction hash of the event. | + +Finding out the total inflow and outflow of USDT token from an address in the past 3 days + +```sql +SELECT CASE WHEN "from" = 0xdfd5293d8e347dfe59e90efd55b2956a1343963d THEN 'Outflow' ELSE 'Inflow' END AS token_flow, + SUM(value/POW(10,decimals)) as total_amount +FROM erc20_ethereum.evt_transfer a JOIN tokens.erc20 b ON a.contract_address = b.contract_address +WHERE a.contract_address = 0xdac17f958d2ee523a2206206994597c13d831ec7 +AND ("from" = 0xdfd5293d8e347dfe59e90efd55b2956a1343963d OR "to" = 0xdfd5293d8e347dfe59e90efd55b2956a1343963d) +AND evt_block_time >= NOW() - interval '3' day +GROUP BY 1 +``` + +Get all holders with their ens and balances of an erc20 token + +```sql +SELECT address, + symbol, + ARRAY_AGG(distinct name) as ens, + contract_address, + SUM(amount) as balance +FROM ( +SELECT tr."from" AS address, + symbol, + contract_address, + -(tr.value/POW(10,decimals)) AS amount +FROM erc20_ethereum.evt_transfer tr JOIN tokens.erc20 USING (contract_address) +WHERE "from" != 0x0000000000000000000000000000000000000000 -- exclude mint/burn addresses +AND contract_address = 0x046eee2cc3188071c02bfc1745a6b17c656e3f3d -- contract_address of the erc20 token +-- AND evt_block_time <= TIMESTAMP '2023-01-01' -- you can use this to get snapshot data +UNION ALL +SELECT tr."to" AS address, + symbol, + contract_address, + (tr.value/POW(10,decimals)) AS amount +FROM erc20_ethereum.evt_transfer tr JOIN tokens.erc20 USING (contract_address) + WHERE "to" != 0x0000000000000000000000000000000000000000 -- exclude mint/burn addresses + AND contract_address = 0x046eee2cc3188071c02bfc1745a6b17c656e3f3d -- contract_address of the erc20 token +-- AND evt_block_time <= TIMESTAMP '2023-01-01' -- you can use this to get snapshot data + ) x LEFT JOIN labels.ens USING (address) + GROUP BY 1,2,4 + HAVING SUM(amount) > 0.1 -- having more than 0.1 balance + ORDER BY 5 DESC +``` + +You can find how to get erc20 balances, mints, and burns using [this guide](https://www.youtube.com/watch?v=LT_PB-Fso3M). + +2. [**`nft.transfers`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.nft_transfers): all transfer events for every erc721 or erc1155 token. + +| Column Name | Data Type | Description | +|-------------------|-------------|---------------------------------------------------| +| `contract_address`| _varbinary_ | The contract address | +| `from` | _varbinary_ | The sender's address | +| `to` | _varbinary_ | The recipient's address | +| `tokenId` | _uint256_ | The token ID of the transaction | +| `evt_block_number`| _bigint_ | The block number of the event | +| `evt_block_time` | _timestamp_ | The timestamp of the event's block | +| `evt_index` | _bigint_ | The index of the event within the block | +| `evt_tx_hash` | _varbinary_ | The transaction hash of the event | + -2. [**`nft.transfers`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.nft_transfers): all transfer events for every erc721 or erc1155 token. You can learn how to leverage this to find nft balances, transfers, and mints in [this guide](https://web3datadegens.substack.com/p/web3-sql-weekly-3-finding-all-nfts) +You can learn how to leverage this to find nft balances, transfers, and mints in [this guide](https://web3datadegens.substack.com/p/web3-sql-weekly-3-finding-all-nfts) If you're looking for how to calculate native token balances like ethereum (ETH) balances then check out [this guide](https://web3datadegens.substack.com/p/web3-sql-weekly-1-how-to-calculate) From 1c1ee6f1588aa943a0b9bfce3cbfab8718b44dde Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 20:28:05 +0800 Subject: [PATCH 09/24] add erc721 example --- .../spellbook/top-tables/tokens.md | 51 +++++++++++++++++-- 1 file changed, 48 insertions(+), 3 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/tokens.md b/docs/data-tables/spellbook/top-tables/tokens.md index e4cc706c..8152c510 100644 --- a/docs/data-tables/spellbook/top-tables/tokens.md +++ b/docs/data-tables/spellbook/top-tables/tokens.md @@ -59,7 +59,7 @@ AND evt_block_time >= NOW() - interval '3' day GROUP BY 1 ``` -Get all holders with their ens and balances of an erc20 token +Get all holders with their ens and balances of an erc20 token contract ```sql SELECT address, @@ -91,6 +91,8 @@ FROM erc20_ethereum.evt_transfer tr JOIN tokens.erc20 USING (contract_address) ORDER BY 5 DESC ``` +If you're looking for how to calculate native token balances like ethereum (ETH) balances then check out [this guide](https://web3datadegens.substack.com/p/web3-sql-weekly-1-how-to-calculate) + You can find how to get erc20 balances, mints, and burns using [this guide](https://www.youtube.com/watch?v=LT_PB-Fso3M). 2. [**`nft.transfers`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.nft_transfers): all transfer events for every erc721 or erc1155 token. @@ -106,8 +108,51 @@ You can find how to get erc20 balances, mints, and burns using [this guide](http | `evt_index` | _bigint_ | The index of the event within the block | | `evt_tx_hash` | _varbinary_ | The transaction hash of the event | +Get all erc721 token transfers to an addresses in the past 30 days -You can learn how to leverage this to find nft balances, transfers, and mints in [this guide](https://web3datadegens.substack.com/p/web3-sql-weekly-3-finding-all-nfts) +```sql +SELECT date_trunc('day',evt_block_time) as date, + COUNT(*) as transfer_count +FROM erc721_ethereum.evt_transfer +where to = 0x6ce82874eaf6e7602fd21cf8bbded82705680a99 +AND evt_block_time >= NOW() - interval '30' day +GROUP BY 1 +order by 1 desc +``` -If you're looking for how to calculate native token balances like ethereum (ETH) balances then check out [this guide](https://web3datadegens.substack.com/p/web3-sql-weekly-1-how-to-calculate) +Get all holders with their ens,labels and balances of an erc721 token contract + +```sql +SELECT z.name, + contract_address, + address, + a.name as ens_name, + ARRAY_AGG(DISTINCT b.name) as labels_list, + nft_list, + number_of_nft_held, + total_nft_supply +FROM ( +SELECT contract_address, + to as address, + ARRAY_AGG(tokenId) as nft_list, + COUNT(*) as number_of_nft_held, + SUM(COUNT(*)) OVER () as total_nft_supply +FROM ( +select contract_address, + to, + tokenId, + ROW_NUMBER() OVER (PARTITION BY contract_address,tokenId ORDER BY evt_block_time DESC,evt_index DESC) as rn +from erc721_ethereum.evt_transfer +where contract_address = 0xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D +) x +WHERE rn = 1 +GROUP BY 1,2 +) p +LEFT JOIN tokens.nft z USING (contract_address) +LEFT JOIN ens.reverse_latest a USING (address) +LEFT JOIN labels.addresses b USING (address) +GROUP BY 1,2,3,4,6,7,8 +ORDER BY 7 DESC +``` +You can learn how to leverage this to find nft balances, transfers, and mints in [this guide](https://web3datadegens.substack.com/p/web3-sql-weekly-3-finding-all-nfts) \ No newline at end of file From 6ebf5da8d34f19463a36b29fa5686b79dfbbe8be Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 20:53:40 +0800 Subject: [PATCH 10/24] remove --- docs/data-tables/spellbook/top-tables/prices.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index e567e1f8..d3a78ad6 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -48,9 +48,7 @@ We created a table that creates price feeds based on decentralized exchange trad **Please keep in mind that this script can generate wrong prices in rare cases.** -This table is very resource intensive and can therefore only be updated every few hours, please keep that in mind when utilizing it. **** Also the resolution is only hourly, so if you need minutely prices do refer to [`prices.usd`](prices.md). - -This table currently only exists for Ethereum on our old database architecture. +This table is very resource intensive and can therefore only be updated every few hours, please keep that in mind when utilizing it. Also the resolution is only hourly, so if you need minutely prices do refer to [`prices.usd`](prices.md). The logic of how this table works can be accessed in our [public github](https://github.com/duneanalytics/spellbook/blob/main/models/dex/dex_prices.sql) repo. From fd0e07776188ee545cea57b2b3e414f98aec7f27 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 21:36:32 +0800 Subject: [PATCH 11/24] Update dex.trades.md --- .../spellbook/top-tables/dex.trades.md | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/dex.trades.md b/docs/data-tables/spellbook/top-tables/dex.trades.md index ee1b394a..c4b88f70 100644 --- a/docs/data-tables/spellbook/top-tables/dex.trades.md +++ b/docs/data-tables/spellbook/top-tables/dex.trades.md @@ -41,7 +41,7 @@ The scripts that generate the table dex.trades can be found in this [public gith | `tx_to` | _varbinary_ | The address of the first smart contract called during this transaction | | `version` | _varchar_ | The version of the DEX used for this trade | -#### Get all transactions of USDC swaps in the past 24 hours +Get all transactions of USDC swaps in the past 24 hours ```sql select * from dex.trades @@ -51,7 +51,7 @@ AND blockchain = 'ethereum' AND block_time >= NOW() - interval '24' hour ``` -#### Get top 100 uniswap pairs' volume from uniswap in the past 3 days +Get top 100 uniswap pairs' volume from uniswap in the past 3 days ```sql select token_pair, @@ -66,4 +66,22 @@ ORDER BY 2 DESC -- order by total_volume limit 100 -- 100 rows ``` +Get top 100 traders' volume of erc20 token using dex.trades + +```sql +SELECT tx_from as address, + SUM(amount_usd) as total_volume, + SUM(CASE WHEN "token_bought_address" = 0x2260fac5e5542a773aa44fbcfedf7c193bc2c599 THEN 1 END) as buy_count, + SUM(CASE WHEN "token_sold_address" = 0x2260fac5e5542a773aa44fbcfedf7c193bc2c599 THEN 1 END) as sell_count, + SUM(CASE WHEN "token_bought_address" = 0x2260fac5e5542a773aa44fbcfedf7c193bc2c599 THEN amount_usd END) as total_buy_volume, + SUM(CASE WHEN "token_sold_address" = 0x2260fac5e5542a773aa44fbcfedf7c193bc2c599 THEN amount_usd END) as total_sell_volume +FROM dex.trades +WHERE ("token_bought_address" = 0x2260fac5e5542a773aa44fbcfedf7c193bc2c599 OR "token_sold_address" = 0x2260fac5e5542a773aa44fbcfedf7c193bc2c599) +AND blockchain = 'ethereum' +AND block_time >= NOW() - interval '7' day +GROUP BY 1 +ORDER BY 2 DESC +limit 100 +``` + From 23421e73330f6c205af66d619161a4605a203b64 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 22:42:31 +0800 Subject: [PATCH 12/24] update nft --- .../spellbook/top-tables/nft.trades.md | 94 ++++++++----------- .../spellbook/top-tables/prices.md | 4 +- 2 files changed, 42 insertions(+), 56 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/nft.trades.md b/docs/data-tables/spellbook/top-tables/nft.trades.md index d109bf6f..7e99ab91 100644 --- a/docs/data-tables/spellbook/top-tables/nft.trades.md +++ b/docs/data-tables/spellbook/top-tables/nft.trades.md @@ -9,14 +9,37 @@ The culmination of this is a dataset which makes it extremely easy to query for You can find the specifications for nft.trades on our [Spellbook documentation](https://dune.com/spellbook#!/model/model.spellbook.nft_trades). -So far we have indexed the data of the following platforms: +## Column Data -- OpenSea -- Rarible -- SuperRare -- CryptoPunks (They get traded in their own contracts) -- Foundation -- LooksRare +| Column Name | Data Type | Description | +|-------------------------|-------------------|----------------------------------------------------------| +| `nft_contract_address` | _varbinary_ | The contract address of the NFT | +| `number_of_items` | _decimal(38,0)_ | The number of NFT items in this trade | +| `project` | _varchar_ | The platform where the NFT is traded | +| `project_contract_address` | _varbinary_ | The contract address of platform | +| `seller` | _varbinary_ | The seller's address of the NFT transaction | +| `token_id` | _varchar_ | The ID of the NFT token | +| `token_standard` | _varchar_ | The standard of the NFT token (ERC-721 / ERC-1155) | +| `trade_category` | _varchar_ | The category of the NFT trade (e.g., Buy/Sell/Auction) | +| `trade_type` | _varchar_ | The type of the NFT trade (e.g., Single Item Trade/Bundle Trade) | +| `tx_from` | _varbinary_ | The address that initiated the transaction | +| `tx_hash` | _varbinary_ | The hash of the NFT transaction | +| `tx_to` | _varbinary_ | The destination address of the NFT transaction | +| `unique_trade_id` | _varchar_ | A unique identifier for the NFT trade | +| `version` | _varchar_ | The version of the platform | +| `aggregator_address` | _varbinary_ | The address of the aggregator | +| `aggregator_name` | _varchar_ | The name of the aggregator | +| `amount_original` | _double_ | The original amount of currency in the trade | +| `amount_raw` | _decimal(38,0)_ | The raw amount of currency in the trade | +| `amount_usd` | _double_ | The USD value of the trade | +| `block_number` | _double_ | The block number of the NFT transaction | +| `block_time` | _timestamp_ | The timestamp of the block of the NFT transaction | +| `blockchain` | _varchar_ | The blockchain of the NFT transaction | +| `buyer` | _varbinary_ | The buyer's address in the NFT trade | +| `collection` | _varchar_ | The NFT collection name | +| `currency_contract` | _varbinary_ | The contract address of the currency used in the trade | +| `currency_symbol` | _varchar_ | The symbol of the currency used in the trade | +| `evt_type` | _varchar_ | The type of event associated with the NFT transaction | ## How it works @@ -61,25 +84,25 @@ Royalty fees are going to the creator, and Platform fees are collected by the NF **_SQL_** ```sql +-- get trade details of specific NFT collection select * from nft.trades - -where nft_contract_address = '\xb47e3cd837ddf8e4c57f05d70ab865de6e193bbb' --this is the cryptopunks address +where nft_contract_address = 0x670fd103b1a08628e9557cd66b87ded841115190 +AND block_time >= NOW() - interval '7' day ``` **_Results_** ![type:video](https://dune.com/embeds/146090/288199/bc835020-f730-4348-b749-abd94277b0f7) -#### Trades in the last 24 hour on a given platform - -**_SQL_** +#### Top collection in terms of volume traded in the last 24 hour on a given platform ```sql -select date_trunc('day', block_time), usd_amount, nft_contract_address, token_id from nft.trades - -where platform = 'OpenSea' --only shows trades on given Platform - -and block_time > now() - interval '24hours' +select COALESCE(collection,CAST(nft_contract_address AS VARCHAR)) as collection,blockchain,SUM(amount_usd) as total_volume from nft.trades +where project = 'opensea' --only shows trades on Opensea +and block_time > now() - interval '24' hour +GROUP BY 1,2 +ORDER BY 3 DESC +limit 100 ``` **_Results_** @@ -119,43 +142,6 @@ group by platform, day - [NFT by @sealaunch](https://dune.com/sealaunch/NFT) -## Column Data - -| Column name | Data type | Description | -| - | :-: | - | -| `block_time` | _timestamp with time zone_ | When was this trade executed | -| `block_time` | _varchar_ | NFT project name (e.g. "the dudes") | -| `nft_token_id` | _varchar_ | The token_id that was traded (e.g. 235) | -| `erc_standard` | _varchar_ | The Token Standard of the traded token `ERC-721` or `ERC-1155` | -| `platform` | _varchar_ | Which Platform the trade was executed on | -| `platform_version` | _varchar_ | Which version of this platform was utilized? | -| `trade_type` | _varchar_ | "Single Item Sale" or "Bundle Sale" | -| `number_of_items` | _integer_ | How many NFTs were included in this trade | -| `category` | _varchar_ | Was this an auction or a direct sale | -| `evt_type` | _varchar_ | Currently not in use, default 'Trade' | -| `aggregator` | _varchar_ | Was this trade made using an aggregator (Yes : Name of aggregator, No : Null) | -| `usd_amount` | _numeric_ | USD value of the trade at time of execution | -| `seller` | _varbinary_ | Seller of NFTs | -| `buyer` | _varbinary_ | Buyer of NFTs | -| `royalty_fees_percent` | _numeric_ | Royalty fees going to the creator (in %) | -| `original_royalty_fees` | _numeric_ | Royalty fees in the currency used for this trade | -| `usd_royalty_fees` | _numeric_ | USD value of royalty fees at time of execution | -| `platform_fees_percent` | _numeric_ | Platform fees (in %) | -| `original_platform_fees` | _numeric_ | Platform fees in the currency used for this trade | -| `usd_platform_fees` | _numeric_ | USD value of platform fees at time of execution | -| `original_currency` | _varchar_ | The Currency used for this trade | -| `original_currency_contract` | _varbinary_ | The ERC-20 address of the currency used in this trade (does not work with raw ETH) | -| `currency_contract` | _varbinary_ | The corrected currency contract | -| `nft_contract_address` | _varbinary_ | The contract address of the NFT traded | -| `exchange_contract_address` | _varbinary_ | The platform contract that facilitated this trade | -| `tx_hash` | _varbinary_ | The hash of this transaction | -| `block_number` | _integer_ | The block_number that this trade was done in | -| `tx_from` | _varbinary_ | Initiated this transaction | -| `tx_to` | _varbinary_ | Received this transaction | -| `trace_address` | _ARRAY_ | n/a | -| `evt_index` | _integer_ | Event index | -| `trade_id` | _integer_ | n/a | - ## Ser, my platform is not indexed The SQL code that processes the data for every market place is open source and available in our [github repository](https://github.com/duneanalytics/spellbook/tree/master/ethereum/nft/trades). Everyone can review the code, make pull requests and submit code to add more marketplaces. diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index d3a78ad6..ef0f6e28 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -31,9 +31,9 @@ and blockchain = 'ethereum' and minute >= NOW() - interval '30' day ``` -## prices.usd_latest +## Getting the latest token price -Just as the table name suggest, this table contains the latest price available, derived from prices.usd +We can use prices.usd_latest. Just as the table name suggest, this table contains the latest price available, derived from prices.usd ```sql -- getting the latest price for 4 tokens From d2d999cfc17fa0ea1f826d29757d2104cead927d Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 22:59:33 +0800 Subject: [PATCH 13/24] Update nft.trades.md removing the embeds --- .../spellbook/top-tables/nft.trades.md | 19 +++++-------------- 1 file changed, 5 insertions(+), 14 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/nft.trades.md b/docs/data-tables/spellbook/top-tables/nft.trades.md index 7e99ab91..318e2fc9 100644 --- a/docs/data-tables/spellbook/top-tables/nft.trades.md +++ b/docs/data-tables/spellbook/top-tables/nft.trades.md @@ -79,9 +79,7 @@ Royalty fees are going to the creator, and Platform fees are collected by the NF ### Queries -#### All trades for a given NFT - -**_SQL_** +#### All trades for a given NFT in the past 7 days ```sql -- get trade details of specific NFT collection @@ -89,26 +87,19 @@ select * from nft.trades where nft_contract_address = 0x670fd103b1a08628e9557cd66b87ded841115190 AND block_time >= NOW() - interval '7' day ``` - -**_Results_** - -![type:video](https://dune.com/embeds/146090/288199/bc835020-f730-4348-b749-abd94277b0f7) - #### Top collection in terms of volume traded in the last 24 hour on a given platform ```sql -select COALESCE(collection,CAST(nft_contract_address AS VARCHAR)) as collection,blockchain,SUM(amount_usd) as total_volume from nft.trades +select COALESCE(collection,CAST(nft_contract_address AS VARCHAR)) as collection, -- using coalesce here will get you the nft_contract_address + blockchain, -- instead of null if collection name does not exist + SUM(amount_usd) as total_volume +FROM nft.trades where project = 'opensea' --only shows trades on Opensea and block_time > now() - interval '24' hour GROUP BY 1,2 ORDER BY 3 DESC limit 100 ``` - -**_Results_** - -![type:video](https://dune.com/embeds/1622909/2690008/ce6aa75e-b94c-4dcf-a1f0-020d2cb5fa9b) - #### Platform volumes in the last year **_SQL_** From d945c5b320edc9251439ab7637054b8ac3136e5d Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Tue, 15 Aug 2023 23:21:06 +0800 Subject: [PATCH 14/24] Update nft.trades.md --- .../spellbook/top-tables/nft.trades.md | 24 +++++++------------ 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/nft.trades.md b/docs/data-tables/spellbook/top-tables/nft.trades.md index 318e2fc9..d5476d34 100644 --- a/docs/data-tables/spellbook/top-tables/nft.trades.md +++ b/docs/data-tables/spellbook/top-tables/nft.trades.md @@ -75,10 +75,6 @@ In the most recent version of `nft.trades`, information about the amount and per Royalty fees are going to the creator, and Platform fees are collected by the NFT platform. Note that royalty fees cannot always be retrieved, and are set to null by default. -## Examples - -### Queries - #### All trades for a given NFT in the past 7 days ```sql @@ -100,23 +96,19 @@ GROUP BY 1,2 ORDER BY 3 DESC limit 100 ``` -#### Platform volumes in the last year - -**_SQL_** +#### Platform daily and cumulative volume over time in the last year ```sql -select sum(usd_amount), - date_trunc('day', block_time) as day, - platform +select date_trunc('day', block_time) as day, + project, + sum(amount_usd) as daily_project_amount_usd, + sum(sum(amount_usd)) OVER (PARTITION BY project ORDER BY date_trunc('day',block_time)) as project_cumulative_volume_usd from nft.trades -where block_time > now() - interval '365 days' -group by platform, day +where block_time > now() - interval '365' day +group by 1,2 +ORDER BY 1 DESC,4 DESC ``` -**_Results_** - -![type:video](https://dune.com/embeds/146160/288002/cc990e4d-21e8-43a7-9bc3-2357a72be7b0) - ### Dashboards #### That utilize parameters From a025bc365f69f351cc260f3e7abfdd4f3468325f Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Wed, 16 Aug 2023 00:13:21 +0800 Subject: [PATCH 15/24] update more --- .../spellbook/top-tables/dex.trades.md | 6 +- .../spellbook/top-tables/labels.md | 97 ++++++++++++++++--- .../spellbook/top-tables/prices.md | 6 +- 3 files changed, 89 insertions(+), 20 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/dex.trades.md b/docs/data-tables/spellbook/top-tables/dex.trades.md index c4b88f70..c21eb137 100644 --- a/docs/data-tables/spellbook/top-tables/dex.trades.md +++ b/docs/data-tables/spellbook/top-tables/dex.trades.md @@ -41,7 +41,7 @@ The scripts that generate the table dex.trades can be found in this [public gith | `tx_to` | _varbinary_ | The address of the first smart contract called during this transaction | | `version` | _varchar_ | The version of the DEX used for this trade | -Get all transactions of USDC swaps in the past 24 hours +#### Get all transactions of USDC swaps in the past 24 hours ```sql select * from dex.trades @@ -51,7 +51,7 @@ AND blockchain = 'ethereum' AND block_time >= NOW() - interval '24' hour ``` -Get top 100 uniswap pairs' volume from uniswap in the past 3 days +#### Get top 100 uniswap pairs' volume from uniswap in the past 3 days ```sql select token_pair, @@ -66,7 +66,7 @@ ORDER BY 2 DESC -- order by total_volume limit 100 -- 100 rows ``` -Get top 100 traders' volume of erc20 token using dex.trades +#### Get top 100 traders' volume of erc20 token using dex.trades ```sql SELECT tx_from as address, diff --git a/docs/data-tables/spellbook/top-tables/labels.md b/docs/data-tables/spellbook/top-tables/labels.md index 0beeece7..4828c65c 100644 --- a/docs/data-tables/spellbook/top-tables/labels.md +++ b/docs/data-tables/spellbook/top-tables/labels.md @@ -69,21 +69,88 @@ Note that there might be a few minutes delay from adding the label on [dune.com] ## The labels table -Labels are stored in the new `labels.labels` table which has the following schema: - -| Column name | Data type | Description | -| - | :-: | - | -| `id` | _int_ | incrementing integer | -| `address` | _varbinary_ | The address of a contract or wallet this label describes | -| `name` | _varchar_ | label name | -| `blockchain` | _varchar_ | the blockchain the label is meant for | -| `author` | _varchar_ | The username of the user who created this label | -| `source` | _varchar_ | The source of this label, autopopulated by Dune | -| `updated_at` | _timestamptz_ | The last time this label was changed | -| `label_type` | _varchar_ | The type of label, defined in the readme | -| `model_name` | _varchar_ | The name of the label model (filename) | +Labels are stored in the new `labels.all` table which has the following schema: + +| Column name | Data type | Description | +|---------------|---------------|-------------------------------------------------------| +| `address` | _varbinary_ | The address of a contract or wallet this label describes | +| `category` | _varchar_ | The category of the label (e.g., dex/contracts/institution) | +| `blockchain` | _varchar_ | The blockchain of the address where the label is given | +| `contributor` | _varchar_ | The name of the contributor that created the label | +| `name` | _varchar_ | The name of the label | +| `source` | _varchar_ | The source reference of the label | +| `label_type` | _varchar_ | The type of label (e.g., persona/usage/identifier) | +| `model_name` | _varchar_ | The name of the model used to create the label | +| `created_at` | _timestamp_ | The time when the label was created | +| `updated_at` | _timestamp_ | The time when the label was last updated | ## Using labels -!!! warning - this section is currently under construction, stay tuned! \ No newline at end of file +#### Using label to identify top token holders + +Using labels to identify top CRV holders + +```sql +SELECT address, + ens, + ARRAY_AGG(DISTINCT name) as label_list, + symbol, + contract_address, + balance +FROM ( +SELECT address, + symbol, + name as ens, + contract_address, + SUM(amount) as balance +FROM ( +SELECT tr."from" AS address, + symbol, + contract_address, + -(tr.value/POW(10,decimals)) AS amount +FROM erc20_ethereum.evt_transfer tr JOIN tokens.erc20 USING (contract_address) +WHERE "from" != 0x0000000000000000000000000000000000000000 -- exclude mint/burn addresses +AND contract_address = 0xD533a949740bb3306d119CC777fa900bA034cd52 -- contract_address of the erc20 token +UNION ALL +SELECT tr."to" AS address, + symbol, + contract_address, + (tr.value/POW(10,decimals)) AS amount +FROM erc20_ethereum.evt_transfer tr JOIN tokens.erc20 USING (contract_address) + WHERE "to" != 0x0000000000000000000000000000000000000000 -- exclude mint/burn addresses + AND contract_address = 0xD533a949740bb3306d119CC777fa900bA034cd52 -- contract_address of the erc20 token + ) x LEFT JOIN ens.reverse_latest USING (address) + GROUP BY 1,2,3,4 + HAVING SUM(amount) > 0.1 -- having more than 0.1 balance + ) p LEFT JOIN labels.all USING (address) + GROUP BY 1,2,4,5,6 + ORDER BY 6 DESC +``` + +#### Using addresses of labels.contract_deployers_ethereum to find the top 100 deployers + +```sql +SELECT "from" as deployer, + COUNT(*) as contracts_deployed +from ethereum.creation_traces +WHERE "from" IN (SELECT distinct address FROM labels.contract_deployers_ethereum) +AND block_time >= NOW() - interval '7' day +GROUP BY 1 +ORDER BY 2 DESC +LIMIT 100 +``` + +#### Using smart_dex_traders label to find tokens traded by traders + +```sql +SELECT tx_from as address, + COALESCE(token_bought_symbol,CAST(token_bought_address AS VARCHAR)) as token_amount, + SUM(amount_usd) as total_volume, + SUM(SUM(amount_usd)) OVER (PARTITION BY COALESCE(token_bought_symbol,CAST(token_bought_address AS VARCHAR))) as total_token_volume +FROM dex.trades +WHERE tx_from IN (select from_hex(address) from labels.smart_dex_traders) +AND block_time >= NOW() - interval '14' day +GROUP BY 1,2 +HAVING SUM(amount_usd) >= 100000 +ORDER BY 4 DESC,3 DESC +``` \ No newline at end of file diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index ef0f6e28..db961f88 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -31,9 +31,9 @@ and blockchain = 'ethereum' and minute >= NOW() - interval '30' day ``` -## Getting the latest token price +#### Getting the latest token price -We can use prices.usd_latest. Just as the table name suggest, this table contains the latest price available, derived from prices.usd +Just as the table name suggest, this table contains the latest price available, derived from prices.usd ```sql -- getting the latest price for 4 tokens @@ -77,6 +77,8 @@ We now have successfully calculated the price of 1 $SPELL. In order to correct for extreme outliers and in order for this table to be performant the script then aggregates all recorded data into one `median_price` per hour. +#### Get daily average price using dex.prices + ```sql -- get daily average price of weth from dex.prices select date_trunc('day',hour) as date, From 78a07c753271a27742680c2461203719fdaebd3b Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Wed, 16 Aug 2023 00:18:30 +0800 Subject: [PATCH 16/24] Update tokens.md --- docs/data-tables/spellbook/top-tables/tokens.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/tokens.md b/docs/data-tables/spellbook/top-tables/tokens.md index 8152c510..da16782b 100644 --- a/docs/data-tables/spellbook/top-tables/tokens.md +++ b/docs/data-tables/spellbook/top-tables/tokens.md @@ -47,7 +47,7 @@ where contract_address = 0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d | `evt_index` | _bigint_ | The index of the event within the block. | | `evt_tx_hash` | _varbinary_ | The transaction hash of the event. | -Finding out the total inflow and outflow of USDT token from an address in the past 3 days +#### Get the total inflow and outflow of USDT token from an address in the past 3 days ```sql SELECT CASE WHEN "from" = 0xdfd5293d8e347dfe59e90efd55b2956a1343963d THEN 'Outflow' ELSE 'Inflow' END AS token_flow, @@ -59,7 +59,7 @@ AND evt_block_time >= NOW() - interval '3' day GROUP BY 1 ``` -Get all holders with their ens and balances of an erc20 token contract +#### Get all holders with their ens and balances of an erc20 token contract ```sql SELECT address, @@ -108,7 +108,7 @@ You can find how to get erc20 balances, mints, and burns using [this guide](http | `evt_index` | _bigint_ | The index of the event within the block | | `evt_tx_hash` | _varbinary_ | The transaction hash of the event | -Get all erc721 token transfers to an addresses in the past 30 days +#### Get all erc721 token transfers to an addresses in the past 30 days ```sql SELECT date_trunc('day',evt_block_time) as date, @@ -120,7 +120,7 @@ GROUP BY 1 order by 1 desc ``` -Get all holders with their ens,labels and balances of an erc721 token contract +#### Get all holders with their ens,labels and balances of an erc721 token contract ```sql SELECT z.name, From 2ffdab3763486954a0f392a9a509601b99551c95 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Wed, 16 Aug 2023 00:27:02 +0800 Subject: [PATCH 17/24] removed this page --- docs/resources/sample-queries.md | 286 ------------------------------- 1 file changed, 286 deletions(-) delete mode 100644 docs/resources/sample-queries.md diff --git a/docs/resources/sample-queries.md b/docs/resources/sample-queries.md deleted file mode 100644 index 323b742e..00000000 --- a/docs/resources/sample-queries.md +++ /dev/null @@ -1,286 +0,0 @@ ---- -title: Sample Queries -description: Learn how to write queries using popular datasets ---- - -# Writing Queries Using Popular Datasets - -There are plenty of dataset created that you can use to query what you need. Here are some commonly requested queries using popular datasets - -## prices.usd - -The prices.usd table contains price feed data derived from [Coinpaprika API](https://coinpaprika.com/). -You can also do a pull request in the [here](https://github.com/duneanalytics/spellbook/tree/main/models/prices) to have your token added in prices.usd (Make sure that the token is active on Coinpaprika!) - -
-
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Column NamesData TypeDescription
contract_addressvarbinarycontract_address of the token
blockchainvarcharblockchain that the token is on
decimalsintnumber of decimals
minutetimestamptimestamp of price feed
pricedoubletoken price
symbolvarchartoken symbol
-
- -```sql --- getting the token price for WETH for the past 30 days -select * from prices.usd -where contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 -and blockchain = 'ethereum' -and minute >= NOW() - interval '30' day -``` - -## prices.usd_latest - -Just as the table name suggest, this table contains the latest price available, derived from prices.usd - -```sql --- getting the latest price for 4 tokens -select * from prices.usd_latest -where blockchain = 'ethereum' -and symbol IN ('WETH','WBTC','USDC','USDT') -``` - -## erc20_ethereum.evt_transfer - -The erc20_ethereum.evt_transfer contains all erc20 token transfer events that occured. With this table, you will be able to get information such as number of token holders,token balances and many more! This table is available for other EVM chain too, just change the suffix to the chain desired(eg.erc20_arbitrum.evt_transfer,erc20_bnb.evt_transfer) - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Column NamesData TypeDescription
contract_addressvarbinaryerc20 token contract address
fromvarbinaryAddress tokens are transferred from
tovarbinaryAddress tokens are transferred to
valueuint256Number of tokens transferred
evt_block_numberbigintBlock number in which the event occurred
evt_block_timetimestampTimestamp of the event occurrence
evt_indexbigintIndex of the event within the block
evt_tx_hashvarbinaryTransaction hash of the event
-
- -```sql --- get total amount of weth sent daily in the past 3 days -SELECT DATE_TRUNC('day', evt_block_time) AS date, - SUM(value / 1e18) AS daily_amount -FROM erc20_ethereum.evt_transfer -WHERE contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 -AND evt_block_time >= NOW() - INTERVAL '3' day -GROUP BY 1 -``` - -You can also get your wallet erc20 token balances! There is a evms.erc20_transfers table that contains all evms erc20 token transfer event - -```sql --- getting erc20 token balances for 0xbe0eb53f46cd790cd13851d5eff43d12404d33e8 -SELECT * FROM ( -select blockchain,symbol,SUM(CASE WHEN "from" = 0xbe0eb53f46cd790cd13851d5eff43d12404d33e8 THEN -(value/POW(10,decimals)) ELSE (value/POW(10,decimals)) END) AS token_balance -from evms.erc20_transfers JOIN tokens.erc20 USING (contract_address,blockchain) -where ("from" = 0xbe0eb53f46cd790cd13851d5eff43d12404d33e8 OR "to" = 0xbe0eb53f46cd790cd13851d5eff43d12404d33e8) -GROUP BY 1,2 -) -WHERE token_balance > 0 -``` - -## dex.trades - -The dex.trades table contains transfer events that occured. With this table, you will be able to get information such as number of token holders,token balances and many more! This table is available for other EVM chain too, just change the suffix to the chain desired(eg.erc20_arbitrum.evt_transfer,erc20_bnb.evt_transfer) - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Column NamesData TypeDescription
amount_usddoubleEquivalent amount in USD
block_datetimestampDate of the block
block_timetimestampTime of the block
blockchainvarcharBlockchain identifier
evt_indexintIndex of the event within the block
makervarbinaryAddress of the maker
projectvarcharProject name
project_contract_addressvarbinaryContract address of the project
takervarbinaryAddress of the taker
token_bought_addressvarbinaryAddress of the token bought
token_bought_amountdoubleAmount of token bought
token_bought_amount_rawdecimal(38,0)Raw amount of token bought
token_bought_symbolvarcharSymbol of token bought
token_pairvarcharToken pair identifier
token_sold_addressvarbinaryAddress of the token sold
token_sold_amountdoubleAmount of token sold
token_sold_amount_rawdecimal(38,0)Raw amount of token sold
token_sold_symbolvarcharSymbol of token sold
trace_addressvarcharTrace address
tx_fromvarbinaryTransaction sender address
tx_hashvarbinaryTransaction hash
tx_tovarbinaryTransaction receiver address
versionvarcharVersion of the trade
-
- - - - From b608196fcd85edb8ad0109586ace367691552e98 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Fri, 18 Aug 2023 02:09:07 +0800 Subject: [PATCH 18/24] added embeds --- .../spellbook/top-tables/labels.md | 27 +++++++++++------- .../spellbook/top-tables/nft.trades.md | 25 +++++++++++++---- .../spellbook/top-tables/prices.md | 10 ++++++- .../spellbook/top-tables/tokens.md | 28 +++++++++++++++---- 4 files changed, 68 insertions(+), 22 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/labels.md b/docs/data-tables/spellbook/top-tables/labels.md index 4828c65c..88114cad 100644 --- a/docs/data-tables/spellbook/top-tables/labels.md +++ b/docs/data-tables/spellbook/top-tables/labels.md @@ -127,30 +127,37 @@ FROM erc20_ethereum.evt_transfer tr JOIN tokens.erc20 USING (contract_address) ORDER BY 6 DESC ``` + + #### Using addresses of labels.contract_deployers_ethereum to find the top 100 deployers ```sql -SELECT "from" as deployer, - COUNT(*) as contracts_deployed +SELECT COUNT(*) as contracts_deployed, + SUM(COUNT(*)) OVER (ORDER BY block_time) as contracts_over_time from ethereum.creation_traces WHERE "from" IN (SELECT distinct address FROM labels.contract_deployers_ethereum) AND block_time >= NOW() - interval '7' day GROUP BY 1 ORDER BY 2 DESC -LIMIT 100 ``` + + #### Using smart_dex_traders label to find tokens traded by traders ```sql -SELECT tx_from as address, - COALESCE(token_bought_symbol,CAST(token_bought_address AS VARCHAR)) as token_amount, - SUM(amount_usd) as total_volume, - SUM(SUM(amount_usd)) OVER (PARTITION BY COALESCE(token_bought_symbol,CAST(token_bought_address AS VARCHAR))) as total_token_volume +SELECT COALESCE(token_bought_symbol,CAST(token_bought_address AS VARCHAR)) as token_traded, + COUNT(DISTINCT tx_from) as address_count, + ARRAY_AGG(tx_from) as address_list, + SUM(amount_usd) as total_volume FROM dex.trades WHERE tx_from IN (select from_hex(address) from labels.smart_dex_traders) +AND token_bought_address != 0x0000000000000000000000000000000000000000 AND block_time >= NOW() - interval '14' day -GROUP BY 1,2 +GROUP BY 1 HAVING SUM(amount_usd) >= 100000 -ORDER BY 4 DESC,3 DESC -``` \ No newline at end of file +ORDER BY 4 DESC +``` + + + diff --git a/docs/data-tables/spellbook/top-tables/nft.trades.md b/docs/data-tables/spellbook/top-tables/nft.trades.md index d5476d34..65e7c9c5 100644 --- a/docs/data-tables/spellbook/top-tables/nft.trades.md +++ b/docs/data-tables/spellbook/top-tables/nft.trades.md @@ -75,27 +75,38 @@ In the most recent version of `nft.trades`, information about the amount and per Royalty fees are going to the creator, and Platform fees are collected by the NFT platform. Note that royalty fees cannot always be retrieved, and are set to null by default. -#### All trades for a given NFT in the past 7 days +#### Trading price for a given NFT in the past 7 days ```sql --- get trade details of specific NFT collection +-- get trade details of specific NFT collection (y00ts on polygon) select * from nft.trades where nft_contract_address = 0x670fd103b1a08628e9557cd66b87ded841115190 AND block_time >= NOW() - interval '7' day ``` -#### Top collection in terms of volume traded in the last 24 hour on a given platform + + + + +#### Top collection in terms of volume traded in the last 72 hour on a given platform ```sql -select COALESCE(collection,CAST(nft_contract_address AS VARCHAR)) as collection, -- using coalesce here will get you the nft_contract_address +SELECT ROW_NUMBER() OVER (ORDER BY total_volume DESC) as ranking, + * +FROM ( +SELECT COALESCE(collection,CAST(nft_contract_address AS VARCHAR)) as collection, -- using coalesce here will get you the nft_contract_address blockchain, -- instead of null if collection name does not exist SUM(amount_usd) as total_volume FROM nft.trades where project = 'opensea' --only shows trades on Opensea -and block_time > now() - interval '24' hour +and block_time > now() - interval '72' hour GROUP BY 1,2 ORDER BY 3 DESC limit 100 +) ``` + + + #### Platform daily and cumulative volume over time in the last year ```sql @@ -108,6 +119,10 @@ where block_time > now() - interval '365' day group by 1,2 ORDER BY 1 DESC,4 DESC ``` + + + + ### Dashboards diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index db961f88..6cf2e9db 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -28,9 +28,11 @@ Note that `WETH` can be used for ETH price as it trades at virtually the same pr select * from prices.usd where contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 and blockchain = 'ethereum' -and minute >= NOW() - interval '30' day +and minute >= NOW() - interval '7' day ``` + + #### Getting the latest token price Just as the table name suggest, this table contains the latest price available, derived from prices.usd @@ -42,6 +44,10 @@ where blockchain = 'ethereum' and symbol IN ('WETH','WBTC','USDC','USDT') ``` + + + + ## How we get prices from DEXs We created a table that creates price feeds based on decentralized exchange trading data. This table covers much more assets than `prices.usd`, since it covers all assets that are traded on any of the decentralized exchanges that are indexed in `dex.trades`. @@ -91,6 +97,8 @@ AND hour >= NOW() - interval '3' day GROUP BY 1,2 ``` + + ### Known issues In rare cases this script will generate price feeds that are based on illiquid pairs and therefore report wrong data. This happens when all liquid trading pools of this token do not have a price feed in `prices.usd`. diff --git a/docs/data-tables/spellbook/top-tables/tokens.md b/docs/data-tables/spellbook/top-tables/tokens.md index da16782b..e0f3c7da 100644 --- a/docs/data-tables/spellbook/top-tables/tokens.md +++ b/docs/data-tables/spellbook/top-tables/tokens.md @@ -23,6 +23,10 @@ select * from tokens.erc20 WHERE contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 ``` +| Blockchain | Contract Address | Symbol | Decimals | +|------------|-------------------------------------------|--------|---------- | +| Ethereum | 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 | WETH | 18 | + 2. [**`tokens.nft`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.tokens_nft): contains the collection `name` and `symbol` for any given `contract_address`. These tables are usually joined on `contract_address` at the end of a query to make everything more human readable. @@ -33,6 +37,10 @@ select * from tokens.nft where contract_address = 0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d ``` +| Blockchain | Contract Address | Name | Symbol | Token Standard | +|------------|-------------------------------------------|---------------------|--------|---------------- | +| Ethereum | 0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d | Bored Ape Yacht Club | BAYC | ERC721 | + ## Transfer tables: 1. [**`erc20_ethereum.evt_Transfer`**](https://spellbook-docs.dune.com/#!/model/model.spellbook.transfers_ethereum_erc20): all transfer events for every erc20 token. @@ -59,13 +67,14 @@ AND evt_block_time >= NOW() - interval '3' day GROUP BY 1 ``` + + #### Get all holders with their ens and balances of an erc20 token contract ```sql SELECT address, symbol, ARRAY_AGG(distinct name) as ens, - contract_address, SUM(amount) as balance FROM ( SELECT tr."from" AS address, @@ -86,11 +95,13 @@ FROM erc20_ethereum.evt_transfer tr JOIN tokens.erc20 USING (contract_address) AND contract_address = 0x046eee2cc3188071c02bfc1745a6b17c656e3f3d -- contract_address of the erc20 token -- AND evt_block_time <= TIMESTAMP '2023-01-01' -- you can use this to get snapshot data ) x LEFT JOIN labels.ens USING (address) - GROUP BY 1,2,4 + GROUP BY 1,2 HAVING SUM(amount) > 0.1 -- having more than 0.1 balance - ORDER BY 5 DESC + ORDER BY 4 DESC ``` + + If you're looking for how to calculate native token balances like ethereum (ETH) balances then check out [this guide](https://web3datadegens.substack.com/p/web3-sql-weekly-1-how-to-calculate) You can find how to get erc20 balances, mints, and burns using [this guide](https://www.youtube.com/watch?v=LT_PB-Fso3M). @@ -112,14 +123,17 @@ You can find how to get erc20 balances, mints, and burns using [this guide](http ```sql SELECT date_trunc('day',evt_block_time) as date, + name, COUNT(*) as transfer_count -FROM erc721_ethereum.evt_transfer -where to = 0x6ce82874eaf6e7602fd21cf8bbded82705680a99 +FROM erc721_ethereum.evt_transfer LEFT JOIN labels.contracts on to = address +where to = 0xdb5485c85bd95f38f9def0ca85499ef67dc581c0 AND evt_block_time >= NOW() - interval '30' day -GROUP BY 1 +GROUP BY 1,2 order by 1 desc ``` + + #### Get all holders with their ens,labels and balances of an erc721 token contract ```sql @@ -155,4 +169,6 @@ GROUP BY 1,2,3,4,6,7,8 ORDER BY 7 DESC ``` + + You can learn how to leverage this to find nft balances, transfers, and mints in [this guide](https://web3datadegens.substack.com/p/web3-sql-weekly-3-finding-all-nfts) \ No newline at end of file From 10a03e0a25adcdcc85bb53cdb05d98156202dd5e Mon Sep 17 00:00:00 2001 From: Boxer Date: Fri, 18 Aug 2023 14:59:43 +0200 Subject: [PATCH 19/24] change dashboard links --- docs/data-tables/spellbook/top-tables/nft.trades.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/nft.trades.md b/docs/data-tables/spellbook/top-tables/nft.trades.md index 65e7c9c5..1b095bcd 100644 --- a/docs/data-tables/spellbook/top-tables/nft.trades.md +++ b/docs/data-tables/spellbook/top-tables/nft.trades.md @@ -129,15 +129,15 @@ ORDER BY 1 DESC,4 DESC #### That utilize parameters
-- [NFT by @0xBoxer](https://dune.com/0xBoxer/NFT) -- [NFT Sales Overview by Project by @rantum](https://dune.com/rantum/NFT-Sales-Overview-by-Project) +- [NFT Business metrics by @0datawolf](https://dune.com/0xdatawolf/nft-business-metrics) +- [NFT Custom Project @sealaunch](https://dune.com/sealaunch/abc)
#### That look across the entire Ecosystem
-- [NFT Collection Dashboard by @rantum](https://dune.com/rantum/NFT-Collection-Dashboard) -- [NFT by @sealaunch](https://dune.com/sealaunch/NFT) +- [NFT market overview by @hildobby](https://dune.com/hildobby/NFTs) +- [NFT Top Collections by @sealaunch](https://dune.com/sealaunch/NFT-Top-Collections)
## Ser, my platform is not indexed From 9e1dddd87e735c98adf5dedabe8bb03c99b70c16 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Sat, 19 Aug 2023 01:38:52 +0800 Subject: [PATCH 20/24] add more embeds --- .../spellbook/top-tables/dex.trades.md | 15 +++++++++++++++ .../spellbook/top-tables/nft.trades.md | 5 +++++ 2 files changed, 20 insertions(+) diff --git a/docs/data-tables/spellbook/top-tables/dex.trades.md b/docs/data-tables/spellbook/top-tables/dex.trades.md index c21eb137..ec390fbb 100644 --- a/docs/data-tables/spellbook/top-tables/dex.trades.md +++ b/docs/data-tables/spellbook/top-tables/dex.trades.md @@ -13,6 +13,10 @@ This table standardizes and normalizes the trading data across virtually all rel The scripts that generate the table dex.trades can be found in this [public github](https://github.com/duneanalytics/spellbook/tree/main/models/dex) repo. +## Indexed Chains And Platforms + + + ## Column Data | Column Name | Data Type | Description | @@ -51,6 +55,9 @@ AND blockchain = 'ethereum' AND block_time >= NOW() - interval '24' hour ``` + + + #### Get top 100 uniswap pairs' volume from uniswap in the past 3 days ```sql @@ -66,9 +73,13 @@ ORDER BY 2 DESC -- order by total_volume limit 100 -- 100 rows ``` + + + #### Get top 100 traders' volume of erc20 token using dex.trades ```sql +-- top traders trading wBTC in the past 7 days SELECT tx_from as address, SUM(amount_usd) as total_volume, SUM(CASE WHEN "token_bought_address" = 0x2260fac5e5542a773aa44fbcfedf7c193bc2c599 THEN 1 END) as buy_count, @@ -84,4 +95,8 @@ ORDER BY 2 DESC limit 100 ``` + + + + diff --git a/docs/data-tables/spellbook/top-tables/nft.trades.md b/docs/data-tables/spellbook/top-tables/nft.trades.md index 1b095bcd..5ff69717 100644 --- a/docs/data-tables/spellbook/top-tables/nft.trades.md +++ b/docs/data-tables/spellbook/top-tables/nft.trades.md @@ -9,6 +9,11 @@ The culmination of this is a dataset which makes it extremely easy to query for You can find the specifications for nft.trades on our [Spellbook documentation](https://dune.com/spellbook#!/model/model.spellbook.nft_trades). +## Indexed Chain And Projects + + + + ## Column Data | Column Name | Data Type | Description | From 43afbad7ffc2e2cb8e9465a23699ce1612c6c941 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Sat, 19 Aug 2023 01:50:18 +0800 Subject: [PATCH 21/24] fix labels page --- docs/data-tables/spellbook/top-tables/labels.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/labels.md b/docs/data-tables/spellbook/top-tables/labels.md index 88114cad..faa1ba23 100644 --- a/docs/data-tables/spellbook/top-tables/labels.md +++ b/docs/data-tables/spellbook/top-tables/labels.md @@ -86,9 +86,7 @@ Labels are stored in the new `labels.all` table which has the following schema: ## Using labels -#### Using label to identify top token holders - -Using labels to identify top CRV holders +#### Identifying CRV Holders Using Labels ```sql SELECT address, From 400448309bfea581edcbae8cb6f0daa26d381913 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Sat, 19 Aug 2023 02:29:03 +0800 Subject: [PATCH 22/24] Update prices.md --- .../spellbook/top-tables/prices.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index 6cf2e9db..343a68ea 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -54,32 +54,32 @@ We created a table that creates price feeds based on decentralized exchange trad **Please keep in mind that this script can generate wrong prices in rare cases.** -This table is very resource intensive and can therefore only be updated every few hours, please keep that in mind when utilizing it. Also the resolution is only hourly, so if you need minutely prices do refer to [`prices.usd`](prices.md). - The logic of how this table works can be accessed in our [public github](https://github.com/duneanalytics/spellbook/blob/main/models/dex/dex_prices.sql) repo. -This script generates median hourly prices based on data from decentralized exchanges found in `dex.trades`. It will assign asset prices based on a trading pair which has a pricefeed in `prices.usd`. +As there could be potential inaccuracy of prices derived from dex.trades due to low liquidity/high slippage trades. Hence, it is recommended to use [`prices.usd`](prices.md),which prices are sourced directly from Coinpaprika API. + +## How Dex.prices work? -Let's take the $SPELL/ETH Pool for example. +For example, let's assume there is a liquidity pool with the token pair $DUNE/ETH * $ETH price is contained in `prices.usd` -* $SPELL price is not contained in `prices.usd` +* $DUNE price is not contained in `prices.usd` In order to get the $SPELL price, the script will dynamically calculate the price of $SPELL based on the price of $ETH that was exchanged for it. -e.g. 5 $ETH were exchanged for 1,086,083 $SPELL. +e.g. 1337 $ETH were exchanged for 69.42 $DUNE. Dex.trades will assign a `usd_amount` to this trade based on the $ETH price data in `prices.usd`. -That `usd_amount` is $23,498. +That `usd_amount` is $13,337,000. -`5 * price of ETH (4.699,6) = $23,498` +`1337 * price of ETH ($10,000) = $13,370,000` Calculating the price of $SPELL is now as simple as dividing the amount of tokens exchanged with the `usd_amount` recorded in `dex.trades`. -`$23,498/1,086,083 ≈ $0,02163` +`$13,337,000/69.42 ≈ $192,595.79` -We now have successfully calculated the price of 1 $SPELL. +We now have successfully calculated the price of 1 $DUNE. In order to correct for extreme outliers and in order for this table to be performant the script then aggregates all recorded data into one `median_price` per hour. From 128038c9ad0b8480fabee6417229d6cf7327da11 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Sat, 19 Aug 2023 15:17:19 +0800 Subject: [PATCH 23/24] fix example $DUNE -> $WBTC --- .../spellbook/top-tables/prices.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index 343a68ea..5892adfc 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -60,26 +60,27 @@ As there could be potential inaccuracy of prices derived from dex.trades due to ## How Dex.prices work? -For example, let's assume there is a liquidity pool with the token pair $DUNE/ETH +For example, there is a liquidity pool with the token pair $WBTC/ETH -* $ETH price is contained in `prices.usd` -* $DUNE price is not contained in `prices.usd` + Assuming the following: +* $ETH price is contained in `prices.usd` +* $WBTC price is not contained in `prices.usd` -In order to get the $SPELL price, the script will dynamically calculate the price of $SPELL based on the price of $ETH that was exchanged for it. +In order to get the $WBTC price, the script will dynamically calculate the price of $WBTC based on the price of $ETH that was exchanged for it. -e.g. 1337 $ETH were exchanged for 69.42 $DUNE. +e.g. 10 $ETH were exchanged for 0.69 $WBTC. Dex.trades will assign a `usd_amount` to this trade based on the $ETH price data in `prices.usd`. -That `usd_amount` is $13,337,000. +That `usd_amount` is $17,000. -`1337 * price of ETH ($10,000) = $13,370,000` +`10 * price of ETH ($1,700) = $17,000` -Calculating the price of $SPELL is now as simple as dividing the amount of tokens exchanged with the `usd_amount` recorded in `dex.trades`. +Calculating the price of $WBTC is now as simple as dividing the amount of tokens exchanged with the `usd_amount` recorded in `dex.trades`. -`$13,337,000/69.42 ≈ $192,595.79` +`$17,000 /0.69 ≈ $24,637.68` -We now have successfully calculated the price of 1 $DUNE. +We now have successfully calculated the price of 1 $WBTC. In order to correct for extreme outliers and in order for this table to be performant the script then aggregates all recorded data into one `median_price` per hour. From 951ec6739599b017347424c3be5751c6b551fcf2 Mon Sep 17 00:00:00 2001 From: jh <0xroll.onchain@gmail.com> Date: Sat, 19 Aug 2023 15:24:03 +0800 Subject: [PATCH 24/24] small fixes --- docs/data-tables/spellbook/top-tables/prices.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/data-tables/spellbook/top-tables/prices.md b/docs/data-tables/spellbook/top-tables/prices.md index 5892adfc..fe57f774 100644 --- a/docs/data-tables/spellbook/top-tables/prices.md +++ b/docs/data-tables/spellbook/top-tables/prices.md @@ -44,7 +44,7 @@ where blockchain = 'ethereum' and symbol IN ('WETH','WBTC','USDC','USDT') ``` - + @@ -63,8 +63,10 @@ As there could be potential inaccuracy of prices derived from dex.trades due to For example, there is a liquidity pool with the token pair $WBTC/ETH Assuming the following: -* $ETH price is contained in `prices.usd` -* $WBTC price is not contained in `prices.usd` + + - $ETH price is contained in `prices.usd` + + - $WBTC price is not contained in `prices.usd` In order to get the $WBTC price, the script will dynamically calculate the price of $WBTC based on the price of $ETH that was exchanged for it. @@ -96,9 +98,10 @@ WHERE blockchain = 'ethereum' AND contract_address = 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 AND hour >= NOW() - interval '3' day GROUP BY 1,2 +ORDER BY 1 DESC ``` - + ### Known issues