[DO NOT MERGE] - Calculate rewards points #75

gianfra-t · 2024-10-18T21:43:52Z

The idea of this PR is NOT to be merged, but to serve as a place to keep and discuss the special purpose indexer code to calculate the points for the campaign.

Overall idea

We only handle contract events, shutting down everything else for this use case.

The overall idea is to integrate the number of points obtained per block. We use the following formula:

points_this_block = (lp_price / total_supply)  * asset_price_usd * (1 / 216000) * (amount_lped / lp_decimals) * (1 / 100 or 50)

To keep the sync time lower, we save 3 entities in memory only and not in the schema. These will keep track of the user's points and mapping between lp tokens and pools. So far we only save Mint, Burn events, and although we could derive the current total lp from them, it would become costly to do so on each block.

The relevant addition and subtraction of the liquidity is handled on the same Mint and Burn events (example).

Similarly, we can replace the in-memory values and put them into the store, yet it seems overkill to implement more schema changes. The only downside is that to get the total current points, the squid must not stop during a run. I believe this is the expected use case for this script/indexer. We do, however, implement a schema to hold the points that will be updated every X blocks. The intention is to be able to easily query the results.

I'll leave relevant comments on the code.

…entity fix

gianfra-t · 2024-10-22T14:22:33Z

src/abi/oracle.ts

@@ -0,0 +1,788 @@
+import { Abi, Bytes, encodeCall, decodeResult } from '@subsquid/ink-abi'


We need this abi to fetch the prices given a pool token.

gianfra-t · 2024-10-22T14:23:37Z

src/mappings/nabla/backstopPoolEventHandler.ts

@@ -143,6 +151,12 @@ export async function handleMint(
        event.amountPrincipleDeposited,
        backstopPool
    )
+    addBackstopLP(


On this implementation, this is how we keep track of the LP tokens per pool and user and avoid storing in the database.

gianfra-t · 2024-10-22T14:32:24Z

src/mappings/points/handlePoints.ts

+    ])
+
+    // Accumulate points from both swap LPs and backstop LPs
+    for (const address of addresses) {


Iterate through addresses to more easily return total points this block for each user.

gianfra-t · 2024-10-22T14:35:49Z

src/mappings/points/helpers.ts

+        block.hash
+    )
+
+    const poolAssetPrice = (await oracleContract.stateCall('0xb3596f07', [


For some reason, the method getAssetPrice is not being generated. So we call it manually.

gianfra-t · 2024-10-22T14:38:22Z

src/mappings/points/handlePoints.ts

+const DUMP_BLOCK_HEIGHT = 3795124 // whenever the campaign finishes.
+
+// global vars to count points
+export const pointsCount = new Map<address, Big>()


We need points to be a Big. The points per block obtained by the current equation can be very small (10**-7)

gianfra-t · 2024-10-22T16:06:19Z

@pendulum-chain/devs I've noticed that the code takes a VERY long term to run. Probably due to the fact that many contract calls are done each block (iteration even) to fetch the price.

Although caching will be super easy to do, still to be 100% correct we would need to call at least once per block the contracts to get the price. If the running time ends up being too long, I propose we used cached values for the prices for N blocks.

TorstenStueber · 2024-10-24T09:56:04Z

Can you give an estimate of the running time? I guess we can just deploy a squid and let it execute on the Subsquid cloud. Even if it takes a day or so, that shouldn't matter too much at the end.

I think we can at least speed up the oracle call. The contract is just a wrapper around a chain extension to read the state from the (dia) oracle pallet. Subsquid should be able to directly read the pallet storage state without making a runtime call, let alone executing a smart contract.

gianfra-t · 2024-10-24T11:50:31Z

The running time was around 1 second per block.. so quite slow. I thought about calling the oracle directly for a different reason but now that you say it that would be way faster.

Since this is indexed by symbol and blockchain and not by address we need to read the contracts the first time and then cache that value for the corresponding pool since that never changes.

I'll try that.

gianfra-t · 2024-10-24T13:44:09Z

Okay done, added caches per block and fetch the price from the diaOracle and not the contract. Still 1 second per block (actually I lied to you before, it was a bit more than 1 second/block before).

I don't know how much else to optimize it if we want exact prices each block. Right now for a 30 day campaign will take 1 hour exactly, as more addresses need to be processed it should scale better since we are caching the prices (in the range I tested, is only 1 address and 3 swap pools).

TorstenStueber · 2024-10-24T14:46:21Z

Do you run it locally? Then it is super slow because it uses the RPC to request information block by block. When running in the Subsquid cloud, it should be much faster as it uses archives and loads information from historical data much faster.

gianfra-t · 2024-10-24T15:06:03Z

Ah! I thought it always used whatever it could, local or not. The we are good.

ebma · 2024-10-24T16:07:38Z

Do you run it locally? Then it is super slow because it uses the RPC to request information block by block.

Are you referring to running the squid locally or indexing a local runtime @TorstenStueber? Indexing a local runtime is slower, yes, but if @gianfra-t was running the squid locally to index pendulum/amplitude/foucoco, then it would also first connect to the archive and eventually switch to the RPC connection. The comment here refers to a local runtime, I might have phrased it poorly. If you check the config, you'll see that the archive field of the local config is undefined. Note that it's also undefined for Foucoco but we can re-enable that. It was just a temporary thing because the subsquid team needed to migrate the foucoco archive after the migration to Paseo.

gianfra-t · 2024-10-24T19:08:25Z

So I ran it on the cloud and we still get 1 block/sec, can see the logs here (need to scroll up a bit).
The normal rate before a liquidity position is minted, is 30 blocks/sec. We could do some profiling to identify what are the slowest operations but I think it is overkill for a process that will only run a couple of times.

We can do this if it turns out it scales poorly with many addresses.

TorstenStueber · 2024-10-25T01:16:20Z

This is rather unexpected. If you run a squid on the Subsquid cloud, then you can see that it catches up very quickly and it is fully synced in I think way less than an hour.

Calling into a contract to request data is slower but the fraction of blocks where interaction with the Forex AMM is taking place is so minimal that I shouldn't have noticeable impact on the total runtime.

Does it make any difference whether you start at a specific block vs. starting from genesis as we usually do with a squid?

TorstenStueber

The algorithm itself looks nice and good to me.

I think the reason for the slow runtime is that you actually do call into smart contracts for each block because you need to get current prices. Although you now query the oracle price directly from the blockchain state, which should be way faster, you still call oracleByAsset of the router contract each block multiple times.

You can either just hard code this information or cache it similar to how you cache getBlockchainSymbolForToken. I assume that if you avoid calling contracts for each single block and only when necessary (possibly not at all), then you will be able to speed up this squid.

TorstenStueber · 2024-10-24T10:00:34Z

src/abi/backstop.ts

-        const input = _abi.encodeMessageInput(selector, args)
-        const data = encodeCall(this.address, input)
-        const result = await this.ctx._chain.rpc.call('state_call', [
+        let input = _abi.encodeMessageInput(selector, args)


This is generated code, right? Strange that they changed this to let statements.

Yes, I thought so too. I did update at some point squid dependencies. They must have changed this.

TorstenStueber · 2024-10-24T10:49:02Z

src/mappings/nabla/backstopPoolEventHandler.ts

@@ -188,8 +202,11 @@ export async function updateBackstopCoverageAndSupply(
    const contractHexAddress = ss58ToHex(backstopPool.id)
    const contract = new BackstopPoolContract(ctx, contractHexAddress)
    const poolTokenAddress = await contract.asset()
-    const poolTokenContract = new Erc20Contract(ctx, poolTokenAddress)
-
+    const poolTokenContract = new Erc20Contract(


Can you merge the latest changes from the main branch or rebase on the latest main branch?

TorstenStueber · 2024-10-24T10:59:09Z

src/mappings/points/helpers.ts

+        ss58ToHex(router.id),
+        block.hash
+    )
+    const relevantOracleHexAddress = await routerContract.oracleByAsset(


This code is really nice and general. But if we want to optimize for speed, it's totally fine to just hardcode all these contracts and addresses as – for now – we will only use it for this use one time use case.

TorstenStueber · 2024-10-24T11:02:03Z

src/mappings/points/helpers.ts

+    )
+    const totalValue = await contract.getTotalPoolWorth()
+
+    const { priceUsdUnits, decimals } = await getBackstopPoolTokenPrice(


The decimals are not used, correct? Same for getSwapPoolLPPrice.

gianfra-t · 2024-10-25T12:34:52Z

@TorstenStueber Thanks for the review. Well in theory I'm caching the LP price on the main loop (which involves the contract-calling) to avoid calling it more than once each block. Also have a token cache in case different Nabla's had pools with shared tokens.

I didn't know we had the possibility to know in advance the pools that would be used. In that case once we have them, I can manually map the symbol and blockchain for each pool to get the price directly.

gianfra-t · 2024-10-25T14:55:24Z

Added more optimizations. Now we are closer to ~~10 blocks/sec.~~

Never mind I did some profiling, getting the info from coinStorage already takes 200ms, that is per pool unless they share token which will not happen for a single deployment of course.

So now we fetch all pairs and filter what we need, this should have a constant 500ms per block regardless of how many pools there are.

TorstenStueber · 2024-10-26T10:46:15Z

getting the info from coinStorage already takes 200ms

So slow. I had the impression that the squids optimize access to historical data as they sync really fast. But it seems that they request block by block from the archive here.

So if we want to be precise I also don't see how we can speed this up any further. That means that running this script will pretty much exactly take one hour per day of the campaign. It's fine, we can just deploy it to the cloud and then come back two days later and check the logs (?). Thanks for all the optimization effort.

Nevertheless, I don't think we need this level of precision. It would be totally fine to cache prices for a couple of blocks as they don't change that much and at the end everything is just for calculating some one-time rewards. If we even just update prices every two minutes we can speed this up 10 times.

Would that speed up block processing? Why does it still take another 300ms to process a single block? Is database access so slow? Is it possible to cache everything just in memory and avoid using the database altogether (apart from processing specific events)?

gianfra-t · 2024-10-28T13:20:22Z

Why does it still take another 300ms to process a single block

It was taking 200ms per coin info (per pool), so reading all pairs together takes 500ms but then it will be constant for any number of pools of course.

This is the real bottleneck. If we cache and update the prices every 2 minutes like you say, or more, then it will be fine. There is also something that will take even longer, which is the call to the backstop-pool contract to get the liabilities. I didn't account for this on the profiling/test because for the range I was testing there was no deposit into backstop pool. So we should probably want to cache that too and update every so often, as prices.

Beyond that, there is nothing else to do. I may be missing something, but I don't know how to fetch the prices faster.

TorstenStueber · 2024-10-30T09:40:50Z

The backstop pool liability can only change when the backstop pool executes some action. We already track that in the event handlers of the indexer and store the liability in the database. That way there should never be a need to request it from the contract (apart from the event handlers).

gianfra-t · 2024-11-15T19:19:05Z

Added two changes to speed up:

Store in the backstop pool entity the totalPoolValue, so that we call the contract only on a relevant event.
Cache the prices not per block, but per a number of blocks.

gianfra-t added 7 commits October 17, 2024 18:06

point calculation script, first pass

1b87357

adjust for pool asset price and decimals, fix errors, refactor

3b4a06a

correct points for lp token decimals

6266595

refactor

49722ce

fix for contract instantiation with relevant block hash, nabla token …

b5c2c16

…entity fix

use Big.js for points calculation and storing

457616e

properly change points schema

0065048

gianfra-t changed the title ~~Calculate rewards points~~ [DO NOT MERGE] - Calculate rewards points Oct 22, 2024

remove logs, reorder helper functions, add csv dump

307f83a

gianfra-t marked this pull request as ready for review October 22, 2024 14:15

gianfra-t requested a review from a team October 22, 2024 14:17

gianfra-t commented Oct 22, 2024

View reviewed changes

small fixes

44db78f

fetch token price from storage, add per-block caches

29efef7

TorstenStueber approved these changes Oct 25, 2024

View reviewed changes

merged w/ main

5119c69

gianfra-t added 2 commits October 25, 2024 10:08

cache router and token

70f36ed

avoid unnessesary contract calls

9507774

fetch coin info once per block all pairs

d22bc54

regenerate schema, cache prices and backstop pool total value

14e6449

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO NOT MERGE] - Calculate rewards points #75

[DO NOT MERGE] - Calculate rewards points #75

gianfra-t commented Oct 18, 2024 •

edited

Loading

gianfra-t Oct 22, 2024

gianfra-t Oct 22, 2024

gianfra-t Oct 22, 2024

gianfra-t Oct 22, 2024

gianfra-t Oct 22, 2024

gianfra-t commented Oct 22, 2024

TorstenStueber commented Oct 24, 2024 •

edited

Loading

gianfra-t commented Oct 24, 2024

gianfra-t commented Oct 24, 2024 •

edited

Loading

TorstenStueber commented Oct 24, 2024

gianfra-t commented Oct 24, 2024

ebma commented Oct 24, 2024

gianfra-t commented Oct 24, 2024

TorstenStueber commented Oct 25, 2024

TorstenStueber left a comment

TorstenStueber Oct 24, 2024

gianfra-t Oct 25, 2024

TorstenStueber Oct 24, 2024

TorstenStueber Oct 24, 2024

TorstenStueber Oct 24, 2024

gianfra-t commented Oct 25, 2024

gianfra-t commented Oct 25, 2024 •

edited

Loading

TorstenStueber commented Oct 26, 2024 •

edited

Loading

gianfra-t commented Oct 28, 2024

TorstenStueber commented Oct 30, 2024

gianfra-t commented Nov 15, 2024

		@@ -0,0 +1,788 @@
		import { Abi, Bytes, encodeCall, decodeResult } from '@subsquid/ink-abi'

[DO NOT MERGE] - Calculate rewards points #75

Are you sure you want to change the base?

[DO NOT MERGE] - Calculate rewards points #75

Conversation

gianfra-t commented Oct 18, 2024 • edited Loading

Overall idea

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gianfra-t commented Oct 22, 2024

TorstenStueber commented Oct 24, 2024 • edited Loading

gianfra-t commented Oct 24, 2024

gianfra-t commented Oct 24, 2024 • edited Loading

TorstenStueber commented Oct 24, 2024

gianfra-t commented Oct 24, 2024

ebma commented Oct 24, 2024

gianfra-t commented Oct 24, 2024

TorstenStueber commented Oct 25, 2024

TorstenStueber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gianfra-t commented Oct 25, 2024

gianfra-t commented Oct 25, 2024 • edited Loading

TorstenStueber commented Oct 26, 2024 • edited Loading

gianfra-t commented Oct 28, 2024

TorstenStueber commented Oct 30, 2024

gianfra-t commented Nov 15, 2024

gianfra-t commented Oct 18, 2024 •

edited

Loading

TorstenStueber commented Oct 24, 2024 •

edited

Loading

gianfra-t commented Oct 24, 2024 •

edited

Loading

gianfra-t commented Oct 25, 2024 •

edited

Loading

TorstenStueber commented Oct 26, 2024 •

edited

Loading