First SIG Analytics Meeting: Brainstorming

[anilmurty]

Hello!

Ahead of our first sig-analytics meeting, I want to get a discussion going on what problems this sig (and any workgroups spun out of it) can tackle. What I've listed below are a list of questions I could think of (with inputs from a couple other folks) that we would look to seek answers to with analytics and data. The goal here is to just describe the problems and what they help us achieve (what and why), without worrying about the solutions (where and how) to those problems for now (we will get to that!).

I've categorized the questions into some high level buckets and added context and would appreciate any additions (via comments to this discussion thread). I'll take any responses and turn this into a ".md" file in the sig-analytics repo, ahead of our call.

Metrics that track and show network usage over time:

1. How many deployments have occurred over time?
2. How much AKT was spent on deployments over time?
3. How many leases are created over time?
4. How many leases are active at a given time?
5. How many contributors did we get to our OSS repositories over time?

Metrics that track growth in the provider network over time:

1. How many active providers are in the network over time?
2. How many total providers (active + inactive) spun up over time?
3. How much compute capacity was added over time?
4. How much memory capacity was added over time?
5. How much storage capacity was added over time?
6. What was the average up time for the providers on the network?
7. How often do providers draw funds out of escrow? (understand provider behavior, so we can figure out how to inform providers about funds so they can avoid volatility)
8. Number of providers with specific special resources - like IP leases, persistent storage, type of memory

User activation & retention metrics:

1. How many new users (new accounts) did the network see over a specific period of time?
2. How many users started a deployment but never completed?
3. How many successfully deployed for the first time?
4. On average how many times does a user deploy per week, month? (are there tiers within this?)
5. What clients are users using to deploy on to akash (percentage breakdown)?
6. How many users (wallets) were connected to akash deployments as a result of a marketing campaign (i.e attribution)?
7. What is the average amount of spend per lease, so that we can try to characterize the types of users (“kicking the tires” vs “using in production”)?
8. How many times was a given type of "special feature" requested by tenants deploying on the providers? (ip leases, persistent storage etc)
9. What was the average size of resource requested (memory, cpu, storage etc)?
10. What types of workloads are running on the providers? (understand our target user segments. Do they tend to be compute heavy or memory heavy? What type of providers should we get?)

What else?

As a side note, some of these metrics are being collected and tracked by some folks today (below).
https://cloudmos.io/akash/dashboard
https://akash.praetorapp.com/provider-status?

A potential goal for this sig to consider, is standardizing on a set of open APIs that anyone can utilize for pulling on-chain data to answer these questions.

[anilmurty]

Notes from the first sig-meeting:

Review of the questions:

1. @cloud-j-luna : Should we consider contributions to other repos (cloudmos, TF provider etc) in the metrics as well?
2. @baktun14 : Wanted to check provider stats (in terms of compute, memory and storage) over time but it wasn't possible to store and query this over time. Artur: Prometheus may be the best way to store and query these metrics.
3. @troian : One more issue with retention metric: If the user gets bids back but never sends manifest, there is no way to know that from the chain perspective but the indirect way of deducing this is to look at if the provider closed the bid over time. Open question to decide if we want to publish this metric.
4. @troian : Will be good to get a working group going for provider metrics. Anil: We have a wg-monitoring group but nothing for growth metrics. Luna: We should strive for a standardized way of displaying these metrics so they're publicly available and consistent everywhere that they are reported.

Implementation suggestions:

1. @cloud-j-luna and team have implemented a caching service on top of praetor that is able to implement an SLA of sorts for a given provider.
2. @troian : makes sense to spec things first.
3. @baktun14: Described the work they did to build the cloudmos indexer. Not OSS currently, will consider open sourcing.
4. @cloud-j-luna : Would love to be able to leverage the indexer, integrate into kafka and be able to report on status of providers.
5. @baktun14: should be able to add custom events (like uptime) for akash and add webhooks. Using the webhook to send deployments to twitter today: https://twitter.com/unusual_cosmos
6. @jinlim : Have used https://numia.xyz/ in the past. They have an indexer for cosmos and can be exported to big table. We don't have some things like what workloads are running.

Spec definition: Would suggest starting a working group focused on defining a spec for the (non-provider-monitoring) metrics once we have defined the spec for the monitoring portion in wg-provider-monitoring. Can also be handled in this sig itself + support issues

[avolon42x]

I would like to add one more metric that might be interesting: overcommitment
One of the biggest wastes' on public cloud platforms - and which I also see as provider on akash - is overcommitment. Most deployments are way more than 100% oversized to their actual usage. If we can - on a platform - level show the rate of overcommitment - this would help to adjust individual behaviour/expectation. Ultimately I'd like to see that metric implemented on a deployment level. Happy to answer any questions and feedbacks.

[anilmurty]

Great point @avolon42x - to rephrase: being able to see the average resource (cpu, mem, storage) utilization per provider, correct?

[avolon42x]

correct.
And from a tenant perspective: being able to see the utilization of my lease.

[nimnathw]

Hi guys,
Clustering the users and providers would be helpful for marketing and future plans. What is the indexer and API endpoint used to retrieve on-chain data?

[anilmurty]

hey @nimnathw - current thinking was that there isn't a use case that requires us to do joins across on-chain and provider data. What is the use case you had in mind?
Re. indexer: we are currently exploring Numia but will also consider SubQuery and Cloudmos (if they are made open source or through an API).