State of Sector Infos

[ZenGround0]

Sector infos by far dominate the size of the filecoin state: https://ipfs.io/ipfs/QmdymZ81atAuvZShbpbmGC1dTiWBBw1W2t4PDkPhw3Ev6r 58G of ~75G of state.

I recently did a quick analysis of what is taking up all the space and have some numbers that will be generally useful for anyone planning to modify sector info fields or looking for opportunities to reduce filecoin state.

Data storage by sector info field

This is the result of adding up the serialized bytes of every field of sector info (live and dead) and summing together:

SectorInfo byte usage by field summary:
        SectorNumber: 2623968119
        SealProof: 594209831
        SealedCID: 26739442395
        DealIDs: 837734183
        Activation: 2954631477
        Expiration: 2971049155
        DealWeight: 604284985
        VerifiedDealWeight: 934359872
        InitialPledge: 5942098310
        ExpectedDayReward: 5348792651
        ExpectedStoragePledge: 5492457502
        ReplacedSectorAge: 600182052
        ReplacedDayReward: 606388570
        SectorKeyCID: 660911367
        SimpleQAPower: 594209831

        Total: 58098930131

About half of sector info storage is used for CommR
InitialPledge and the two termination fee calculation fees each contribute ~10% for a total of 30%

SectorNumber, Activation and Expiration are each about 5% for a total of 15%

The remaining fields share the last 10%

SectorInfo categorizations

601212 total miners processed
live: 304455373
dead: 289754458
fault: 1349870
has-deals: 43126445

Almost half of sector infos are dead and could be cleaned up with compaction

Reducing state size

Some ideas

Do better at removing dead sectors

It is probably scalable to remove expired and terminated sector infos during cron. We should investigate whether there is some correctness reason not to do this, address any correctness issues, double check that it is not too much cron gas and then implement this.

Simplify termination fee

The termination fee calculation is taking 20% or 12GiB of state. We can likely reduce this by sacrificing some precision in the termination fee calculation.

Move sector info off chain

This one is not easy or concrete. It is a design direction not a concrete proposal.
We could make the whole sector info private as explored in the Private Sector Info section here: #530

[ZenGround0]

One last idea: we could potentially reduce precision of pledge and termination fee fields by a lot and lose almost nothing. There's not much value in the long list of digits near the attofil range.

[ribasushi]

One more idea. Given:

• how expensive various hashes are to store
• many new sector infos will carry an additional, equally expensive, CommD post-direct-data

Would it not make sense to have a general "CID Actor" which gives out c01234 adreses for various CIDs one expects to be repeated multiple times within the state, just like the general f0xxxx actor registry?

[ZenGround0]

That is a really cool idea. Users could register a CID if they expect to reference / store it more than once.

I'm not necessarily expecting lots of repeated cids in state but if that does come to pass we should do this.

[ZenGround0]

many new sector infos will carry an additional, equally expensive, CommD post-direct-data

Also this isn't for sure yet though Im hoping to get it included

[ribasushi]

I'm not necessarily expecting lots of repeated cids in state

Replicated CommD's in state would be referring to multiple sectors... ideally 😅

[jennijuju]

Replicated CommD's in state would be referring to multiple sectors... ideally 😅

what does this mean? like have a map knowing all the sectors that has the same piece of data?

[ZenGround0]

Other ideas that have been floating around that I forgot to write down:

• Increase incentive for compaction somehow, for example forcing compaction before move or paying out from a small pool of funds for compaction (could implement this on FVM for example)
• Make compaction safe for public calling so that anyone can go in and clean this up if willing to pay the gas costs
• Solve rent generally to create the incentive for compaction by consistently charging for byte usage
• efficient GC of miner actors that no longer have any active sectors -- i.e. by just zeroing out their sectors amt

[jennijuju]

can people say get a small discount on the termination penalty if they do the compaction along the termination as well?

[ZenGround0]

Not a bad idea to explore. However most sectors expire naturally so there's only so much termination incentives will do.

[anorth]

Termination fee is 20% of state?! OMG so keen to simplify it.

[steven004]

I still think that we could significantly simply this by removing slashing and termination fee from the core, but let the user-defined market to do it, which could be off-chain.

[steven004]

This is not only for simplification of the core, reducing the state storage, but significantly lowering the barrier of miners to join the network by reducing the complex of finance planning and risk cost, which will immediately bring more miners/investors joining the network, and we will see the network power growth to a new level.

[arajasek]

Thanks for writing this up! The large number of dead sector infos has been motivating some discussion in the Lotus team about simply cleaning them all up in a one-time migration that runs in a network upgrade.

I prototyped this over the weekend in order to see the impact it would have, and the results are encouraging. Running the compact partitions logic on every single mutable deadline[1], of every single miner, results in the size of a single state tree decreasing from ~65 GiB to ~42 GiB, a saving of 35%! In addition to reduced disk usage, this saving would also be felt in the time to download & import a snapshot, as well as the monthly $$ of infrastructure that hosts snapshots. Further, this operation is fast enough to run in a migration, taking only on the order of a few minutes without any cached premigrations.

It may of course be contentious to "operate" on every miner actor in a network upgrade, though I believe this effect should have no impact on miner operations. If we wanted to consider a more conservative approach, we can only apply this compaction to "inactive" miners, which I define as those miners whose DeadlineCronActive field is false[2]. Applying this operation to this subset of miners results in smaller, but still substantial savings: shrinking from 65 GiB to 56 GiB, a reduction of 14%.

[1] One important caveat here is that the results are based on skipping "immutable" deadlines (which, IIUC, at any given time are 2/48 deadlines), but NOT skipping disputable deadlines (which at any given time are 30/48 deadlines). If we do this, we may not be able to correctly dispute proofs for a 15 hour window after the migration (specifically invalid proofs that were submitted, but not yet disputed, before the migration). I would like to argue that this is fine, but we may need to resolve this.

[2] If we were to adopt this approach, to be completely safe, we will probably want to only compact those miners whose DeadlineCronActive field is false in the current epoch and the epoch 24 hours ago.

[arajasek]

Can you expand on why you think we can't do that? I don't necessarily disagree, but AFAIC tell it's safe and unintrusive to do so.

We can certainly also consider just clearing out sectors AMTs.

[anorth]

Can you expand on why you think we can't [automatically compact partitions]?

Partition configuration is something that might affect SP operations. While they have only a little control over partition content at the moment, the fact that certain sectors must be Window PoSTed together could well affect hardware configuration, datacentre layout etc. Automatically re-arranging them could mess with operations.

The partition mapping could also be stored in a miner's off-chain metadata (e.g. indexed sector->partition). Even if Lotus doesn't do that, some SPs might. That metadata would be invalidated by a forced compaction.

Yes, if we give SPs long and loud enough warnings, they could prepare both these things. But it strikes me as a risk and burden that's not worth the network benefit, especially if we can achieve most of the benefit (dropping sector infos) without re-arranging partitions.

[arajasek]

Yup, that makes sense. The Lotus team is currently asking users what the impact of such a hypothetical "compact all partitions" would be, but I think I agree that a more scoped change is appropriate here (compacting all partitions was just easier to prototype since the code was directly useable).

[jennijuju]

Partition configuration is something that might affect SP operations....the fact that certain sectors must be Window PoSTed together could well affect hardware configuration, datacentre layout etc. Automatically re-arranging them could mess with operations.

@filecoin-project/lotus-tse How big of a concern is this in practice/real operation for SP? remove dead sectors would not only save network states, but also save gas cost with any SPs d2d operation that does write/read to sector info (most operations lolol) - would that be something SP actually be okay with?

[rjan90]

impact of such a hypothetical "compact all partitions" would be

I have maybe some concerns about compacting all partitions and its potential impacts on SP operations:

• It could potentially lead to unwanted holes in the sector bitfields in partitions. This is true for SPs that has taken on deals with a shorter expiration (and these sectors have expired) compared to their usual lifetime of sectors for example.
  • It would be nice to understand how bad a bitfield has to be before the aditional gas-cost from it would outweigh the gas-savings. If even possible.
• We had a case where a recovering windowPoSt was unable to prove due to GasLimit was higher then the block limit. One initial theory was that the sector number allocation was so bad that it was the cause. This theory was invalidated, but it would be good to understand where the bounderies are.