Purpose

Make sure we’re aware of all operational readiness items we need to complete before passive testing and activation. This helping drive https://github.com/filecoin-project/go-f3/issues/722to close.

2025-03-24

  1. Review the recovery steps in F3 Failure Scenarios and Disaster Recovery
    1. Do they seem right?
    2. Which ones do we need runbooks for?
    3. Desired output: list of runbooks we need to create
  2. Notes
    1. (Assuming a post activation world)
    2. Producing snapshots
      1. What data is missing from snapshots today? 1. 2. If f3 freezes 1. People have a habbit of wiping Lotus nodes 2. We’ll need to provide a way for SPs to get this data (power tables) 1. This is a new tool we’d need to create 3. We think it would take a few hours to create the tool 4. The impact if we don’t have the tool in advance 1. F3 is already frozen, and this is just pushing out the time to resolution 5. Cacade of things that would need to fail 1. Ohshitstore : activates when 900 epochs behind and keeps foreven until we recover 2. Users need to wipe their nodes
      2. Summary: edge case and not going to hit a cliff. We’ll just be reactive here.
    3. Disabling/pausing F3
      1. Use environment variables to disable F3 (e.g, Use LOTUS_DISABLE_F3_ACTIVATION=ContractAddress)
      2. TODO: create this small runbook
    4. Adjusting parameters
      1. Depends on what is the root cause / what happened
      2. Generally these are going to need to be a coordinated effort to upgrade together
    5. Topic of how to “emergency upgrade” (very slim scope)
    6. Snapshot discussion
      1. Usecases
        1. Distribute cert chain
        2. Where you need to have some state in order to progress
      2. With headloockback = 4, the possibility of “By the time finality processing ends a heavier chain exists that is selected by non-participating SPs” is low
      3. Today
        1. download snapshot
        2. sync cert chain
      4. Future (https://github.com/filecoin-project/go-f3/issues/480)

2025-03-20

  1. Review F3 Launch Planning Notes from 2024 and see if there’s anything we need to carry over and do for this latest round.
    1. Example: Make a new node release with the power table that the first f3 instance is based on (TODO: we need operator docs on how we expect people to update their nodes and/or config their nodes with env car)
      1. 🎬 Create a v1.32.1 release placeholder issue that will have initial power table CID and the activation parameters
        1. ✅ https://github.com/filecoin-project/lotus/issues/12970
    2. Discussion on power table metric
      1. Total power in bytes as seen by F3 for the currently instance being processed
      2. Purpose: for detecting power swings
      3. 🎬 Create task to add this metric
        1. ✅ https://github.com/filecoin-project/go-f3/issues/925
    3. Discussion on how we can do cleanup after the activation (but this doesn’t have to get rushed out right after activation. It should be done right away but can go out with the next normal Lotus release)
      1. 🎬 Create cleanup issue to start accumulating the things to delete! w00t w00t!
        1. ✅ https://github.com/filecoin-project/lotus/issues/12971
  2. Ensure we’re good with how we’ll start an incident
    1. Start an incident
    2. Who else do we need to communicate this to?
  3. What are the potential things that could go wrong?
    1. Starter list: ‣
    2. For each item
      1. How will we know?
      2. What would we do?
  4. F3 Failure Scenarios and Disaster Recovery
    1. Discussion around CPU usage
      1. Idea: run a ddos against our devnet
      2. Knobs that we have at our disposal?
        1. None
    2. 🎬 Need to update based on openings from chain exchange
      1. ✅ Tracking item: https://github.com/orgs/filecoin-project/projects/114/views/2?pane=issue&itemId=102954830
  5. Dicussion of next steps
    1. Metrics
    2. Unit tests
    3. L/R side hashing
    4. ddos testing our infra network (this informs grouping)
    5. Grouping (planning to punt this - too complex of a change too close to the release)
  6. On the attack
    1. We have opened ourselves to Sybil
      1. To handle messages that anyone can send
      2. Reduce potency by reduce the CPU consumption
        1. Kuba’s work (merkle tree optimization)
          1. We did 2x reduction
          2. We think we can get another 2x squeeze
        2. Also, reduce the amount you need to hash (grouping)
      3. Goal here is to not make spam attack vector worse
    2. Not critical right now but we want to have it ready to release at the minimum
      1. Continuing with release train items
    3. We’ll get data by having the adversary