# WG Paid / MGTS — R20O R02 failed refresh, durable DEGRADED_POOL and controlled retry recovery proof
## R02 continuation

R20O R01 stopped in the VM130 mandatory POSIX fixture before the installer started because BusyBox is not installed on router-ops. No snapshot, VM101 mutation, failure injection, or runtime test occurred. R02 keeps the runtime test unchanged and makes the POSIX fixture require `dash`, while checking BusyBox `ash` only when that binary is available on VM130.


## Baseline

- VM101 Machine Git commit: `d0c0a77486665b0abfef8133f0cccd20c6232b46`.
- Active state after R20N R02: `NORMAL`, repair counter `0`, five healthy slots.
- R20N R02 proved four real LOCAL_REPAIR transactions, threshold `5`, watcher-owned automatic full refresh, generation replacement and counter reset.

## Scope

R20O proves only:

`FULL_POOL_REFRESH failure -> durable DEGRADED_POOL -> controlled retry -> NORMAL`

It does not implement or test SLOT_EXHAUSTED, consolidation, Direct emergency or bootstrap recovery.

## Failure injection

1. Perform one real watcher-owned LOCAL_REPAIR on `vpn2`, establishing counter `1` through the production path.
2. Create a temporary provider download command which exits nonzero.
3. Temporarily point `PROVIDER_DOWNLOAD_CMD` in `/etc/router-egress-vm101.conf` to that command.
4. Run one forced full refresh using the production orchestrator.

The injection affects only acquisition of a new HideMyName pool. It does not modify the active VPN interfaces or client data plane.

## Required failed-refresh proof

After the provider failure:

- mode is durable `DEGRADED_POOL` in runtime and persistent state;
- active generation is unchanged;
- all five runtime endpoints are unchanged and healthy;
- repair counter remains `1`;
- quarantine is preserved;
- `full_refresh_due=true` and `next_refresh_epoch` are recorded;
- Direct remains disabled.

## Controlled retry proof

- The test temporarily sets `HMN_REFRESH_RETRY_INTERVAL_SEC=120`, while leaving the production retry tick and service unchanged.
- A real-time controller tick before `next_refresh_epoch` must be a NOOP.
- The already running procd retry service must own the first due attempt while the provider blocker remains; it must launch exactly one refresh and fail, incrementing retry count once while preserving generation, endpoints and counter.
- The original VM101 config is then restored byte-for-byte and the blocker removed before the second due time.
- The same procd service must own the second due attempt and complete a full provider download/test/rank/build/activate transaction.
- No synthetic `--now-epoch` is used, so persisted refresh timestamps remain aligned with the real clock.
- Success requires a new generation, five healthy slots, `NORMAL`, counter `0`, cleared retry metadata and Direct still disabled.

## Safety and rollback

A Proxmox snapshot and a VM101 file backup are created before runtime mutation. Cleanup always restores `/etc/router-egress-vm101.conf` and removes the temporary blocker. After a production refresh transaction starts, the harness never races it with an external generation rollback. The snapshot is operator-only rollback.

## Next step

After PASS, continue with reboot persistence while degraded, then SLOT_EXHAUSTED/consolidation scenarios from the canonical local plan.