# WG Paid / MGTS — HMN Egress Recovery & Failover Architecture Status: locked for implementation. Scope: VM101 egress recovery, HideMyName pool refresh, top-5 rebalance, degraded mode, direct fail-open, external HMN access recovery. Production context: WG Paid / MGTS / pve-wg-paid. This document records the architecture agreed after STEP_049H. It is intended to be the source of truth for the next implementation phase. --- ## 1. Proven baseline before this architecture lock As of STEP_049H: - VM101 has five production HMN/AmneziaWG egress slots: - egress1 / vpn1 - egress2 / vpn2 - egress3 / vpn3 - egress4 / vpn4 - egress5 / vpn5 - Health repair watcher is installed and running. - Daily HMN refresh cron exists at 04:20. - Daily top-5 rebalance cron exists at 04:40. - Health-triggered repair was proven by intentional ifdown vpn2. - Repair replaced vpn2 endpoint and restored strict connectivity. - Post-repair checkpoint snapshots exist: - VM100 S049H100_0919 - VM101 S049H101_0919 - VM121 S049H121_0919 Important caveat found after STEP_049H: - After repair, rebalance saw the old top-ranked endpoint as desirable again. - That is incorrect if the endpoint was removed by repair after the current pool was created. - Therefore endpoint quarantine must become part of the rebalance contract. --- ## 2. Core concepts ### Slot A slot is a production egress lane, for example: egress2 / vpn2 / table 202 / fwmark 0x202 / DSCP cs5 Slot health answers: Does this production lane currently pass traffic? ### Endpoint / tunnel An endpoint is a concrete HideMyName tunnel endpoint, for example: 192.121.163.235:42530 Endpoint health answers: Is this tunnel still safe to reuse? A slot can be repaired by replacing its endpoint. ### Pool A pool is a downloaded/tested HideMyName list. Current relevant pool files: /root/hmn/cache/ok-awg1-strict-foreign-latest.tsv /root/hmn/cache/ok-awg1-strict-all-latest.tsv /root/hmn/cache/working-awg1-latest.tsv The strict TSV contract is: rank file endpoint avg_ms ping_loss config_path ### Fresh pool A pool is fresh relative to an endpoint failure if the pool mtime is newer than the quarantine/failure time of that endpoint. Fresh pool is the only source that may restore eligibility of a previously quarantined endpoint. ### Quarantine Quarantine is local state recording endpoints removed by health repair. Proposed file: /var/lib/router-egress-recovery/quarantine.tsv Proposed fields: endpoint failed_at_epoch failed_at_iso slot iface reason old_pool_mtime_epoch replacement_endpoint Meaning: This endpoint must not be selected by rebalance unless a newer successful HMN strict pool contains it again. ### Daily failure counter Counts endpoint replacements/failures per day. Proposed path: /var/lib/router-egress-recovery/fail-counter/YYYY-MM-DD.count A successful or attempted repair replacement increments this counter according to implementation policy. ### User plane Normal customer traffic. ### Control / recovery plane Internal recovery traffic: probes, HMN refresh attempts, rescue tunnels. In direct fail-open these must be separate: users -> direct recovery/control -> tries VPN/rescue paths --- ## 3. High-level state machine NORMAL Fresh pool exists. Top-5 slots are healthy. Users go through VPN egress. SLOT_REPAIR One slot fails. Old endpoint is quarantined. Replacement endpoint is selected. Daily failure counter increments. Slot is restored if replacement works. EMERGENCY_REFRESH Daily failure counter >= threshold, initially 5. Run HMN refresh/test/validate early. If success: fresh pool becomes source of truth, top-5 rebalance runs. If fail but current VPN still works: enter DEGRADED_NO_FRESH_POOL. DEGRADED_NO_FRESH_POOL HMN fresh pool cannot be obtained. Some VPN egress still works. Users still use remaining VPN egress. System periodically retries HMN refresh, for example every 10 minutes. Failed retry does not by itself trigger direct. VPN_EXHAUSTED Fresh HMN pool cannot be obtained. Current/remaining VPN egress cannot keep users on VPN. Repair cannot find working replacements. DIRECT_FAILOPEN Users go direct. This is a user-plane fail-open, not the end of recovery. Recovery/control-plane continues in background. EXTERNAL_HMN_ACCESS_RECOVERY Pluggable recovery strategies try to regain any path to HideMyName. If a strategy succeeds, run HMN refresh/test/validate through that path. If fresh pool succeeds and top-5 strict checks pass, return users to VPN. RECOVERED Fresh pool obtained. Top-5 rebuilt. Strict checks OK. Users return from direct to VPN. State returns to NORMAL. --- ## 4. Normal daily workflow Expected normal morning sequence: 04:20 HMN refresh/test/validate 04:40 top-5 rebalance Daily rebalance rules: 1. Use strict pool by rank/avg_ms. 2. Prefer ok-awg1-strict-foreign-latest.tsv. 3. Fallback to ok-awg1-strict-all-latest.tsv. 4. Select five unique endpoints. 5. Keep current endpoints if already in top-5 to minimize churn. 6. Apply only changed slots. 7. Per-slot apply must have: - backup - ifdown/ifup - route/table check - strict ping check - rollback on failure Critical rule: Rebalance must exclude quarantined endpoints unless the pool file is newer than the quarantine event. --- ## 5. Health repair workflow When a slot fails: 1. Watcher sees slot unhealthy for threshold N cycles. 2. Dispatcher prepares repair. 3. Before replacement, old endpoint is written to quarantine. 4. Daily failure counter increments. 5. Adapter commits replacement endpoint. 6. Slot is strict-checked. 7. If success: slot is healthy again. 8. If failure: rollback or try next available candidate according to policy. The repair event must record: old endpoint new endpoint slot iface failure time reason pool used pool mtime repair result --- ## 6. Quarantine and rebalance interaction Important scenario: Pool at 04:20 says endpoint A is rank #1. At 12:00 endpoint A fails. Repair replaces A with endpoint B. Then: Daily/top-5 rebalance must NOT immediately return A just because the old 04:20 pool still ranks it #1. Correct rule: If endpoint A was quarantined at 12:00 and the pool file is older than 12:00, A is ineligible. If a later successful HMN refresh creates a strict pool at 12:30 and A appears in that fresh pool, A is eligible again. No separate per-endpoint revalidation is required in the first implementation. --- ## 7. Emergency refresh trigger Instead of individually retesting quarantined tunnels all day: If daily failure counter >= 5: run emergency HMN refresh/test/validate If emergency refresh succeeds: fresh pool becomes new source of truth quarantine older than pool is effectively cleared/reconciled top-5 rebalance runs state returns toward NORMAL If emergency refresh fails: do not immediately direct fail-open enter DEGRADED_NO_FRESH_POOL if VPN still works Guardrails: refresh lock: no parallel HMN refresh cooldown: avoid repeated emergency refresh storms log every attempt record state.kv Proposed state file: /var/lib/router-egress-recovery/state.kv Fields: mode=normal|slot_repair|emergency_refresh|degraded_no_fresh_pool|direct_failopen|external_hmn_access_recovery reason=... since_epoch=... last_refresh_attempt_epoch=... last_refresh_rc=... last_successful_pool_mtime_epoch=... daily_failure_count=... --- ## 8. DEGRADED_NO_FRESH_POOL behavior Meaning: HMN refresh failed, but VPN egress is not fully dead. Behavior: users -> still use working VPN slots repair -> may use remaining last-known candidates rebalance -> does not return quarantined endpoints from stale pool HMN refresh retry -> every ~10 minutes with lock/cooldown Important: A failed retry does not by itself cause direct fail-open. Direct fail-open only happens when: HMN fresh pool is unavailable AND VPN egress is exhausted VPN exhausted means one or more of: no healthy slots repair cannot find working replacement strict checks fail for all usable VPN egress remaining last-known candidates are exhausted or fail --- ## 9. DIRECT_FAILOPEN behavior Critical nuance: HideMyName/serverlist access is only possible through VPN. Therefore: After users are moved to direct, HMN refresh through direct is expected to be useless. Direct fail-open means: user traffic -> direct recovery/control traffic -> continues trying to obtain VPN/rescue path It does NOT mean: stop all VPN recovery --- ## 10. External HMN access recovery layer The last recovery step must be pluggable. It must not be hardcoded as "probe old HMN pool only". Abstract contract: external_hmn_access_recovery: input: current state last-known pools quarantine available rescue interfaces configured rescue strategies output: success=true|false recovery_strategy=name recovery_iface=... hmn_access_ok=true|false If a strategy provides HMN access: run HMN refresh/test/validate through that strategy/path if fresh strict pool succeeds: run top5 rebuild strict-check VPN slots switch users back to VPN Initial strategy: last_known_hmn_pool_probe Possible future strategies: dedenevo_wg_rescue client_rescue_wg known_good_external_egress manual_operator_rescue The global state machine must not change when new strategies are added. Only the strategy registry/adapter list changes. --- ## 11. Last-known HMN pool probe strategy First implementation candidate for external_hmn_access_recovery. Behavior: 1. Users remain direct. 2. Pick a rescue lane, initially vpn1. 3. Iterate candidates from last-known pools: - strict foreign - strict all - working pool 4. For each candidate: - set candidate on rescue iface - ifdown/ifup rescue iface - strict ping through rescue iface 5. If any candidate works: - run HMN refresh/test/validate through this rescue VPN path 6. If refresh success: - top-5 rebuild - users back to VPN 7. If all candidates fail: - remain direct_failopen - retry later Routing requirement: User traffic stays direct. Recovery HMN traffic must be forced through the rescue VPN path. This may require one of: curl/wget bound to interface if supported policy mark/table for recovery process temporary route/table network namespace-like isolation if available Do not implement by globally moving default route back to VPN while users are in direct. --- ## 12. Direct fail-open recovery exit criteria Exit from direct only after all are true: 1. HMN access restored through a recovery strategy. 2. HMN refresh/test/validate succeeded. 3. Fresh strict pool exists. 4. Top-5 rebuild applied or confirmed noop. 5. Production VPN slots strict OK. 6. User policy can safely return to VPN. Never exit direct based only on: one ping succeeded one tunnel came up operator hope old pool metadata --- ## 13. Implementation phases from here ### Phase 1 — State and quarantine Implement: quarantine.tsv fail-counter/YYYY-MM-DD.count state.kv Patch repair path: on successful/attempted endpoint replacement: record old endpoint in quarantine increment daily fail counter Patch rebalance: exclude quarantined endpoints if pool_mtime <= quarantine_failed_at Acceptance test: after 049G2 repair, rebalance dry-run must NOT try to return old vpn2 endpoint until a newer pool exists. ### Phase 2 — Emergency refresh trigger Implement: if daily failures >= 5: run guarded emergency refresh if success -> top5 rebalance if fail -> degraded_no_fresh_pool Acceptance test: simulate counter >=5 emergency runner attempts refresh once lock/cooldown works state.kv updates ### Phase 3 — Degraded retry timer Implement: degraded mode retry every ~10 minutes no parallel refresh failed retry keeps degraded successful retry exits degraded Acceptance test: force degraded flag timer executes guarded retry does not touch user direct unless VPN exhausted ### Phase 4 — Direct fail-open state Implement: user traffic policy -> direct state=direct_failopen recovery remains active Acceptance test: users/direct policy toggles without disabling recovery daemons clear exit condition documented and guarded ### Phase 5 — Pluggable external HMN access recovery Implement strategy runner: strategies.d/ 10-last-known-hmn-pool-probe future: 20-dedenevo-wg-rescue future: 30-client-rescue-wg Acceptance test: direct_failopen can run strategy dry-run strategy success triggers HMN refresh path global state machine unchanged by adding strategies --- ## 14. Open design questions 1. Exact user-plane direct implementation: - Which VM101 policy/routing switch controls users direct? - How to ensure recovery traffic is isolated? 2. HMN refresh through specific iface: - Does downloader support binding to iface? - If not, implement recovery mark/table wrapper. 3. Last-known pool order in direct recovery: - strict foreign first? - strict all second? - working third? - include quarantined or not? - likely yes only in direct rescue mode, because all normal options are exhausted. 4. Emergency threshold: - initial value: 5 failures/day. - cooldown: initial value 2 hours for emergency refresh. - degraded retry: 10 minutes. 5. Direct fail-open threshold: - define VPN exhausted precisely in code. --- ## 15. Non-goals for first implementation Do not implement first: per-endpoint hospital/revalidation daemon complex weighted scoring of failed endpoints automatic use of future Dedenёvo/client rescue tunnels global network restart unbounded refresh loops First implementation should be simple: quarantine daily counter emergency refresh degraded retry direct failopen state pluggable recovery strategy interface one initial strategy: last-known HMN pool probe --- ## 16. Safety principles 1. Never return a just-failed endpoint based on stale pool. 2. Never direct fail-open merely because HMN refresh failed. 3. Never stop recovery just because users are in direct. 4. Keep user-plane and recovery-plane separate. 5. Every apply path needs backup and rollback. 6. Every state transition must be logged. 7. New recovery strategies must be pluggable, not hardwired into the core state machine.