=== STEP_034H2 AGENT/JOB DIAGNOSTICS === timestamp=20260708-121548 mode=read-only peer_id=376dfa07-0ba4-4182-af39-389a2cb3badd job_id=95463317-b0a0-4117-84fd-2a0dcf387741 tunnel_ip=10.253.1.10 forbidden: manual peer write, job run, cancel, disable, wg reload === host === wg-access-dev Wed Jul 8 12:15:48 UTC 2026 === backend peer/job exact JSON redacted === peer_found=True { "id": "376dfa07-0ba4-4182-af39-389a2cb3badd", "user_id": "5af6e227-e772-4a29-9290-7a74aea8bb18", "node_id": "ddn-test", "public_key": "fmcqSUy1fErBV5RrcoVoex2VAgaxz2i79GzK8qd88UI=", "tunnel_ip": "10.253.1.10", "paid_until": "2026-08-07T12:12:18.368562Z", "enabled": true, "created_at": "2026-07-08T12:12:18.380667Z" } job_found=True { "id": "95463317-b0a0-4117-84fd-2a0dcf387741", "node_id": "ddn-test", "action": "enable_peer", "status": "pending", "attempts": 0, "created_at": "2026-07-08T12:12:18.382277Z", "last_error": null } job_status=pending job_action=enable_peer job_node_id=ddn-test job_peer_id_top=None job_payload=None === agent/systemd/docker inventory === cloud-init-local.service loaded active exited Initial cloud-init job (pre-networking) cloud-init.service loaded active exited Initial cloud-init job (metadata service crawler) qemu-guest-agent.service loaded active running QEMU Guest Agent wg-access-agent.service loaded inactive dead WG Access Agent real poller wg-access-expire-subscriptions.service loaded inactive dead WG Access expire subscriptions maintenance wg-access-expire-subscriptions.timer loaded active waiting Run WG Access expire subscriptions maintenance periodically --- systemctl status wg-access-agent.service --- ○ wg-access-agent.service - WG Access Agent real poller Loaded: loaded (/etc/systemd/system/wg-access-agent.service; static) Active: inactive (dead) since Mon 2026-07-06 18:48:38 UTC; 1 day 17h ago Process: 440716 ExecStart=/opt/wg-access/agent/wg_access_agent.py (code=, status=0/SUCCESS) Main PID: 440716 (code=, status=0/SUCCESS) CPU: 134ms Jul 06 18:48:38 wg-access-dev wg_access_agent.py[440716]: [2026-07-06T18:48:38.432384+00:00] fetched pending jobs: 1 Jul 06 18:48:38 wg-access-dev wg_access_agent.py[440716]: [2026-07-06T18:48:38.432485+00:00] starting job_id=c2d55542-713c-4cb2-9040-7d64bdf511bb action=disable_peer Jul 06 18:48:38 wg-access-dev wg_access_agent.py[440716]: [2026-07-06T18:48:38.904617+00:00] remote stdout: interface: wg_paid Jul 06 18:48:38 wg-access-dev wg_access_agent.py[440716]: public key: FD1J+KVZ965pCjgtO0YWxJsKANY9uz6v1P54JLkiZzY= Jul 06 18:48:38 wg-access-dev wg_access_agent.py[440716]: private key: Jul 06 18:48:38 wg-access-dev wg_access_agent.py[440716]: listening port: 51830 Jul 06 18:48:38 wg-access-dev wg_access_agent.py[440716]: [2026-07-06T18:48:38.905015+00:00] REAL disable_peer peer_id=3d03a7a2-c009-48a4-b6ff-214326d26fc0 Jul 06 18:48:38 wg-access-dev wg_access_agent.py[440716]: [2026-07-06T18:48:38.911811+00:00] completed job_id=c2d55542-713c-4cb2-9040-7d64bdf511bb status=completed attempts=1 Jul 06 18:48:38 wg-access-dev systemd[1]: wg-access-agent.service: Deactivated successfully. Jul 06 18:48:38 wg-access-dev systemd[1]: Finished wg-access-agent.service - WG Access Agent real poller. --- systemctl cat wg-access-agent.service --- # /etc/systemd/system/wg-access-agent.service [Unit] Description=WG Access Agent real poller After=network-online.target docker.service Wants=network-online.target [Service] Type=oneshot User=root WorkingDirectory=/opt/wg-access/agent Environment=WG_ACCESS_AGENT_ENV=/opt/wg-access/agent/agent.env ExecStart=/opt/wg-access/agent/wg_access_agent.py --- systemctl status wg-access-worker.service --- --- systemctl cat wg-access-worker.service --- --- systemctl status wg-access-jobs.service --- --- systemctl cat wg-access-jobs.service --- --- systemctl status wg-access-expire-subscriptions.service --- ○ wg-access-expire-subscriptions.service - WG Access expire subscriptions maintenance Loaded: loaded (/etc/systemd/system/wg-access-expire-subscriptions.service; static) Active: inactive (dead) since Wed 2026-07-08 12:13:28 UTC; 2min 20s ago TriggeredBy: ● wg-access-expire-subscriptions.timer Process: 749478 ExecStart=/usr/local/sbin/wg-access-expire-subscriptions.sh (code=, status=0/SUCCESS) Main PID: 749478 (code=, status=0/SUCCESS) CPU: 20ms Jul 08 12:13:28 wg-access-dev systemd[1]: Starting wg-access-expire-subscriptions.service - WG Access expire subscriptions maintenance... Jul 08 12:13:28 wg-access-dev wg-access-expire-subscriptions.sh[749483]: == 2026-07-08T12:13:28+00:00 expire-subscriptions == Jul 08 12:13:28 wg-access-dev wg-access-expire-subscriptions.sh[749483]: http_code= Jul 08 12:13:28 wg-access-dev wg-access-expire-subscriptions.sh[749483]: {"expired_subscription_count":0,"disabled_peer_count":0,"job_ids":[],"subscription_ids":[]} Jul 08 12:13:28 wg-access-dev systemd[1]: wg-access-expire-subscriptions.service: Deactivated successfully. Jul 08 12:13:28 wg-access-dev systemd[1]: Finished wg-access-expire-subscriptions.service - WG Access expire subscriptions maintenance. --- systemctl cat wg-access-expire-subscriptions.service --- # /etc/systemd/system/wg-access-expire-subscriptions.service [Unit] Description=WG Access expire subscriptions maintenance Wants=network-online.target After=network-online.target docker.service wg-access-agent.service [Service] Type=oneshot ExecStart=/usr/local/sbin/wg-access-expire-subscriptions.sh User=root Group=root --- systemctl status wg-access-expire-subscriptions.timer --- ● wg-access-expire-subscriptions.timer - Run WG Access expire subscriptions maintenance periodically Loaded: loaded (/etc/systemd/system/wg-access-expire-subscriptions.timer; enabled; preset: enabled) Active: active (waiting) since Sun 2026-07-05 10:19:47 UTC; 3 days ago Trigger: Wed 2026-07-08 12:18:33 UTC; 2min 44s left Triggers: ● wg-access-expire-subscriptions.service Jul 05 10:19:47 wg-access-dev systemd[1]: Started wg-access-expire-subscriptions.timer - Run WG Access expire subscriptions maintenance periodically. --- systemctl cat wg-access-expire-subscriptions.timer --- # /etc/systemd/system/wg-access-expire-subscriptions.timer [Unit] Description=Run WG Access expire subscriptions maintenance periodically [Timer] OnBootSec=2min OnUnitActiveSec=5min AccuracySec=30s RandomizedDelaySec=30s Persistent=true [Install] WantedBy=timers.target === docker containers === NAMES IMAGE STATUS PORTS wgaccess-backend python:3.12-slim Up 18 minutes 0.0.0.0:18080->8000/tcp wgaccess-postgres postgres:16-alpine Up 41 hours (healthy) 127.0.0.1:15432->5432/tcp === backend/agent logs filtered === --- docker logs wgaccess-backend --- INFO: 172.18.0.1:50034 - "GET /admin/jobs HTTP/1.1" 200 OK INFO: 172.18.0.1:56840 - "GET /admin/jobs HTTP/1.1" 200 OK INFO: 172.18.0.1:32896 - "GET /admin/jobs HTTP/1.1" 200 OK INFO: 172.18.0.1:32928 - "GET /admin/jobs HTTP/1.1" 200 OK INFO: 172.18.0.1:45022 - "GET /admin/jobs HTTP/1.1" 200 OK INFO: 172.18.0.1:58938 - "GET /admin/jobs HTTP/1.1" 200 OK INFO: 172.18.0.1:58970 - "GET /admin/jobs HTTP/1.1" 200 OK INFO: 172.18.0.1:50050 - "GET /admin/jobs HTTP/1.1" 200 OK INFO: 172.18.0.1:51202 - "GET /admin/jobs HTTP/1.1" 200 OK --- docker logs wgaccess-postgres --- === source/tools for agent processing === /opt/wg-access/backend/app/models/core.py:85: # uq_peers_node_tunnel_ip_enabled (node_id, tunnel_ip) WHERE enabled = true, /opt/wg-access/backend/app/models/core.py:91: node_id: Mapped[str] = mapped_column(String(64), default="ddn-test", nullable=False, index=True) /opt/wg-access/backend/app/models/core.py:106: __tablename__ = "provisioning_jobs" /opt/wg-access/backend/app/models/core.py:110: node_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True) /opt/wg-access/backend/app/models/core.py:120: completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) /opt/wg-access/backend/app/main.py:8:from app.api.agent import router as agent_router /opt/wg-access/backend/app/main.py:14:app.include_router(agent_router) /opt/wg-access/backend/app/services/wireguard.py:41:def next_tunnel_ip(db: Session, node_id: str) -> str: /opt/wg-access/backend/app/services/wireguard.py:59: Peer.node_id == node_id, /opt/wg-access/backend/app/config.py:7: agent_token: = "dev-agent-token" /opt/wg-access/backend/app/api/admin.py:66: node_id: str /opt/wg-access/backend/app/api/admin.py:88: node_id: str = "ddn-test" /opt/wg-access/backend/app/api/admin.py:97: node_id: str /opt/wg-access/backend/app/api/admin.py:109: # It creates a user, an active subscription, a WG peer and a pending enable_peer job. /opt/wg-access/backend/app/api/admin.py:127: tunnel_ip = next_tunnel_ip(db, payload.node_id) /opt/wg-access/backend/app/api/admin.py:134: node_id=payload.node_id, /opt/wg-access/backend/app/api/admin.py:145: node_id=payload.node_id, /opt/wg-access/backend/app/api/admin.py:146: action="enable_peer", /opt/wg-access/backend/app/api/admin.py:169: node_id=peer.node_id, /opt/wg-access/backend/app/api/admin.py:199: jobs = [] /opt/wg-access/backend/app/api/admin.py:219: node_id=peer.node_id, /opt/wg-access/backend/app/api/admin.py:220: action="disable_peer", /opt/wg-access/backend/app/api/admin.py:230: jobs.append(job) /opt/wg-access/backend/app/api/admin.py:234: for job in jobs: /opt/wg-access/backend/app/api/admin.py:240: job_ids=[job.id for job in jobs], /opt/wg-access/backend/app/api/admin.py:276: jobs = [] /opt/wg-access/backend/app/api/admin.py:282: node_id=peer.node_id, /opt/wg-access/backend/app/api/admin.py:283: action="disable_peer", /opt/wg-access/backend/app/api/admin.py:293: jobs.append(job) /opt/wg-access/backend/app/api/admin.py:297: for job in jobs: /opt/wg-access/backend/app/api/admin.py:306: job_ids=[job.id for job in jobs], /opt/wg-access/backend/app/api/admin.py:313: node_id: str /opt/wg-access/backend/app/api/admin.py:320:def disable_peer(peer_id: UUID, db: Session = Depends(get_db)): /opt/wg-access/backend/app/api/admin.py:330: node_id=peer.node_id, /opt/wg-access/backend/app/api/admin.py:331: action="disable_peer", /opt/wg-access/backend/app/api/admin.py:348: node_id=peer.node_id, /opt/wg-access/backend/app/api/admin.py:358: node_id: str /opt/wg-access/backend/app/api/admin.py:369:@router.get("/jobs", response_model=list[JobResponse]) /opt/wg-access/backend/app/api/admin.py:370:def list_jobs(db: Session = Depends(get_db)): /opt/wg-access/backend/app/api/admin.py:392: node_id: str /opt/wg-access/backend/app/api/admin.py:406: node_id: str /opt/wg-access/backend/app/api/admin.py:436: jobs: list[AdminUserJobResponse] /opt/wg-access/backend/app/api/admin.py:480: jobs = db.execute( /opt/wg-access/backend/app/api/admin.py:486: jobs = [] /opt/wg-access/backend/app/api/admin.py:494: jobs=jobs, /opt/wg-access/backend/app/api/dev.py.BEFORE_CLIENT_CONFIG_RESPONSE.20260704-151636:22: node_id: str = "ddn-test" /opt/wg-access/backend/app/api/dev.py.BEFORE_CLIENT_CONFIG_RESPONSE.20260704-151636:63:def next_tunnel_ip(db: Session, node_id: str) -> str: /opt/wg-access/backend/app/api/dev.py.BEFORE_CLIENT_CONFIG_RESPONSE.20260704-151636:66: select(func.count()).select_from(Peer).where(Peer.node_id == node_id) /opt/wg-access/backend/app/api/dev.py.BEFORE_CLIENT_CONFIG_RESPONSE.20260704-151636:78: tunnel_ip = next_tunnel_ip(db, payload.node_id) /opt/wg-access/backend/app/api/dev.py.BEFORE_CLIENT_CONFIG_RESPONSE.20260704-151636:85: node_id=payload.node_id, /opt/wg-access/backend/app/api/dev.py.BEFORE_CLIENT_CONFIG_RESPONSE.20260704-151636:96: node_id=payload.node_id, /opt/wg-access/backend/app/api/dev.py.BEFORE_CLIENT_CONFIG_RESPONSE.20260704-151636:97: action="enable_peer", /opt/wg-access/backend/app/api/dev.py:29: node_id: str = "ddn-test" /opt/wg-access/backend/app/api/dev.py:58: tunnel_ip = next_tunnel_ip(db, payload.node_id) /opt/wg-access/backend/app/api/dev.py:66: node_id=payload.node_id, /opt/wg-access/backend/app/api/dev.py:77: node_id=payload.node_id, /opt/wg-access/backend/app/api/dev.py:78: action="enable_peer", /opt/wg-access/backend/app/api/dev.py.BEFORE_REAL_WG_KEYS.20260704-132545:19: node_id: str = "ddn-test" /opt/wg-access/backend/app/api/dev.py.BEFORE_REAL_WG_KEYS.20260704-132545:36:def next_tunnel_ip(db: Session, node_id: str) -> str: /opt/wg-access/backend/app/api/dev.py.BEFORE_REAL_WG_KEYS.20260704-132545:39: select(func.count()).select_from(Peer).where(Peer.node_id == node_id) /opt/wg-access/backend/app/api/dev.py.BEFORE_REAL_WG_KEYS.20260704-132545:51: tunnel_ip = next_tunnel_ip(db, payload.node_id) /opt/wg-access/backend/app/api/dev.py.BEFORE_REAL_WG_KEYS.20260704-132545:55: node_id=payload.node_id, /opt/wg-access/backend/app/api/dev.py.BEFORE_REAL_WG_KEYS.20260704-132545:66: node_id=payload.node_id, /opt/wg-access/backend/app/api/dev.py.BEFORE_REAL_WG_KEYS.20260704-132545:67: action="enable_peer", /opt/wg-access/backend/app/api/agent.py:13:router = APIRouter(prefix="/agent", tags=["agent"]) /opt/wg-access/backend/app/api/agent.py:16:def check_agent_token(x_agent_token: | None = Header(default=None)): /opt/wg-access/backend/app/api/agent.py:17: if not x_agent_token x_agent_token settings.agent_token: /opt/wg-access/backend/app/api/agent.py:18: raise HTTPException(status_code=, detail="invalid agent token" /opt/wg-access/backend/app/api/agent.py:23: node_id: str /opt/wg-access/backend/app/api/agent.py:36: node_id: str, /opt/wg-access/backend/app/api/agent.py:38: _: None = Depends(check_agent_token), /opt/wg-access/backend/app/api/agent.py:42: .where(Peer.node_id == node_id) /opt/wg-access/backend/app/api/agent.py:52: node_id: str /opt/wg-access/backend/app/api/agent.py:68:@router.get("/jobs", response_model=list[AgentJobResponse]) /opt/wg-access/backend/app/api/agent.py:69:def get_pending_jobs( /opt/wg-access/backend/app/api/agent.py:70: node_id: str, /opt/wg-access/backend/app/api/agent.py:73: _: None = Depends(check_agent_token), /opt/wg-access/backend/app/api/agent.py:77: .where(ProvisioningJob.node_id == node_id) /opt/wg-access/backend/app/api/agent.py:85:@router.post("/jobs/{job_id}/start", response_model=AgentJobResponse) /opt/wg-access/backend/app/api/agent.py:89: _: None = Depends(check_agent_token), /opt/wg-access/backend/app/api/agent.py:106:@router.post("/jobs/{job_id}/complete", response_model=AgentJobResponse) /opt/wg-access/backend/app/api/agent.py:107:def complete_job( /opt/wg-access/backend/app/api/agent.py:110: _: None = Depends(check_agent_token), /opt/wg-access/backend/app/api/agent.py:118: job.status = "completed" /opt/wg-access/backend/app/api/agent.py:119: job.completed_at = datetime.now(timezone.utc) /opt/wg-access/backend/app/api/agent.py:127:@router.post("/jobs/{job_id}/fail", response_model=AgentJobResponse) /opt/wg-access/backend/app/api/agent.py:132: _: None = Depends(check_agent_token), /opt/wg-access/backend/app/api/agent.py:140: job.status = "failed" /opt/wg-access/backend/app/api/agent.py:141: job.completed_at = datetime.now(timezone.utc) /opt/wg-access/backend/app/api/admin.py.BEFORE_DISABLE_PEER.20260704-151253:59: node_id: str /opt/wg-access/backend/app/api/admin.py.BEFORE_DISABLE_PEER.20260704-151253:78: node_id: str /opt/wg-access/backend/app/api/admin.py.BEFORE_DISABLE_PEER.20260704-151253:89:@router.get("/jobs", response_model=list[JobResponse]) /opt/wg-access/backend/app/api/admin.py.BEFORE_DISABLE_PEER.20260704-151253:90:def list_jobs(db: Session = Depends(get_db)): /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:13:router = APIRouter(prefix="/agent", tags=["agent"]) /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:16:def check_agent_token(x_agent_token: | None = Header(default=None)): /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:17: if not x_agent_token x_agent_token settings.agent_token: /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:18: raise HTTPException(status_code=, detail="invalid agent token" /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:23: node_id: str /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:39:@router.get("/jobs", response_model=list[AgentJobResponse]) /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:40:def get_pending_jobs( /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:41: node_id: str, /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:44: _: None = Depends(check_agent_token), /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:48: .where(ProvisioningJob.node_id == node_id) /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:56:@router.post("/jobs/{job_id}/start", response_model=AgentJobResponse) /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:60: _: None = Depends(check_agent_token), /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:77:@router.post("/jobs/{job_id}/complete", response_model=AgentJobResponse) /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:78:def complete_job( /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:81: _: None = Depends(check_agent_token), /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:89: job.status = "completed" /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:90: job.completed_at = datetime.now(timezone.utc) /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:98:@router.post("/jobs/{job_id}/fail", response_model=AgentJobResponse) /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:103: _: None = Depends(check_agent_token), /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:111: job.status = "failed" /opt/wg-access/backend/app/api/agent.py.BEFORE_AGENT_PEERS_SYNC.20260704-133954:112: job.completed_at = datetime.now(timezone.utc) /usr/local/sbin/wg-access-expire-subscriptions.sh:48: *) echo "ERROR: expire-subscriptions failed"; exit 1 ;; === API paths agent/jobs from openapi === /admin/jobs ['get'] /admin/maintenance/expire-subscriptions ['post'] /admin/peers ['get'] /admin/peers/{peer_id}/disable ['post'] /agent/jobs ['get'] /agent/jobs/{job_id}/complete ['post'] /agent/jobs/{job_id}/fail ['post'] /agent/jobs/{job_id}/start ['post'] /agent/peers ['get'] /dev/create-test-peer ['post'] === health === {"status":"ok","service":"wg-access-backend","environment":"dev"} {"status":"ok","db":1} === no forbidden actions proof === no_manual_peer_write=1 no_job_run=1 no_cancel=1 no_disable=1 no_wg_reload=1 vm121_agent_diag_done=1 === VM100 read-only wg_paid snapshot === date=Wed Jul 8 12:15:49 UTC 2026 === wg_paid counts === wg_paid_peer_count=0 vm100_tunnel_ip_visible_rc=1 === recent wg_paid snippets === interface: wg_paid public key: FD1J+KVZ965pCjgtO0YWxJsKANY9uz6v1P54JLkiZzY= private key: (hidden) listening port: 51830 no_manual_peer_write_vm100=1 no_wg_reload_vm100=1 vm100_wg_paid_snapshot_done=1 === STEP_034H2 RESULT === peer_found=True job_found=True job_status=pending job_action=enable_peer job_node_id=ddn-test job_peer_id_top=None agent_or_processing_related_lines=23 error_related_lines=6 api_agent_related_lines=6 vm100_tunnel_ip_visible_rc=1 backend_health_ok_lines=2 decision=PASS_AGENT_JOB_DIAGNOSTICS