After Dark Systems / Service Automation HubCompletely Programmable Infrastructure

OpenSAH / Farm tool guides

goodfarm: reviewed repairs.

Bind badfarm findings to a vpscfgfarm contract, get the exact request approved, submit it once, and verify recovery with a later scan.

Available Ships with badfarm v0.4.0 Apache 2.0

What it is

goodfarm is the repair-planning and recovery module of badfarm. It turns selected badfarm findings into an immutable, content-addressed repair plan bound to a vpscfgfarm service contract, asks vpscfgfarm to approve the exact request, submits it once, reconciles the result without blind retries, and then checks recovery against a fresh badfarm scan. Use it when a finding should be fixed through a reviewed, recorded change rather than by hand.

What it is not. goodfarm never turns a finding's recommendation into a shell command, and it holds no authority of its own. It invokes the sanctioned vpscfgfarm wrapper only for explicit repair-protocol operations, and every vpscfgfarm gate (governor, deny floor, scopes, freeze, host posture) still applies. Only the service-deploy/v1 recipe can execute today. Plan files and checksums are integrity checks, not signatures; actor fields are audit labels.

Requirements and install

  • badfarm installed (goodfarm ships in the same distribution; see install badfarm). PyYAML is needed for executor and YAML work: install with '.[yaml]' or '.[yaml,iac]'.
  • An existing badfarm store. goodfarm needs one for everything except help, version and the recipe catalog.
  • A vpscfgfarm checkout with a governor, the service's contract, and a clean, committed service source tree, passed as --vpscfg-repo.
goodfarm --version
goodfarm recipes

Without installing, use python3 -m badfarm.goodfarm. --db works before or after the command and defaults to the same per-user store as badfarm ($BADFARM_DB overrides it).

The workflow

plan→ validate→ request→ approve→ submit→ scan→ verify

At any point, goodfarm next PLAN_OR_ID prints where a plan is and the exact next command. It is read-only.

Recipes

A recipe defines its prerequisites, changes, rollback and checks in code. goodfarm recipes lists them:

RecipeBound fromStatus
service-deploy/v1A service finding with explicit ownershipExecutes through the contract's existing deploy path: lock, live outage check, fresh backup, sync, build, contract verify, drift confirmation, unlock.
distro-security-upgrade/v1software.pending-security-updatesCan be planned, described and approved. Submitting fails closed: no package is installed until execution is graduated.
reboot/v1health.reboot-requiredCan be planned and approved. Submitting fails closed: no reboot command is sent.

Host-fault findings (kernel faults, disk faults, OOM, reboot storms, failed units) stay in the badfarm store for review and investigation cases. They do not become recipes.

Ownership and restore evidence

Start from a fresh, complete badfarm scan using the same check policy and datasets you will use to verify. A recipe plan needs that versioned evidence; historical scans and missing coverage do not qualify.

An ownership file binds service contracts to target scopes and assets. Service names must match vpscfgfarm contracts, dependencies must be acyclic, criticality is 1 to 5, and exposure is public, internal, isolated or unknown:

{
  "schema_version": 1,
  "services": [
    {
      "service": "shop",
      "target": "ssh://web1",
      "asset_ids": ["asset-REPLACE_WITH_OBSERVED_ID"],
      "depends_on": [],
      "shared_resources": ["volume:shared-assets"],
      "criticality": 4,
      "exposure": "public"
    }
  ]
}

For service-deploy/v1, the vpscfgfarm contract must have repo, sync.include, sync.never, backup and verify steps, and an explicit blast_radius.max_other_containers. Supply a restore drill record from the last 30 days. It is operator-supplied evidence, reviewed as part of the request:

{
  "kind": "restore-drill",
  "host": "web1",
  "service": "shop",
  "verified": true,
  "tested_at": "REPLACE_WITH_ACTUAL_RFC3339_TIME",
  "reference": "REPLACE_WITH_RESTORE_DRILL_EVIDENCE_REFERENCE",
  "verifier": "alice"
}

Plan and validate

goodfarm plan bf-FINDING_ID \
  --target ssh://web1 --service shop --vpscfg-repo ../vpscfgfarm \
  --ownership ownership.json --rollback-proof drill.json \
  --actor alice --rollback 'Restore the tested image and pre-change archive' \
  --ttl 3600 --output repair.json

goodfarm validate repair.json --vpscfg-repo ../vpscfgfarm

A plan binds the assessment, the selected findings, ownership, recipe, restore record, expected asset replacements and executor and source descriptors. The TTL is 60 to 86400 seconds, and the source assessment can be at most 24 hours old. validate checks stored evidence, expiry and current executor inputs. A new scan invalidates a pending plan even if the findings look unchanged. A passing validation means the plan can be reviewed, not that it is authorised.

To review a replacement in advance (for example a container that will come back with a new identity), add --transitions transitions.json. Each replacement must match exactly one newly observed asset, and the old asset must be gone.

Approve and submit

goodfarm request gf-PLAN_ID --output request.json
goodfarm approve gf-PLAN_ID --vpscfg-repo ../vpscfgfarm \
  --approve-digest sha256:REPLACE_WITH_REQUEST_DIGEST
goodfarm submit gf-PLAN_ID --vpscfg-repo ../vpscfgfarm \
  --approval REPLACE_WITH_EXECUTOR_APPROVAL
  • request exports the exact versioned executor request and prints its request_digest.
  • approve asks the vpscfgfarm governor to approve that digest. For service-deploy/v1 it runs a read-only live preflight through the wrapper. It returns a single-use approval value: keep it private and out of tickets and logs. Approval is governed by local OS access to the executor, not a separate identity service. Changing the plan needs a new approval.
  • submit sends the approved request once. The approval is consumed atomically with the creation of the executor operation. Preflight expires after 300 seconds and is repeated under the host lock; changed policy, source or live state refuses execution.

Status, history and resume

goodfarm status gf-PLAN_ID
goodfarm history gf-PLAN_ID
goodfarm resume gf-PLAN_ID --vpscfg-repo ../vpscfgfarm

goodfarm records submitted before calling the executor. A timeout or dropped connection becomes execution-unknown, and resume only queries the operation ID; it never resubmits. Submitting the same request again returns the existing receipt instead of repeating the change. A missing receipt is not permission to retry: inspect the executor journal and host lock first. The executor's operation database is journal/repair.sqlite3 in the vpscfgfarm checkout. Back it up with SQLite's backup facilities and never delete it to clear an error.

Verify recovery

badfarm scan ssh://web1 --checkpoint after-repair
goodfarm verify repair.json --checkpoint after-repair

Verification compares stored checkpoints; it does not scan by itself. Use the same data and policy options as the original scan.

  • recovery_verified needs every selected finding resolved, complete comparable checks, no unknown findings, no new or worse findings, and the selected assets observed.
  • execution_verified also needs a matching applied executor receipt and a scan taken after execution.

Only both claims move a repair to recovered. A clean scan without a receipt is observed-recovery. Recovery is scoped to the selected findings; it does not certify the whole target.

Canary rollouts

badfarm prioritize --ownership ownership.json
goodfarm rollout create gf-CANARY gf-FOLLOWER --canary gf-CANARY --max-hosts 3
goodfarm rollout next rollout-ID
goodfarm rollout submit rollout-ID --vpscfg-repo ../vpscfgfarm \
  --approval REPLACE_WITH_NEXT_PLAN_APPROVAL

A rollout orders recipe plans by dependency, with one canary first. Host bounds are 1 to 10 and concurrency is one. Every member needs its own exact approval, and only verified recovery of the earlier member releases the next. Unknown execution, refusal, rollback, incomplete verification or a regression halts the rollout. goodfarm does not scan in the background: you collect fresh evidence between members.

Service source IaC review

Every new plan records an iac-review gate. When the contract names a repo, goodfarm scans that tree in-process with badfarm's IaC scanner; nothing in it runs. Critical and high findings, author-written suppressions of them, and an incomplete scan refuse the plan. Accept a reviewed location explicitly:

goodfarm plan bf-FINDING_ID ... --accept-iac iac.tf.open_ingress:net/sg.tf:12 \
  --accept-iac-reason 'Bastion ingress, reviewed by ops'

Exit codes

ExitMeaning
0Success
1Unresolved, halted, refused or uncertain operation result
2Malformed or stale input, or a controller error
130Interrupted

An explicit executor refusal without a receipt can appear as a controller error; inspect it before acting again.

How it fits with the other tools

badfarm supplies the findings and the later checkpoint. goodfarm plans and coordinates. vpscfgfarm approves and executes through its repair protocol (--repair-action describe, approve, apply and status on its deploy command), with every gate in force. Scheduling repairs as routine work is farmops's planned role and does not exist yet.

Built from the public badfarm repository (GOODFARM.md, README.md and the goodfarm --help and recipes output) at commit 99bad35, and the vpscfgfarm repository (docs/REPAIR-PROTOCOL.md) at commit e899b93.