SOP authoring

An SOP (standard operating procedure) tells adoe how to handle one kind of alert. It says which alerts it covers, what must be true first, which runbook to run, and how to verify the fix.

A runbook is the part that acts: a GitHub Actions workflow, an AWS Systems Manager (SSM) document, a script or an SSH command. In short, an SOP decides and a runbook acts.

Where SOPs live

In the dashboard or the API. POST /api/sops creates or replaces an SOP by name. It needs the admin or editor role. The change applies to the next alert. See the API reference.
As YAML files, on a self-hosted server. adoe loads every file ending in .yaml from the top level of the sops/ folder when it starts. It ignores .yml files and subfolders.
As drafts from your existing documents. See Discovery from GitHub and Confluence.

All three use the same format.

Anatomy of an SOP

Here is a complete example:

name: cpu_high
description: Restart the service when CPU stays high
alert_types:                 # signals this SOP handles
  - cpu_high
  - cpu_spike
preconditions:               # all must match
  - field: env
    operator: in
    value: [prod, staging]
actions:                     # runbook steps, run in order
  - type: github_action
    repo: infra
    workflow: restart-service.yml
    inputs:
      service: "{{ service }}"
      env: "{{ env }}"
    timeout: 300
validation:                  # how adoe verifies the fix
  check_type: metric
  metric: cpu_percent
  condition: "< 80"
  timeout: 180
confidence_threshold: 0.85
auto_execute: true           # false = recommend only

Field reference

Field	Purpose	Required
`name`	A unique name for the SOP.	Yes
`description`	A short summary. adoe uses it when several SOPs match one alert.	No
`alert_types`	The signals this SOP handles. adoe compares them with the alert's signal.	Yes
`preconditions`	Conditions that must all match before the SOP applies.	No
`actions`	The runbook steps, run in order.	Yes, at least one
`validation`	How adoe verifies the fix. See Verification.	Strongly recommended
`confidence_threshold`	The lowest confidence at which adoe may run this SOP on its own. Default `0.8`. Through `POST /api/sops` the default is `0.85`.	No
`auto_execute`	`true` lets adoe run the SOP without a person. `false` means adoe only recommends it. Default `false`.	No
`enabled`	`false` stops adoe from loading the SOP. Default `true`.	No
`tags`	Free-form labels for your own use.	No
`rollback`	Not run. See The rollback block.	No

Preconditions

Each precondition has a field, an operator and a value. All preconditions must match.

Operator	Meaning
`eq`	The field equals the value. This is the default.
`ne`	The field does not equal the value.
`in`	The field is one of a list of values.
`not_in`	The field is not in a list of values.
`regex`	The field matches a regular expression, starting at the first character.

You can test service, env, severity, signal, source, or any value stored with the alert, such as host. If the alert does not have the field, the precondition fails.

service ignores case, hyphens, underscores and spaces. api-backend matches api_backend.
env treats common names as the same: production matches prod, and stage matches staging.
severity preconditions apply only to SOPs with auto_execute: true. Recommend-only SOPs ignore them.

An unknown environment passes env checks. If adoe cannot tell the alert's environment, it skips env preconditions instead of failing them. This happens when the environment is unknown or a Sensu namespace such as default. Do not rely on an env precondition alone to keep an SOP away from production.

How adoe decides

adoe finds the SOPs whose alert_types match the alert's signal. An entry also matches when it appears word for word in a PagerDuty incident title.
It removes SOPs whose preconditions fail.
If no SOP is left, it escalates to a person. If one is left, its confidence is 0.95. If several are left, adoe picks one and gives it a confidence score. It uses past outcomes once it has at least 10 for that signal, and Claude before that.
It makes one of three decisions:
- Run the SOP on its own. Every condition must hold: auto_execute: true, confidence at or above the SOP's confidence_threshold, DRY_RUN off, and the organization's execution mode set to auto.
- Recommend it to a person, when confidence is at or above CONFIDENCE_THRESHOLD (default 0.8). A person can then run it from the dashboard or from Slack.
- Escalate to a person in every other case.

One more check can only make the decision more careful. If the SOP is verified by a metric and the organization has no Grafana connection, adoe recommends instead of running. It will not run an SOP on its own when it cannot read the metric that verifies it.

Runbook steps

Each entry in actions is one runbook step. The type field picks how it runs. Each step has a timeout in seconds. The default is 300.

Type	What it runs
`github_action`	A GitHub Actions workflow
`ssm`	An AWS Systems Manager Automation or Run Command document
`shell`	A command on the adoe server
`ssh`	A command on a remote host over SSH
`manual`	Nothing. It is a logged step for a person to do by hand.

Runbook steps use the credentials set on the adoe server, such as GITHUB_TOKEN and the AWS variables. TODO(owner): how hosted customers provide credentials for runbook steps

GitHub Actions

- type: github_action
  repo: org/repo            # or just "repo" to use GITHUB_ORG
  workflow: restart-service.yml
  inputs:
    service: "{{ service }}"

adoe triggers a workflow_dispatch event on the main branch. The workflow must accept workflow_dispatch and the inputs you pass.

AWS SSM

How adoe runs a document depends on its name. A name that starts with AWS-, Automation- or Custom- runs as an SSM Automation. Any other name runs as a Run Command, and needs instance_ids or targets in parameters.

# Automation document
- type: ssm
  document: AWS-RestartEC2Instance
  parameters:
    InstanceId: i-0123456789abcdef0

# Your own Run Command document
- type: ssm
  document: RestartAppService
  parameters:
    instance_ids: [i-0123456789abcdef0]
    serviceName: "{{ service }}"

Because of this naming rule, AWS Run Command documents such as AWS-RunShellScript are started as an Automation and do not run as a command today. Use an Automation document, or your own Command document with a name that does not start with these prefixes.

Shell

- type: shell
  command: "kubectl rollout restart deployment/{{ service }}"

A shell step runs on the machine that runs adoe, not on your hosts. To run a command on your own server, use ssh or ssm.

Fixed safety rules in the code check every shell command. You cannot configure them.

The first word must be on an allow-list: systemctl, service, docker, kubectl, aws, curl, wget, python, pip, npm, yarn, redis-cli, psql, mysql, mongosh, or a read-only command such as cat, grep, ls, ps or df.
Dangerous patterns are always blocked, for example rm -rf /, mkfs and dd if=/dev/zero.
Command chaining and redirection are rejected: ;, &&, ||, &, >, <, $(, backticks and line breaks.
Pipes (|) are allowed only when every part of the pipe starts with an allowed command.

SSH

- type: ssh
  command: "sudo systemctl restart {{ service }}"
  parameters:
    host: "{{ host }}"     # put a templated host here
    use_bastion: true       # connect through the bastion host

Put the target host under parameters. adoe fills in templates there, but not in a top-level host: field. Other optional parameters are username and port (default 22).

The SSH key and default user come from SSH_KEY_PATH and SSH_DEFAULT_USER (default deploy). A bastion is a jump host that adoe connects through first. Set it with SSH_BASTION_HOST, SSH_BASTION_USER and SSH_BASTION_PORT. Without SSH_BASTION_HOST, adoe connects directly even when use_bastion is true. SSH commands are checked against the same blocked patterns as shell commands, but not against the allow-list.

Manual

- type: manual
  command: "Check the queue depth in the admin console before restarting"

A manual step describes something a person does by hand. adoe writes it to the execution log and runs nothing. Use manual steps in SOPs with auto_execute: false.

Templating

Runbook steps support Jinja2 templates: {{ name }} is replaced with a value from the alert. One SOP can then serve many services.

Variables: service, env, severity, signal, source, fingerprint, alert_id, and every simple value stored with the alert, such as host.
adoe fills in templates in inputs, parameters and command.
It uses other fields as written, including repo, workflow, document and a top-level host.
Text inside a list is not templated. For example, commands: ["restart {{ service }}"] stays as written.

Verification

The validation block says how adoe checks that the fix worked. When adoe runs an SOP on its own, it resolves the incident only if this check passes. If the check fails, adoe escalates to a person and posts the reason in the alert's Slack thread.

Add a validation: block. Without one, adoe cannot verify the fix, so it will not resolve the incident on its own.

Set check_type to one of these. timeout is in seconds, with a default of 300.

metric: set metric to a PromQL query (Prometheus query language) or a metric name. Set condition to an operator and a number, such as "< 80". The operators are <, <=, >, >=, == and !=. adoe queries Grafana every 15 seconds until the condition holds or the timeout passes. It reads only the series for the alert's own host or service. No Grafana connection, no data, several series that do not match the alert, or a timeout all count as a failed check.
http: set endpoint to a URL. adoe sends a GET request from its server. The check passes on any status code below 400.
command: set command. adoe runs it on its own server. The check passes on exit code 0. The shell allow-list does not apply to this command.

When DRY_RUN is on, nothing runs, so adoe skips the metric check.

The rollback block

adoe does not run the rollback: block, and it does not undo runbook steps. When a runbook step or the verification fails, adoe escalates the incident to a person.

The schema still accepts rollback with notify_human, pagerduty_escalate, slack_channel and custom_action. The dashboard's execution preview shows notify_human and pagerduty_escalate as the plan on failure. They do not change what adoe does. To undo a change, write a separate SOP whose runbook reverts it.

Discovery from GitHub and Confluence

adoe can read the runbooks you already have and draft SOPs from them with AI. The onboarding steps offer this, or you can call the API with a login token (see Authentication).

# GitHub: scans Markdown, reStructuredText and text files
# in folders such as runbooks/ and docs/runbooks/
curl -X POST -H "Authorization: Bearer $JWT" \
  "http://localhost:8000/sops/discover/github?repos=myorg/runbooks&repos=myorg/infra-docs"

# Confluence: scans pages with SOP labels, plus any labels you add
curl -X POST -H "Authorization: Bearer $JWT" \
  "http://localhost:8000/sops/discover/confluence?spaces=OPS&labels=runbook"

# Both sources, in the background
curl -X POST -H "Authorization: Bearer $JWT" \
  "http://localhost:8000/sops/discover?github_repos=myorg/runbooks&confluence_spaces=OPS"

Discovery uses GITHUB_TOKEN and the CONFLUENCE_URL, CONFLUENCE_USERNAME and CONFLUENCE_API_TOKEN settings.

adoe saves discovered SOPs as drafts. A draft cannot run. An admin must approve it first, on the SOPs & Runbooks page in the dashboard or through the API:

# List drafts waiting for review
curl -H "Authorization: Bearer $JWT" http://localhost:8000/api/sops/drafts

# Read one draft in full
curl -H "Authorization: Bearer $JWT" http://localhost:8000/sops/{sop_id}/content

# Approve it (admin only), or reject it
curl -X POST -H "Authorization: Bearer $ADMIN_JWT" http://localhost:8000/api/sops/{sop_id}/approve
curl -X POST -H "Authorization: Bearer $ADMIN_JWT" http://localhost:8000/api/sops/{sop_id}/reject

Review each draft before you approve it. Check its runbook steps, its preconditions and its validation block. A rejected draft is kept for the record and never runs.

Reloading SOP files

After you edit SOP files on disk, reload them without a restart. This needs an admin login token.

curl -X POST -H "Authorization: Bearer $ADMIN_JWT" http://localhost:8000/sops/reload