Integration setup

Connect your monitoring and notification tools to adoe. Each monitoring tool sends alerts to a webhook URL. adoe turns every alert into one standard format, with a service, an environment, a severity and a signal (the type of problem).

How webhook URLs work

Every alert source has its own URL. The last part of the URL is a token:

https://<your-adoe-host>/webhooks/{source}/{token}

• The token identifies your organization and authenticates the request. Your tool does not need to send an extra header.
• Create the token on the Integrations page, on the Webhooks tab. You need the admin or editor role.
• Copy the full URL from that tab. It includes your host and the token.
• Generating a new token replaces the old one. Update your tools when you do this.
• A request with an unknown token gets a 401 response.

The sources are sensu, splunk, pagerduty, grafana, uptime-com and railway.

Sensu Go

1. Create the handler

Create a pipe handler that forwards Sensu events to your adoe webhook URL. Replace the URL with the one from the dashboard.

# adoe-handler.yml (apply with: sensuctl create -f adoe-handler.yml)
type: Handler
api_version: core/v2
metadata:
  name: adoe
  namespace: default
spec:
  type: pipe
  command: >
    curl -s -X POST
    -H "Content-Type: application/json"
    -d @-
    https://<your-adoe-host>/webhooks/sensu/<your-token>
  timeout: 30
  filters:
    - is_incident

Then add the handler to the checks you want adoe to see.

2. Fields adoe reads

adoe does not read a severity label. Severity always comes from check.status. Add service and environment labels to your Sensu entities, so adoe can match each alert to the right SOP.

To let adoe read your check definitions, also connect the Sensu API on the Integrations page.

Splunk

1. Create a webhook alert action

In Splunk, open Settings, then Searches, Reports and Alerts. Create or edit a saved search. Then:

1. Under Trigger Actions, choose Add Actions, then Webhook.
2. Set the URL to your adoe webhook URL: https://<your-adoe-host>/webhooks/splunk/<your-token>.

You do not need an Authorization header. The token in the URL authenticates the request.

2. Fields adoe reads

adoe reads these fields from the top level of the JSON body. It does not read them from the result object.

adoe also stores result.host as the alert's host. TODO(owner): confirm how customers add top-level service, environment, severity and signal fields to Splunk's webhook payload

3. Name your saved searches so adoe can parse them

When the fields are missing, adoe reads them from the saved search name. Use this pattern:

Alert - <service> - <signal words> - <environment>

• Service: the word after Alert -.
• Environment: the first of prod, production, staging, stage, dev, development, qa or test found anywhere in the name.
• Signal: one of these phrases. "CPU high" or "CPU spike" gives cpu_high. "Memory high" gives memory_high. "Disk full" gives disk_full. "Error rate" gives error_rate_high. "Latency high" gives latency_high. "500 error" gives http_5xx. Any other name gives splunk_alert.

Word order matters. "CPU high" gives cpu_high, but "High CPU" gives splunk_alert.

4. Example saved search

Save this search as Alert - api-backend - CPU high - prod. adoe then reads the service, signal and environment from its name.

index=metrics host=api-backend-* earliest=-5m
| stats avg(cpu_usage) as avg_cpu by host
| where avg_cpu > 90

To let adoe read saved search definitions and recent events, also connect the Splunk API on the Integrations page.

PagerDuty

1. Create a V3 webhook subscription

1. Go to Services, then Service Directory, and select the service.
2. Open the Integrations tab. Under Generic Webhooks (V3), choose Add a webhook.
3. Set the webhook URL to https://<your-adoe-host>/webhooks/pagerduty/<your-token>. Set the scope type to Service.
4. Select the incident.triggered and incident.resolved event types.

2. Add the signing secret (optional)

After you create the subscription, PagerDuty shows a signing secret. Paste it into the PagerDuty card on the Webhooks tab of the Integrations page. Each webhook token has its own secret.

adoe then checks the X-PagerDuty-Signature header on every request. This is an HMAC-SHA256 signature: a hash of the request body made with the shared secret. A missing or wrong signature gets a 401 response. If you do not set a secret, adoe skips this check.

The same card has a Send Test Alert button. It sends a test event through your webhook, signed with your secret if you set one.

3. Events adoe handles

• incident.triggered creates an alert and starts processing it.
• incident.resolved resolves every open alert linked to that PagerDuty incident.
• adoe accepts other event types and ignores them.

adoe accepts both the V3 format (an event object) and the older V2 format (a messages list).

4. Fields adoe reads

To import past incidents and keep them in sync, also connect the PagerDuty API on the Integrations page.

Grafana

1. Add a webhook contact point

1. In Grafana, go to Alerting, then Contact points, and add a contact point.
2. Choose the Webhook integration.
3. Set the URL to https://<your-adoe-host>/webhooks/grafana/<your-token>.

Firing alerts start processing in adoe. A resolved notification closes the matching open alert. The webhook accepts any payload in the Alertmanager shape: an alerts list whose entries carry labels, annotations and status.

2. Labels adoe reads

3. Connect the Grafana API

Also connect Grafana on the Integrations page. adoe uses the API to poll for firing alerts, read metrics during an investigation, and verify a fix. If an SOP is verified by a metric and Grafana is not connected, adoe will not run that SOP on its own. It recommends the SOP to a person instead.

Uptime.com

In Uptime.com, add a custom postback URL under Integrations. Use https://<your-adoe-host>/webhooks/uptime-com/<your-token>.

• Only alert_raised events create alerts. adoe accepts other events and ignores them.
• The service is the Uptime.com check name, or else the device name.
• The signal is uptime_<check type>_<state>, for example uptime_http_down.
• A check that is down gives critical. A check that is up gives info.

Railway

In Railway, open Project Settings, then Webhooks, and add a webhook. Use https://<your-adoe-host>/webhooks/railway/<your-token>.

• Events whose type starts with Deployment., Volume., CPU. or RAM. create alerts. adoe acknowledges other events and ignores them.
• The signal is the event type in lower case with underscores. Deployment.failed becomes deployment_failed.
• The service is the Railway service name, or else the project name. The environment is the Railway environment name.
• Deployment.failed and Deployment.crashed are critical unless the event sets its own severity.

Slack

adoe posts each alert to a Slack thread and updates it as work progresses. People can approve actions from Slack and use the /adoe command.

1. Create a Slack app at api.slack.com/apps. Give the bot these scopes: chat:write, channels:history, groups:history and channels:read.
2. Install the app to your workspace. Invite the bot to your alert channel.
3. On the adoe Integrations page, open Slack. Paste the bot token (it starts with xoxb-) and the app's signing secret.
4. In the Slack app settings, point these URLs at your adoe host:
   • Interactivity request URL: /api/slack/interactions
   • Slash command URL: /api/slack/commands
   • Event subscriptions URL: /api/slack/events

adoe uses the signing secret to verify every button click, command and event from Slack.

Semaphore CI

Semaphore adds recent deployment context to alerts. adoe can then tell whether a deploy caused the alert. It looks for pipelines in the hour before the alert.

Semaphore is configured with environment variables on the adoe server, not in the dashboard. TODO(owner): how hosted customers connect Semaphore

1. Get your API token

In Semaphore CI, open Settings, then API Tokens, and create a token with read access. Then set:

SEMAPHORE_API_TOKEN=<your-api-token>
SEMAPHORE_ORG_URL=https://your-org.semaphoreci.com

2. Map services to projects

adoe finds a project on its own when its name exactly matches the service name in the alert. When the names differ, map them. First list your projects and their IDs:

curl -H "Authorization: Token <your-api-token>" \
  https://your-org.semaphoreci.com/api/v1alpha/projects \
  | jq '.[] | {name: .metadata.name, id: .metadata.id}'

Then map each service name to a project ID:

SEMAPHORE_PROJECT_MAP={"api-backend": "proj-abc123", "web-frontend": "proj-def456"}