Integration Setup

← back to docs

Sensu Go

1. Create the webhook handler

In your Sensu Go configuration, create a handler that forwards events to the agent:

# /etc/sensu/conf.d/handlers/l1-agent.yml
type: Handler
api_version: core/v2
metadata:
  name: l1-devops-agent
  namespace: default
spec:
  type: pipe
  command: >
    curl -s -X POST
    -H "Content-Type: application/json"
    -H "Authorization: Bearer $SENSU_WEBHOOK_TOKEN"
    -d @-
    https://your-agent-host/webhooks/sensu
  timeout: 30
  filters:
    - is_incident

2. Configure the auth token

# Generate a random token
TOKEN=$(openssl rand -hex 32)

# Set it in the L1 Agent .env
echo "SENSU_WEBHOOK_TOKEN=$TOKEN" >> .env

# Set the same value in the Sensu environment
export SENSU_WEBHOOK_TOKEN=$TOKEN

3. Map checks to alerts

The agent normalizes Sensu events using these fields:

Sensu field	Agent field	Example
entity.labels.service	service	api-backend
entity.namespace or entity.labels.environment	env	prod
check.name	signal	cpu_high
check.labels.severity	severity	critical
check.output	message	CPU usage is 95%

Ensure your Sensu checks include service and environment labels on the entity for proper routing.

Splunk

1. Create a webhook alert action

In Splunk, go to Settings → Searches, Reports and Alerts and create or edit a saved search:

1. Under Trigger Actions, click Add Actions → Webhook.
2. Set the URL to https://your-agent-host/webhooks/splunk.
3. Under Advanced, add a custom header: Authorization: Bearer <your-splunk-webhook-token>.

2. Configure custom fields

The agent expects these fields in the Splunk payload — add them via Custom Fields on the alert action, or include them in the search results:

Field	Required	Description	Example
service	Yes	Service name	api-backend
environment	Yes	Environment	prod
severity	Yes	Alert severity	critical, warning, info
signal	Yes	Alert signal type	cpu_high, error_rate_high
search_name	Auto	Name of the saved search	Alert - api-backend - High CPU

3. Set the auth token

TOKEN=$(openssl rand -hex 32)
echo "SPLUNK_WEBHOOK_TOKEN=$TOKEN" >> .env

Use this same value in the Authorization: Bearer <token> header in Splunk's webhook configuration.

4. Example saved search

index=metrics host=api-backend-* earliest=-5m
| stats avg(cpu_usage) as avg_cpu by host
| where avg_cpu > 90
| eval service="api-backend"
| eval environment="prod"
| eval severity="critical"
| eval signal="cpu_high"

PagerDuty

1. Create a V3 webhook subscription

1. Go to Services → Service Directory and select the service to monitor.
2. Open the Integrations tab → Add a webhook under Generic Webhooks (V3).
3. Set the Webhook URL to https://your-agent-host/webhooks/pagerduty, scope type Service, and select the incident.triggered event type.

2. Get the HMAC secret

After creating the subscription, PagerDuty provides an HMAC signing secret. Copy it and set it in the agent:

echo "PAGERDUTY_WEBHOOK_SECRET=<your-signing-secret>" >> .env

3. Event filtering

The agent only processes incident.triggered events; other types (acknowledged, resolved, etc.) are received but ignored. It extracts:

PagerDuty field	Agent field	Example
incident.service.name	service	api-backend
incident.title	Used for signal extraction	High CPU on api-backend in prod
incident.urgency	severity	high → critical

Semaphore CI

The Semaphore integration enriches alerts with recent deployment context, so the decision engine can tell whether an alert was caused by a deploy.

1. Get your API token

Log in to Semaphore CI → Settings → API Tokens, create a read-access token, then:

echo "SEMAPHORE_API_TOKEN=<your-api-token>" >> .env
echo "SEMAPHORE_ORG_URL=https://your-org.semaphoreci.com" >> .env

2. Map services to projects

List your projects to get their IDs, then map service names (as they appear in alerts) to project IDs:

curl -H "Authorization: Token <your-api-token>" \
  https://your-org.semaphoreci.com/api/v1alpha/projects \
  | jq '.[] | {name: .metadata.name, id: .metadata.id}'

echo 'SEMAPHORE_PROJECT_MAP={"api-backend": "proj-abc123", "web-frontend": "proj-def456"}' >> .env

If a service name exactly matches a Semaphore project name, the agent auto-discovers it — the project map is only needed when names differ.