Gregor
中文
← All guides

Get an alert when a cron job fails

Wrap any scheduled command so a failure lands in your inbox with its exit code and the last lines of output.

Published cronshell

Scheduled jobs fail quietly: the backup that stopped running three weeks ago, the certificate renewal nobody noticed. A few lines of shell turn every failure into an alert.

1. Create a send URL

Create a channel for your server (or use your device’s own channel), then create a send URL for it named after the machine, for example nas cron. Store it in a file only root can read:

Terminal window
sudo install -m 600 /dev/null /etc/gregor-send-url
echo 'https://<your-server>/v1/send/<token>' | sudo tee /etc/gregor-send-url >/dev/null

2. Add a wrapper script

Save this as /usr/local/bin/alert-on-failure and make it executable with chmod +x:

#!/bin/sh
# Usage: alert-on-failure <job name> <command> [args...]
name="$1"; shift
output=$("$@" 2>&1)
status=$?
[ "$status" -eq 0 ] && exit 0


tail=$(printf '%s' "$output" | tail -n 5)
body=$(printf 'Exit code %s on %s\n%s' "$status" "$(hostname)" "$tail")


jq -n --arg title "$name failed" --arg body "$body" --arg subject "cron:$(hostname):$name" \
  '{alert: {title: $title, body: $body}, level: "high",
    event: {type: "cron.failed", subject: $subject}}' |
  curl -fsS -X POST "$(cat /etc/gregor-send-url)" \
    -H 'Content-Type: application/json' --data-binary @- >/dev/null
exit "$status"

The script runs the job, does nothing when it succeeds, and otherwise sends a high alert with the exit code and the last five lines of output. It uses jq to build the JSON, so quotes and newlines in the output cannot break the request.

3. Wrap your jobs

Prefix each crontab entry with the wrapper and a short name:

30 2 * * * /usr/local/bin/alert-on-failure "Nightly backup" /usr/local/bin/backup.sh
0 4 * * 1 /usr/local/bin/alert-on-failure "Renew certificates" certbot renew --quiet

Test it once with a command that always fails:

Terminal window
/usr/local/bin/alert-on-failure "Test job" false

Variations

  • Make it urgent. For jobs where a failure needs someone right away, change level to "urgent". Gregor then reminds the recipients every minute until someone acknowledges the alert.
  • Group repeats. The event.subject is the same every time a job fails, so repeated failures of one job are recognized as the same event.
  • Keep secrets out of output. The last lines of output are sent in the alert body. If a job prints tokens or passwords, reduce tail -n 5 or send only the exit code.