Observability

Monitor Herald with structured logging via slog, the message delivery log, and the event hook system.

Herald provides observability through three mechanisms: structured logging using Go's log/slog, a persistent message delivery log stored in the database, and event hooks that fire on key lifecycle events.

Structured Logging with slog

Herald uses Go's standard log/slog package for all internal logging. Log messages are structured with key-value pairs, making them easy to filter, search, and aggregate in log management systems.

Configuring the Logger

When using Herald directly, pass a logger with the WithLogger option:

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelDebug,
}))

h, err := herald.New(
    herald.WithStore(myStore),
    herald.WithLogger(logger),
    // ... drivers
)

If no logger is provided, Herald defaults to slog.Default().

Log Messages

Herald emits structured log messages at key points in the delivery pipeline:

Debug level -- Provider resolution and delivery details:

h.logger.Debug("herald: resolved provider via fallback",
    "provider", p.Name,
    "channel", channel,
    "app_id", appID,
)

h.logger.Debug("herald: user opted out",
    "user_id", req.UserID,
    "template", req.Template,
    "channel", channel,
)

Warn level -- Non-fatal issues during delivery:

h.logger.Warn("herald: notify channel failed",
    "channel", ch,
    "template", req.Template,
    "error", err,
)

h.logger.Warn("herald: failed to seed template",
    "slug", tmpl.Slug,
    "channel", tmpl.Channel,
    "error", err,
)

h.logger.Warn("herald: invalid provider ID in scoped config",
    "provider_id", pidStr,
    "scope", scopeType,
    "scope_id", scopeID,
)

These structured fields make it easy to build queries in your logging platform:

Filter by channel to see all email delivery events
Filter by template to track a specific notification type
Filter by app_id to isolate a single application
Filter by error to find all delivery failures

Message Delivery Log

Every notification delivery attempt is persisted in the messages table. This provides a complete audit trail of all notifications sent through Herald.

Message Entity

type Message struct {
    ID          id.MessageID      `json:"id"`
    AppID       string            `json:"app_id"`
    EnvID       string            `json:"env_id,omitempty"`
    TemplateID  string            `json:"template_id,omitempty"`
    ProviderID  string            `json:"provider_id,omitempty"`
    Channel     string            `json:"channel"`
    Recipient   string            `json:"recipient"`
    Subject     string            `json:"subject,omitempty"`
    Body        string            `json:"body,omitempty"`
    Status      Status            `json:"status"`
    Error       string            `json:"error,omitempty"`
    Metadata    map[string]string `json:"metadata,omitempty"`
    Async       bool              `json:"async"`
    Attempts    int               `json:"attempts"`
    SentAt      *time.Time        `json:"sent_at,omitempty"`
    DeliveredAt *time.Time        `json:"delivered_at,omitempty"`
    CreatedAt   time.Time         `json:"created_at"`
}

What Gets Logged

For every recipient of every Send call, Herald creates a message record capturing:

Who: Recipient, AppID, EnvID
What: Channel, TemplateID, Subject, Body (truncated)
How: ProviderID, Attempts
Result: Status, Error, SentAt, DeliveredAt
Context: Metadata (application-specific key-value pairs)

The Body field is truncated to TruncateBodyAt characters (default: 4096) to keep the delivery log compact while retaining enough content for debugging.

Status Tracking

Messages progress through a lifecycle of states:

Status	Description
`queued`	Message created, pending delivery (async mode)
`sending`	Delivery attempt in progress
`sent`	Driver accepted the message
`delivered`	Delivery confirmed by the provider
`failed`	Delivery failed
`bounced`	Recipient address bounced

The status is updated via UpdateMessageStatus as the delivery progresses:

// Before sending
msg.Status = message.StatusSending
_ = h.store.CreateMessage(ctx, msg)

// After driver returns
if sendErr != nil {
    _ = h.store.UpdateMessageStatus(ctx, msg.ID, message.StatusFailed, sendErr.Error())
} else {
    _ = h.store.UpdateMessageStatus(ctx, msg.ID, message.StatusSent, "")
}

Querying the Delivery Log

Messages can be queried through the store or API with filtering and pagination:

messages, err := store.ListMessages(ctx, appID, message.ListOptions{
    Channel: "email",
    Status:  message.StatusFailed,
    Limit:   50,
    Offset:  0,
})

API Endpoints

Method	Path	Operation
`GET`	`/messages`	List messages (query: `app_id`, `channel`, `status`, `limit`, `offset`)
`GET`	`/messages/:id`	Get a single message by ID

Example: list all failed email deliveries:

GET /herald/messages?app_id=app_01abc...&channel=email&status=failed&limit=20

Store Interface

type Store interface {
    CreateMessage(ctx context.Context, m *Message) error
    GetMessage(ctx context.Context, messageID id.MessageID) (*Message, error)
    UpdateMessageStatus(ctx context.Context, messageID id.MessageID, status Status, errMsg string) error
    ListMessages(ctx context.Context, appID string, opts ListOptions) ([]*Message, error)
}

Monitoring Patterns

Failed Delivery Alerting

Query the messages table periodically for failed deliveries to trigger alerts:

failed, err := store.ListMessages(ctx, appID, message.ListOptions{
    Status: message.StatusFailed,
    Limit:  100,
})

if len(failed) > threshold {
    // Trigger alert
}

Delivery Rate Tracking

Use the CreatedAt and SentAt timestamps to measure delivery latency:

for _, msg := range messages {
    if msg.SentAt != nil {
        latency := msg.SentAt.Sub(msg.CreatedAt)
        // Record latency metric
    }
}

Per-Channel Health

List messages filtered by channel and status to monitor the health of individual channels:

// Check email delivery success rate
allEmail, _ := store.ListMessages(ctx, appID, message.ListOptions{Channel: "email", Limit: 1000})
failedEmail, _ := store.ListMessages(ctx, appID, message.ListOptions{Channel: "email", Status: message.StatusFailed, Limit: 1000})

successRate := 1.0 - float64(len(failedEmail)) / float64(len(allEmail))

Per-Provider Tracking

Each message records the ProviderID that was used for delivery. This allows you to track provider-level reliability:

msg, _ := store.GetMessage(ctx, messageID)
// msg.ProviderID tells you which provider handled this delivery
// msg.Error tells you what went wrong (if failed)

Health Check

Herald exposes a Health method that pings the underlying store to verify database connectivity:

err := h.Health(ctx)
if err != nil {
    // Store is unreachable
}

When running as a Forge extension, this is integrated into the Forge health check system automatically.

Observability

On this page