Skip to main content
zolotoy.dev

Service case study

Notification Service

An event-driven delivery service: durable inbox, at-least-once consumption, bounded delivery workers and retry with backoff — designed around duplicates, poison messages and recovery.

PlannedBackend runtime: No deployed runtime — specification stageDemo data source: Mock

Scope
A backend engineering case study of reliable event consumption and outbound notification delivery for the zolotoy.dev demo environment.
Not in scope
  • Not a campaign or marketing platform.
  • Not an email/HTML template builder.
  • Not an aggregate analytics product.
Engineering focus
At-least-once processing, durable jobs, retry/DLQ semantics, bounded workers.

Engineering Focus

At-least-once processing, durable jobs, retry/DLQ semantics, bounded workers.

Current Implementation

Backend implementation is not present yet. The demo module renders the planned flows with deterministic mock data; the content below is the proposed boundary, not a shipped system.

  • Current

    No backend implementation exists yet: the demo renders deterministic MSW scenarios of the planned event → job → attempts model.

    • Notification service specification (target contract)docs/backend-specs/03_notification_service.mdzolotoy-dev-frontend (this repository)reviewed 2026-09-17

Target Architecture

  • Target

    A durable inbox with deduplication by event ID so at-least-once delivery from the broker never produces duplicate processing.

    • Notification service specification (target contract)docs/backend-specs/03_notification_service.mdzolotoy-dev-frontend (this repository)reviewed 2026-09-17
  • Target

    Transactional job creation: delivery jobs are recorded atomically with inbox commits, so a crash never loses or orphans a notification.

    • Notification service specification (target contract)docs/backend-specs/03_notification_service.mdzolotoy-dev-frontend (this repository)reviewed 2026-09-17
  • Target

    Bounded worker pool with per-attempt records, exponential backoff with jitter, terminal dead-job states and lease-based recovery of abandoned work.

    • Notification service specification (target contract)docs/backend-specs/03_notification_service.mdzolotoy-dev-frontend (this repository)reviewed 2026-09-17

Architecture / Core Flow

Target
  • Order servicePublishes order events via outbox
  • KafkaEvent transport (planned infrastructure)
  • Inbox + dedupIdempotent event consumption
  • Delivery jobsTransactional records with attempts
  • Delivery workersBounded concurrency, backoff, leases
  • Email providerExternal API
  • order events: order → kafka (planned)
  • at-least-once consumption: kafka → inbox (planned)
  • transactional job creation: inbox → jobs (planned)
  • lease + attempt records: jobs → worker (planned)
  • send + record result: worker → provider (planned)
Target notification flow: events to durable delivery jobs.

Engineering Decisions

At-least-once everywhere; deduplication is the consumer’s job

Target design: the broker and workers only promise at-least-once. The inbox deduplicates by event ID, and send-result recording is idempotent per attempt.

Kafka DLQ vs dead delivery jobs are different tools

Target design: broker-level DLQ handles structurally unprocessable messages; application-level dead jobs handle provider-rejected or permanently failing deliveries with retry policies of their own.

Leases recover abandoned processing jobs

Target design: a worker holds a time-boxed lease; if it crashes, the lease expires and another worker takes over. Send-success-then-crash duplicates are resolved by idempotent result recording.

Reliability / Failure Modes

Duplicate events
The same event ID delivered twice (broker retry, consumer rebalance) must commit once in the inbox; the second copy is acknowledged as already-processed.
Poison messages
Messages that fail deserialization or validation permanently must not block the partition: bounded retries, then dead-letter with the reason recorded.
Transient vs permanent provider errors
429/5xx/timeouts retry with backoff and jitter; 4xx validation failures of the provider are terminal and move the job to dead with the provider reason preserved.
Send success then process crash
Provider accepted the email but the worker died before recording it: retry produces a duplicate email. Bounded by idempotency keys where the provider supports them and by honest "at-least-once" wording.
Abandoned processing jobs
Jobs stuck in "processing" after a worker death are reclaimed after lease expiry; the number of reclaim cycles is capped before the job is declared dead.

Evidence

  • Notification service specification (target contract)docs/backend-specs/03_notification_service.mdzolotoy-dev-frontend (this repository)reviewed 2026-09-17

Interactive Demo

Next Milestone: Order events → Notification

Transactional outbox/event contract, inbox/jobs, delivery workers, retries and recovery.

Acceptance evidence (planned):

  • Outbox records commit atomically with order state changes
  • Duplicate event delivery commits once in the inbox