IoT ingestion that expects duplicates, disorder, and disconnections
A listener receiving messages is not yet an ingestion platform. Reliability appears when the flow preserves identity, time, errors, and replay capability.
Reference flow
- Devicesidentity · timestamp · sequence
- Listenersprotocol adapters
- Buspartitions · retention · backpressure
- Processingvalidation · deduplication · windows
- Persistenceseries · events · state
- Consumptionalerts · APIs · analytics
Semantics before technology
Each event needs stable identity, source, device time, and receive time. Without those fields, duplicates and ordering cannot be resolved defensibly.
- Idempotency through event keys or sequences.
- Ordering per device or partition; never promise global order.
- Windows and tolerance for late events.
- Schema validation before enrichment.
Failures must be recoverable
Retries must be bounded. Exhausted events move to dead-letter with enough context for diagnosis and replay; they are neither discarded nor retried forever.
- Backpressure when input exceeds processing.
- Dead-letter with cause, safe payload, and correlation.
- Controlled replay without duplicate effects.
- Disconnections treated as normal behavior.
Operational observability
Observability distinguishes received, valid, duplicate, late, rejected, and pending messages. A single throughput metric hides the failures that matter.
- Lag by partition or consumer.
- Age of the oldest pending event.
- Duplicate and rejection rates.
- Alerts around flow SLOs, not individual noise.