The defining characteristic of IoT data is that it is time-sensitive. A temperature spike in a server room, a pressure drop in a pipeline, or a machine vibration anomaly on a production line requires immediate response — not a report the next morning. This real-time requirement drives fundamentally different architectural choices compared to traditional batch analytics.
The Lambda architecture — which combines a speed layer for real-time processing and a batch layer for historical analysis — was the dominant pattern for IoT data systems for many years. Apache Kafka served as the ingestion backbone, Apache Storm or Spark Streaming handled real-time processing, and Hadoop or Spark batch processed historical data. This architecture works but requires maintaining two processing codebases.
The Kappa architecture simplifies this by using a streaming system for both real-time and historical processing. Kafka's log retention capability allows historical data to be replayed through the same streaming pipeline that handles live data, eliminating the batch layer. Apache Flink's stateful stream processing API handles the complex aggregations and pattern detection that would previously have required a batch system.
For edge-to-cloud IoT architectures, the processing hierarchy is: on-device filtering removes noise and reduces transmission volume, edge gateways perform local aggregation and time-critical alerting, and the cloud handles long-term storage, ML model training, and cross-site analytics.
Time-series databases deserve special mention as the storage layer for IoT real-time data. InfluxDB's line protocol and TimescaleDB's PostgreSQL extension are both optimised for high-write-throughput time-stamped data and provide efficient time-range queries that relational and document databases handle poorly at IoT scales.
The measurement that matters for real-time IoT systems is end-to-end latency: the time from a sensor reading to an operational response. For control applications this should be under 100ms. For alerting applications, under five seconds. For operational dashboards, under thirty seconds. Design your architecture to meet the latency requirements of your highest-priority use case.