8 best practices for IoT data management
IoT data management applies stewardship practices to handle large volumes of sensor data, ensuring security and compliance while coordinating across teams for reliable data access.
Data management is necessary when there is a large amount of data, when data will be shared or when data needs special handling, high levels of reliability or continuous availability.
The large amount of data produced by IoT systems can come from frequent sensor sampling, for example. The sensor might only send a few bytes of data each time it's checked, but it generates a lot of data because it's sampled every microsecond. That large volume of data can also result from having many sensors or from capturing high-definition video.
Even if an IoT data project isn't generating a lot of data, it might be generating operationally crucial data, sensitive data or data important to business performance evaluation, resource planning, environmental management and compliance efforts. It might be generating data that other systems need to perform their work or that the organization must make available to partners, suppliers or customers.
What is IoT data management?
IoT data management is the application of good information stewardship practices to the data associated with IoT systems. Information stewardship derives from the idea that every byte of information entering an organization will be stored, managed and protected throughout its life according to set policy and standardized practices. It comprises several data management disciplines: data quality management, information lifecycle management, information protection, information resilience and compliance management.
Key features of IoT data management
In an IoT context, all the major aspects of information stewardship are relevant:
- Governance. This underpins a coherent system of IoT data management rather than a haphazard collection of possibly adequate practices. Without a solid framework of policy guiding activities such as data quality evaluation, data retention and disaster recovery planning, organizations have trouble ensuring that IoT data will be available and trustworthy when needed and visible only to those who need to see it.
- Data quality management. This includes both the core IoT system design -- for example, making sure a sensor can produce data with the required accuracy at the needed frequency -- and interactions between the IoT devices and the rest of the infrastructure. In an IoT context, data quality management often involves ensuring that data is stored and transmitted without losing accuracy or data points, and at whatever volume the overall system requires. This might involve adding extra error correction and ensuring lossless data transmissions, building large-scale parallel storage systems capable of handling very high data acquisition rates or preventing the use of lossy compression on some data streams.
- Information lifecycle management. This addresses all aspects of data acquisition, storage and ultimate disposition. In an IoT context, this means knowing where data streams will be coming from and at what volume, guiding each stream of data to where it will be stored for however long it's to be retained and then making sure it's archived or deleted when it should be. Data might come to a central database directly from sensors, aggregated in storage in edge facilities first or using some intermediate controllers. Data can persist for minutes, hours, days or years in databases and might be published to data lakes for analytics or integrations with other systems.
- Information protection. This requires everything from properly encrypting data in motion -- e.g., from sensors to controllers or controllers to databases -- to properly securing access to data in back-end systems. It requires clear policies defining which data needs to be protected and how, and specifying which people, roles or systems are to have access to that data and at what points in its lifecycle.
- Information resilience. This covers all aspects of guaranteeing both short-term and long-term availability of IoT data in the event of service disruptions and disasters. So, it encompasses activities ranging from redundant storage of data in an edge facility prior to its consolidation into a central database or data lake to ensuring up-to-the-minute backups of key data to external facilities to create air-gapped archives for data that must be kept available long term, e.g., for compliance reasons.
- Compliance management. This can overlap with other areas because it involves ensuring and verifiably auditing that IoT data is managed according to policy, regulation and law. It includes aspects such as data quality management, or making sure data is captured where and how it should be and not where or how it shouldn't be; information lifecycle management, which ensures data is stored or disposed of as required; information protection, which protects data according to policies and restricts access to authorized people or systems; and resilience, which guarantees data availability as dictated by policy.
Benefits of IoT data management
The main benefit of IoT data management, like all data stewardship, is that the organization has the data it needs, when it needs it, and can trust that data because it has been properly controlled and protected. Additionally, effective IoT data management helps ensure that the organization's customers, partners and suppliers can trust that the organization is handling their data and any data shared with them responsibly and appropriately. It also enables the organization to demonstrate that it's a good steward of the data if needed.
Challenges of IoT data management
The main challenge of effective IoT data management is coordinating many people and systems. IoT data management involves not just teams within different parts of IT but also groups from other areas of the organization, located in various places and even in different countries, and possibly in other organizations altogether.
At a minimum, IoT data management involves or affects the work of the following roles:
- Whoever owns the IoT systems in question. The primary stakeholders are usually outside IT and might be part of the operational technology team in a line of business.
- Data managers.
- Storage management teams.
- Network and cybersecurity teams.
- Edge, data center and cloud infrastructure teams.
- Disaster planners.
- Risk and compliance managers.
Best practices for IoT data management
Not many of these best practices are specific to IoT situations; instead, they are broader best practices that organizations must understand and apply to IoT, too:
- Understand the anticipated volume, velocity and variety of data streams stemming from IoT projects and systems.
- Ensure excess capacity in all the key components of the data management system, from end to end, whether for processing, encryption, transmission or storage of data.
- Apply zero-trust principles to IoT data; allow access only to those entities requiring it and authenticate every request for access.
- Secure IoT endpoints to protect data at the point of capture or creation, again following zero-trust principles, and encrypt data in motion and at rest.
- Understand the compliance and privacy requirements that might apply to any IoT data so that an appropriate management policy can be written.
- Understand when data can be removed from active systems and how it should be archived or removed.
- Audit for compliance with data management policies, including access controls in particular.
- Regularly conduct tabletop "war game" exercises or red-team probes to stress-test systems, policies and people's understandings of their roles and responsibilities.
John Burke is CTO and a research analyst at Nemertes Research. Burke joined Nemertes in 2005 with nearly two decades of technology experience. He has worked at all levels of IT, including as an end-user support specialist, programmer, system administrator, database specialist, network administrator, network architect and systems architect.
