The Data Archipelago: architecture for the Internet of Things

Data Archipelago

On this day, I proclaim: modern data architecture is an archipelago.

It’s 2016 and time to recognize the failings of data lakes and warehouses. In a world where we connect wine bottles to the internet, we need a new cliché metaphor: one for the Internet of Things.

It’s time to recognize the Data Archipelago.

The archipelago solves the problem of other ideas of data storage.

All your data sources no longer flow into one location (sorry, James Dixon, but you got it wrong with Data Lake). No. Data floods into siloes: islands. These islands are your wearables, your connected toothbrushes, thousands and thousands and thousands of things all there, all separate data siloes. Separated by water.

Of course, your data is also in the cloud. Our archipelago needs clouds, too, raining data from the devices to the local storage represented by the islands.

In the archipelago data flows around like water. Water is like the air around us. It contains the information. It flows freely. Polluted water corrupts your data and Pirates, literally, can land on your islands and steal your stuff.

Modern data architecture is an archipelago.

Or maybe it’s a Data Swimming Pool.

data swimming pool

Your data’s the water. Data needs to be kept clean. Chlorine and Vacuums represent the ETL layer. Lifeguards are your disaster recovery processes. And kids who pee? Well that’s the SQL Injection attack, of course.


Leave a Reply