In the first post of our Datascape: Edge-to-Cloud blog series I shared how IoT devices, such as autonomous cars, surveillance cameras, drones, wearables, sensors, and more, capture and process lots of data (Big Data) quickly for real-time action (Fast Data). In this blog post, let’s dive into one of the biggest issues facing IoT and autonomous things. I like to call it “The Trouble with Fog.”
One hint to the cause can be found in our DataScape infographic, which shows a “gap” between edges (IoT, Digital Life, Enterprise, and Research), as well as a gap between the edge and the cloud. It may look like artistic use of white space, but that space is intentional. It represents one of the largest barriers to autonomy and to the acceleration of atmospheric and oceanic research; it is a huge hassle for Hollywood remote shoots, and, bringing it closer to home, it gets in my way, making it difficult to stay connected when I drive through the valleys nestled at the foot of the Rocky Mountains on tough commute days. What is it? Yeah, it’s that darn fog.
Fog is what we call the “last mile” or the weakest bandwidth link between the edges or between the edges and clouds. As the name suggests, it extends the power of data in the cloud closer to the edge or to end users, but the trouble is that it is a thin layer. The edge simply does not have the bandwidth for every “thing” to interact with every other “thing” and perform analytics of longer tails. Data can be stored and processed at the edge for real-time decision-making, but when it comes to large amounts of data or long-term archival, the fog is literally bogged. It can consume large amounts of bandwidth and create problems with time in transit (latency), crashing, and the quality of the transit itself. In the case of on-location movie shoots, the fog doesn’t have the bandwidth, speed, security, and reliable connections that are necessary for large mobile uploads.
Here’s a single example of Big Data, Fast Data and the impact of the fog on data: look at autonomous vehicles. This type of car can generate 1GBps[1] of data from an assortment of cameras and sensors. Today’s 4G[2] edge networks, at best, can simultaneously stream 1/8th of that data to the cloud with a whopping 1Gbps of bandwidth. We are told that 5G will introduce more speed, stability, and versatility on the lines of ten times faster than 4G LTE, yet we have to wait for that. Most carriers are targeting a slow rollout beginning in 2020[3], and even with new 5G[4] edge networks, expect that car to upload significantly less than 50% of the data it generates simultaneously.
Think about it. If we need to store all 1GBps, over a year we find that easily 1PB of data needs to live in the car. That’s a lot. My car’s trunk would be full of disk drives and I’d have nowhere to keep my golf clubs. That’s not acceptable!
The good news is that not everything needs to stay on board. Using the same storage technology we have used for 50 years (caching) we can intelligently manage what stays onboard, what goes to the cloud for instant processing, and what goes to the archive. (For more on what to store at the edge versus in the cloud, see our recent blog post “Storage and the IoT: What kind of storage is needed, and where?”)
[TWEET “As IoT pushes more #bigdata to the edge, how can we move through the fog? #cloud”]The 3D View—Devices, Data, and Data Center
Data management is key. Even with NAND’s astronomical increase in capacity, thanks to technology like Western Digital’s 96-layer 3D NAND, if we golf enthusiasts want to fit our prized golf clubs in our trunks we still have to be smart about the data we keep. Some safety-critical info (i.e. black box recordings) must be kept in the car for a relatively long time and doesn’t need to be shared with the cloud. Conversely, engine performance data and traffic conditions have great utility when aggregated centrally and will be sent over the fog for storage in data centers. Finally, we want to keep 256GB to 1TB of navigation and mapping data completely in-car to ensure service even when a network is unavailable.
Gateways – how data will get from your car to the cloud
New 5G communications standards will still support mobile broadband, just like existing 4G networks, but the really revolutionary changes will be the support for massive machine-to-machine (mM2M) and ultra-reliable, low-latency communications (URLLC). These modes will allow for the data enabling connected cars, smart cities, industrial automation, and other machine-centric communications to be delivered rapidly and reliably, which means no outages while driving through the Rocky Mountain valleys, among other things. These modes will also ensure queueing of lower-priority messaging to ensure safety-critical and real-time data coming from these connected cars take priority over less critical bits of data.
…So the good news is that while there are known limitations in the 4G environment, we have our fog lights on and can see some relief in the 5G distance— (and I’ll be able to keep the golf clubs in my trunk for a while!)
In my next blog post we will take a look inside the Cloud IoT data center to see how to manage massive fleets of cars, drones, and sensors in what I like to call the Data Tsunami.
Western Digital enables the capture, access, aggregation, transformation and preservation of data through a wealth of technologies and products that bring to life new ideas and possibilities. To hear more about the exciting role of data from edge to cloud and everywhere in between, check out our CTO, Martin Fink’s keynote address from the recent Flash Memory Summit.