Amazon Web Services now offers a way to move data into its cloud by the truckload. The reigning cloud king used its annual AWS re:Invent conference in Las Vegas as a launch pad to reveal a big data big rig, called Snowmobile. The exabyte-scale data transfer service will enable users to move up to 100 PB inside a 45-foot long ruggedized shipping container, pulled by a semi-trailer truck.
“This secure data truck stores up to 100 PB of data and can help you to move exabytes to AWS in a matter of weeks (you can get more than one if necessary),” wrote AWS Chief Evangelist Jeff Barr in a blog post. “Designed to meet the needs of our customers in the financial services, media & entertainment, scientific, and other industries, Snowmobile attaches to your network and appears as a local, NFS-mounted volume. You can use your existing backup and archiving tools to fill it up with data destined for Amazon Simple Storage Service (S3) or Amazon Glacier.”
During the unveiling at AWS Reinvent yesterday (Nov. 30), AWS CEO Andy Jassy observed that moving large volumes of data is not as convenient or quick as it needs to be to match the demands of this data-driven age. According to the CEO, uploading an exabyte of data would take 26 years using a 10 Gbps link. But that same amount of data could be moved in just six months using the throughput of 10 Snowmobile deliveries.
Each climate controlled shipping container is equipped with a network cable and high-speed switch permitting one terabit-per-second data dumps across multiple 40 Gbps connections. At this transfer rate, datacenters can fill up a Snowmobile unit in roughly 10 days. The truck requires about 350 kW of AC power and AWS will provide an on-site generator upon request. AWS can also arrange for dedicated security guards at the customer site and for a security vehicle escort during transit.
The transfer of electronic information using conventional ground or air transportation methods isn’t a new concept. The low-tech solution to a high-tech problem is affectionately referred to as a sneakernet, since these data transfers were often carried out on foot within companies or from one site to another. It’s an approach that seemed to be on its way out as network bandwidth increased over the last couple decades, but the pressing need to ingest, process and analyze massive data streams has given sneakernets new life.
When a semi-truck pulled an AWS Snowmobile trailer onto the stage at the Amazon re:Invent conference, the audience cheered. AWS launched Snowmobile as the much larger companion to AWS Snowball, a rugged suitcase-sized device that lets customers transfer 50 TB of data to the AWS cloud at a time. The demand for Snowball exceeded company expectations, according to Jassy, who said Amazon had to up its device orders by 10x after the rollout.
Commercial satellite vendor DigitalGlobe has already signed on to use Snowmobile. The company has a 16-year archive of high-resolution satellite imagery, visualizing 6 billion square kilometers of the Earth’s surface, that it has been transitioning to the AWS cloud.
“We have slowly been migrating our archive to AWS but that process has been slow and inefficient,” stated Jay Littlepage, vice president of infrastructure & operations at DigitalGlobe. “Our constellation of satellites generate more earth imagery each year (10 PB) than we have been able to migrate by these methods. We needed a solution that could move our 100 PB archive but could not find one until now with AWS Snowmobile. DigitalGlobe is currently migrating our entire raw imagery archive with one Snowmobile transfer directly into an Amazon Glacier Vault.”
Snowmobile is currently available in all regions and is offered as a fully-supported service. The AWS site lists the price of provisioned Snowmobile capacity at $0.005/GB per month (the math comes out to $500,000 for 100 PB). The full cost of the service has not been disclosed, but Barr reports, “we intend to make sure that Snowmobile is both faster and less expensive than using a network-based data transfer model.”
For more coverage of Snowmobile’s launch, check out “Exabytes Hit the Road with AWS Snowmobile” by Alex Woodie, managing editor of HPCwire sister publication, Datanami.