The Cloud-Enabled Space Weather Platform

By Everett Toews

February 14, 2011

Space weather is the result of solar winds interacting with the Earth’s magnetosphere. The most visible effect of space weather is the phenomenon of the Aurora Borealis (i.e. the Northern Lights). Space weather research is diverse in scope and includes, among a host of related questions to explore, the study of the impact of space weather on satellites orbiting Earth. Now more than ever scientists require a scalable, robust platform to study the magnetosphere.

The purpose of the Cloud-Enabled Space Weather Platform (CESWP) project is to bring the power and flexibility of cloud computing to space weather physicists.

The goal is to lower the barriers for the physicists to conduct their science–that is, to make it easier to collaborate with other scientists, develop space weather models, run simulations, produce visualizations and enable provenance. Success of the project is measured by the broad acceptance and use of the platform by the space weather science community.

The community of platform users includes space weather physicists who are developing models to help us better understand space weather and the magnetosphere. The principal investigator for CESWP is Dr. Robert Rankin, Professor in the Department of Physics at the University of Alberta. In addition to the University of Alberta, the institutions that are connected are Peking University, the University of California Los Angeles (UCLA), the University of New Brunswick, and Sharcnet, which is a high performance computing center run out of the University of Windsor. The availability zone at the University of Alberta acts as the CESWP Cloud Controller, initially handling all requests to operate on cloud resources.

The CESWP application itself is running in a virtual machine on a node controller in the CESWP cloud. Users visit the application as they would any normal web site from a web browser on their desktop machine, laptop, tablet or smart phone. Users of the cloud platform are presented with a view that is a simple HTML web page rendered in their browser and interact with the application by submitting requests to the controller from the view. Depending on the nature of the request, the controller may load models from the database or initiate an asynchronous call to the CloudService to perform a cloud-based operation. The results of the request are then passed to a view, which is sent back to the users as the response.

In essence, this project is building a cloud for this international community of physicists and given the nature of cloud computing, infrastructure can be geographically distributed. For the purposes of this project, a wide area network (WAN) is required to carry the traffic. For the Cloud-Enabled Space Weather Platform (CESWP), Canada’s Advanced Research and Innovation Network (CANARIE) fills this role. CANARIE is a dedicated network of high-speed, fiber optic cable that stretches across Canada and links researchers throughout Canada and around the world.

To operate an IaaS cloud you require a software framework on which it will run. For CESWP, Eucalyptus was initially selected as the cloud framework but the Virtual Computing Lab, OpenNebula, Nimbus and Eucalyptus were also considered during the survey of cloud management software at the project outset during the end of 2009.

Ultimately, Eucalyptus was selected based on both technical merits and long-term prospects. Among the technical reasons was its support for the Kernel Virtual Machine (KVM) and the Amazon Web Services (AWS) application programming interface (API). Support for the AWS API was particularly attractive, as the option to operate as a hybrid cloud with AWS was important.

Infrastructure as a Service is a complex issue and, as a consequence, Eucalyptus is a complex piece of software. Eucalyptus provided an environment to experiment with IaaS, however, given that IaaS is still a relatively green field, new prospects, such as OpenStack, did not exist at the time of the initial survey even though they still warrant an evaluation.

The scientists can connect to their virtual machines via secure shell (SSH) or NoMachine.  NoMachine is remote access software with support for a graphical user interface (GUI). This means that researchers can use NoMachine to access their VMs and be presented with a GUI desktop as if they were sitting in front of a physical machine. NoMachine has been of significant value to the CESWP project and for these purposes has been a good choice for connecting to cloud based VMs with a graphical user interface.

To ease the interaction with the CESWP cloud a web application written in Groovy and Grails was developed. Groovy is an open source dynamic language for the Java Virtual Machine that is fully compatible with the Java language itself. Grails is an open source web application framework that applies principles like convention over configuration to improve development productivity.
To develop CESWP, the team followed the principles of Scrum. Scrum is an agile software development methodology focused on the iterative development of a product. Iterations are organized into a three-week “sprint” wherein the team creates a working product increment. That product increment is delivered to the researchers as a demonstration in order to solicit feedback.

Technically speaking, CESWP itself is composed of three distinct parts. First, there is the CESWP cloud.  The cloud is comprised of servers, network and Eucalyptus. These resources combine together to provide the Infrastructure as a Service that the physicists will use to conduct their science.

Second, there is the CESWP toolkit. This toolkit is included on every VM image in the cloud and is comprised of scripts, both bash and Python, which ease interaction with the CESWP cloud.  These scripts perform such functions as running simulations in parallel to perform parameter sweeps, bundling VM images and enabling provenance.

Third, there is the CESWP application. The application is primarily developed with of the Groovy programming language, the Grails web framework and the Scrum agile development methodology. These resources combine together to create a GUI web application with a wizard-based interface that the physicists will use to interact with the CESWP cloud.

The primary motivation of the Cloud-Enabled Space Weather Platform is to lower the barriers for the physicists to conduct their science. There are a number of factors that obstruct the space weather physicists from concentrating on their science. The CESWP cloud paired with the CESWP application and toolkit addresses each of these barriers.

First, the specification and acquisition of hardware is consolidated in the CESWP cloud. Instead of specifying and purchasing desktop or server hardware for individual researchers on a piecemeal basis, the purchasing can be done in bulk for the benefit of the entire group. The procurement of this bulk hardware could still take weeks or months but is done only once per period as opposed to multiplying this effort for every researcher whenever the need arises.

Second, the maintenance of the hardware is consolidated in the CESWP cloud. Naturally, the scientists will always require hardware with which to access the cloud resources. However, there will no longer be a need for each researcher to have a laptop on which to develop their models and a high-powered desktop machine on which to run their simulations. The cloud effectively replaces the need for each researcher to maintain separate sets of hardware.

Third, the CESWP cloud has currently limited the selection of operating systems to Ubuntu and CentOS on the virtual machine images. This prevents a proliferation of operating systems and enables the cloud administrator to focus support effort on these two. In our experience, the physicists do not particularly care what operating system they are running, as long as it comes with the precompiled and preconfigured software that they prefer to use so we tailored the operating system VM images built for our community of users.

Fourth, one of the most time consuming aspects of the scientists work is getting code to compile properly. Be it their code when they have moved to a new machine or the code of a framework or tool they would like to use. The CESWP cloud can reduce the amount of time consumed in both cases. Once a researcher has compiled their own code in a VM, they can then bundle that VM into another VM image. The new image can be instantiated any number of times without having to recompile the code.  Likewise, once a framework (e.g. the Space Weather Modeling Framework) or tool has been compiled on a VM, it to can be bundled and made available to all of the physicists without recompilation.

Fifth, when a researcher’s model has reached a certain level of maturity he may want to run that model as a simulation on a grid or cluster to take advantage of high performance computing.  This is of great benefit to the researchers as their code can be processed many times more quickly than on their local machine. However, grid resources are in high demand and, therefore, scientists often must submit their jobs to a queue in order to be processed. The wait time can be hours to weeks. Instead, utilizing the CESWP toolkit, the researchers can run their simulations on the CESWP cloud now. If the CESWP cloud is low on resources the researchers can burst onto the Amazon Web Services (AWS) cloud, for a price.

In addition to lowering barriers CESWP also enables global access to a researcher’s virtual machine and access to their VM from a variety of devices. By default the virtual machines are globally accessible via secure channels. The VMs can be secured with either usernames and passwords or usernames and private keys. All traffic to and from the VM is encrypted. This global access can come from a variety of devices such as smart phones, tablets, laptops, netbooks or desktops.  Global access also enables collaboration as the researcher’s colleagues can be invited to access the VM to share data and collaborate on model development.

The space weather physics community is internationally distributed. Having cloud computing resources geographically close to its users lowers latency. This improves their experience interacting with the cloud. In addition, having all of the researchers using a single internationally distributed cloud will make it easier to share resources and collaborate.

Provenance is also enabled in the CESWP cloud. When a VM is bundled for provenance, using the CESWP toolkit, all of a physicist’s model code and data are bundled along with it. That is, the virtual machine’s file system is saved into another VM image. Also, a snapshot is taken of all of the virtual machine’s attached storage.Thus the preserved instance becomes a reproducible record of the scientists work. It is exact, complete, and unambiguous.

Work continues on Cloud-Enabled Space Weather Platform. The CESWP application is at the prototype stage and requires further refinement for production usage. The CESWP toolkit has only begun and scripts are routinely being added to enhance its utility. Naturally, we would like to expand the CESWP cloud by adding compute and storage capacity. The projected date for putting the Cloud-Enabled Space Weather Platform into production is June 2011.

Although this is the Cloud-Enabled Space Weather Platform, what we are building is not necessarily specific to space weather. Ultimately, we hope to build a platform that could potentially be used by other scientific disciplines.

About the Author

Everett Toews received his Bachelor of Science in Computing Science from the University of Alberta and has been working in the software development industry for the past decade. 

Everett has been working at Cybera for the past year and a half where he is building a cloud for use by space weather physicists at the University of Alberta on a project that goes by the name CESWP.

More available about Cybera can be found here.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “pre-exascale” award), parsed out additional information ab Read more…

By Tiffany Trader

Tsinghua Crowned Eight-Time Student Cluster Champions at ISC

June 22, 2017

Always a hard-fought competition, the Student Cluster Competition awards were announced Wednesday, June 21, at the ISC High Performance Conference 2017. Amid whoops and hollers from the crowd, Thomas Sterling presented t Read more…

By Kim McMahon

GPUs, Power9, Figure Prominently in IBM’s Bet on Weather Forecasting

June 22, 2017

IBM jumped into the weather forecasting business roughly a year and a half ago by purchasing The Weather Company. This week at ISC 2017, Big Blue rolled out plans to push deeper into climate science and develop more gran Read more…

By John Russell

Intersect 360 at ISC: HPC Industry at $44B by 2021

June 22, 2017

The care, feeding and sustained growth of the HPC industry increasingly is in the hands of the commercial market sector – in particular, it’s the hyperscale companies and their embrace of AI and deep learning – tha Read more…

By Doug Black

HPE Extreme Performance Solutions

Creating a Roadmap for HPC Innovation at ISC 2017

In an era where technological advancements are driving innovation to every sector, and powering major economic and scientific breakthroughs, high performance computing (HPC) is crucial to tackle the challenges of today and tomorrow. Read more…

At ISC – Goh on Go: Humans Can’t Scale, the Data-Centric Learning Machine Can

June 22, 2017

I've seen the future this week at ISC, it’s on display in prototype or Powerpoint form, and it’s going to dumbfound you. The future is an AI neural network designed to emulate and compete with the human brain. In thi Read more…

By Doug Black

Cray Brings AI and HPC Together on Flagship Supers

June 20, 2017

Cray took one more step toward the convergence of big data and high performance computing (HPC) today when it announced that it’s adding a full suite of big data and artificial intelligence software to its top-of-the-l Read more…

By Alex Woodie

AMD Charges Back into the Datacenter and HPC Workflows with EPYC Processor

June 20, 2017

AMD is charging back into the enterprise datacenter and select HPC workflows with its new EPYC 7000 processor line, code-named Naples, announced today at a “global” launch event in Austin TX. In many ways it was a fu Read more…

By John Russell

Hyperion: Deep Learning, AI Helping Drive Healthy HPC Industry Growth

June 20, 2017

To be at the ISC conference in Frankfurt this week is to experience deep immersion in deep learning. Users want to learn about it, vendors want to talk about it, analysts and journalists want to report on it. Deep learni Read more…

By Doug Black

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Tsinghua Crowned Eight-Time Student Cluster Champions at ISC

June 22, 2017

Always a hard-fought competition, the Student Cluster Competition awards were announced Wednesday, June 21, at the ISC High Performance Conference 2017. Amid wh Read more…

By Kim McMahon

GPUs, Power9, Figure Prominently in IBM’s Bet on Weather Forecasting

June 22, 2017

IBM jumped into the weather forecasting business roughly a year and a half ago by purchasing The Weather Company. This week at ISC 2017, Big Blue rolled out pla Read more…

By John Russell

Intersect 360 at ISC: HPC Industry at $44B by 2021

June 22, 2017

The care, feeding and sustained growth of the HPC industry increasingly is in the hands of the commercial market sector – in particular, it’s the hyperscale Read more…

By Doug Black

At ISC – Goh on Go: Humans Can’t Scale, the Data-Centric Learning Machine Can

June 22, 2017

I've seen the future this week at ISC, it’s on display in prototype or Powerpoint form, and it’s going to dumbfound you. The future is an AI neural network Read more…

By Doug Black

Cray Brings AI and HPC Together on Flagship Supers

June 20, 2017

Cray took one more step toward the convergence of big data and high performance computing (HPC) today when it announced that it’s adding a full suite of big d Read more…

By Alex Woodie

AMD Charges Back into the Datacenter and HPC Workflows with EPYC Processor

June 20, 2017

AMD is charging back into the enterprise datacenter and select HPC workflows with its new EPYC 7000 processor line, code-named Naples, announced today at a “g Read more…

By John Russell

Hyperion: Deep Learning, AI Helping Drive Healthy HPC Industry Growth

June 20, 2017

To be at the ISC conference in Frankfurt this week is to experience deep immersion in deep learning. Users want to learn about it, vendors want to talk about it Read more…

By Doug Black

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Just how close real-wo Read more…

By John Russell

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the cam Read more…

By John Russell

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its a Read more…

By Tiffany Trader

Google Pulls Back the Covers on Its First Machine Learning Chip

April 6, 2017

This week Google released a report detailing the design and performance characteristics of the Tensor Processing Unit (TPU), its custom ASIC for the inference Read more…

By Tiffany Trader

CPU-based Visualization Positions for Exascale Supercomputing

March 16, 2017

In this contributed perspective piece, Intel’s Jim Jeffers makes the case that CPU-based visualization is now widely adopted and as such is no longer a contrarian view, but is rather an exascale requirement. Read more…

By Jim Jeffers, Principal Engineer and Engineering Leader, Intel

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Nvidia’s Mammoth Volta GPU Aims High for AI, HPC

May 10, 2017

At Nvidia's GPU Technology Conference (GTC17) in San Jose, Calif., this morning, CEO Jensen Huang announced the company's much-anticipated Volta architecture a Read more…

By Tiffany Trader

Facebook Open Sources Caffe2; Nvidia, Intel Rush to Optimize

April 18, 2017

From its F8 developer conference in San Jose, Calif., today, Facebook announced Caffe2, a new open-source, cross-platform framework for deep learning. Caffe2 is the successor to Caffe, the deep learning framework developed by Berkeley AI Research and community contributors. Read more…

By Tiffany Trader

Leading Solution Providers

MIT Mathematician Spins Up 220,000-Core Google Compute Cluster

April 21, 2017

On Thursday, Google announced that MIT math professor and computational number theorist Andrew V. Sutherland had set a record for the largest Google Compute Engine (GCE) job. Sutherland ran the massive mathematics workload on 220,000 GCE cores using preemptible virtual machine instances. Read more…

By Tiffany Trader

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

US Supercomputing Leaders Tackle the China Question

March 15, 2017

Joint DOE-NSA report responds to the increased global pressures impacting the competitiveness of U.S. supercomputing. Read more…

By Tiffany Trader

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

Groq This: New AI Chips to Give GPUs a Run for Deep Learning Money

April 24, 2017

CPUs and GPUs, move over. Thanks to recent revelations surrounding Google’s new Tensor Processing Unit (TPU), the computing world appears to be on the cusp of Read more…

By Alex Woodie

DOE Supercomputer Achieves Record 45-Qubit Quantum Simulation

April 13, 2017

In order to simulate larger and larger quantum systems and usher in an age of “quantum supremacy,” researchers are stretching the limits of today’s most advanced supercomputers. Read more…

By Tiffany Trader

Messina Update: The US Path to Exascale in 16 Slides

April 26, 2017

Paul Messina, director of the U.S. Exascale Computing Project, provided a wide-ranging review of ECP’s evolving plans last week at the HPC User Forum. Read more…

By John Russell

Knights Landing Processor with Omni-Path Makes Cloud Debut

April 18, 2017

HPC cloud specialist Rescale is partnering with Intel and HPC resource provider R Systems to offer first-ever cloud access to Xeon Phi "Knights Landing" processors. The infrastructure is based on the 68-core Intel Knights Landing processor with integrated Omni-Path fabric (the 7250F Xeon Phi). Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This