The Cloud-Enabled Space Weather Platform

By Everett Toews

February 14, 2011

Space weather is the result of solar winds interacting with the Earth’s magnetosphere. The most visible effect of space weather is the phenomenon of the Aurora Borealis (i.e. the Northern Lights). Space weather research is diverse in scope and includes, among a host of related questions to explore, the study of the impact of space weather on satellites orbiting Earth. Now more than ever scientists require a scalable, robust platform to study the magnetosphere.

The purpose of the Cloud-Enabled Space Weather Platform (CESWP) project is to bring the power and flexibility of cloud computing to space weather physicists.

The goal is to lower the barriers for the physicists to conduct their science–that is, to make it easier to collaborate with other scientists, develop space weather models, run simulations, produce visualizations and enable provenance. Success of the project is measured by the broad acceptance and use of the platform by the space weather science community.

The community of platform users includes space weather physicists who are developing models to help us better understand space weather and the magnetosphere. The principal investigator for CESWP is Dr. Robert Rankin, Professor in the Department of Physics at the University of Alberta. In addition to the University of Alberta, the institutions that are connected are Peking University, the University of California Los Angeles (UCLA), the University of New Brunswick, and Sharcnet, which is a high performance computing center run out of the University of Windsor. The availability zone at the University of Alberta acts as the CESWP Cloud Controller, initially handling all requests to operate on cloud resources.

The CESWP application itself is running in a virtual machine on a node controller in the CESWP cloud. Users visit the application as they would any normal web site from a web browser on their desktop machine, laptop, tablet or smart phone. Users of the cloud platform are presented with a view that is a simple HTML web page rendered in their browser and interact with the application by submitting requests to the controller from the view. Depending on the nature of the request, the controller may load models from the database or initiate an asynchronous call to the CloudService to perform a cloud-based operation. The results of the request are then passed to a view, which is sent back to the users as the response.

In essence, this project is building a cloud for this international community of physicists and given the nature of cloud computing, infrastructure can be geographically distributed. For the purposes of this project, a wide area network (WAN) is required to carry the traffic. For the Cloud-Enabled Space Weather Platform (CESWP), Canada’s Advanced Research and Innovation Network (CANARIE) fills this role. CANARIE is a dedicated network of high-speed, fiber optic cable that stretches across Canada and links researchers throughout Canada and around the world.

To operate an IaaS cloud you require a software framework on which it will run. For CESWP, Eucalyptus was initially selected as the cloud framework but the Virtual Computing Lab, OpenNebula, Nimbus and Eucalyptus were also considered during the survey of cloud management software at the project outset during the end of 2009.

Ultimately, Eucalyptus was selected based on both technical merits and long-term prospects. Among the technical reasons was its support for the Kernel Virtual Machine (KVM) and the Amazon Web Services (AWS) application programming interface (API). Support for the AWS API was particularly attractive, as the option to operate as a hybrid cloud with AWS was important.

Infrastructure as a Service is a complex issue and, as a consequence, Eucalyptus is a complex piece of software. Eucalyptus provided an environment to experiment with IaaS, however, given that IaaS is still a relatively green field, new prospects, such as OpenStack, did not exist at the time of the initial survey even though they still warrant an evaluation.

The scientists can connect to their virtual machines via secure shell (SSH) or NoMachine.  NoMachine is remote access software with support for a graphical user interface (GUI). This means that researchers can use NoMachine to access their VMs and be presented with a GUI desktop as if they were sitting in front of a physical machine. NoMachine has been of significant value to the CESWP project and for these purposes has been a good choice for connecting to cloud based VMs with a graphical user interface.

To ease the interaction with the CESWP cloud a web application written in Groovy and Grails was developed. Groovy is an open source dynamic language for the Java Virtual Machine that is fully compatible with the Java language itself. Grails is an open source web application framework that applies principles like convention over configuration to improve development productivity.
To develop CESWP, the team followed the principles of Scrum. Scrum is an agile software development methodology focused on the iterative development of a product. Iterations are organized into a three-week “sprint” wherein the team creates a working product increment. That product increment is delivered to the researchers as a demonstration in order to solicit feedback.

Technically speaking, CESWP itself is composed of three distinct parts. First, there is the CESWP cloud.  The cloud is comprised of servers, network and Eucalyptus. These resources combine together to provide the Infrastructure as a Service that the physicists will use to conduct their science.

Second, there is the CESWP toolkit. This toolkit is included on every VM image in the cloud and is comprised of scripts, both bash and Python, which ease interaction with the CESWP cloud.  These scripts perform such functions as running simulations in parallel to perform parameter sweeps, bundling VM images and enabling provenance.

Third, there is the CESWP application. The application is primarily developed with of the Groovy programming language, the Grails web framework and the Scrum agile development methodology. These resources combine together to create a GUI web application with a wizard-based interface that the physicists will use to interact with the CESWP cloud.

The primary motivation of the Cloud-Enabled Space Weather Platform is to lower the barriers for the physicists to conduct their science. There are a number of factors that obstruct the space weather physicists from concentrating on their science. The CESWP cloud paired with the CESWP application and toolkit addresses each of these barriers.

First, the specification and acquisition of hardware is consolidated in the CESWP cloud. Instead of specifying and purchasing desktop or server hardware for individual researchers on a piecemeal basis, the purchasing can be done in bulk for the benefit of the entire group. The procurement of this bulk hardware could still take weeks or months but is done only once per period as opposed to multiplying this effort for every researcher whenever the need arises.

Second, the maintenance of the hardware is consolidated in the CESWP cloud. Naturally, the scientists will always require hardware with which to access the cloud resources. However, there will no longer be a need for each researcher to have a laptop on which to develop their models and a high-powered desktop machine on which to run their simulations. The cloud effectively replaces the need for each researcher to maintain separate sets of hardware.

Third, the CESWP cloud has currently limited the selection of operating systems to Ubuntu and CentOS on the virtual machine images. This prevents a proliferation of operating systems and enables the cloud administrator to focus support effort on these two. In our experience, the physicists do not particularly care what operating system they are running, as long as it comes with the precompiled and preconfigured software that they prefer to use so we tailored the operating system VM images built for our community of users.

Fourth, one of the most time consuming aspects of the scientists work is getting code to compile properly. Be it their code when they have moved to a new machine or the code of a framework or tool they would like to use. The CESWP cloud can reduce the amount of time consumed in both cases. Once a researcher has compiled their own code in a VM, they can then bundle that VM into another VM image. The new image can be instantiated any number of times without having to recompile the code.  Likewise, once a framework (e.g. the Space Weather Modeling Framework) or tool has been compiled on a VM, it to can be bundled and made available to all of the physicists without recompilation.

Fifth, when a researcher’s model has reached a certain level of maturity he may want to run that model as a simulation on a grid or cluster to take advantage of high performance computing.  This is of great benefit to the researchers as their code can be processed many times more quickly than on their local machine. However, grid resources are in high demand and, therefore, scientists often must submit their jobs to a queue in order to be processed. The wait time can be hours to weeks. Instead, utilizing the CESWP toolkit, the researchers can run their simulations on the CESWP cloud now. If the CESWP cloud is low on resources the researchers can burst onto the Amazon Web Services (AWS) cloud, for a price.

In addition to lowering barriers CESWP also enables global access to a researcher’s virtual machine and access to their VM from a variety of devices. By default the virtual machines are globally accessible via secure channels. The VMs can be secured with either usernames and passwords or usernames and private keys. All traffic to and from the VM is encrypted. This global access can come from a variety of devices such as smart phones, tablets, laptops, netbooks or desktops.  Global access also enables collaboration as the researcher’s colleagues can be invited to access the VM to share data and collaborate on model development.

The space weather physics community is internationally distributed. Having cloud computing resources geographically close to its users lowers latency. This improves their experience interacting with the cloud. In addition, having all of the researchers using a single internationally distributed cloud will make it easier to share resources and collaborate.

Provenance is also enabled in the CESWP cloud. When a VM is bundled for provenance, using the CESWP toolkit, all of a physicist’s model code and data are bundled along with it. That is, the virtual machine’s file system is saved into another VM image. Also, a snapshot is taken of all of the virtual machine’s attached storage.Thus the preserved instance becomes a reproducible record of the scientists work. It is exact, complete, and unambiguous.

Work continues on Cloud-Enabled Space Weather Platform. The CESWP application is at the prototype stage and requires further refinement for production usage. The CESWP toolkit has only begun and scripts are routinely being added to enhance its utility. Naturally, we would like to expand the CESWP cloud by adding compute and storage capacity. The projected date for putting the Cloud-Enabled Space Weather Platform into production is June 2011.

Although this is the Cloud-Enabled Space Weather Platform, what we are building is not necessarily specific to space weather. Ultimately, we hope to build a platform that could potentially be used by other scientific disciplines.

About the Author

Everett Toews received his Bachelor of Science in Computing Science from the University of Alberta and has been working in the software development industry for the past decade. 

Everett has been working at Cybera for the past year and a half where he is building a cloud for use by space weather physicists at the University of Alberta on a project that goes by the name CESWP.

More available about Cybera can be found here.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Geospatial Data Research Leverages GPUs

August 17, 2017

MapD Technologies, the GPU-accelerated database specialist, said it is working with university researchers on leveraging graphics processors to advance geospatial analytics. The San Francisco-based company is collabor Read more…

By George Leopold

Intel, NERSC and University Partners Launch New Big Data Center

August 17, 2017

A collaboration between the Department of Energy’s National Energy Research Scientific Computing Center (NERSC), Intel and five Intel Parallel Computing Centers (IPCCs) has resulted in a new Big Data Center (BDC) that Read more…

By Linda Barney

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last week the cloud giant released deeplearn.js as part of that in Read more…

By John Russell

HPE Extreme Performance Solutions

Leveraging Deep Learning for Fraud Detection

Advancements in computing technologies and the expanding use of e-commerce platforms have dramatically increased the risk of fraud for financial services companies and their customers. Read more…

Spoiler Alert: Glimpse Next Week’s Solar Eclipse Via Simulation from TACC, SDSC, and NASA

August 17, 2017

Can’t wait to see next week’s solar eclipse? You can at least catch glimpses of what scientists expect it will look like. A team from Predictive Science Inc. (PSI), based in San Diego, working with Stampede2 at the Read more…

By John Russell

Microsoft Bolsters Azure With Cloud HPC Deal

August 15, 2017

Microsoft has acquired cloud computing software vendor Cycle Computing in a move designed to bring orchestration tools along with high-end computing access capabilities to the cloud. Terms of the acquisition were not disclosed. Read more…

By George Leopold

HPE Ships Supercomputer to Space Station, Final Destination Mars

August 14, 2017

With a manned mission to Mars on the horizon, the demand for space-based supercomputing is at hand. Today HPE and NASA sent the first off-the-shelf HPC system i Read more…

By Tiffany Trader

AMD EPYC Video Takes Aim at Intel’s Broadwell

August 14, 2017

Let the benchmarking begin. Last week, AMD posted a YouTube video in which one of its EPYC-based systems outperformed a ‘comparable’ Intel Broadwell-based s Read more…

By John Russell

Deep Learning Thrives in Cancer Moonshot

August 8, 2017

The U.S. War on Cancer, certainly a worthy cause, is a collection of programs stretching back more than 40 years and abiding under many banners. The latest is t Read more…

By John Russell

IBM Raises the Bar for Distributed Deep Learning

August 8, 2017

IBM is announcing today an enhancement to its PowerAI software platform aimed at facilitating the practical scaling of AI models on today’s fastest GPUs. Scal Read more…

By Tiffany Trader

IBM Storage Breakthrough Paves Way for 330TB Tape Cartridges

August 3, 2017

IBM announced yesterday a new record for magnetic tape storage that it says will keep tape storage density on a Moore's law-like path far into the next decade. Read more…

By Tiffany Trader

AMD Stuffs a Petaflops of Machine Intelligence into 20-Node Rack

August 1, 2017

With its Radeon “Vega” Instinct datacenter GPUs and EPYC “Naples” server chips entering the market this summer, AMD has positioned itself for a two-head Read more…

By Tiffany Trader

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Nvidia’s Mammoth Volta GPU Aims High for AI, HPC

May 10, 2017

At Nvidia's GPU Technology Conference (GTC17) in San Jose, Calif., this morning, CEO Jensen Huang announced the company's much-anticipated Volta architecture a Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Just how close real-wo Read more…

By John Russell

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Groq This: New AI Chips to Give GPUs a Run for Deep Learning Money

April 24, 2017

CPUs and GPUs, move over. Thanks to recent revelations surrounding Google’s new Tensor Processing Unit (TPU), the computing world appears to be on the cusp of Read more…

By Alex Woodie

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its a Read more…

By Tiffany Trader

Leading Solution Providers

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the cam Read more…

By John Russell

CPU-based Visualization Positions for Exascale Supercomputing

March 16, 2017

In this contributed perspective piece, Intel’s Jim Jeffers makes the case that CPU-based visualization is now widely adopted and as such is no longer a contrarian view, but is rather an exascale requirement. Read more…

By Jim Jeffers, Principal Engineer and Engineering Leader, Intel

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

Six Exascale PathForward Vendors Selected; DoE Providing $258M

June 15, 2017

The much-anticipated PathForward awards for hardware R&D in support of the Exascale Computing Project were announced today with six vendors selected – AMD Read more…

By John Russell

Top500 Results: Latest List Trends and What’s in Store

June 19, 2017

Greetings from Frankfurt and the 2017 International Supercomputing Conference where the latest Top500 list has just been revealed. Although there were no major Read more…

By Tiffany Trader

IBM Clears Path to 5nm with Silicon Nanosheets

June 5, 2017

Two years since announcing the industry’s first 7nm node test chip, IBM and its research alliance partners GlobalFoundries and Samsung have developed a proces Read more…

By Tiffany Trader

Messina Update: The US Path to Exascale in 16 Slides

April 26, 2017

Paul Messina, director of the U.S. Exascale Computing Project, provided a wide-ranging review of ECP’s evolving plans last week at the HPC User Forum. Read more…

By John Russell

Graphcore Readies Launch of 16nm Colossus-IPU Chip

July 20, 2017

A second $30 million funding round for U.K. AI chip developer Graphcore sets up the company to go to market with its “intelligent processing unit” (IPU) in Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This