Start-up Agnostiq Tackles HPC-Quantum Workflow Development

By John Russell

June 9, 2022

Imagine some years hence the HPC landscape, already confusing in its heterogeneity, also includes quantum computing resources. How does a user choose from among those resources the best workflow and computing engines for a project? Agnostiq, a Toronto-based start-up, is focused on solving that broad problem and its initial product, Covalent, is focused on the intersection of HPC and quantum resources.

Agnostiq presented at last week’s ISC conference, and this week Will Cunningham, Agnostiq head of software, briefed HPCwire on the company’s nascent but ambitious plans. Co-founded in 2018 by CEO Oktay Goktas (physicist trained at Max-Planck-Institute in Stuttgart) and COO Elliot MacGowan (MBA from university of Toronto), Agnostiq is one of a growing number of Canadian quantum computing startups.

Will Cunningham, Agnostiq

Cunningham told HPCwire, “Agnostic right now is a seed stage startup. We have about 25 fulltime employees across the globe – Canada, North America, and beyond – so we’re fully remote right now and intend to stay that way. We position ourselves at the intersection of quantum and high-performance computing, since we see these technologies growing and eventually merging together into a broader continuum of compute technology.”

This idea of quantum computing becoming part of the broader HPC landscape is starting to take hold with more companies like Agnostiq seeking to provide orchestrating software and services to knit the technologies together. Covalent, the company’s first product, is a workflow orchestration tool that hides much of the underlying quantum system complexity from users and allows fast prototyping among many resources to choose a preferred workflow (time-to-solution, cost, viability).

Noted Cunningham, “Right now we’re focused on Covalent and putting all of our effort into developing tools at the HPC-quantum intersection. We started as a pure quantum software company, trying to develop finance applications. What we found is that it’s a little bit too early to build out enterprise applications that are really ready to be consumed by large companies. Of course, the reason being that the hardware is getting there, but it’s not quite there yet.

“When it comes to classical HPC, one of the things that we’re interested in [and] that’s in our roadmap is understanding how people can better provision and manage and schedule tasks on compute in general. We view quantum as part of this broader landscape of compute, but one of the things that we’re interested in, both with Covalent and beyond, is understanding how to appropriately map software to hardware and that hardware can be high compute and supercomputers or it could be general compute or low compute,” said Cunningham.

Released last January, Covalent is open-sourced and freely available on github; it is a workflow orchestration tool designed for rapid iteration and pre-production R&D workflows. The idea is to be able to quickly build, test and compare workflows.

Complicated workflows using multiple compute resources is a growing part of HPC.

“For example, a user may perform some data pre-processing on a local laptop, then transfer it to a supercomputer where it is used as an input to a high compute simulation. Finally, results are collected, some post-processing may be performed to remove corrupt or missing entries, and the results are visualized in some plot. In more complex settings, users may be interacting with multiple supercomputing, clusters, cloud HPC resources, and now even quantum computers,” noted Cunningham during his ISC presentation.

“In the era of hybrid computing, there are more options than ever when it comes to how you interact with HPC devices. Experiments become heterogeneous in so many ways. They can be classical or quantum, use high compute or general compute resources, use cloud or on-prem compute clusters, involve serial or parallel algorithms, and even deterministic or probabilistic algorithms. We are transitioning to a world where all of these options can be considered together for a single application, it can become very difficult to make sense of so many options.”

Covalent is intended to prototype and test drive workflows using quantum resources. Agnostiq says that users can quickly prototype and experiment, try it out and iterate using different input parameters, software environments and hardware resources.

“At the top of the stack above Covalent, we find commercial workflow orchestration tools such as Prefect, Dagster, Airflow, and Luigi. These tools are used in large scale enterprise machine learning and data analytics applications where certain tasks must run on a time-based schedule. Covalent is a layer in between the distributed computation and workflow management layers, which is why we call it a distributed workflow tool. In Covalent, instances of the workflow rather than the workflow itself, are the primary objects. This core design principle enables the type of rapid iteration needed for pre-production R&D workflows while remaining compatible with tools at the other layers in the stack,” said Cunningham.

Users can start from a Jupyter notebook, which has become the platform of choice for prototyping in high performance settings, or use standard scripts. The first step is to functionalize their existing code. “While this is good practice anyways, in that it increases maintainability and code quality, it is required for Covalent as functions are what will be mapped to tasks in a workflow,” said Cunningham.

Users then add one-line decorators to each function. “Internally, this converts the functions to callable, class objects, which have the ability to save loads of metadata about the function and how it ran. However, these functions remain callable as standard Python functions in the way that you might expect. This differs from some workflow tools which require the use of Yaml rather than Python. Notice we use two different decorators – the electron decorator refers to a task while the lattice decorator refers to a workflow. Next, users can add execution information to the decorators and executors are used to tell Covalent where and how to run the tasks,” explained Cunningham.

Shown (slide) above are two examples. One uses the Slurm executor, which submits a high compute task to a Slurm partition in an HPC cluster managed by Slurm. The second uses the AWS Fargate executor, which submits low compute tasks to the AWS elastic container service, where it can run using AWS Fargate. With these two different executors, a workflow using these tasks can flexibly send tasks to both high compute and low compute resources.

Cunningham said Agnostiq understands that Python is not the first language of choice for many HPC users, “We don’t ask that you rewrite legacy code in a new language. Instead, we provide bindings between Covalent and other common HPC languages, so that Python is used for orchestration, while C or C++, Bash, Julia and Fortran are used for the actual compute task.”

Focusing for a moment on the quantum portion, Cunningham said, “In today’s NISQ (noisy intermediate scale quantum) era, quantum computers are in high demand, and it is commonplace to wait for hours in queues. Practically, this means both hardware providers and research groups need to be careful about how HPC and quantum reservations are utilized. Quantum computers are expensive, difficult to maintain, and have a relatively short lifetime compared to classical resources.”

Obviously, these are early days for Agnostiq. “We’re engaging with a variety of potential customers before we go for any enterprise or monetized version. We’re really trying to get good user feedback from a variety of use cases. So, this includes software startups in deep tech like machine learning, and pharmaceuticals. It includes national supercomputing centers. We’re sort of exploring how this performs, you know, in some of these larger systems in North America, and after ISC, hopefully in Europe, as well.”

Agnostiq agrees that broad quantum advantage – the time when many applications will perform significantly better on quantum computers than classical one – is some years away. Cunningham suggests the middle of the decade for select use cases and maybe another five years after that for wider spread adoption. Agnostiq hopes to produce tools that help hybrid HPC-Quantum users make better compute choices at the workflow prototype and R&D stage.

Recalling his early days at Agnostiq, Cunningham said, “My first task was to go onto AWS, and learn how to provision infrastructure, so that we could actually start doing some experiments and ultimately do them at scale. What we found is that very quickly, it became difficult to manage costs. As you know, we’re a small organization. We found it was kind of difficult to manage this interplay between classical and quantum, and all of today’s quantum algorithms are going to require some classical compute. We’ve seen various cloud providers like IBM and AWS starting to provide these sort of hybrid platforms through Qiskit runtime and AWS has Braket hybrid jobs.”

Now, Agnostiq is developing tools to help users with their hybrid HPC-quantum workflows.

“This problem of trying to understand where to invest money, not only on which applications are going to be good for long term, but also where to invest money in a particular technology, but without going all in. So today, you might have superconducting quantum computers that are the best for a particular application, and maybe you get some sort of incremental speed up from using these devices, or at least you can use them to train employees. In a couple of years from now, maybe you’re going to be looking at photonic quantum computers, or maybe you’re going to be looking at neutral atoms. Our take on this is that it’s very dynamic. Nobody really knows who’s going to be the winner.”

Stay tuned.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Solving Heterogeneous Programming Challenges with Python, Today

June 30, 2022

You may be surprised how ready Python is for heterogeneous programming, and how easy it is to use today. Our first three articles about heterogeneous programming focused primarily on C++ as we ponder “how to enable programming in the face of an explosion of hardware diversity that is coming?” For a refresher on what motivates this question... Read more…

The Mainstreaming of MLPerf? Nvidia Dominates Training v2.0 but Challengers Are Rising

June 29, 2022

MLCommons’ latest MLPerf Training results (v2.0) issued today are broadly similar to v1.1 released last December. Nvidia still dominates, but less so (no grand sweep of wins). Relative newcomers to the exercise – AI Read more…

NOAA Launches Twin Supercomputers, Tripling Operational Forecasting Capacity

June 29, 2022

In February 2020, the United States’ National Oceanic and Atmospheric Administration (NOAA) announced that it would be procuring two HPE Cray systems, allowing the organization to triple its operational supercomputing Read more…

US Pursues Next-gen Exascale Systems with 5-10x the Performance of Frontier

June 28, 2022

With the Linpack exaflops milestone achieved by the Frontier supercomputer at Oak Ridge National Laboratory, the United States is turning its attention to the next crop of exascale machines, some 5-10x more performant than Frontier. At least one such system is being planned for the 2025-2030 timeline, and the DOE is soliciting input from the vendor community... Read more…

HPE’s New Arm Server Signals Shift in x86 Mindset

June 28, 2022

HPE's early stab at ARM servers close to a decade ago didn't pan out, but the company is hoping the second time is a charm. The company introduced the ProLiant RL300 Gen11 server, which has Ampere's ARM server processor. The one-socket server is designed for cloud-based applications, with the ability to scale up applications in a power efficient... Read more…

AWS Solution Channel

Shutterstock 1413860048

Indivumed Boosts Cancer Research With Powerful Analytics Built on AWS

Hamburg-based Indivumed specializes in using the highest quality biospecimen and comprehensive clinical data to advance research and development in precision oncology. Its IndivuType discovery solution uses AWS to store data and support analysis to decipher the complexity of cancer. Read more…

Microsoft/NVIDIA Solution Channel

Shutterstock 1179306271

Using Cloud-Based, GPU-Accelerated AI to Track Identity Fraud

Consumers use many accounts for financial transactions, ordering products, and social media—a customer’s identity can be stolen using any of these accounts. Identity fraud can happen when setting up or using financial accounts, but it can also occur with communications such as audio, images, and chats. Read more…

What’s New in HPC Research: EXA2PRO, DQRA, and HiCMA-PaRSE Frameworks & More

June 28, 2022

In this regular feature, HPCwire highlights newly published research in the high-performance computing community and related domains. From parallel programming to exascale to quantum computing, the details are here. Read more…

Solving Heterogeneous Programming Challenges with Python, Today

June 30, 2022

You may be surprised how ready Python is for heterogeneous programming, and how easy it is to use today. Our first three articles about heterogeneous programming focused primarily on C++ as we ponder “how to enable programming in the face of an explosion of hardware diversity that is coming?” For a refresher on what motivates this question... Read more…

Shutterstock 1874021860

The Mainstreaming of MLPerf? Nvidia Dominates Training v2.0 but Challengers Are Rising

June 29, 2022

MLCommons’ latest MLPerf Training results (v2.0) issued today are broadly similar to v1.1 released last December. Nvidia still dominates, but less so (no gran Read more…

NOAA Launches Twin Supercomputers, Tripling Operational Forecasting Capacity

June 29, 2022

In February 2020, the United States’ National Oceanic and Atmospheric Administration (NOAA) announced that it would be procuring two HPE Cray systems, allowin Read more…

US Pursues Next-gen Exascale Systems with 5-10x the Performance of Frontier

June 28, 2022

With the Linpack exaflops milestone achieved by the Frontier supercomputer at Oak Ridge National Laboratory, the United States is turning its attention to the next crop of exascale machines, some 5-10x more performant than Frontier. At least one such system is being planned for the 2025-2030 timeline, and the DOE is soliciting input from the vendor community... Read more…

HPE’s New Arm Server Signals Shift in x86 Mindset

June 28, 2022

HPE's early stab at ARM servers close to a decade ago didn't pan out, but the company is hoping the second time is a charm. The company introduced the ProLiant RL300 Gen11 server, which has Ampere's ARM server processor. The one-socket server is designed for cloud-based applications, with the ability to scale up applications in a power efficient... Read more…

Is Time Running Out for Compromise on America COMPETES/USICA Act?

June 22, 2022

You may recall that efforts proposed in 2020 to remake the National Science Foundation (Endless Frontier Act) have since expanded and morphed into two gigantic bills, the America COMPETES Act in the U.S. House of Representatives and the U.S. Innovation and Competition Act in the U.S. Senate. So far, efforts to reconcile the two pieces of legislation have snagged and recent reports... Read more…

Cerebras Systems Thinks Forward on AI Chips as it Claims Performance Win

June 22, 2022

Cerebras Systems makes the largest chip in the world, but is already thinking about its upcoming AI chips as learning models continue to grow at breakneck speed. The company’s latest Wafer Scale Engine chip is indeed the size of a wafer, and is made using TSMC’s 7nm process. The next chip will pack in more cores to handle the fast-growing compute needs of AI, said Andrew Feldman, CEO of Cerebras Systems. Read more…

AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Read more…

Nvidia R&D Chief on How AI is Improving Chip Design

April 18, 2022

Getting a glimpse into Nvidia’s R&D has become a regular feature of the spring GTC conference with Bill Dally, chief scientist and senior vice president of research, providing an overview of Nvidia’s R&D organization and a few details on current priorities. This year, Dally focused mostly on AI tools that Nvidia is both developing and using in-house to improve... Read more…

Royalty-free stock illustration ID: 1919750255

Intel Says UCIe to Outpace PCIe in Speed Race

May 11, 2022

Intel has shared more details on a new interconnect that is the foundation of the company’s long-term plan for x86, Arm and RISC-V architectures to co-exist in a single chip package. The semiconductor company is taking a modular approach to chip design with the option for customers to cram computing blocks such as CPUs, GPUs and AI accelerators inside a single chip package. Read more…

The Final Frontier: US Has Its First Exascale Supercomputer

May 30, 2022

In April 2018, the U.S. Department of Energy announced plans to procure a trio of exascale supercomputers at a total cost of up to $1.8 billion dollars. Over the ensuing four years, many announcements were made, many deadlines were missed, and a pandemic threw the world into disarray. Now, at long last, HPE and Oak Ridge National Laboratory (ORNL) have announced that the first of those... Read more…

AMD/Xilinx Takes Aim at Nvidia with Improved VCK5000 Inferencing Card

March 8, 2022

AMD/Xilinx has released an improved version of its VCK5000 AI inferencing card along with a series of competitive benchmarks aimed directly at Nvidia’s GPU line. AMD says the new VCK5000 has 3x better performance than earlier versions and delivers 2x TCO over Nvidia T4. AMD also showed favorable benchmarks against several Nvidia GPUs, claiming its VCK5000 achieved... Read more…

Top500: Exascale Is Officially Here with Debut of Frontier

May 30, 2022

The 59th installment of the Top500 list, issued today from ISC 2022 in Hamburg, Germany, officially marks a new era in supercomputing with the debut of the first-ever exascale system on the list. Frontier, deployed at the Department of Energy’s Oak Ridge National Laboratory, achieved 1.102 exaflops in its fastest High Performance Linpack run, which was completed... Read more…

Newly-Observed Higgs Mode Holds Promise in Quantum Computing

June 8, 2022

The first-ever appearance of a previously undetectable quantum excitation known as the axial Higgs mode – exciting in its own right – also holds promise for developing and manipulating higher temperature quantum materials... Read more…

Nvidia Launches Hopper H100 GPU, New DGXs and Grace Superchips

March 22, 2022

The battle for datacenter dominance keeps getting hotter. Today, Nvidia kicked off its spring GTC event with new silicon, new software and a new supercomputer. Speaking from a virtual environment in the Nvidia Omniverse 3D collaboration and simulation platform, CEO Jensen Huang introduced the new Hopper GPU architecture and the H100 GPU... Read more…

PsiQuantum’s Path to 1 Million Qubits

April 21, 2022

PsiQuantum, founded in 2016 by four researchers with roots at Bristol University, Stanford University, and York University, is one of a few quantum computing startups that’s kept a moderately low PR profile. (That’s if you disregard the roughly $700 million in funding it has attracted.) The main reason is PsiQuantum has eschewed the clamorous public chase for... Read more…

Leading Solution Providers

Contributors

ISC 2022 Booth Video Tours

AMD
AWS
DDN
Dell
Intel
Lenovo
Microsoft
PENGUIN SOLUTIONS

AMD Opens Up Chip Design to the Outside for Custom Future

June 15, 2022

AMD is getting personal with chips as it sets sail to make products more to the liking of its customers. The chipmaker detailed a modular chip future in which customers can mix and match non-AMD processors in a custom chip package. "We are focused on making it easier to implement chips with more flexibility," said Mark Papermaster, chief technology officer at AMD during the analyst day meeting late last week. Read more…

Intel Reiterates Plans to Merge CPU, GPU High-performance Chip Roadmaps

May 31, 2022

Intel reiterated it is well on its way to merging its roadmap of high-performance CPUs and GPUs as it shifts over to newer manufacturing processes and packaging technologies in the coming years. The company is merging the CPU and GPU lineups into a chip (codenamed Falcon Shores) which Intel has dubbed an XPU. Falcon Shores... Read more…

AMD’s MI300 APUs to Power Exascale El Capitan Supercomputer

June 21, 2022

Additional details of the architecture of the exascale El Capitan supercomputer were disclosed today by Lawrence Livermore National Laboratory’s (LLNL) Terri Read more…

India Launches Petascale ‘PARAM Ganga’ Supercomputer

March 8, 2022

Just a couple of weeks ago, the Indian government promised that it had five HPC systems in the final stages of installation and would launch nine new supercomputers this year. Now, it appears to be making good on that promise: the country’s National Supercomputing Mission (NSM) has announced the deployment of “PARAM Ganga” petascale supercomputer at Indian Institute of Technology (IIT)... Read more…

Nvidia, Intel to Power Atos-Built MareNostrum 5 Supercomputer

June 16, 2022

The long-troubled, hotly anticipated MareNostrum 5 supercomputer finally has a vendor: Atos, which will be supplying a system that includes both Nvidia and Inte Read more…

Nvidia Dominates MLPerf Inference, Qualcomm also Shines, Where’s Everybody Else?

April 6, 2022

MLCommons today released its latest MLPerf inferencing results, with another strong showing by Nvidia accelerators inside a diverse array of systems. Roughly fo Read more…

Covid Policies at HPC Conferences Should Reflect HPC Research

June 6, 2022

Supercomputing has been indispensable throughout the Covid-19 pandemic, from modeling the virus and its spread to designing vaccines and therapeutics. But, desp Read more…

AMD Lines Up Alternate Chips as It Eyes a ‘Post-exaflops’ Future

June 10, 2022

Close to a decade ago, AMD was in turmoil. The company was playing second fiddle to Intel in PCs and datacenters, and its road to profitability hinged mostly on Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire