May 15, 2020 — The National Science Foundation has awarded over $480,000 to the University of Illinois at Urbana-Champaign to develop and deploy funcX. This new “function-as-a-service” (FaaS) platform is hosted on the cloud and will enable researchers to build applications from functions that they can easily distribute across a variety of academic and commercial computing resources.
“Exploding data volumes and velocities, new computational methods and platforms, and ubiquitous connectivity demand new approaches to computation in the sciences,” says Ian Foster, Professor of Computer Science at the University of Chicago. “funcX makes it easy for scientists to run computations wherever it makes the most sense and move computations between resources. It will enable new science, reduce barriers and democratize access to advanced cyberinfrastructure, and enable researchers to compute wherever is most efficient.”
This four-year grant is part of a $3.14 million collaborative effort between the National Center for Supercomputing Applications (NCSA) at the University of Illinois and the University of Chicago (UChicago). The project is led by Principal Investigator (PI) Ian Foster, UChicago, and co-PIs, Daniel S. Katz, NCSA, and Kyle Chard, UChicago. The team will work with a wide range of scientific and cyberinfrastructure partners to build and test this new system for “computationally fluid” research.
“The research ecosystem comprises a range of existing systems: from HPC clusters to clouds and supercomputers,” says Chard, Research Assistant Professor at UChicago. “Researchers have varied workloads, large amounts of distributed data, and dynamic collaborations, so there isn’t a one-size-fits-all mapping from workloads to systems. funcX will make it easy to distribute these functions to different computing systems without needing to think about their differences, and will integrate with the scientific ecosystem for authentication, data management, and heterogeneous computing infrastructure.”
“Since its creation by NSF and the state of Illinois in 1986, NCSA has been a leader in working with and supporting scientific and engineering communities through the development, deployment, and use of new computing and software technologies,” says Katz, Assistant Director for Scientific Software and Applications at NCSA. “This project exemplifies our effort to develop leading-edge software that enables new research on our systems.”
Collaborative Research: Frameworks: Funcx: A Function Execution Service for Portability and Performance
Abstract: The funcX project is developing, deploying, and operating a new distributed computing cyberinfrastructure platform to enable researchers to build applications from programming functions that execute on different computing resources, from laptops to supercomputers. This cloud-hosted service democratizes access to advanced computing by providing intuitive interfaces for both registering remote computers as function executors and executing functions on these computers reliably, securely, and with high performance. Researchers can thus decompose monolithic applications into collections of reusable lightweight functions that can be run wherever makes the most sense, for example where data resides or where excess capacity is available. By simplifying access to specialized and high performance cyberinfrastructure and decreasing the time to discovery, the project serves the national interest, as stated in NSF’s mission, by promoting the progress of science. A total of 33 diverse science, cyberinfrastructure, and software institute partners working with cutting-edge science applications and research cyberinfrastructure will directly benefit from the funcX platform.
This project develops funcX, a scalable and high-performance federated platform for managing the remote execution of (often short-duration) functions across diverse cyberinfrastructure systems, from edge accelerators to clusters, supercomputers, and clouds. funcX allows developers to decompose applications into collections of functions that can each be executed in the best location, in terms of cost, execution time, data movement costs, and/or energy consumption. It thus integrates the extreme convenience of the function as a service (FaaS) model, developed in industry for specific industry applications, with support for the specialized needs of scientific research. funcX addresses important barriers to these new uses of research cyberinfrastructure systems, by enabling the intuitive, flexible, and scalable execution of functions without regard to physical location, scheduler architecture, virtualization technology, administrative domain, or data location. Flexible open-source funcX agent software makes it easy to expose arbitrary computing systems as funcX computing platforms, thereby transforming existing cyberinfrastructure systems into high-performance function serving environments (endpoints). The cloud-hosted funcX service provides a REST interface for registering functions, discovering available endpoints, and managing the execution of functions on endpoints, all via a universal trust fabric and standard web authentication and authorization mechanisms. It dynamically creates and deploys containers that incorporate function dependencies and provide a secure and isolated environment for safe function execution. The project engages a diverse set of 11 science partners, 18 research computing and cyberinfrastructure projects, and four NSF Software Institutes, each supporting many NSF-funded researchers, to provide use cases for funcX, shape its design, and evaluate its implementation.
This award reflects NSF’s statutory mission and has been deemed worthy of support through evaluation using the Foundation’s intellectual merit and broader impacts review criteria.
Source: National Center for Supercomputing Applications (NCSA)