Current Research Projects in the CCL

xGFabric is one of several projects recently awarded by the U.S. Department of Energy for exploratory research in extreme scale science. This project will explore the design of systems for connecting remote wireless sensor networks with high performance computing systems. The driving applications are in digital agriculture, where extensive sensors are already used to observe weather, growing conditions, and the current state of crops. However, exploiting this data to improve agriculture in a timely way is challenging, due to energy and computing limitations of remote settings. Only a high performance computing (HPC) center has the capability to run the simulations needed to predict micro-weather and recommend interventions. This project will develop XGFabric, a novel software system that will allow complex workflows to span remote 5G/6G sensor networks, edge computers, and powerful HPC facilities, enabling timely access to powerful computing capabilities from remote agricultural sites.

The SADE project is a large collaborative undertaking across the University of Notre Dame, Iowa State University, St Louis University, University of Texas El Paso, and DePaul University. The overall goal of the project is to develop technology for safety zones that permit only trusted drones to operate in congested or sensitive airspace. The CCL is contributing to this project with the development of complex simulation infrastructure that will allow for in-silico testing of drone control software. This infrastructure includes flight controllers (PX4), physical simulators (Gazebo) gaming technology (Unreal), and other services all dynamically deployed in HPC infrastructure. This allows developers of drone control software to evaluate the correctness of systems at scale before deploying to real-world scenarios.

Advanced discovery in scientific computing increasingly depends upon the successful execution of complex workflows that make use of a shared filesystem. We observe that workflows rarely need the full power of global sequential consistency. Rather, they tend to follow several common I/O patterns that can be characterized at a high level. Rather than depend upon the filesystem to "figure things out" at the last minute, we propose consistency contracts as a means of expressing the I/O intentions of complex workflows. With a contract in hand, the runtime system can then perform a variety of optimizations that exploit the internal storage and I/O capacity of the cluster as a whole.

NBFlow is our newest CSSI project that brings together interactive notebook technologies (Jupyter), with reproducibility tools (SciUnit), and distributed workflows (TaskVine). The combined result will be a new system (NBFlow) that makes it easy to execute large scale workflows on thousands of cluster nodes, through the familiar interface of the executable notebook. While easily stated, this presents a number of challenges to reconcile the high throughput but unpredictable nature of clusters with the immediate feedback expected by notebooks. We will develop a number of new techniques to convert notebooks into workflows, execute them reliably on the cluster, and then return them back again to the user. This project is a collaboration with Kevin Lannon (U Notre Dame), Tanu Malik (DePaul U), and Shaowen Wang (U Illinois).

TaskVine is open source software for building large scale data intensive dynamic workflows that run on HPC clusters, GPU clusters, and commercial clouds. As tasks access external data sources and produce their own outputs, more and more data is pulled into local storage on workers. This data is used to accelerate future tasks and avoid re-computing exisiting results. Data gradually grows "like a vine" through the cluster. By using a variety of pioneering techniques such as distant futures, distributed provenance, and graph pruning, TaskVine is able to outperform traditional distributed computing techniques. It has been used to build large scale applications in scientific fields such as high energy physics, bioinformatics, molecular dynamics, and machine learning. We continue to develop new techniques within TaskVine, and extend it use to new applications.

Expresses workflow I/O intentions using consistency contracts to enable efficient, cluster-wide storage and execution optimizations.