TaskVine is open source software for building large scale data intensive dynamic workflows that run on HPC clusters, GPU clusters, and commercial clouds. As tasks access external data sources and produce their own outputs, more and more data is pulled into local storage on workers. This data is used to accelerate future tasks and avoid re-computing exisiting results. Data gradually grows “like a vine” through the cluster. By using a variety of pioneering techniques such as distant futures, distributed provenance, and graph pruning, TaskVine is able to ourperform traditional distributed computing techniques. It has been used to build large scale applications in scientific fields such as high energy physics, bioinformatics, molecular dynamics, and machine learning. We continue to develop new techniques within TaskVine, and extend it use to new applications.
Related Publications
Reshaping High Energy Physics Applications for Near-Interactive Execution Using TaskVine
Barry Sly-Delgado, Ben Tovar, Jin Zhou, and Douglas Thain
@inproceedings{reshaping-sc-2024,author={Sly-Delgado, Barry and Tovar, Ben and Zhou, Jin and Thain, Douglas},title={{Reshaping High Energy Physics Applications for Near-Interactive Execution Using TaskVine}},booktitle={{ACM/IEEE Supercomputing}},pages={1-11},year={2024},cclpaperid={996},keywords={taskvine},doi={10.1109/SC41406.2024.00068}}
Poster: Leveraging Intermediate Data Management with Parsl/TaskVine
Colin Thomas and Douglas Thain
In Greater Chicago Area Systems Research Workshop, 2024
@inproceedings{data-gcasr-2024,author={Thomas, Colin and Thain, Douglas},title={{Poster: Leveraging Intermediate Data Management with Parsl/TaskVine}},booktitle={{Greater Chicago Area Systems Research Workshop}},pages={1},year={2024},cclpaperid={998},keywords={taskvine},}
Poster: Reshaping High Energy Physics Applications for Near-Interactive Execution Using TaskVine
Barry Sly-Delgado and Douglas Thain
In Greater Chicago Area Systems Research Workshop, 2024
@inproceedings{reshaping-gcasr-2024,author={Sly-Delgado, Barry and Thain, Douglas},title={{Poster: Reshaping High Energy Physics Applications for Near-Interactive Execution Using TaskVine}},booktitle={{Greater Chicago Area Systems Research Workshop}},pages={1},year={2024},cclpaperid={1002},keywords={taskvine},}
TaskVine: Managing In-Cluster Storage for High-Throughput Data Intensive Workflows
Barry Sly-Delgado, Thanh Son Phung, Colin Thomas, David Simonetti, Andrew Hennessee, Ben Tovar, and Douglas Thain
In 18th Workshop on Workflows in Support of Large-Scale Science, 2023
@inproceedings{taskvine-works-2023,author={Sly-Delgado, Barry and Phung, Thanh Son and Thomas, Colin and Simonetti, David and Hennessee, Andrew and Tovar, Ben and Thain, Douglas},title={{TaskVine: Managing In-Cluster Storage for High-Throughput Data Intensive Workflows}},booktitle={{ 18th Workshop on Workflows in Support of Large-Scale Science}},year={2023},cclpaperid={991},keywords={taskvine},}
Poster: TaskVine: A User-Level Framework for Data Intensive Scientific Applications
@inproceedings{taskvine-cssi-2023,author={Thain, Douglas},title={{Poster: TaskVine: A User-Level Framework for Data Intensive Scientific Applications}},booktitle={{CSSI PI Meeting}},year={2023},cclpaperid={989},keywords={taskvine},}
Poster: Mixed Modality Workflows in TaskVine
David Simonetti, Ben Tovar, and Douglas Thain
In ACM High Performance Distributed Computing, 2023
@inproceedings{mixed-hpdc-2023,author={Simonetti, David and Tovar, Ben and Thain, Douglas},title={{Poster: Mixed Modality Workflows in TaskVine}},booktitle={{ACM High Performance Distributed Computing}},pages={331-332},year={2023},cclpaperid={988},keywords={taskvine},}
