NBFlow: From Notebook to Workflow and Back Again (CSSI Framework)

NBFlow is our newest CSSI project that brings together interactive notebook technologies (Jupyter), with reproducibility tools (SciUnit), and distributed workflows (TaskVine). The combined result will be a new system (NBFlow) that makes it easy to execute large scale workflows on thousands of cluster nodes, through the familiar interface of the executable notebook. While easily stated, this presents a number of challenges to reconcile the high throughput but unpredictable nature of clusters with the immediate feedback expected by notebooks. We will develop a number of new techniques to convert notebooks into workflows, execute them reliably on the cluster, and then return them back again to the user. This project is a collaboration with Kevin Lannon (U Notre Dame), Tanu Malik (DePaul U), and Shaowen Wang (U Illinois).

Related Publications

Backpacks for Notebooks: Enabling Containerized Notebook Workflows in Distributed Environments

Md Saiful Islam, Talha Azaz, Raza Ahmad, A D M Shahadat Hossain, Furqan Baig, Shaowen Wang, Kevin Lannon, Tanu Malik, and Douglas Thain

In IEEE Conference on eScience, 2025

Bib PDF

@inproceedings{floability-escience-2025,
  author = {Islam, Md Saiful and Azaz, Talha and Ahmad, Raza and Hossain, A D M Shahadat and Baig, Furqan and Wang, Shaowen and Lannon, Kevin and Malik, Tanu and Thain, Douglas},
  title = {{Backpacks for Notebooks: Enabling Containerized Notebook Workflows in Distributed Environments}},
  booktitle = {{IEEE Conference on eScience}},
  pages = {9},
  year = {2025},
  cclpaperid = {1003},
  keywords = {floability}
}