[論文レビュー] Workflows Community Summit: Bringing the Scientific Workflows Community Together

Rafael Ferreira da Silva, Henri Casanova+43 more|arXiv (Cornell University)|Mar 16, 2021

Scientific Computing and Data Management参考文献 12被引用数 27

ひとこと要約

本論文は Workflows Community Summit (Jan 2021) を報告し、六つのテーマ別ディスカッションを要約し、科学的ワークフロー管理システムとより広範なワークフローエコシステムを前進させるための短期および長期のコミュニティの取り組みを提案しています。

ABSTRACT

Scientific workflows have been used almost universally across scientific domains, and have underpinned some of the most significant discoveries of the past several decades. Many of these workflows have high computational, storage, and/or communication demands, and thus must execute on a wide range of large-scale platforms, from large clouds to upcoming exascale high-performance computing (HPC) platforms. These executions must be managed using some software infrastructure. Due to the popularity of workflows, workflow management systems (WMSs) have been developed to provide abstractions for creating and executing workflows conveniently, efficiently, and portably. While these efforts are all worthwhile, there are now hundreds of independent WMSs, many of which are moribund. As a result, the WMS landscape is segmented and presents significant barriers to entry due to the hundreds of seemingly comparable, yet incompatible, systems that exist. As a result, many teams, small and large, still elect to build their own custom workflow solution rather than adopt, or build upon, existing WMSs. This current state of the WMS landscape negatively impacts workflow users, developers, and researchers. The "Workflows Community Summit" was held online on January 13, 2021. The overarching goal of the summit was to develop a view of the state of the art and identify crucial research challenges in the workflow community. Prior to the summit, a survey sent to stakeholders in the workflow community (including both developers of WMSs and users of workflows) helped to identify key challenges in this community that were translated into 6 broad themes for the summit, each of them being the object of a focused discussion led by a volunteer member of the community. This report documents and organizes the wealth of information provided by the participants before, during, and after the summit.

研究の動機と目的

• Document the state of the scientific workflows and WMS landscape and its fragmentation.
• Identify six key thematic challenges facing the workflow community.
• Summarize the summit structure, participants, and outputs.
• Propose short- and long-term community efforts to address identified challenges.
• Outline how two NSF/DOE projects (WorkflowsRI and ExaWorks) collaborate to advance the field.

提案手法

• Pre-summit Community Research Infrastructure Survey to identify requirements and challenges.
• Online summit with 48 invited participants from international WMS developers and users; plenary lightning talks followed by breakout discussions.
• Thematic synthesis of breakout discussions to identify challenges and proposed actions.
• Documentation of outcomes, including short- and long-term community efforts for each theme.
• Cross-project collaboration (WorkflowsRI and ExaWorks) to inform infrastructure and SDK development.

実験結果

リサーチクエスチョン

• RQ1 What are the core challenges across FAIR computational workflows for lifecycle, reuse, provenance, and labeling?
• RQ2 What training and educational needs exist for workflow users and how can they be addressed?
• RQ3 What are the unique requirements and challenges of AI/ML-enabled workflows within scientific workflows?
• RQ4 What exascale and beyond-HPC considerations affect workflow execution, resource management, and fault tolerance?
• RQ5 How can interoperability, APIs, and standards be advanced to reduce fragmentation of WMSs?
• RQ6 How can a cohesive workflows community be built and sustained across developers and users?

主な発見

• Identified six themes (FAIR workflows, training/education, AI workflows, exascale challenges, APIs/interoperability/standards, building a workflows community) and associated challenges.
• Proposed concrete short-term and long-term community efforts for each theme to tackle identified challenges.
• Documented the summit structure, including surveys, lightning talks, and breakout sessions, and the involvement of NSF/DoE projects WorkflowsRI and ExaWorks.
• Outlined the need for a common knowledge-base and community-driven guidelines to reduce fragmentation in the WMS landscape.
• Recommended leveraging existing registries, workflows repositories, and curricula to advance FAIRness, training, and standards.
• Suggested creating AI workflow use cases and eventual benchmarking mini-apps to guide HPC co-design and evaluation.