New Plugin Tracks Scientific Workflows in Real Time

A new monitoring plugin for Nextflow, a widely used workflow management system in fields like bioinformatics and earth observation, provides researchers with a more detailed and flexible way to track their data processing tasks. This tool addresses of monitoring distributed workflows that rely on containerization, orchestration, or batch processing systems, which is crucial for performance analysis, debugging, and ensuring data provenance. By offering an alternative to the standard wf-commons tool, it enhances the ability to observe workflow executions in real time, making it easier for scientists to optimize their computational processes without needing to modify the underlying workflow code.

The key finding of this research is that the plugin enables comprehensive tracing of Nextflow executions by constructing a physical execution graph online during runtime. It gathers detailed information for each task, such as start and end times, execution s, container images, and working directories, which can be used by other tools like schedulers for online data processing. This approach provides a broader view of workflow executions compared to basic monitoring, as illustrated in Figure 1, which shows the assignment of physical tasks in a partial run of the rnaseq workflow. The plugin outputs data in a valid and extended wf-instance JSON format, ensuring compatibility with existing tools while adding richer context.

Ology involves implementing the plugin in Groovy, leveraging Nextflow's plug-in mechanism available in version 21.10, which avoids the need for a custom fork of Nextflow. It uses callbacks and JVM reflections on Nextflow's internal state to monitor tasks dynamically, and it can be activated through Nextflow's config file or a command-line override. For deeper insights, an optional patch allows node-level monitoring by injecting code or tools into the wrapper script used for task execution, enabling the collection of hardware data like CPU, RAM, network information, and IP addresses, potentially using external tools such as collectl for time-series resource usage.

From using the plugin on real workflows, such as those from the nf-core repository, demonstrate that data can be seamlessly collected without modifying the workflow itself. For example, monitoring a six-node execution of the rnaseq workflow revealed detailed task assignments and node interactions, as shown in Figure 1. This data aids in identifying performance bottlenecks and understanding execution patterns, which is valuable for optimizing resource allocation and scheduling in high-performance computing environments where traditional s may fall short.

Of this work are significant for scientific communities that rely on Nextflow for large-scale data analysis, such as in bioinformatics and earth observation. By providing detailed monitoring capabilities, the plugin helps researchers gain insights into workflow performance, potentially leading to more efficient use of computational resources and faster cycles. It also supports the German Research Foundation's CRC 1404: FONDA project, which focuses on foundations of workflows for large-scale scientific data analysis, highlighting its relevance to advancing scientific research through improved workflow management.

Limitations of the plugin include its reliance on Nextflow's available internal data, which may not expose all hardware details without the optional patch. The patch itself requires additional setup and may not be suitable for all deployment environments, potentially limiting the depth of monitoring in some cases. Additionally, while the plugin offers enhanced flexibility, it may introduce overhead in highly dynamic or large-scale executions, though the paper does not specify performance impacts. Future work could address these constraints by integrating more seamless node-level monitoring or expanding compatibility with other workflow systems.