Release Notes

2026 Releases

JAWS Release 2026.03 - 03/31/2026

🐛 Bug Fixes

Improved exception handling for the sbatch command to ensure failures are properly surfaced rather than failing silently, making debugging easier (pool-manager-244).
Resolved an issue with GPU node request handling that was causing GPU workflows to stall indefinitely (pool-manager-239).

JAWS Release 2026.02 - 03/26/2026

✨ New Features

RNome Site at NERSC: JAWS has been successfully deployed to the new RNome site at NERSC, expanding compute capacity. This site is dedicated to supporting the RNome project, expanding the user base for JAWS.
Enhanced GPU Support: Added gpuCount runtime attribute and mapped request_gpus from both gpu and gpuCount for improved GPU workflow support (cromwell-258).

Below is an example of how to request GPU resources in a WDL workflow using the new gpuCount attribute:
```
version 1.0

workflow GPU_Test_Workflow {
  call GPU_Test_Task
}

task GPU_Test_Task {

  command <<<
    echo "Testing GPU runtime capabilities"
    ## add your GPU commands here.
  >>>

  runtime {
    docker: "pytorch/pytorch:latest"
    memory: "20GiB"
    cpu: 1
    gpu: true
    gpuCount: 3
    runtime_minutes: 10
  }
}
```
Key Notes:
- gpu: true enables GPU resource allocation for the task.
- gpuCount: 3 requests 3 GPUs from the scheduler. If gpuCount is not specified, JAWS will request 1 GPU by default when gpu: true.
- The request_gpus attribute is also supported and maps to gpuCount for backward compatibility.
- Ensure the Docker container used supports GPU execution (e.g., pytorch/pytorch:latest includes CUDA).
Resource Management: Reduced the maximum number of nodes allocated at NMDC NERSC instance (pool-manager-236).

🐛 Bug Fixes

Fixed a regression where jaws submit could fail with KeyError: ‘library_id’ when submitting valid workflows with nested struct inputs (client-323).
Fixed an issue where team output folder permissions were not set to site-specific team directory permissions (site-1918).
Fixed condor_status command for improved node cleanup (pool-manager-240).

🔧 Internal Changes

This release also includes internal bug fixes and platform improvements to enhance reliability, performance, and overall user experience.

📦 Client Version Update

JAWS Client has been updated to version 2.12.1. To take advantage of the latest capabilities:

If you’re using the JAWS Client Python library, upgrade with:

pip install --upgrade jaws-client==2.12.1

JAWS Release 2026.01 - 02/25/2026

✨ New SITE: Superbio at NERSC

New Site: Superbio: A new JAWS site, superbio, has been added. - This site is located at the NERSC and its purpose is to support the SSLR at SLAC.

🔧 Additional Improvements

This release also includes internal bug fixes and platform improvements to enhance reliability, performance, and overall user experience.

📦 Client Version Update

JAWS Client has been updated to version 2.12.0. To take advantage of the latest capabilities:

If you’re using the JAWS Client Python library, upgrade with:

pip install --upgrade jaws-client==2.12.0

2025 Releases

JAWS Release 2025.14 - 10/29/2025

🐛 Bug Fixes

Fixed NMDC refdata location to ensure correct data paths for NMDC workflows. This update resolves issues where workflows couldn’t access required refdata (Issue client-314).
Resolved HTCondor Access Point port conflict that prevented proper communication between JAWS components. (Issue pool-manager-219).
Fixed an issue with HTCondor queue processing that incorrectly set the number of needed nodes to 0 when the first queued task required GPU resources. This caused GPU workflows to stall indefinitely waiting for resources (Issue pool-manager-220).
Ensured that the version reported by jaws info command matches the actual JAWS_CLIENT_VERSION for accurate version tracking and debugging (Issue client-315).

📦 Client Version Update

JAWS Client has been updated to version 2.11.0. To take advantage of the latest capabilities:

If you’re using the JAWS Client Python library, upgrade with:

pip install --upgrade jaws-client==2.11.0

JAWS Release 2025.13 - 10/22/2025

🔧 Internal Improvements

Downgrade Globus SDK library to version < 4.0.0, since version 4.0.0 introduced breaking changes that affected JAWS functionality.

JAWS Release 2025.12 - 10/14/2025

🐛 Bug Fixes

Fixed an issue with HTCondor queue processing that caused the number of needed nodes to be underestimated (Issue pool-manager-216).

JAWS Release 2025.11 - 10/03/2025

✨ New Features and Improvements

New Site: Defiant: A new JAWS site, defiant, has been added. This site is located at the Defiant system at the Oak Ridge Leadership Computing Facility (OLCF).
New Site: Crux: A new JAWS site, crux, has been added. This site is located at the Crux system at the Argonne Leadership Computing Facility (ALCF).
Improved detection of potential user group and/or file permission issues: An explicit check has been added to verify that the user account running jaws belongs to the expected user groups. If this is not the case, a helpful message is shown. Previously, this type of issues would only be detected indirectly by seemingly unrelated errors, which made the messages misleading and harder to trace back to the underlying cause (Issue client-264).
Support for Dockerhub private registry: JAWS now supports using private container images hosted on the JGI organization account on Dockerhub.
Improved handling for site GPU support: If a user attempts to submit a workflow with one or more tasks requiring GPU to a site where GPU computation is not available, an error message is promptly shown and the run cancelled. Previously, such tasks would remain pending for a long time (72 hours) before resulting in an error (Issue cromwell-228).

🐛 Bug Fixes

Fixed issues with jaws team add and delete subcommands (Issue central-276)
Using jaws update-user to set Slack notifications works correctly instead of displaying a server error message (Issue client-301)
Resolved issue preventing files located in the JAWS staging area from being submitted (Issue client-289)
jaws history --user <user_id> now returns history for the specified user_id (Issue client-302)
Duplicated rows in the output of jaws tasks --fmt tab have been removed (Issue client-290)
Fixed port conflict affecting NMDC and KBASE sites (Issue site-1889)
Improved GPU handling at NERSC: CPU jobs now explicitly use module=none, while GPU jobs use module=gpu for proper resource allocation (Issue pool-manager-206).

JAWS Release 2025.10 - 07/17/2025

✨ Improvements

The JAWS Client Python library has been updated to Python version >=3.10,<3.14, ensuring compatibility with newer environments. (Issue client-296).
JAWS Client has been updated to version 2.9.0. To take advantage of the latest capabilities:
- If you’re using the JAWS Client Python library, upgrade with:
```
pip install --upgrade jaws-client==2.9.0
```

🐛 Bug Fixes

Fixed a bug that affected workflows using Array[WomCompositeType] inputs (Structs). (Issue client-299).

JAWS Release 2025.9 - 06/17/2025

✨ Enhanced WDL Subworkflow Support in Validation

The jaws validate command now supports the –sub flag, allowing users to validate WDLs with subworkflow dependencies packaged as .zip files. (Issue client-281).

This feature is particularly useful for validating WDLs that include subworkflows, ensuring that all dependencies are correctly resolved and validated together. For example, if you have a main WDL file that imports a subworkflow from a .zip file, you can validate it as follows:

jaws validate WDLs/test-wdl-subworkflow/main.wdl --sub WDLs/test-wdl-subworkflow/child1.zip

JAWS will automatically resolve imports and validate across all files within the .zip.

Note: You can use the same –sub option during submission with jaws submit:

jaws submit main.wdl inputs.json --sub child1.zip

✨ Improvements

jaws info now reports the JAWS client version for easier debugging and environment tracking. (Issue client-288).

jaws info
{
  "JAWS_Central_version": "2.7.0",
  "JAWS_Client_version": "2.8.0",
  "deployment": "prod",
  "docs_url": "https://jaws-docs.readthedocs.io/en/latest/"
}

Added support for dynamic mounting of SDM file systems. (Issue cromwell-217).

🐛 Bug Fixes

jaws.submit now correctly suppresses “Using cached copy…”` when the --quiet flag is used. (Issue client-291).
Fixed a bug affecting local transfers to the JAWS teams folder. (Issue site-1819).
Fixed container mount errors when using dynamic refdata paths. (Issue cromwell-216).
Resolved intermittent failures in Cromwell file downloads by increasing Akka HTTP’s response-entity-subscription-timeout. This prevents premature termination of HTTP streams when downloading large input files, particularly affecting NMDC workflows. (Issue cromwell-220).
Increased Akka HTTP’s connection pool size (akka.http.host-connection-pool.max-connections) for improved concurrency on NMDC sites. (Issue cromwell-222).

✨ New Client Version

JAWS Client has been updated to version 2.8.0. To take advantage of the latest capabilities:

If you’re using the JAWS Python library, upgrade with:

pip install --upgrade jaws-client==2.8.0

🚧 In Progress

We are working on the new version of JAWS Central, which will include a new REST API version.
Ongoing deployment of new JAWS Sites at: - Argonne Leadership Computing Facility (ALCF) – Crux - Oak Ridge Leadership Computing Facility (OLCF) – Defiant

JAWS Release 2025.8 - 04/24/2025

✨ Improvements

Increased the number of worker and RMQ connections for JAWS Central. This change improves the performance of the JAWS Central service and reduces the number of errors related to connection limits. (Issue central-256).

JAWS Release 2025.7 - 04/15/2025

✨ New Features and Improvements

jaws health Command Improvements:

Introduced a new –site option for the jaws health command, allowing users to check the status of specific JAWS sites. The command now accepts a comma-separated list of sites (e.g., perlmutter,dori) to check their health status. (Issue client-283).
jaws health --site perlmutter,dori { "Central": "UP", "RabbitMQ": "UP", "dori-Site": "UP", "perlmutter-Site": "UP" }
Usage information is also available via jaws health -h:
```
Usage: jaws health [OPTIONS]

  Display the current system status and health.

Options:
  --site TEXT  A list of site(s) to check; default=all
  -h, --help   Show this message and exit.
```
This option is available for the jaws-client==2.7.0 version and later. If you are using the jaws-client python library, you can update it using the following command:
```
pip install --upgrade jaws-client==2.7.0  --index-url https://code.jgi.doe.gov/api/v4/projects/312/packages/pypi/simple
```
Improved Task Return Code Visibility:

The jaws tasks command now displays the return code for each task, providing users with clearer insights into task outcomes. Additionally, an issue where jaws tasks failed to display return code 79 has been resolved. This update improves debugging and facilitates better metric collection. (Issue Support#281).

🐛 Bug Fixes

Fixed the jaws cancel issue where runs got stuck in the canceled state and would not transition to `done. The cancellation lifecycle now completes as expected, ensuring proper state transitions. (Issue central-239).

JAWS Release 2025.6 - 04/07/2025

🐛 Bug Fixes

Resolved issues that limited the download of URL-based inputs by increasing the akka.http.client.parsing.max-content-length and adding the akka.http.routing.decode-max-size setting, both configured to 45 GB. These changes enhance support for large input files for the NMDC sites. (Issue cromwell-210).

JAWS Release 2025.5 - 04/01/2025

✨ New Features and Improvements

Introduced a new admin-only command to renew user tokens and deactivate user accounts. (Issue client-276 and Issue central-234).
Removed deprecated functions and improved help documentation. (Issue client-271)
- jaws task-log and jaws task-summary were replaced by jaws tasks.
- Previously, jaws get was responsible for downloading the output files from the Cromwell execution folder to the user space. Now, this process is automatic for successful runs. For failed runs, files are not automatically copied to the team directory, but you can force the download using the jaws download command.

🐛 Bug Fixes

Fixed a bug that prevented the inclusion of subworkflows as zip files when using the --sub flag. (Issue client-270)
jaws submit <WDL> <inputs.json> --sub <subworkflows.zip> <SITE>
Improved error handling when arguments (inputs.json, subworkflows, or site) are missing for the jaws submit command. (Issue client-274)
Improved error messages when configuration files (jaws-site.conf or jaws.conf) are missing. (Issue client-279)
Fixed an issue with pulling GitLab registry images on Dori. (Issue cromwell-207)
Fixed an issue with the max CPU variable for exvivo resources on Dori. (Issue site-1881)

✨ JAWS Resiliency

Improved end-to-end testing and the monitoring system. These tests are no longer run in the SLURM queue, making the use of allocations more efficient. (Issue cromwell-205)

🚧 In Progress

Started working the deployment of a new JAWS SITE at ALCF - Crux Computing.
Continue efforts to deployment of a new JAWS SITE at OLCF - Defiant Computing.

JAWS Release 2025.4 - 02/28/2025

🐛 Bug Fixes

Fixed the cleanup routine where the JAWS pool-manager was not canceling unused nodes on NERSC systems. (Issue 188).

JAWS Release 2025.3- 02/11/2025

✨ Improvements

JAWS Staging Input Optimization: The JAWS client now identifies existing inputs at the JAWS Staging Area (Inputs files) and avoids copying them again. (Issue 260).
Consistent Error Response Formatting: All error responses from the JAWS Python Library are now returned in a structured JSON format. This change improves error handling and makes it easier for users to parse and debug issues. (Issue 259).

JAWS Release 2025.2 - 02/04/2025

✨ New Features and Improvements

Upgraded Cromwell and Womtool versions to 87 (Issue 202).

JAWS now supports GPU-enabled tasks for Perlmutter NERSC site. (Issue 54).

Below is an example of how to request GPU resources in a WDL workflow:

version 1.0

workflow GPU_Test_Workflow {
  call GPU_Test_Task
}

task GPU_Test_Task {
  command {
    echo "Testing GPU runtime capabilities"
    python3 -c "import torch; print('CUDA Available:', torch.cuda.is_available()); print('Number of GPUs:', torch.cuda.device_count()); print('1 GPU Available:', torch.cuda.device_count() >= 1)"
  }
  output {
    File gpu_test_output = stdout()
  }
  runtime {
    docker: "pytorch/pytorch:latest"
    memory: "1GiB"
    cpu: 1
    gpu: true
    runtime_minutes: 10
  }
}

Enabling GPU support: The gpu: true setting in the runtime block enables GPU resources for this task.
JAWS will request 1 GPU node in the underlying scheduler.
Docker container compatibility: The container must support GPUs (e.g., pytorch/pytorch:latest includes CUDA).

Adjusted small worker configuration for jgi site:
- 40 nodes with 16 CPUs / 58 GB RAM for small workers.
- 6 nodes allocated for 32 CPUs / 512 GB RAM.
- Please refer to the Compute Resources page for details.

🐛 Bug Fixes

Fixed dynamic mount path issue for KBase (Issue Cromwell-199).
Added dynamic mounting location for KBase at the JGI JAWS Site (Issue cromwell-188).
Fixed Refdata for NMDC Tahoma (Issue 198).
Fixed JAWS tasks folder permissions for group access (Issue 1857).

🚧 In Progress

Started working the deployment of a new JAWS SITE at ANL - Crux Computing.

JAWS Release 2025.1 - 01/16/2025

✨ New Features and Improvements

WDL Variable Validation: JAWS now validates memory and CPU values in the runtime section when they are provided as WDL variables. If requested resources exceed system limits, an error is displayed in stderr.submit, helping users quickly identify and address resource allocation issues. (Issue 167)
ERROR:check_cpu_mem_requests.cli:Requested number of CPUs cannot be supported: max CPUs supported, 36 < 64 Requesed cpu/memory is larger than the supported compute resource.
Dynamic Mounting for KBase: Added support for dynamic mounting location for KBase. (Issue 179)
Input Type Warning: Introduced a warning when users define an input as a String, but the value resembles a file path (e.g., /some/filepath/sample.fastq). (Issue 249)
Extended Task Time Limit: Since queue times have increased, we have also increased the time limit for tasks to remain IDLE in the queue from 72 hours to 144 hours. (Issue 171)

🐛 Bug Fixes

Duplicate Slack Messages Resolved: Fixed an issue where JAWS would send duplicate Slack notifications for canceled runs. (Issue 216)
Container Pull Fix: Resolved an issue with container image pulls on the Dori. (Issue 195)
Error Message Cleanup: Removed the unnecessary “chmod: operation not permitted” error message in stderr.submit. (Issue 185)
Task Issue Fixes: Fixed an issue with the jaws tasks command to ensure all tasks are displayed correctly in the table.
Improved Resilience for IO Errors: Improve local file transfer processes by adding automatic retries for IO errors, reducing the risk of incomplete file transfers.

2024 Releases

JAWS Release 7/2024 - 11/12/2024

✨ New Features and Improvements

KBase Site: A new JAWS site, kbase, has been added. This site is located at the NERSC cluster and is dedicated to supporting the KBase project.
Team-Based Site Permissions: Teams are now required to obtain permissions to use certain sites. Below is a summary of permissions granted to each team across sites: (Issue 237)
- JAWS Teams with Access to Dori, Perlmutter, JGI, and Tahoma:
  - gt-ga, dsi-ii, bionano, sc-mcr, dsi-syseng, workshop, plant_ha, mycocosm, plantcompgen, gaa, gt-seqtech, funcgen, sc-mds, gt-usa, micro-scale, rqc, dsi-aa, gt-syn, phytzm
- KBase Team: Can submit to jgi and kbase sites
- NMDC Team: Can submit to nmdc and nmdc_tahoma sites
- ORNL Team: Can submit to the defiant site
WDL URLs and Subworkflows: Users can now submit WDLs as URLs, with support for subworkflows that can either be relative paths or full URLs. (Issue 224)
jaws submit https://code.jgi.doe.gov/advanced-analysis/jaws/test-repo/-/raw/main/main.wdl dori

Please make sure the repository is public and provide the raw URL to the WDL argument.

jaws purge Command: Added the jaws purge command, allowing users to delete cromwell-execution folders for their own runs or those of team members, improving storage management. (Issue 235)
jaws status command is now brief by default, with the –verbose flag available for additional details. The help message for the –verbose flag has also been updated for clarity. (Issue 239 and Issue 229)
A clearer error message is now provided when a user submits jaws submit <wdl> <inputs> but omits the SITE argument. (Issue Client-214)
jaws-history command now has a –verbose flag, which defaults to False for a streamlined output. (Issue 227)
If the WDL tasks runtime_minutes is not specified, Cromwell will default to 35 minutes. We highly recommend specifying the runtime_minutes in the WDL file. (Issue Cromwell-175).
jaws tasks command now displays input and output sum sizes for each task in bytes. (Issue 1841).

🐛 Bug Fixes

jaws tasks command: - jaws tasks command now works on Tahoma. - Fixed an issue where jaws tasks did not display all tasks for a run.
The copy progress bar is now functional again for the jaws submit command.
Links in the documentation have been updated for deprecated functions to ensure accurate guidance (Issue 245).
Resolved an issue with the jaws validate <WDL> <inputs> command (Issue 232).
The result field in jaws status now resets to NULL when a run is resubmitted, providing accurate tracking of resubmissions (Issue 209).
Fixed an error where JAWS would attempt to validate subworkflows located in nested directories, ensuring proper workflow validation (Issue 213).
Improved the error message for failed container pulls, providing clearer troubleshooting information. (Issue 183).
Fixed an issue where JAWS did not update the container image when a tag was reused with a new SHA, ensuring that updated containers are recognized (Issue 176).

JAWS Release 6/2024 - 09/17/2024

✨ New Features

Supplementary files are now downloaded to user JAWS Teams directory upon run failure (Issue 210).

🐛 Bug Fixes

Users can now delete their own folders, specifically the <JAWS RUN ID> folder, after completing a local transfer on Dori (input and compute site are the same). (Issue 1840).
jaws submit command will now throw an error if the user is not part of a JAWS team, preventing unauthorized access and enhancing security. (Issue 222).

JAWS Release 5/2024 - 09/10/2024

✨ New Features and Improvements

JAWS Client Python Library and Documentation:
- Added the JAWS Python library to the JAWS Client. This library provides a Python interface to interact with JAWS, allowing users to submit runs, check run status, and download outputs programmatically.
- The JAWS Client Python library documentation is available at JAWS Python Library section. (Issue 205).
JAWS Staging Area Overview
- Input File Cache Validation:
  
  When a run is submitted, the JAWS Client copies the input files to a designated “JAWS staging area.” For subsequent submissions using the same input files, JAWS will reference the cached copies instead of re-copying them. If an input file content has changed but its filename remains the same, JAWS will detect this and provide an error message. Here is an example of the error message:
```
jaws submit --no-cache align_final.wdl inputs.json dori
Error initializing Run: Unable to copy input files:
/clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/data/sample.fasta is different from its cached version.
Submitting with this input file can affect previous runs.
Use --overwrite-inputs to force update the cached input files.
```
  This ensures that input consistency is maintained and prevents files linked to previous runs from being unintentionally modified. (Issue 220 and Issue 206).
- Overwrite Inputs:
  
  Users can explicitly provide the –overwrite-inputs flag if they wish to update cached input files with a new version. (Issue 220).
```
jaws submit --no-cache align_final.wdl inputs.json dori --overwrite-inputs
```

For more information about the JAWS Staging Area, please refer to JAWS documentation.

Display Return Code (RC) to the jaws tasks Command:

The jaws tasks command now includes the RC file content for each task. This improvement provides users with additional context when troubleshooting task failures. (Issue Jaws-Cromwell-168 and Issue 1832).
Updated 48-Hour Maximum Walltime on Perlmutter:

The maximum walltime for jobs on the Perlmutter compute system has been updated to 48 hours. (Issue 150).

Please check the walltime limit for all available JAWS sites.

✨ Jaws Resiliency

Local Output File Transfer Optimization:

JAWS has transitioned from using parallel-sync to Python’s built-in shutil.copy for local output file transfers. This change addresses several performance and compatibility issues:
- parallel-sync was rewritten in version 2.0; this new version did not include a local transfer function.
- The last compatible version of parallel-sync was incompatible with Python 3.11 due to outdated dependencies like the brunch package.
- By switching to shutil.copy, JAWS now leverages optimized kernel-level fast copy calls, improving performance and reducing dependency on external libraries.
- In addition to shutil.copy, JAWS now uses filecmp.cmp to verify the integrity of local file transfers by comparing file attributes such as size, type, and modification time. This method is more CPU efficient than using md5sum and provides the same level of accuracy. (Issue 1819 and Issue 1818 and Issue 214).

🐛 Bug Fixes

Improved error messaging for JAWS cancel command to provide clearer guidance when the RUN_ID is missing. (Issue 199).
Display RC File Content with Timestamp:
- Added a print statement to stderr.submit that outputs the contents of the RC file along with a timestamp.
- This helps in debugging issues such as the Cromwell error (rc=79), where the RC file might initially show rc=0 but later be overwritten by Cromwell with a rc=79. (Issue 160).
Users can now delete their own folders, specifically the run-id folder, after completing a Globus download. (Issue 238).

⚖️ Policy Updates

Fair Share Policy Update:
- Updated the fair share policy for Dori, allowing a maximum of 30 runs per user on the Dori cluster. (Issue 1826).
Please notice that we update our time limits for all the sites. Please check the walltime limit for all available JAWS sites. (Issue 1836).

Time Overhead

When Cromwell submits a task, HTCondor manages job scheduling by checking the queue for available resources. The JAWS Pool Manager monitors HTCondor and, when needed, requests new Slurm nodes. Once a compute node is available, HTCondor submits the task.

Due to a slight delay (a few seconds) in resource allocation, we build in a time buffer to ensure jobs get the full requested time. For example, instead of requesting the maximum 48 hours on Perlmutter, we request 47 hours and 45 minutes to account for the delay.

JAWS Release 4/2024 - 05/21/2024

✨ Jaws Resiliency

JAWS Central and its databases have been migrated to IGB.
We have deployed a new site, jaws defiant, at the Advanced Computing Ecosystem (ACE) testbed located at the Oak Ridge Leadership Computing Facility (OLCF).

JAWS Release 3/2024 - 05/08/2024

🐛 Bug Fixes

Fixed bug here multiple users sharing the same input file caused conflicts. (Issue 208)

JAWS Release 2/2024 - 05/01/2024

✨ New Features

NMDC Tahoma Site: We’ve added a new JAWS site: nmdc_tahoma. This new site is located at the EMSL cluster, and it is dedicated to supporting the NMDC project.

Enhanced Error Messaging: Improvements have been made to the error messages for backend task failures, providing clearer guidance for troubleshooting. (Issue Pool-Manager-118)

jaws history command has new filtering options, allowing searches by JSON file name (e.g., my_inputs.json), WDL file name, user, or tag, using flags such as –json-file, –wdl, –user, and –tag. (Issue 88)

🆕 New behavior for JAWS

Dori exvivo nodes will now be dedicated solely to large memory tasks, optimizing resource allocation and enhancing performance. (Issue 122)

If a Cromwell task returns a code of 79 (error code related to file system issues), JAWS will automatically retry the task once more. (Issue 139)

The JAWS_SITE environment variable is now exported within container environments for running tasks. This means you can use $JAWS_SITE from your WDL commands{} section for conditional statements (if $JAWS_SITE == "dori" then do something). (Issue 142)

Slack notification now contains the workflow_root field. (Issue 191)

🐛 Bug Fixes

Fixed bug where JAWS was not able to create a user subdirectory under the Teams dir. (Issue 1790)

Fixed bug for parallel copy files. Parallel copy happens in lieu of globus when submitting jobs to the same site as you submitted from. (Issue 1782)

Added retries for Jaws-site retrieving metadata from cromwell. (Issue 1787)

JAWS First Winter Release - 02/07/2024

✨ New Features

JAWS will issue a warning if a File type is hardcoded in the WDL file. (Issue 128)
We have implemented a validation process to ensure the WDL file aligns with the inputs.json file, using womtool. This validation occurs automatically upon submission and can also be triggered manually by executing the jaws validate command. (Issue 175)
jaws validate <WDL_FILE> <INPUTS_JSON>

🆕 New behavior for JAWS

We update the Fair-Share policies: (Issue 136)

Task-Level Control:

Limit: There’s now a maximum limit of 200 concurrent tasks per user.

Queue: Tasks submitted above the 200-task limit will enter a queue.

Only successful runs will be copied to the team’s directory. (Issue 154)

Our backend system (HTCondor) will attempt to execute a task twice under certain conditions. If your task exceeds its allocated time, or if JAWS-site needs to be reassigned to a new node (Perlmutter site), or due to some instability in the backend, we will automatically initiate a second attempt to run the task.

Important to note:

When the backend or Cromwell retries a task, the cromwell-execution folder will not be reset. In specific cases, this requires adding a verification process in your WDL command stanza. For example, if the command includes creating directories or specific files. Depending on the outcome, you’ll need to determine whether to force the directory or file to be recreated or to skip this step. (Issue Pool-Manager-110)

🐛 Bug Fixes

Fixed issue related to retrieving metadata from runs that contain over 1 million rows. (Issue Cromwell-137)

A user-friendly error message is displayed when an invalid run_id is entered in the jaws status <RUN_ID> or jaws log <RUN_ID> commands. (Issue 177)

Fixed file permission issue for files transferred to the team’s directory via Globus from a compute site that is different from the input site. (Issue 168 and 1739)

Fixed issue related to transferring very large output datasets. (Issue 1746)

Return an error message to the user when Cromwell submission fails. (Issue 1568)

Updated the refdata path for the NMDC site (/global/cfs/cdirs/m3408/refdata). (Issue 1756)

workflow_root: null bug is fixed. (Issue 1747)

Added a tag to the summary.json supplementary file. (Issue 1758)

Increased the length of the JSON basename field. (Issue 1776)

Fixed job_id column in the jaws task output. (Issue 1749)

2023 Releases

JAWS Second Fall Release - 11/09/2023

✨ New Features

JAWS has incorporated support for specifying execution time in the runtime section. In addition to memory and cpu, you can now include the time required to run each task. The mandatory key for specifying this information is runtime_minutes.

runtime_minutes:

Accepted types: Int

Int: minutes. Example:
runtime {
  docker: "ubuntu@sha256:c9cf959fd83770dfdefd8fb42cfef0761432af36a764c077aed54bbc5bb25368"
  runtime_minutes: 60
  memory: "5G"
  cpu: 4
}
The benefit of specifying runtime_minutes is that it provides a guarantee that the task will be put on a node with sufficient time.

⚠️ If the WDL runtime section uses time as a key or doesn’t specify any time value, the workflow will still be accepted but without the assurance it will be allocated to a suitable node.
✨ JAWS Client Container is Available ✨
How to use:
Dori:
JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/clusterfs/jgi/groups/dsi/homes/svc-jaws/dori-prod/jaws-prod.conf apptainer run docker://doejgi/jaws-client:latest jaws --help
Append to the end of your ~/.bashrc on DORI:
jaws() {
  JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/clusterfs/jgi/groups/dsi/homes/svc-jaws/dori-prod/jaws-prod.conf apptainer run docker://doejgi/jaws-client:latest jaws "$@"
}
Perlmutter:
JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/global/cfs/cdirs/jaws/perlmutter-prod/jaws-prod.conf shifter --image=doejgi/jaws-client:latest jaws --help
Append to the end of your ~/.bashrc on NERSC:
jaws() {
  JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/global/cfs/cdirs/jaws/perlmutter-prod/jaws-prod.conf shifter --image=doejgi/jaws-client:latest jaws "$@"
}

🆕 New behavior for JAWS

In our ongoing efforts to enhance performance and maintain a robust, up-to-date environment, we will be discontinuing support for WDL Draft-2 version 12/1/2023.

We will display a ⚠️ warning message to notify you if you are still using the old version when submitting a run.

We strongly urge all users to update their WDL workflows to the Version 1.0 specification. If you have any questions, please schedule a pair programming session with our Team.

🐛 Bug Fixes

We added a new flag to jaws validate to show shellCheck linter validation. In addition, the output also now prints newlines for better readability.

Addressed the issue “noise” being printed to stderr for Dori runs. (Issue 130)

Increased allowed length of WDL and JSON filenames. (Issue 1744)

jaws health command is now operational again.

workflow_root: null bug is fixed. (Issue Jaws-Support-148)

JAWS First Fall Release - 10/09/2023

✨ New Features

Dori Exvivo nodes are now available for JAWS. JAWS is using the long queue (14 days) and can access up to 1.5 TB of memory. (Issue 83)

Globus endpoint is available for Perlmutter! 🎉 (Issue 1683)

This means that you are now able to submit from Perlmutter to Dori/Tahoma/JGI sites and vice-versa.
Teams output directory will include JAWS user ID and will have the following structure (Issue 137):
/<TEAM PATH>/<JAWS_USER_ID>/<RUN_ID>/<Cromwell_ID>
New command: jaws tasks <RUN_ID>:
jaws tasks 18279
#TASK_DIR       STATUS     QUEUE_START          RUN_START            RUN_END              QUEUE_MIN  RUN_MIN  CACHED  TASK_NAME          REQ_CPU  REQ_GB  REQ_MIN  CPU_HRS
call-alignment  succeeded  2023-10-02 16:55:07  2023-10-02 16:56:16  2023-10-02 16:56:22  1          0        False   bbtools.alignment  1        1                0.0
call-samtools   succeeded  2023-10-02 16:56:36  2023-10-02 16:56:55  2023-10-02 16:56:57  0          0        False   bbtools.samtools   1        1                0.0
jaws tasks integrate two previously distinct commands, jaws task-log and jaws task-summary. Be sure to explore the newly unified and enhanced features:

“cpu_hours” metric is now included in the jaws tasks command for each task and jaws status as an aggregation for the entire workflow upon run completion.

Cached tasks will now be recorded in the jaws tasks post run completion.

jaws tasks status is now updated appropriately when a run is cancelled.

jaws tasks now uses timestamps derived from the task log instead of using Cromwell metadata.
New command: jaws download <RUN_ID>:

If a run fails, JAWS will skip the transfer to team’s directory. However, if you need the cromwell-execution for debugging reasons, you can use our new command to ‘force’ the download of the run:
jaws download 18386
{
    "download_id": 8272,
    "id": 18386,
    "status": "download queued"
}


- Please notice that the run output will be transferred to the team's directory.
New command jaws get-user (Issue 107)

This command gets current user’s settings.
jaws get-user
{
    "email": "[email protected]",
    "name": "Daniela Cassol",
    "slack_id": "<Member_ID>",
    "teams": [
        "dsi-aa",
        "nmdc"
    ],
  "uid": "dcassol"
}
Bash commands employing curl is now working on Tahoma. (Issue 121)

Output directory will include a copy of the original WDL, input.json, and subworkflows-ZIP files, for reproducibility. (Issue 1710)

jaws status is now verbose by default. If you prefer the short version, please use jaws status --brief. To ensure backward compatibility, when you use the command jaws status --verbose, it will issue a warning and additionally display the output of jaws status.

JAWS is now using the latest version of Cromwell-85.

🐛 Bug Fixes

JAWS will not transfer workflow outputs to teams output location runs that were canceled by the user. (Issue 156)

Fixed teams output directory permissions when the submission and compute site are the same. (Issue 1712)

Fixed issue when workflow name and site are the same. (Issue 23)

jaws resubmit command is restricted to the members of the team who own the run. (Issue 148)

We have improved the error messaging for jaws resubmit, especially when the original submission failed. (Issue 116)

jaws status is now displaying local time zone. (Issue 151)

jaws status will report when a run was canceled by the user in the result field. (Issue 1714)

We are ignoring extra attributes to the runtime{} stanza, for example "runtime_minutes": "20". (Issue 118)

When a run is canceled, it will be recorded to jaws tasks <RUN_ID>. (Issue 120)

Fixed issues reporting the wrong task status in jaws tasks <RUN_ID> command.

Fixed error for transfers when file was named pipe instead of regular file. (Issue 1725)

When a run is resubmitted (jaws resubmit <RUN_ID>), JAWS will ensure that all the input files required are still available. (e.g., haven’t been purged). (Issue Jaws-Support-110)

When a run is resubmitted (jaws resubmit <RUN_ID>), JAWS will update access timestamp (atime) for input files, in order to avoid purging files prematurely. (Issue 1689)

Fixed bug when Cromwell submission fails during input processing and was not recognized by JAWS. (Issue 1711)

outputs.json supplementary file contain relpaths instead of abspaths. (Issue 1652)

🆕 New behavior for JAWS

In our ongoing efforts to enhance performance and maintain a robust, up-to-date environment, we will be discontinuing support for WDL Draft-2 version 11/1/2023.

We will display a ⚠️ warning message to notify you if you are still using the old version when submitting a run.

We strongly urge all users to update their WDL workflows to the Version 1.0 specification. If you have any questions, please schedule a pair programming session with our Team.

JAWS Cromwell configuration was updated, and now Container tags can be used for call caching (Issue Jaws-Cromwell-122, Issue Jaws-Client-156).

We recommended referencing containers by their SHA256 instead of tag (e.g., doejgi/bbtools@sha256:64088.. instead of doejgi/bbtools:latest). While using mutable or “floating” tags in tasks can be convenient in certain scenarios, it adversely impacts the reproducibility of a workflow. For instance, executing the same workflow with doejgi/bbtools:latest now, and then rerunning it in a month or a year, could result in the use of different container images.

If the command stanza uses $TMPDIR, it will have access to /tmp. Previous we set to execution/ directory (e.g., NFS). (Issue 110)

❌ Deprecated Commands

jaws task-log and jaws task-summary have been deprecated (Issue 161, Issue 1719).

These two commands have been merged and into a new command, jaws tasks.

For backward compatibility, jaws task-log will be reporting the output of jaws tasks.

⚠️ Known issues

If the submission and compute sites are different (for example, from Dori to Tahoma), there could still be permission issues in the team’s output directory. In this case, the transfer to the team’s directory happens via Globus, we need are looking into solutions for this problem.

JAWS Summer Release - 09/05/2023

We are releasing a new version of JAWS that includes breaking changes.

JAWS Teams

We’re introducing a new feature in JAWS called “JAWS Teams.” This feature allows for easier management of users and offers a centralized location for sharing and delivering output files for each team.

List all the teams available:

jaws teams list
[
    "gt-ga",
    "nmdc",
    "dsi-ii",
    "sc-mcr",
    "gt-seqtech",
    "dsi-aa",
    "gt-syn",
    "phytzm"
]

List the teams to which you belong:

jaws teams my-teams
[
    "dsi-aa"
]

List the users associated with a team:

jaws teams members dsi-aa
[
    "dcassol",
...
]

Get a team’s site config - Outputs path:

jaws teams get-site dsi-aa dori
"/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/dsi-aa"

Team’s owner has power and responsibilities, such as setting the path for each site, adding and deleting users from the team:

jaws teams set-site <TEAM_ID> <SITE_ID> <PATH>
jaws teams add-user <TEAM_ID> <USER_ID>
jaws teams del-user <TEAM_ID> <USER_ID>

How to use Jaws Teams?

When submitting a new run, you can use --team flag. The outputs for this run will be transferred to the team’s path.

jaws submit align_final.wdl inputs.json dori --team=dsi-aa

As an easy alternative, you can set the default_team at your jaws.conf file:

vi ~/jaws.conf

[USER]
token = <TOKEN>
default_team = dsi-aa

Important: Do not use quotes for the team’s name.

The Jaws Team’s new feature will transfer the final workflow outputs from cromwell-execution to the path defined for your team.

How do I find the output data?

jaws status -–verbose <ID> will provide the path to final path. Please check output_dir.

Then, you can find the expected file tree structure:

/<TEAM PATH>/<RUN_ID>/<Cromwell_ID>

JAWS will copy the following in case of a successful run:

Workflow outputs;
Supplementary files:
- errors.json;
- metadata.json;
- output_manifest.json;
- outputs.json;
- task_summary.json.

JAWS will copy the following in case of a FAILED run:

Workflow outputs;
Supplementary files;
Failed tasks’ cromwell-execution folder only.

As we transition to copying the output files, we are deprecating out the jaws get command. To ensure backward compatibility, this command will remain functional for a few more months, serving solely to copy the final workflow outputs. However, please note that we plan to discontinue this command entirely in upcoming cycles.

We have implemented parallel copying capabilities when both the submission and compute site are the same, for example, from Dori to Dori. This effectively resolves the delay issues associated with the ‘download queue.’

Slack Notifications

We replaced Email with Slack notifications when the run is completed.

How to set up Slack Notification?

Please set up your slack_webhook and update your JAWS account. Instructions on how to get your Slack webhook are available here.

jaws update-user –[email protected] --slack_id <Member_ID> --slack_webhook <WebHook_URL>

Call-caching Strategy

The call-caching strategy that we used was “xxh64”, and that required a lot of I/O operation to calculate the hash of the entire file content. Now, we replace that with “fingerprint”. fingerprint will take the last modified time, size, and hash from the first 10 mb with xxh64 to create a file fingerprint. Please be aware of that, and please let us know if you think that can cause any file collision when the task will use call-caching. If you want to read more about the call-caching and all the strategies please check here.

Additional minor features

Added --forcequeue flag option to jaws submit command. Users can force run submission when the site has been disabled;
Resubmitting a run will change the “result” filed to “resubmitted” for jaws status <run id> command;
We now parse the runtime{} parameter values correctly when there is a space (i.e., memory: “5 G”) ticket: #111;
/refdata is mounted correctly and accessible to the WDLs.

Special note about the DORI TEAMS Folders: For this release, Teams’ folders on Dori must be located at /clusterfs/jgi/scratch/dsi/aa/jaws/teams/. However, once the Globus endpoint changes have been completed, teams’ folders may be edited under the team’s scratch.

JAWS Summer Release Slides

Link to JAWS Calendar

JAWS Sprint Release - 04/12/2023

We released a new version of JAWS.

Here are the changes that are now on PROD:

Dori Jaws site is available for testing;

Create a config file in your HOME directory:
touch ~/jaws.conf
chmod 600 ~/jaws.conf

Copy your token from CORI:

[USER]
token = <copy your token from CORI and paste it here>

Module load jaws

module use /clusterfs/jgi/groups/dsi/homes/svc-jaws/modulefiles
module load jaws/dori-prod
jaws submit <WDL_FILE> <INPUTS> dor

NOTE:

Dori and Perlmutter have temporary Globus limitations (IT has not configured JAWS application Globus endpoints yet).
Temporary workaround:
- Because data transfer isn’t available, you must log in to the cluster (Dori or Perlmutter) and submit the run from there.
- You can use Globus endpoint to transfer your data from Cori to Dori, for example.

Deprecated commands

jaws outfiles
jaws outputs
jaws metadata
jaws errors

This is part of the effort to refactor jaws metadata command (uses cromwell metadata). As users are submitting large workflows (> 10k tasks), cromwell metadata became too expensive to query and would sometimes timeout. Instead, we now wait for the run to finish and then write some associated reports (e.g. errors, outfiles, and outputs) to disc. These json files are written to the run’s execution directory and returned to users via jaws get.

To find the report files:

jaws get <RUN_ID> <DEST>
ls <DEST>
<workflow>.wdl
<inputs>.json
errors.json
metadata.json
outfiles.json
outputs.json
task_summary.json

jaws task-log changed:
- task-log is now much faster and robust than before and will support runs with greater than 10k tasks.
- Even more than before, it is a real-time reflection of a run’s current status since it gets its information directly from the backend instead of using an intermediate metadata cromwell, which was a bottleneck.
- However, because it no longer uses cromwell’s metadata, there are no longer records for cached tasks since they did not actually execute. Therefore, the “cached” column was deleted.
- Limitation for Tahoma site: jaws task-log isn’t working properly for Tahoma due to the firewall between the compute and workflow node. We have a ticket open for this issue. However, this will not affect your run!
jaws task-summary
It only will be available after the run is finished.
jaws status –verbose
Two more fields: workflow_name and workflow_root
jaws validate
miniwdl is used instead of womtool for WDL validation.
jaws resubmit
New command available! Resubmit a run (at the same compute site).

Fixed bugs reported by users

runtime parameters can be on separate lines;
Fixed jaws get --flatten;
symlinks can be used for the path to the WDL (jaws submit <symlink>/my.wdl my.json dori)
When restarting cromwell service (i.e. upon release) will not interrupt active Runs;
Added job purging policies to the HTCondor backend so that it can automatically cancel tasks that stay in HOLD status for too long time.
Added --time-min SLURM option for requesting SLURM compute nodes with a shorter wall-clock-time on Perlmutter when maintenance is scheduled.