============= Release Notes ============= .. role:: bash(code) :language: bash 2026 Releases ============= JAWS Release 2026.03 - 03/31/2026 --------------------------------- |:bug:| Bug Fixes +++++++++++++++++ - Improved exception handling for the ``sbatch`` command to ensure failures are properly surfaced rather than failing silently, making debugging easier (`pool-manager-244 `_). - Resolved an issue with GPU node request handling that was causing GPU workflows to stall indefinitely (`pool-manager-239 `_). JAWS Release 2026.02 - 03/26/2026 --------------------------------- |:sparkles:| New Features +++++++++++++++++++++++++ - **RNome Site at NERSC**: JAWS has been successfully deployed to the new **RNome** site at NERSC, expanding compute capacity. This site is dedicated to supporting the `RNome project `_, expanding the user base for JAWS. - **Enhanced GPU Support**: Added ``gpuCount`` runtime attribute and mapped ``request_gpus`` from both ``gpu`` and ``gpuCount`` for improved GPU workflow support (`cromwell-258 `_). Below is an example of how to request GPU resources in a WDL workflow using the new ``gpuCount`` attribute: .. code-block:: text version 1.0 workflow GPU_Test_Workflow { call GPU_Test_Task } task GPU_Test_Task { command <<< echo "Testing GPU runtime capabilities" ## add your GPU commands here. >>> runtime { docker: "pytorch/pytorch:latest" memory: "20GiB" cpu: 1 gpu: true gpuCount: 3 runtime_minutes: 10 } } **Key Notes:** - ``gpu: true`` enables GPU resource allocation for the task. - ``gpuCount: 3`` requests 3 GPUs from the scheduler. If ``gpuCount`` is not specified, JAWS will request **1 GPU** by default when ``gpu: true``. - The ``request_gpus`` attribute is also supported and maps to ``gpuCount`` for backward compatibility. - Ensure the Docker container used supports GPU execution (e.g., ``pytorch/pytorch:latest`` includes CUDA). - **Resource Management**: Reduced the maximum number of nodes allocated at NMDC NERSC instance (`pool-manager-236 `_). |:bug:| Bug Fixes +++++++++++++++++ - Fixed a regression where ``jaws submit`` could fail with ``KeyError: ‘library_id’`` when submitting valid workflows with nested struct inputs (`client-323 `_). - Fixed an issue where team output folder permissions were not set to site-specific team directory permissions (`site-1918 `_). - Fixed ``condor_status`` command for improved node cleanup (`pool-manager-240 `_). |:wrench:| Internal Changes +++++++++++++++++++++++++++ - This release also includes internal bug fixes and platform improvements to enhance reliability, performance, and overall user experience. |:package:| Client Version Update ++++++++++++++++++++++++++++++++++ JAWS Client has been updated to version **2.12.1**. To take advantage of the latest capabilities: - **If you're using the JAWS Client Python library, upgrade with:** .. code-block:: bash pip install --upgrade jaws-client==2.12.1 JAWS Release 2026.01 - 02/25/2026 --------------------------------- |:sparkles:| New SITE: **Superbio** at NERSC ++++++++++++++++++++++++++++++++++++++++++++ - **New Site: Superbio**: A new JAWS site, ``superbio``, has been added. - This site is located at the `NERSC` and its purpose is to support the SSLR at SLAC. |:wrench:| Additional Improvements ++++++++++++++++++++++++++++++++++ - This release also includes internal bug fixes and platform improvements to enhance reliability, performance, and overall user experience. |:package:| Client Version Update ++++++++++++++++++++++++++++++++++ JAWS Client has been updated to version **2.12.0**. To take advantage of the latest capabilities: - **If you're using the JAWS Client Python library, upgrade with:** .. code-block:: bash pip install --upgrade jaws-client==2.12.0 2025 Releases ============= JAWS Release 2025.14 - 10/29/2025 --------------------------------- |:bug:| Bug Fixes +++++++++++++++++ - Fixed NMDC `refdata` location to ensure correct data paths for NMDC workflows. This update resolves issues where workflows couldn't access required refdata (`Issue client-314 `_). - Resolved HTCondor Access Point port conflict that prevented proper communication between JAWS components. (`Issue pool-manager-219 `_). - Fixed an issue with HTCondor queue processing that incorrectly set the number of needed nodes to 0 when the first queued task required GPU resources. This caused GPU workflows to stall indefinitely waiting for resources (`Issue pool-manager-220 `_). - Ensured that the version reported by ``jaws info`` command matches the actual ``JAWS_CLIENT_VERSION`` for accurate version tracking and debugging (`Issue client-315 `_). |:package:| Client Version Update ++++++++++++++++++++++++++++++++++ JAWS Client has been updated to version **2.11.0**. To take advantage of the latest capabilities: - **If you're using the JAWS Client Python library, upgrade with:** .. code-block:: bash pip install --upgrade jaws-client==2.11.0 JAWS Release 2025.13 - 10/22/2025 --------------------------------- |:wrench:| Internal Improvements +++++++++++++++++++++++++++++++++ - Downgrade Globus SDK library to version < 4.0.0, since version 4.0.0 introduced breaking changes that affected JAWS functionality. JAWS Release 2025.12 - 10/14/2025 --------------------------------- |:bug:| Bug Fixes +++++++++++++++++ - Fixed an issue with HTCondor queue processing that caused the number of needed nodes to be underestimated (`Issue pool-manager-216 `_). JAWS Release 2025.11 - 10/03/2025 --------------------------------- |:sparkles:| New Features and Improvements ++++++++++++++++++++++++++++++++++++++++++ - **New Site: Defiant**: A new JAWS site, ``defiant``, has been added. This site is located at the `Defiant `_ system at the Oak Ridge Leadership Computing Facility (OLCF). - **New Site: Crux**: A new JAWS site, ``crux``, has been added. This site is located at the Crux system at the Argonne Leadership Computing Facility (ALCF). - **Improved detection of potential user group and/or file permission issues**: An explicit check has been added to verify that the user account running `jaws` belongs to the expected user groups. If this is not the case, a helpful message is shown. Previously, this type of issues would only be detected indirectly by seemingly unrelated errors, which made the messages misleading and harder to trace back to the underlying cause (`Issue client-264 `_). - **Support for Dockerhub private registry**: JAWS now supports using private container images hosted on the JGI organization account on Dockerhub. - **Improved handling for site GPU support**: If a user attempts to submit a workflow with one or more tasks requiring GPU to a site where GPU computation is not available, an error message is promptly shown and the run cancelled. Previously, such tasks would remain pending for a long time (72 hours) before resulting in an error (`Issue cromwell-228 `_). |:bug:| Bug Fixes +++++++++++++++++ - Fixed issues with ``jaws team`` ``add`` and ``delete`` subcommands (`Issue central-276 `_) - Using `jaws update-user` to set Slack notifications works correctly instead of displaying a server error message (`Issue client-301 `_) - Resolved issue preventing files located in the JAWS staging area from being submitted (`Issue client-289 `_) - ``jaws history --user `` now returns history for the specified ``user_id`` (`Issue client-302 `_) - Duplicated rows in the output of ``jaws tasks --fmt tab`` have been removed (`Issue client-290 `_) - Fixed port conflict affecting NMDC and KBASE sites (`Issue site-1889 `_) - **Improved GPU handling at NERSC**: CPU jobs now explicitly use ``module=none``, while GPU jobs use ``module=gpu`` for proper resource allocation (`Issue pool-manager-206 `_). JAWS Release 2025.10 - 07/17/2025 --------------------------------- |:sparkles:| Improvements +++++++++++++++++++++++++ - The JAWS Client Python library has been updated to Python version `>=3.10,<3.14`, ensuring compatibility with newer environments. (`Issue client-296 `_). - JAWS Client has been updated to version **2.9.0**. To take advantage of the latest capabilities: - **If you're using the JAWS Client Python library, upgrade with:** .. code-block:: bash pip install --upgrade jaws-client==2.9.0 |:bug:| Bug Fixes +++++++++++++++++ - Fixed a bug that affected workflows using `Array[WomCompositeType]` inputs (`Structs `_). (`Issue client-299 `_). JAWS Release 2025.9 - 06/17/2025 -------------------------------- |:sparkles:| **Enhanced WDL Subworkflow Support in Validation** ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - The `jaws validate` command now supports the `--sub` flag, allowing users to validate WDLs with subworkflow dependencies packaged as `.zip` files. (`Issue client-281 `_). This feature is particularly useful for validating WDLs that include subworkflows, ensuring that all dependencies are correctly resolved and validated together. For example, if you have a main WDL file that imports a subworkflow from a `.zip` file, you can validate it as follows: .. code-block:: bash jaws validate WDLs/test-wdl-subworkflow/main.wdl --sub WDLs/test-wdl-subworkflow/child1.zip JAWS will automatically resolve imports and validate across all files within the `.zip`. **Note:** You can use the same `--sub` option during submission with `jaws submit`: .. code-block:: bash jaws submit main.wdl inputs.json --sub child1.zip |:sparkles:| Improvements +++++++++++++++++++++++++ - `jaws info` now reports the JAWS client version for easier debugging and environment tracking. (`Issue client-288 `_). .. code-block:: text jaws info { "JAWS_Central_version": "2.7.0", "JAWS_Client_version": "2.8.0", "deployment": "prod", "docs_url": "https://jaws-docs.readthedocs.io/en/latest/" } - Added support for dynamic mounting of SDM file systems. (`Issue cromwell-217 `_). |:bug:| Bug Fixes +++++++++++++++++ - :bash:`jaws.submit` now correctly suppresses `"Using cached copy..."`` when the :bash:`--quiet` flag is used. (`Issue client-291 `_). - Fixed a bug affecting local transfers to the JAWS teams folder. (`Issue site-1819 `_). - Fixed container mount errors when using dynamic `refdata` paths. (`Issue cromwell-216 `_). - Resolved intermittent failures in Cromwell file downloads by increasing Akka HTTP's `response-entity-subscription-timeout`. This prevents premature termination of HTTP streams when downloading large input files, particularly affecting NMDC workflows. (`Issue cromwell-220 `_). - Increased Akka HTTP's connection pool size (`akka.http.host-connection-pool.max-connections`) for improved concurrency on NMDC sites. (`Issue cromwell-222 `_). |:sparkles:| **New Client Version** +++++++++++++++++++++++++++++++++++ JAWS Client has been updated to version **2.8.0**. To take advantage of the latest capabilities: - **If you're using the JAWS Python library, upgrade with:** .. code-block:: bash pip install --upgrade jaws-client==2.8.0 |:construction:| In Progress ++++++++++++++++++++++++++++ - We are working on the new version of JAWS Central, which will include a new REST API version. - Ongoing deployment of new JAWS Sites at: - **Argonne Leadership Computing Facility (ALCF)** – *Crux* - **Oak Ridge Leadership Computing Facility (OLCF)** – *Defiant* JAWS Release 2025.8 - 04/24/2025 -------------------------------- |:sparkles:| Improvements +++++++++++++++++++++++++ - Increased the number of worker and RMQ connections for JAWS Central. This change improves the performance of the JAWS Central service and reduces the number of errors related to connection limits. (`Issue central-256 `_). JAWS Release 2025.7 - 04/15/2025 -------------------------------- |:sparkles:| New Features and Improvements ++++++++++++++++++++++++++++++++++++++++++ - `jaws health` Command Improvements: Introduced a new `--site` option for the `jaws health` command, allowing users to check the status of specific JAWS sites. The command now accepts a comma-separated list of sites (e.g., `perlmutter,dori`) to check their health status. (`Issue client-283 `_). .. code-block:: text jaws health --site perlmutter,dori { "Central": "UP", "RabbitMQ": "UP", "dori-Site": "UP", "perlmutter-Site": "UP" } Usage information is also available via :bash:`jaws health -h`:: Usage: jaws health [OPTIONS] Display the current system status and health. Options: --site TEXT A list of site(s) to check; default=all -h, --help Show this message and exit. This option is available for the `jaws-client==2.7.0` version and later. If you are using the `jaws-client` python library, you can update it using the following command: .. code-block:: text pip install --upgrade jaws-client==2.7.0 --index-url https://code.jgi.doe.gov/api/v4/projects/312/packages/pypi/simple - Improved Task Return Code Visibility: The `jaws tasks` command now displays the return code for each task, providing users with clearer insights into task outcomes. Additionally, an issue where `jaws tasks` failed to display return code `79` has been resolved. This update improves debugging and facilitates better metric collection. (`Issue Support#281 `_). |:bug:| Bug Fixes +++++++++++++++++ - Fixed the `jaws cancel` issue where runs got stuck in the `canceled state and would not transition to `done`. The cancellation lifecycle now completes as expected, ensuring proper state transitions. (`Issue central-239 `_). JAWS Release 2025.6 - 04/07/2025 -------------------------------- |:bug:| Bug Fixes +++++++++++++++++ - Resolved issues that limited the download of URL-based inputs by increasing the `akka.http.client.parsing.max-content-length` and adding the `akka.http.routing.decode-max-size` setting, both configured to 45 GB. These changes enhance support for large input files for the NMDC sites. (`Issue cromwell-210 `_). JAWS Release 2025.5 - 04/01/2025 -------------------------------- |:sparkles:| New Features and Improvements ++++++++++++++++++++++++++++++++++++++++++ - Introduced a new admin-only command to renew user tokens and deactivate user accounts. (`Issue client-276 `_ and `Issue central-234 `_). - Removed deprecated functions and improved help documentation. (`Issue client-271 `_) - :bash:`jaws task-log` and :bash:`jaws task-summary` were replaced by :bash:`jaws tasks`. - Previously, :bash:`jaws get` was responsible for downloading the output files from the Cromwell execution folder to the user space. Now, this process is automatic for successful runs. For failed runs, files are not automatically copied to the team directory, but you can force the download using the `jaws download` command. |:bug:| Bug Fixes +++++++++++++++++ - Fixed a bug that prevented the inclusion of subworkflows as zip files when using the :bash:`--sub` flag. (`Issue client-270 `_) .. code-block:: text jaws submit --sub - Improved error handling when arguments (`inputs.json`, `subworkflows`, or `site`) are missing for the :bash:`jaws submit` command. (`Issue client-274 `_) - Improved error messages when configuration files (`jaws-site.conf` or `jaws.conf`) are missing. (`Issue client-279 `_) - Fixed an issue with pulling GitLab registry images on Dori. (`Issue cromwell-207 `_) - Fixed an issue with the `max CPU` variable for exvivo resources on Dori. (`Issue site-1881 `_) |:sparkles:| JAWS Resiliency ++++++++++++++++++++++++++++ - Improved end-to-end testing and the monitoring system. These tests are no longer run in the SLURM queue, making the use of allocations more efficient. (`Issue cromwell-205 `_) |:construction:| In Progress ++++++++++++++++++++++++++++ - Started working the deployment of a new JAWS SITE at ALCF - Crux Computing. - Continue efforts to deployment of a new JAWS SITE at OLCF - Defiant Computing. JAWS Release 2025.4 - 02/28/2025 -------------------------------- |:bug:| Bug Fixes +++++++++++++++++ - Fixed the cleanup routine where the `JAWS pool-manager` was not canceling unused nodes on NERSC systems. (`Issue 188 `_). JAWS Release 2025.3- 02/11/2025 ------------------------------- |:sparkles:| Improvements +++++++++++++++++++++++++ - JAWS Staging Input Optimization: The JAWS client now identifies existing inputs at the JAWS Staging Area (Inputs files) and avoids copying them again. (`Issue 260 `_). - Consistent Error Response Formatting: All error responses from the JAWS Python Library are now returned in a structured JSON format. This change improves error handling and makes it easier for users to parse and debug issues. (`Issue 259 `_). JAWS Release 2025.2 - 02/04/2025 -------------------------------- |:sparkles:| New Features and Improvements ++++++++++++++++++++++++++++++++++++++++++ - Upgraded Cromwell and Womtool versions to `87 `_ (`Issue 202 `_). - JAWS now supports GPU-enabled tasks for `Perlmutter` NERSC site. (`Issue 54 `_). Below is an example of how to request GPU resources in a WDL workflow: .. code-block:: text version 1.0 workflow GPU_Test_Workflow { call GPU_Test_Task } task GPU_Test_Task { command { echo "Testing GPU runtime capabilities" python3 -c "import torch; print('CUDA Available:', torch.cuda.is_available()); print('Number of GPUs:', torch.cuda.device_count()); print('1 GPU Available:', torch.cuda.device_count() >= 1)" } output { File gpu_test_output = stdout() } runtime { docker: "pytorch/pytorch:latest" memory: "1GiB" cpu: 1 gpu: true runtime_minutes: 10 } } - Enabling GPU support: The `gpu: true` setting in the runtime block enables GPU resources for this task. - JAWS will request `1 GPU` node in the underlying scheduler. - Docker container compatibility: The container must support GPUs (e.g., pytorch/pytorch:latest includes CUDA). - Adjusted small worker configuration for `jgi` site: - 40 nodes with 16 CPUs / 58 GB RAM for small workers. - 6 nodes allocated for 32 CPUs / 512 GB RAM. - Please refer to the :ref:`Compute Resources ` page for details. |:bug:| Bug Fixes +++++++++++++++++ - Fixed dynamic mount path issue for KBase (`Issue Cromwell-199 `_). - Added dynamic mounting location for KBase at the JGI JAWS Site (`Issue cromwell-188 `_). - Fixed Refdata for NMDC Tahoma (`Issue 198 `_). - Fixed JAWS tasks folder permissions for group access (`Issue 1857 `_). |:construction:| In Progress ++++++++++++++++++++++++++++ - Started working the deployment of a new JAWS SITE at ANL - Crux Computing. JAWS Release 2025.1 - 01/16/2025 -------------------------------- |:sparkles:| New Features and Improvements ++++++++++++++++++++++++++++++++++++++++++ - **WDL Variable Validation:** JAWS now validates `memory` and `CPU` values in the `runtime` section when they are provided as WDL variables. If requested resources exceed system limits, an error is displayed in `stderr.submit`, helping users quickly identify and address resource allocation issues. (`Issue 167 `_) .. code-block:: text ERROR:check_cpu_mem_requests.cli:Requested number of CPUs cannot be supported: max CPUs supported, 36 < 64 Requesed cpu/memory is larger than the supported compute resource. - **Dynamic Mounting for KBase:** Added support for dynamic mounting location for KBase. (`Issue 179 `_) - **Input Type Warning:** Introduced a warning when users define an input as a `String`, but the value resembles a file path (e.g., `/some/filepath/sample.fastq`). (`Issue 249 `_) - **Extended Task Time Limit:** Since queue times have increased, we have also increased the time limit for tasks to remain IDLE in the queue from 72 hours to 144 hours. (`Issue 171 `_) |:bug:| Bug Fixes +++++++++++++++++ - **Duplicate Slack Messages Resolved:** Fixed an issue where JAWS would send duplicate Slack notifications for canceled runs. (`Issue 216 `_) - **Container Pull Fix:** Resolved an issue with container image pulls on the Dori. (`Issue 195 `_) - **Error Message Cleanup:** Removed the unnecessary `"chmod: operation not permitted"` error message in `stderr.submit`. (`Issue 185 `_) - **Task Issue Fixes:** Fixed an issue with the `jaws tasks` command to ensure all tasks are displayed correctly in the table. - **Improved Resilience for IO Errors:** Improve local file transfer processes by adding automatic retries for IO errors, reducing the risk of incomplete file transfers. 2024 Releases ============= JAWS Release 7/2024 - 11/12/2024 -------------------------------- |:sparkles:| New Features and Improvements ++++++++++++++++++++++++++++++++++++++++++ - **KBase Site**: A new JAWS site, `kbase`, has been added. This site is located at the NERSC cluster and is dedicated to supporting the KBase project. - **Team-Based Site Permissions**: Teams are now required to obtain permissions to use certain sites. Below is a summary of permissions granted to each team across sites: (`Issue 237 `_) - **JAWS Teams with Access to Dori, Perlmutter, JGI, and Tahoma**: - `gt-ga`, `dsi-ii`, `bionano`, `sc-mcr`, `dsi-syseng`, `workshop`, `plant_ha`, `mycocosm`, `plantcompgen`, `gaa`, `gt-seqtech`, `funcgen`, `sc-mds`, `gt-usa`, `micro-scale`, `rqc`, `dsi-aa`, `gt-syn`, `phytzm` - **KBase Team**: Can submit to `jgi` and `kbase` sites - **NMDC Team**: Can submit to `nmdc` and `nmdc_tahoma` sites - **ORNL Team**: Can submit to the `defiant` site - **WDL URLs and Subworkflows**: Users can now submit WDLs as URLs, with support for subworkflows that can either be relative paths or full URLs. (`Issue 224 `_) .. code-block:: text jaws submit https://code.jgi.doe.gov/advanced-analysis/jaws/test-repo/-/raw/main/main.wdl dori Please make sure the repository is **public** and provide the raw URL to the WDL argument. - `jaws purge` Command: Added the `jaws purge` command, allowing users to delete `cromwell-execution` folders for their own runs or those of team members, improving storage management. (`Issue 235 `_) - `jaws status` command is now `brief` by default, with the `--verbose` flag available for additional details. The help message for the `--verbose` flag has also been updated for clarity. (`Issue 239 `_ and `Issue 229 `_) - A clearer error message is now provided when a user submits `jaws submit ` but omits the `SITE` argument. (`Issue Client-214 `_) - `jaws-history` command now has a `--verbose` flag, which defaults to `False` for a streamlined output. (`Issue 227 `_) - If the WDL tasks runtime_minutes is not specified, Cromwell will default to 35 minutes. We highly recommend specifying the `runtime_minutes` in the WDL file. (`Issue Cromwell-175 `_). - `jaws tasks` command now displays input and output sum sizes for each task in bytes. (`Issue 1841 `_). |:bug:| Bug Fixes +++++++++++++++++ - `jaws tasks` command: - `jaws tasks` command now works on Tahoma. - Fixed an issue where jaws tasks did not display all tasks for a run. - The copy progress bar is now functional again for the jaws submit command. - Links in the documentation have been updated for deprecated functions to ensure accurate guidance (`Issue 245 `_). - Resolved an issue with the `jaws validate ` command (`Issue 232 `_). - The result field in `jaws status` now resets to `NULL` when a run is resubmitted, providing accurate tracking of resubmissions (`Issue 209 `_). - Fixed an error where JAWS would attempt to validate subworkflows located in nested directories, ensuring proper workflow validation (`Issue 213 `_). - Improved the error message for failed container pulls, providing clearer troubleshooting information. (`Issue 183 `_). - Fixed an issue where JAWS did not update the container image when a tag was reused with a new SHA, ensuring that updated containers are recognized (`Issue 176 `_). JAWS Release 6/2024 - 09/17/2024 -------------------------------- |:sparkles:| New Features +++++++++++++++++++++++++ - Supplementary files are now downloaded to user JAWS Teams directory upon run failure (`Issue 210 `_). |:bug:| Bug Fixes +++++++++++++++++ - Users can now delete their own folders, specifically the folder, after completing a local transfer on Dori (input and compute site are the same). (`Issue 1840 `_). - `jaws submit` command will now throw an error if the user is not part of a JAWS team, preventing unauthorized access and enhancing security. (`Issue 222 `_). JAWS Release 5/2024 - 09/10/2024 -------------------------------- |:sparkles:| New Features and Improvements ++++++++++++++++++++++++++++++++++++++++++ - JAWS Client Python Library and Documentation: - Added the JAWS Python library to the JAWS Client. This library provides a Python interface to interact with JAWS, allowing users to submit runs, check run status, and download outputs programmatically. - The JAWS Client Python library documentation is available :doc:`at JAWS Python Library section `. (`Issue 205 `_). - JAWS Staging Area Overview - Input File Cache Validation: When a run is submitted, the JAWS Client copies the input files to a designated "JAWS staging area." For subsequent submissions using the same input files, JAWS will reference the cached copies instead of re-copying them. If an input file content has changed but its filename remains the same, JAWS will detect this and provide an error message. Here is an example of the error message: .. code-block:: text jaws submit --no-cache align_final.wdl inputs.json dori Error initializing Run: Unable to copy input files: /clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/data/sample.fasta is different from its cached version. Submitting with this input file can affect previous runs. Use --overwrite-inputs to force update the cached input files. This ensures that input consistency is maintained and prevents files linked to previous runs from being unintentionally modified. (`Issue 220 `_ and `Issue 206 `_). - Overwrite Inputs: Users can explicitly provide the `--overwrite-inputs` flag if they wish to update cached input files with a new version. (`Issue 220 `_). .. code-block:: text jaws submit --no-cache align_final.wdl inputs.json dori --overwrite-inputs For more information about the JAWS Staging Area, please refer to :doc:`JAWS documentation `. - Display Return Code (RC) to the `jaws tasks` Command: The `jaws tasks` command now includes the `RC` file content for each task. This improvement provides users with additional context when troubleshooting task failures. (`Issue Jaws-Cromwell-168 `_ and `Issue 1832 `_). - Updated 48-Hour Maximum Walltime on Perlmutter: The maximum walltime for jobs on the Perlmutter compute system has been updated to 48 hours. (`Issue 150 `_). Please check the walltime limit for all available :ref:`JAWS sites `. |:sparkles:| Jaws Resiliency ++++++++++++++++++++++++++++ - Local Output File Transfer Optimization: JAWS has transitioned from using `parallel-sync` to Python’s built-in `shutil.copy` for local output file transfers. This change addresses several performance and compatibility issues: - `parallel-sync` was rewritten in version 2.0; this new version did not include a local transfer function. - The last compatible version of `parallel-sync` was incompatible with Python 3.11 due to outdated dependencies like the brunch package. - By switching to `shutil.copy`, JAWS now leverages optimized kernel-level fast copy calls, improving performance and reducing dependency on external libraries. - In addition to `shutil.copy`, JAWS now uses `filecmp.cmp` to verify the integrity of local file transfers by comparing file attributes such as size, type, and modification time. This method is more CPU efficient than using `md5sum` and provides the same level of accuracy. (`Issue 1819 `_ and `Issue 1818 `_ and `Issue 214 `_). |:bug:| Bug Fixes +++++++++++++++++ - Improved error messaging for :bash:`JAWS cancel` command to provide clearer guidance when the `RUN_ID` is missing. (`Issue 199 `_). - Display RC File Content with Timestamp: - Added a print statement to `stderr.submit` that outputs the contents of the `RC` file along with a `timestamp`. - This helps in debugging issues such as the Cromwell error (`rc=79`), where the `RC` file might initially show `rc=0` but later be overwritten by Cromwell with a `rc=79`. (`Issue 160 `_). - Users can now delete their own folders, specifically the run-id folder, after completing a Globus download. (`Issue 238 `_). |:scales:| Policy Updates +++++++++++++++++++++++++ - Fair Share Policy Update: - Updated the fair share policy for Dori, allowing a maximum of 30 runs per user on the Dori cluster. (`Issue 1826 `_). - Please notice that we update our time limits for all the sites. Please check the walltime limit for all available :ref:`JAWS sites `. (`Issue 1836 `_). .. admonition:: Time Overhead When Cromwell submits a task, HTCondor manages job scheduling by checking the queue for available resources. The JAWS Pool Manager monitors HTCondor and, when needed, requests new Slurm nodes. Once a compute node is available, HTCondor submits the task. Due to a slight delay (a few seconds) in resource allocation, we build in a time buffer to ensure jobs get the full requested time. For example, instead of requesting the maximum 48 hours on Perlmutter, we request 47 hours and 45 minutes to account for the delay. JAWS Release 4/2024 - 05/21/2024 -------------------------------- |:sparkles:| Jaws Resiliency ++++++++++++++++++++++++++++ - JAWS Central and its databases have been migrated to IGB. - We have deployed a new site, `jaws defiant`, at the Advanced Computing Ecosystem (ACE) testbed located at the Oak Ridge Leadership Computing Facility (OLCF). JAWS Release 3/2024 - 05/08/2024 -------------------------------- |:bug:| Bug Fixes +++++++++++++++++ - Fixed bug here multiple users sharing the same input file caused conflicts. (`Issue 208 `_) JAWS Release 2/2024 - 05/01/2024 -------------------------------- |:sparkles:| New Features +++++++++++++++++++++++++ - **NMDC Tahoma Site**: We've added a new JAWS site: `nmdc_tahoma`. This new site is located at the EMSL cluster, and it is dedicated to supporting the NMDC project. - **Enhanced Error Messaging**: Improvements have been made to the error messages for backend task failures, providing clearer guidance for troubleshooting. (`Issue Pool-Manager-118 `_) - :bash:`jaws history` command has new filtering options, allowing searches by JSON file name (e.g., my_inputs.json), WDL file name, user, or tag, using flags such as `--json-file`, `--wdl`, `--user`, and `--tag`. (`Issue 88 `_) |:new:| New behavior for JAWS +++++++++++++++++++++++++++++ - Dori exvivo nodes will now be dedicated solely to large memory tasks, optimizing resource allocation and enhancing performance. (`Issue 122 `_) - If a Cromwell task returns a code of `79` (error code related to file system issues), JAWS will automatically retry the task once more. (`Issue 139 `_) - The :bash:`JAWS_SITE` environment variable is now exported within container environments for running tasks. This means you can use :bash:`$JAWS_SITE` from your WDL :bash:`commands{}` section for conditional statements (if :bash:`$JAWS_SITE == "dori"` then do something). (`Issue 142 `_) - Slack notification now contains the `workflow_root` field. (`Issue 191 `_) |:bug:| Bug Fixes +++++++++++++++++ - Fixed bug where JAWS was not able to create a user subdirectory under the Teams dir. (`Issue 1790 `_) - Fixed bug for parallel copy files. Parallel copy happens in lieu of globus when submitting jobs to the same site as you submitted from. (`Issue 1782 `_) - Added retries for Jaws-site retrieving metadata from cromwell. (`Issue 1787 `_) JAWS First Winter Release - 02/07/2024 -------------------------------------- |:sparkles:| New Features +++++++++++++++++++++++++ - JAWS will issue a warning if a `File` type is hardcoded in the WDL file. (`Issue 128 `_) - We have implemented a validation process to ensure the `WDL` file aligns with the `inputs.json` file, using `womtool`. This validation occurs automatically upon submission and can also be triggered manually by executing the :bash:`jaws validate` command. (`Issue 175 `_) .. code-block:: text jaws validate |:new:| New behavior for JAWS +++++++++++++++++++++++++++++ - We update the Fair-Share policies: (`Issue 136 `_) - Task-Level Control: - Limit: There's now a maximum limit of 200 concurrent tasks per user. - Queue: Tasks submitted above the 200-task limit will enter a queue. - Only successful runs will be copied to the team's directory. (`Issue 154 `_) - Our backend system (HTCondor) will attempt to execute a task **twice** under certain conditions. If your task exceeds its allocated time, or if JAWS-site needs to be reassigned to a new node (Perlmutter site), or due to some instability in the backend, we will automatically initiate a second attempt to run the task. Important to note: - When the backend or Cromwell retries a task, the `cromwell-execution` folder will **not** be reset. In specific cases, this requires adding a verification process in your WDL command stanza. For example, if the command includes creating directories or specific files. Depending on the outcome, you’ll need to determine whether to force the directory or file to be recreated or to skip this step. (`Issue Pool-Manager-110 `_) |:bug:| Bug Fixes +++++++++++++++++ - Fixed issue related to retrieving metadata from runs that contain over 1 million rows. (`Issue Cromwell-137 `_) - A user-friendly error message is displayed when an invalid `run_id` is entered in the :bash:`jaws status ` or :bash:`jaws log ` commands. (`Issue 177 `_) - Fixed file permission issue for files transferred to the team's directory via Globus from a compute site that is different from the input site. (`Issue 168 `_ and `1739 `_) - Fixed issue related to transferring very large output datasets. (`Issue 1746 `_) - Return an error message to the user when Cromwell submission fails. (`Issue 1568 `_) - Updated the `refdata` path for the NMDC site (:bash:`/global/cfs/cdirs/m3408/refdata`). (`Issue 1756 `_) - :bash:`workflow_root: null` bug is fixed. (`Issue 1747 `_) - Added a `tag` to the `summary.json` supplementary file. (`Issue 1758 `_) - Increased the length of the `JSON` basename field. (`Issue 1776 `_) - Fixed `job_id` column in the :bash:`jaws task` output. (`Issue 1749 `_) 2023 Releases ============= JAWS Second Fall Release - 11/09/2023 ------------------------------------- |:sparkles:| New Features +++++++++++++++++++++++++ - JAWS has incorporated support for specifying **execution time** in the `runtime` section. In addition to `memory` and `cpu`, you can now include the time required to run each task. The mandatory key for specifying this information is `runtime_minutes`. - `runtime_minutes`: - Accepted types: `Int` - `Int`: minutes. Example: .. code-block:: text runtime { docker: "ubuntu@sha256:c9cf959fd83770dfdefd8fb42cfef0761432af36a764c077aed54bbc5bb25368" runtime_minutes: 60 memory: "5G" cpu: 4 } The benefit of specifying `runtime_minutes` is that it provides a guarantee that the task will be put on a node with sufficient time. |:warning:| If the WDL `runtime` section uses `time` as a key or doesn't specify any time value, the workflow will still be accepted but without the assurance it will be allocated to a suitable node. - |:sparkles:| JAWS Client Container is Available |:sparkles:| - How to use: - **Dori**: .. code-block:: text JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/clusterfs/jgi/groups/dsi/homes/svc-jaws/dori-prod/jaws-prod.conf apptainer run docker://doejgi/jaws-client:latest jaws --help - Append to the end of your :bash:`~/.bashrc` on DORI: .. code-block:: text jaws() { JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/clusterfs/jgi/groups/dsi/homes/svc-jaws/dori-prod/jaws-prod.conf apptainer run docker://doejgi/jaws-client:latest jaws "$@" } - **Perlmutter**: .. code-block:: text JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/global/cfs/cdirs/jaws/perlmutter-prod/jaws-prod.conf shifter --image=doejgi/jaws-client:latest jaws --help - Append to the end of your :bash:`~/.bashrc` on NERSC: .. code-block:: text jaws() { JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/global/cfs/cdirs/jaws/perlmutter-prod/jaws-prod.conf shifter --image=doejgi/jaws-client:latest jaws "$@" } |:new:| New behavior for JAWS +++++++++++++++++++++++++++++ - In our ongoing efforts to enhance performance and maintain a robust, up-to-date environment, **we will be discontinuing support for WDL Draft-2 version** 12/1/2023. - We will display a |:warning:| warning message to notify you if you are still using the old version when submitting a run. - We strongly urge all users to update their WDL workflows to the `Version 1.0 specification `_. If you have any questions, please schedule a pair programming session with our Team. |:bug:| Bug Fixes +++++++++++++++++ - We added a new flag to :bash:`jaws validate` to show :bash:`shellCheck` linter validation. In addition, the output also now prints newlines for better readability. - Addressed the issue "noise" being printed to :bash:`stderr` for Dori runs. (`Issue 130 `_) - Increased allowed length of WDL and JSON filenames. (`Issue 1744 `_) - :bash:`jaws health` command is now operational again. - :bash:`workflow_root: null` bug is fixed. (`Issue Jaws-Support-148 `_) JAWS First Fall Release - 10/09/2023 ------------------------------------ |:sparkles:| New Features +++++++++++++++++++++++++ - Dori Exvivo nodes are now available for JAWS. JAWS is using the long queue (14 days) and can access up to 1.5 TB of memory. (`Issue 83 `_) - Globus endpoint is available for Perlmutter! |:tada:| (`Issue 1683 `_) - This means that you are now able to submit from `Perlmutter` to `Dori/Tahoma/JGI` sites and vice-versa. - Teams output directory will include JAWS user ID and will have the following structure (`Issue 137 `_): .. code-block:: text //// - New command: :bash:`jaws tasks `: .. code-block:: text jaws tasks 18279 #TASK_DIR STATUS QUEUE_START RUN_START RUN_END QUEUE_MIN RUN_MIN CACHED TASK_NAME REQ_CPU REQ_GB REQ_MIN CPU_HRS call-alignment succeeded 2023-10-02 16:55:07 2023-10-02 16:56:16 2023-10-02 16:56:22 1 0 False bbtools.alignment 1 1 0.0 call-samtools succeeded 2023-10-02 16:56:36 2023-10-02 16:56:55 2023-10-02 16:56:57 0 0 False bbtools.samtools 1 1 0.0 - :bash:`jaws tasks` integrate two previously distinct commands, :bash:`jaws task-log` and :bash:`jaws task-summary`. Be sure to explore the newly unified and enhanced features: - "cpu_hours" metric is now included in the :bash:`jaws tasks` command for each task and :bash:`jaws status` as an aggregation for the entire workflow upon run completion. - Cached tasks will now be recorded in the :bash:`jaws tasks` post run completion. - :bash:`jaws tasks` status is now updated appropriately when a run is cancelled. - :bash:`jaws tasks` now uses timestamps derived from the task log instead of using Cromwell metadata. - New command: :bash:`jaws download `: - If a run fails, JAWS will skip the transfer to team's directory. However, if you need the cromwell-execution for debugging reasons, you can use our new command to 'force' the download of the run: .. code-block:: text jaws download 18386 { "download_id": 8272, "id": 18386, "status": "download queued" } - Please notice that the run output will be transferred to the team's directory. - New command :bash:`jaws get-user` (`Issue 107 `_) - This command gets current user's settings. .. code-block:: text jaws get-user { "email": "dcassol@lbl.gov", "name": "Daniela Cassol", "slack_id": "", "teams": [ "dsi-aa", "nmdc" ], "uid": "dcassol" } - Bash commands employing :bash:`curl` is now working on Tahoma. (`Issue 121 `_) - Output directory will include a copy of the original `WDL`, `input.json`, and `subworkflows-ZIP files`, for reproducibility. (`Issue 1710 `_) - :bash:`jaws status` is now verbose by default. If you prefer the short version, please use :bash:`jaws status --brief`. To ensure backward compatibility, when you use the command :bash:`jaws status --verbose`, it will issue a warning and additionally display the output of :bash:`jaws status`. - JAWS is now using the latest version of `Cromwell-85 `_. |:bug:| Bug Fixes +++++++++++++++++ - JAWS will **not** transfer workflow outputs to teams output location runs that were canceled by the user. (`Issue 156 `_) - Fixed teams output directory permissions when the submission and compute site are the same. (`Issue 1712 `_) - Fixed issue when workflow name and site are the same. (`Issue 23 `_) - :bash:`jaws resubmit` command is restricted to the members of the team who own the run. (`Issue 148 `_) - We have improved the error messaging for :bash:`jaws resubmit`, especially when the original submission failed. (`Issue 116 `_) - :bash:`jaws status` is now displaying local time zone. (`Issue 151 `_) - :bash:`jaws status` will report when a run was canceled by the user in the `result` field. (`Issue 1714 `_) - We are ignoring extra attributes to the :bash:`runtime{}` stanza, for example :bash:`"runtime_minutes": "20"`. (`Issue 118 `_) - When a run is canceled, it will be recorded to :bash:`jaws tasks `. (`Issue 120 `_) - Fixed issues reporting the wrong task status in :bash:`jaws tasks ` command. - Fixed error for transfers when file was named pipe instead of regular file. (`Issue 1725 `_) - When a run is resubmitted (:bash:`jaws resubmit `), JAWS will ensure that all the input files required are still available. (e.g., haven't been purged). (`Issue Jaws-Support-110 `_) - When a run is resubmitted (:bash:`jaws resubmit `), JAWS will update access timestamp (`atime`) for input files, in order to avoid purging files prematurely. (`Issue 1689 `_) - Fixed bug when Cromwell submission fails during input processing and was not recognized by JAWS. (`Issue 1711 `_) - :bash:`outputs.json` supplementary file contain relpaths instead of abspaths. (`Issue 1652 `_) |:new:| New behavior for JAWS +++++++++++++++++++++++++++++++++ - In our ongoing efforts to enhance performance and maintain a robust, up-to-date environment, **we will be discontinuing support for WDL Draft-2 version** 11/1/2023. - We will display a |:warning:| warning message to notify you if you are still using the old version when submitting a run. - We strongly urge all users to update their WDL workflows to the `Version 1.0 specification `_. If you have any questions, please schedule a pair programming session with our Team. - JAWS Cromwell configuration was updated, and now Container tags can be used for call caching (`Issue Jaws-Cromwell-122 `_, `Issue Jaws-Client-156 `_). - We recommended referencing containers by their `SHA256` instead of `tag` (e.g., :bash:`doejgi/bbtools@sha256:64088..` instead of :bash:`doejgi/bbtools:latest`). While using mutable or "floating" tags in tasks can be convenient in certain scenarios, it adversely impacts the reproducibility of a workflow. For instance, executing the same workflow with :bash:`doejgi/bbtools:latest` now, and then rerunning it in a month or a year, could result in the use of different container images. - If the command stanza uses :bash:`$TMPDIR`, it will have access to :bash:`/tmp`. Previous we set to :bash:`execution/` directory (e.g., `NFS`). (`Issue 110 `_) |:x:| Deprecated Commands +++++++++++++++++++++++++ - :bash:`jaws task-log` and :bash:`jaws task-summary` have been deprecated (`Issue 161 `_, `Issue 1719 `_). - These two commands have been merged and into a new command, :bash:`jaws tasks`. - For backward compatibility, :bash:`jaws task-log` will be reporting the output of :bash:`jaws tasks`. |:warning:| Known issues ++++++++++++++++++++++++ - If the submission and compute sites are different (for example, from Dori to Tahoma), there could still be permission issues in the team's output directory. In this case, the transfer to the team's directory happens via Globus, we need are looking into solutions for this problem. JAWS Summer Release - 09/05/2023 -------------------------------- We are releasing a new version of JAWS that includes **breaking** changes. JAWS Teams ++++++++++ We're introducing a new feature in JAWS called "JAWS Teams." This feature allows for easier management of users and offers a centralized location for sharing and delivering output files for each team. - List all the teams available: .. code-block:: text jaws teams list [ "gt-ga", "nmdc", "dsi-ii", "sc-mcr", "gt-seqtech", "dsi-aa", "gt-syn", "phytzm" ] - List the teams to which you belong: .. code-block:: text jaws teams my-teams [ "dsi-aa" ] - List the users associated with a team: .. code-block:: text jaws teams members dsi-aa [ "dcassol", ... ] - Get a team's site config - Outputs path: .. code-block:: text jaws teams get-site dsi-aa dori "/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/dsi-aa" - Team's owner has power and responsibilities, such as setting the path for each site, adding and deleting users from the team: .. code-block:: text jaws teams set-site jaws teams add-user jaws teams del-user How to use Jaws Teams? ++++++++++++++++++++++ When submitting a new run, you can use :bash:`--team` flag. The outputs for this run will be transferred to the team's path. .. code-block:: text jaws submit align_final.wdl inputs.json dori --team=dsi-aa As an easy alternative, you can set the :bash:`default_team` at your :bash:`jaws.conf` file: .. code-block:: text vi ~/jaws.conf [USER] token = default_team = dsi-aa Important: Do not use quotes for the team's name. The Jaws Team's new feature will transfer the final workflow outputs from cromwell-execution to the path defined for your team. How do I find the output data? ++++++++++++++++++++++++++++++ :bash:`jaws status -–verbose ` will provide the path to final path. Please check :bash:`output_dir`. Then, you can find the expected file tree structure: .. code-block:: text /// JAWS will copy the following in case of a successful run: - Workflow outputs; - Supplementary files: - errors.json; - metadata.json; - output_manifest.json; - outputs.json; - task_summary.json. JAWS will copy the following in case of a **FAILED** run: - Workflow outputs; - Supplementary files; - Failed tasks' cromwell-execution folder only. As we transition to copying the output files, we are deprecating out the :bash:`jaws get` command. To ensure backward compatibility, this command will remain functional for a few more months, serving solely to copy the final workflow outputs. However, please note that we plan to discontinue this command entirely in upcoming cycles. We have implemented parallel copying capabilities when both the submission and compute site are the same, for example, from Dori to Dori. This effectively resolves the delay issues associated with the ‘*download queue.*’ Jaws Fair-share policies ++++++++++++++++++++++++ We have implemented a JAWS execution throttling, allowing us to have fair-share policies in place. Specifically, there are two layers of control: **Run-Level Control** - *Limit*: A maximum of 10 runs can be executed concurrently per site per user. - *Queue*: Any additional run submissions beyond this limit will be automatically placed in the JAWS Queue. - *Processing*: These queued runs will be initiated as soon as one of the currently active runs is completed. **Task-Level Control** - *Limit*: A cap of 600 concurrent *tasks* is set per user. - *Queue*: Any tasks submitted beyond this 600-task limit will be placed in a queue. Slack Notifications ++++++++++++++++++++ We replaced Email with Slack notifications when the run is completed. **How to set up Slack Notification?** Please set up your :bash:`slack_webhook` and update your JAWS account. Instructions on how to get your Slack webhook are available `here `_. .. code-block:: text jaws update-user –email=dcassol@lbl.gov --slack_id --slack_webhook Call-caching Strategy +++++++++++++++++++++ The call-caching strategy that we used was “xxh64", and that required a lot of I/O operation to calculate the hash of the entire file content. Now, we replace that with “fingerprint". fingerprint will take the last modified time, size, and hash from the first 10 mb with xxh64 to create a file fingerprint. Please be aware of that, and please let us know if you think that can cause any file collision when the task will use call-caching. If you want to read more about the call-caching and all the strategies please check here. Additional minor features +++++++++++++++++++++++++ - Added :bash:`--forcequeue` flag option to :bash:`jaws submit` command. Users can force run submission when the site has been disabled; - Resubmitting a run will change the “result” filed to “resubmitted” for :bash:`jaws status ` command; - We now parse the :bash:`runtime{}` parameter values correctly when there is a space (i.e., memory: “5 G”) ticket: #111; - :bash:`/refdata` is mounted correctly and accessible to the WDLs. **Special note about the DORI TEAMS Folders:** For this release, Teams’ folders on Dori must be located at :bash:`/clusterfs/jgi/scratch/dsi/aa/jaws/teams/`. However, once the Globus endpoint changes have been completed, teams’ folders may be edited under the team’s *scratch*. .. button-link:: https://docs.google.com/presentation/d/1-eRWFQE0zXEC8gsadmaXytCVe8LwpGBoQVyqH7mSFJY/edit?usp=sharing :color: primary :shadow: JAWS Summer Release Slides .. button-link:: https://calendar.google.com/calendar/embed?height=600&wkst=1&bgcolor=%23ffffff&ctz=America%2FLos_Angeles&src=Y18xY2NyMG5vM3JmcWp1djNkc2U3a2o2Z2R2b0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t&color=%23039BE5 :color: primary :shadow: Link to JAWS Calendar JAWS Sprint Release - 04/12/2023 -------------------------------- *We released a new version of JAWS.* Here are the changes that are now on PROD: ++++++++++++++++++++++++++++++++++++++++++ - Dori Jaws site is available for testing; - Create a config file in your HOME directory: .. code-block:: text touch ~/jaws.conf chmod 600 ~/jaws.conf - Copy your token from CORI: .. code-block:: text [USER] token = - *Module load jaws* .. code-block:: text module use /clusterfs/jgi/groups/dsi/homes/svc-jaws/modulefiles module load jaws/dori-prod jaws submit dor NOTE: - Dori and Perlmutter have temporary Globus limitations (IT has not configured JAWS application Globus endpoints yet). - Temporary workaround: - Because data transfer isn’t available, you must log in to the cluster (Dori or Perlmutter) and submit the run from there. - You can use Globus endpoint to transfer your data from Cori to Dori, for example. Deprecated commands ++++++++++++++++++++ .. code-block:: text jaws outfiles jaws outputs jaws metadata jaws errors This is part of the effort to refactor :bash:`jaws metadata` command (uses cromwell metadata). As users are submitting large workflows (> 10k tasks), cromwell metadata became too expensive to query and would sometimes timeout. Instead, we now wait for the run to finish and then write some associated reports (e.g. errors, outfiles, and outputs) to disc. These json files are written to the run’s execution directory and returned to users via :bash:`jaws get`. - To find the report files: .. code-block:: text jaws get ls .wdl .json errors.json metadata.json outfiles.json outputs.json task_summary.json - :bash:`jaws task-log` changed: - :bash:`task-log` is now much faster and robust than before and will support runs with greater than 10k tasks. - Even more than before, it is a real-time reflection of a run’s current status since it gets its information directly from the backend instead of using an intermediate metadata cromwell, which was a bottleneck. - However, because it no longer uses cromwell’s metadata, there are no longer records for cached tasks since they did not actually execute. Therefore, the “cached” column was deleted. - *Limitation for Tahoma site:* :bash:`jaws task-log` isn’t working properly for Tahoma due to the firewall between the compute and workflow node. We have a ticket open for this issue. However, this will not affect your run! - :bash:`jaws task-summary` It only will be available after the run is finished. - :bash:`jaws status –verbose` Two more fields: workflow_name and workflow_root - :bash:`jaws validate` miniwdl is used instead of womtool for WDL validation. - :bash:`jaws resubmit` New command available! Resubmit a run (at the same compute site). Fixed bugs reported by users ++++++++++++++++++++++++++++ - *runtime* parameters can be on separate lines; - Fixed :bash:`jaws get --flatten`; - symlinks can be used for the path to the WDL (:bash:`jaws submit /my.wdl my.json dori`) - When restarting cromwell service (i.e. upon release) will not interrupt active Runs; - Added job purging policies to the HTCondor backend so that it can automatically cancel tasks that stay in HOLD status for too long time. - Added :bash:`--time-min` SLURM option for requesting SLURM compute nodes with a shorter wall-clock-time on Perlmutter when maintenance is scheduled.