====
FAQs
====

.. role:: bash(code)
  :language: bash

This document provides solutions to frequently asked questions about JAWS, WDL development, and workflow execution. 
You'll find code snippets, explanations, and best practices organized by topic.

Table of Contents
=================

- `WDL Development <#id1>`_
- `Cromwell <#id2>`_
- `Compute Systems <#id3>`_
- `JAWS <#id4>`_
- `Known Limitations <#id5>`_

Resources
---------

- Official WDL specification: `OpenWDL Version 1.0 <https://github.com/openwdl/wdl/blob/legacy/versions/1.0/SPEC.md>`_

WDL Development
===============

.. dropdown:: How to use bash commands with curly braces in WDL :ref:`🔗 <How to use bash commands with curly braces in WDL>`
    :color: info
    :animate: fade-in
    :name: How to use bash commands with curly braces in WDL
    
    If you ever need to use curly braces in bash to strip a suffix 'txt' or set a default, consider using the `WDL Version 1.0 Specification <https://github.com/openwdl/wdl/blob/legacy/versions/1.0/SPEC.md#versioning>`_.
    In WDL document, the first line should indicate version 1.0. You'll specify the command section like :bash:`command <<< >>>` instead of using curly braces. Additionally, you'll need to adjust some other formats. Refer to the Version 1.0 specification link for more details.
   

    .. code-block:: text   

        command <<<
            # setting a default value in bash
            VAR=${VAR:=25}

            # strip a suffix
            myvar=${somefile%.txt}
        >>>

.. dropdown:: How can I output a file that has been named dynamically as a bash variable :ref:`🔗 <How can I output a file that has been named dynamically as a bash variable>`
    :color: info
    :animate: fade-in
    :name: How can I output a file that has been named dynamically as a bash variable

    Bash variables created in the :bash:`command { }` block cannot be seen outside the block, for example, in the :bash:`output { }` section. Therefore, you can write the name(s) of any output files to another file which will be read inside the :bash:`output { }` block.
    
    This is the official WDL way, using `glob`:
    
    .. code-block:: text   
    
          output {
              Array[File] output_bams = glob("*.bam")
          }

    This is another method:
    
    .. code-block:: text       
    
         command{
             echo $lib.bam > list_of_files
          }
          output {
             Array[File] = read_lines("list_of_files")
          }
          
    To see more about `read_lines()` and other WDL functions, see `openwdl/wdl <https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md>`_.
    

.. dropdown:: Using conditionals :ref:`🔗 <Using conditionals>`
    :color: info
    :animate: fade-in
    :name: Using conditionals

    .. code-block:: text      
    

        workflow conditional_example {
          File infile

          call wc as wc_before { input: infile = infile }

          Int num_lines = wc_before.num_lines

          if (num_lines > 10) {
            call truncate { input: infile = infile }
          }

          # This function will return false if the defined() argument is an 
          # unset optional value. It will return true in all other cases.
          Boolean has_head_file = defined(truncate.outfile)

          if (has_head_file) {
            call wc as wc_after { input: infile = truncate.outfile }
          }

          # notice the '?' after File. These are required since these files may not exist.
          output {
            File wc_before_file = wc_before.outfile
            File? head_file = truncate.outfile
            File? wc_after_file = wc_after.outfile
          }
        }

        task wc {
          File infile
          command { wc -l < ${infile} | tee wc.txt }
          output {
            Int num_lines = read_int(stdout())
            File outfile = "wc.txt"
          }
        }
    
.. dropdown:: How to scatter over arrays and maps :ref:`🔗 <How to scatter over arrays and maps>`
    :color: info
    :animate: fade-in
    :name: How to scatter over arrays and maps

    Although you can scatter over arrays and maps, there is different syntax for each.
    You can only scatter over an array with this syntax

    .. code-block:: text        
        
        Array[String] some_array
        scatter (e in some_array) {
          String value = some_array[e]
          call some_task {input: value = value}
        }

    But you can iterate over a map by using the 'pair' keyword and then '.left' and '.right' as such

    .. code-block:: text    
    
        Map[String,String] some_map
        scatter (pair in some_map) {
          String key= pair.left
          String value = pair.right # or String val = some_map[key]
          call some_task {input: value = value}
        }

    You can see working examples for `scattering an array and scattering a map <https://code.jgi.doe.gov/official-jgi-workflows/jaws-tutorial-examples/-/tree/master/scatter_gather_example>`_. 


.. dropdown:: Custom data structures :ref:`🔗 <Custom data structures>`
    :color: info
    :animate: fade-in
    :name: Custom data structures
    
    Besides Map, Array, Pair you can create a custom data structure using "struct". This will be similar to a hash but can contain any combination of data types.
    
    - Documentation for `Custom Type "Struct" <https://github.com/openwdl/wdl/blob/main/versions/1.1/SPEC.md#struct-definition>`_.
    
    - Example `main.wdl <https://code.jgi.doe.gov/official-jgi-workflows/jaws-tutorial-examples/-/blob/main/custom_datastructure/main.wdl>`_ && `inputs.json <https://code.jgi.doe.gov/official-jgi-workflows/jaws-tutorial-examples/-/blob/main/custom_datastructure/inputs.json>`_.
      

.. dropdown:: Get Keys from a Map :ref:`🔗 <Get Keys from a Map>`
    :color: info
    :animate: fade-in
    :name: Get Keys from a Map

    As of version 1.0 of the wdl spec, there is no direct way to get an array of Map keys. This will become available in version 1.1.  As a work-around for now, you can use the :bash:`Pair` data type instead of :bash:`Map` as follows.
    
    .. code-block:: text
    
      version 1.0 
      workflow test {
        input {
          Array[Pair[Float,String]] my_map = [(0.1, "mouse"), (3, "cat"), (15, "dog")]
        }
      
        scatter (pairs in my_map) {
          String keys = pairs.left
        }
      
        output {Array[String] allouts = keys}
      }

    Note that the default format for :bash:`my_map` in the WDL is different that in an :bash:`input.json`
        
    .. code-block:: text
    
      {
        "test.my_map": [{"Left": 0.1, "Right": "mouse"}, {"Left": 3, "Right": "cat"}, {"Left": 15, "Right": "dog"}]
      }

 
.. dropdown:: An example of a scatter/gather model when a scattered task is optional :ref:`🔗 <An example of a scatter/gather model when a scattered task is optional>`
    :color: info
    :animate: fade-in
    :name: An example of a scatter/gather model when a scattered task is optional

    You can use this example to see where you need to declare optional varaibles (i.e. Array[Array[String?]]) and 
    how you can use two wdl functions, :bash:`flatten` and :bash:`select_all` to convert an optional 
    variable (Array[Array[String?]) to an Array[String].

    See this line: 

    .. code-block:: text

        Array[String] flat_array = flatten(select_all(num_array))
        # note that this line is not within any stanza, but between input{} and command<<<>>>

    and note that

    | :bash:`flatten` is used to convert an array-of-an-array to an array.
    | :bash:`select_all`, used in this example, converts Array[String?] to Array[String].
    | If you don't use select_all here, you get the error:
    |   Expected 'Array[Array[_]]' but got 'Array[Array[String]?]'


    .. code-block:: text

       version 1.0

       workflow flatten_it {
           input {
               Boolean try_it = false
               Array[Int] numbers = [1,2,3]
           }
       
           scatter (num in numbers) {
             if (try_it == true) {
               call do_if_true {
                 input: num = num
               }
             }
       
             # call something_else {}
           }
       
           call gather  {
             input: num_array = do_if_true.out
           }
       
          output {
            Array[File] out_array = gather.final_array
          }
       }
       
       task do_if_true {
            input { 
               Int num 
            }
       
            command <<<
               echo "~{num}.one"  
               echo "~{num}.two"  
               echo "~{num}.three"    
            >>>
       
            output {
                Array[String] out = read_lines(stdout())
           }
       }
       
       task gather {
           input {
               Array[Array[String]?] num_array
           }
       
           Array[String] flat_array = flatten(select_all(num_array))
       
           command <<<
               echo ~{sep=', ' flat_array}
           >>>
       
           output {
              Array[String] final_array = read_lines(stdout()) 
           }
       }

.. dropdown:: Why does `jaws validate` fail when using `"string" + <WDL variable>` in the runtime block? :ref:`🔗 <Why does jaws validate fail when using "string" WDL variable in the runtime block>`
    :color: info
    :animate: fade-in
    :name:  Why does jaws validate fail when using "string" WDL variable in the runtime block

    🛑 Issue

    You may see the following error when trying to concatenate a string and a WDL variable in a runtime block:

    .. code-block:: text

       Cannot add/concatenate String and String?
       docker: "ubuntu:" + tag

    This issue is not detected by `womtool`, so workflows may still submit successfully using `jaws submit` (which internally uses `womtool` validation).

    However, `jaws validate` uses `miniwdl` under the hood, which enforces stricter WDL 1.0 compliance and fails validation when expressions like :bash:`"ubuntu:" + tag` are used in runtime attributes.

    💡 Explanation

    The problem arises because the variable `tag` is declared as an optional string (`String?`), which makes the expression :bash:`"ubuntu:" + tag` ambiguous if `tag` is missing.

    Removing the optional (`?`) can fix the immediate error for validation.


    WDL Example that causes the issue:

    .. code-block:: text

      version 1.0

      workflow HelloWorld {
        call helloWDL
      }

      task helloWDL {
        input {
          Int cpu = 2
          Int mem = 8
          String? tag = "latest"
        }
        command <<<
          echo "Hello WDLs" >> test.txt
        >>>

        output {
          File output_wdl = "test.txt"
        }
        runtime{
            docker: "ubuntu:" + tag
            memory: mem + " GiB"
            runtime_minutes: 10
            cpu: cpu /1
          }
      }


    🔍 Summary

    - `jaws validate` uses `miniwdl` and will reject runtime expressions like :bash:`"ubuntu:" + tag` if `tag` is optional. 
    - `jaws submit` uses `womtool`, which is more permissive and may allow the submission.


Cromwell
========

.. dropdown:: Handling HTTP URLs Inputs in Cromwell Workflows :ref:`🔗 <Handling HTTP Inputs in Cromwell Workflows>`
    :color: info
    :animate: fade-in
    :name: Handling HTTP Inputs in Cromwell Workflows
    
    **Problem**: When using HTTP URLs as inputs in Cromwell workflows, Cromwell automatically downloads and stores the files locally. 
    However, during this process, files are renamed, and file extensions may be lost. This can cause issues for programs that rely on specific file extensions or naming conventions to function correctly.

    This is a known issue in Cromwell. For more details, refer to the `Cromwell Filesystems - HTTP documentation <https://cromwell.readthedocs.io/en/stable/filesystems/HTTP/>`_. 

    **Solution**
    To make your workflow more portable and compatible with both HTTP and local files, you can copy the downloaded file to match its expected file name and extension. 
    A useful approach is to use a `Pair` data type to pass the URL and the desired basename for the file. Then, in the task's `command stanza`, copy the file to the correct name.
    
    Example WDL Workflow:
    
    .. code-block:: text

        version 1.0

        workflow HelloWorld {
          input {
            Pair[File, String] testURL
          }
          call helloWDL {
            input: testURL=testURL
          }
        }

        task helloWDL {
          input {
            Pair[File, String] testURL
          }

          command <<<
            echo "INPUT FOLDER: $(ls ~{testURL.left})" >> test.txt
            cp  ~{testURL.left}  ~{testURL.right}
            ls -latr  >> test.txt
          >>>

          output {
            File output_wdl = "test.txt"
          }

          runtime{
            docker: "ubuntu@sha256:4b1d0c4a2d2aaf63b37111f34eb9fa89fa1bf53dd6e4ca954d47caebca4005c2"
            memory: "2GiB"
            runtime_minutes: 10
            cpu: 2
            }
        }

    Input JSON Example:

    .. code-block:: text
      
        {
          "HelloWorld.testURL": { "Left": "https://portal.nersc.gov/cfs/m342/jaws/test_data/trinity/reads.left.fa.gz", "Right": "reads.left.fa.gz" }
        }

    Why This Works?

     - Manual File Renaming: Ensures the file retains its correct basename and extension, preventing issues with tools that rely on specific file naming conventions.
     - Portability: The solution works seamlessly for both local files and HTTP-based inputs, making your WDL workflow more flexible and robust.

.. dropdown:: Does Cromwell offer checkpointing? :ref:`🔗 <Does Cromwell offer checkpointing?>`
    :color: info
    :animate: fade-in
    :name: Does Cromwell offer checkpointing?
    
    Cromwell has call caching instead which accomplishes the same thing. When a task completes successfully, it's results are capable of being reused if the same task and inputs are run again. Use :bash:`jaws submit --no-cache` to turn caching off.
    

.. dropdown:: Why didn't call caching work for me? :ref:`🔗 <Why didn't call caching work for me?>`
    :color: info
    :animate: fade-in
    :name: Why didn't call caching work for me?
    
    Changes to the WDL, the name contents of the inputs.json, or the name of the inputs.json will prevent call-caching.
    
    For example, if you set your task's runtime attributes using input variables, changes to the values of these variables count as changes to the inputs, resulting in a different hash for the task (the wdl and inputs.json are hashed).

    Call caching may have failed if your files are being fed in as String rather than File inputs. The hashes of two identical Files stored in different locations would be the same. The hashes of the String values for the different locations would be different, even though the contents of the file are the same.

    Call caching also requires consistency in the outputs of the task, both the count (number of outputs) and the output expressions. If you publish a new version of your WDL that has one extra or one fewer output, it will not be able to benefit from a previously successful run of the same task, even if the inputs are the same.
    

.. dropdown:: Why do JAWS jobs fail if filenames contain special characters like `\` or `;`? :ref:`🔗 <Why do JAWS jobs fail if filenames contain special characters?>`
    :color: info
    :animate: fade-in
    :name: Why do JAWS jobs fail if filenames contain special characters?

    JAWS jobs fail when input file names contain special characters such as `\` (apostrophe) or `;` (semicolon). 
    Cromwell, the workflow execution engine used by JAWS, does not handle special characters properly. 
    To avoid failures please don't use special characters in your filenames.


    For example, the following error might occur:

    .. code-block:: bash

      cat ~/cromwell-executions/test_weird_chars/d277a390-4552-490a-8bcf-af02a80c7718/call-file/execution/stderr.submit
      ~/script: line 52: syntax error near unexpected token `&&'
      ~/script: line 52: `find . -type d -exec sh -c '[ -z "$(ls -A '"'"'{}'"'"')" ] && touch '"'"'{}'"'"'/.file' \;'


.. dropdown:: Can I use WDL version 1.1 with JAWS? :ref:`🔗 <Can I use WDL version 1.1 with JAWS?>`
    :color: info
    :animate: fade-in
    :name: Can I use WDL version 1.1 with JAWS?

    No, JAWS currently does **not support WDL version 1.1**.

    Cromwell, the workflow execution engine used by JAWS, does not fully support `WDL 1.1 <https://github.com/openwdl/wdl/blob/legacy/versions/1.1/SPEC.md>`_ yet. 
    Workflows written in that version may fail to parse or execute correctly. 

    Please use `WDL version 1.0 <https://github.com/openwdl/wdl/blob/legacy/versions/1.0/SPEC.md>`_, which is officially supported and well-tested within the JAWS ecosystem.

    You can check your WDL version by looking at the first line of the workflow file:

    .. code-block:: bash

        version 1.0

    For more information, refer to the Cromwell documentation `here <https://cromwell.readthedocs.io/en/stable/LanguageSupport/#wdl-10>`_.


Compute Systems
===============

.. dropdown:: What flavor of linux do the compute nodes run? :ref:`🔗 <What flavor of linux do the compute nodes run?>`
    :color: info
    :animate: fade-in
    :name: What flavor of linux do the compute nodes run?
    
    JAWS makes multiple computing resources available, using various linux distros. Thus, we recommend that a docker container be specified for every task; if not, the default container is Ubuntu.

.. dropdown:: What would the :bash:`runtime{}` section look like if your task required 8 threads and "5G" RAM.
    :color: info
    :animate: fade-in

    .. code-block:: text
    
        runtime {
            memory: "5G"
            cpu: 8
        }

    For a `dori` node (492G usable ram, threads: 64), you could run 8 tasks in parallel because (64 threads/8 = 8) and you would have more than the required 5G of ram per task since (492G/8 = 61.5G).


.. dropdown:: What happens if I request 64 threads but only 2G of the possible 128G of ram?
    :color: info
    :animate: fade-in

    A regular `dori` node has 492G and 64 threads. So what happens if you request 

    .. code-block:: text
    
        runtime {
            memory: "2G"
            cpu: 64
        }

    Will you be restricted to 2G or will you have access to 492G? 
    Since we don't have any memory limits in place in HTCondor you would be allowed to use all 492G on the node. 
    The reverse should be true as well, if you ask for 492G but only 2 threads (cpu: 2), you should still have access to 64 threads.

.. dropdown:: How should I set the :bash:`runtime{}` section when I want to run many scattering jobs and use multiple nodes.
    :color: info
    :animate: fade-in

    For example, if you request ~500 tasks each with :bash:`runtime{}`:

    .. code-block:: text
    
        runtime {
            memory: "8G"
            cpu: 4
        }

    HTCondor will put them in the queue and the pool manager will start getting new nodes. 
    If we had access to a maximum of 30 nodes, it would grab 30 nodes and start running as many parts of the scatter it can. 
    If nothing else is running you could get 480 of them to run at the same time.


JAWS
====
.. dropdown:: How do I set up JAWS? :ref:`🔗 <How do I set up JAWS?>`
    :color: info
    :animate: fade-in
    :name: How do I set up JAWS?

    To set up JAWS, follow the instructions in the :doc:`JAWS Setup Guide </jaws/jaws_config>`.
.. dropdown:: Will my container's entrypoint script be executed by JAWS? :ref:`🔗 <Will my container's entrypoint script be executed by JAWS?>`
    :color: info
    :animate: fade-in
    :name: Will my container's entrypoint script be executed by JAWS?
    
    JAWS does not execute entrypoint scripts, and users cannot alter this behavior.

    The ENTRYPOINT instruction sets the default executable for the container, and any arguments passed to the docker run command are appended to it.

    However, Cromwell generates a script file that the container runs instead. 
    This script includes the command specified in the command stanza, with all variables expanded, 
    as well as additional Cromwell-specific instructions. 
    As a result, the container's entrypoint script is ignored by both Cromwell and the JAWS backend, 
    even if specified.


.. dropdown:: What should I do if I encounter a timezone offset warning when using JAWS Container? :ref:`🔗 <What should I do if I encounter a timezone offset warning when using JAWS Container?>`
    :color: info
    :animate: fade-in
    :name: What should I do if I encounter a timezone offset warning when using JAWS Container?

    If you're using the JAWS Client Container, you might see a warning similar to the following when running the JAWS commands:

    .. code-block:: bash

      JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/clusterfs/jgi/groups/dsi/homes/svc-jaws/dori-prod/jaws-prod.conf apptainer run docker://doejgi/jaws-client:latest jaws queue
      INFO:   Using cached SIF image
      /usr/local/lib/python3.11/site-packages/local/utils.py:43: UserWarning: Timezone offset does not match system offset: 0 != -25200. Please check your config files.
      warnings.warn(msg)
      []

    This warning occurs because of a mismatch between the detected `timezone` offset and the system's offset. 
    While this doesn't affect the functionality of JAWS commands, you can remove the warning by setting the `TZ` environment variable.

    1. Add the following line to your `~/.bashrc` file:

    .. code-block:: bash

        export TZ="America/Los_Angeles"

    2. Reload your `~/.bashrc` by running:
    
     .. code-block:: bash
  
        source ~/.bashrc

    After doing this, the warning should no longer appear, and your system's timezone will be correctly aligned.

.. dropdown:: Why doesn't JAWS copy the entire Cromwell execution for successful runs? :ref:`🔗 <Why doesn't JAWS copy the entire Cromwell execution for successful runs?>`
  :color: info
  :animate: fade-in
  :name: Why doesn't JAWS copy the entire Cromwell execution for successful runs?

  To optimize performance and avoid unnecessary data movement—especially when workflows generate
  terabytes of data—JAWS only transfers files explicitly defined in the WDL ``output`` section. This approach:

  - Reduces network load and file system congestion
  - Helps users manage only the data they actually need

  If you need additional outputs like logs for successful runs, you must declare them as workflow outputs.


.. dropdown:: How can I access ``stderr`` and ``stdout`` for debugging purposes? :ref:`🔗 <How can I access stderr and stdout for debugging purposes?>`
  :color: info
  :animate: fade-in
  :name: How can I access stderr and stdout for debugging purposes?

  **For failed runs:**

  Use the ``jaws download <run_id>`` command to retrieve the full Cromwell execution directory,
  including all ``stderr`` and ``stdout`` logs for every task that failed. Additionally, JAWS generates an
  ``error.json`` file summarizing the failure and embedding these logs for quick reference.

  **For successful runs:**

  To access logs like ``stderr`` and ``stdout``, explicitly define them in the WDL task's output section. Example:

  .. code-block:: text

      task example_task {
        command <<<
          ./run_analysis.sh > stdout 2> stderr
        >>>

        output {
          File stdout_log = "stdout"
          File stderr_log = "stderr"
        }
      }

  This ensures that JAWS copies those logs back to the submission site along with your final outputs.

Troubleshooting 
---------------

.. dropdown:: What happens if a run fails with JAWS? :ref:`🔗 <What happens if a run fails with JAWS?>`
  :color: info
  :animate: fade-in
  :name: What happens if a run fails with JAWS?

  If a workflow run fails, JAWS does **not automatically transfer** the Cromwell execution folder
  to the submission site. However, you can retrieve the full execution directory for the failed tasks for debugging by running:

  .. code-block:: bash

      jaws download <run_id>

  This command triggers a secure transfer of all intermediate files, including ``stderr``, ``stdout``, task
  inputs, from the compute site back to the origin site (e.g., Perlmutter or Dori).


.. dropdown:: Why does `jaws.validate()` fail with a message about ShellCheck? :ref:`🔗 <Why does jaws.validate fail with a message about ShellCheck?>`
   :color: info
   :animate: fade-in
   :name: Why does jaws.validate fail with a message about ShellCheck?

   `ShellCheck` is an optional dependency used to validate shell commands inside WDL task blocks.  
   If it is not installed and you use :bash:`shell_check=True`, validation will return a warning or failure.

   See :ref:`ShellCheck Validation <shellcheck-validation>` for how to install and use it.


.. dropdown:: My job failed but I can't find an rc file :ref:`🔗 <My job failed but I can't find an rc file>`
  :color: info
  :animate: fade-in
  :name: My job failed but I can't find an rc file

   If your run fails and you can't find the `RC` (return code) file in the task's execution directory, 
   you may see an error in `error.json` like:
  
   .. code-block:: text

      "message": "Unable to determine that 4572306 is alive, and /clusterfs/.../execution/rc does not exist."

   This happens when HTCondor (JAWS backend) has a network or scheduler issue that interrupts communication 
   with the compute node. In these cases, the job may disconnect and reconnect, but if HTCondor 
   cannot cleanly finish the task, the `RC` file is never created.

   Without the `RC` file, Cromwell/JAWS cannot determine the task's final status, so the run is 
   reported as failed even if the underlying command was running normally. 
   You may also see disconnection/reconnection messages in the `execution.log`, for example:

   .. code-block:: text

     Job disconnected, attempting to reconnect
     DisconnectReason: Local schedd and job shadow died, schedd now running again

   **What should I do?**

   In most cases, simply resubmit:

   .. code-block:: text

      jaws resubmit <run_id>


   The task usually succeeds on retry.

   If the error repeats, contact the JAWS team with the run ID and the error details so we can 
   investigate HTCondor and site logs.

   **Key points**

    - This error is not caused by your workflow or WDL script.
    - It is a transient HTCondor/site issue.
    - Resubmission almost always resolves the problem.


Data Access
-----------

.. dropdown:: How can I check the Cromwell execution for a run on a site that I don't have access to, for example Tahoma? :ref:`🔗 <How can I check the Cromwell execution?>`
  :color: info
  :animate: fade-in
  :name: How can I check the Cromwell execution?

  
  If you want full access to the Cromwell execution folder, you need access to the compute cluster. However, for these cases, JAWS transfers data back to the input site.
  If you submitted a run from NERSC (e.g., `perlmutter` JAWS site) to the Tahoma cluster and don't have access to Tahoma:

  - **For successful runs**: JAWS will automatically transfer the expected output files back to the input site (e.g., `perlmutter`). Use the `jaws status` command to find the path to the output directory:

    .. code-block:: text

      jaws status <run_id> | grep output_dir
      "output_dir": "<JAWS TEAMS DIRECTORY>/<USER>/<JAWS_RUN_ID>/<CROMWELL_ID>",


  - **For failed runs**: JAWS will **not** automatically transfer the entire Cromwell execution. Instead, it only transfers supplementary files, including the `error.json`, which can help debug the issue. 
    To find the path to the output directory, use the `jaws status` command. 
    If the `error.json` is insufficient for debugging, you can transfer the entire Cromwell execution **task** (note that JAWS only transfers the tasks that failed) using the following command:

    .. code-block:: text
      
        jaws download <run_id>


  Why doesn't JAWS copy the entire Cromwell execution for successful/failed runs?
  
  To prevent unnecessary data movement, JAWS does not automatically copy the entire Cromwell execution for successful runs. 
  Some workflows generate terabytes of data with many intermediate files, and transferring all data would be inefficient. Only the output files specified in the `output stanza` are copied.
  For failed runs, JAWS does not copy the entire Cromwell execution by default either. Sometimes, runs fail due to simple system errors, and the `error.json` file is sufficient to debug the issue.


Known Limitations
=================

.. dropdown:: Outputs outside the `cromwell-execution/execution/` directory :ref:`🔗 <Outputs outside the cromwell-execution/execution/ directory>`
    :color: info
    :animate: fade-in
    :name: Outputs outside the cromwell-execution/execution/ directory
    
    When using JAWS, there are some limitations related to output file handling that users should be aware of. Example Workflow:

    .. code-block:: text
    
          version 1.0

          workflow example {
            call dump { }
            output {
              File log = write_lines(['foo'])
            }
          }

          task dump {
            command <<<
              echo 
            >>>
            runtime { docker: "debian:bullseye-slim" }
          }
    
    **Issue Description**

    JAWS is unable to identify any outputs outside the :bash:`cromwell-execution/execution/` directory to transfer to the JAWS Teams directory. 
    For example, if a file is created in the :bash:`/tmp` directory, JAWS will not be able to recognize and transfer it, as shown in the example above.

    Similarly, if a user identifies a file from the inputs folder as a final ouput, JAWS will also be unable to copy it.
    
    **Workaround**

    To ensure that JAWS correctly identifies and transfers the output files, you should:

    - Save the string to a file and in the outputs stanza of your workflow.
    - Explicitly copy the final outputs files to :bash:`cromwell-execution/execution/` directory.

    Here’s how you can adjust the example workflow:

    .. code-block:: text

          version 1.0

          workflow example {
            call dump
            output {
              File log = dump.log
            }
          }

          task dump {
            command <<<
              echo foo > out.log
            >>>
            output {
              File log = "out.log"
            }
            runtime {
              docker: "debian:bullseye-slim"
            }
          }

    By following this approach, you ensure that all output files are correctly saved within the :bash:`cromwell-execution/execution/` directory and are explicitly defined, 
    allowing JAWS to identify and transfer them back to the user without issues.