FAQs

This document provides solutions to frequently asked questions about JAWS, WDL development, and workflow execution. You’ll find code snippets, explanations, and best practices organized by topic.

Table of Contents

Resources

WDL Development

How to use bash commands with curly braces in WDL 🔗

If you ever need to use curly braces in bash to strip a suffix ‘txt’ or set a default, consider using the WDL Version 1.0 Specification. In WDL document, the first line should indicate version 1.0. You’ll specify the command section like command <<< >>> instead of using curly braces. Additionally, you’ll need to adjust some other formats. Refer to the Version 1.0 specification link for more details.

command <<<
    # setting a default value in bash
    VAR=${VAR:=25}

    # strip a suffix
    myvar=${somefile%.txt}
>>>
How can I output a file that has been named dynamically as a bash variable 🔗

Bash variables created in the command { } block cannot be seen outside the block, for example, in the output { } section. Therefore, you can write the name(s) of any output files to another file which will be read inside the output { } block.

This is the official WDL way, using glob:

output {
    Array[File] output_bams = glob("*.bam")
}

This is another method:

command{
    echo $lib.bam > list_of_files
 }
 output {
    Array[File] = read_lines("list_of_files")
 }

To see more about read_lines() and other WDL functions, see openwdl/wdl.

Using conditionals 🔗
workflow conditional_example {
  File infile

  call wc as wc_before { input: infile = infile }

  Int num_lines = wc_before.num_lines

  if (num_lines > 10) {
    call truncate { input: infile = infile }
  }

  # This function will return false if the defined() argument is an
  # unset optional value. It will return true in all other cases.
  Boolean has_head_file = defined(truncate.outfile)

  if (has_head_file) {
    call wc as wc_after { input: infile = truncate.outfile }
  }

  # notice the '?' after File. These are required since these files may not exist.
  output {
    File wc_before_file = wc_before.outfile
    File? head_file = truncate.outfile
    File? wc_after_file = wc_after.outfile
  }
}

task wc {
  File infile
  command { wc -l < ${infile} | tee wc.txt }
  output {
    Int num_lines = read_int(stdout())
    File outfile = "wc.txt"
  }
}
How to scatter over arrays and maps 🔗

Although you can scatter over arrays and maps, there is different syntax for each. You can only scatter over an array with this syntax

Array[String] some_array
scatter (e in some_array) {
  String value = some_array[e]
  call some_task {input: value = value}
}

But you can iterate over a map by using the ‘pair’ keyword and then ‘.left’ and ‘.right’ as such

Map[String,String] some_map
scatter (pair in some_map) {
  String key= pair.left
  String value = pair.right # or String val = some_map[key]
  call some_task {input: value = value}
}

You can see working examples for scattering an array and scattering a map.

Custom data structures 🔗

Besides Map, Array, Pair you can create a custom data structure using “struct”. This will be similar to a hash but can contain any combination of data types.

Get Keys from a Map 🔗

As of version 1.0 of the wdl spec, there is no direct way to get an array of Map keys. This will become available in version 1.1. As a work-around for now, you can use the Pair data type instead of Map as follows.

version 1.0
workflow test {
  input {
    Array[Pair[Float,String]] my_map = [(0.1, "mouse"), (3, "cat"), (15, "dog")]
  }

  scatter (pairs in my_map) {
    String keys = pairs.left
  }

  output {Array[String] allouts = keys}
}

Note that the default format for my_map in the WDL is different that in an input.json

{
  "test.my_map": [{"Left": 0.1, "Right": "mouse"}, {"Left": 3, "Right": "cat"}, {"Left": 15, "Right": "dog"}]
}
An example of a scatter/gather model when a scattered task is optional 🔗

You can use this example to see where you need to declare optional varaibles (i.e. Array[Array[String?]]) and how you can use two wdl functions, flatten and select_all to convert an optional variable (Array[Array[String?]) to an Array[String].

See this line:

Array[String] flat_array = flatten(select_all(num_array))
# note that this line is not within any stanza, but between input{} and command<<<>>>

and note that

flatten is used to convert an array-of-an-array to an array.
select_all, used in this example, converts Array[String?] to Array[String].
If you don’t use select_all here, you get the error:
Expected ‘Array[Array[_]]’ but got ‘Array[Array[String]?]’
version 1.0

workflow flatten_it {
    input {
        Boolean try_it = false
        Array[Int] numbers = [1,2,3]
    }

    scatter (num in numbers) {
      if (try_it == true) {
        call do_if_true {
          input: num = num
        }
      }

      # call something_else {}
    }

    call gather  {
      input: num_array = do_if_true.out
    }

   output {
     Array[File] out_array = gather.final_array
   }
}

task do_if_true {
     input {
        Int num
     }

     command <<<
        echo "~{num}.one"
        echo "~{num}.two"
        echo "~{num}.three"
     >>>

     output {
         Array[String] out = read_lines(stdout())
    }
}

task gather {
    input {
        Array[Array[String]?] num_array
    }

    Array[String] flat_array = flatten(select_all(num_array))

    command <<<
        echo ~{sep=', ' flat_array}
    >>>

    output {
       Array[String] final_array = read_lines(stdout())
    }
}
Why does jaws validate fail when using “string” + <WDL variable> in the runtime block? 🔗

🛑 Issue

You may see the following error when trying to concatenate a string and a WDL variable in a runtime block:

Cannot add/concatenate String and String?
docker: "ubuntu:" + tag

This issue is not detected by womtool, so workflows may still submit successfully using jaws submit (which internally uses womtool validation).

However, jaws validate uses miniwdl under the hood, which enforces stricter WDL 1.0 compliance and fails validation when expressions like "ubuntu:" + tag are used in runtime attributes.

💡 Explanation

The problem arises because the variable tag is declared as an optional string (String?), which makes the expression "ubuntu:" + tag ambiguous if tag is missing.

Removing the optional (?) can fix the immediate error for validation.

WDL Example that causes the issue:

version 1.0

workflow HelloWorld {
  call helloWDL
}

task helloWDL {
  input {
    Int cpu = 2
    Int mem = 8
    String? tag = "latest"
  }
  command <<<
    echo "Hello WDLs" >> test.txt
  >>>

  output {
    File output_wdl = "test.txt"
  }
  runtime{
      docker: "ubuntu:" + tag
      memory: mem + " GiB"
      runtime_minutes: 10
      cpu: cpu /1
    }
}

🔍 Summary

  • jaws validate uses miniwdl and will reject runtime expressions like "ubuntu:" + tag if tag is optional.

  • jaws submit uses womtool, which is more permissive and may allow the submission.

Cromwell

Handling HTTP URLs Inputs in Cromwell Workflows 🔗

Problem: When using HTTP URLs as inputs in Cromwell workflows, Cromwell automatically downloads and stores the files locally. However, during this process, files are renamed, and file extensions may be lost. This can cause issues for programs that rely on specific file extensions or naming conventions to function correctly.

This is a known issue in Cromwell. For more details, refer to the Cromwell Filesystems - HTTP documentation.

Solution To make your workflow more portable and compatible with both HTTP and local files, you can copy the downloaded file to match its expected file name and extension. A useful approach is to use a Pair data type to pass the URL and the desired basename for the file. Then, in the task’s command stanza, copy the file to the correct name.

Example WDL Workflow:

version 1.0

workflow HelloWorld {
  input {
    Pair[File, String] testURL
  }
  call helloWDL {
    input: testURL=testURL
  }
}

task helloWDL {
  input {
    Pair[File, String] testURL
  }

  command <<<
    echo "INPUT FOLDER: $(ls ~{testURL.left})" >> test.txt
    cp  ~{testURL.left}  ~{testURL.right}
    ls -latr  >> test.txt
  >>>

  output {
    File output_wdl = "test.txt"
  }

  runtime{
    docker: "ubuntu@sha256:4b1d0c4a2d2aaf63b37111f34eb9fa89fa1bf53dd6e4ca954d47caebca4005c2"
    memory: "2GiB"
    runtime_minutes: 10
    cpu: 2
    }
}

Input JSON Example:

{
  "HelloWorld.testURL": { "Left": "https://portal.nersc.gov/cfs/m342/jaws/test_data/trinity/reads.left.fa.gz", "Right": "reads.left.fa.gz" }
}

Why This Works?

  • Manual File Renaming: Ensures the file retains its correct basename and extension, preventing issues with tools that rely on specific file naming conventions.

  • Portability: The solution works seamlessly for both local files and HTTP-based inputs, making your WDL workflow more flexible and robust.

Does Cromwell offer checkpointing? 🔗

Cromwell has call caching instead which accomplishes the same thing. When a task completes successfully, it’s results are capable of being reused if the same task and inputs are run again. Use jaws submit --no-cache to turn caching off.

Why didn’t call caching work for me? 🔗

Changes to the WDL, the name contents of the inputs.json, or the name of the inputs.json will prevent call-caching.

For example, if you set your task’s runtime attributes using input variables, changes to the values of these variables count as changes to the inputs, resulting in a different hash for the task (the wdl and inputs.json are hashed).

Call caching may have failed if your files are being fed in as String rather than File inputs. The hashes of two identical Files stored in different locations would be the same. The hashes of the String values for the different locations would be different, even though the contents of the file are the same.

Call caching also requires consistency in the outputs of the task, both the count (number of outputs) and the output expressions. If you publish a new version of your WDL that has one extra or one fewer output, it will not be able to benefit from a previously successful run of the same task, even if the inputs are the same.

Why do JAWS jobs fail if filenames contain special characters like ` or `;? 🔗

JAWS jobs fail when input file names contain special characters such as ` (apostrophe) or `; (semicolon). Cromwell, the workflow execution engine used by JAWS, does not handle special characters properly. To avoid failures please don’t use special characters in your filenames.

For example, the following error might occur:

cat ~/cromwell-executions/test_weird_chars/d277a390-4552-490a-8bcf-af02a80c7718/call-file/execution/stderr.submit
~/script: line 52: syntax error near unexpected token `&&'
~/script: line 52: `find . -type d -exec sh -c '[ -z "$(ls -A '"'"'{}'"'"')" ] && touch '"'"'{}'"'"'/.file' \;'
Can I use WDL version 1.1 with JAWS? 🔗

No, JAWS currently does not support WDL version 1.1.

Cromwell, the workflow execution engine used by JAWS, does not fully support WDL 1.1 yet. Workflows written in that version may fail to parse or execute correctly.

Please use WDL version 1.0, which is officially supported and well-tested within the JAWS ecosystem.

You can check your WDL version by looking at the first line of the workflow file:

version 1.0

For more information, refer to the Cromwell documentation here.

Compute Systems

What flavor of linux do the compute nodes run? 🔗

JAWS makes multiple computing resources available, using various linux distros. Thus, we recommend that a docker container be specified for every task; if not, the default container is Ubuntu.

What would the runtime{} section look like if your task required 8 threads and “5G” RAM.
runtime {
    memory: "5G"
    cpu: 8
}

For a dori node (492G usable ram, threads: 64), you could run 8 tasks in parallel because (64 threads/8 = 8) and you would have more than the required 5G of ram per task since (492G/8 = 61.5G).

What happens if I request 64 threads but only 2G of the possible 128G of ram?

A regular dori node has 492G and 64 threads. So what happens if you request

runtime {
    memory: "2G"
    cpu: 64
}

Will you be restricted to 2G or will you have access to 492G? Since we don’t have any memory limits in place in HTCondor you would be allowed to use all 492G on the node. The reverse should be true as well, if you ask for 492G but only 2 threads (cpu: 2), you should still have access to 64 threads.

How should I set the runtime{} section when I want to run many scattering jobs and use multiple nodes.

For example, if you request ~500 tasks each with runtime{}:

runtime {
    memory: "8G"
    cpu: 4
}

HTCondor will put them in the queue and the pool manager will start getting new nodes. If we had access to a maximum of 30 nodes, it would grab 30 nodes and start running as many parts of the scatter it can. If nothing else is running you could get 480 of them to run at the same time.

JAWS

How do I set up JAWS? 🔗

To set up JAWS, follow the instructions in the JAWS Setup Guide.

Will my container’s entrypoint script be executed by JAWS? 🔗

JAWS does not execute entrypoint scripts, and users cannot alter this behavior.

The ENTRYPOINT instruction sets the default executable for the container, and any arguments passed to the docker run command are appended to it.

However, Cromwell generates a script file that the container runs instead. This script includes the command specified in the command stanza, with all variables expanded, as well as additional Cromwell-specific instructions. As a result, the container’s entrypoint script is ignored by both Cromwell and the JAWS backend, even if specified.

What should I do if I encounter a timezone offset warning when using JAWS Container? 🔗

If you’re using the JAWS Client Container, you might see a warning similar to the following when running the JAWS commands:

JAWS_USER_CONFIG=~/jaws.conf JAWS_CLIENT_CONFIG=/clusterfs/jgi/groups/dsi/homes/svc-jaws/dori-prod/jaws-prod.conf apptainer run docker://doejgi/jaws-client:latest jaws queue
INFO:   Using cached SIF image
/usr/local/lib/python3.11/site-packages/local/utils.py:43: UserWarning: Timezone offset does not match system offset: 0 != -25200. Please check your config files.
warnings.warn(msg)
[]

This warning occurs because of a mismatch between the detected timezone offset and the system’s offset. While this doesn’t affect the functionality of JAWS commands, you can remove the warning by setting the TZ environment variable.

  1. Add the following line to your ~/.bashrc file:

export TZ="America/Los_Angeles"

  1. Reload your ~/.bashrc by running:

source ~/.bashrc

After doing this, the warning should no longer appear, and your system’s timezone will be correctly aligned.

Why doesn’t JAWS copy the entire Cromwell execution for successful runs? 🔗

To optimize performance and avoid unnecessary data movement—especially when workflows generate terabytes of data—JAWS only transfers files explicitly defined in the WDL output section. This approach:

  • Reduces network load and file system congestion

  • Helps users manage only the data they actually need

If you need additional outputs like logs for successful runs, you must declare them as workflow outputs.

How can I access stderr and stdout for debugging purposes? 🔗

For failed runs:

Use the jaws download <run_id> command to retrieve the full Cromwell execution directory, including all stderr and stdout logs for every task that failed. Additionally, JAWS generates an error.json file summarizing the failure and embedding these logs for quick reference.

For successful runs:

To access logs like stderr and stdout, explicitly define them in the WDL task’s output section. Example:

task example_task {
  command <<<
    ./run_analysis.sh > stdout 2> stderr
  >>>

  output {
    File stdout_log = "stdout"
    File stderr_log = "stderr"
  }
}

This ensures that JAWS copies those logs back to the submission site along with your final outputs.

Troubleshooting

What happens if a run fails with JAWS? 🔗

If a workflow run fails, JAWS does not automatically transfer the Cromwell execution folder to the submission site. However, you can retrieve the full execution directory for the failed tasks for debugging by running:

jaws download <run_id>

This command triggers a secure transfer of all intermediate files, including stderr, stdout, task inputs, from the compute site back to the origin site (e.g., Perlmutter or Dori).

Why does jaws.validate() fail with a message about ShellCheck? 🔗

ShellCheck is an optional dependency used to validate shell commands inside WDL task blocks. If it is not installed and you use shell_check=True, validation will return a warning or failure.

See ShellCheck Validation for how to install and use it.

My job failed but I can’t find an rc file 🔗

If your run fails and you can’t find the RC (return code) file in the task’s execution directory, you may see an error in error.json like:

"message": "Unable to determine that 4572306 is alive, and /clusterfs/.../execution/rc does not exist."

This happens when HTCondor (JAWS backend) has a network or scheduler issue that interrupts communication with the compute node. In these cases, the job may disconnect and reconnect, but if HTCondor cannot cleanly finish the task, the RC file is never created.

Without the RC file, Cromwell/JAWS cannot determine the task’s final status, so the run is reported as failed even if the underlying command was running normally. You may also see disconnection/reconnection messages in the execution.log, for example:

Job disconnected, attempting to reconnect
DisconnectReason: Local schedd and job shadow died, schedd now running again

What should I do?

In most cases, simply resubmit:

jaws resubmit <run_id>

The task usually succeeds on retry.

If the error repeats, contact the JAWS team with the run ID and the error details so we can investigate HTCondor and site logs.

Key points

  • This error is not caused by your workflow or WDL script.

  • It is a transient HTCondor/site issue.

  • Resubmission almost always resolves the problem.

Data Access

How can I check the Cromwell execution for a run on a site that I don’t have access to, for example Tahoma? 🔗

If you want full access to the Cromwell execution folder, you need access to the compute cluster. However, for these cases, JAWS transfers data back to the input site. If you submitted a run from NERSC (e.g., perlmutter JAWS site) to the Tahoma cluster and don’t have access to Tahoma:

  • For successful runs: JAWS will automatically transfer the expected output files back to the input site (e.g., perlmutter). Use the jaws status command to find the path to the output directory:

    jaws status <run_id> | grep output_dir
"output_dir": "<JAWS TEAMS DIRECTORY>/<USER>/<JAWS_RUN_ID>/<CROMWELL_ID>",

  • For failed runs: JAWS will not automatically transfer the entire Cromwell execution. Instead, it only transfers supplementary files, including the error.json, which can help debug the issue. To find the path to the output directory, use the jaws status command. If the error.json is insufficient for debugging, you can transfer the entire Cromwell execution task (note that JAWS only transfers the tasks that failed) using the following command:

    jaws download <run_id>

Why doesn’t JAWS copy the entire Cromwell execution for successful/failed runs?

To prevent unnecessary data movement, JAWS does not automatically copy the entire Cromwell execution for successful runs. Some workflows generate terabytes of data with many intermediate files, and transferring all data would be inefficient. Only the output files specified in the output stanza are copied. For failed runs, JAWS does not copy the entire Cromwell execution by default either. Sometimes, runs fail due to simple system errors, and the error.json file is sufficient to debug the issue.

Known Limitations

Outputs outside the cromwell-execution/execution/ directory 🔗

When using JAWS, there are some limitations related to output file handling that users should be aware of. Example Workflow:

version 1.0

workflow example {
  call dump { }
  output {
    File log = write_lines(['foo'])
  }
}

task dump {
  command <<<
    echo
  >>>
  runtime { docker: "debian:bullseye-slim" }
}

Issue Description

JAWS is unable to identify any outputs outside the cromwell-execution/execution/ directory to transfer to the JAWS Teams directory. For example, if a file is created in the /tmp directory, JAWS will not be able to recognize and transfer it, as shown in the example above.

Similarly, if a user identifies a file from the inputs folder as a final ouput, JAWS will also be unable to copy it.

Workaround

To ensure that JAWS correctly identifies and transfers the output files, you should:

  • Save the string to a file and in the outputs stanza of your workflow.

  • Explicitly copy the final outputs files to cromwell-execution/execution/ directory.

Here’s how you can adjust the example workflow:

version 1.0

workflow example {
  call dump
  output {
    File log = dump.log
  }
}

task dump {
  command <<<
    echo foo > out.log
  >>>
  output {
    File log = "out.log"
  }
  runtime {
    docker: "debian:bullseye-slim"
  }
}

By following this approach, you ensure that all output files are correctly saved within the cromwell-execution/execution/ directory and are explicitly defined, allowing JAWS to identify and transfer them back to the user without issues.