==== FAQs ==== .. role:: bash(code) :language: bash Frequently asked questions about JAWS, WDL development, and workflow execution. Find quick answers, code examples, and troubleshooting tips organized by category. .. contents:: Quick Navigation :local: :depth: 2 Setup, Authentication, and Environment ======================================= These cover the most-asked setup questions by the users. The authoritative setup walk-through is in :doc:`../jaws/jaws_config` and :doc:`../jaws/jaws_quickstart`; this section answers the recurring "but what about…" follow-ups. .. dropdown:: How do I get a JAWS token? :ref:`🔗 ` :color: primary :icon: key :name: faq-how-do-i-get-a-jaws-token Fill out the `JAWS Token Request form `_. The JAWS team will send your token. Place it in :bash:`~/jaws.conf` under the ``[USER]`` section and set the file mode: .. code-block:: bash vi ~/jaws.conf [USER] token = default_team = chmod 600 ~/jaws.conf The ``chmod 600`` step is important: it restricts the file so that only you can read or write it. Your token is effectively a password, anyone who can read ``~/jaws.conf`` can act as you in JAWS. On a shared cluster, default file modes are often world-readable, so without ``chmod 600`` your token may be visible to everyone with an account on the system. **Never post your token in a public channel.** If it's posted by accident, ping the team and they'll rotate it. Full setup including the per-site ``jaws.conf`` template paths is in :doc:`../jaws/jaws_config`. .. dropdown:: My token isn't working, I get ``401 Unauthorized`` :ref:`🔗 ` :color: danger :icon: x-circle :name: faq-my-token-isn-t-working-i-get-401-unauthorized The full error looks like: .. code-block:: text {'detail': 'Provided token is not valid', 'status': 401, 'title': 'Unauthorized', ...} Check these in order: 1. **No quotes around the token value** in ``~/jaws.conf``. A literal ``token = "abc..."`` is invalid; remove the quotes. 2. No stray whitespace around ``=`` or trailing on the value. 3. **You're sourcing the right per-site setup.** Each site has its own ``jaws.conf`` template and its own ``module use`` path, see the tabs in :doc:`../jaws/jaws_config`. 4. The token may have expired or been rotated. Ask in ``#jaws channel`` for a refresh. .. dropdown:: How do I activate the JAWS environment on my site? :ref:`🔗 ` :color: primary :icon: terminal :name: faq-how-do-i-activate-the-jaws-environment-on-my-site Use :bash:`module load jaws` after pointing :bash:`module use` at your site's modulefiles directory. The per-site tabs in :doc:`../jaws/jaws_config` show the exact paths for DORI, NERSC (Perlmutter / NMDC / KBase / RNome / SuperBio) and Tahoma (NMDC). .. code-block:: bash # Example: NERSC Perlmutter (JGI) module use /global/cfs/cdirs/jaws/modulefiles module load jaws If you see :bash:`Warning: a python module is already loaded`, run :bash:`module unload python` before loading JAWS. You can also use JAWS container or jaws client python library (see :doc:`../jaws/jaws_config` for details). Sites and Site Selection ========================= .. dropdown:: How do I list available compute sites? :ref:`🔗 ` :color: primary :icon: server :name: faq-how-do-i-list-available-compute-sites Use the ``jaws teams list-sites`` command with your team ID. The list of sites are depending on the team you are part of. For example, for the team ``dsi-aa``: .. code-block:: bash jaws teams list-sites dsi-aa [ "defiant", "crux", "jgi", "nmdc_tahoma", "perlmutter", "tahoma", "dori" ] Use the returned site IDs (``dori``, ``perlmutter``, ``tahoma``, ``nmdc``, ``nmdc_tahoma``, ``kbase``, ``defiant``, and others as they come online) when you submit: .. code-block:: text jaws submit my.wdl my.json perlmutter See :doc:`../jaws/jaws_quickstart` for the current list. .. dropdown:: Which compute site should I use? :ref:`🔗 ` :color: info :icon: question :name: faq-which-compute-site-should-i-use Guidelines (verify against ``jaws teams list-sites dsi-aa`` and the site's queue limits in :doc:`../Resources/compute_resources`). If you have question or a big batch of data, ask in ``#jaws``. .. dropdown:: Where are outages and maintenance windows announced? :ref:`🔗 ` :color: info :icon: megaphone :name: faq-where-are-outages-and-maintenance-windows Three places, in order of immediacy: 1. ``#jaws`` Slack channel, ``@channel`` announcements from the JAWS team for ongoing outages and same-day downtime. 2. :doc:`JAWS Outages calendar <../training/outages>`, planned maintenance windows and known downtime, scheduled in advance. Best place to look when you're planning a run a week or two out. (Office hours, user meetings, and workshops live on the separate :doc:`JAWS Events calendar `.) 3. ``jaws health``, for the **current** state of each site right now. The calendar and Slack tell you what's *expected*; ``jaws health`` tells you what's *actually happening*. .. code-block:: bash jaws health WDL Development =============== Using AI Assistants to Write or Debug WDL ------------------------------------------ LLMs (CBorg, ChatGPT, Claude, Copilot, etc.) can be a real productivity boost when writing WDL, but they also produce a class of errors that is worth watching for. This section covers what to know before pasting AI-generated WDL into ``jaws submit``. .. dropdown:: Tell the AI which WDL spec to target, always 1.0 :ref:`🔗 ` :color: warning :icon: alert :name: faq-tell-the-ai-which-wdl-spec-to-target-always-1-0 **This is the single most important thing.** Out of the box, most AI assistants will generate **WDL 1.0** or later or using bash syntax, because that's what their training data biases toward. JAWS currently supports **WDL 1.0 only**, because the Cromwell version JAWS runs doesn't parse 1.1 features. (We'll upgrade Cromwell and lift that restriction, see "Can I use WDL 1.1 with JAWS?" below, but for now, 1.0 is the answer.) Always start your prompt with something like: .. code-block:: text Write a WDL workflow using the WDL 1.0 specification (https://github.com/openwdl/wdl/blob/legacy/versions/1.0/SPEC.md). Do not use WDL 1.1 features such as keys(), values(), None, Directory type, or enhanced struct features. The first line of every file must be `version 1.0`. Without this, the AI will often produce ``keys(my_map)`` or ``None`` literals that look correct but fail validation when you submit. The first symptom is usually a ``jaws validate`` error you can't immediately explain. .. dropdown:: Generating a new WDL with AI :ref:`🔗 ` :color: primary :icon: light-bulb :name: faq-generating-a-new-wdl-with-ai A workflow that works well: 1. **Prompt the AI** with: the WDL 1.0 constraint above, a clear description of the task, the input/output types you expect, and the resources the task will need (CPU, memory, ``runtime_minutes``). 2. **Validate** before submitting: .. code-block:: bash jaws validate workflow.wdl inputs.json If validation fails, paste the error back to the AI along with a reminder to target WDL 1.0. 3. **Submit a tiny test run** with minimal inputs before committing to a big batch. Things the AI is good at: - Boilerplate (task structure, runtime blocks, scatter patterns) - Translating a shell script into a ``command <<< ... >>>`` block - Suggesting input/output type signatures - Writing the matching ``inputs.json`` Things the AI is unreliable at: - JAWS-specific behavior; for example, ``runtime {runtime_minutes}`` attribute is a JAWS Cromwell engine features, and the AI might not know that. - Whether a particular Docker image is the right one, verify registry, tag, and that it actually exists on Docker Hub or ``library.jgi.doe.gov``. .. dropdown:: Debugging a failing WDL with AI :ref:`🔗 ` :color: primary :icon: bug :name: faq-debugging-a-failing-wdl-with-ai When a run fails and you want AI help diagnosing it, paste: 1. The **exact** error text. JAWS writes an ``errors.json`` file to the run's output directory; that's the first place to look. To find the path: .. code-block:: bash jaws status # look for "output_dir" cat /errors.json If ``errors.json`` doesn't tell you enough, fall back to the task's own ``stderr`` and ``stdout`` in the Cromwell execution directory. For runs that failed before the execution dir was copied back, you may need ``jaws download `` first to pull the failed task directories. Don't paraphrase the error text, the AI needs the literal output. 2. The WDL section that's failing (just the relevant task is enough; don't paste a 500-line workflow). 3. The relevant inputs (with sensitive values redacted). 4. **Tell the AI the JAWS context**: WDL 1.0, Cromwell as the executor, Shifter or Apptainer as the container runtime depending on the site. The AI is most useful for: - Suggesting alternate patterns when you hit a WDL 1.0 limitation (e.g. "can't scatter over a Map", see the "How do I get keys from a Map?" entry). - Explaining what a particular runtime attribute does. The AI is **least** reliable for telling you whether something is a JAWS infrastructure issue vs. a workflow bug. When in doubt, ask in ``#jaws`` with your run ID before assuming the AI is correct. .. dropdown:: A prompt template you can copy :ref:`🔗 ` :color: info :icon: copy :name: faq-a-prompt-template-you-can-copy .. code-block:: text I'm writing a WDL workflow for JAWS (JGI Analysis Workflow Service). Constraints: - WDL spec: 1.0 ONLY (https://github.com/openwdl/wdl/blob/legacy/versions/1.0/SPEC.md) - The first line of every file must be `version 1.0` - Do not use WDL 1.1 features: no keys(), values(), None, Directory type, or enhanced struct features - Each task must have a `runtime {}` block with at least: docker, cpu, memory, runtime_minutes - Use the heredoc command style: `command <<< ... >>>` - Use ~{var} for WDL interpolation, ${var} for bash inside the command block - Outputs must live inside the task's working directory (no /tmp, no absolute paths outside) Now: . Syntax and Language Features ----------------------------- .. dropdown:: How do I use bash commands with curly braces in WDL? :ref:`🔗 ` :color: primary :icon: code :name: faq-how-do-i-use-bash-commands-with-curly-braces-in **Problem**: Bash uses curly braces for parameter expansion (e.g., ``${VAR:-default}``), but WDL also uses ``${}`` for variable interpolation, causing conflicts. **Solution**: Use the heredoc-style command block (``<<<`` and ``>>>``) with WDL 1.0+: .. code-block:: text version 1.0 task example { input { String? optional_var } command <<< # Set bash default value VAR=${VAR:-25} # Strip file extension myvar=${somefile%.txt} # Use WDL variable with ~{} notation echo "WDL variable: ~{optional_var}" >>> runtime { docker: "ubuntu:22.04" } } **Key Points**: - Use ``<<<`` ``>>>`` delimiters (heredoc style) instead of ``{`` ``}`` - Use ``~{variable}`` for WDL interpolation inside heredoc - Bash ``${variable}`` syntax works normally inside heredoc 📖 `WDL 1.0 Spec: Command Section `_ .. dropdown:: How do I output dynamically named files? :ref:`🔗 ` :color: primary :icon: file :name: faq-how-do-i-output-dynamically-named-files **Problem**: Bash variables created in the ``command`` block aren't accessible in the ``output`` block. **Solution 1**: Use ``glob()`` to match patterns (recommended): .. code-block:: text task align_reads { command <<< # Generate output with dynamic name minimap2 ... > "${SAMPLE_ID}.bam" >>> output { Array[File] bam_files = glob("*.bam") File first_bam = glob("*.bam")[0] } } **Solution 2**: Write filenames to a manifest: .. code-block:: text task process_samples { command <<< for sample in ~{sep=' ' samples}; do analyze.sh "$sample" > "${sample}.result" echo "${sample}.result" >> output_manifest.txt done >>> output { Array[File] results = read_lines("output_manifest.txt") } } **Available WDL Functions**: - ``glob(pattern)`` - Match file patterns - ``read_lines(file)`` - Read file into Array[String] - ``read_tsv(file)`` - Parse TSV into Array[Array[String]] - ``read_json(file)`` - Parse JSON into WDL types 📖 `WDL Standard Library `_ .. dropdown:: How do I use conditionals in workflows? :ref:`🔗 ` :color: primary :icon: git-branch :name: faq-how-do-i-use-conditionals-in-workflows Use ``if`` statements to conditionally execute tasks: .. code-block:: text version 1.0 workflow conditional_processing { input { File input_file Boolean filter_data = true Int min_quality = 30 } call qc_check { input: infile = input_file } # Conditional task execution if (qc_check.pass_qc && filter_data) { call filter_reads { input: infile = input_file, min_qual = min_quality } } # Use select_first() to handle optional outputs File final_file = select_first([filter_reads.filtered_file, input_file]) call downstream_analysis { input: infile = final_file } output { File result = downstream_analysis.output_file File? filtered = filter_reads.filtered_file Boolean was_filtered = defined(filter_reads.filtered_file) } } **Important Functions**: - ``defined(variable)`` - Check if optional value is set - ``select_first([opt1, opt2, default])`` - Pick first defined value - ``select_all(array)`` - Remove undefined elements from Array 📖 `WDL Conditionals `_ Data Structures and Iteration ------------------------------ .. dropdown:: How do I scatter over arrays and maps? :ref:`🔗 ` :color: primary :icon: stack :name: faq-how-do-i-scatter-over-arrays-and-maps **Arrays** (simple scatter): .. code-block:: text workflow process_samples { input { Array[File] fastq_files File reference } scatter (fastq in fastq_files) { call align { input: reads = fastq, ref = reference } } # Gather scattered outputs into array output { Array[File] all_bams = align.bam } } **Maps** (iterate over key-value pairs): .. code-block:: text workflow process_configs { input { Map[String, String] sample_to_barcode = { "sample1": "ATCG", "sample2": "GCTA" } } scatter (pair in as_pairs(sample_to_barcode)) { String sample_id = pair.left String barcode = pair.right call demultiplex { input: sample = sample_id, barcode = barcode } } } **Cross-product scatter** (nested): .. code-block:: text scatter (sample in samples) { scatter (replicate in replicates) { call process { input: s = sample, r = replicate } } } # Flatten nested array: Array[Array[File]] → Array[File] Array[File] all_results = flatten(process.output) 📚 `Example workflows `_ .. dropdown:: How do I create custom data structures? :ref:`🔗 ` :color: primary :icon: package :name: faq-how-do-i-create-custom-data-structures Use ``struct`` to define custom types: .. code-block:: text version 1.0 struct SampleMetadata { String sample_id File fastq_r1 File fastq_r2 String sequencing_center Int read_length Float? quality_score } workflow analyze_samples { input { Array[SampleMetadata] samples } scatter (sample in samples) { call process { input: id = sample.sample_id, r1 = sample.fastq_r1, r2 = sample.fastq_r2 } } } **Input JSON**: .. code-block:: json { "analyze_samples.samples": [ { "sample_id": "S001", "fastq_r1": "s001_R1.fastq.gz", "fastq_r2": "s001_R2.fastq.gz", "sequencing_center": "JGI", "read_length": 150, "quality_score": 35.2 } ] } 📖 `WDL Structs `_ .. dropdown:: How do I get keys from a Map? :ref:`🔗 ` :color: primary :icon: key :name: faq-how-do-i-get-keys-from-a-map **Workaround for WDL 1.0** (use Pairs): .. code-block:: text version 1.0 workflow extract_keys { input { Array[Pair[String, Int]] data = [ ("sample1", 100), ("sample2", 200) ] } scatter (pair in data) { String key = pair.left Int value = pair.right } output { Array[String] all_keys = key Array[Int] all_values = value } } **Input JSON**: .. code-block:: json { "extract_keys.data": [ {"Left": "sample1", "Right": 100}, {"Left": "sample2", "Right": 200} ] } **Note**: WDL 1.1+ has ``keys(map)`` and ``values(map)`` functions, but JAWS currently supports WDL 1.0 only. Advanced Patterns ----------------- .. dropdown:: How do I handle optional outputs in scatter-gather? :ref:`🔗 ` :color: primary :icon: git-merge :name: faq-how-do-i-handle-optional-outputs-in-scatter-gather **Problem**: Scattered tasks may conditionally produce outputs, creating ``Array[File?]``. **Solution**: Use ``select_all()`` and ``flatten()``: .. code-block:: text version 1.0 workflow optional_scatter { input { Array[File] samples Boolean run_qc = false } scatter (sample in samples) { if (run_qc) { call quality_check { input: sample_file = sample } } } # quality_check.report is Array[File?] # Remove undefined elements Array[File] valid_reports = select_all(quality_check.report) call summarize { input: reports = valid_reports } output { File summary = summarize.output_file Int num_reports = length(valid_reports) } } **Complex nested example**: .. code-block:: text # Nested scatter with conditionals scatter (batch in batches) { scatter (sample in batch.samples) { if (sample.needs_processing) { call process { input: s = sample } } } } # process.output is Array[Array[File?]] # Flatten and remove undefined Array[File] all_outputs = flatten(select_all(process.output)) 📖 `WDL Array Functions `_ .. dropdown:: Why does ``jaws validate`` fail with string concatenation in runtime? :ref:`🔗 ` :color: warning :icon: alert :name: faq-why-does-jaws-validate-fail-with-string **Problem**: ``jaws validate`` (using miniwdl) is stricter than ``womtool``: .. code-block:: text task example { input { String? tag = "latest" } runtime { docker: "ubuntu:" + tag # ❌ Fails validation } } **Error**: .. code-block:: text Cannot add/concatenate String and String? **Solutions**: 1. **Remove optional** (if always provided): .. code-block:: text input { String tag = "latest" # Not optional } runtime { docker: "ubuntu:" + tag # ✅ Works } 2. **Use select_first()** (with fallback): .. code-block:: text input { String? tag } runtime { docker: "ubuntu:" + select_first([tag, "latest"]) # ✅ Works } 3. **Use sub()** for complex string building: .. code-block:: text runtime { docker: sub("ubuntu:TAG", "TAG", select_first([tag, "latest"])) } **Why it matters**: - ``jaws submit`` uses ``womtool`` (permissive) - ``jaws validate`` uses ``miniwdl`` (strict WDL 1.0 compliance) - Use ``jaws validate`` before submission to catch issues early Cromwell and Execution ====================== Call Caching ------------ .. dropdown:: Does Cromwell support checkpointing? :ref:`🔗 ` :color: primary :icon: sync :name: faq-does-cromwell-support-checkpointing **Answer**: Cromwell uses **call caching** instead of traditional checkpointing. **How it works**: When a task completes successfully, Cromwell stores: 1. Command template hash 2. Input file content hashes (via MD5) 3. Docker image digest 4. Runtime attributes If you rerun with identical inputs, Cromwell reuses previous outputs. **Enable/disable caching**: .. code-block:: bash # Default: caching enabled jaws submit workflow.wdl inputs.json # Disable caching for one run jaws submit workflow.wdl inputs.json --no-cache **Check which tasks were served from cache**: .. code-block:: bash jaws tasks The output table has a ``CACHED`` column (``True`` / ``False``) for each task, that's the authoritative view of what hit cache and what executed fresh. 📖 `Cromwell Call Caching `_ .. dropdown:: Why didn't call caching work? :ref:`🔗 ` :color: warning :icon: x-circle :name: faq-why-didn-t-call-caching-work Call caching fails if **any** of these change: 1. **WDL file contents** (even whitespace/comments) 2. **Input values** (variable names or values in inputs.json) 3. **Input file contents** (different MD5 hash) 4. **Docker image** (different digest) 5. **Runtime attributes** (memory, CPU, disk) 6. **Task outputs** (number or type of outputs) **Common mistakes**: ❌ Using String paths instead of File: .. code-block:: text # Bad: cache won't work if path string changes input { String bam_path = "/path/to/file.bam" } ✅ Use File type: .. code-block:: text # Good: caching works based on file content hash input { File bam_file = "/path/to/file.bam" } ❌ Dynamic runtime attributes: .. code-block:: text runtime { memory: if (large_input) then "32GB" else "16GB" } Each value creates a different cache key. **Debug caching issues**: Use ``jaws tasks `` to see the ``CACHED`` column per task. For deeper inspection of *why* a task didn't cache, JAWS writes a ``metadata.json`` supplementary file to the run's output directory (find the path with ``jaws status `` → ``output_dir``); Cromwell records the cache decision and the contributing hashes there. .. code-block:: bash jaws tasks jaws status # find output_dir cat /metadata.json | jq '.calls[] | .[].callCaching' .. dropdown:: How do I handle HTTP URLs as inputs? :ref:`🔗 ` :color: primary :icon: link :name: faq-how-do-i-handle-http-urls-as-inputs **Problem**: Cromwell downloads HTTP files but strips extensions, breaking tools that rely on filenames. **Solution**: Use ``Pair[File, String]`` to preserve original name: .. code-block:: text version 1.0 workflow http_input_example { input { Pair[File, String] remote_file } call process { input: file_pair = remote_file } } task process { input { Pair[File, String] file_pair } command <<< # Copy to preserve extension cp ~{file_pair.left} ~{file_pair.right} # Now tool can read correct filename analyze.sh ~{file_pair.right} >>> output { File result = "result.txt" } } **Input JSON**: .. code-block:: json { "http_input_example.remote_file": { "Left": "https://data.org/sample.fastq.gz", "Right": "sample.fastq.gz" } } 📖 `Cromwell HTTP Filesystems `_ .. dropdown:: Does using a private registry image (e.g. ``library.jgi.doe.gov``) prevent call caching? :ref:`🔗 ` :color: warning :icon: alert :name: faq-does-using-a-private-registry-image-e-g-library **Yes, call caching will not work for images pulled from a private registry.** Cromwell can't perform the SHA256 digest lookup against a private registry it doesn't have read credentials for, and call caching depends on that lookup to decide whether the container has changed. **Why this matters in practice.** Cromwell uses the image digest (SHA256) as part of the cache key. When you reference an image by tag (e.g. ``mytool:latest``), Cromwell resolves the tag to a digest at submission time. If it can't reach the registry to do that resolution, as is the case with ``library.jgi.doe.gov`` and other private registries, it can't compute the cache key, so every run starts from scratch even when nothing about your inputs or WDL has changed. A new push to the same tag (different digest) would also miss the cache, but for private-registry images you don't even get to that point, the lookup itself fails. **What to do.** - For workflows where call caching matters, use images from Docker Hub, Quay.io, or another public registry. Reference by ``image@sha256:`` for the strongest guarantee. - For private-registry images, expect every run to execute fully. That's a fine trade-off when the image content is sensitive; it's a problem when the image is large and the workflow is repeated. - For images on Docker Hub or other public registries, call-caching works as normal, see "Why didn't call caching work?" above for the other things that can break it. .. dropdown:: Will I get cache hits if I resubmit the same workflow on a different site? :ref:`🔗 ` :color: danger :icon: x-circle :name: faq-will-i-get-cache-hits-if-i-resubmit-the-same **No.** Each JAWS site runs its own Cromwell instance, and Cromwell has no API for importing cache results between instances. A run on DORI followed by an identical run on Perlmutter will start from scratch, even though the WDL and inputs are byte-identical. WDL Version Support ------------------- .. dropdown:: Can I use WDL 1.1 with JAWS? :ref:`🔗 ` :color: danger :icon: stop :name: faq-can-i-use-wdl-1-1-with-jaws **Not yet**, JAWS currently supports **WDL 1.0 only**. The Cromwell version JAWS runs today doesn't parse WDL 1.1 features reliably, so workflows declared with ``version 1.1`` will fail validation (often with confusing errors that don't mention the version mismatch). Use ``version 1.0`` on the first line of every WDL file. **A Cromwell upgrade is on the roadmap.** Recent Cromwell releases do support WDL 1.1, and we plan to upgrade the JAWS-bundled Cromwell soon. Until Cromwell is upgraded and fully tested, please continue to use WDL 1.0 only. **Check your version**: .. code-block:: text version 1.0 # ✅ Supported # version 1.1 # ❌ Not supported **WDL 1.1 features NOT available**: - ``keys()`` and ``values()`` Map functions - ``None`` type - Directory type - Enhanced struct features **Workarounds**: - Use Pairs instead of Maps for key extraction - Use ``File`` type for directories (pass as paths) - Use ``struct`` with optional fields instead of None 📖 `WDL 1.0 Spec `_ JAWS-Specific Issues ==================== Container Configuration ----------------------- .. dropdown:: Will my container's ENTRYPOINT script run? :ref:`🔗 ` :color: warning :icon: container :name: faq-will-my-container-s-entrypoint-script-run **No**, JAWS/Cromwell does not execute container ENTRYPOINT scripts. **How Cromwell runs containers**: .. code-block:: bash # Your Dockerfile ENTRYPOINT is ignored apptainer run /cromwell-executions/.../script # Cromwell creates script from WDL command block # Any ENTRYPOINT is bypassed **Solution**: Move logic from ENTRYPOINT to command block: ❌ Bad (ENTRYPOINT won't run): .. code-block:: dockerfile FROM ubuntu:22.04 COPY setup.sh / ENTRYPOINT ["/setup.sh"] ✅ Good (explicit in WDL): .. code-block:: text command <<< # Run setup explicitly /setup.sh # Then your actual command analyze.sh input.txt >>> .. dropdown:: How do I fix timezone offset warnings? :ref:`🔗 ` :color: info :icon: clock :name: faq-how-do-i-fix-timezone-offset-warnings **Warning message**: .. code-block:: text Timezone offset does not match system offset: 0 != -25200 **Solution**: Set ``TZ`` environment variable: .. code-block:: bash # Add to ~/.bashrc export TZ="America/Los_Angeles" # Reload source ~/.bashrc **For JAWS container users**: .. code-block:: bash export TZ="America/Los_Angeles" apptainer run docker://doejgi/jaws-client:latest jaws queue Output Management ----------------- .. dropdown:: Why doesn't JAWS copy the entire Cromwell execution? :ref:`🔗 ` :color: info :icon: file-directory :name: faq-why-doesn-t-jaws-copy-the-entire-cromwell **Reason**: To avoid transferring terabytes of intermediate files. **What JAWS transfers**: ✅ **For successful runs**: - Files declared in workflow ``output`` section - Final outputs only (not intermediate files) ✅ **For failed runs**: - ``error.json`` with failure details - Full execution directory if you run ``jaws download `` **How to preserve additional files**: .. code-block:: text task analysis { command <<< analyze.sh > output.txt 2> stderr.log >>> output { File result = "output.txt" File stderr_log = "stderr.log" # ✅ Explicitly declared File stdout_log = stdout() # ✅ Special function } } .. dropdown:: How do I access stderr and stdout logs? :ref:`🔗 ` :color: info :icon: note :name: faq-how-do-i-access-stderr-and-stdout-logs **For failed runs**: .. code-block:: bash # Download full execution directory jaws download # error.json includes embedded logs cat error.json | jq '.failures[0].message' **For successful runs**: Declare logs as outputs: .. code-block:: text task example { command <<< ./pipeline.sh > stdout.log 2> stderr.log >>> output { File stdout = "stdout.log" File stderr = "stderr.log" } } **Use WDL functions**: .. code-block:: text output { String stdout_text = read_string(stdout()) String stderr_text = read_string(stderr()) } Run Lifecycle and Monitoring ============================ These cover the recurring "is something wrong?" questions. The full lifecycle state diagram is in :doc:`../jaws/jaws_usage` (under "Understanding the Stages") and :doc:`../jaws/jaws_troubleshooting`. .. dropdown:: My ``jaws history`` returns ``[]`` even though I've run jobs :ref:`🔗 ` :color: info :icon: clock :name: faq-my-jaws-history-returns-even-though-i-ve-run-jobs Default lookback is short (1 day). Pass ``--days``: .. code-block:: text jaws history --days 30 # last 30 days, all sites jaws history --days 30 --site dori # last 30 days, only Dori jaws history --days 30 --result failed # last 30 days, only failed runs Full flag reference is in :doc:`../jaws/jaws_usage`. .. dropdown:: What's the difference between ``workflow_root`` and ``output_dir``? :ref:`🔗 ` :color: info :icon: file-directory :name: faq-what-s-the-difference-between-workflow-root-and Both appear in ``jaws status`` output and confuse a lot of users: - **``workflow_root``** is the *Cromwell execution* directory: ``cromwell-executions///``. It contains every intermediate file, log, and task working directory. It's auto-purged on a fixed schedule (currently ~10 days on Dori). - **``output_dir``** is the *JAWS Teams* directory where your *final* declared outputs are copied: ``////``. It is **not** auto-purged, your team manages its lifecycle. When you need an intermediate file, look in ``workflow_root``. When you need a final output, look in ``output_dir``. .. dropdown:: How is my output storage managed? Will my files be deleted? :ref:`🔗 ` :color: warning :icon: archive :name: faq-how-is-my-output-storage-managed-will-my-files-be Two different things happen, with two different timelines: 1. **``workflow_root`` (``cromwell-executions/``)**, the full Cromwell execution directory, is **auto-purged after ~10 days on DORI**. This is JAWS-managed; you don't need to clean it. 2. **``output_dir`` (JAWS Teams directory)**, where the final declared outputs land, is **not auto-purged**. Each team is responsible for managing its space. The recommendation for DORI users: periodically review your team's output directory and remove anything you don't need. .. dropdown:: A compute site went into "degraded" state while my run was active. Are my outputs safe? :ref:`🔗 ` :color: danger :icon: alert :name: faq-a-compute-site-went-into-degraded-state-while-my **Not necessarily, and JAWS won't warn you.** When a compute site (Perlmutter, Tahoma, Dori, etc.) enters a *degraded* state, the underlying filesystem or compute nodes can corrupt files that are being read or written at that moment. Neither JAWS nor Cromwell detects this kind of silent, mid-run corruption, the run can still finish with a normal "succeeded" status and outputs that look complete but are actually wrong. **How to know a site is degraded**: - The site's own status page is the source of truth (for example, `NERSC Live Status `_, or the #dori channel status announcements). - The JAWS team posts ``@channel`` messages in ``#jaws`` when a site they're aware of enters a degraded state. **What to do if your run overlapped with a degraded window**: 1. **Spot-check the outputs** that the run produced during the degraded window. Compare file sizes, line counts, or content hashes against a known-good prior run if you have one. 2. **Look at the run timeline** with ``jaws log `` and cross-reference against the site's status timeline. 3. **When in doubt, resubmit** with ``jaws submit--no-cache``, to trigger the workflow to run fresh and produce new outputs. We've had previous user reports of incorrect outputs from Perlmutter runs that finished during degraded states, so this is worth checking proactively, not just when something obviously looks off. .. dropdown:: How do I clean up old runs on demand? :ref:`🔗 ` :color: info :icon: trash :name: faq-how-do-i-clean-up-old-runs-on-demand .. code-block:: text jaws purge This removes the ``cromwell-executions/`` directory for a specific run early (before the auto-purge would). Your final outputs in ``output_dir`` stay where they are. .. dropdown:: How do I cancel everything I have queued or running? :ref:`🔗 ` :color: info :icon: stop :name: faq-how-do-i-cancel-everything-i-have-queued-or .. code-block:: text jaws cancel-all # cancel all your active runs jaws cancel # cancel one specific run ``cancel-all`` is intended for "I made a mistake, kill everything." See :doc:`../jaws/jaws_usage` for the full command list. Troubleshooting =============== Common Errors ------------- .. dropdown:: My job failed with "OutOfMemory" :ref:`🔗 ` :color: danger :icon: flame :name: faq-my-job-failed-with-outofmemory **Cause**: Task exceeded requested memory. **Solution**: Increase memory allocation: .. code-block:: text runtime { memory: "64 GB" # Was 32 GB cpu: 8 } .. dropdown:: Cannot find rc file after failure :ref:`🔗 ` :color: warning :icon: x-circle :name: faq-cannot-find-rc-file-after-failure **Error message**: .. code-block:: text Unable to determine that job is alive, and .../rc does not exist **Cause**: HTCondor lost connection to compute node before task completed. **Solution**: .. code-block:: bash # Simply resubmit jaws resubmit This is a transient scheduler issue, not a problem with your workflow. The task usually succeeds on retry. **If it persists**: Contact JAWS support with run ID. .. dropdown:: ``500 Internal Server Error`` / ``408 Request Timeout`` / ``502`` from ``jaws`` commands :ref:`🔗 ` :color: warning :icon: alert :name: faq-500-internal-server-error-408-request-timeout-502 These are *usually* transient and *usually* happen in two situations: - **Right after a JAWS deploy** (the team announces these in ``#jaws``). Services need a few minutes to come back up. Wait 5–15 minutes and retry. - **During an external incident** (Cloudflare hiccup, NERSC maintenance, AMQP/RabbitMQ flap). ``jaws status`` returning ``502``, or ``jaws tasks`` returning ``Connection dead, no heartbeat or data received in >= 60s``, both fall into this bucket. Your runs are still progressing at the site level even when central is briefly unreachable; once the API is back, your state catches up. If you see the same error for more than ~15 minutes and there's no announcement in ``#jaws``, ping the team with your run ID. .. dropdown:: My run says ``submission failed`` and ``jaws task`` returns ``None... cromwell_run_id`` :ref:`🔗 ` :color: warning :icon: alert :name: faq-my-run-says-submission-failed-and-jaws-task The run never made it to Cromwell. Run ``jaws log `` to see where the lifecycle stopped. Examples: .. code-block:: text jaws log #STATUS_FROM STATUS_TO TIMESTAMP COMMENT created upload queued ... upload queued upload complete ... upload complete submission failed ... Server timeout: The service is unable to respond at this time Common causes: - **Brief central/site outage** during submission. The run row exists but never got submitted to Cromwell. The fix is to ``jaws resubmit `` (or re-``jaws submit``) once the team confirms services are back. The ``status`` field will read ``done`` even for these, JAWS always terminates in ``done``. Check the ``result`` field: ``null`` means the run never reached Cromwell. Data Access Issues ------------------ .. dropdown:: How do I access Cromwell execution on a remote site? :ref:`🔗 ` :color: info :icon: server :name: faq-how-do-i-access-cromwell-execution-on-a-remote **Scenario**: You submitted from Perlmutter to Tahoma but don't have Tahoma access. **For successful runs**: .. code-block:: bash # Outputs automatically transferred back to Perlmutter jaws status | jq -r '.result.output_dir' **For failed runs**: .. code-block:: bash # Transfer failed task execution jaws download # Check error summary cat error.json | jq '.failures' .. dropdown:: Why are special characters in filenames causing failures? :ref:`🔗 ` :color: danger :icon: stop :name: faq-why-are-special-characters-in-filenames-causing **Problem**: Files with ``'`` (apostrophe), ``;`` (semicolon), or ``\`` fail. **Cause**: Cromwell doesn't properly escape special characters in bash commands. **Solution**: **Avoid special characters** in input filenames. ❌ Bad: .. code-block:: text sample's_data.fastq file;with;semicolons.txt ✅ Good: .. code-block:: text sample_data.fastq file_with_underscores.txt **If you must use existing files**: Rename them first: .. code-block:: text command <<< # Sanitize filename SAFE_NAME=$(echo "~{input_file}" | tr "';\\\"" "_") cp ~{input_file} "$SAFE_NAME" process.sh "$SAFE_NAME" >>> Known Limitations ================= .. dropdown:: Outputs outside cromwell-executions/execution/ directory :ref:`🔗 ` :color: warning :icon: file-directory :name: faq-outputs-outside-cromwell-executions-execution **Problem**: JAWS only transfers files from ``cromwell-executions/.../execution/``. ❌ **Won't be transferred**: .. code-block:: text command <<< echo "data" > /tmp/output.txt >>> output { File result = "/tmp/output.txt" # ❌ Outside execution dir } ✅ **Will be transferred**: .. code-block:: text command <<< echo "data" > output.txt # ✅ In execution dir >>> output { File result = "output.txt" } **Workaround for existing files**: .. code-block:: text command <<< # Process creates output in /tmp process.sh --output /tmp/result.txt # Copy to execution directory cp /tmp/result.txt ./result.txt >>> output { File result = "result.txt" } Additional Resources ==================== External Documentation ---------------------- .. grid:: 3 .. grid-item-card:: 📖 WDL 1.0 Specification :link: https://github.com/openwdl/wdl/blob/legacy/versions/1.0/SPEC.md Official OpenWDL language spec .. grid-item-card:: 🚀 Dockstore WDL Guide :link: https://docs.dockstore.org/en/stable/getting-started/getting-started-with-wdl.html Beginner-friendly WDL tutorial .. grid-item-card:: 🔧 Cromwell Documentation :link: https://cromwell.readthedocs.io/ Workflow engine reference JAWS Documentation ------------------ .. seealso:: **Related Guides** - :doc:`../jaws/jaws_quickstart` - Get started with JAWS - :doc:`../jaws/jaws_troubleshooting` - Comprehensive debugging guide - :doc:`../Tutorials/writing_wdls` - Step-by-step WDL development - :doc:`../Resources/best_practices` - WDL coding standards and optimization - :doc:`../Resources/compute_resources` - HPC resource allocation Need More Help? =============== If you didn't find your answer: .. button-link:: help.html :color: primary :expand: 💬 Contact JAWS Support .. button-link:: https://joint-genome-inst.slack.com/archives/jaws :color: secondary :expand: #jaws Slack Channel