JAWS Commands

Warning

Before these commands will work, you need to set up everything: See Quickstart Example for full setup instructions.

Example submitting a workflow:

jaws submit <WDL> [inputs json] <SITE>

Usage: jaws [OPTIONS] COMMAND [ARGS]…

Run a command with --help to see it’s options.

  jaws --help

  Usage: jaws [OPTIONS] COMMAND [ARGS]...
  JGI Analysis Workflows Service

  Options:
    --config TEXT     JAWS config file
    --user TEXT       User config file
    --log TEXT        Log file
    --log-level TEXT  Logging level [debug|info|warning|error|critical]
    -h, --help        Show this message and exit.

Commands:
  * admin        Administrator Commands
  cancel       Cancel a running workflow; prints whether aborting was...
  cancel-all   Cancel all active workflows for the user.
  download     Force download of failed tasks in a run.
  get-user     Get current user's settings.
  health       Display the current system status and health.
  * history      List past workflow runs submitted by the user.
  info         JAWS version and info.
  inputs       Generate inputs template (JSON) from workflow (WDL) file.
  list-sites   List of all available compute sites for workflow execution.
  * log          View the state transition log for a specific workflow.
  * purge        Remove the cromwell-execution directory for a run.
  * queue        List of the user's currently running workflows.
  resubmit     Resubmit a run (at same compute-site).
  * status       Check the current status of a specific workflow run.
  * submit       Submit a run for execution
  * tasks        Get detailed information about workflow tasks.
  * teams        Manage user teams and permissions.
  * update-user  Update current user's settings.
  validate     Validate a WDL using miniwdl.

The commands with an asterik have additional options. Expand the below commands to see them.

Examples

See status of the JAWS services

Not all sites are represented here yet.

jaws health
{
    "Central": "UP",
    "RabbitMQ": "UP",
    "defiant-Site": "UP",
    "dori-Site": "UP",
    "jgi-Site": "UP",
    "nmdc-Site": "UP",
    "nmdc_tahoma-Site": "UP",
    "perlmutter-Site": "UP",
    "tahoma-Site": "UP"
}

To run a wdl

To run a workflow, you will need two files: a WDL file (.wdl) and an inputs (.json) file. For the latter, you can generate one from scratch by running jaws inputs <WDL>, which will print out a template based on the WDL, you just need to fill out the values.

A simple example template would look like:

{
  fq_count.fastq_file: File
}

To submit a run

jaws submit my.wdl my.json dori

# output looks like
{
  run_id: 7235,
}

Include a tag for your run to help keep track of things. jaws status will display the tag.

jaws submit --tag 'some useful info' my.wdl my.json dori

Run with Cromwell’s call-caching off. Call-caching will allow you to re-run JAWS submissions without re-running tasks that completed successfully. However, this is not always desirable and you can turn caching off, as follows:

jaws submit --no-cache my.wdl my.json dori

Monitor your Run

The examples are in order of verbosity.

# overview of the whole WDL
jaws status 7235

or

# the times each stage started, e.g. queued, running, etc.
jaws log 7235

or

# the times each task entered each stage
jaws tasks 7235

Understanding the Stages

These are the possible states, in order, that a JAWS run passes through.

created:            The Run was accepted and a run_id assigned.
upload queued:      The Run's input files are waiting to be transferred to the compute-site.
uploading:          Your Run's input files are being transferred to the compute-site.
upload failed:      The transfer of your run to the compute-site failed.
upload inactive:    Globus transfer stalled.
upload complete:    Your Run's input files have been transferred to the compute-site successfully.
ready:              The Run has been transferred to the compute-site.
submitted:          The run has been submitted to Cromwell and tasks should start to queue within moments.
submission failed:  The run was submitted to Cromwell but rejected due to invalid input.
queued:             At least one task has requested resources but no tasks have started running yet.
running:            The run is being executed; you can check `tasks` for more detail.
succeeded:          The run has completed successfully.
failed:             The run has failed and will not be transferred unless you run the 'download' command.
complete:           Supplementary files have been written to the run's output dir.
finished:           The task-summary has been published to the performance metrics service.
cancel:             Your run is in the process of being canceled.
cancelled:          The run was cancelled by either the user or an admin.
download queued:    Your Run's output files are waiting to be transferred from the compute-site.
downloading:        The Run's output files are being transferred from the compute-site.
download failed:    The Run's output files could not be transferred from the compute-site.
download inactive:  Globus transfer stalled.
download complete:  Your Run's output (succeeded or failed) have been transferred to the team outdir.
download skipped:   The run was not successful so the results were not downloaded.
fix perms queued:   The fix permissions request has been queued.
fix perms complete: The permissions of the outputs have been changed.
fix perms failed:   The permissions of the outputs could not be changed.
sync complete:      Inode metadata sync delay complete.
slack succeeded:    Notification of run completion was sent via slack.
slack failed:       Notification of run completion could not be sent via slack.
post succeeded:     Notification of run completion was POSTed to webhook URL.
post failed:        Notification of run completion could not be POSTed to webhook URL.
done:               The run is complete.

Get current or old history of jobs owned by you

# get list of your currently running jobs
jaws queue

# view history of your jobs for last 7 days on site Dori
jaws history --days=7 --site=dori

Debugging

The errors.json file is a catch-all command for viewing errors. It is created by JAWS after a run has completed and it will be transfer to the teams output directory.

The errors.json file should be capturing errors from:

cromwell

the WDL tasks

HTCondor backend

Slurm

Sometimes there is additional information in the files created by cromwell:

script.submit - contains the commands used by cromwell to run the script file.
script        - the commands representing cromwell boilerplate stuff in addition to your commands from the wdl-task.
stderr        - the stderr from running script.
stdout        - the stdout from running script.
stderr.submit - the stderr from submission script.
stdout.submit - the stdout from submission script.

These files will be transferred to the team’s output directory solely for tasks that have failed, for the purpose of debugging.

Getting your output

The final outputs from the workflow will be moved to your team’s directory. If a run fails, the resulting final outputs will still be transferred. Furthermore, for any tasks that fail, the complete Cromwell execution folder for that task will be copied over.

jaws status <RUN_ID> display the output_dir for a specific run.

Specialty Commands

This command uses the miniwdl as a linter for your WDLs. You would use this when developing a WDL.

jaws validate my.wdl

The --user flag allows someone to use a different jaws token than the default. This way, you can have a token representing a user like ‘rqc’ or ‘jaws-admin’ with certain permissions. Then multiple people from a group can use this token to have access to certain files.

jaws --user <~/jaws.conf> <command>