JAWS Commands
Warning
Before these commands will work, you need to set up everything: See Quickstart Example for full setup instructions.
Example submitting a workflow:
jaws submit <WDL> [inputs json] <SITE>
Usage: jaws [OPTIONS] COMMAND [ARGS]…
Run a command with --help
to see it’s options.
jaws --help
Usage: jaws [OPTIONS] COMMAND [ARGS]...
JGI Analysis Workflows Service
Options:
--config TEXT JAWS config file
--user TEXT User config file
--log TEXT Log file
--log-level TEXT Logging level [debug|info|warning|error|critical]
-h, --help Show this message and exit.
Commands:
* admin Administrator Commands
cancel Cancel a running workflow; prints whether aborting was...
cancel-all Cancel all active workflows for the user.
download Force download of failed tasks in a run.
get-user Get current user's settings.
health Display the current system status and health.
* history List past workflow runs submitted by the user.
info JAWS version and info.
inputs Generate inputs template (JSON) from workflow (WDL) file.
list-sites List of all available compute sites for workflow execution.
* log View the state transition log for a specific workflow.
* purge Remove the cromwell-execution directory for a run.
* queue List of the user's currently running workflows.
resubmit Resubmit a run (at same compute-site).
* status Check the current status of a specific workflow run.
* submit Submit a run for execution
* tasks Get detailed information about workflow tasks.
* teams Manage user teams and permissions.
* update-user Update current user's settings.
validate Validate a WDL using miniwdl.
The commands with an asterik have additional options. Expand the below commands to see them.
admin
Usage: jaws admin [OPTIONS] COMMAND [ARGS]...
Administrator Commands
Options:
-h, --help Show this message and exit.
Commands:
add-team Create a new user team.
chown-team Change the team's owner.
create-user Create new user and get JAWS OAuth access token.
deactivate-user Mark user inactive.
del-team Delete a user team
disable-sites Turn off site(s) ex) jaws disable-sites cori jaws...
enable-sites Turn on site(s) ex) jaws enable-sites cori jaws...
queue-wait List compute Sites' estimated queue-wait times (in...
update-token Update user's token.
history
Usage: jaws history [OPTIONS]
Print a list of the user's past runs.
Options:
--days INTEGER history going back this many days; default=1
--site TEXT limit results to this compute-site; default=all
--result TEXT limit results to this result; default=any
--tag TEXT List runs from single tag; default=False
--wdl TEXT Filter runs by wdl file name; default=any
--json-file TEXT Filter runs by json file name; default=any
--user TEXT Filter runs by user; default=any
--verbose Return more fields; default=False
--all List runs from all users; default=False
-h, --help Show this message and exit.
log
Usage: jaws log [OPTIONS] RUN_ID
View the log of Run state transitions for the workflow as a whole.
Options:
--fmt TEXT the desired output format: [text|json|tab].
-h, --help Show this message and exit.
purge
Usage: jaws purge [OPTIONS] RUN_ID
remove cromwell-execution directory for a specific run
Options:
-h, --help Show this message and exit.
queue
Usage: jaws queue [OPTIONS]
List of user's current runs
Options:
--site TEXT limit results to this compute-site; default=all
--all-users, --all List runs from all users; default=False
--fmt TEXT the desired output format: [text|json|tab]
-h, --help Show this message and exit.
status
Usage: jaws status [OPTIONS] RUN_ID
Check the current status of a specific workflow run.
Options:
--verbose Returns ids that are useful for debugging if you have admin privileges
-h, --help Show this message and exit.
submit
Usage: jaws submit [OPTIONS] WDL_FILE [INPUTS]... SITE
Submit a run for execution.
Options:
--tag TEXT identifier for the run.
--no-cache Disable call-caching for this run.
--quiet Don't print copy progress bar.
--sub TEXT Subworkflows zip (optional; by default, auto-generate).
--team TEXT User-team to associate with this run.
--webhook TEXT Deprecated. Please use `jaws status` or `jaws log` to retrieve updates. If provided, JAWS will POST to this URL when Run completes.
--forcequeue Submit Run to an unavailable site.
--overwrite-inputs Overwrite input files that already exist in JAWS cache.
-h, --help Show this message and exit.
tasks
Usage: jaws tasks [OPTIONS] RUN_ID
Get Task logs and current status
Options:
--fmt TEXT the desired output format: [txt|json|tab].
-h, --help Show this message and exit.
teams
Usage: jaws teams [OPTIONS] COMMAND [ARGS]...
Teams Commands
Options:
-h, --help Show this message and exit.
Commands:
add-site Grant team access to a site (restricted)
add-user Add a user to a team
get-site-path Get a team's site config
list List all teams
list-sites List a team's sites.
members List the users associated with a team.
my-teams List the teams to which you belong
remove-site Remove team access to a site (restricted)
remove-user Remove a user from a team
set-site-path Configure a team's folder at a jaws-site (restricted)
update-user
Usage: jaws update-user [OPTIONS]
Update existing user in JAWS
Options:
--email TEXT Your email address.
--name TEXT Your full name.
--slack_id TEXT Your slack Member ID.
--slack_webhook TEXT Your slack webhook.
-h, --help Show this message and exit.
Examples
See status of the JAWS services
Not all sites are represented here yet.
jaws health
{
"Central": "UP",
"RabbitMQ": "UP",
"defiant-Site": "UP",
"dori-Site": "UP",
"jgi-Site": "UP",
"nmdc-Site": "UP",
"nmdc_tahoma-Site": "UP",
"perlmutter-Site": "UP",
"tahoma-Site": "UP"
}
To run a wdl
To run a workflow, you will need two files: a WDL file (.wdl) and an inputs (.json) file.
For the latter, you can generate one from scratch by running jaws inputs <WDL>
, which will print out a template based on the WDL, you just need to fill out the values.
A simple example template would look like:
{
fq_count.fastq_file: File
}
To submit a run
jaws submit my.wdl my.json dori
# output looks like
{
run_id: 7235,
}
Include a tag for your run to help keep track of things. jaws status
will display the tag.
jaws submit --tag 'some useful info' my.wdl my.json dori
Run with Cromwell’s call-caching off. Call-caching will allow you to re-run JAWS submissions without re-running tasks that completed successfully. However, this is not always desirable and you can turn caching off, as follows:
jaws submit --no-cache my.wdl my.json dori
Monitor your Run
The examples are in order of verbosity.
# overview of the whole WDL
jaws status 7235
or
# the times each stage started, e.g. queued, running, etc.
jaws log 7235
or
# the times each task entered each stage
jaws tasks 7235
Understanding the Stages
These are the possible states, in order, that a JAWS run passes through.
created: The Run was accepted and a run_id assigned.
upload queued: The Run's input files are waiting to be transferred to the compute-site.
uploading: Your Run's input files are being transferred to the compute-site.
upload failed: The transfer of your run to the compute-site failed.
upload inactive: Globus transfer stalled.
upload complete: Your Run's input files have been transferred to the compute-site successfully.
ready: The Run has been transferred to the compute-site.
submitted: The run has been submitted to Cromwell and tasks should start to queue within moments.
submission failed: The run was submitted to Cromwell but rejected due to invalid input.
queued: At least one task has requested resources but no tasks have started running yet.
running: The run is being executed; you can check `tasks` for more detail.
succeeded: The run has completed successfully.
failed: The run has failed and will not be transferred unless you run the 'download' command.
complete: Supplementary files have been written to the run's output dir.
finished: The task-summary has been published to the performance metrics service.
cancel: Your run is in the process of being canceled.
cancelled: The run was cancelled by either the user or an admin.
download queued: Your Run's output files are waiting to be transferred from the compute-site.
downloading: The Run's output files are being transferred from the compute-site.
download failed: The Run's output files could not be transferred from the compute-site.
download inactive: Globus transfer stalled.
download complete: Your Run's output (succeeded or failed) have been transferred to the team outdir.
download skipped: The run was not successful so the results were not downloaded.
fix perms queued: The fix permissions request has been queued.
fix perms complete: The permissions of the outputs have been changed.
fix perms failed: The permissions of the outputs could not be changed.
sync complete: Inode metadata sync delay complete.
slack succeeded: Notification of run completion was sent via slack.
slack failed: Notification of run completion could not be sent via slack.
post succeeded: Notification of run completion was POSTed to webhook URL.
post failed: Notification of run completion could not be POSTed to webhook URL.
done: The run is complete.
Get current or old history of jobs owned by you
# get list of your currently running jobs
jaws queue
# view history of your jobs for last 7 days on site Dori
jaws history --days=7 --site=dori
Debugging
The errors.json
file is a catch-all command for viewing errors.
It is created by JAWS after a run has completed and it will be transfer to the teams output directory.
The errors.json
file should be capturing errors from:
cromwell
the WDL tasks
HTCondor backend
Slurm
Sometimes there is additional information in the files created by cromwell:
script.submit - contains the commands used by cromwell to run the script file.
script - the commands representing cromwell boilerplate stuff in addition to your commands from the wdl-task.
stderr - the stderr from running script.
stdout - the stdout from running script.
stderr.submit - the stderr from submission script.
stdout.submit - the stdout from submission script.
These files will be transferred to the team’s output directory solely for tasks that have failed, for the purpose of debugging.
Getting your output
The final outputs from the workflow will be moved to your team’s directory. If a run fails, the resulting final outputs will still be transferred. Furthermore, for any tasks that fail, the complete Cromwell execution folder for that task will be copied over.
jaws status <RUN_ID>
display the output_dir
for a specific run.
Specialty Commands
This command uses the miniwdl as a linter for your WDLs. You would use this when developing a WDL.
jaws validate my.wdl
The --user
flag allows someone to use a different jaws token`
than the default.
This way, you can have a token representing a user like ‘rqc’ or ‘jaws-admin’ with certain permissions.
Then multiple people from a group can use this token to have access to certain files.
jaws --user <~/jaws.conf> <command>
JAWS Deprecated Commands
In this section, you’ll find a compilation of commands deprecated in the previous releases. For each command, we have also provided guidance on finding its current alternative. If you want more details about each release, please check the Release notes.
jaws task-log
andjaws task-summary
These two commands have been merged and into a new command,
jaws tasks <RUN_ID>
.For backward compatibility,
jaws task-log
will be reporting the output ofjaws tasks
.
jaws get
As we transition to copying the output files to the JAWS Teams directory, we are deprecating out the
jaws get
command. To ensure backward compatibility, this command will remain functional for a few more cycles, serving solely to copy the final workflow outputs.jaws status <ID>
will provide the path to final path for the output files. Please checkoutput_dir
. Then, you can find the expected file tree structure:/<TEAM PATH>/<RUN_ID>/<Cromwell_ID>
jaws outfiles
,jaws outputs
,jaws metadata
andjaws errors
All these commands were deprecated as part of the effort to refactor
jaws metadata
command (that was using cromwell metadata). As users are submitting large workflows (> 10k tasks), cromwell metadata became too expensive to query and would sometimes timeout. Instead, we now wait for the run to finish and then write some associated reports (e.g. errors, outfiles, and outputs) to disc. These json files are written to the run’s execution directory and transfer to the Team’s directory.How to find the report files:
ls /<TEAM PATH>/<RUN_ID>/<Cromwell_ID> <workflow>.wdl <inputs>.json errors.json metadata.json output_manifest.json outputs.json summary.json tasks.json