=================== JAWS Python Library =================== .. role:: listsize .. role:: textborder .. role:: bash(code) JAWS Python Library Usage Guide =============================== The JAWS Python Library provides a programmatic interface to interact with the JAWS system. This guide provides an overview of the available commands, along with examples to help users interact with JAWS programmatically using Python. Getting Started with the JAWS Python Library -------------------------------------------- - **Requiriments** - Python version: `>=3.11, <3.14` - **Installation** To use the JAWS Python Library, you need to install the JAWS client package. You can install it using pip: .. code-block:: bash pip install jaws-client --index-url https://code.jgi.doe.gov/api/v4/projects/312/packages/pypi/simple - **Initialization** Before using any commands, you need to initialize the JAWS Python Library by configuring your environment. .. code-block:: python # Import the JAWS Python Library and configuration modules from jaws_client import api from jaws_client.config import Configuration # Define the paths to the JAWS configuration and user token files jaws_config_file_path = '/path/to/jaws-prod.conf' user_token_file_path = '/path/to/jaws.conf' # Initialize the Configuration object config = Configuration.from_files(jaws_config_file_path, user_token_file_path) # Initialize the JAWS Python Library jaws = api.JawsApi(config) **JAWS Configuration Paths** .. tab-set:: .. tab-item:: DORI :sync: key1 .. code-block:: text jaws_config_file_path = '/clusterfs/jgi/groups/dsi/homes/svc-jaws/dori-prod/jaws-prod.conf' .. tab-item:: NERSC - Perlmutter :sync: key2 .. code-block:: text jaws_config_file_path = '/global/cfs/cdirs/jaws/perlmutter-prod/jaws-prod.conf' .. tab-item:: NERSC - NMDC :sync: key3 If you are part of `NMDC` project, please use: .. code-block:: text jaws_config_file_path = '/global/cfs/cdirs/m3408/jaws-install/jaws-client/nmdc-prod/jaws-prod.conf' .. tab-item:: NERSC - KBASE :sync: key4 If you are part of `KBase` project, please use: .. code-block:: text jaws_config_file_path = '/global/cfs/cdirs/kbase/jaws/jaws-client/kbase-prod/jaws-prod.conf' .. tab-item:: TAHOMA - NMDC :sync: key5 If you are part of `NMDC` project on Tahoma, please follow: .. code-block:: text jaws_config_file_path = '/tahoma/mscnmdc/jaws-install/jaws-client/nmdc_tahoma-prod/jaws-prod.config' Command Reference ----------------- Below is a list of available commands you can use in your scripts. **Check System Health** +++++++++++++++++++++++ To retrieve the status of the JAWS services, you can use the :bash:`jaws.health()` command. Here's how it works: .. code-block:: python # Retrieve the status of all JAWS services jaws.health() {'Central': 'UP', 'RabbitMQ': 'UP', 'crux-Site': 'DOWN', 'defiant-Site': 'DOWN', 'dori-Site': 'UP', 'jgi-Site': 'UP', 'kbase-Site': 'UP', 'nmdc-Site': 'UP', 'nmdc_tahoma-Site': 'UP', 'perlmutter-Site': 'UP', 'tahoma-Site': 'UP'} # Retrieve the status of specific sites (e.g., "nmdc" and "dori") jaws.health(site="nmdc,dori") {'Central': 'UP', 'RabbitMQ': 'UP', 'dori-Site': 'UP', 'nmdc-Site': 'UP'} - :bash:`jaws.health()`: This retrieves the health status of all JAWS services and sites. The returned dictionary contains the status ('UP' or 'DOWN') for each service or site. - :bash:`jaws.health(site="nmdc,dori")`: This filters the health check to only include the specified sites (`nmdc` and `dori` in this case). **Submit a Workflow** +++++++++++++++++++++ Submit a workflow to be executed at a specific site. Parameters: - `wdl_file`: Path to the WDL file. - `site`: Compute site where the workflow will run. - `inputs`: Path to the JSON file containing input parameters. Optional Parameters: - `team`: The team under which the workflow run is being submitted. - `tag`: A tag to identify the run. - `no_cache`: Disable Cromwell call-caching for this run. - `quiet`: Suppress non-essential output. - `sub`: Path to a ZIP file containing subworkflows. - `overwrite_inputs`: Overwrite inputs in the staging area. .. code-block:: python jaws.submit( wdl_file="workflow.wdl", # Path to the WDL file or to the URL WDL file site="dori", # Site where the workflow will run inputs="inputs.json", # (Optional) Path to the input JSON file team="dsi-aa", # (Optional) Team under which the run is submitted tag="experiment1", # (Optional) Tag to identify the run no_cache=True, # (Optional) Disable caching quiet=False, # (Optional) Suppress non-essential output sub="subworkflows.zip", # (Optional) Subworkflow file overwrite_inputs=False # (Optional) Overwrite inputs in staging ) jaws.submit(wdl_file="align_final.wdl", site="dori", inputs="inputs.json", tag="experiment1", no_cache=True) Using cached copy of sample.fastq.bz2 Using cached copy of sample.fasta {'run_id': 21811} **List Active Runs** ++++++++++++++++++++ Get the list of active workflow runs for the current user. .. code-block:: python jaws.queue(site="dori", all=False) [ { "compute_site_id": "dori", "id": 21811, "result": null, "status": "running", "tag": "experiment1", "team_id": "dsi-aa", "updated": "2024-09-08 16:41:08" } ] **List Past Runs** ++++++++++++++++++ Get the list of past workflow runs submitted by the user. .. code-block:: python jaws.history(days=1, site="dori", result="any", verbose=False, all=False) [ { "compute_site_id": "dori", "cpu_hours": null, "cromwell_run_id": "d33be53a-1905-4214-99f9-9b9f3d2a42a6", "id": 21811, "input_site_id": "dori", "json_file": "/clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/5min_example/inputs.json", "output_dir": null, "result": null, "status": "running", "status_detail": "The run is being executed; you can check `tasks` for more detail", "submitted": "2024-09-08 16:38:54", "tag": "experiment1", "team_id": "dsi-aa", "updated": "2024-09-08 16:41:08", "user_id": "dcassol", "wdl_file": "/clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/5min_example/align_final.wdl", "workflow_name": "bbtools", "workflow_root": "/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/cromwell-executions/bbtools/d33be53a-1905-4214-99f9-9b9f3d2a42a6" } ] **Get Run Status** ++++++++++++++++++ Retrieve the current status of a specific workflow run using its `run_id`. .. code-block:: python jaws.status("21811", verbose=False) { "compute_site_id": "dori", "cpu_hours": 0.0, "cromwell_run_id": "d33be53a-1905-4214-99f9-9b9f3d2a42a6", "id": 21811, "input_site_id": "dori", "json_file": "/clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/5min_example/inputs.json", "output_dir": "/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/dsi-aa/dcassol/21811/d33be53a-1905-4214-99f9-9b9f3d2a42a6", "result": "succeeded", "status": "done", "status_detail": "The run is complete.", "submitted": "2024-09-08 16:38:54", "tag": "experiment1", "team_id": "dsi-aa", "updated": "2024-09-08 16:44:20", "user_id": "dcassol", "wdl_file": "/clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/5min_example/align_final.wdl", "workflow_name": "bbtools", "workflow_root": "/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/cromwell-executions/bbtools/d33be53a-1905-4214-99f9-9b9f3d2a42a6" } **View Run Logs** +++++++++++++++++ Get the log of state transitions for a specific workflow run. .. code-block:: python jaws.log("21811") {'data': [['created', 'upload queued', '2024-09-08 16:38:57', ''], ['upload queued', 'upload complete', '2024-09-08 16:38:57', ''], ['upload complete', 'ready', '2024-09-08 16:39:08', ''], ['ready', 'submitted', '2024-09-08 16:39:16', ''], ['submitted', 'queued', '2024-09-08 16:39:27', ''], ['queued', 'running', '2024-09-08 16:41:08', ''], ['running', 'succeeded', '2024-09-08 16:42:59', ''], ['succeeded', 'complete', '2024-09-08 16:43:09', ''], ['complete', 'finished', '2024-09-08 16:43:19', ''], ['finished', 'download queued', '2024-09-08 16:43:29', ''], ['download queued', 'download complete', '2024-09-08 16:43:39', ''], ['download complete', 'fix perms complete', '2024-09-08 16:43:49', ''], ['fix perms complete', 'sync complete', '2024-09-08 16:44:00', ''], ['sync complete', 'slack succeeded', '2024-09-08 16:44:10', ''], ['slack succeeded', 'done', '2024-09-08 16:44:20', '']], 'header': ['STATUS_FROM', 'STATUS_TO', 'TIMESTAMP', 'COMMENT']} **View Tasks Details** ++++++++++++++++++++++ Retrieve detailed information about individual tasks within a workflow run. .. code-block:: python jaws.tasks("21811") {'data': [['call-alignment', '24135', 'succeeded', '2024-09-08 16:39:21', '2024-09-08 16:41:02', '2024-09-08 16:41:07', 2, 0, False, 'bbtools.alignment', 1, 8, 10, 0.0, 0], ['call-samtools', '24136', 'succeeded', '2024-09-08 16:42:09', '2024-09-08 16:42:21', '2024-09-08 16:42:23', 0, 0, False, 'bbtools.samtools', 1, 4, 20, 0.0, 0]], 'header': ['TASK_DIR', 'JOB_ID', 'STATUS', 'QUEUE_START', 'RUN_START', 'RUN_END', 'QUEUE_MIN', 'RUN_MIN', 'CACHED', 'TASK_NAME', 'REQ_CPU', 'REQ_GB', 'REQ_MIN', 'CPU_HRS', 'RETURN_CODE']} **Cancel Workflow Run** +++++++++++++++++++++++ Cancel a specific workflow run or multiple runs. .. code-block:: python jaws.cancel(run_ids=("21811",)) {'21811': True} **Cancel All Active Runs** ++++++++++++++++++++++++++ Cancel all currently active workflow runs. .. code-block:: python jaws.cancel_all() {'21647': True, '21661': True, '21738': True, '21747': True, '21801': True, '21812': True} **Resubmit a Workflow** +++++++++++++++++++++++ Resubmit a failed workflow run. Please investigate the cause of the failure before resubmitting. If the issue is related to the WDL or inputs, please fix them and use the `jaws.submit` command. To resubmit a failed workflow run caused by system issues, you can use the `jaws.resubmit` command. JAWS will reuse the same WDL and inputs in the JAWS staging area. Example: .. code-block:: python jaws.resubmit(run_id="21811") {'21811': 'resubmitted'} **Download Run Results** ++++++++++++++++++++++++ Get detailed task information for a specific workflow run. .. code-block:: python jaws.download(run_id="21811") {'download_id': 12297, 'id': 21811, 'status': 'download queued'} **List Available Compute Sites** ++++++++++++++++++++++++++++++++ List all available compute sites for submitting workflows. .. code-block:: python jaws.list_sites() { "defiant": { "max_cpu": "128", "max_ram_gb": "256", "max_time": "1425", "status": true }, "dori": { "max_cpu": "64", "max_ram_gb": "1480", "max_time": "4320", "status": true }, "jgi": { "max_cpu": "32", "max_ram_gb": "492", "max_time": "4320", "status": true }, "nmdc": { "max_cpu": "128", "max_ram_gb": "492", "max_time": "2865", "status": true }, "nmdc_tahoma": { "max_cpu": "36", "max_ram_gb": "1480", "max_time": "2865", "status": true }, "perlmutter": { "max_cpu": "128", "max_ram_gb": "492", "max_time": "1440", "status": true }, "tahoma": { "max_cpu": "36", "max_ram_gb": "1480", "max_time": "2865", "status": true } } **Generate Inputs Template** ++++++++++++++++++++++++++++ Generate an input template (JSON) from a WDL file. .. code-block:: python jaws.inputs(wdl_file="align.wdl") '{\n "bbtools.ref": "File",\n "bbtools.reads1": "File"\n}' **Validate WDL and Inputs** +++++++++++++++++++++++++++ Validate a WDL file and its corresponding inputs. .. code-block:: python jaws.validate(shell_check=True, wdl_file="align.wdl", inputs_file="inputs.json") {'result': 'succeeded', 'message': 'align_final.wdl\n workflow bbtools\n call alignment\n call samtools\n task alignment\n task samtools\n', 'error': ''} **Get User Information** ++++++++++++++++++++++++ Retrieve the current user's information. .. code-block:: python jaws.get_user() {'email': 'dcassol@lbl.gov', 'name': 'Daniela Cassol', 'slack_id': 'XXXXXXX', 'teams': ['dsi-aa'], 'uid': 'dcassol'} **Update User Information** +++++++++++++++++++++++++++ Update the current user's information. .. code-block:: python jaws.update_user(data=UserUpdate(slack_id="", slack_webhook="")) **List All Teams** ++++++++++++++++++ List all teams, including those the current user does not belong to. .. code-block:: python jaws.list_teams() [ "gt-ga", "nmdc", "dsi-ii", "bionano", "sc-mcr", "ornl", "workshop", "plant_ha", "plantcompgen", "gaa", "gt-seqtech", "sc-mds", "gt-usa", "micro-scale", "rqc", "dsi-aa", "gt-syn", "phytzm" ] **List User Teams** +++++++++++++++++++ List all teams the current user belongs to. .. code-block:: python jaws.my_teams() ['dsi-aa'] **List Team Members** +++++++++++++++++++++ List all members of a specific team. .. code-block:: python jaws.team_get_users(team_id="dsi-aa") ['dcassol', 'ekirton', 'elais', 'jfroula', 'mmelara', 'ramanik', 'ssul'] **Get JAWS Teams Path** +++++++++++++++++++++++ Get the path to the JAWS Team directory in a specific site. .. code-block:: python jaws.team_get_site("dsi-aa", "dori") '/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/dsi-aa' JAWS CENTRAL REST API Documentation =================================== Discover the detailed definition and specifications of the CENTRAL REST API for the JAWS by clicking `here `_. .. raw:: html