========
Cromwell
========

Installing and Running Cromwell locally
=======================================
.. _install cromwell:

.. role:: bash(code)
   :language: bash


Summary
+++++++

Here we will go over getting `Cromwell <https://cromwell.readthedocs.io/en/stable/>`_ setup on your local machine so you can test & develop WDLs.


Required
++++++++

.. admonition:: Important! 

    You need to have a `java --version` that is compatible with Cromwell or Cromwell will complain. If you get an error that says some component of Cromwell's was compiled with a newer version of java than you have, then update java: `java downloads <https://www.oracle.com/java/technologies/downloads/>`_.


Install Cromwell
++++++++++++++++

The below steps are copied from Cromwell's official github 
`repository <https://github.com/broadinstitute/cromwell/releases>`_:

1. Download the latest cromwell and womtool jar file from the above repository.
2. Copy the files to some place like `~/cromwell/cromwell-<VERSION>.jar` and `~/cromwell/womtool-<VERSION>.jar`.

Test Run Cromwell
+++++++++++++++++

To make sure `cromwell` is working, copy the WDL file example below to a file called :bash:`my.wdl` and then run:

.. code-block:: text

    java -jar ~/cromwell/cromwell-<VERSION>.jar run my.wdl

.. dropdown::  :bash:`my.wdl` Example
    :color: info
    :animate: fade-in
    
    .. code-block:: text 

        workflow myTest {
            call TaskOne
        }

        task TaskOne {
            command {
                echo "hello jackson" > output.txt
            }
            output {
                String out = "output.txt"
            }
        }

The cromwell Config File
========================

.. role:: bash(code)
   :language: bash

You can configure `cromwell` settings in 3 ways.
The default settings are in a file called :bash:`cromwell.conf` file which is packaged with every :bash:`cromwell.jar`.
However, you don't set the variables directly in this file. Instead, when you run the :bash:`cromwell.jar` command, you can:

    1. Point to your own :bash:`cromwell.conf` file.
    2. Include an `options.json` file that has the variables set.
    3. Specify the variables from the command line itself.

.. note::
   Check out the tutorial on `How to Configure Cromwell <https://cromwell.readthedocs.io/en/stable/tutorials/ConfigurationFiles/>`_ for more information.

1. Overriding Variables with a `cromwell.conf` File
+++++++++++++++++++++++++++++++++++++++++++++++++++

.. _cromwell config:

The :bash:`cromwell.conf` file that is packaged with the `jar` is much larger than this example. This example only includes variables you are likely to alter.

.. dropdown:: An example `cromwell.conf` file
    
    :title: jgi-purple text-white font-weight-bold  box
    :body: bg-light
    :animate: fade-in
      
      .. code-block:: text
      
          include required(classpath("application"))
      
          webservice
          {
            port = 50011
          }
      
          workflow-options
          {
            workflow-log-dir: "cromwell-workflow-logs"
            workflow-log-temporary: false
          }
      
          call-caching
          {
            enabled = true
            invalidate-bad-cache-result = true
          }
          docker {
            hash-lookup {
                enabled = true
            }
          }

      
          backend
          {
            default = "Local"
      
            providers
            {
      
              Local
              {
                actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
      
                config
                {
      
                  concurrent-job-limit = 5
                  run-in-background = true
                  #temporary-directory = "`mktemp -d \"/global/cscratch1/sd/jaws/cromwell-tmp\"/tmp.XXXXXX`"
      
                  # The list of possible runtime custom attributes.
                  runtime-attributes = """
                  String? docker
                  String? docker_user
                  """
      
                  # Submit string when there is no "docker" runtime attribute.
                  submit = "/usr/bin/env bash ${script}"
      
                  # Submit string when there is a "docker" runtime attribute.
                  submit-docker = "docker run --volume=${cwd}:${docker_cwd} ${docker} /usr/bin/env bash ${script}"
      
                  root = "/cromwell-executions"
                  dockerRoot = "/cromwell-executions"
      
                  filesystems
                  {
                    local
                    {
                      localization: [ "soft-link", "hard-link", "copy" ]
                      http {}
      
                      caching {
                        duplication-strategy: [ "soft-link" ]
                        hashing-strategy: "path"
                      }
                    }
                  }
      
                  default-runtime-attributes
                  {
                    failOnStderr: false
                    continueOnReturnCode: 0
                  }
                }
              }
            }
          }
      
          database
          {
            profile = "slick.jdbc.MySQLProfile$"
            db
            {
              driver = "com.mysql.cj.jdbc.Driver"
            url = "jdbc:mysql://localhost:3306/cromwell?rewriteBatchedStatements=true&useSSL=false&autoReconnect=true&useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC&allowPublicKeyRetrieval=true"
              user = "userName"
              password = "somePassword"
              connectionTimeout = 5000
            }
            insert-batch-size = 2000
          }

.. note::
   The example config has different sections or "stanzas" defined by :bash:`{}` and can be nested.


One of the larger stanzas is called :bash:`backend`. It has only one backend defined, :bash:`default = "Local"`, but could have more like slurm, and custom backends.

The official `cromwell configuration documentation <https://cromwell.readthedocs.io/en/stable/Configuring/>`_ is the best place to learn about the configs, which includes:

    1. a tutorial on creating your own config.
    2. description of the different parameter settings.
    3. examples using different backends beside :bash:`Local`.
    4. how to set variables using the command line arguments (also described below).
    5. how to setup the mysql database (we have a tutorial under :ref:`Configuring MySQL Database for Cromwell <mysql setup>`)

2. Overriding Variables with a `options.json` File
++++++++++++++++++++++++++++++++++++++++++++++++++

See the official documentation from https://cromwell.readthedocs.io/ concerning `options.json <https://cromwell.readthedocs.io/en/stable/wf_options/Overview/#setting-default-runtime-attributes>`_.

3. Overriding Variables from the Command Line
+++++++++++++++++++++++++++++++++++++++++++++

Setting variables on the command line is done with the :bash:`-D` flag, with no spaces.
The variable name must include the full path as it exists in the `cromwell_docker.config `file, for example, the variable :bash:`dockerRoot` exists under:

.. code-block:: text

    backend {
        providers {
            Local {
                config {
                    dockerRoot = "/cromwell-execution"


Following that logic for other variables.

.. code-block:: text

    java -Dconfig.file=cromwell_docker.conf \
         -Dbackend.providers.Local.config.dockerRoot=$(pwd)/cromwell-executions \
         -Dbackend.providers.Local.config.root=$(pwd)/cromwell-executions \
         -Dbackend.default=Local \
         -jar ~/cromwell/cromwell.jar run my.wdl

where

    :bash:`-Dconfig.file`
    points to a cromwell conf file that is used to overwrite the default configurations

    :bash:`-Dbackend.providers.Local.config.dockerRoot`
    this overwrites a variable 'dockerRoot' that is in cromwell_docker.conf and tells cromwell to mount this path within the docker container.

    :bash:`-Dbackend.providers.Local.config.root`
    this overwrites a variable 'root' that is in cromwell_docker.conf and tells cromwell to use this path for its output outside of the docker container.

    :bash:`-Dbackend.default=[Local|Slurm|AWS|etc.]`
    this will allow you to choose between the Local and other backends.
    No other backends are included in the :bash:`cromwell_docker.conf` example.



Configure Backends
==================

One thing you can customize in a `Cromwell <https://cromwell.readthedocs.io/en/stable/>`_ configuration file is the backend which, among other things, specifies where your jobs will run. You can, for example, choose to use SLURM or your local machine. 

.. note:: **Running in JAWS**

      when running in JAWS, the `HTCondor` backend is set for you by default and can't be changed. 

This page describes how you can change backend parameters in the config file for testing your WDL. When running JAWS you cannot change the configuration settings at all.

How Cromwell uses the Configuration file
++++++++++++++++++++++++++++++++++++++++

Cromwell is installed with a complete default config file. However, for testing, you can overwrite parts of it by specifying another config file that has, for example, docker backend specifications. By including the :bash:`-Dconfig.file=<your.config>` option to your Cromwell command you can overwrite the default config. 

Example configs

  * complete config:  `Cromwell.examples.conf <https://github.com/broadinstitute/cromwell/blob/develop/cromwell.example.backends/cromwell.examples.conf>`_  
  
  * slurm config: `slurm.conf <https://github.com/broadinstitute/cromwell/blob/develop/cromwell.example.backends/slurm.conf>`_   
  
  * backend example using: `Singularity containers with Cromwell <https://github.com/broadinstitute/cromwell/blob/develop/cromwell.example.backends/singularity.slurm.conf>`_
  
  * all official backend examples from Cromwell docs: `all examples <https://github.com/broadinstitute/cromwell/tree/develop/cromwell.example.backends>`_

Backend Examples
++++++++++++++++

Cromwell has two modes of operation: server and non-server (client). During development, the client mode is the simplest way to test WDLs.

You can find examples of backends for our various sites `here <https://code.jgi.doe.gov/official-jgi-workflows/jaws-tutorial-examples/-/tree/main/config_files>`_.

Example of Running a WDL with Slurm Config
++++++++++++++++++++++++++++++++++++++++++

You can see a working example of how to run a task on slurm.
`working example using slurm <https://code.jgi.doe.gov/official-jgi-workflows/jaws-tutorial-examples/-/tree/main/using_slurm_and_local>`_

Follow the README.md in the above repository to run the WDL (`test.wdl`) and inspect the WDL and `cromwell_dori.conf` file. 

Essentially, you specify that you want to use slurm in the runtime section of your WDL. If you don't specify anything for the `backend:` then the default will run jobs locally, as set in the config (see below).

.. code-block:: text

	runtime {	
		backend: "SLURM"
	}

And in the config file you would define `SLURM` to be one of the backend `providers`.

.. code-block:: text

	backend {
  		default = "Local"
  		providers {
   			SLURM {
				submit = "sbatch <command>"
			}
	}