Getting Started

v1.13.0 addresses several WDL 1.1 spec interoperability issues, with thanks to the reporting community members.

⚠️ These changes, while improving WDL spec compliance, bear slightly elevated risks of backwards-incompatibility with prior miniwdl versions. Please report any issues or unexpected behaviors for investigation.

Interoperability

• dedent task commands before eval/interpolation (#674 @adamnovak)
• allow float values for task runtime.cpu (#690 @adamnovak)
• round() "half up" instead of "to even" (#698 @stxue1)
• fix Array runtime typing issues (#699 #700 @stxue1)
• allow trailing commas in meta arrays (#708 @adthrasher)
• fix ValueError reading empty JSON into Map (#726 @ahvigil)
• refactor typechecking for equatability & comparability (#736)
• allow non-typesafe read_json(...) within arrays and if-then-else branches (#740 @ahvigil)

Runner

• allow namespace prefix in task-only input JSON files (#693 @stxue1)
• fix cache of task with empty outputs (#715 @adamnovak)
• fix recognition of exponential notation in JSON/YAML run input files (#717 @sach244n)
• --no-quant-check applies to comparison operators (#752 @stxue1)

Python WDL API

• Multi-line strings (development/1.2 feature) get first-class AST representation as WDL.Expr.MultiLineString, which is dedented upon evaluation (instead of during parsing into WDL.Expr.String).
• WDL.Type.* classes expose ty1.equatable(ty2) and ty1.comparable(ty2) operators, indicating whether it is valid to use the WDL == != or < <= > >= operators, respectively.

v1.13.0 addresses several WDL 1.1 spec interoperability issues, with thanks to the reporting community members.

Because these changes bear small backwards-compatibility risks, we begin with release candidate v1.13.0rc0 that must be expressly requested with: pip install miniwdl==1.13.0rc0. Please report any issues or unexpected behaviors for investigation over the next few weeks.

Interoperability

• dedent task commands before eval/interpolation (#674 @adamnovak)
• allow float values for task runtime.cpu (#690 @adamnovak)
• round() "half up" instead of "to even" (#698 @stxue1)
• fix Array runtime typing issues (#699 #700 @stxue1 )
• allow trailing commas in meta arrays (#708 @adthrasher)
• fix ValueError reading empty JSON into Map (#726 @ahvigil)
• refactor typechecking for equatability & comparability (#736)
• allow non-typesafe read_json(...) within arrays and if-then-else branches (#740 @ahvigil)

Runner

• allow namespace prefix in task-only input JSON files (#693 @stxue1)
• fix cache of task with empty outputs (#715 @adamnovak)
• fix recognition of exponential notation in JSON/YAML run input files (#717 @sach244n)

Python WDL API

• Multi-line strings (development/1.2 feature) get first-class AST representation as WDL.Expr.MultiLineString, which is dedented upon evaluation (instead of during parsing into WDL.Expr.String).
• WDL.Type.* classes expose ty1.equatable(ty2) and ty1.comparable(ty2) operators, indicating whether it is valid to use the WDL == != or < <= > >= operators, respectively.

This maintenance release has no user-facing changes, but updates dependencies and modernizes the Python project tooling (#668) in preparation for the next development cycle.

• Fix coercions of Array[File] and Array[Directory] to Array[String] (#669 @john-smith, #681 @markjschreiber)
• Ensure presence of optional fields in structs read from runner input JSON file (#686 @adthrasher)
• miniwdl run: accept CLI inputs of compound types (pairs, maps, structs) as single-quoted JSON; also accept Array[Directory]
• miniwdl check: warn when a document of an older WDL version imports a document of a newer WDL version
• Dependency updates

Getting Started

Maintenance release:

• miniwdl input-template: add --no-namespace option to omit the top-level workflow name prefix
• fix download caching when config [file_io] root is used (miniwdl-ext/miniwdl-aws#20 @staskh)
• logging improvements and dependency updates

• WDL spec compatibility:
  • == and != accept optional operands (#634 @jdidion)
  • fix resolution of imported struct type aliases when the original struct name is reused in current document (#635 @jdidion)
  • WDL 1.1: allow placeholder options like sep= in string expressions outside of task commands (#633 @adamnovak)
  • WDL 1.1: disable a few version development features that had been inadvertently enabled for version 1.1 (multi-line strings and optional call: keyword)
• allow extra JSON keys initializing structs (#524)
• fix bug in WDL.Env.merge() that could resurrect previously-shadowed bindings (#637 @adamnovak)
• miniwdl zip --additional to include any specified file(s) in the zip (#641 @rhpvorderman)

• Change singularity invocation from singularity run to singularity exec (#628 @adthrasher)
• WDL version development:
  • Make the input: keyword optional in calls (openwdl/wdl#524)
  • miniwdl check: reduce spurious warnings on env task inputs used as environment variables instead of WDL interpolations (openwdl/wdl#504)

• Two potentially breaking changes:
  • Remove the default -l (login shell) flag that had been supplied to each task command bash interpreter starting in v1.5.3, but caused interoperability problems (#603). New config options mentioned below can restore this flag if necessary.
  • Disallow implicit coercions of Boolean? Int? Float? File? Directory? to (non-optional) String, which should never have been permitted (#596). These had been subject to miniwdl check deprecation warning since v1.7.1. They will still validate with --no-quant-check.
• New config options (see default.cfg)
  • [task_runtime] command_shell and [task_runtime] command_preamble customize the task command bash interpreter. For example, to restore the login shell flag, set [task_runtime] command_shell = /bin/bash -l or environment MINIWDL__TASK_RUNTIME__COMMAND_SHELL='/bin/bash -l'
  • [file_io] chown to opt-out of the post-task chown of the working directory
  • [podman] exe to customize the invocation for the podman backend (for example, to remove sudo)
• Relax PyYAML version requirements (#625 @adamnovak)
• Improve MultipleDefinitions error messages (#622 @crabba)
• Allow underscores in the short tags used to identify each iteration of a scatter
• miniwdl zip: fix assertion error importing WDL from a sibling directory sharing a common basename prefix with the main directory

• In scatters, embed a string representation of the scatter variable in each call's logs and run subdirectory structure.
  • Facilitates identification of the call processing a given sample, but the string tag must be derived heuristically.
  • The tag length can be adjusted by config [scheduler] scatter_max_tag (0 to disable the new tagging).
• Add config [file_io] use_relative_output_paths = true to flatten the out/ links structure. (#606 @rhpvorderman)
  • The generated link path relative to out/ reflects the path relative to the original task working directory.
  • Fails in the event of an output link path collision (e.g. a workflow outputting files with colliding names generated by different calls).
• Fix out/ link directory structure for Maps (where subfolders used numeric indexes even if the map keys were suitable path components).
• Fix postprocessing absent File?/Directory? task outputs. (#614 @rhpvorderman)
• Fix parsing string literals ending with escaped backslash, e.g. "foo\\".
• miniwdl zip: Fix archival of local WDL that imports a remote URI. (#610 @yunhailuo)

WDL version development features (experimental & subject to change)

• openwdl/wdl#414 Multi-line strings: the <<< >>> task command delimiters can now be used to write multi-line string literals wherever needed.
• openwdl/wdl#504 Support Defining Environment Variables Within a Task: task input declarations can be prefixed by the keyword env to bind the input as an environment variable for the command (an alternative to WDL command interpolations/placeholders).