Abstract: We present Kliko, a Docker based container specification for running one or multiple related compute jobs. The key concepts of Kliko is the encapsulation of data processing software into a container and the formalisation of the input, output and task parameters. Formalisation is realised by bundling a container with a Kliko file, which describes the IO and task parameters. This Kliko container can then be opened and run by a Kliko runner. The Kliko runner will parse the Kliko definition and gather the values for these parameters, for example by requesting user input or pre defined values in a script. Parameters can be various primitive types, for example float, int or the path to a file. This paper will also discuss the implementation of a support library named Kliko which can be used to create Kliko containers, parse Kliko definitions, chain Kliko containers in workflows using, for example, Luigi a workflow manager. The Kliko library can be used inside the container interact with the Kliko runner. Finally this paper will discuss two reference implementations based on Kliko: RODRIGUES, a web based Kliko container schedular and output visualiser specifically for astronomical data, and VerMeerKAT, a multi container workflow data reduction pipeline which is being used as a prototype pipeline for the commisioning of the MeerKAT radio telescope.
Credit: Gijs Molenaar