• MOTEUR workflow managersupport
• DIET middleware with new “MA DAG” workflow

A new agent called the MADAG is used to manage workflows in the DIET architecture. This agent receives requests from clients containing the description of a workflow in a specific language (the MADAG XML workflow language). The role of the MADAG is to determine how to schedule the tasks contained in the workflow in order to follow the precedence and data constraints between tasks, and how to map the tasks to appropriate resources in the DIET hierarchy.

see this video to understand how the MA_DAG provides Workflow support to the Client throught the diet_wf_call API.

A workflow application is considered as a Directed Acyclic Graph (DAG) The workflow is described with an XML representation which is close to Diet profile representation. In addition to profile description (problem path and arguments), this description represents also the data dependencies between ports (source/sink), the node identifier (unique) and the precedences between nodes. This last information can be removed since it can be retrieved from the dependencies between ports, however it can be useful to define a temporal dependency without port linking. The general structure of this description is: <dag>

<node id=“…” path=“…”>

<arg name=“…” type=“……..” value=“..”/>

<in name=“…” type=“……..” source=“…….”/>

<out name=“….” type=“……..” sink=“……”/>

<out name=“….” type=“……..” sink=“……”/>

</node>

….

The main idea of the MA_DAG is to provide a common point for clients to deal with concurrent workflow submissions to the DIET hierarchy. If the MA_DAG becomes a bottleneck, multiple MA_DAG can be connected to the DIET hierarchy, but in this case the multi-workflow scheduling behavior is lost.

The available MA_DAG workflow schedulers are:

• A basic scheduler (option -basic or default choice) : this scheduler manages the precedence constraints between the tasks. The priority between tasks within a dag is set according (Heterogeneous Earliest Finish Time) HEFT heuristic. When a task is ready to be executed (ie the preceding tasks are completed) the ready task with the higher HEFT rank is sent to the client for execution without specifying a resource. Then The client performs a standard DIET request that will use the scheduler configured by the SeD.
• A Multi-HEFT scheduler (option -heft) : this scheduler applies the HEFT heuristic to all workflows submitted by different clients to the MADAG. This means that the priorities assigned by the HEFT heuristic are used to order the tasks of all dags processed by the MADAG and following this order the tasks are mapped to the first available resource.
• A Multi-AgingHEFT scheduler (option -aging heft) : this scheduler is similar to Multi-HEFT but it applies a correction factor to the priorities calculated by the HEFT algorithm. This factor is based on the age of the dag ie the time since it was submitted to the scheduler. Compared to Multi-HEFT this scheduler will increase the priority of the tasks of a workflow that has been submitted earlier than other dags.
• A FOFT (Fairness on Finish Time) scheduler (option -fairness) : this scheduler uses another heuristic to apply a correction factor to the priorities calculated by the HEFT algorithm. This factor is based on the slowdown of the dag that is calculated by comparing the earliest finish time of the tasks in the same environment without any other concurrent workflow and the actual estimated finish time.
• A SRPT (Smallest Remaining Processing Time) scheduler (option -srpt) : this scheduler changes the priority of the tasks managed by the MA_DAG in order to select the tasks from the applications (DAG) which have the smallest remaining processing time.

All these heuristics are based on HEFT : rank the task then select the appropriate resource to compute the task. It is important to notice that all these heuristics provide the same scheduling if there are no concurrent workflows submission. The workflow schedulers (Basic, Multi-HEFT, Multi-AgingHEFT, FOFT and SRPT) use dynamic information provided by the SeDs to be able to run the HEFT heuristic. The SeD programmer must provide the required data in the estimation vector by implementing a plugin scheduler into DIET. Some basic examples are given in the DIET distribution. You can find more information about DIET on the web pages of the project. You can also download the new version of DIET-2.3 which have all support for workflow execution.