Jobs can be linked with each other in a Job Flow, and the execution dependency of one job on another job can be set based on link condition.

Note: To create a new Job Flow, refer Creating New Job Flow

To create link between Jobs, and to assign link condition follow the below steps.

Step I: Create jobs in Job Flow

From Jobs menu, drag and drop the jobs as required on canvas.

  • Diyotta provides various types of jobs as shown under Jobs menu. To create various jobs, refer respective job pages in Working with Job Flow.
  • Links can be created between any of the two jobs under Jobs menu.

Step II: Create link between the jobs 

1. Select the job on canvas from where link needs to start. Then, drag the "connector out" from this start job to the "connector in" of the job subsequent job. This creates the link between the jobs.

You can also have multiple pipelines in a single Job Flow with multiple out and in links as shown below.

Step III: Assign link condition between jobs

The Link condition is an expression which defines the condition that should be met for execution of subsequent job to begin. This expression may include Job Flow parameters, Job Flow sql parameters, Project Parameters, Diyotta System Parameters, Runtime Status and Runtime Statistics.

1. Double click on the link between two Jobs, or select the link and click Open Expression Editor option. 

2. The Expression Editor window opens.

By default, the condition defined on any link is to check the status of previous job has succeeded.

You can change the expression to define what condition needs to be met for the linked job to run. The Link condition can consists of combination of multiple conditions. The AND and OR conditions can be concatenated using parentheses to check multiple conditionsOnce you enter the expression, click Validate to verify that there are no syntax errors. If everything is correct then, a success message is displayed. 

Here for reference, we are checking for the status of the previous job to succeed or defer, and the source count post execution to be 400. If both conditions are met, only then the next job will start executing.

  • The Link condition defined can include Parameters, Runtime Status and Statistics.
  • The Parameters can consists of Job Flow Parameter, Job Flow sql Parameter, Project Parameter, or System Parameter. For more information, refer Working with Job Flow Parameters, Working with Job Flow SQL Parameters, Working with Project Parameters, and Working with Studio System Parameters.
  • Runtime Status and Statistics: It consists of Runtime Status and Runtime Statistics of Job. You can use Runtime Status and Runtime Statistics to get particular details of Job. For more information, refer Working with runtime status and statistics.
    • If any job is dependent on a Data Flow instance, then, in Expression Editor you get the option of both Runtime Status and Runtime Statistics.
    • If any job is dependent on simple Job, then in Expression Editor you get the option of only Parameters and Runtime Status.
  • If the link condition is defined in a Loop Job, then the link condition can include Looper Parameters such as $LP_Loop_CMD_1, $LP_ITERATOR_ID and $LP_ITERATOR_NUMBER. Looper Parameters are shown in the jobs and link conditions present in between Looper Start and Looper End. For more information, refer Working with Loop Job.

3. If the link condition is invalid, then on canvas the link is red highlighted, and an error message is shown under properties.

  • If there are multiple incoming links for a job then the condition defined on each link has to be met for the job to start executing.
  • If the condition defined on link is not met when the link start job completes, then the jobs linked to it do not run and are marked as deferred.
  • When the link start job is executing, then the dependent job does not execute and will be in waiting or dependency state.

Note:

  • To align various Jobs in Job Flow, click Arrange on the Actions menu. For more information, refer Arranging Job Flow
  • To delete the link between jobs, click Delete Link on the Actions menu. For more information, refer Deleting links in Job Flow
  • To save the Job Flow, click Save on the Actions menu. For more information, refer Saving Job Flow
  • To revert the changes before saving the Job Flow, click Revert on the Actions menu. For more information, refer Reverting changes in Job Flow
  • To execute individual job in Job Flow, click Run Job on the Actions menu. For more information, refer Executing individual job in Job Flow.
  • To execute the Job Flow, click Run on the Actions menu. For more information, refer Executing Job Flow.
  • Once you create and save the Job Flow, then, close or unlock the Job Flow so that it is editable by other users. For more information, refer Closing Job Flow and Unlocking Job Flow.
  • To delete the link between jobs, click Delete Link on the Actions menu. For more information, refer Deleting links in Job Flow