In a data flow, a pipeline is defined by source instance transform as starting point, and the target instance transform as an ending point. You can have multiple pipelines in a data flow and a pipeline can have multiple targets. The load order feature in data flow is used to specify the order in which the data should be loaded into the targets. The data load can happen in parallel into all targets or sequenced to load one after the other.
To define a load order, in data flow edit page, navigate to Load Order tab and define the order. For each target in the data flow, an entry can be seen here. If there are multiple pipelines, then you can specify when the pipeline to which a target belongs should be executed. Also, if one pipeline has multiple target instances then, you can specify the order in which each instance load should be executed. Below are the fields that can be used to define this.
- You can auto set the pipeline order by selecting the order of execution from the drop down. Select Sequential or Parallel to define the order in which pipelines should be executed. To apply the selected order to all the pipelines click on Apply Batch icon. The order of the pipelines can be seen under the field Pipeline Order. The numbering of the pipeline execution order starts from 0. If Parallel is selected then, all pipelines will get number 0 and all pipelines will start executing together. If Sequential is selected then, the numbering will start from 0 and increase in sequence. The pipeline marked 0 will start executing first during runtime. If a pipeline fails during execution, then the subsequent pipelines will not execute.
- You can also manually set the order by setting the order of execution under the fields Pipeline Order and Target Order. From the Pipeline Order drop down select the sequence number at which the pipeline should be executed. If there are multiple targets in single pipeline then, to define the order in which each target should be loaded, select the sequence number from the Target Order drop down. The target order numbering starts from 1 and the target marked 1 will be loaded first. If any target load fails then subsequent target loads and any subsequent pipelines will not be executed. You can also change the pipeline order by selecting target and ordering it using the Move up and Move down icons.