The Rest Job is used to extract data from RESTful API services or SOAP integrated Web services. It provides option to request data using GET, PUT, DELETE and POST methods. The response from the application is placed as a file on the server defined in the job in the same format as sent by the application.

Note: To create a new Job Flow, refer Creating New Job Flow

To create the Rest Job, follow the below steps. 

Step I: From the Jobs menu, drag and drop the Web Services Job on canvas.

Step II: Provide the General details of Job.

On Canvas, select the Web Services Job, and then under Properties, provide the General details.

  • Name - The Name field consists of default name and is editable..
  • Description - In the text box, you can provide a description and is optional.
  • Disable task - Check Disable task, if the job need not be executed as part of the Job Flow, and you do not want to delete the Job.

Step III: Optionally specify the retry attempts.

Under Properties, select the Properties tab to enable retry attempts for Web Services job execution.

Retry EnabledCheck the retry option if you want to enable retry attempts for Rest job.

  • No. of Retry Attempts: Specify the number of attempts to retry Rest job execution if in case the job fails to execute. By default, the retry attempts is set to 2.
  • Retry Wait Time (in Seconds): Specify the duration in seconds for the job to retry next execution. By default, the duration is set to 60 seconds. If the Job fails to execute, it retries again for next execution attempt after the specified wait time. 

Step IV: Provide the properties that need to be used to form the request to extract data from the application service. 

1. Under Properties, click Web Services Properties tab. To add the Web Services Data Point, click Add.

2. The window lists all the available Web Services Data Points. Choose the required Data Point 

  • If there is any Global project that the user has access to, then the window displays the Project drop-down, which lists all the global projects. You can choose the global project from the Project drop-down, and select the global Rest data point as required.
  • To create Rest Data point and for more information about Rest Data Point, refer Working with Web Services Data Point.

3. Provide the following details in the Web Services Properties.

  • Service Type: Choose service type as REST or SOAP. The properties vary based on type of webservice API being connected to.

Option I: Service Type - REST

  • Method Type: Provide the request type to extract data from application. The request types can be GET, PUT , DELETE or POST.
  • REST webservices URI: Provide the URL to access resources on REST API.
  • End Point Relative Path: Provide additional extension for the above URL to retrieve the required data. 
  • Body: Provide body in raw format to be included in the request sent to the Rest application.

    The Rest webservices URI, End Point Relative Path and Body can be parameterized using Project Parameter and System Parameter. For more information, refer Working with Project Parameters and Diyotta System Parameters.
  • Choose the option Write response to the file if the response needs to be saved in a output file or choose Save response as parameter option to save response in memory and use it as parameter in the next jobs.

  • File Properties tab is seen when Write response to file option is chosen and Response Parameters tab is seen when Save response as parameter option is chosen. These tabs are see below Web Services properties.
  • Save response as parameter is supported only when the response from the API call is of Json format.

Re-run Properties: 

Re-run is supported only when the response from the API call is of Json format.
  • Rerun Based on Response Data : Set this to yes to rerun rest command based on the response of the prior call. By default, Rerun Based on Response Data is disabled by setting it to NO. If the specific value mentioned against the URL needs to be monitored to avoid connectivity and Job to run until it finds the required response value then, Rerun Based on Response Data needs to be enabled by setting it to Yes. By default, Rerun Based on Response Data is disabled by setting it to NO.
  • Response Data to Check: Provide the attribute name and the value to check and determine if rest command needs to be submitted again. Ensure the fully qualified parameter name is provided. Example: data.status=Done. Mention the specific parameter and value to be monitored and send response when Rerun Based on Response Data is enabled. ex: status.state=Done, httpsstatuscode=200
  • Rerun Interval (in seconds): Provide the duration in seconds to attempt the next execution of rest command. Mention the interval to check specific value to be monitored against URL until it finds the response value when Rerun Based on Response Data is enabled.

Pagination Properties: If the response from the application has multiple pages and all the pages need to be extracted then, Pagination needs to be enabled by setting it to On. By default, pagination is disabled by setting it to OFF.

  • Pagination is supported only when the response body has "paging" or "page" element indicating multiple pages in the response and the next page is indicated by "next" element.
  • Pagination is supported only when the response from the API call is of Json format.

Pagination: Enable pagination by specifying if it is header based or body based. By default, pagination is disabled by setting it to OFF.  

  • Request Type : Provide which method will be used to request the next page. Depending on this the rest command will be prepared to request subsequent pages. 

URL: A fully qualified URL. 
Request Param:  Key, value pair will be added to the initial rest command URL.
Request Header:  Header will be added to the initial rest command URL.

  • Response Key: This is applicable only for header based pagination. Specify the attribute to look for in the header of the response from the prior rest command execution. The value against this attribute will be used to request next page. 
  • Response Body attribute: This is applicable only for body based pagination. Specify the attribute to look for in the body of the response from the prior rest command execution. The value against this attribute will be used to request next page.
  • Request Key: This is applicable when request type is request param or request header. In case of request param, provide the key name to be used in rest command URL and in case of request header specify the parameter name to be used in the header. Specify the key request value to request next page. 
  • Repeat attribute: Specify the attribute and the value to look for in the response from the prior rest command execution to determine if request for next page should be submitted. Ensure the fully qualified parameter name is provided.
  • Enable request template: Enable this to provide a template to form the request for next page.
  • Request Template : Specify the template depending on the request type. Use predefined constants to specify any conditions and URL in the template Specify the URL value in standard template and give relative attribute keyword on which pagination searches for next page. This is enabled only when Enable request template is YES.

Decryption Properties: 

  • Decrypt Response: Enable this to decrypt the response from the rest command execution.
  • Portion of response to decrypt:  Specify the attribute in the response to be decrypted.
  • Get Intialization vector from response: Enable this when the response includes the Initialization Vector used to decrypt.
  • Intialization vector file: This is displayed when "Get Intialization vector from response" is set to No. Specify absolute path of the Initialization Vector file.
  • Intialization vector from field: This is displayed when "Get Intialization vector from response" is set to Yes. Specify the attribute in the response which contains the initialization vector.
  • Get AES key location from response: Enable this when the response includes the AES key file location used to decrypt.
  • AES key file: This is displayed when "Get AES key location from response" is set to No. Specify absolute path of the AES key file.
  • AES key from field: This is displayed when "Get AES key location from response" is set to Yes. Specify the attribute in the response which contains the path of the AES key file.
  • Is AES key encrypted: Enable this when the AES key used to decrypt the response is encrypted.
  • Private Key file: This is displayed when "Is AES key encrypted" is set to YES. Specify absolute path of private key file to decrypt the AES key. 
  • Cipher Operation Mode: Enable Cipher Operation Mode to use GMC or CBC algorithms for decryption.
  • Authentication Tag size: Specify the tag size to be used for decryption.
  • Associated text: Specify the text in the encrypted value that needs to be ignored during decryption.

Web services Parameters

1. To add Web services Parameters, click on the Add icon. Click to enter the key and value. 

2. Under Web Services Parameters, add all the key name and its value that need to be included in the API request. 

Note: Following operations are allowed on the Parameter entries: Add and Delete.

Rest Headers

Provide the header properties which consists of key name and value. Add this by either selecting from the predefined list by clicking on Select or add a new entry by clicking on Click Here and modifying the name manually.

  • The "Key" field is header property name to be appended to the Rest call URL. 
  • Provide the value to be used for the header in the "Value" field against the header key name

Option II: Service Type - SOAP

  • WebservicesURIProvide the URL to access resources on Webservice API.
  • Version: Choose the version of SOAP service.
  • Wsdl Relative Path: Provide fully qualified wsdl URL for the webservice API specified. Click on arrow next to it to load the operations.
  • Operation: Choose the request and a response service from the operation drop down displayed based on wsdl Relative Path. Click on arrow next to it to load the service, binding and the template body. 
  • Service: This property is populated automatically based on the selected operation. This is a collection of related end-points encompassing the service definitions in the file.
  • Binding: This property is populated automatically based on the selected operation. Binding is an direction to use the SOAP HTTP transport protocol and data formats for the operations and messages defined for a particular operation chosen. 
  • Request Body: The template body is populated automatically based on the operation chosen. The template can be modified by clicking the highlighted arrow and this will be used in SOAP command to request data.

The Webservices URI, Wsdl Relative Path and Operation can be parameterized using Project Parameter and System Parameter. For more information, refer Working with Project Parameters and Diyotta System Parameters.

Rest Headers

Provide the header properties which consists of key name and value. Add this by either selecting from the predefined list by clicking on Select or add a new entry by clicking on Click Here and modifying the name manually.

  • The "Key" field is header property name to be appended to the Rest call URL. 
  • Provide the value to be used for the header in the "Value" field against the header key name

The Web Services Properties can be parameterized using Job Flow Parameter, Job Flow sql Parameter, Project Parameter, or System Parameter. For more information, refer Working with Job Flow ParametersWorking with Job Flow SQL ParametersWorking with Project Parameters, and Diyotta System Parameters.

You can also use Runtime Status and Runtime Statistics from linked Job to parameterize the properties. For more information, refer Working with runtime status and statistics.

Step V: For service type, SOAP, response can be written only to a file and File properties tab will be displayed to provide the output file related details. For service type, REST, depending on the option chosen to write the response either File properties tab will be displayed or Response Parameter tab will be displayed.

Option I: File properties

1. Under Propertiesclick File Properties tab. To assign a File Data Point, click Add

2. The window lists all the available File Data Points. Choose the required Data Point.   

To create File Data point and for more information about File Data Point, refer Working with File Data Point.

3. Provide the details for File Properties.

  • File Name - You can specify the name of the file with which you want to create the generated response.  
  • File Name Prefix - Specify the prefix that you want to add to the file name.
  • File Name Suffix - Specify the suffix that you want to add to the file name. 
  • Enable overwrite - Select Yes, if you want to overwrite the existing file. Select No, if you do not want to overwrite the file. If the file exists then the job will fail if the file is already present.

Option II: Response Parameters

You can create parameters here and assign the value of any attribute from response of the Rest command execution. These parameters can then be used in subsequent linked jobs.

1. Create a Response parameter

  • Under Propertiesclick Response Parameters tab, then to add a new parameter, click Click Here, or click Paste to paste the parameter you have already copied.

2. Provide the name and value for the parameter

The list shows the added parameter. 

  • Param Name: Click on the param name text box, to modify the name. The names for each parameter in the list has to be different.
  • Param: This is the internal param name generated by Diyotta. The response parameter name is prefixed with '$RT_' .
  • Header/Body: Using this option you can pull the parameter value either from Header or Body by providing the property name from header or attribute name from the body as the value. Click on the header/body text box, to modify it. 

  • Value: Provide the fully qualified attribute name to be looked for in the response to get the value to be assigned to the parameter.

1. Click on the arrow icon next to the Value field. 

2. The Expression Editor window will open, and you can define the response value as parameter value here by giving exact object hierarchy of the value. 

Once you enter the expression, click Validate to verify that there are no syntax errors. If everything is correct then, a success message is displayed.

Here for example, the parameter is being defined as jobReference.jobId.InputFileName in parent to child hierarchy format to capture exact value of InputFileName attribute from the response.

The expression defined for parameter value can include other parameters such as Job Flow SQL Parameters, Project Parameters, and Diyotta System Parameters. For more information, refer Working with Job Flow sql parametersWorking with Project Parameters, and Working with Studio System Parameters.

Use the Response parameter to parameterize different properties in the Job flow, or the Jobs in it. For illustrating the use of Response parameter in the Job Flow, refer the use cases below.

  • In an expression editor of jobs and options in the job flow, the Response parameters can be selected from the Response parameters list in the parameters section.
  • When Response parameter needs to be specified in fields, where expression editor is not available, then type in the Response parameter name with $RT_ prefixed.

Use Case IUsing response parameter to define the name of the target file generated as part of execution of associated data flow instance. 

Step IOn Canvas, click on the job, and then under Properties, click Connections. Click 'L', that is the Load properties of target object 'TGT_SRC_DEPARTMENT'. 

Step IIThe Load Properties window opens. 

Specify the response parameter in the "value" field for the property "File Name". Here there is no expression editor so, the response parameter needs to be typed in. Prefix it with $RT_JOBNAME_ as - $RT_RESTJOB_GA_FILE_NAME. During run-time, the value for property "File Name" will be replaced with the response value from the attribute defined from json format. Here jobReference.jobId.InputFileName returns TGT_SRC_DEPARTMENT.dat value during runtime.

Use Case II: Using Response parameter in the Task Command.

Step IOn Canvas, click on the task command job, and then under Properties, click on the Command Tab.  Open the editor for the command by clicking on the arrow.

The Expression Editor window opens.

Step IIProvide the command to be executed. For illustration, here we are checking if the file as specified in the response parameter exists and deleting it.

The expression editors will have section to display all the applicable parameters. To list all the response parameters, click on the RestJob Parameters.

Click on the RestJob Parameter name to use it in the expression.During run-time, the value for property "File Name" will be replaced with the response value from the attribute defined from json format. Here jobReference.jobId.InputFileName returns TGT_SRC_DEPARTMENT.dat value during runtime.

If the parameter name ($RT_****) used anywhere in the job flow, or associated jobs does not match any of the response parameters defined in the job flow then, during the execution these references will be replaced by default value of -1.

Note:

  • To save the Job Flow, on the Actions menu, click Save. For more information, refer Saving Job Flow
  • To revert the changes before saving the Job Flow, on the Actions menu, click Revert. For more information, refer Reverting changes in Job Flow
  • To execute individual job in the Job Flow, on the Actions menu, click Run Job. For more information, refer Executing individual Job in Job Flow.
  • To execute the Job Flow, on the Actions menu, click Run. For more information, refer Executing Job Flow.
  • Once the Job is created and the changes are saved, then, close or unlock the Job Flow so that it is editable by other users. For more information, refer Closing Job Flow and Unlocking Job Flow