File Watcher command is used in Job Flow, to watch Files. Most common use of File Watcher Command job is to watch single or multiple files before or after the data flow completes.

The syntax and validity of commands used in File Watcher Command job should be validated by the user before execution. Any issues with the command will result in failure during runtime.
Note:  To create a new Job Flow, refer Creating New Job Flow

To create the File Watcher Job follow the below steps. 

Step I: From the Jobs menu, drag and drop the Task Command on canvas.

Step II: Provide the General details of Job.

On Canvas, select the Task Command Job, and then under Properties, provide the General details.

Name - The Name field consists of default name and is editable.

Description - In the text box, you can provide a description and is optional.

Disable task - Check Disable task, if the job need not be executed as part of the Job Flow, and you do not want to delete the Job. 

Step III: Provide the Properties for Job.

Under Properties, click Properties and change the entries for below. 

Success Codes: Define the exit codes from the script that should be treated as successful exit. By default only when the script exists with codes 0 and 99 the Task Command will be marked as success. If the script exits with any other code then the Task Command will be marked as Failed. You can add any numeric value here separated by comma (,) ans these will be considered as successful exit codes.

Retry Enabled: Check the retry option if you want to enable retry attempts for file watcher task command job execution.

  • No. of Retry Attempts: Specify the number of attempts to retry task command job execution if in case the job fails to execute. By default, the retry attempts is set to 2.
  • Retry Wait Time (in Seconds): Specify the duration in seconds for the job to retry next execution. By default, the duration is set to 60 seconds. If the Job fails to execute, it retries again for next execution attempt after the specified wait time. 

Step IV: Provide the command to be executed

1. Under Properties, click CommandSelect the Command Type as File Watcher.

The Command Type drop-down lists the command types such as External and File Watcher. For more information about External for Linux commands, refer Working with Task Command Job.

2. Under Command, click on the Expression Editor arrow. The Expression Editor window opens.

The command defined contains the File Watcher command, and may include Parameters, Runtime Status and Runtime Statistics.

The File Watcher command defined consists of various arguments as listed below. You can provide values for these arguments based on your requirement. 

File Watcher Command syntax: <<file to watch>> [ -w 1] [-a <<FILE ARCHIVE DICRECTORY>>] [-t 60<<timeout after specified seconds>>] [-i <<For Amazon S3 only. Agent FilePath-only for Indirect>>] [-s <sleep time for each triggering to server in seconds >]

  • file to watch : Specify the file name with the file path, which needs to be watched for arrival.
  • -w : Specify the wait time after the file arrives and before it is archived. By default it is 1 second and you can change it as needed.
  • -a : Specify the path where the file needs to be archived once the file to watch has arrived. 
  • -t : By default the timeout is 60 seconds and you can change as needed. If the file does not arrive in specified timeout seconds, then it goes to timeout mode. 
  • -i : Indirect file which has path of the file being looked up. This is specific to Amazon S3. 
  • -s : Specify the sleep time in seconds for triggering to server to check if the file to watch has arrived. 

All arguments are not mandatory and you can specify only the arguments that are needed.

Once you enter the expression, click Validate to verify that there are no syntax errors. If everything is correct then, a success message is displayed. 

Here for reference, the Command get executed in the assigned Data Point and the file 'BRANCH1.csv' in the specified path is watched every 2 seconds for arrival. If the file does not arrive in the timeout specified that is 10 second then it goes to timeout mode. Once the file arrives it waits for 3 seconds and the file gets archived to the path specified /incoming/archive.

Step V: Add the Data Point 

1. Add a Data Point for the File Watcher command job to be executed. Next to the Data Points, click Add.

2. The window lists all the available Data Points. Choose the required Data Point.

If there is any Global project that the user has access to, then the window displays the Project drop-down, which lists all the global projects. You can choose the global project from the Project drop-down, and select the global data point as required.

Note:

  • To save the Job Flow, on the Actions menu, click Save. For more information, refeSaving Job Flow
  • To revert the changes before saving the Job Flow, on the Actions menu, click Revert. For more information, refer Reverting changes in Job Flow
  • To execute individual job in the Job Flow, on the Actions menu, click Run Job. For more information, refer Executing individual job in Job Flow.
  • To execute the Job Flow, on the Actions menu, click Run. For more information, refer Executing Job Flow.
  • Once the Job is created and the changes are saved, then, close or unlock the Job Flow so that it is editable by other users. For more information, refer Closing Job Flow and Unlocking Job Flow