Page tree
Skip to end of metadata
Go to start of metadata

dicmd is Diyotta command line utility that provides capability to access Diyotta from Linux machine and perform operations in Diyotta without logging into the web user interface.

dicmd is installed as part of controller installation. By default, this command can be run only from the user under which Diyotta controller is installed. You can run the dicmd from a different user as well by providing appropriate folder permissions.

This utility can also be installed externally on separate Linux machine and configured to connect and work with remote Diyotta installation. 

For performing actions in Diyotta using dicmd it is mandatory to pass the username and password of the Diyotta user as parameter. The actions performed in Diyotta using dicmd will be logged under the username provided. Alternately, the credentials can be set onetime in the profile file of the Linux user. If credentials are set in profile file then, the username and password parameters need not be used when executing dicmd. If the username and password parameters are provided even when these are defined in the profile file then, the parameters used will override the values in the profile file. To define the credentials in profile file follow below steps. Here, for reference .bash_profile is being considered as the profile file.

Step I: Open the profile file.

vi ~/.bash_profile

Step II: Add username and password parameter in this file. Below parameters needs to be added.

export DI_HOME="<<Diyotta login user>>" (Example: "Administrator")

export DI_PASSWORD="<<password>>" (Example: "ysAxD_324")

The plain text password can be masked before adding in the profile file or passing through dicmd. This helps to secure the password by not exposing the password in plain text in profile file or dicmd command parameter. To mask or encrypt the password, Diyotta provides an option in dicmd called passwd. You can provide the plain text as input to it and the result will be a masked text. See example below. When this masked text is provided as password then, Diyotta internally converts it to actual plain password to authenticate and execute the command.

$ dicmd passwd -e 'P2wd_4321'

M2FThB8QQAsv8fTo9KxhnSmLsjXgI18POI0qCQDUYiE=

It is required that the user specified has sufficient privileges to perform the operation. 

Below are various options available in dicmd to perform different actions in Diyotta.

  • version: This option is used to view the current version of the Diyotta Controller.

Syntax

dicmd version [-u Username -w UserPassword]

Arguments

-u Username (optional)

Username of the Diyotta user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)

Password of the Diyotta user. This parameter is required if username parameter is used in the command.

Privileges

The user requires "Studio Read" privilege for performing this action.

Example: 

$ dicmd version

4.1.0.3114.002

ExitCode:0

  • serverstatus: This option is used to view the status of Diyotta controller at any moment. If the Controller is up and running without any issues then, it exists without any messages. In case of issues the command exists with appropriate error message.

Syntax

dicmd serverstatus

Arguments

None

Privileges

Not restricted by user specific privileges.

Example: Output when server is working fine.

$ dicmd serverstatus

ExitCode:0

  • agentstatus: This option is used to view the status of a specific agent registered with the Controller. Alternately, this option can provide status of all agents registered with the Controller.

Syntax

dicmd agentstatus [-u Username -w UserPassword] [-a AgentName]

Arguments

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-a AgentName (optional)

Name of the agent to check the status. Only one agent name can be specified with this argument. If this argument is not provided then, this option prints the status of all the agents registered with the Controller.

Privileges

Not restricted by user specific privileges.

Example: Print the status of all the agents registered with the Controller.

$ dicmd agentstatus

-------------------------------------------------------------------
Agent Name Status
-------------------------------------------------------------------
Default Active
src_agnt_1 Not Active
tgt_agnt_1 Not Active

ExitCode:0

  • status: This option is used to view the status of the last execution of dataflow and jobflow in a particular project or layer. You can also view the output based on a specific status type. The output is synonymous to the status that can be seen on the Monitor module. 

Syntax

dicmd status [-u Username -w UserPassword] [-c ACTIVE|FAILED|SUCCESS|ABORT|STOP|ALL] -p ProjectName [-l LayerName] [-d DataFlowName|-s JobFlowName]

Arguments

 -u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-c ACTIVE|FAILED|SUCCESS|ABORT|STOP|ALL (optional) 

Restrict the output based on the execution status. Only one of these values can be specified with this argument. If this argument is not provided then, the command will consider all the statuses.

ACTIVE - Jobs with status as activeFAILED - Jobs with status as failedSUCCESS - Jobs with status as succeeded
ABORT - Jobs with status as abortedSTOP - Jobs with status as stopped

-p ProjectName (optional)

Restrict the output to a specific project. Only one project can be specified with this argument. If this argument is not provided then, the command will consider all the projects to which the user has access.

-l LayerName (optional)

Restrict the output to a specific layer in a project. Only one layer can be specified with this argument. If this argument is provided then, it is mandatory to provide the project name also. If this argument is not passed and project name is specified as argument then, the command will consider all the layers in the projects to which the user has access.

-d DataFlowName (optional) 

Restrict the output to a specific data flow in a layer in a project. Only one data flow can be specified with this argument. If this argument is provided then, it is mandatory to provide the project name and the layer name to which it belongs. You can either provide data flow name argument or job flow name argument for this option. If this argument and job flow name argument is not provided and project name and/or layer name is specified as argument then, the command will consider all the data flows in the project/layer specified.

-s JobFlowName (optional) 

Restrict the output to a specific job flow in a layer in a project. Only one job flow can be specified with this argument. If this argument is provided then, it is mandatory to provide the project name and the layer name to which it belongs. You can either provide data flow name argument or job flow name argument for this option. If this argument and data flow name argument is not provided and project name and/or layer name is specified as argument then, the command will consider all the job flows in the project/layer specified.

Privileges

The user requires "Monitor Read" privilege for performing this action.

Example: Print all the failed job flows and data flows executed in a layer.

$ dicmd status -c FAILED -p Project_1 -l Layer_1

RunId,FldrName,LayerName,Name,Status,StartTime,EndTime

7042,Project_1,Layer_1,ds_Kaf_DFS,F,20180605102925,20180605103029

7276,Project_1,Layer_1,DATA_FLOW_NAME_1,F,20180611022954,20180611023011

TotalCount:2

ExitCode:0

  • compileThis option is used to recompile the data flows or job flows in a project or layer. 

Syntax

dicmd compile [-u Username -w UserPassword] -p ProjectName [-l LayerName] [-d [DataFlowName | DataStreamName]|-s JobFlowName]

Arguments

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-p ProjectName

Compile all the data flows and job flows in a specific project. Only one project can be specified with this argument. It is mandatory to provide this argument.

-l LayerName (optional)

Compile all the data flows and job flows in a specific layer in a project. Only one layer can be specified with this argument. If this argument is provided then, it is mandatory to provide the project name also. If this argument is not passed and project name is specified as argument then, the command will consider all the layers in the projects to which the user has access.

-d DataFlowName (optional)

Compile a specific data flow in a layer in a project. Only one data flow can be specified with this argument. If this argument is provided then, it is mandatory to provide the project name and the layer name to which it belongs. You can either provide data flow name argument or job flow name argument for this option. If this argument and job flow name argument is not provided and project name and/or layer name is specified as argument then, the command will consider all the data flows in the project/layer specified.

-s JobFlowName (optional)

Compile a specific job flow in a layer in a project. Only one job flow can be specified with this argument. If this argument is provided then, it is mandatory to provide the project name and the layer name to which it belongs. You can either provide data flow name argument or job flow name argument for this option. If this argument and data flow name argument is not provided and project name and/or layer name is specified as argument then, the command will consider all the job flows in the project/layer specified.

Privileges

The user requires "Studio Read" and "Studio Write" privilege for performing this action.

Example: Compile a specific dataflow in a project.

$ dicmd compile -p Project_1 -l Layer_1 -d df_Snowflake_to_Hive_jnr

ExitCode:0

  • execute: This option is used to either run, rerun, abort or stop the execution of dataflows and jobflows.

Syntax

dicmd execute [-u Username -w UserPassword] -c start|abort|stop|rerun|restartfromfailure -p ProjectName -l LayerName [-d [DataFlowName] |-s JobFlowName] [-j jobName] [-f ParamName] [-param Parameters] [-i instanceName] [-email [-mailTo mail@example.com] [-cc mail@example.com] [-subject subject] [-message message] [-logs]]

Arguments 

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-c start|abort|stop|rerun|restartfromfailure 

Specify the operation to be performed on any data flow or job flow. It is mandatory to specify this argument. Only one of these operation can be specified with the argument. 

start - To execute a job flow or data flowabort - To abort an active job flow or data flow executionstop - To stop an active job flow or data flow after completing execution
rerun - To execute a job flow or data flow with same Diyotta generated internal run id as prior run.restartfromfailure - To start executing a failed job flow or data flow from point of failure or prior run

-p ProjectName

Specify the name of the project to which the data flow or job flow which needs to be executed belongs. It is mandatory to provide this argument. Only one project name can be specified with this argument.

-l LayerName

Specify the name of the layer to which the data flow or job flow which needs to be executed belongs. It is mandatory to provide this argument. Only one layer name can be specified with this argument.

-d DataFlowName

Specify the name of the data flow that needs to be executed. You can either provide data flow name argument or job flow name argument for this option. It is mandatory to provide data flow name argument or job flow name argument. Only one data flow name can be specified with this argument.

-s JobFlowName

Specify the name of the job flow that needs to be executed. You can either provide data flow name argument or job flow name argument for this option. It is mandatory to provide data flow name argument or job flow name argument. Only one job flow name can be specified with this argument.

-j jobName (optional)  

Specify the name of the job within the job flow that needs to be executed. This argument can be specified only when job flow name argument is provided. Only one job name can be specified with this argument.

-f ParamName (optional)  

Applicable only for -c start|rerun|restartfromfailure. Specify the parameter file name with fully qualified path to be used as part of job flow execution. This argument can be specified only when job flow name argument is provided.

-param Parameters (optional)

Applicable only for -c start|rerun|restartfromfailure. Specify the parameters and the value that need to be used to override the default parameter value defined in the data flow or job flow during execution. This argument allows you to override project parameter, data flow parameter and Job Flow parameter. 

The parameter name and value needs to be encapsulated in single quotes ('') and multiple parameters need to be separated by comma (,).

The parameter names should be suffixed with identifiers based on the type of the parameter being passed with the argument. The project parameter should be suffixed with $PP_, the data flow parameter should be suffixed with $MP_ and job flow parameter should be suffixed with $FL_.

-i instanceName (optional) 

Applicable only for -c start|rerun|restartfromfailureSpecify a name for the instance to be associated with the execution of job flow. This argument is used to run multiple instances a job flow in parallel. This argument can be specified only when job flow name argument is provided. If no instance name is specified then in the monitor the job flow name will appear as it is. When instance name is specified then the job flow name will appear with [instance name] suffixed to the job flow name.

-email (optional)

Applicable only for -c start|rerun|restartfromfailureUse this argument to send email notification in case of failure of data flow or job flow being executed. Below are the associated arguments that need to be specified with this argument.

-mailTo mail@example.com

Specify the email id of the recipient to whom email should be sent. Multiple email ids can be specified as comma separated values. It is mandatory to provide this argument to send the email. You can use project parameters, job flow parameters and system parameters to specify the value to be passed with this argument. 

-cc mail@example.com (optional)

Specify the email id of the recipient to whom copy of the email should be sent. Multiple email ids can be specified as comma separated values. You can use project parameters, job flow parameters and system parameters to specify the value to be passed with this argument. 

-subject subject (optional)

Specify the subject of the email to be sent. You can use project parameters, job flow parameters and system parameters to specify the value to be passed with this argument. 

-message message (optional)

Specify the message to be included in the the email to be sent. You can use project parameters, job flow parameters and system parameters to specify the value to be passed with this argument. 

-logs (optional)

Use this option if you want monitor logs associated with this failed execution to be attached to the email sent.

Privileges

The user requires "Studio Read" and "Studio Execute" privilege to perform this action.

Example 1: Execute a Job Flow 

$ dicmd execute -c start -p Project_1 -l Layer_1 -s S_Netezza_to_Hive

ExitCode:0

Example 2: Execute a Job Flow by overriding project parameters used in job flow.

$ dicmd execute -c start -p Project_1 -l Layer_1 -s S_Netezza_to_Hive -param '$PP_PRD_FLTR=250','$PP_START_TIME=20181231000101'

ExitCode:0

Example 3: Execute an instance of job flow and send notification with email attachment in case of failure.

$ dicmd execute -c start -p Project_1 -l Layer_1 -s S_Netezza_to_Hive -i ins_eu -email -mailTo USER2@diyotta.com -cc USER3@diyotta.com -subject '$$JobFlowName Failed' -message 'Job flow $$JobFlowName in the project $$ProjectName failed. Please see attached logs for more details.' -logs

ExitCode:0

Example 4: Abort a Job Flow.

$ dicmd execute -c abort -p Project_1 -l Layer_1 -s S_Netezza_to_Hive

ExitCode:0

  • exportlog: This option is used to save monitor log of a specific data flow or job flow executed. 

Syntax

dicmd exportlog [-u Username -w UserPassword] -o ERROR|WARN|INFO|DEBUG|TRACE|ALL -p ProjectName -l LayerName -r runId [-d DataFlowName | -s JobFlowName [-j JobName]] [-t UnitName] -f OutFileName

Arguments

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-o ERROR|WARN|INFO|DEBUG|TRACE|ALL

Specify the log level at which the monitor log needs to be saved. Only if the log has the details at the required level only then, the logs will be saved. It is mandatory to provide this argument.

ERROR - This will have all error entries in the logWARN - This will have all the warning entries in the logINFO - This will only have informational log that highlight the progress of the application
DEBUG - This will have all entries in the log that provide more granular and diagnostic informationTRACE - This will have all entries in the log that are finer-grained informational events than DEBUGALL - This will have all the entries in the log irrespective of the level at which it is generated

-p ProjectName

Specify the name of the project to which the data flow or job flow for which the log needs to saved. It is mandatory to provide this argument. Only one project name can be specified with this argument.

-l LayerName

Specify the name of the layer to which the data flow or job flow for which the log needs to saved. It is mandatory to provide this argument. Only one layer name can be specified with this argument.

-r runId

Specify the Diyotta generated internal run id for the data flow or job flow execution for which the log needs to saved. It is mandatory to provide this argument. Only one run id can be specified with this argument.

-d DataFlowName

Specify the name of the executed data flow for which the log needs to saved. You can either provide data flow name argument or job flow name argument for this option. It is mandatory to provide data flow name argument or job flow name argument. Only one data flow name can be specified with this argument.

-s JobFlowName

Specify the name of the executed job flow for which the log needs to saved. You can either provide data flow name argument or job flow name argument for this option. It is mandatory to provide data flow name argument or job flow name argument. Only one job flow name can be specified with this argument.

-j jobName (optional)  

Specify the name of the job within the executed job flow for which the log needs to saved. This argument can be specified only when job flow name argument is provided. Only one job name can be specified with this argument.

-t UnitName (optional)

Specify the name of the unit within the executed job flow for which the log needs to saved. The unit name is the name of the transform that executed within the data flow. This argument can be specified only when data flow name argument or job flow name argument is provided. Only one unit name can be specified with this argument.

-f OutFileName (optional)

Specify the name with which the monitor log needs to be saved. The file name should be appended with the path where it should be saved. The user with which dicmd is executed should have write access permission to the path specified. If this argument is not specified then, the log will be saved with Diyotta generated file name in the folder from where the dicmd is executed.

Privileges

The user requires "Monitor Read" and "Monitor Execute" privilege to perform this action.

Example: Save Monitor logs for the specified data flow.

$ dicmd exportlog -o ALL -p Project_1 -l Layer_1 -r 13052 -d d_Oracle_to_Hive -f /home/disupport/log_13052.txt

ExitCode:0

  • export: This option is used to export the Diyotta objects in JSON file specification. You can export an entire project or layer or individual objects. Whenever an object is exported all the lower level objects that are used in it are also included in the exported file. Which means if a job flow is exported then all the jobs and data flows and it's associated objects like, data objects, data points, reusable expression will be included in the exported file.

Syntax 

dicmd export [-u Username -w UserPassword] -p ProjectName [-l LayerName] [-o DOBJ|DATAPOINT|NSEQ|EXPR|UDF|DATASUBFLOW|DATAFLOW|DATASTREAM|JOBFLOW|SCHTASK|SCHCAL|SCHEMAIL|SCHFILE|SCHEDULER|PROJECTPARAMS] [-t NZ|TD|OR|FF|PG|DB|HV|MS|BI|CO|SF|ED|GP|HQ|HD|JS|HB|XD|SP|TW|FB|SK|JM|KK|CS|SY|SS|TS|SN|MY|BQ|RT|AV] [-c DataPointName] [-g GroupName] [-n ObjectName] -f OutJSONFileName

Arguments

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-p ProjectName

Specify the name of the project which needs to be referred when exporting. It is mandatory to provide this argument when exporting any object. Only one project name can be specified with this argument. If only project name argument is provided then the entire project and the objects within it will be exported.

-l LayerName (optional)

Specify the name of the layer which needs to be referred when exporting. If this argument is provided then, it is mandatory to provide the project name also. Only one layer name can be specified with this argument. If only layer name and project name is provided then the entire layer and the objects within it will be exported. It is mandatory to provide this argument when exporting data flow or job flow.

-o DOBJ|DATAPOINT|NSEQ|EXPR|UDF|DATASUBFLOW|DATAFLOW|JOBFLOW|SCHTASK|SCHCAL|SCHEMAIL|SCHFILE|SCHEDULER|PROJECTPARAMS (optional)

Specify the type of object to be exported. This argument is mandatory to be provided when exporting a specific object type. You can specify one of these types for this argument.

DOBJ - To export data objectDATAPOINT - To export data pointNSEQ - To export sequence 
EXPR - To export reusable expressionUDF - To export user defined functionsDATASUBFLOW - To export data subflow
DATAFLOW - To export data flowJOBFLOW - To export job flowSCHTASK - To export scheduler task
SCHCAL - To export scheduler calendarSCHEMAIL - To export scheduler email eventSCHFILE - To export scheduler file watcher event
SCHEDULER - To export all tasks in schedulerPROJECTPARAMS - To export Project Parameters

-t NZ|TD|OR|FF|PG|DB|HV|MS|BI|CO|SF|ED|GP|HQ|HD|JS|HB| XD|SP|TW|FB|SK|JM|KK|CS|SY|SS|TS|SN|MY|BQ|RT|AV (optional)

Specify the database type for the database object to be exported. This argument is mandatory to be provided when exporting group level objects - data point, data object, sequence, expression and udf. You can specify one of these data types for this argument.

NZ - NetezzaTD - TeradataOR - OracleFF - FlatfilePG - PostgreSQLDB - DB2
BI - BiginsightsCO - CobolSF - SalesforceHD - HDFSJS - JSONHB - Hadoop
MS - MSSQLSP - Splice MachineSS - SASHV - HiveXD - XSDSY - Sybase
TW- TwitterFB - FacebookSK -  SparkJM - JMSKK - KafkaCS - Cassandra
TS - thoughspotSN - SnowflakeMY - MysqlBQ - big QueryRT - RestFulAV - Avro

-c DataPointName (optional)

Specify the data point associated with the database object to be exported. This argument is mandatory to be provided when exporting group level objects - data point, data object, sequence, expression and udf.

-g GroupName (optional)

Specify the name of the group to which the database object to be exported belongs. This argument is mandatory to be provided when exporting group level objects - data point, data object, sequence, expression and udf.

-n ObjectName (optional)

Specify the name of the object which needs to be exported. When exporting group level objects - data point, data object, sequence, expression and udf, it is mandatory to provide project name, object type, database type, data point name and group name. When exporting layer level objects - data flow and job flow, it is mandatory to provide project name and layer name. When exporting scheduler objects it is mandatory to provide project name.

-f OutJSONFileName (optional)

Specify the name with which the exported json file should to saved. You can specify the path along with the file name where the exported file should be placed. Make sure the user with which the export option is being run has permission to write in the path specified. If path is not specified then the file will be saved in folder from where dicmd is being executed. If this argument is not specified the the file will be named same as the name of the object being exported.

Privileges

The user requires "Studio Read" privilege to export studio objects and "Scheduler Read" privilege to export scheduler objects.

Example 1: Export data object

$ dicmd export -p Project_1 -o DOBJ -t NZ -c CONN_NZ -g INGESTION -n CUSTOMER -f /home/disupport/customer_nz.json

ExitCode:0

Example 2: Export data flow

$ dicmd export -p Project_1 -l Layer_1 -o DATAFLOW -n df_oracle_to_pg -f /home/disupport/df_oracle_to_pg.json

ExitCode:0

  • import: This option is used to import the Diyotta Studio and Scheduler code in JSON format. The json file to be imported could have code corresponding to entire project or layer or individual objects. 

Syntax

dicmd import [-u Username -w UserPassword] -p ProjectName [-l LayerName] -f InJSONFileName [-o ImportOptionsFile] [-g GlobalProjectName] [-h GlobalLayerName]

Arguments

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-p ProjectName 

Specify the name of the project into which the json file needs to be imported. It is mandatory to provide this argument when importing any object. Only one project name can be specified with this argument.

-l LayerName (optional)

Specify the name of the layer into which the json file needs to be imported. It is mandatory to provide this argument when importing layer level objects - data flow and job flow. Only one layer name can be specified with this argument.

-f InJSONFileName

Specify the the file with the absolute path to be imported. The user running the dicmd command should have permission to read the file.

-o ImportOptionsFile (optional)

Specify the options file to be used to override the default import behavior for specific objects in the json file. By default all the data points if exist in Diyotta Studio will be reused and all the other objects if exist will be replaced. If the object does not exist in Diyotta then, it will be imported. This file can be manually created or generated using genoptions option in dicmd. 

-g GlobalProjectName (optional)

Specify the name of the global project into which the global objects needs to be imported. It is mandatory to provide this argument when json file includes global objects. Only one global project name can be specified with this argument.

-h GlobalLayerName (optional)

Specify the name of the layer in the global project into which the global objects needs to be imported. It is mandatory to provide this argument when json file includes global data objects. This argument need not be provided when importing global objects at group level - data point, data object, sequence, reusable expression and udf. Only one layer name can be specified with this argument.

Privileges

The user requires "Studio Write" privilege to perform this action.

Example: Import a data flow 

$ dicmd import -p Project_2 -l Layer_1 -f /home/disupport/df_oracle_to_pg.json

ExitCode:0

  • genoptions: This option is used to generate import override file to override the default import behavior. The file generated will be in json format. By default the data points if exist in Diyotta Studio will be reused and all the other objects if exist will be replaced. You need to provide the Diyotta import json file as input for which the options file need to be generated and corresponding override options json file will be generated. The override options file will have name of each object that is present in the import json file. You can specify what import option needs to be used for each object in the import file. Use below indicators to specify the import option against the required objects.
C - Create or import the object if object with same name does not exist in DiyottaR - Replace the object if object with same name exists in DiyottaA - Rename the file during import. Import the object in Diyotta with a name different than in import file.
E - Reuse the object in Diyotta if same object existsS - Skip the object from importing

Syntax

dicmd genoptions [-u Username -w UserPassword] -p ProjectName [-g GlobalProjectName] -i InJSONFileName -f OutJSONFileName

Arguments

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-p ProjectName

Specify the name of the project into which the json file will be imported. It is mandatory to provide this argument to generate options file. Only one project name can be specified with this argument.

-g GlobalProjectName (optional)

Specify the name of the global project into which the global objects will be imported. It is mandatory to provide this argument when import json file includes global objects. Only one global project name can be specified with this argument.

-i InJSONFileName

Specify the import json file along with the absolute path for which the override options file needs to be created. It is mandatory to provide this argument. The user running the dicmd command should have permission to read this file.

-f OutJSONFileName

Specify the name along with the absolute path for options file to be created. It is mandatory to provide this argument. The user running the dicmd command should have permission to write the file in the path specified. If only path is specified then the file will be created with same name as the input import json file.

Privileges

Not restricted by user specific privileges.

Example: Generate options file for the import json file corresponding to a data flow

$ dicmd genoptions -p Project_2 -i /home/disupport/df_oracle_to_pg.json -f /home/disupport/df_oracle_to_pg_ovr.json

ExitCode:0

The generated override file will include relevant details for all the objects which are part of the data flow. For example, below is part of the file corresponding to a PostgreSQL data point. Here, the object name can be identified by the value for the "nm" field. The import option for the object needs to be specified against the "change_Ind" field. If rename option is specified for the object then specify the new name against "reName" field. Remaining fields are only to map the object back to the original import file.

{

"obj_Id" : 6003,

"nm" : "POSTGRES_Conn",

"aliasNm" : "POSTGRESQL",

"layer_Id" : -1,

"layerNm" : "",

"fldr_Id" : -1,

"fldrNm" : "",

"reusableFlg" : "N",

"obj_Cd" : "",

"parent_Ids" : [ ],

"imageUrl" : "",

"change_Ind" : "E",

"reName" : "POSTGRES_Conn",

"mapped_Id" : -1,

"selected" : true,

"groupNm" : "",

"exists" : true,

"obj_Type" : "",

"mappedNm" : "",

"ref_Cd" : "POSTGRES_Conn",

"dbTyp_Id" : 4,

"layers" : [ ],

"conn_Db_Nm" : "",

"conn_Db_Id" : -1,

"mapped_Conn_Db_Id" : -1,

"streamingFlg" : "N"

}

  • cleanup: This option is used to perform adhoc cleanup of Diyotta logs, temporary data files, temporary tables and operational logs. Diyotta generates logs and entries in metadata operational tables for each execution of the data flow or job flow. Similarly, temporary stage files and stage tables are not deleted for the failed execution of data flow or job flow. These need to be cleaned up periodically by running the cleanup option. 

Syntax

dicmd cleanup [-u Username -w UserPassword] -c tform|local|applogs|opsruns|srvlogs|stage [-p ProjectNm] [-n days]

Arguments

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-c tform|local|applogs|opsruns|srvlogs|stage

Specify the type of cleanup that needs to be performed. You can specify one of these types with this argument.

tform – To drop temporary tables from assigned tform database/schema created during execution of data flow.local - To drop transient tables created during execution of data flow.stage - To delete left over data files from failed jobs created when extracting data from external systems.
applogs - To delete monitor log files created to log and display the progress of data flow and job flow execution.opsruns - To delete the entries from metadata operational tables created for data flow, job flow and scheduler task execution.srvlogs - To delete the Diyotta generated Controller and Agent logs.

-p ProjectName (optional)

Specify the name of the project which needs to be referred when cleaning up. Only one project name can be specified with this argument. If this argument is provided then the cleanup will be performed only for those tables and files that were created as part of execution of jobs in the project specified. If this argument is not specified then cleanup will be performed irrespective of the project where the jobs were executed.

-n NumberOfDays

Specify the number of days prior to which the files, tables and operation table entries created should be deleted. The number of days of history that needs to be shown in run history of Monitor and Scheduler is set in Diyotta Admin. Make sure the value specified with this argument for the cleanup types applogs and opsruns is more than that defined in Admin.

Privileges

The User Requires "Studio Write" privilege to perform this action.

Example: The below command cleans up the stage files older than a day from the project Project_1

$ dicmd cleanup -c stage -p Project_1 -n 1

No.of Files Deleted From Controller : 0

No.of Files Deleted From Agent[test] : 0

No.of Files Deleted From Agent[Default] : 21

ExitCode:0

  • passwd: This option is used to mask any text passed as argument to it. This is usually used to mask the plain text password for Diyotta login ids before adding in the profile file or using in CLI arguments.

Syntax

dicmd passwd -e 'TextToEncrypt'

Arguments

-e 'TextToEncrypt'

Specify the text that needs to be masked or encrypted. 

Privileges

Not restricted by user specific privileges.

Example: Generate masked text for a password.

$ dicmd passwd -e 'P2wd_4321'

M2FThB8QQAsv8fTo9KxhnSmLsjXgI18POI0qCQDUYiE=

ExitCode:0

  • genkey: This option is used to generate RSA, AES and PGP encryption keys. These are used to configure Diyotta agents to encrypt the data at source while extracting and decrypt the data at target before loading.

Syntax 

dicmd genkey rsa|pgp|aes

Arguments

-genkey rsa|pgp|aes

Choose the type of key to be and used for encryption. Only one of these types can be specified with this argument.

rsa - Generates keys based on rsa algorithm. This is a asymmetric key and generates a public and private key files.pgp - Generates keys based on pgp algorithm. This will prompt to provide identity of the signer, secure passphrase and the agent to which the key will be associated. This generates public and private key along with the secret key.aes - Generates keys based on aes algorithm. This is a symmetric key. This will prompt to provide passphrase, key size and the agent to which the key will be associated. This generates a single key file.

Privileges

Not restricted by user specific privileges.

Example 1: Generate aes 192 key for agent named Default.

$ dicmd genkey aes

Enter AES Passphrase: gvrt_84s

Enter AES keysize(128/192/256): 192

Enter Agent Name: Default

diaes Key generated at /home/disupport/Default_diaes.key

Example 2: Generate rsa key.

$ dicmd genkey rsa

PublicKey generated at /home/disupport/dipublic.key

PrivateKey generated at /home/disupport/diprivate.key

Example 3: Generate pgp key for agent named Default.

$ dicmd genkey pgp

Enter Identity: Administrator

Enter Passphrase: gvrt_84s

Enter Agent Name: Default

PublicKey generated at /home/disupport/PGPdipublic.key

PrivateKey generated at /home/disupport/PGPdiprivate.key

SecretKey generated at /home/disupport/Default_secretKey.key

  • encrypt: This option is used to futher encrypt AES Key generated by genkey option using the RSA public key. The RSA key used to encrypt could be the one generated using genkey option or generated externally.  

Syntax

dicmd encrypt -f aesKeyFile -k rsaPublicKeyFile

Arguments

-f aesKeyFile

Specify the AES key file with fully qualified path to be encrypted. The command will prompt to specify the agent name to which the encrypted key file generated will be associated.

-k rsaPublicKeyFile

Specify the RSA public key file to be used to encrypt the AES key.

Privileges

Not restricted by user specific privileges.

Example: Generate encrypted AES key file.

$ dicmd encrypt -f /home/disupport/Default_diaes.key -k /home/disupport/dipublic.key

Enter Agent Name:default

diaes_enc key generated at /home/disupport/default_diaes_enc.key

  • scheduler: This option is used to pause and resume the entire scheduler or a specific task or task group in a project. 

Syntax 

dicmd scheduler [-u Username -w UserPassword] -c pause|resume [-p ProjectName -t TaskName|TaskGroupName]

Arguments

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-c pause|resume

Specify if pause or resume operation needs to be performed. When paused then, the active tasks will complete the execution and wait for the scheduler to resume for next scheduled run to execute.

-p ProjectName (optional)

Specify the name of the project to which the task or task group specified belongs. It is mandatory to provide this argument if task or task group name argument is provided. Only one project name can be specified with this argument.

-t TaskName|TaskGroupName (optional)

Specify either task name or task group name to be paused or resumed. It is mandatory to provide the project name argument if this argument is provided.

Privileges

The user requires "Scheduler Execute" privilege to perform this action.

Example: Pause a specific task group in the scheduler

$ dicmd scheduler -c pause -p Project_1 -t tg_sales_dm_incremental

ExitCode:0

  • refresh: This option is used to refresh the Diyotta Lineage component. If there are any modifications to the code in Diyotta Studio then the lineage needs to be refreshed for these changes to take effect.

Syntax

dicmd refresh [-u Username -w UserPassword] -c mvw

Arguments 

-u Username (optional) 

Username of the user. This parameter is required if username is not predefined in the profile file.

-w UserPassword (optional)  

Password of the user. This parameter is required if username parameter is used in the command.

-c mvw

This argument refreshes the lineage component.

Privileges

The user requires "Lineage Read" and "Lineage Write" privilege for performing this action.

Example: Refresh the view in lineage.

$ dicmd refresh -c mvw

ExitCode:0

  • No labels