Below are the details of how each Diyotta component is integrated into the high availability architecture.
- The Diyotta Controller is setup in active-passive mode. At any point only one Controller will be active and accessible to the user.
- A heartbeat mechanism is setup between the active and passive Controllers. The heartbeat is a program that runs on both the Controller instances.
- The Controller that is started first marks itself as active. When started it starts the heartbeat program as well. The other instance when started, marks itself as passive. Only heartbeat program starts on the passive Controller.
- The heartbeat on the active Controller checks the status of itself at regular interval. As soon as it notices any issue with functioning of the Controller it initiates failover. It force shuts down the active Controller and waits for the passive Controller to become active to mark itself as passive.
- The heartbeat on the passive Controllers keeps checking the availability of the active Controller. When it notices that the active Controller is not accessible then it starts the passive Controller and marks it as active. The active Controller can become inaccessible if the heartbeat on the active Controller force shut it down or if the server on which the active Controller was running crashed.
- The active and passive Controller share single metadata repository. The PostgreSQL database needs to be setup externally for this and should be accessible to both the Controllers. Only the active Controller would access the metadata repository.
- A heartbeat mechanism is setup between the Agents and the Controllers. The heartbeat is a program that runs on the Agent and checks the status of the Controller.
- During startup the Agent detects the Controller that is active and brings itself up by pointing to it.
- When the heartbeat program detects that the active Controller is not accessible then it initiates failover. The heartbeat waits for the passive Controller to become active and restarts the Agents by pointing to the now active Controller.
- If the heartbeat cannot access both the Controllers after the number of retries specified in configuration then, it force shuts down the Agent and the heartbeat program.
- The user can access only the active Controller at any time. The URL for two Controllers would be different based on the host and ports assigned to them.
- To abstract the difference in the URL for two Controllers to the user, virtual IP or load-balancer can be setup to resolve to the active Controller URL. The user will always use this virtual URL to connect to the Controller. The Controller to which the virtual URL should be redirected can be determined by looking for the accessible startup port of the Controller.