ee.telekom.workflow.executor.lifecycle

Class HealthCheckServiceImpl

java.lang.Object

ee.telekom.workflow.executor.lifecycle.HealthCheckServiceImpl

All Implemented Interfaces:

HealthCheckService

@Component
public class HealthCheckServiceImpl
extends Object
implements HealthCheckService

Constructor Summary

Constructors

Constructor and Description
HealthCheckServiceImpl()

Method Summary

Modifier and Type	Method and Description
void	healFailedNodes() "Cluster healing" is meant to repair inconsistent database state that results from an unclean cluster shutdown or network hardware failure.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HealthCheckServiceImpl

public HealthCheckServiceImpl()

Method Detail

healFailedNodes

public void healFailedNodes()

"Cluster healing" is meant to repair inconsistent database state that results from an unclean cluster shutdown or network hardware failure. Inconsistent database state results from non-transactional operations in the cluster, such as adding/taking work units to/from the work unit queue. Let's recall the possible states of (locked, nodeName) of a work instance and how they relate to the non-transactional operations.

1. After being a added to the work unit queue -> (true, null)
2. After being taken from the work unit queue -> (true, 'the consuming node's name)
3. After the work unit is successfully -> (false, null).

Scenario 1: A node takes a work unit from the queue and updates the associated workflow instance node_name field. Maybe it also updates the workflow instance's or work item's status field. Afterwards the node is found dead/failed.
Scenario 2: The master node locks a workflow instance but fails before it can add the workflow instance to the work unit queue.

Resolution for scenario 1:

1. If the node updated the workflow instance's or the work item's status field, then this field is left in the temporary execution status and needs to be recovered. For workflow instances: STARTING->NEW or ABORTING->ABORT. For work items: EXECUTING->(is task ? NEW : ERROR) or COMPLETING->EXECUTED.
2. The fields (locked, nodeName) need to be set to (false, null).

NB! EXECUTING task work items cannot be automatically recovered, since their execution is not guaranteed to be transactional. Therefore, they need to be handled manually. To this end, their status field is set to EXECUTING_ERROR and an error message is created in the EXER table.

Resolution for scenario 2:
First of all, we need to make sure that the working queue is empty and that every consumer has had sufficient time to assign its most recently taken work unit to itself. To this extend, every consumer is granted the so called "maximum node assignment time."

Two different kind of errors may cause the work unit queue to be empty for at least the duration of the maximum node assignment and at the same time (locked, nodeName) = (true, null) exist. The first kind is that the distributed queue failed. The second kind is that the node which took the element from the queue failed between taking the element and assigning the process execution to itself.

The recovery of this scenario is expensive because we need to suspend the producer, wait for the queue to become empty, wait for the maximum node assignment grace period, do the* recovery and resume the producer. For this reason, this advanced recovery is not run on every health check.

Specified by:

healFailedNodes in interface HealthCheckService