Skip to content

Commit 39818f8

Browse files
csviricsviri
committed
feat: add default delete condition to managed dependent resources
Signed-off-by: Attila Mészáros <[email protected]>
1 parent ab89e69 commit 39818f8

File tree

8 files changed

+229
-1
lines changed

8 files changed

+229
-1
lines changed

‎docs/documentation/workflows.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -312,6 +312,28 @@ containing all the related exceptions.
312312
The exceptions can be handled
313313
by [`ErrorStatusHandler`](https://github.com/java-operator-sdk/java-operator-sdk/blob/14620657fcacc8254bb96b4293eded84c20ba685/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/reconciler/ErrorStatusHandler.java)
314314

315+
## Waiting for the actual deletion of Kubernetes Dependent Resources
316+
317+
Let's consider a case when a Kubernetes Dependent Resources (KDR) depends on another resource, on cleanup
318+
the resources will be deleted in reverse order, thus the KDR will be deleted first.
319+
However, the workflow implementation currently simply asks the Kubernetes API server to delete the resource. This is,
320+
however, an asynchronous process, meaning that the deletion might not occur immediately, in particular if the resource
321+
uses finalizers that block the deletion or if the deletion itself takes some time. From the SDK's perspective, though,
322+
the deletion has been requested and it moves on to other tasks without waiting for the resource to be actually deleted
323+
from the server (which might never occur if it uses finalizers which are not removed).
324+
In situations like these, if your logic depends on resources being actually removed from the cluster before a
325+
cleanup workflow can proceed correctly, you need to block the workflow progression using a delete post-condition that
326+
checks that the resource is actually removed or that it, at least, doesn't have any finalizers any longer. JOSDK
327+
provides such a delete post-condition implementation in the form of
328+
[`KubernetesResourceDeletedCondition`](https://github.com/java-operator-sdk/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/workflow/KubernetesResourceDeletedCondition.java)
329+
330+
Also, check usage in an [integration test](https://github.com/java-operator-sdk/java-operator-sdk/blob/main/operator-framework/src/test/java/io/javaoperatorsdk/operator/sample/manageddependentdeletecondition/ManagedDependentDefaultDeleteConditionReconciler.java).
331+
332+
In such cases the Kubernetes Dependent Resource should extend `CRUDNoGCKubernetesDependentResource`
333+
and NOT `CRUDKubernetesDependentResource` since otherwise the Kubernetes Garbage Collector would delete the resources.
334+
In other words if a Kubernetes Dependent Resource depends on another dependent resource, it should not implement
335+
`GargageCollected` interface, otherwise the deletion order won't be guaranteed.
336+
315337
## Notes and Caveats
316338

317339
- Delete is almost always called on every resource during the cleanup. However, it might be the case

‎operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/workflow/DefaultManagedWorkflow.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ public Workflow<P> resolve(KubernetesClient client,
8585
resolve(spec, client, configuration));
8686
alreadyResolved.put(node.getName(), node);
8787
spec.getDependsOn()
88-
.forEach(depend -> node.addDependsOnRelation(alreadyResolved.get((String) depend)));
88+
.forEach(depend -> node.addDependsOnRelation(alreadyResolved.get(depend)));
8989
}
9090

9191
final var bottom =

‎operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/workflow/KubernetesResourceDeletedCondition.java

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
package io.javaoperatorsdk.operator.processing.dependent.workflow;
2+
3+
import io.fabric8.kubernetes.api.model.HasMetadata;
4+
import io.javaoperatorsdk.operator.api.reconciler.Context;
5+
import io.javaoperatorsdk.operator.api.reconciler.dependent.DependentResource;
6+
7+
/**
8+
* A condition implementation meant to be used as a delete post-condition on Kubernetes dependent
9+
* resources to prevent the workflow from proceeding until the associated resource is actually
10+
* deleted from the server (or, at least, doesn't have any finalizers anymore). This is needed in
11+
* cases where a cleaning process depends on resources being actually removed from the server
12+
* because, by default, workflows simply request the deletion but do NOT wait for the resources to
13+
* be actually deleted.
14+
*/
15+
public class KubernetesResourceDeletedCondition implements Condition<HasMetadata, HasMetadata> {
16+
17+
@Override
18+
public boolean isMet(DependentResource<HasMetadata, HasMetadata> dependentResource,
19+
HasMetadata primary, Context<HasMetadata> context) {
20+
var optionalResource = dependentResource.getSecondaryResource(primary, context);
21+
if (optionalResource.isEmpty()) {
22+
return true;
23+
} else {
24+
return optionalResource.orElseThrow().getMetadata().getFinalizers().isEmpty();
25+
}
26+
}
27+
}

‎operator-framework/src/test/java/io/javaoperatorsdk/operator/ManagedDependentDeleteConditionIT.java

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
package io.javaoperatorsdk.operator;
2+
3+
import java.time.Duration;
4+
import java.util.Set;
5+
6+
import org.junit.jupiter.api.Test;
7+
import org.junit.jupiter.api.extension.RegisterExtension;
8+
9+
import io.fabric8.kubernetes.api.model.ConfigMap;
10+
import io.fabric8.kubernetes.api.model.ObjectMetaBuilder;
11+
import io.fabric8.kubernetes.api.model.Secret;
12+
import io.javaoperatorsdk.operator.junit.LocallyRunOperatorExtension;
13+
import io.javaoperatorsdk.operator.sample.manageddependentdeletecondition.ManagedDependentDefaultDeleteConditionCustomResource;
14+
import io.javaoperatorsdk.operator.sample.manageddependentdeletecondition.ManagedDependentDefaultDeleteConditionReconciler;
15+
16+
import static org.assertj.core.api.Assertions.assertThat;
17+
import static org.awaitility.Awaitility.await;
18+
19+
public class ManagedDependentDeleteConditionIT {
20+
21+
public static final String RESOURCE_NAME = "test1";
22+
public static final String CUSTOM_FINALIZER = "test/customfinalizer";
23+
24+
@RegisterExtension
25+
LocallyRunOperatorExtension extension =
26+
LocallyRunOperatorExtension.builder()
27+
.withConfigurationService(o -> o.withDefaultNonSSAResource(Set.of()))
28+
.withReconciler(new ManagedDependentDefaultDeleteConditionReconciler()).build();
29+
30+
31+
@Test
32+
void resourceNotDeletedUntilDependentDeleted() {
33+
var resource = new ManagedDependentDefaultDeleteConditionCustomResource();
34+
resource.setMetadata(new ObjectMetaBuilder()
35+
.withName(RESOURCE_NAME)
36+
.build());
37+
resource = extension.create(resource);
38+
39+
await().timeout(Duration.ofSeconds(300)).untilAsserted(() -> {
40+
var cm = extension.get(ConfigMap.class, RESOURCE_NAME);
41+
var sec = extension.get(Secret.class, RESOURCE_NAME);
42+
assertThat(cm).isNotNull();
43+
assertThat(sec).isNotNull();
44+
});
45+
46+
var secret = extension.get(Secret.class, RESOURCE_NAME);
47+
secret.getMetadata().getFinalizers().add(CUSTOM_FINALIZER);
48+
secret = extension.replace(secret);
49+
50+
extension.delete(resource);
51+
52+
// both resources are present until the finalizer removed
53+
await().pollDelay(Duration.ofMillis(250)).untilAsserted(() -> {
54+
var cm = extension.get(ConfigMap.class, RESOURCE_NAME);
55+
var sec = extension.get(Secret.class, RESOURCE_NAME);
56+
assertThat(cm).isNotNull();
57+
assertThat(sec).isNotNull();
58+
});
59+
60+
secret.getMetadata().getFinalizers().clear();
61+
extension.replace(secret);
62+
63+
await().untilAsserted(() -> {
64+
var cm = extension.get(ConfigMap.class, RESOURCE_NAME);
65+
var sec = extension.get(Secret.class, RESOURCE_NAME);
66+
assertThat(cm).isNull();
67+
assertThat(sec).isNull();
68+
});
69+
}
70+
71+
}

‎operator-framework/src/test/java/io/javaoperatorsdk/operator/sample/manageddependentdeletecondition/ConfigMapDependent.java

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
package io.javaoperatorsdk.operator.sample.manageddependentdeletecondition;
2+
3+
import java.util.Map;
4+
5+
import io.fabric8.kubernetes.api.model.ConfigMap;
6+
import io.fabric8.kubernetes.api.model.ConfigMapBuilder;
7+
import io.javaoperatorsdk.operator.api.reconciler.Context;
8+
import io.javaoperatorsdk.operator.processing.dependent.kubernetes.CRUDNoGCKubernetesDependentResource;
9+
10+
public class ConfigMapDependent extends
11+
CRUDNoGCKubernetesDependentResource<ConfigMap, ManagedDependentDefaultDeleteConditionCustomResource> {
12+
13+
public ConfigMapDependent() {
14+
super(ConfigMap.class);
15+
}
16+
17+
@Override
18+
protected ConfigMap desired(ManagedDependentDefaultDeleteConditionCustomResource primary,
19+
Context<ManagedDependentDefaultDeleteConditionCustomResource> context) {
20+
21+
return new ConfigMapBuilder()
22+
.withNewMetadata()
23+
.withName(primary.getMetadata().getName())
24+
.withNamespace(primary.getMetadata().getNamespace())
25+
.endMetadata()
26+
.withData(Map.of("key", "val"))
27+
.build();
28+
}
29+
}

‎operator-framework/src/test/java/io/javaoperatorsdk/operator/sample/manageddependentdeletecondition/ManagedDependentDefaultDeleteConditionCustomResource.java

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
package io.javaoperatorsdk.operator.sample.manageddependentdeletecondition;
2+
3+
import io.fabric8.kubernetes.api.model.Namespaced;
4+
import io.fabric8.kubernetes.client.CustomResource;
5+
import io.fabric8.kubernetes.model.annotation.Group;
6+
import io.fabric8.kubernetes.model.annotation.ShortNames;
7+
import io.fabric8.kubernetes.model.annotation.Version;
8+
9+
@Group("sample.javaoperatorsdk")
10+
@Version("v1")
11+
@ShortNames("mdcc")
12+
public class ManagedDependentDefaultDeleteConditionCustomResource
13+
extends CustomResource<Void, Void>
14+
implements Namespaced {
15+
}

‎operator-framework/src/test/java/io/javaoperatorsdk/operator/sample/manageddependentdeletecondition/ManagedDependentDefaultDeleteConditionReconciler.java

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
package io.javaoperatorsdk.operator.sample.manageddependentdeletecondition;
2+
3+
import org.slf4j.Logger;
4+
import org.slf4j.LoggerFactory;
5+
6+
import io.javaoperatorsdk.operator.api.reconciler.*;
7+
import io.javaoperatorsdk.operator.api.reconciler.dependent.Dependent;
8+
import io.javaoperatorsdk.operator.processing.dependent.workflow.KubernetesResourceDeletedCondition;
9+
10+
@ControllerConfiguration(dependents = {
11+
@Dependent(name = "ConfigMap", type = ConfigMapDependent.class),
12+
@Dependent(type = SecretDependent.class, dependsOn = "ConfigMap",
13+
deletePostcondition = KubernetesResourceDeletedCondition.class)
14+
})
15+
public class ManagedDependentDefaultDeleteConditionReconciler
16+
implements Reconciler<ManagedDependentDefaultDeleteConditionCustomResource> {
17+
18+
private static final Logger log =
19+
LoggerFactory.getLogger(ManagedDependentDefaultDeleteConditionReconciler.class);
20+
21+
@Override
22+
public UpdateControl<ManagedDependentDefaultDeleteConditionCustomResource> reconcile(
23+
ManagedDependentDefaultDeleteConditionCustomResource resource,
24+
Context<ManagedDependentDefaultDeleteConditionCustomResource> context) {
25+
26+
log.debug("Reconciled: {}", resource);
27+
28+
return UpdateControl.noUpdate();
29+
}
30+
31+
}

‎operator-framework/src/test/java/io/javaoperatorsdk/operator/sample/manageddependentdeletecondition/SecretDependent.java

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
package io.javaoperatorsdk.operator.sample.manageddependentdeletecondition;
2+
3+
import java.nio.charset.StandardCharsets;
4+
import java.util.Base64;
5+
import java.util.Map;
6+
7+
import io.fabric8.kubernetes.api.model.Secret;
8+
import io.fabric8.kubernetes.api.model.SecretBuilder;
9+
import io.javaoperatorsdk.operator.api.reconciler.Context;
10+
import io.javaoperatorsdk.operator.processing.dependent.kubernetes.CRUDNoGCKubernetesDependentResource;
11+
12+
public class SecretDependent
13+
extends
14+
CRUDNoGCKubernetesDependentResource<Secret, ManagedDependentDefaultDeleteConditionCustomResource> {
15+
16+
public SecretDependent() {
17+
super(Secret.class);
18+
}
19+
20+
@Override
21+
protected Secret desired(ManagedDependentDefaultDeleteConditionCustomResource primary,
22+
Context<ManagedDependentDefaultDeleteConditionCustomResource> context) {
23+
24+
return new SecretBuilder()
25+
.withNewMetadata()
26+
.withName(primary.getMetadata().getName())
27+
.withNamespace(primary.getMetadata().getNamespace())
28+
.endMetadata()
29+
.withData(Map.of("key",
30+
new String(Base64.getEncoder().encode("val".getBytes(StandardCharsets.UTF_16)))))
31+
.build();
32+
}
33+
}

0 commit comments

Comments
 (0)