Skip to content

ClusterObjectSets

This document explains what ClusterObjectSets are, how they enable safe phased rollouts of Kubernetes resources, and how operator-controller uses them to manage ClusterExtensions.

What is a ClusterObjectSet?

A ClusterObjectSet is a cluster-scoped Kubernetes API that represents a versioned set of Kubernetes resources organized into ordered phases. It provides a declarative way to roll out a group of related resources sequentially, with built-in readiness checks between phases.

The revision content — the group identifier, revision number, collisionProtection strategy, phases (including all objects within them), and optional progressionProbes — is immutable from creation. This guarantees that the record of what was deployed at a given revision cannot change after creation. Progression probes cannot be added, changed, or removed after creation. Other fields like lifecycleState can change over the object's lifecycle (e.g. transitioning from Active to Archived).

Each ClusterObjectSet has:

  • A group — a required, immutable identifier linking revisions into one sequence, regardless of their owner references
  • A revision number — an immutable, sequential integer that identifies the version
  • A lifecycle state — either Active (being reconciled) or Archived (inactive); transitions from Active to Archived but not back
  • A list of phases — immutable, ordered groups of Kubernetes objects that are applied sequentially
  • A collision protection strategy — immutable, controlling how pre-existing objects are handled
  • A Ready condition — reporting resource reconciliation, with first rollout completion recorded separately in status.completedAt

ClusterObjectSets can be used by any controller or system that needs to manage the rollout of a set of Kubernetes resources in a controlled, phased manner. Within OLM, when BoxcutterRuntime is enabled, operator-controller creates and updates a ClusterObjectDeployment (COD) for each ClusterExtension. The COD controller creates the immutable ClusterObjectSet revisions that deploy and upgrade the extension.

For deployment-level revision management and rollout policy, see ClusterObjectDeployments. For a standalone walkthrough that manages mutable desired content through COS revisions, see Manage standalone ClusterObjectDeployments.

The group value must be 1–52 characters long, start with a lowercase letter, contain only lowercase letters, digits or hyphens, and end with a letter or digit. Set metadata.name explicitly to <group>-<revision>, with the revision in decimal and without leading zeros. Revisions range from 1 to 9999999999, inclusive: ten digits plus the separating hyphen fit within a 63-character name even for a 52-character group. This naming rule ensures that a group cannot contain duplicate revision numbers. Name generation through metadata.generateName is not supported.

Revisions in the same group can transfer managed objects between them even when they have different controller owners or no owner references. They share the server-side apply field manager cos-group/<group>. Different groups remain subject to normal object collision protection.

Revision membership and management

The immutable spec.group identifies a COS's revision chain. Controller owner references record management relationships: a COD owns the COS revisions it creates, and each COS owns the Kubernetes objects it manages.

A standalone COS, a COD-owned COS, and a COS owned by another controller are siblings when their groups match. Adding, changing, or removing a COS's controller owner reference leaves its revision chain unchanged. For example, deleting a COD with orphan propagation removes its owner references from the surviving COSs; those COSs remain in the same group and retain the same handoff behavior. Archived and deleting COSs are excluded from the active sibling list.

Controller deployment

The experimental object-controller runs in its own Deployment with its own service account and leader-election lease. It watches ClusterObjectDeployments, ClusterObjectSets, and the objects they manage; it does not require ClusterExtension or ClusterCatalog resources or controllers. When BoxcutterRuntime is enabled, operator-controller resolves and unpacks bundles, applies ClusterObjectDeployments, and observes their rollout status.

The Helm chart automatically enables object-controller alongside operator-controller when BoxcutterRuntime is enabled. To deploy it independently, first build an image and make it available to your cluster (push it to a registry, or load it into your local cluster). Set the image reference below, then render and apply the manifests:

helm template object-controller helm/olmv1 \
  --set options.featureSet=experimental \
  --set options.objectController.enabled=true \
  --set options.operatorController.enabled=false \
  --set options.catalogd.enabled=false \
  --set-string options.objectController.deployment.image='<your-built-image>' \
  > object-controller.yaml
kubectl apply -f object-controller.yaml
kubectl -n olmv1-system rollout status deployment/object-controller-controller-manager --timeout=180s

Set options.objectController.deployment.image to the built image when using a local or downstream image. make go-build-local builds bin/object-controller; make docker-build builds its image alongside the existing controller images. OBJECT_CONTROLLER_IMAGE_REPO controls its build and release repository.

Object-controller reads immutable referenced Secrets directly from the API server in the namespace specified by each reference. Metrics require a TLS certificate and key; the chart configures these through cert-manager or the OpenShift service CA when enabled. The minimal standalone example above enables neither certificate provider, so metrics are disabled. Health and readiness probes remain enabled. The default service account has cluster-admin privileges because ClusterObjectSets can manage arbitrary Kubernetes resources, matching the existing experimental runtime.

For downstream builds enabling BoxcutterRuntime, package the new binary and configure its image in addition to the operator-controller image. It is possible to package both binaries in one image while running them in separate Deployments. Standard installations and default Helm installations without BoxcutterRuntime do not enable object-controller.

Why ClusterObjectSets?

ClusterObjectSets solve several problems that arise when managing sets of related Kubernetes resources:

  • Immutable revision content — The revision number, phases, objects, and collision protection strategy are immutable once set, providing a clear, auditable record of exactly what was deployed at each revision. There is no ambiguity about what resources belong to a given version.
  • Phased rollout — Resources are grouped into phases (e.g. CRDs before Deployments) and applied sequentially. A phase only progresses after all its objects pass readiness probes, preventing partially-applied states.
  • Safe transitions — During a revision change, both the old and new revisions remain active until the new revision fully rolls out. Object ownership transitions from one revision to the next, ensuring no resource is left unmanaged.
  • Single ownership — Each Kubernetes resource can only be managed by one ClusterObjectSet at a time. This prevents conflicts between revisions fighting over the same object.
  • Large resource support — Object manifests can be stored inline or externalized into Secrets via references, allowing ClusterObjectSets to manage bundles that would otherwise exceed etcd's 1.5 MiB object size limit.

Lifecycle

Active state

When a ClusterObjectSet is Active, the controller actively reconciles it using the boxcutter library:

  • Phases are applied sequentially, each waiting for readiness before proceeding
  • Objects are managed via server-side apply
  • Status conditions are updated to reflect rollout progress

Revision transitions

When transitioning from one revision to the next:

  1. A new ClusterObjectSet is created with the same group and the next sequential revision number
  2. Both old and new revisions remain Active during the transition
  3. Objects transition ownership from the old revision to the new one
  4. Once the new revision has rolled out successfully, the old revision can be archived

Archival

When a revision's lifecycle state is set to Archived:

  • It is removed from the owner list of all previously managed objects
  • Objects that did not transition to a succeeding revision are deleted
  • The revision cannot be un-archived
  • It remains in the cluster for historical reference until garbage collected

Phases

Objects within a ClusterObjectSet are organized into phases. Each phase groups related resources, and phases are applied sequentially. Within a phase, all objects are applied simultaneously in no particular order.

A phase only progresses to the next after all of its objects pass their readiness probes. This ensures dependencies are satisfied before dependents are created — for example, CRDs are established before Deployments that use custom resources.

Phase names follow the DNS label standard (RFC 1123) and must be unique within a ClusterObjectSet. A ClusterObjectSet supports up to 20 phases, with up to 50 objects per phase.

Phase ordering in operator-controller

When operator-controller renders a ClusterObjectDeployment template for a ClusterExtension, it automatically assigns objects to phases based on their GroupKind. The COD controller copies those phases into each COS revision:

Order Phase Resource Kinds
1 namespaces Namespace
2 policies NetworkPolicy, PodDisruptionBudget, PriorityClass
3 identity ServiceAccount
4 configuration Secret, ConfigMap
5 storage PersistentVolume, PersistentVolumeClaim, StorageClass
6 crds CustomResourceDefinition
7 roles ClusterRole, Role
8 bindings ClusterRoleBinding, RoleBinding
9 infrastructure Service, Issuer (cert-manager)
10 deploy Certificate (cert-manager), Deployment
11 scaling VerticalPodAutoscaler
12 publish PrometheusRule, ServiceMonitor, PodMonitor, Ingress, Route, ConsoleYAMLSample, ConsoleQuickStart, ConsoleCLIDownload, ConsoleLink, ConsolePlugin
13 admission ValidatingWebhookConfiguration, MutatingWebhookConfiguration

Any resource kind not listed above defaults to the deploy phase.

Note

This phase ordering is specific to how operator-controller renders COD templates. The APIs do not enforce any particular phase ordering — phases are applied in the order they appear in the template's spec.phases list.

Readiness probes

A phase only progresses to the next after all of its objects pass readiness probes. Several resource kinds have built-in probes:

Resource Kind Readiness Criteria
CustomResourceDefinition Condition Established = True
Namespace status.phase = "Active"
PersistentVolumeClaim status.phase = "Bound"
Deployment status.updatedReplicas == status.replicas and condition Available = True
StatefulSet status.updatedReplicas == status.replicas and condition Available = True
Certificate (cert-manager) Condition Ready = True
Issuer (cert-manager) Condition Ready = True

For custom resources or other objects that need tailored checks, you can define custom progression probes in the spec.progressionProbes field. These are experimental and support three assertion types:

ConditionEqual
Checks that an object has a condition of the specified type and status (e.g. Ready = True).
FieldsEqual
Checks that the values at two field paths match (e.g. spec.replicas == status.readyReplicas).
FieldValue
Checks that a field has a specific value (e.g. status.phase = "Bound").

Probes use selectors to target objects by GroupKind or by label. A probe only runs against objects matching its selector — if no objects in a phase match, the probe is considered to have passed.

Collision protection

Collision protection controls whether a ClusterObjectSet can adopt pre-existing objects on the cluster. This is configured at three levels, with the most specific taking precedence:

object > phase > spec

The available strategies are:

Prevent
Only manages objects the revision created itself. This is the safest option and prevents ownership collisions entirely.
IfNoController
Can adopt pre-existing objects that are not owned by another controller. Useful when taking over management of manually-created resources.
None
Can adopt any pre-existing object, even if owned by another controller. Use with extreme caution — this can cause multiple controllers to fight over the same resource.

Object storage: inline vs. references

Each object in a phase must set type and its corresponding content field. Exactly one of inlineManifest or secretRef must be set:

type: InlineManifest with inlineManifest
The full Kubernetes manifest is embedded inline in the ClusterObjectSet. Simple and self-contained, but contributes to the overall size of the ClusterObjectSet resource in etcd.
type: SecretReference with secretRef
A reference to an immutable Secret that holds the serialized object manifest. The secretRef specifies the Secret name, namespace, and data key containing either a JSON-encoded manifest or gzip-compressed JSON bytes. This allows ClusterObjectSets to manage bundles that would otherwise exceed etcd's 1.5 MiB object size limit. Object-controller reads referenced Secrets directly from the API server in their declared namespaces.

When operator-controller renders ClusterObjectDeployment templates for ClusterExtensions, it automatically externalizes objects into immutable Secrets:

  • Secrets are packed up to 900 KiB each, leaving headroom below the etcd limit
  • Objects larger than 900 KiB are gzip-compressed before storage (ref resolution auto-detects and transparently decompresses gzip-compressed values so consumers see the original JSON manifest)
  • Content-addressable naming (based on SHA-256 hashes of the data) ensures that Secret names and data keys are deterministic, making creation idempotent and safe to retry

Referenced Secret lifetime

For ClusterExtension-managed deployments, operator-controller creates the referenced Secrets before applying the COD. Each Secret is owned by the ClusterExtension, not by the COD or an individual COS. Secret names use the extension name and a content hash, so unchanged packed content can be reused across revisions. Pruning archived COS revisions does not delete these Secrets; they are garbage collected when the owning ClusterExtension is deleted.

Standalone COD and COS callers manage the lifetime of their referenced Secrets. Keep each Secret available for every retained revision that references it. Do not assign a shared Secret exclusively to one COS if deleting that COS would remove content still needed by other revisions. A content change requires a new Secret reference, not an in-place edit to an immutable Secret.

For historical design alternatives, see Large Bundle Support. The current behavior is described here and in the standalone COD tutorial.

Status conditions

ClusterObjectSets report a single Ready condition describing their current state. Rollout completion is recorded separately as a timestamp in status.completedAt, not as a condition.

Ready

Indicates whether all objects in the revision have been successfully rolled out and reached their desired state.

Status Reason Meaning
True AllObjectsReady All objects are at the desired state
False ObjectsNotReady One or more objects are still being applied or have not yet passed their readiness probes during rollout
False Superseded One or more objects have been adopted by a newer revision of the object set
False ObjectCollision One or more objects are controlled by another owner, or are unowned and collision protection prevents adoption
False PreconditionsNotMet A precondition for applying the revision is not yet satisfied (a referenced Secret is mutable or has invalid or changed content, or a preflight dry-run was rejected, e.g. an immutable field, a conflicting object, or RBAC/dependency CRDs not yet propagated); the revision keeps retrying and recovers once the precondition is met
False InvalidSpecification The revision spec is invalid (a malformed revision/phase object or progression probe); requires a spec change
False Archiving The revision is being archived; owned objects are still being torn down
False Archived Objects torn down after archival
Unknown Pending The revision has not yet been evaluated (initial state)

On a revision's first reconciliation the Ready condition is initialized to Unknown/Pending, so it is always present. A transient internal error (e.g. an API read failure) leaves the Ready condition unchanged and is retried with backoff; it is not reported as a distinct reason (while the revision is still Pending, it stays Unknown/Pending).

Completion (status.completedAt)

The first time a revision becomes ready, the controller stamps status.completedAt with the current time. This timestamp is set once and never changes, marking that this revision successfully rolled out at least once — it persists even if the revision later becomes not ready. The ClusterExtension controller uses it to determine the installed revision.

For deployment-level rollout policy and conditions, see ClusterObjectDeployments.

How operator-controller uses ClusterObjectSets

With BoxcutterRuntime enabled, ClusterObjectSets serve as the deployment mechanism for ClusterExtensions. The operator-controller is one consumer of the ClusterObjectSet API.

Installation

When a ClusterExtension is installed:

  1. The operator-controller resolves a bundle from the catalog
  2. The bundle's Kubernetes manifests are sorted into phases by resource kind
  3. Objects are packed into immutable Secrets
  4. The operator-controller applies a ClusterObjectDeployment with the desired content in its template
  5. When a new revision is needed, the COD controller creates an Active ClusterObjectSet with a revision number one greater than the highest existing revision in the group (1 for an empty group)
  6. Phases roll out sequentially — each phase waits for all its objects to become ready before the next phase begins

Upgrades

When a ClusterExtension is upgraded to a new version:

  1. The operator-controller updates the ClusterObjectDeployment's template with the new bundle content
  2. The COD controller creates a new ClusterObjectSet with a revision number one greater than the highest existing revision in the group, including archived and foreign-owned ClusterObjectSets
  3. Both old and new revisions are Active during the transition
  4. Objects transition ownership from the old revision to the new one
  5. Once the new revision has rolled out successfully, the old revision is archived
  6. The COD controller garbage collects older archived revisions, keeping the last 5 by default for auditing

Once a group reaches revision 9999999999, the ClusterObjectDeployment controller reports an error instead of allocating another revision. Reconciliation of unchanged content still works when no higher active, non-deleting COS is controlled by another controller.

Relationship with ClusterExtension

A ClusterExtension owns a ClusterObjectDeployment with the same name. The COD owns the ClusterObjectSet revisions it creates, and each COS owns its managed Kubernetes objects. These controller owner references enable cascading garbage collection when the extension is deleted.

The COD controller sets each revision's spec.group to the COD's name, which is also the ClusterExtension's name. Revision lookup uses the group; labels and owner references do not determine membership.

The COD template propagates labels that identify the parent extension to its COS revisions:

metadata:
  labels:
    olm.operatorframework.io/owner-kind: ClusterExtension
    olm.operatorframework.io/owner-name: my-extension

Examples

Inline objects

This example shows a ClusterObjectSet with objects embedded directly in the spec. This is the simplest approach and works well when the total size of all objects stays within etcd's 1.5 MiB limit. The first phase creates the my-app-system namespace before the namespaced resources in later phases. Replace the illustrative controller image with one available to your cluster.

apiVersion: olm.operatorframework.io/v1
kind: ClusterObjectSet
metadata:
  name: my-app-1
spec:
  group: my-app
  revision: 1
  lifecycleState: Active
  collisionProtection: Prevent
  phases:
  - name: namespaces
    objects:
    - type: InlineManifest
      inlineManifest:
        apiVersion: v1
        kind: Namespace
        metadata:
          name: my-app-system
  - name: crds
    objects:
    - type: InlineManifest
      inlineManifest:
        apiVersion: apiextensions.k8s.io/v1
        kind: CustomResourceDefinition
        metadata:
          name: widgets.example.com
        spec:
          group: example.com
          names:
            kind: Widget
            plural: widgets
          scope: Namespaced
          versions:
          - name: v1
            served: true
            storage: true
            schema:
              openAPIV3Schema:
                type: object
                properties:
                  spec:
                    type: object
  - name: rbac
    objects:
    - type: InlineManifest
      inlineManifest:
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: my-app-controller
          namespace: my-app-system
    - type: InlineManifest
      inlineManifest:
        apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRole
        metadata:
          name: my-app-manager-role
        rules:
        - apiGroups: ["example.com"]
          resources: ["widgets"]
          verbs: ["get", "list", "watch", "create", "update", "delete"]
  - name: deploy
    objects:
    - type: InlineManifest
      inlineManifest:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: my-app-controller
          namespace: my-app-system
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: my-app-controller
          template:
            metadata:
              labels:
                app: my-app-controller
            spec:
              serviceAccountName: my-app-controller
              containers:
              - name: manager
                image: example.com/my-app-controller:v1.0.0

Secret references

This example shows a ClusterObjectSet that references objects stored in Secrets. Operator-controller uses the same reference mechanism in COD templates, which the COD controller copies into COS revisions. It is necessary when managing bundles with large resources like CRDs with extensive schemas.

For ClusterExtension-managed content, Secret names are content-addressable, computed as <extensionName>-<16 hex chars> where the suffix is the first 8 bytes of a SHA-256 digest over the Secret's sorted keys and values. Each data key is a 43-character base64url-encoded (no padding) SHA-256 hash of the stored object bytes.

The following Secrets and hashes are illustrative placeholders, not a runnable manifest. Standalone callers may choose their own Secret names. These examples omit owner references; a standalone producer is responsible for their cleanup. Operator-controller instead sets a controller owner reference to the ClusterExtension, as described in Referenced Secret lifetime.

apiVersion: v1
kind: Secret
metadata:
  name: my-app-3a7f1b9c0e2d4f68
  namespace: olmv1-system
immutable: true
data:
  # key: 43-char base64url SHA-256 of the object JSON bytes
  # value: base64-encoded JSON-serialized Kubernetes manifest
  K8sTl2xQa0vNpR7mW4dYcJfHbE9gUiAoZX1sDj6wFCy: <base64-encoded CRD JSON>
---
apiVersion: v1
kind: Secret
metadata:
  name: my-app-8b2e4a6c1d3f5097
  namespace: olmv1-system
immutable: true
data:
  # Multiple objects can be packed into the same Secret
  Pm5tRqLwXv8sCj3kFdNhYe7bUiA0oZG1x2W9acDfHJy: <base64-encoded ServiceAccount JSON>
  Vn4mTsLwXp8rCk3jFdNhYe7bUiA0oZG1x2W9acDfHQy: <base64-encoded ClusterRole JSON>
  Bq9nRtLxYw2sDk4lGePhZf8cVjB1pUiA0oZH3x5WaCdy: <base64-encoded Deployment JSON>

The ClusterObjectSet references these Secrets:

apiVersion: olm.operatorframework.io/v1
kind: ClusterObjectSet
metadata:
  name: my-app-1
spec:
  group: my-app
  revision: 1
  lifecycleState: Active
  collisionProtection: Prevent
  phases:
  - name: crds
    objects:
    - type: SecretReference
      secretRef:
        name: my-app-3a7f1b9c0e2d4f68
        namespace: olmv1-system
        key: "K8sTl2xQa0vNpR7mW4dYcJfHbE9gUiAoZX1sDj6wFCy"
  - name: rbac
    objects:
    - type: SecretReference
      secretRef:
        name: my-app-8b2e4a6c1d3f5097
        namespace: olmv1-system
        key: "Pm5tRqLwXv8sCj3kFdNhYe7bUiA0oZG1x2W9acDfHJy"
    - type: SecretReference
      secretRef:
        name: my-app-8b2e4a6c1d3f5097
        namespace: olmv1-system
        key: "Vn4mTsLwXp8rCk3jFdNhYe7bUiA0oZG1x2W9acDfHQy"
  - name: deploy
    objects:
    - type: SecretReference
      secretRef:
        name: my-app-8b2e4a6c1d3f5097
        namespace: olmv1-system
        key: "Bq9nRtLxYw2sDk4lGePhZf8cVjB1pUiA0oZH3x5WaCdy"

Multiple objects can reference different keys within the same Secret, allowing efficient packing of smaller objects.

Inspecting ClusterObjectSets

# List all ClusterObjectSets
kubectl get clusterobjectsets

# List revisions in a specific extension's group
kubectl get clusterobjectsets --field-selector spec.group=my-extension

# View full details for a specific revision
kubectl get clusterobjectset <name> -o yaml

# Inspect reconciliation in the independent controller
kubectl -n olmv1-system logs deployment/object-controller-controller-manager

Example output:

NAME                   GROUP          READY     AGE
my-extension-1         my-extension   Unknown   2d
my-extension-2         my-extension   True      1h