Skip to content

Automating Longhorn Recurring Jobs Assignment Using Labels

Article Number: 000021915

Environment

SUSE Enterprise Storage - Longhorn

Situation

As the cluster scales and the number of volumes increases, there is a need to automate the assignment of Longhorn recurring jobs to a set of volumes. To streamline this process, the approach involves using volume labels to group volumes, allowing recurring jobs to be automatically applied based on those labels.

Resolution

Labels on the PVCs can be used to automate the assignment of Longhorn recurring jobs to a set of volumes. Add the labels recurring-job.longhorn.io/source=enabled and recurring-job-group.longhorn.io/<GROUP_NAME>=enabled to the PVCs. For example:

    1. Create a recurring job with a group name (e.g., "test-group" shown below).
    > kubectl get recurringjobs.longhorn.io -n longhorn-system -oyaml
apiVersion: v1
items:
- apiVersion: longhorn.io/v1beta2
  kind: RecurringJob
  metadata:
    creationTimestamp: "2025-07-11T08:08:14Z"
    generation: 3
    name: c-mptbs9
    namespace: longhorn-system
    resourceVersion: "1247498"
    uid: 6d88a149-1ca9-489e-a358-7f2b0d326610
  spec:
    concurrency: 1
    cron: '*/2 * * * *'
    groups:
    - test-group
    labels: {}
    name: c-mptbs9
    parameters: {}
    retain: 0
    task: filesystem-trim
  status:
    executionCount: 28
    ownerID: node-xxxxx
kind: List
metadata:
  resourceVersion: ""

    Above example creates a recurring job for filesystem-trim on the volumes from group test-group. Alternatively, it can be created via the UI: Longhorn UI >> Recurring Job >> Create Recurring Job. 2. Label the PVC with the following labels:

    > kubectl -n longhorn-training get pvc --show-labels|grep test-pvc
test-pvc              Bound    pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f   1Gi        RWO            longhorn       <unset>                 97m   recurring-job-group.longhorn.io/test-group=enabled,recurring-job.longhorn.io/source=enabled
    3. Verify in the longhorn-manager pod logs that the PVC is added to the recurring job:

    kubectl -n longhorn-system logs -l app=longhorn-manager

time="2025-07-11T09:25:34Z" level=info msg="Adding Volume pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f recurring job label recurring-job-group.longhorn.io/test-group: enabled" func=controller.syncRecurringJobLabelsToTargetResource file="recurring_job_controller.go:604" controller=longhorn-volume node=node-xxxx
    4. Confirm that the cronjob executed fstrim from the job pod logs:

    kubectl -n longhorn-system logs -l job-name=<job_name>

time="2025-07-11T09:26:01Z" level=info msg="Found 1 volumes with recurring job c-mptbs9" func=app.recurringJob file="recurring_job.go:156"
Fri, Jul 11 2025 2:56:01 pm
time="2025-07-11T09:26:01Z" level=info msg="Creating job" func=app.startVolumeJob file="recurring_job.go:195" concurrent=1 executionCount=23 groups=test-group job=c-mptbs9 labels="{\"RecurringJob\":\"c-mptbs9\"}" parameters="map[]" retain=0 task=filesystem-trim volume=pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f
Fri, Jul 11 2025 2:56:01 pm
time="2025-07-11T09:26:01Z" level=info msg="job starts running" func="app.(*Job).run" file="recurring_job.go:294" executionCount=23 labels="map[RecurringJob:c-mptbs9]" namespace=longhorn-system retain=0 snapshotName=c-mptbs9-588e71a7-9e9e-4756-b0ce-34875f8d4575 task=filesystem-trim volumeName=pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f
Fri, Jul 11 2025 2:56:01 pm
time="2025-07-11T09:26:01Z" level=info msg="Running recurring filesystem trim for volume pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f" func="app.(*Job).run" file="recurring_job.go:321" executionCount=23 labels="map[RecurringJob:c-mptbs9]" namespace=longhorn-system retain=0 snapshotName=c-mptbs9-588e71a7-9e9e-4756-b0ce-34875f8d4575 task=filesystem-trim volumeName=pvc-646002c0-9fc7-4940-90b9-fdeffdaba44f

    Additional Details of recurring jobs are explained here. Once the source label on the PVC is enabled, Longhorn starts to sync the recurring-job label from PVC to Longhorn Volume, since Volume is the real resource that Longhorn manages. So, Longhorn needs to sync the label from PVCs to Volumes and further use these labels to select Volumes to do recurring jobs.