Automating Artifactory Cleanup Policies with AQL to Reduce Storage Costs

If you've been running JFrog Artifactory for more than a few months, you've almost certainly had the conversation: "Why is our storage bill so high?" Artifacts accumulate fast. CI pipelines push builds every few minutes, Docker layers pile up, and before you know it you're sitting on terabytes of data that nobody has touched in six months.

The good news is that Artifactory gives you a genuinely powerful tool to fix this: Artifactory Query Language (AQL). Combined with the REST API and a bit of scripting automation, you can build cleanup policies that are precise, auditable, and safe — none of which you get from just clicking "delete old artifacts" in the UI.

Let me walk you through how I approach this in production environments.

---

Why AQL Over Built-in Cleanup Policies?

Artifactory Pro and above include built-in cleanup policies, and they're fine for simple cases. But they hit a wall quickly when you need logic like:

• Delete build artifacts older than 30 days unless they're tagged as release or production
• Clean up Docker images that haven't been pulled in 90 days except the last 3 tags per repository
• Remove snapshot artifacts but preserve anything referenced by a currently deployed version

AQL gives you SQL-like expressiveness against Artifactory's artifact database. You query first, review what you'd delete, then execute — which is exactly the kind of workflow you want before running mass deletions in production.

---

Understanding AQL Basics

AQL queries run against the /api/search/aql endpoint and look like this:

items.find({
    "repo": "my-repo",
    "type": "file",
    "created": { "$lt": "2024-01-01" }
})
.include("name", "repo", "path", "created", "size")
.sort({"$desc": ["created"]})
.limit(100)

The main item types you'll work with are file, folder, and any. You can filter on properties, statistics (download count, last downloaded date), and build information.

The statistics domain is the secret weapon for cleanup. It tells you when an artifact was last downloaded — which is a much better signal for "is this still needed?" than creation date alone.

---

Setting Up Your Environment

You'll need:

• Artifactory URL and an API key or access token with delete permissions
• curl or a scripting language (I'll use Python here)
• A dry-run mindset before you touch anything in production

export ARTIFACTORY_URL="https://your-instance.jfrog.io/artifactory"
export ARTIFACTORY_TOKEN="your-access-token"

---

Recipe 1: Clean Up Old Snapshot Artifacts

This is the most common cleanup scenario. Maven and Gradle snapshot repositories grow without bound if left unchecked.

import requests
import json
import os
from datetime import datetime, timedelta

ARTIFACTORY_URL = os.environ["ARTIFACTORY_URL"]
TOKEN = os.environ["ARTIFACTORY_TOKEN"]

headers = {
    "Authorization": f"Bearer {TOKEN}",
    "Content-Type": "text/plain"
}

# Find snapshot artifacts older than 30 days with zero downloads
aql_query = """
items.find({
    "repo": {"$match": "*-snapshots"},
    "type": "file",
    "created": {"$lt": "2024-06-01T00:00:00.000Z"},
    "stat.downloads": {"$eq": 0}
})
.include("name", "repo", "path", "created", "size", "stat.downloads")
.sort({"$desc": ["size"]})
"""

response = requests.post(
    f"{ARTIFACTORY_URL}/api/search/aql",
    headers=headers,
    data=aql_query
)

results = response.json()
artifacts = results.get("results", [])

print(f"Found {len(artifacts)} artifacts to review")

total_size = sum(a.get("size", 0) for a in artifacts)
print(f"Total reclaimable space: {total_size / (1024**3):.2f} GB")

# Dry run — print what we'd delete
for artifact in artifacts[:10]:  # Preview first 10
    path = f"{artifact['repo']}/{artifact['path']}/{artifact['name']}"
    size_mb = artifact.get('size', 0) / (1024**2)
    print(f"  Would delete: {path} ({size_mb:.1f} MB)")

Run this first, review the output, then flip to actual deletion once you're satisfied.

---

Recipe 2: Docker Image Cleanup by Last Download Date

Docker repositories are the biggest storage hogs in most environments. This query targets images that haven't been pulled in 60 days while preserving the most recent tags.

# Find Docker manifests not downloaded in 60+ days
aql_query = """
items.find({
    "repo": "docker-local",
    "name": "manifest.json",
    "type": "file",
    "stat.downloaded": {"$lt": "2024-04-01T00:00:00.000Z"}
})
.include("name", "repo", "path", "created", "stat.downloaded", "stat.downloads")
.sort({"$asc": ["stat.downloaded"]})
"""

response = requests.post(
    f"{ARTIFACTORY_URL}/api/search/aql",
    headers=headers,
    data=aql_query
)

results = response.json()
manifests = results.get("results", [])

def delete_docker_image(repo, path):
    """Delete a Docker image by deleting its folder path."""
    # Docker images are stored as folders — delete the image tag folder
    image_folder = path.rsplit("/", 1)[0]  # strip manifest.json
    delete_url = f"{ARTIFACTORY_URL}/{repo}/{image_folder}"
    
    resp = requests.delete(delete_url, headers={"Authorization": f"Bearer {TOKEN}"})
    return resp.status_code

# Execute deletions — remove the dry_run check when ready
dry_run = True

for manifest in manifests:
    image_path = f"{manifest['repo']}/{manifest['path']}"
    last_downloaded = manifest.get("stat.downloaded", "never")
    
    print(f"{'[DRY RUN] ' if dry_run else ''}Deleting: {image_path} (last pulled: {last_downloaded})")
    
    if not dry_run:
        status = delete_docker_image(manifest["repo"], manifest["path"])
        print(f"  Status: {status}")

Important: After bulk-deleting Docker layers, run Artifactory's garbage collection to actually reclaim the storage:

curl -X POST \
  -H "Authorization: Bearer $ARTIFACTORY_TOKEN" \
  "$ARTIFACTORY_URL/api/system/storage/gc"

---

Recipe 3: Keep the Last N Builds Per Artifact

This is where AQL really earns its keep. Instead of time-based cleanup, you keep the last 5 builds of each artifact and delete everything older.

from collections import defaultdict

# Get all artifacts in a release repo, grouped by module
aql_query = """
items.find({
    "repo": "libs-release-local",
    "type": "file",
    "name": {"$match": "*.jar"},
    "path": {"$nmatch": "*sources*"}
})
.include("name", "repo", "path", "created", "size")
.sort({"$desc": ["created"]})
"""

response = requests.post(
    f"{ARTIFACTORY_URL}/api/search/aql",
    headers=headers,
    data=aql_query
)

artifacts = response.json().get("results", [])

# Group by module path (everything except the version folder)
modules = defaultdict(list)
for artifact in artifacts:
    # Path looks like: com/example/my-service/1.2.3/my-service-1.2.3.jar
    parts = artifact["path"].split("/")
    if len(parts) >= 2:
        module_key = f"{artifact['repo']}/{'/'.join(parts[:-1])}"  # exclude version
        modules[module_key].append(artifact)

KEEP_LAST_N = 5
to_delete = []

for module, versions in modules.items():
    # Already sorted by created desc, so keep first N
    candidates_for_deletion = versions[KEEP_LAST_N:]
    to_delete.extend(candidates_for_deletion)
    
    if candidates_for_deletion:
        print(f"Module: {module}")
        print(f"  Keeping: {len(versions[:KEEP_LAST_N])} versions")
        print(f"  Deleting: {len(candidates_for_deletion)} versions")

total_reclaimable = sum(a.get("size", 0) for a in to_delete)
print(f"\nTotal reclaimable: {total_reclaimable / (1024**3):.2f} GB across {len(to_delete)} artifacts")

---

Operationalizing This: Run It on a Schedule

A one-time script isn't a cleanup policy. You need this running automatically. Here's how I typically wire this up:

Option 1: Jenkins/CI Pipeline

pipeline {
    agent any
    triggers {
        cron('0 2 * * 0')  // Every Sunday at 2 AM
    }
    stages {
        stage('Artifactory Cleanup') {
            steps {
                withCredentials([string(credentialsId: 'artifactory-token', variable: 'ARTIFACTORY_TOKEN')]) {
                    sh '''
                        python3 cleanup_snapshots.py \
                            --dry-run=false \
                            --older-than-days=30 \
                            --repo="*-snapshots"
                    '''
                }
            }
        }
    }
    post {
        always {
            archiveArtifacts artifacts: 'cleanup-report.json'
        }
    }
}

Option 2: Kubernetes CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: artifactory-cleanup
  namespace: devops-tools
spec:
  schedule: "0 3 * * 0"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cleanup
            image: python:3.11-slim
            command: ["python3", "/scripts/cleanup.py"]
            env:
            - name: ARTIFACTORY_TOKEN
              valueFrom:
                secretKeyRef:
                  name: artifactory-credentials
                  key: token
            - name: ARTIFACTORY_URL
              value: "https://your-instance.jfrog.io/artifactory"
            volumeMounts:
            - name: scripts
              mountPath: /scripts
          volumes:
          - name: scripts
            configMap:
              name: cleanup-scripts
          restartPolicy: OnFailure

---

Safety Rails You Should Always Have

Before you run any of this in production, build in these safeguards:

1. Always log what you're deleting. Write a JSON report before executing deletions. If something goes wrong, you have a record.

2. Set deletion rate limits. Don't fire off 10,000 delete requests at once. Add a small sleep between deletions and cap the number processed per run.

import time

MAX_DELETIONS_PER_RUN = 500
SLEEP_BETWEEN_DELETIONS = 0.1  # seconds

for i, artifact in enumerate(to_delete[:MAX_DELETIONS_PER_RUN]):
    # delete logic here
    time.sleep(SLEEP_BETWEEN_DELETIONS)

3. Exclude artifacts with specific properties. Tag important artifacts with cleanup.exempt=true and filter them out in your AQL:

items.find({
    "repo": "libs-release-local",
    "type": "file",
    "@cleanup.exempt": {"$ne": "true"}
})

4. Test on a non-production instance first. Obvious, but worth saying.

---

What Kind of Savings Should You Expect?

In environments I've worked with, a well-tuned cleanup policy typically recovers:

• 40-60% of snapshot repository storage within the first run
• 20-40% of Docker registry storage when targeting images older than 90 days
• Ongoing steady-state reduction of 15-25% monthly storage growth

The exact numbers depend heavily on your release cadence and how long the repository has been running without cleanup. One client went from 8TB to 3.2TB after a single cleanup pass targeting two-year-old snapshots. Their Artifactory Cloud bill dropped proportionally.

---

Final Thoughts

AQL-based cleanup isn't glamorous work, but it's the kind of infrastructure hygiene that quietly saves tens of thousands of dollars a year in storage costs while keeping your Artifactory instance performant. The key principles are simple: query before you delete, log everything, and automate it so it runs without anyone thinking about it.

Start with a dry run, validate the output looks sane, then turn it loose. Your finance team will notice.