Quality-Aware Image Conversion: Using Perceptual Metrics to Automate Format Fallbacks

Why a quality-aware image conversion pipeline matters

Simple size-based conversion rules are brittle. A one-size-fits-all "convert everything to AVIF at quality 60" approach will sometimes produce dramatic savings and sometimes produce unacceptable artifacts. A quality-aware image conversion pipeline evaluates converted candidates using perceptual image metrics and only accepts a conversion when it passes visual quality thresholds. When it does not pass, the system automatically falls back to a safer format and/or a higher-quality encoding.

Practical scenarios where this matters:

• For e-commerce product images, subtle texture or fabric detail is critical to buyers. A pipeline must avoid aggressive conversions that introduce banding or texture loss.
• Photographers archiving work need predictable visual fidelity across large batches and formats, and they often require metadata and color-profile preservation.
• Web developers optimizing Core Web Vitals want reliable filesize reductions without risking poor Largest Contentful Paint experiences due to visible degradation.

Perceptual metrics primer: what to measure and why

Objective metrics attempt to predict visual differences a human would notice. Classic metrics like PSNR correlate poorly with perceived quality for modern codecs. Better options include SSIM, MS-SSIM, Butteraugli, and VMAF. Each has tradeoffs in sensitivity, runtime, and interpretability.

Short metric summaries

• SSIM / MS-SSIM — Structural similarity measures that focus on luminance, contrast and structure. Easy to compute and broadly useful, but can miss some color shifts.
• VMAF — A machine-learned metric from Netflix that blends multiple elementary metrics to model human perception. More reliable for modern encoders but heavier to compute.
• Butteraugli — Developed by Google, emphasizes color perception and is sensitive to banding and color shifts. Slower than SSIM but helpful for color-critical images.
• PSNR — Peak signal-to-noise ratio. Fast but poorly correlated with perception for many use cases.

When discussing decision logic I will use the phrase SSIM VMAF image decision to describe logic that combines SSIM and VMAF scores into one accept/reject rule. Combining metrics helps cover different failure modes.

How to compute metrics in practice

The most pragmatic approach is to use FFmpeg builds with libvmaf and the ssim filter. Below is a short command-line example for comparing an original image against a converted candidate. Note that working with still images requires using image2 or treating files as single-frame videos.

# Create temporary single-frame IVF/PNG sequences or use ffmpeg -loop 1 -t 1 to generate single-frame videos
ffmpeg -loop 1 -i original.png -loop 1 -i candidate.avif   -filter_complex "[0:v]scale=iw:ih:flags=lanczos[ref];[1:v]scale=iw:ih:flags=lanczos[dist];[ref][dist]libvmaf=model_path=/usr/share/model/vmaf_v0.6.1.pkl:log_fmt=json:psnr=1:ssim=1"   -f null -

The command above writes a JSON VMAF report to stdout and includes PSNR and SSIM. In production, you typically write that JSON to a file and parse it. For batch jobs you can spawn FFmpeg processes from Node.js, Python, or a compiled service.

For extremely high throughput, compute lighter-weight descriptors first (e.g., SSIM or a fast feature hash) and only compute VMAF for candidates that are close to thresholds.

Designing a quality-aware pipeline: architecture and stages

A robust pipeline has stages from input validation to final artifact selection. Below is a recommended high-level flow that balances accuracy and performance.

1. Ingest and normalize: validate inputs, extract metadata and color profiles, and create a working master image (e.g., 16-bit linear if you need HDR).
2. Create conversion candidates: encode to target formats and quality presets (AVIF/WebP/JPEG/PNG) using multiple quality settings for each format.
3. Compute perceptual metrics: run SSIM and VMAF (and optionally Butteraugli) comparing each candidate to the master image.
4. Decision logic: apply your SSIM VMAF image decision rules to accept or reject each candidate. Include fallback rules based on format capabilities (alpha, color depth).
5. Persist accepted artifacts and metadata: store both chosen artifacts and the metric results for auditing and future tuning.
6. Cache and CDN policies: set cache-control, content negotiation headers, and origin hints if using client hints or server-side device detection.

In the next section we will look at concrete code for the candidate creation and decision step.

Candidate generation with Sharp (Node.js example)

Sharp is an excellent starting point for conversion tasks. It supports WebP, AVIF, JPEG and PNG and preserves ICC profiles by default when asked. The sample below demonstrates how to create multiple candidates at different quality levels.

const sharp = require("sharp");
const fs = require("fs").promises;

async function generateCandidates(inputPath, outDir) {
  const qualities = { jpeg: [75, 85, 95], webp: [60, 75, 90], avif: [40, 60, 80] };

  await fs.mkdir(outDir, { recursive: true });

  await Promise.all([
    ...qualities.jpeg.map(q => sharp(inputPath).jpeg({ quality: q }).toFile(`${outDir}/image-q${q}.jpg`)),
    ...qualities.webp.map(q => sharp(inputPath).webp({ quality: q }).toFile(`${outDir}/image-q${q}.webp`)),
    ...qualities.avif.map(q => sharp(inputPath).avif({ quality: q }).toFile(`${outDir}/image-q${q}.avif`)),
  ]);
}

module.exports = { generateCandidates };

The next step is to compare each candidate with the original master using FFmpeg VMAF and SSIM to make an SSIM VMAF image decision.

Automating format fallback: decision logic examples

A fallback strategy uses perceptual thresholds and format capability constraints. I recommend a two-stage rule:

1. Perceptual acceptance: candidate must meet SSIM >= 0.985 and VMAF >= 95. These values are conservative and suitable for consumer-facing visuals.
2. Feature checks: if the original has alpha or requires 10-bit color, disallow formats that do not support them or adjust acceptance thresholds accordingly.

When a candidate fails, fall back to the next safest option in your prioritized list. Example priority for many sites: AVIF → WebP → JPEG. For images with alpha the chain becomes AVIF → WebP (lossless/alpha) → PNG.

Node.js decision function (simple)

Below is a lightweight decision function that reads metric JSON produced by FFmpeg and returns the best accepted format or a fallback.

const fs = require("fs").promises;

// Example metric file format:
// { format: "avif", ssim: 0.988, vmaf: 96.2, size: 12345, hasAlpha: false }

async function chooseBestCandidate(metricsList, originalFeatures) {
  const priorities = [
    { fmt: "avif", supportsAlpha: true, supports10bit: true },
    { fmt: "webp", supportsAlpha: true, supports10bit: false },
    { fmt: "jpeg", supportsAlpha: false, supports10bit: false },
    { fmt: "png", supportsAlpha: true, supports10bit: false }
  ];

  // Filter out formats that cannot represent the original features
  const compatiblePriorities = priorities.filter(p => {
    if (originalFeatures.hasAlpha && !p.supportsAlpha) return false;
    if (originalFeatures.needs10bit && !p.supports10bit) return false;
    return true;
  });

  // Combine SSIM and VMAF into a simple score
  function passesThresholds(m) {
    return m.ssim >= 0.985 && m.vmaf >= 95;
  }

  // Try to find the highest-priority candidate that passes
  for (const p of compatiblePriorities) {
    const candidate = metricsList.find(m => m.format === p.fmt && passesThresholds(m));
    if (candidate) return candidate;
  }

  // If none pass, pick the highest-quality candidate available (fallback)
  metricsList.sort((a, b) => {
    if (a.vmaf !== b.vmaf) return b.vmaf - a.vmaf;
    return a.size - b.size;
  });

  return metricsList[0];
}

module.exports = { chooseBestCandidate };

The function above is intentionally simple. In production you will want to include confidence bands, metrics for thumbnails vs full images, and logging for auditability.

Format capabilities and comparison

Use format capability knowledge to avoid accepting a technically smaller file that is functionally incompatible with your requirements (for example, losing alpha or reducing bit depth). The table below summarizes key properties from format specifications and common implementations.

The table above is based on format specifications and broad usage patterns rather than benchmarks. For browser adoption and feature detection consult resources like Can I Use and MDN Web Docs.

Helpful external references:

• MDN Web Docs: Image formats overview
• Can I Use
• W3C / WHATWG specifications
• Cloudflare Learning Center
• web.dev

Tuning visual quality thresholds

Thresholds are context-dependent. Below are practical starting points and how to adapt them by use case.

Suggested starting thresholds

• SSIM >= 0.985 — conservative for product photos and sites where detail matters
• VMAF >= 95 — indicates near-indistinguishable quality in most cases
• For thumbnails: SSIM >= 0.96 and VMAF >= 90 are often acceptable
• For archival masters: prefer lossless or VMAF >= 99 with preserved metadata

Use A/B testing for user-facing UX tradeoffs. For e-commerce, small losses in image fidelity can negatively affect conversions; test a control group carefully before relaxing thresholds.

Adaptive threshold strategies

Rather than global fixed thresholds you can adapt thresholds based on:

• Image content type detection (photograph vs screenshot vs logo)
• Device class and connection speed (using client hints to prefer smaller files on slow connections)
• Importance score (product hero image vs thumbnail)

Integration and deployment patterns

You can integrate a quality-aware pipeline at several points in your stack: build-time, on-upload, or at request-time. Each has tradeoffs.

Build-time

Convert and evaluate assets during your static build or CI. This is efficient for sites with a finite set of images and yields the lowest runtime cost. Use GitHub Actions or CI runners to run conversion and metric checks. Below is a minimal GitHub Actions job sketch for running conversions and VMAF checks.

# .github/workflows/convert.yml (sketch)
name: convert-images
on:
  push:
    paths:
      - "assets/images/**"

jobs:
  convert:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies
        run: |
          sudo add-apt-repository ppa:savoury1/ffmpeg4 -y
          sudo apt-get update -y
          sudo apt-get install -y ffmpeg
          npm ci
      - name: Run conversion and metrics
        run: node scripts/convert_and_evaluate.js
      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: converted-images
          path: converted/

Build-time pipelines are a good fit for WebP2JPG.com style tools where you want consistent outputs per source image.

On-upload

Running conversions when users upload images balances compute and user expectations. You can queue a background job that generates candidates and computes metrics, returning a safe default to the uploader immediately while generating optimized assets asynchronously.

Request-time

Real-time conversion is possible but expensive. Use it when you must serve a format that depends on dynamic client signals. To keep latency low, perform a fast candidate creation step and cache evaluated artifacts aggressively at the CDN edge.

Performance, scaling, and cost considerations

VMAF and Butteraugli are computationally expensive. For thousands of images per hour you need orchestration and caching strategies.

• Prioritize cheap filters first: run SSIM and size checks before VMAF. If SSIM is high and size reduction is small, you may skip VMAF.
• Sample pixels for very large images, compute metrics on downscaled versions to reduce CPU, and adjust thresholds accordingly.
• Use GPU-accelerated encoders where available for AVIF/AV1 to speed up candidate generation.
• Cache metric results keyed by source image hash and conversion parameters so repeated uploads or rebuilds reuse prior decisions.

For heavy workloads, consider running metric computation on dedicated workers and only storing final artifacts in S3 or your object store. Logging metric results helps refine thresholds later without rerunning expensive jobs.

Troubleshooting common issues

Here are real problems I've seen in production and how to resolve them.

Color shifts after conversion

Cause: color profile (ICC) was stripped or a conversion used a different color space. Solution: preserve or embed ICC profiles during conversion and ensure conversions operate in the correct color space (sRGB vs display-p3). With Sharp, include the option to preserve metadata or explicitly pass the profile.

Banding and posterization

Cause: aggressive quantization or reduced bit depth. Solution: increase quality settings, enable dither if available, or use a format with higher bit depth like AVIF for gradient-heavy images.

Metrics disagree with visual perception

Cause: a single metric misses a failure mode. Solution: combine metrics such as SSIM and VMAF. For color-focused differences use Butteraugli. Also visually audit a sample set regularly.

Small images and thumbnails produce noisy metric values

Cause: metrics are less stable on tiny images. Solution: compute metrics on upscaled versions of the thumbnail or use simpler heuristics like file size ratio and SSIM with relaxed thresholds.

Technical deep-dive: how compression and color handling affect decisions

Understanding the internals of image formats helps explain why some conversions fail perceptual checks. Here are the key points to watch.

JPEG: DCT, quantization and chroma subsampling

JPEG uses block-based DCT transforms and quantization tables. Quantization is where most loss comes from and is frequency-dependent. Chroma subsampling (e.g., 4:2:0) reduces color resolution which is usually acceptable for photographs but can ruin sharp edges and text. When a conversion results in color bleeding or fuzzy text, it is often due to chroma subsampling combined with aggressive quantization.

AVIF/AV1: transform coding, tools, and high bit depth

AVIF is based on AV1 intra-frame compression which uses more sophisticated transforms and tools than JPEG. It supports higher bit depths and better handling of gradients, which is why it can often achieve smaller sizes for the same perceived quality. However AV1 encoders are more sensitive to encoder settings and slower by default.

Color profiles and metadata

Always decide whether to preserve ICC profiles and EXIF metadata. Consumer web delivery often strips metadata for size, but professional photography workflows require preservation. Use tools like exiftool to extract, reapply, or inspect metadata when automated conversions appear to change color.

# Preserve ICC with Sharp when converting
const sharp = require("sharp");

await sharp("input.tiff")
  .withMetadata() // preserves ICC and orientation
  .avif({ quality: 60 })
  .toFile("out.avif");

When you see color drift after conversion, the first step is to verify the ICC profile is present and interpreted by the tools you use for measurement.

Example workflow: e-commerce product images

For product images the priority is fidelity. Here is a practical workflow tailored to e-commerce:

1. Upload time: create a lossless master (preserve ICC and EXIF).
2. Background worker: generate candidates at multiple quality points for AVIF, WebP and JPEG.
3. Compute SSIM and VMAF for full-size hero images. For thumbnails compute SSIM on 2x upscaled thumbnails.
4. Accept candidate if it passes both SSIM and VMAF thresholds. Otherwise fallback to next format or quality tier.
5. Store accepted artifacts and expose them via CDN. Use Content Negotiation or client hints for delivery.

This workflow minimizes the chance that a product hero loses detail. It also captures metric history useful for auditing and retraining thresholds later.

Choosing tools and services

Several open-source libraries and tools support the building blocks you need. A few recommendations based on real-world use:

• Sharp (Node.js) — fast image operations and multi-format support.
• FFmpeg with libvmaf — compute VMAF and SSIM from the CLI.
• exiftool — inspect and preserve metadata reliably.
• For a web UI or batch tool, consider WebP2JPG.com as an example of browser-based conversion tools with strong usability.

For many teams, combining these into a small microservice is a practical way to run quality-aware conversions. WebP2JPG.com demonstrates how a browser-based workflow can simplify user interaction while delegating quality checks to a backend worker.

Decision auditability and observability

Track every decision. Store metric outputs, chosen format, original features and a diff image or zoomable visual diff for manual inspection. These logs are gold when tuning thresholds or diagnosing a user complaint.

Build dashboards that surface:

• Distribution of SSIM and VMAF scores over time
• Failure rates by image type and by encoder quality setting
• Average filesize reductions for accepted artifacts

These observability signals let you confidently relax thresholds for low-risk content and tighten them for critical assets.

FAQ

Q: How expensive is VMAF to compute at scale?

A: VMAF is heavier than SSIM. On a modern CPU a single-frame VMAF calculation is measurable in hundreds of milliseconds to a few seconds depending on image resolution and FFmpeg build. Use SSIM, size checks and sampling to reduce the number of VMAF computations, and cache results indexed by image hash and conversion parameters.

Q: Can perceptual metrics fully replace human QA?

A: Not entirely. Metrics are strong filters but periodic human audits are critical, especially after encoder upgrades or format changes. Treat metrics as a squad of fast, consistent reviewers and human QA as the final judge for edge cases.

Q: What if a conversion passes metrics but users still complain about quality?

A: Start capturing the exact image hash, metric results, and contextual signals like device and zoom level. Oftentimes complaints arise from unusual device color management or a missing ICC profile on certain browsers, which metrics computed in a linear sRGB pipeline may not capture.

Q: Should I use different thresholds per format?

A: Yes. Some formats have artifacts that particular metrics detect differently. For example, Butteraugli is more sensitive to color shifts often introduced by WebP, so you might require a higher Butteraugli score for WebP/AVIF acceptance.

Final checklist and next steps

Before you ship a quality-aware image conversion pipeline, run through this checklist:

• Preserve color profiles and required metadata for your use case.
• Implement fast prefilters (size reduction checks and SSIM) to limit expensive VMAF runs.
• Define clear fallback chains based on format capabilities.
• Log metric results and decisions for auditing and tuning.
• Run human audits and A/B tests for critical images such as product heroes.

If you want a practical starting point, try building a small proof-of-concept: take ten representative images from your catalog, generate candidates (AVIF/WebP/JPEG), compute SSIM and VMAF via FFmpeg, and iterate your thresholds. If you prefer a browser-first workflow for smaller teams, WebP2JPG.com is a helpful reference for UI and conversion shortcuts and can be part of your toolkit.

Building a quality-aware image conversion pipeline is an investment that pays back in user experience, storage savings and predictable image quality. Start conservatively, instrument heavily, and iterate based on data.

— Alexander Georges, founder of WebP2JPG.com, Techstars '23 alum and former full-stack/UX lead