Codec private options: squeezing every bit of quality

The Complete Codecs Guide gave you the foundation. Now we go deeper. Every serious encoder offers private options—tuning knobs that control motion estimation, rate control, psychovisual optimizations, and stream structure. These are the settings that differentiate a "good" encode from a "masterpiece" encode.

libx264 – the deep tuning guide

To pass private options to libx264, use -x264-params with a colon‑separated list of key=value pairs.

ffmpeg -i input.mov -c:v libx264 -preset medium -crf 23 \
  -x264-params "refs=5:bframes=3:b-adapt=2:rc-lookahead=60:deblock=-1,-1:psy-rd=1.0,0.15" \
  output.mp4

Breaking down the critical x264 options

• `refs` (default varies by preset, max 16) – Number of reference frames. Higher values improve compression for static scenes but increase decoding complexity. Use 5–8 for high‑quality encodes.
• `b-adapt` (0, 1, or 2) – B‑frame decision algorithm. 2 (optimal) gives the best quality but is slow. Use 1 (fast) for faster encodes.
• `bframes` (default 3) – Max number of consecutive B‑frames. Values like 5–8 can improve compression in slow scenes.
• `rc-lookahead` (default 40, max 250) – Number of frames for the rate‑controller to look ahead. Higher = better bitrate distribution, especially for scene changes. Use 60–120 for high‑quality VBR.
• `deblock` (format: `strength,threshold`, default `0,0`) – Controls the in‑loop deblocking filter. Negative values (e.g., `-1,-1`) preserve fine detail but may show more compression artifacts. Positive values smooth more.
• `psy-rd` (format: `rd,trellis`, default `1.0,0.0`) – Psychovisual rate‑distortion optimization. RD (0.5–1.5) preserves textures; higher values sharpen but can introduce noise. Trellis (0.0–0.25) enhances detail in flat areas.
• `no-mbtree` – Disables macroblock tree rate control (enabled by default). Disabling can sometimes improve quality in extremely complex scenes but increases file size.

Pro tip for grain preservation (film, noisy content): -x264-params "deblock=-2,-2:psy-rd=1.2,0.2:no-mbtree=1"

libx265 – HEVC advanced tuning

With H.265, use -x265-params. The options are similar but often more sensitive.

ffmpeg -i input.mov -c:v libx265 -crf 24 \
  -x265-params "ref=5:bframes=5:b-adapt=2:rc-lookahead=60:rd-refine=1:limit-refs=3" \
  output.mp4

• `ref` – Similar to x264, but 4–6 is a good range for HEVC.
• `rd-refine` (0 or 1) – Enables R‑D refinement at the cost of speed. Set to 1 for the best quality.
• `limit-refs` (0–3) – Limits the number of reference frames used by motion estimation. 3 offers a good speed/quality trade‑off.
• `rect` and `amp` – Enable rectangular motion partitions (rect) and asymmetric motion partitions (amp). Both improve compression for edges but slow encoding significantly.
• `rd-level` (0–6) – Rate‑distortion optimization level. Use 5 or 6 for high quality, but expect it to be very slow.
• `no-strong-intra-smoothing` – Disables strong intra smoothing. Enable this to preserve fine edges in sharp content.

VP9 – libvpx-vp9 tuning

VP9 options are passed directly as flags, not via a params string.

ffmpeg -i input.mov -c:v libvpx-vp9 -crf 30 -b:v 0 \
  -row-mt 1 -tile-columns 2 -tile-rows 1 -frame-parallel 1 \
  -speed 2 -c:a libopus output.webm

• `row-mt` (0 or 1) – Enables row‑based multi‑threading. Always set to 1 for massive speedups on multi‑core CPUs.
• `tile-columns` – Sets the number of tile columns (log2). 2 = 4 tiles, 3 = 8 tiles. Improves multi‑threading and allows for frame‑parallel decoding.
• `tile-rows` – Same for rows. Use 0 or 1.
• `frame-parallel` – Enables frame‑parallel encoding/decoding. Set to 1 for faster encoding.
• `cpu-used` – Deprecated; use -speed instead.
• `auto-alt-ref` (default 1) – Enables alternate reference frames. Keep enabled for better compression.

AV1 – libsvtav1 tuning

The SVT‑AV1 encoder is gaining popularity for its speed/quality balance.

ffmpeg -i input.mov -c:v libsvtav1 -crf 30 -preset 8 \
  -svtav1-params "enable-tf=1:tile-columns=2:tile-rows=1:enable-overlays=1" \
  -c:a libopus output.mkv

• `enable-tf` (0 or 1) – Enables temporal filtering. Reduces noise and improves compression. Set to 1 for grainy/noisy sources.
• `enable-overlays` (0 or 1) – Enables overlapped block motion compensation. Improves quality for complex motion. Set to 1.
• `enable-qm` – Enables quantization matrices. Improves compression for high resolutions.
• `enable-altref` – Enables alternate reference frames. Always keep enabled.
• `tile-columns` / `tile-rows` – Similar to VP9; set based on your core count (e.g., 2/1 for 4 cores).

Niche & legacy codecs

Not every project needs H.264 or HEVC. Here are the codecs you'll encounter in legacy broadcast, archival, or specialized pipelines.

MPEG‑2 Video (`mpeg2video`)

Still used in broadcast, DVD, and MPEG‑TS streams. It lacks the advanced tools of modern codecs.

# MPEG‑2 with high bitrate (DVD quality)
ffmpeg -i input.mov -c:v mpeg2video -b:v 8M -maxrate 9M -bufsize 3.5M \
  -mpeg2video_psi 1 -g 15 -bf 2 output.m2v

• `-g` – GOP size (keyframe interval). 12–15 for standard definition.
• `-bf` – Number of B‑frames (0–3).
• `-mpeg2video_psi` – Inserts Program Specific Information (for broadcast).

MJPEG (`mjpeg`)

Motion JPEG – each frame is a separate JPEG image. Used in IP cameras and legacy video mixers. High bitrate required.

# MJPEG with custom quantization (quality 2–31, lower = better)
ffmpeg -i input.mov -c:v mjpeg -q:v 5 -huffman 1 output.avi

• `-q:v` – Quality (2 = best, 31 = worst).
• `-huffman` – Use optimized Huffman tables (better compression).

PNG (`png`)

Lossless, but huge file sizes. Used for screenshot sequences or lossless intermediates.

# PNG with maximum compression (slower)
ffmpeg -i input.mov -c:v png -compression_level 9 -pred mixed output/%05d.png

WMV (Windows Media Video) – `wmv2`

Legacy Microsoft codec. Rarely used today, but still encountered in corporate archives.

ffmpeg -i input.mov -c:v wmv2 -b:v 2M -c:a wmav2 output.wmv

Codec tuning cheat sheet

Pair these advanced settings with the right container. Read the Advanced Muxer Flags guide to ensure your container isn't the bottleneck.