Frame Extraction & Selection

Frame Extraction Pipeline Overview

The entire process is a pipeline: raw video → coarse extraction → blur filter → duplicate filter → exposure filter → selected frames → (optional) mask generation.

Typical count reduction:

Step 1: FFmpeg Coarse Extraction

FFmpeg is the Swiss Army knife of frame extraction. Core idea: extract at a fixed frame rate while preserving maximum quality.

Basic Command Template

-vf "fps=2" \
-qscale:v 2 \
-start_number 001 \
output/frame_%04d.jpg

Parameter explanation:

Extraction Rate by Scenario

Rule of thumb: Extraction rate = movement speed (m/s) × 2 ÷ frame width coverage (m). Target: ≥70% overlap between adjacent frames.

Advanced: mpdecimate Auto-Deduplication

FFmpeg's built-in mpdecimate filter can automatically drop near-duplicate frames during extraction (redundant frames from pauses or very slow movement):

# Extract + auto-deduplicate ffmpeg -i input.mp4 \
-vf "fps=2,mpdecimate=hi=64*200:lo=64*50:frac=0.33" \
-vsync vfr \
-qscale:v 2 \
output/frame_%04d.jpg

Preserving EXIF Metadata

FFmpeg strips metadata by default. Add -map_metadata 0 to preserve source file global metadata:

ffmpeg -i input.mp4 \
-vf "fps=2" \
-qscale:v 2 \
-map_metadata 0 \
output/frame_%04d.jpg

Note: Video files have limited EXIF (usually only camera model and date). For precise focal length, record it in meta.yaml and batch-write with ExifTool:

# Batch write focal length (assuming 24mm lens) exiftool -FocalLength=24 -FocalLengthIn35mmFilm=24 ./output/*.jpg

Step 2: Blur Detection & Removal

Motion blur is the #1 enemy of video frame extraction. Even 5–10 severely blurred frames among 300 good ones will cause visible blur artifacts at corresponding angles.

Laplacian Variance Method

Principle: Apply the Laplacian operator (second derivative) to the image and compute the variance of the result. Sharp images have many edges → high variance. Blurry images have few edges → low variance.

import cv2
def laplacian_variance(image_path):
"""Calculate sharpness score for a single image"""
img = cv2.imread(image_path)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
return cv2.Laplacian(gray, cv2.CV_64F).var()

Threshold Selection

Practical tip: Run once without a threshold, output all scores to CSV. Plot a histogram — find the clear "bimodal" distribution. Low peak = blurry frames, high peak = sharp frames. Set threshold at the valley between peaks.

File Size Quick-Screen Method

A faster rough filter: motion-blurred JPEG frames are typically 20–40% smaller than sharp frames (blur reduces detail → higher compression ratio).

# Sort by file size — smallest 20% are likely blurry ls -lS ./raw_frames/ \

Step 3: Duplicate Frame Removal

When you pause during capture, hesitate while turning, or move very slowly, consecutive frames are nearly identical. These redundant frames provide no new information — they only slow COLMAP and waste training time.

SSIM (Structural Similarity) Deduplication

from skimage.metrics import structural_similarity as ssim import cv2
def is_duplicate(frame_a_path, frame_b_path, threshold=0.95):
"""Check if two frames are duplicates via SSIM"""
a = cv2.cvtColor(cv2.imread(frame_a_path), cv2.COLOR_BGR2GRAY)
b = cv2.cvtColor(cv2.imread(frame_b_path), cv2.COLOR_BGR2GRAY)
a = cv2.resize(a, (640, 480))
# Downscale for speed
b = cv2.resize(b, (640, 480))
return ssim(a, b) > threshold

SSIM Threshold Selection

Step 4: Exposure Anomaly Detection

Overexposed (highlights blown white) and underexposed (shadows crushed black) frames must also be removed. They have few feature points and lost color information, causing color discontinuities at corresponding angles.

import cv2 import numpy as np
def check_exposure(image_path, low_thresh=30, high_thresh=225, max_ratio=0.25):
"""Detect over/underexposure. Returns: 'ok', 'overexposed', 'underexposed'"""
img = cv2.imread(image_path)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
total = gray.size
over = np.sum(gray > high_thresh) / total
under = np.sum(gray < low_thresh) / total
if over > max_ratio:
return 'overexposed'
elif under > max_ratio:
return 'underexposed'
return 'ok'

How Many Frames to Keep?

More isn't always better, and fewer isn't always worse. There's a sweet spot:

Recommended Frame Count by Subject Type

Problems with Too Many vs Too Few

COLMAP processing speed reference (RTX 4090):

• 100 frames: ~30 minutes

• 300 frames: ~2 hours

• 500 frames: ~6 hours

• 1000 frames: ~24 hours+

Frame count vs COLMAP time is approximately O(N²) because feature matching is pairwise.

Mask Generation: Tell the Algorithm "Ignore This"

A mask is a black-and-white image the same size as the original: white area = train normally, black area = ignore during training. It solves the problem of scene elements you don't want the model to learn (pedestrians, vehicles, glass reflections, construction barriers).

When Masks Are Needed

Method 1: SAM2 / SAM3 Automatic Segmentation

Meta's SAM2 and SAM3 (released March 2026) can automatically detect and segment any object in images. Combined with Grounding DINO for text-guided detection, you can batch-generate masks for "person," "car," "animal."

from transformers import pipeline
# Initialize SAM3 pipeline generator = pipeline("mask-generation", model="facebook/sam3", device=0) outputs = generator(image_path, points_per_batch=64)

Method 2: Manual Painting (Photoshop / GIMP)

For cases automatic segmentation can't handle (glass reflections, specific light spots):

1. Open frame in Photoshop

2. New layer, paint black over areas to ignore

3. Export as matching _mask.png (pure black and white, no gradients)

4. Ensure mask dimensions exactly match the original frame

Batch tip: If glass/mirror positions are relatively fixed across frames (e.g., display cases), paint one mask template, then use SAM2's video propagation to automatically apply across all frames.

Mask File Naming Convention

01_selected/ ├── 001.jpg ├── 002.jpg └── ...
masks/ ├── 001_mask.png ├── 002_mask.png └── ...

Format requirements:

• PNG format (lossless, no compression artifacts)

• Pure black and white (0 or 255 only, no gray gradients)

• Dimensions exactly match corresponding frame

• Filenames correspond one-to-one with frames

Common Mistakes & Troubleshooting

Next Steps

• Frame selection complete, colors inconsistent → Enter 06-Color Grading & Consistency

• Want to train directly → Enter 08-Training

• Need to review asset organization → Back to 04-Asset Organization & Archival

• ← Previous chapter: 04-Asset Organization & Archival