Video Labeling with DNN Filters – Object Detection & Classification Pipeline

Q: What models are used?

The pipeline supports multiple models including YOLO for object detection, ResNet for classification, and face detection models. The default configuration uses OpenVINO models for face detection (face-detection-adas-0001), classification (emotions-recognition-retail-0003), and can be extended for general object detection.

Q: What is the output format?

The pipeline outputs a JSON file containing all detection results: frame number, bounding box coordinates, label/class name, confidence score, and timestamp. This can be used for search, analytics, or further processing.

Video understanding is one of the most powerful applications of AI. Being able to automatically detect objects, faces, and scenes in videos opens up a world of possibilities — from content moderation and search to analytics and accessibility.

FFmpeg's DNN filters — dnn_detect and dnn_classify — make this possible. With pre‑trained models like YOLO, ResNet, and face detection models, you can label every frame of your video with rich metadata.

This guide shows you how to build a fully automated video labeling pipeline that:

Detects objects and faces in uploaded videos
Classifies scenes and emotions
Generates JSON metadata with bounding boxes and labels
Stores results alongside the original video

Key takeaways

dnn_detect — object and face detection with bounding boxes
dnn_classify — classification of scenes, emotions, and more
YOLO, ResNet, and face detection — proven models for video understanding
JSON metadata — results stored for search, analytics, and further processing
One SQL script — everything is set up with a single execution

The Gap: Video Understanding at Scale

Video is the most data‑rich medium we have. But without understanding what's in the video, it's just pixels. Manually labeling videos is impossible at scale. Automated video labeling using DNN filters makes it practical:

Content moderation — detect inappropriate content automatically
Search and discovery — find videos by content (e.g., "videos with dogs")
Analytics — understand what's in your video library
Accessibility — auto‑generate alt text and descriptions

This guide shows you how to build that pipeline.

Architecture Overview

User Uploads Video → Supabase Storage → PostgreSQL Trigger → pgmq Queue → ffmpeglab-runner
dnn_detect → Bounding Boxes → dnn_classify → Labels
JSON Metadata → Public Folder → pg_notify → User Notified

The pipeline consists of:

Supabase Storage — two buckets: private-uploads (per‑user) and public-processed (per‑user).
RLS policies — restrict access to each user's own folders.
PostgreSQL trigger — fires on INSERT into storage.objects.
pgmq — message queue for job processing.
ffmpeglab-runner — runs DNN detection and classification.
pg_notify — real‑time status updates.

What the Pipeline Delivers

Output	Format	Location
Detection Metadata	JSON (bounding boxes, labels, confidence)	`public-processed/{userId}/labels/`
Labeled Video	MP4 (with bounding boxes overlaid)	`public-processed/{userId}/labeled/`
Real‑time notifications	`pg_notify` channels	N/A
Job tracking	`render` table	Existing FFmpegLab table
Logs	`logpiece` table	Existing FFmpegLab table

Prerequisites

A Supabase project (cloud or self‑hosted).
ffmpeglab-server and ffmpeglab-runner deployed.
FFmpeg compiled with DNN support (OpenVINO or TensorFlow).
DNN model files downloaded (see Model Management).
render and logpiece tables must exist.
Access to your Supabase database.

Quick Start – The Setup Script

Step 1

Save the SQL script

Copy the complete SQL script from the section below and save it as setup_labeling_pipeline.sql.

Step 2

Run the script

Execute the script against your Supabase database.

        # Via psql
        psql -U postgres -d your_database -f setup_labeling_pipeline.sql
        
        # Or via the Supabase SQL Editor
        # Copy and paste the entire script into the SQL Editor and run it.

Step 3

Download models

Download the required DNN models (see Model Management).

Step 4

Configure the runner

Add the labeling queue to your runner's environment.

        # Add to your .env or docker-compose.yml
        LABELING_QUEUE_NAME=labeling_queue
        
        # Model paths
        DNN_DETECT_MODEL=/app/models/detection/face-detection-adas-0001.xml
        DNN_CLASSIFY_MODEL=/app/models/classification/emotions-recognition-retail-0003.xml
        DNN_LABELS=/app/models/detection/face-detection-adas-0001.label
        
        # DNN backend
        DNN_BACKEND=openvino

Step 5

Restart the runner

Restart the runner to pick up the new queue.

docker compose restart ffmpeglab-runner

The Complete SQL Setup Script

      -- ============================================================
      -- VIDEO LABELING WITH DNN FILTERS
      -- Complete SQL Setup Script
      -- ============================================================
      -- This script adds the video labeling pipeline components.
      -- It assumes the render and logpiece tables already exist.
      -- ============================================================

      -- ============================================================
      -- 1. Create storage buckets
      -- ============================================================
      INSERT INTO storage.buckets (id, name, public, avif_autodetection, file_size_limit, allowed_mime_types)
      VALUES
          ('private-uploads', 'private-uploads', false, false, 5368709120, ARRAY['video/mp4', 'video/quicktime', 'video/x-msvideo', 'video/webm', 'video/mpeg']),
          ('public-processed', 'public-processed', true, false, 5368709120, ARRAY['video/mp4', 'application/json'])
      ON CONFLICT (id) DO NOTHING;

      -- ============================================================
      -- 2. RLS policies for private-uploads bucket
      -- ============================================================

      DROP POLICY IF EXISTS "Users can upload to their own folder" ON storage.objects;
      CREATE POLICY "Users can upload to their own folder"
      ON storage.objects
      FOR INSERT
      TO authenticated
      WITH CHECK (
          bucket_id = 'private-uploads' AND
          (storage.foldername(name))[1] = auth.uid()::text
      );

      DROP POLICY IF EXISTS "Users can download from their own folder" ON storage.objects;
      CREATE POLICY "Users can download from their own folder"
      ON storage.objects
      FOR SELECT
      TO authenticated
      USING (
          bucket_id = 'private-uploads' AND
          (storage.foldername(name))[1] = auth.uid()::text
      );

      -- ============================================================
      -- 3. RLS policies for public-processed bucket
      -- ============================================================

      DROP POLICY IF EXISTS "Public read access to processed media" ON storage.objects;
      CREATE POLICY "Public read access to processed media"
      ON storage.objects
      FOR SELECT
      USING (bucket_id = 'public-processed');

      DROP POLICY IF EXISTS "Service role can manage processed media" ON storage.objects;
      CREATE POLICY "Service role can manage processed media"
      ON storage.objects
      FOR ALL
      TO service_role
      USING (bucket_id = 'public-processed');

      DROP POLICY IF EXISTS "Users can read their own processed media" ON storage.objects;
      CREATE POLICY "Users can read their own processed media"
      ON storage.objects
      FOR SELECT
      TO authenticated
      USING (
          bucket_id = 'public-processed' AND
          (storage.foldername(name))[1] = auth.uid()::text
      );

      -- ============================================================
      -- 4. Enable pgmq extension and create the queue
      -- ============================================================
      CREATE EXTENSION IF NOT EXISTS pgmq;
      SELECT pgmq.create('labeling_queue');

      -- ============================================================
      -- 5. Create the trigger function for video labeling
      -- ============================================================
      DROP FUNCTION IF EXISTS handle_video_labeling() CASCADE;

      CREATE OR REPLACE FUNCTION handle_video_labeling()
      RETURNS TRIGGER AS $$
      DECLARE
          user_id text;
          file_path text;
          file_name text;
          file_extension text;
          base_filename text;
          msg jsonb;
          commands jsonb := '[]';
          video_mime_types text[] := ARRAY['video/mp4', 'video/quicktime', 'video/x-msvideo', 'video/webm', 'video/mpeg'];
      BEGIN
          IF NEW.bucket_id = 'private-uploads' AND NEW.metadata->>'mimetype' = ANY(video_mime_types) THEN
              user_id := (storage.foldername(NEW.name))[1];
              file_path := NEW.name;
              file_name := (storage.filename(NEW.name));
              file_extension := split_part(file_name, '.', array_length(string_to_array(file_name, '.'), 1));
              base_filename := replace(file_name, '.' || file_extension, '');

              -- Command 1: Detect faces and objects (dnn_detect)
              commands := commands || jsonb_build_object(
                  'type', 'detect',
                  'output_path', user_id || '/labels/' || base_filename || '_detections.json',
                  'command', 'ffmpeg -i "INPUT_FILE" -vf "dnn_detect=dnn_backend=openvino:model=DETECT_MODEL:input=data:output=detection_out:confidence=0.6:labels=DETECT_LABELS,showinfo" -f null -'
              );

              -- Command 2: Classify emotions/scenes (dnn_classify)
              commands := commands || jsonb_build_object(
                  'type', 'classify',
                  'output_path', user_id || '/labels/' || base_filename || '_classifications.json',
                  'command', 'ffmpeg -i "INPUT_FILE" -vf "dnn_classify=dnn_backend=openvino:model=CLASSIFY_MODEL:input=data:output=prob_emotion:confidence=0.3:labels=CLASSIFY_LABELS,showinfo" -f null -'
              );

              -- Command 3: Overlay bounding boxes on video
              commands := commands || jsonb_build_object(
                  'type', 'overlay',
                  'output_path', user_id || '/labeled/' || base_filename || '_labeled.mp4',
                  'command', 'ffmpeg -i "INPUT_FILE" -vf "dnn_detect=dnn_backend=openvino:model=DETECT_MODEL:input=data:output=detection_out:confidence=0.6:labels=DETECT_LABELS,drawbox=x=1005:y=813:w=81:h=92:color=red" -c:v libx264 -crf 18 "OUTPUT_FILE"'
              );

              msg := jsonb_build_object(
                  'userId', user_id,
                  'inputPath', file_path,
                  'inputBucket', NEW.bucket_id,
                  'outputBucket', 'public-processed',
                  'fileName', file_name,
                  'baseFilename', base_filename,
                  'commands', commands,
                  'timestamp', NOW()
              );

              PERFORM pgmq.send('labeling_queue', msg::jsonb);

              INSERT INTO "render" (id, title, project, status, public, user_id, data)
              VALUES (
                  gen_random_uuid(),
                  file_name,
                  'video-labeling',
                  'queued',
                  false,
                  user_id::uuid,
                  msg
              );

              PERFORM pg_notify(
                  'labeling_channel',
                  jsonb_build_object(
                      'userId', user_id,
                      'filePath', file_path,
                      'status', 'queued',
                      'timestamp', NOW()
                  )::text
              );
          END IF;
          RETURN NEW;
      END;
      $$ LANGUAGE plpgsql SECURITY DEFINER;

      -- ============================================================
      -- 6. Attach the trigger to storage.objects
      -- ============================================================
      DROP TRIGGER IF EXISTS labeling_upload_trigger ON storage.objects;
      CREATE TRIGGER labeling_upload_trigger
      AFTER INSERT ON storage.objects
      FOR EACH ROW
      EXECUTE FUNCTION handle_video_labeling();

      -- ============================================================
      -- 7. Helper views for monitoring
      -- ============================================================
      DROP VIEW IF EXISTS labeling_queue_view;
      CREATE OR REPLACE VIEW labeling_queue_view AS
      SELECT
          msg_id,
          read_ct,
          enqueued_at,
          vt,
          message::jsonb as payload,
          (message::jsonb->>'userId') as user_id,
          (message::jsonb->>'fileName') as file_name
      FROM pgmq.q_labeling_queue
      ORDER BY msg_id DESC;

      -- ============================================================
      -- 8. Initialize notification channels
      -- ============================================================
      DO $$
      BEGIN
          PERFORM pg_notify('labeling_channel', '{"init": true}');
          PERFORM pg_notify('render_status_channel', '{"init": true}');
          PERFORM pg_notify('log_channel', '{"init": true}');
      END $$;
    

FFmpeg Commands

The trigger function generates the following FFmpeg commands:

1. Object / Face Detection (dnn_detect)

      ffmpeg -i "INPUT_FILE" -vf "dnn_detect=dnn_backend=openvino:model=DETECT_MODEL:input=data:output=detection_out:confidence=0.6:labels=DETECT_LABELS,showinfo" -f null -
    

💡

dnn_detect Parameters Explained

dnn_detect — FFmpeg's object detection filter
dnn_backend=openvino — DNN backend (openvino, tensorflow, native)
model — Path to the model file
input=data — Input tensor name
output=detection_out — Output tensor name
confidence=0.6 — Confidence threshold
labels — Path to labels file
showinfo — Prints detection results to console

2. Classification (dnn_classify)

      ffmpeg -i "INPUT_FILE" -vf "dnn_classify=dnn_backend=openvino:model=CLASSIFY_MODEL:input=data:output=prob_emotion:confidence=0.3:labels=CLASSIFY_LABELS,showinfo" -f null -
    

💡

dnn_classify Parameters Explained

dnn_classify — FFmpeg's classification filter
output=prob_emotion — Output tensor for probabilities
confidence=0.3 — Confidence threshold

3. Overlay Detection Results (drawbox)

      ffmpeg -i "INPUT_FILE" -vf "dnn_detect=...,drawbox=x=1005:y=813:w=81:h=92:color=red" -c:v libx264 -crf 18 "OUTPUT_FILE"
    

💡

Overlay Parameters Explained

drawbox — FFmpeg filter for drawing rectangles
x, y, w, h — Position and size of the bounding box
color=red — Color of the bounding box
-c:v libx264 -crf 18 — High-quality encoding

Model Management

Directory Structure

      /app/models/
      ├── detection/
      │   ├── face-detection-adas-0001.xml
      │   ├── face-detection-adas-0001.bin
      │   └── face-detection-adas-0001.label
      ├── classification/
      │   ├── emotions-recognition-retail-0003.xml
      │   ├── emotions-recognition-retail-0003.bin
      │   └── emotions-recognition-retail-0003.label
      └── object-detection/
          ├── yolo-v3-tiny.xml
          ├── yolo-v3-tiny.bin
          └── coco.names
    

Downloading Models

      # Create directories
      mkdir -p models/detection models/classification models/object-detection

      # Face Detection Model (OpenVINO)
      wget -O models/detection/face-detection-adas-0001.xml \
        https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/face-detection-adas-0001.xml
      wget -O models/detection/face-detection-adas-0001.bin \
        https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/face-detection-adas-0001.bin
      wget -O models/detection/face-detection-adas-0001.label \
        https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/face-detection-adas-0001.label

      # Emotion Recognition Model (OpenVINO)
      wget -O models/classification/emotions-recognition-retail-0003.xml \
        https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/emotions-recognition-retail-0003.xml
      wget -O models/classification/emotions-recognition-retail-0003.bin \
        https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/emotions-recognition-retail-0003.bin
      wget -O models/classification/emotions-recognition-retail-0003.label \
        https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/emotions-recognition-retail-0003.label
    

Configure ffmpeglab-runner

Step 1

Add the queue to your environment

Add the following to your .env file or Docker Compose configuration.

        # Labeling queue
        LABELING_QUEUE_NAME=labeling_queue

        # Detection model
        DNN_DETECT_MODEL=/app/models/detection/face-detection-adas-0001.xml
        DNN_DETECT_LABELS=/app/models/detection/face-detection-adas-0001.label

        # Classification model
        DNN_CLASSIFY_MODEL=/app/models/classification/emotions-recognition-retail-0003.xml
        DNN_CLASSIFY_LABELS=/app/models/classification/emotions-recognition-retail-0003.label

        # DNN backend
        DNN_BACKEND=openvino

        # Confidence thresholds
        DETECT_CONFIDENCE=0.6
        CLASSIFY_CONFIDENCE=0.3
      

Step 2

Implement the processing loop

The runner should execute the following steps:

        # 1. Connect to Supabase and listen to the queue
        
        # 2. For each job:
        
        #    a. Download the input file
        
        #    b. Parse the 'commands' array
        
        #    c. Execute each command:
        
        #        - Run dnn_detect (extract JSON metadata)
        
        #        - Run dnn_classify (extract JSON metadata)
        
        #        - Run overlay (generate labeled video)
        
        #    d. Upload outputs to public-processed
        
        #    e. Update render table

Step 3

Restart the runner

After updating the environment, restart the runner service.

docker compose restart ffmpeglab-runner

Monitor the Pipeline

Step 1

Check queued jobs

Use the helper view to see all queued jobs.

SELECT * FROM labeling_queue_view;

Step 2

Check render status

Query the existing render table for job status.

        SELECT id, title, status, progress, data
        FROM "render"
        WHERE project = 'video-labeling'
        ORDER BY created_at DESC;
      

Step 3

Listen to notifications

In your application, listen for real‑time updates.

        -- In your PostgreSQL client:
        LISTEN labeling_channel;
        LISTEN render_status_channel;
        LISTEN log_channel;
      

Frequently Asked Questions (FAQ)

What does the video labeling pipeline do?

The pipeline automatically detects and labels objects, faces, and scenes in uploaded videos using FFmpeg's dnn_detect and dnn_classify filters. It generates bounding boxes, labels, and confidence scores, and stores them as JSON metadata alongside the video.

What models are used?

The pipeline supports multiple models including YOLO for object detection, ResNet for classification, and face detection models. The default configuration uses OpenVINO models for face detection (face-detection-adas-0001), classification (emotions-recognition-retail-0003), and can be extended for general object detection.

What is the output format?

The pipeline outputs a JSON file containing all detection results: frame number, bounding box coordinates, label/class name, confidence score, and timestamp. This can be used for search, analytics, or further processing.

Can I use custom models?

Yes. You can use any OpenVINO or TensorFlow model that works with FFmpeg's dnn_detect or dnn_classify filters. You'll need to provide the model files and configure the input/output tensor names.

Final Word

You now have a fully automated video labeling pipeline that uses FFmpeg's DNN filters to detect objects, faces, and scenes in videos. With PostgreSQL triggers, pgmq, and Supabase Storage, you get:

Object and face detection — bounding boxes with confidence scores
Scene classification — labels for what's in the video
JSON metadata — rich, structured data for search and analytics
Labeled video output — visual overlay of detection results
Real‑time notifications — know when processing is complete
Full observability — monitoring views and logs

The pipeline is production‑ready, scalable, and extensible — you can swap in any DNN model for your specific use case.