Video generation optimization runbook

​

Status today

​

Planned stack target

​

Quality gates

​

Planned detection, routing, and application

​

Verification before production

1. The plan should show the selected video-generation technique.
2. Benchmarks should report per-frame latency, total video latency, peak VRAM, and quality deltas.
3. Generated runtime files should preserve the same precision and compile settings.

​

Local verification added

• Engine and dashboard local capability mirrors now expose video-gen as testable:true, optimizable:true, deployable:false, serving backend diffusers, and allowed optimizer serving only.
• The video-generation recipe now declares hardware, baseline, and Diffusers serving phases with benchmarkAdapter:"video-generation", /video-gen-benchmark, video_second billing, and videoSemanticAlignment as the lab quality metric. Temporal consistency remains canary telemetry, not production no-loss proof.
• Live lab intake now accepts video-gen plus Diffusers manifests with a runnable baseline and serving candidate on wired video-capable Modal GPUs, while non-Diffusers video backends and L4 video runs stay blocked before paid work.
• Modal Diffusers video runtime methods export MP4 output through Diffusers export_to_video and report video-output integrity before counting benchmark successes.
• Modal Diffusers video runtime methods compute a frame-derived temporal-consistency score and threshold. /video-gen-benchmark aggregates the score and fails below-threshold results even when MP4 integrity passes.
• Modal Diffusers video runtime methods compute video_semantic_alignment when semantic eval is requested by scoring sampled generated frames against the prompt with CLIP. /video-gen-benchmark aggregates the semantic score and the load-test adapter exposes it as qualityGate.source: "modal:<callId>:video_semantic_alignment" when requested. Semantic eval requires an explicit min_semantic_alignment; the live lab runner now supplies the local threshold for video-gen Diffusers serving manifests, and the final five-model certifier still requires saved Modal evidence from approved paid runs.
• The TypeScript Modal client posts video-native fields to /video-gen-diffusers and /video-gen-benchmark, including num_frames, fps, image-to-video conditioning, decode_chunk_size, motion_bucket_id, and noise_aug_strength.
• The TypeScript Modal client also posts optional semantic fields: semantic_quality, min_semantic_alignment, and clip_model_id.
• The dashboard Test tab now has a local video-generation preview path. Optimized and deployed video previews send Modal Diffusers decode chunking, attention slicing, VAE tiling, and channels-last through /api/infer; explicit requests can also pass model CPU offload. Engine routes the request to Modal /video-gen-diffusers, maps applied-knob telemetry back to the dashboard with the returned MP4 data for preview/download, propagates request aborts, and cancels submitted Modal video calls on abort, timeout, or poll failure. This is a local Modal-first preview path, not production certification.
• When a video-generation Test target has a real optimized version and execution trace, the Test tab exposes a Compare action that starts /api/infer/dual-run with the same prompt and video request shape for baseline and optimized runtimes. The compare view renders the returned MP4 artifacts, dimensions, frame count, FPS, video semantic score, MP4 integrity, cache-applied evidence, GPU, and latency side by side. This is a user testing and diagnosis surface; production certification still requires the source-bound Modal streak and saved /result verification gates below.
• The dashboard Deploy surface recognizes native video-generation nodes as video-gen/Diffusers workloads, but managed and BYOC deploy remain disabled with the same five-model Modal proof gate. Completed video-generation runbooks route the post-runbook action to Export instead of Deploy, and Export/codegen remains the testable hosting-prep path until production certification exists.
• runLoadTest({ modality: "video-gen", servingBackend: "diffusers" }) now routes to the dedicated video benchmark adapter instead of image-gen or generic stress paths, and the live lab serving runner requests semantic video quality before accepting measured candidates.
• Generic benchmark_serving_config video-generation runs are video-native now: they use total video latency, videos/sec, video_second billing, and a default semantic-alignment gate before any measured candidate can be applied. They do not treat temporal consistency as production no-loss proof.
• The generic promotion quality-gate registry now has a config-gated videoSemanticAlignment adapter for video generation. With the flag off it is indistinguishable from an unwired gate and returns pending without a Modal call; with MODAL_VIDEO_GEN_QUALITY_EVAL_ENABLED=true, it runs the existing Modal /video-gen-benchmark semantic path and returns the measured video_semantic_alignment score only when Modal reports a passed semantic gate.
• Engine export and the dashboard Code tab now generate portable Diffusers runtime artifacts for video-gen models: Dockerfile, Docker Compose, and K8s config that preserve the base image entrypoint and point at /v1/videos/generations. The base Diffusers runtime uses DiffusionPipeline, exports MP4 output with export_to_video, reads the exported video optimization envs, and fails /health until the pipeline is loaded.
• The generated portable runtime carries video-native request defaults such as num_frames, fps, duration_seconds, width, height, inference steps, and guidance scale. It also persists Modal-proven video optimization defaults through DIFFUSERS_VIDEO_CACHE_STRATEGY, DIFFUSERS_VIDEO_DECODE_CHUNK_SIZE, DIFFUSERS_VIDEO_ATTENTION_SLICING, DIFFUSERS_ENABLE_CHANNELS_LAST, DIFFUSERS_ENABLE_VAE_TILING, and DIFFUSERS_ENABLE_MODEL_CPU_OFFLOAD; requested unsupported self-host knobs fail closed instead of silently serving a weaker baseline. The managed/BYOC RunPod Diffusers env builder now preserves the same explicit video defaults and optimization envs when that delivery path is later validated. Portable and managed/BYOC runtime artifacts remain for local review or later delivery validation only and do not replace the Modal evidence loop.
• The load-test adapter exposes Modal semantic alignment as the returned quality gate when requested, with source: modal:<callId>:video_semantic_alignment; temporal consistency remains canary telemetry and MP4 integrity remains separate as videoOutputIntegrity.
• Hardware profiling delegates video-gen candidates through the dedicated load-test adapter.
• Invalid prompt, dimensions, frame count, total frame-pixel budget, fps, and video-specific knobs are gated before paid Modal submission.
• Explicit Wan GGUF Modal requests now have a no-paid loader boundary: gguf_family:"wan", gguf_base_model_id, and gguf_transformer_file must travel together through preflight, the TypeScript Modal client, and /video-gen-benchmark; Modal reports pipeline_loader:"gguf-wan-transformer" plus the same GGUF metadata when that path runs. This is loader scaffolding only by default. GGUF rows become supportable in the HF coverage gate only when --gguf-loader-evidence-file supplies matching source-bound Modal loader evidence. Those rows become owner-runnable only when an explicit dry-run plan carries the same GGUF loader metadata and the saved Modal output reports the same loader.
• scripts/live-modal-optimization-certify.ts now threads --prompt, --num-frames, --fps, --decode-chunk-size, --attention-slicing, --enable-channels-last, --enable-vae-tiling, --enable-model-cpu-offload, --video-cache-strategy, --quantization, --compile, and --compile-mode into the Diffusers video benchmark request. Model CPU offload is rejected before paid work when paired with FP8, torch.compile, GGUF loader metadata, or multi-GPU targets. The video load-test result now preserves the Modal call id in result metadata so saved evidence can bind to the measured Modal run.
• scripts/live-modal-optimization-certify.ts now threads --video-semantic-eval, --min-semantic-alignment, and --clip-model-id into the Diffusers video benchmark request. --min-semantic-alignment is required and must be greater than 0 when --video-semantic-eval is set. This is still no-paid local wiring until an owner approves live Modal runs.
• scripts/live-modal-optimization-certify.ts now accepts --source-commit-sha <40hex>, refuses paid live certification when that value does not match the current Engine checkout or the checkout has uncommitted changes, and writes the source commit into the saved video Modal evidence.
• scripts/certify-optimization-techniques.ts emits source-bound video-gen/diffusers Modal commands with a real 40-character Engine commit SHA, so the generic technique planner and the dedicated streak planner both feed the same direct live-certifier safety check before paid work.
• The default dedicated streak planner now uses video-pyramid-attention-broadcast as the optimized speed candidate on high-VRAM single-GPU runs. video-first-block-cache is emitted as an explicit canary only. Attention slicing, channels-last, model CPU offload, and VAE tiling remain separate Modal-attested lab or fit candidates, not hidden extras on the cache command.
• scripts/live-modal-optimization-certify.ts now accepts and emits --seed plus the request shape used by video benchmarks: prompt, dimensions, frame count, FPS, inference steps, guidance scale, seed, image-conditioning SHA-256 plus content type for image-to-video requests, and optional semantic-eval fields. Raw image payloads are not serialized into saved evidence. The dry-run planner includes a fixed seed so baseline and optimized rows are comparable.
• Modal /video-gen-benchmark, the load-test adapter, live-cert output, and the saved-result certifier now bind applied optimization proof. Baseline rows must match the planned unoptimized request shape. Optimized PAB and First-Block rows must prove Modal reported video_cache_applied=true with the requested video_cache_strategy. Channels-last and fit rows must prove channels_last_applied=true, attention_slicing_applied=true, model_cpu_offload_applied=true, or enable_vae_tiling_applied=true when those knobs are selected. If a worker fails to apply a requested knob, certification fails.
• The video candidate generator currently emits Modal-attested or canary candidates: FP16 baseline, Pyramid Attention Broadcast, First-Block Cache, high-VRAM channels-last, attention slicing, model CPU offload for single-GPU fit, and VAE tiling. FP8, torch.compile, decode chunking, and USP context-parallel candidates are intentionally not emitted until Modal applies and reports those techniques.
• Model CPU offload follows the Diffusers enable_model_cpu_offload() path and is treated as a memory/fit lever, not a speed claim. Modal evidence must report model_cpu_offload_applied:true before the candidate can count.
• The RunInfra dashboard runbook execution UI mirrors that execution boundary: A100-class, L4, B200, and multi-GPU video-gen plans do not list the FP8 plus torch.compile native technique. L40S, H100, and H200 single-GPU plans can show it when the rest of the Modal Diffusers path is selected.
• scripts/hf-video-gen-coverage-probe.ts creates the source-bound no-paid HF support report required before paid command output is usable:

pnpm exec tsx scripts/hf-video-gen-coverage-probe.ts --limit 50 --candidate-search-limit 100 --threshold 0.9 --source-commit-sha <sha> --json-output --output runs/video-gen/hf-video-coverage.json

pnpm exec tsx scripts/hf-video-gen-coverage-probe.ts --limit 50 --threshold 0.9 --source-commit-sha <sha> --gguf-loader-evidence-file runs/video-gen/wan-gguf-loader-evidence.json --json-output --output runs/video-gen/hf-video-coverage.json

{
  "source": "video-gen-gguf-loader-evidence",
  "measured": true,
  "provider": "modal",
  "backend": "diffusers",
  "modality": "video-gen",
  "sourceCommitSha": "<sha>",
  "rows": [
    {
      "model": "city96/Wan2.1-T2V-14B-gguf",
      "status": "passed",
      "modalCallId": "fc-...",
      "pipelineLoader": "gguf-wan-transformer",
      "ggufFamily": "wan",
      "ggufBaseModelId": "Wan-AI/Wan2.1-T2V-14B-Diffusers",
      "ggufTransformerFile": "wan2.1-t2v-14b-Q4_K_M.gguf"
    }
  ]
}

{
  "ali-vilab/i2vgen-xl": {
    "imageFile": "fixtures/i2vgen-xl.png",
    "imageContentType": "image/png"
  }
}

pnpm exec tsx scripts/hf-video-gen-coverage-probe.ts --models <id,id,...> --image-fixtures-file fixtures/video-gen-image-fixtures.json --source-commit-sha <sha> --json-output --output runs/video-gen/hf-video-coverage.json

• scripts/certify-video-gen-optimization-streak.ts can print the no-paid command plan without launching paid work, but paid command output requires the source-bound coverage report:

pnpm exec tsx scripts/certify-video-gen-optimization-streak.ts --hf-coverage-report runs/video-gen/hf-video-coverage.json --expected-source-commit-sha <sha> --json-output --output runs/video-gen-optimization-streak/summary.json

pnpm exec tsx scripts/certify-video-gen-optimization-streak.ts --hf-coverage-report runs/video-gen/hf-video-coverage.json --expected-source-commit-sha <sha> --env-file <engine-env-file> --json-output --output runs/video-gen-optimization-streak/summary.json

pnpm exec tsx scripts/certify-video-gen-optimization-streak.ts --models <id,id,...> --image-fixtures-file fixtures/video-gen-image-fixtures.json --hf-coverage-report runs/video-gen/hf-video-coverage.json --expected-source-commit-sha <sha> --json-output --output runs/video-gen-optimization-streak/summary.json

• scripts/run-video-gen-optimization-streak.ts is the operator path for approved paid Modal execution:

pnpm exec tsx scripts/run-video-gen-optimization-streak.ts --plan runs/video-gen-optimization-streak/summary.json --results-dir runs/video-gen-optimization-streak/live-results
pnpm exec tsx --env-file=<engine-env-file> scripts/run-video-gen-optimization-streak.ts --plan runs/video-gen-optimization-streak/summary.json --results-dir runs/video-gen-optimization-streak/live-results --confirm-live --max-total-cost-usd <usd>

• scripts/certify-video-gen-optimization-streak.ts can validate saved Modal evidence after approved Modal runs:

pnpm exec tsx scripts/certify-video-gen-optimization-streak.ts --results-dir runs/video-gen-optimization-streak/live-results --expected-source-commit-sha <sha> --verify-modal-results

​

Availability notes

• Video-generation optimization is hosted and testable for owner-approved Modal canaries, but still requires the five-model quality-preserving 2x-4x Modal streak before production certification.
• Modal is the required optimization and verification loop for this work. RunPod BYOC can only validate a later delivery path after Modal evidence exists.
• Context parallelism will depend on multi-GPU runtime support.
• First-Block Cache is functional as a Modal-applied canary, but it remains blocked from production promotion until it proves a 2x-4x quality-preserving five-model streak.
• This page will move from in-development to available only after the video-gen path produces owner-approved Modal measurements, quality-preserving 2x-4x wins, synchronized UI/export support, and production certification evidence.