Assessing the Viability of Lossy Compression for On-Set Virtual Production

Core Concepts

Modern lossy video codecs like AV1 and HEVC can replace intermediate codecs like HAP and NotchLC in virtual production workflows, offering comparable visual quality at significantly reduced bitrates, potentially leading to more efficient storage and real-time playback capabilities.

Abstract

This research paper investigates the feasibility of replacing commonly used intermediate codecs in virtual production (VP) with modern lossy compression formats like H.264, HEVC, and AV1. The authors argue that despite the industry's preference for pristine image quality, the inherent limitations of LED wall rendering and camera capture systems introduce a certain level of degradation, making the impact of lossy compression less significant than traditionally believed.

Research Objective:

The study aims to evaluate the visual quality of various compression formats when used for displaying content on LED walls in a VP setting and compare their performance against established intermediate codecs.

Methodology:

The researchers designed an experiment involving the playback of compressed video sequences on an LED wall and capturing the output using a high-quality camera. They tested different codecs (H.264, HEVC, AV1, HAP, NotchLC, Daniel2) across various bitrates and GOP sizes. Objective quality metrics like PSNR, VMAF, and CVVDP were employed to assess the visual fidelity of the compressed videos relative to the uncompressed reference footage.

Key Findings:

• Hybrid codecs (H.264, HEVC, AV1) demonstrated comparable or superior visual quality to intermediate codecs (HAP, NotchLC, Daniel2) at significantly lower bitrates.
• AV1 and HEVC achieved near-lossless quality at bitrates 10 times lower than intermediate codecs for certain content types.
• Encoding speed using hardware-accelerated implementations of hybrid codecs surpassed that of intermediate codecs.
• The study revealed that the VP setup itself introduces a certain level of visual degradation, making the impact of lossy compression less pronounced.

Main Conclusions:

The findings suggest that modern lossy codecs, particularly HEVC and AV1, are viable alternatives to intermediate codecs in VP workflows. Their ability to deliver high visual quality at significantly reduced bitrates presents opportunities for optimizing storage requirements and real-time playback capabilities.

Significance:

This research challenges the prevailing notion that VP necessitates the use of lossless or near-lossless codecs. The adoption of lossy compression could lead to more efficient and cost-effective VP workflows without compromising visual fidelity.

Limitations and Future Research:

The study primarily relied on objective quality metrics. Future research could incorporate subjective evaluations to validate the findings from a human perception perspective. Further investigation into the impact of different LED wall configurations and camera settings on the perceived quality of compressed video is also warranted.

Stats

Hybrid codecs achieved the same quality at an order of magnitude less bitrate (x10) depending on the chosen QP setting and content.
For simpler content, x10 bitrate reduction is possible, while for content with more texture, the benefit is in the x2-x5 range.
The upper ranges of VMAF and PSNR are about 10-20% less than the theoretical maximum possible regardless of the encoder.
The maximum PSNR observed was not 50dB but 37 dB within a very small range of 0.02 dB.
Average gain of about 12x in bitrate savings for HEVC or AV1 in Intra only mode.
Using motion prediction mode of the encoders, dramatically improved savings are now possible for no loss of objective quality, e.g., for 2P5D we approach 230x savings.
CVVDP is at least 9.4 out of 10 max, indicating good color preservation across all codecs.

Quotes

"the impact of the quality of the data source might be much less than expected."
"the bigger limiting factor in picture quality for VP is the LED Panel rendering and the aliasing between the panel and the camera capture, not so much the hybrid codec choice."

Key Insights Distilled From

Demystifying the use of Compression in Virtual Production

by Anil... at arxiv.org 11-04-2024

https://arxiv.org/pdf/2411.00547.pdf

Deeper Inquiries

How might advancements in LED panel technology and camera resolution affect the viability of using lossy compression in future VP setups?

Advancements in LED panel technology and camera resolution are a double-edged sword for lossy compression in VP. On one hand, these advancements push for even higher fidelity, potentially making the subtle differences introduced by lossy compression more apparent. Here's how:

Increased Resolution: Higher resolution LED panels and cameras mean more pixels to process. While this allows for more detailed virtual environments and captures, it also means that even small compression artifacts could become visible. The differences in encoding algorithms, particularly at lower bitrates, might become more pronounced as those minute details are rendered and captured with greater precision.
High Dynamic Range (HDR): As LED panels and cameras embrace wider color gamuts and higher dynamic range, the nuances of color and luminance become more critical. Lossy compression, especially at lower bitrates, might struggle to preserve these subtle gradations, potentially leading to banding or a loss of detail in highlights and shadows.
Frame Rate: Higher frame rates, such as 60fps or even 120fps, are becoming increasingly common in VP for smoother motion. This increase in temporal information puts additional strain on compression algorithms. Maintaining visual fidelity at higher frame rates requires more bits, potentially making the benefits of lossy compression less significant.
On the other hand, these advancements also provide opportunities to further leverage lossy compression:

Improved Perceptual Encoding: Codec development is constantly evolving. Future codecs might become even more sophisticated in exploiting the limitations of human perception, allowing for higher compression ratios without sacrificing perceived quality. These advancements could leverage the characteristics of new display technologies and higher resolutions to further optimize the compression process.
Focus on Experience: Even with higher fidelity displays and cameras, the ultimate goal of VP is to create a believable and immersive experience for the viewer. As long as lossy compression artifacts remain imperceptible within the context of the overall production, their presence might be inconsequential. The focus could shift from pixel-perfect fidelity to a perceptually optimized viewing experience.
Ultimately, the viability of lossy compression in future VP setups will depend on a delicate balance between technological advancements, compression algorithms, and the perceptual thresholds of human vision.

Could the use of lossy compression in VP introduce any unforeseen challenges during post-production, particularly in color grading or visual effects compositing?

Yes, using lossy compression in VP could introduce challenges during post-production, particularly in color grading and VFX compositing:

Color Grading: Lossy compression algorithms discard color information deemed less perceptually important. While this is generally acceptable for final delivery, it can limit the flexibility of color grading in post-production. The reduced color information might lead to banding or posterization when pushing grades, especially in areas with subtle color transitions. This limitation could make it harder to achieve the desired artistic look or match the footage to other elements in the scene.
Visual Effects Compositing: VFX compositing often involves manipulating and layering images with high precision. Lossy compression can introduce subtle artifacts, such as blockiness or ringing around edges, which become more apparent during compositing. These artifacts might require additional cleanup or limit the ability to seamlessly integrate VFX elements into the scene. Additionally, if the compression algorithm introduces temporal inconsistencies, it could complicate tasks like motion tracking or object removal.
Generation Loss: Each time a lossy compressed image is decoded and re-encoded, further information is lost. This generation loss can accumulate through multiple stages of post-production, potentially degrading the final image quality. This issue is particularly relevant in VP workflows, where content might be compressed and decompressed multiple times during on-set visualization, editing, and final VFX work.
To mitigate these challenges, several strategies can be employed:

Higher Bitrate and Quality Settings: Using higher bitrate and quality settings during compression can minimize information loss and preserve more flexibility for post-production.
Intermediate Codecs: Employing visually lossless or intermediate codecs, while still offering compression benefits, can preserve more image data for post-production.
Careful Workflow Management: Implementing a robust workflow that minimizes unnecessary compression and decompression cycles can help preserve image quality throughout post-production.
By understanding the potential challenges and adopting appropriate mitigation strategies, post-production teams can work effectively with lossy compressed VP footage while achieving the desired creative outcomes.

If the human eye cannot perceive the difference between lossless and high-quality lossy compression in a VP setting, does it change our understanding of "perfect" image fidelity in media production?

This question cuts to the heart of a philosophical debate within media production: is "perfect" image fidelity about achieving a bit-for-bit replica of reality, or is it about creating an experience indistinguishable from reality for the viewer?
If we accept that the human eye cannot perceive the difference, then the definition of "perfect" shifts away from purely technical measurements like PSNR and towards a more nuanced understanding of human perception. This has several implications:

Prioritizing Perceptual Quality: The focus shifts from eliminating all differences to eliminating perceptible differences. This prioritizes codecs and techniques that excel at exploiting the limitations of human vision, even if they result in technically "imperfect" representations of the source.
Optimizing for Experience: The goal becomes delivering the most compelling visual experience within the constraints of the medium. This might involve making strategic trade-offs, such as accepting minor compression artifacts in exchange for higher frame rates or reduced storage requirements.
Evolving Standards: The industry might need to re-evaluate its reliance on traditional metrics of image fidelity. New standards and evaluation methods that better align with human perception could emerge, emphasizing subjective testing and perceptual quality metrics.
However, this doesn't mean abandoning the pursuit of technical excellence. Even if imperceptible to the human eye, lossy compression can have downstream consequences in post-production or archival.
Ultimately, the understanding of "perfect" image fidelity is evolving alongside technology and our understanding of human perception. While lossy compression challenges the traditional definition, it also pushes the industry towards a more nuanced and holistic approach to visual quality, where the viewer's experience takes center stage.