HDR video is coming to Firefox for Windows users (and has been available for some time on macOS). This blog post explains how we developed the feature and gives a retrospective on the technical choices we made.
A primer on video playback for the web:
- Video file demux and decode: A video stream generally consists of parallel image and audio streams, along with captions, HDR scene metadata, and the like. “Container” formats like MP4 or MKV specify how these streams are combined, or multiplexed, into a single byte stream for transmission. On receipt, Firefox needs to divide that byte stream back into the individual media streams; this is de-multiplexing or “demuxing”. Then Firefox must uncompress the data to get images, audio samples, and so on. Firefox’s media team provides the demuxers, and pulls in appropriate codecs to decode them. We prefer using hardware video decoders if they work reasonably well. Video decompression usually produces roughly a YUV 4:2:0 image in . (If you visit about:support in Firefox, and search for Codec Support Information (or one of the codec names like AV1), you can see a whole feature matrix of support details for which codecs are hardware and software on your system.)
- Gecko displaylist building: Given a demultiplexed, uncompressed frame of video, Gecko displaylist building incorporates it into a video element in the displaylist being sent to WebRender. If the frame was decoded in hardware, it is generally represented by a texture in GPU memory. Or, if it was decoded in software, then it is represented by a memory mapping holding some raw pixel data in system memory shared with Firefox’s media decoder process.
- WebRender: Given the video element in the displaylist, WebRender decides whether to promote it to a desktop compositor overlay, or whether it must instead be rendered using a pathway more like an ordinary HTML element. A compositor overlay is faster and uses less power; on Windows this uses DWM with the . But if complex CSS is involved (rounded corners, blur filters, or similar features), Firefox must use WebRender’s ordinary rendering pathway. Currently the latter is not HDR capable, so Firefox favors the desktop compositor overlay for animated elements such as video and canvas.
As we began designing Firefox’s HDR support, we had to lay out some assumptions and found many complications:
- Initially, we had hoped that on a modern system, :: or function and the latter is ARIB-STD-B67 also known as HLG, the , calling SetColorSpace1 with this value seems to be ignored on P010 (at least in testing on AMD), so it incorrectly displays BT2100 PQ video as if it were BT709, which makes the video dull and muddy, since BT709 is a narrower gamut than BT2020, and the BT1886 transfer function used by BT709 is very different from PQ defined by BT2100. SetColorSpace1 may work on other vendors with P010, so it may be a valid optimization, but we were looking for a universal solution.
- For the future, Windows 11 23H2 has added a new interface called IDCompositionTexture which may serve our purposes better; from what we have been told, it is universally supported for all formats and color spaces. We haven’t used it for video so far, but it’s an interesting future direction.
- Until HDR came along, Gecko and WebRender only used desktop compositor overlays as a power/performance optimization. With HDR, overlays become a necessity as the pixel format and color space differ from classic sRGB.
- Fortunately, HDR videos tend to be shown without particularly fancy CSS rendering such as clip masks and rounded corners, which would require WebRender to perform further copies. Technically, DirectComposition does support all of those features, but Firefox doesn’t use that functionality much.
- In the future, we expect to upgrade WebRender for HDR rendering, allowing us to deal with complex cases like clip masks or blur filters on video elements.
- In favor of VideoProcessorBlt:
- It uses less power on GPUs that have a video processor unit.
- We discovered in testing (using by Erik Reinhard et al, and configuring it for a fixed brightness ratio of 400 cd/m^2 -> 100 cd/m^2 when used on SDR displays, and see if that fits all HDR content on the web well enough for a good user experience – and if it does not, we will iterate based on feedback from users on Firefox Nightly.
- We are hoping that we will never have to apply tonemapping for HDR content on HDR displays, there are multiple factors in this decision:
- Varying the brightness limit would make it a significant fingerprinting vector if not handled very carefully if the script can inspect pixels or parameters related to that. There are ways to mitigate this but they are all awkward restrictions to impose, and queries would have to get a different answer than what the rendering is using.
- Phones and laptops with light sensors may vary the reference brightness in real time, and this changes the maximum displayable ratio (aka HDR headroom) every refresh, which is also a major battery drain if we keep redrawing all of the time.
- Documents composed of multiple images (a gallery or some form of art composition) would apply different tonemapping to each image if the brightest pixel in each image is different brightness). We’d have to do something about that to make it controllable via CSS.
- In general the detailed parts of an image are within a certain brightness band – see , bug reports have guided us to focus on the use-cases that matter to people using Firefox. When we succeed, it’s a great feeling.
We’re working on extending HDR support to photos, apps/games and general web content.
↗ Original-Artikel auf mozillagfx.wordpress.com lesenVollständiger Original-BerichtAusführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf mozillagfx.wordpress.com.
SOCIAL SHARE CARD GENERATOR