How can UIs keep up with higher resolution video?

If you buy a new TV today, the chances are it is 4K. It’s quite strange that the UIs on these TVs are still rendered at 1080p and upscaled, but that’s the reality. It means that any text and graphics in the video stream, such as football scores, are visually sharper and clearer than the TV’s own menus and app UIs.

A UI at lower resolution than the video

At IBC this year there were several stands showing the latest innovations in 8K TVs, but none were focusing on the user interface. This is surprising given that the UI is an integral part of the viewing experience and is often considered by the customer as an indication of overall product quality.

For most products the UI is the main opportunity for differentiation. This ought to be the area where higher resolution TVs shine, but instead UI resolutions lag behind video resolutions. Today, 4K TVs render their UI at HD resolution, just as UIs on HD TVs remained at SD for several years; even 8K TVs currently only have upscaled HD UIs.

As the availability of higher resolution video increases and the average size of TVs grows, the disparity between video resolution and UI resolution becomes increasingly noticeable. Despite this, the next generation of 8K TVs won’t jump straight to an 8K UI. Instead it is likely that they will migrate to a 4K UI first and remain there for some time.

The challenges of increasing UI resolutions

The technical challenges created by a step change in UI resolutions offer new and interesting problems. The amount of raw data that makes up a 4K display is four times that of an HD display. Refreshing this data at 60 fps is more than all but the fastest TV SoCs are capable of today. Fortunately, increases in bus bandwidth and GPU performance will make this more widely available before long. However, in order to deliver 4K at 60 fps, the software needs to be able to use the hardware effectively.

Software frameworks and rendering engines aim to shield the UI and app developers from these hardware issues. However, if these can’t keep up with the increases in resolution, the visual performance of the UIs will degrade significantly.

HTML is commonly used to author UIs and apps. Unfortunately, for traditional HTML browsers, 4K presents something of a problem. For higher resolution UIs, the hardware acceleration techniques that traditional browsers use to provide smooth animations are significantly more hardware demanding than the developments in TV SoCs will be able to handle.

Accelerated compositing is the most common method of hardware acceleration used in browsers today; this caches areas of the screen in temporary pixmaps and manipulates them using the GPU. Unfortunately, the extra memory needed to store these pixmaps can be significant; at 4K resolutions it’s not uncommon to require more than 100MB. So long as the contents of the pixmaps don’t change, the GPU can be used to animate them with little or no load on the CPU. However, if the contents of the pixmaps need to change mid-animation, they must be redrawn from scratch which will cause UI glitches.

The right tools for the job

By contrast, Ekioh’s clean-room HTML Flow browser was designed with the evolution of silicon in mind. Instead of relying upon accelerated compositing, Flow uses the GPU more efficiently – it redraws the entire screen for each frame update. This may seem like it ought to be slower, but it isn’t because GPUs are specifically optimised for this. 

By redrawing everything each frame, Flow doesn’t suffer from UI glitches and saves memory by not using any cached pixmaps. It also eliminates the need to calculate which screen elements require updating; this greatly reduces the CPU load. Because of this approach, which mirrors techniques from the gaming industry, Flow’s graphical rendering performance is many times faster than the fastest traditional browser.

By enabling the optimal use of the GPU and using all the CPU cores for layout, Flow is the only browser capable of rendering a 4K UI on TV silicon. This makes it the optimal platform to deliver the next generation of exciting applications and UIs that this significantly increased resolution will enable.