Treviño's Blog

As previously announced, few days ago I attended the GNOME Fractional Scaling Hackfest that me and Red Hat‘s Jonas Ådahl organized at the Canonical office in Taipei 101.
Although the location was chosen mostly because it was the one closest to Jonas and near enough to my temporary place, it turned out to be the best we could use, since the huge amount of hardware that was available there, including some 4k monitors and HiDPI laptops.
Being there also allowed another local Caonical employee (Shih-Yuan Lee) to join our efforts!

As this being said I’ve to thank my employer, for allowing me to do this and for sponsoring the event in order to help making GNOME a better desktop for Ubuntu (and not only).

Going deeper into the event (for which we tracked the various more technical items in a WIP journal), it has been a very though week, hard working till late while trying to look for the various edge cases and discovering bugs that the new “logically sized” framebuffer and actors were causing.

In fact, as I’ve already quickly explained, the whole idea is to paint all the screen actors at the maximum scale value across the displays they intersect and then using scaled framebuffers when painting, so that we can redefine the screen coordinates in logical pixels, more than using pixel units. However, since we want to be able to use any sized element scaled at (potentially any) fractional value, we might incur in problems when eventually we go back to the pixel level, where everything is integer-indexed.

We started by defining the work items for the week and setting up some other HiDPI laptops (Dell XPS 15 and XPS 13 mostly) we got from the office with jhbuild, then as you can see we defined some list of things to care about:

• Supporting multiple scaling values: allowing to scale up and down (< 1.0) the interface, not only to well-known value, but providing a wider range of floats we support
• Non-perfect-scaling: covering the cases in which the actor (or the whole monitor) when scaled up/down to a fractional level has not anymore a pixel-friendly size, and thus there are input and outputs issues to handle due to rounding.
• GNOME Shell UI: the shell StWidget‘s need to be drawn at proper resource scaling value, so that when they’re painted they won’t look blurred.
• Toolkit supports: there are some Gtk issues when scaling more than 2x, while Qt has support for Fractional scaling.
• Wayland protocol improvements: related to the point above we might define a way to tell toolkits the actual fractional scaling value, so that they could be scaled at the real value, instead of asking them to scale up to the upper integer scaling level. Also when it comes to games and video players, they should not be scaled up/down at all.
• X11 clients: supporting XWayland clients

What we did

As you see the list of things we meant to work or plan was quite juicy, so more than enough for one week, but even if we didn’t finish all the tasks (despite the Super-Joans powers :-)), we have been able to start or address the work for most of them so that we’ll know what to work on for the next weeks.

As a start, we had to ensure mutter was supporting various scaling values (including the ones < 1.0), we decided (this might change, but given the Unity experience, it proved to work well) to support 8 intermediate values per integer, from 0.5 to 4.0. This, as said, would lead to troubles when it comes to many resolutions (as you see in the board picture, 1280×720 is an example of a case that doesn’t work well when scaled at 1.5 for instance). So we decided to make mutter to expose a list of supported scaling values per each mode, while we defined an algorithm to compute the closest “good” scaling level to get a properly integer sized logical screen.
This caused a configuration API change, and we updated accordingly gnome-settings-daemon and gnome-control-center adding also some UI changes to reflect and control this new feature.
Not only, the ability of having such fractional values, caused various glitches in mutter, mostly related to the damage algorithm, which Jonas refactored. Other issues in screenshots or gnome-shell fullscreen animations have also been found and fixed.

Speaking of Gnome Shell toolkit, we started some work to fix the drawing of cairo-based areas, while I had already something done for labels, that needs to be tuned. Shih-Yuan fixed a scaling problem of the workspace thumbnails.

On toolkits support, we didn’t do much (a part Gnome Shell) as Gtk problem is not something that affects us much in normal scenarios yet, but still we debugged the issue, while it’s probably a future optimization to support fractional-friendly toolkits using an improved wayland protocol. Instead it’s quite important to define a such protocol for apps that don’t need to be scaled, such as games, but in order to do it we need feedback from games developers too, so that we can define it in the best way.

Not much has been also done in XWayland world (right now everything is scaled to the required value by mutter, but the toolkit  will also use scale 1, which would lead to some blurred result), but we agreed that we’d probably need to define an X11 protocol for this.

We finally spent some time for defining an algorithm for picking the preferred scaling per mode. This is a quite controversial aspect, and anyone might have their ideas on this (especially OEMs), so far we defined some DPI limits that we’ll use to evaluate weather a fractional scaling level has to be done or not: outside these limits (which change depending we’re handling a laptop panel or an external monitor [potentially in a docked setup]) we use integer scaling, in between them we use instead proportional (fractional) values.
One idea I had was to see the problem the other way around and define instead the physical size (in tenth of mm) we want for a pixel at least to be, and then scale to ensure we reach those thresholds instead of defining DPIs (again, that physical size should be weighted for external monitors differently, though). Also, hardware vendors might want to be able to tune these defaults, so one idea was also to provide a way for them to define defaults by panel serial.
In any case, the final and most important goal, to me, is to provide defaults that guarantee usable and readable HiDPI environments, so that people would be able to use gnome-control-center and adjust these values if needed.
And I think could be quite also quite useful to add to the gnome-shell intro-wizard an option to chose the scaling level if an high DPI monitor is detected.
For this reason, we also filled this wiki page, with display technical infos for all the hardware we had around, and we encourage you to do add your infos (if you don’t have write access to the Wiki, just send it to us).

What to do

As you can see in our technical journal TODO, we’ve plenty of things to do but the main thing is currently fixing the Shell toolkit widgets, while going through various bugs and improving the XWayland clients situation. Then there multiple optimizations to do at mutter level too.

When we ship

Our target is to get this landed by GNOME 3.26, even if this might be under an experimentalgsettings key, as right now the main blocker is the X11 clients support.

How to help

The easiest thing you can do is help testing the code (using jhbuild build gnome-shell with a config based on this) should be enough), also filling the scale factor tests wiki page might help. If you want to get involved with code, these are the git branches to look at.

Read More

You can read a more schematic report that Jonas wrote for this event on the gnome-shell mailing list.

Conclusions

It has been a great event, we did and discussed about many things but first of all I’ve been able to get more closely familiar in the GNOME code with who has wrote most of it, which indeed helped.
We’ve still lots of things to do, but we’re approaching to a state that would allow everyone to get differently scaled monitors at various fractional values, with no issues.

Finally, I’ve to say thanks a lot to Jonas who initially proposed the event and, a part being a terrific engineer, has been a great mate to work and hang out with, making me discover (and survive in) Taipei and its food!