Andrew Morgan's Blog

It's a blog

GSoC Weekly Progress Report #8

August 08, 2017 — Andrew Morgan

Hey there, welcome to another report! Since the last one, you may remember that I had finished setting up the build environment for Nautilus inside of GNOME Builder and was just getting started on properly doing the patch for supporting file trust attributes. Well, I'm happy to report that the patch is now finished! Screenshots and details below.

Working Demo

As screen recording in dom0 still isn't really doable yet, you'll have to settle for some screenshots of it in action :)

Double-clicking a trusted file opens normally... Double-clicking a trusted file opens normally...

Everyone knows gedit is the ultimate text editor. Everyone knows gedit is the ultimate text editor.

Whereas an untrusted file summons a DispVM! Whereas an untrusted file summons a DispVM!

Virus free, yes-sir-ee! Virus free, yes-sir-ee!

This short demo makes use of a majority of all the different components that have been worked on throughout the project. The now-patched Nautilus makes a call to the patched NautilusPython which asks our qvm_trust python extension to open a file, which it gives a response based on what our cli tool, qvm-file-trust, says about it, and if it is determined an untrusted file, open-file-trust-based will open it securely in a DisposableVM. The contents of the file were never parsed on the local machine!

The low-level part of this process was already working, but it took the patch to Nautilus allowing us to set off the chain from the GUI to really bring it all together. Let's talk about how that was done and how it works.

Nautilus and NautilusPython

By default, Nautilus does not support extensions written in Python, but rather only those written in C. To get around this, a Nautilus C extension called NautilusPython was created. This extension has the ability to provide a bridging interface from Nautilus' extension infastructure to Python, handling any calls Python programs make to Nautilus and calling them from C. Arguments and return variables are all passed along seamlessly.

This is a really nice feature, as writing extensions in Python are about 90% less work than writing them in C, due to the large amount of boiler-plate associated with setting up and tearing down an extension. If you recall, the point of patching Nautilus was to allow extensions to be able to be notified of when a file is being opened, and block that request if necessary. I didn't want to subject myself or anyone else to only being able to write an extension that made use of this new functionality in C, thus patching NautilusPython as well as Nautilus itself to add our new file_open function was necessary. So how was each one done then?

Patching NautilusPython to add a new method was relatively simple, you only need look at the existing methods that an extension can subclass, find one that is functionally similar your own (I used update_file_info as my guide), then insert your own method (file_open in this case) at all the relevant points as well as create the appropriate function bodies and set argument/return variables appropriately.

So overall NautilusPython wasn't too bad really. The real time sink was Nautilus itself.

Patching Nautilus

First of all, I almost immediately ditched GNOME Builder after my last blog post. It's not that it was a bad IDE, far from it actually, I quite liked the interface and the various tools it contained. The ability to download a project from git, package and run it inside a Flatpak, and then send patches upstream all from within the editor was quite appealing! The first major showstopper however, was that the version of Nautilus that Qubes makes use of (v3.22.3), doesn't support Flatpak. While building and running from GNOME Builder without Flatpak seemed possible, no matter what I tried I could not get extensions to function properly with the GNOME Builder version.

After a lot of fiddling about, I settled on what I probably should have been using since the very beginning: qubes-builder!

Not only does it build the correct version (and supports extensions), but I knew that whatever I did with this build system would be exactly what users would be using when they built Qubes' templates or got them from the build servers. So, unfortunately no Flatpak'd builds, but I do have pre-packaged RPMs with debug-symbols! That's probably even better, right?

I'll post them in a separate blog post with instructions on how to set everything up soon.

qubes-builder worked very well and was quite easy to drill into and mess about with, as it is mostly just a collection of various scripts. I eventually built a script that ran in dom0 that would build Nautilus in my development VM, transfer it to my testing VM, delete the old version and install the new version.

The only downside to this process was that building Nautilus took quite a while each time (2-3m) which made quick changes quite an agonizing process. I believe this was due to both having to rebuild every file each time (as they were packaged in a .tar.xz before being built, meaning all files were considered to be modified) as well as a call to dnf to fetch any updates was made, every time. I found out the latter when working without internet and realizing my could would not build! Each of these things probably could have been worked out or changed if I had felt the need to, but I just stuck with the loading times and usually had internet, so it was not too terrible.

Was was terrible and time-consuming were a couple of very strange issues I had during the creation of the patch.

The first was the dreaded provider issue. You see, every time Nautilus calls a method that is meant to be subclassed by a Nautilus extension, you must provide a provider. This can be an INFO provider, MENU provider, and so on. Each method only accepts one of each type, typically corresponding to the category of method you were calling. For the file_open method I decided to stick it under the INFO provider category, as that had to do with information about files, which I felt 'when they were opened' best fell into.

Well, these providers don't just pop out of nowhere. You have to create them! You can do that by calling the intuitively-named nautilus_module_get_extensions_for_type method, providing it the NAUTILUS_TYPE_INFO_PROVIDER enum. It is then supposed to return a provider with a type that matches the one you gave it, but no matter what I did it would always just return a NULL pointer! I spent days on this problem, trying loads of different things. Every other call of nautilus_module_get_extensions_for_type had no special setup or initialization, leading me confused why my own calling of it brought such different results.

After picking through many lines of code, talking with GNOME people over IRC who were currently GUADEC'ing, and trying to understand the whole GNOME development ecosystem, I eventually traced it all back to NautilusPython not finding an extension that made use of a subclassed method from the INFO method type. But wait, we do have an extension that does that with qvm_trust, so why doesn't it work??

Well I hit my head after this one folks. The version of NautilusPython that Nautilus was loading in my test VM was the original, from-repo version, not our modified version. D'oh!

"Well that's just a simple fix, I just have to build it and import it into Nautilus in the testing VM!", I assumed, before realizing that while my modified NautilusPython built, Nautilus would not accept it. Nautilus would just hang with little to no output.

After a few emails, Marek pointed me in the correct direction with the nautilus-3.0 branch of NautilusPython, which, while 6 years since the last commit, worked with the latest version. Alright no problem. Rebase my commits on that branch, rebuild, transfer the newly-built shared-library over and bam, Nautilus runs again! And we have a provider that's not NULL, hooray!

And then, ladies and gentlemen, problem #2 reared its ugly head.

Here's the code to get a list (GList*) of providers to use with a Nautilus extension:

providers = nautilus_module_get_extensions_for_type (NAUTILUS_TYPE_INFO_PROVIDER);

While we were getting a non-null provider value, the value we received was different from the one returned by the method we called. "Wat?" you say, and I too said the same. For some reason we were returning a 64-bit GList pointer from nautilus_module_get_extensions_for_type, but storing a 32-bit GList pointer in providers.

I tried all sorts of weird tricks to get this fixed. Nobody I talked to seemed to have a solution, and after wrestling with it for a few days I begrudingly settled on this very elegant workaround:

// TODO: Providers ends up as 0x55f65ac0
// when method is returning 0x555555f65ac0, very odd...
providers = (GList*) (nautilus_module_get_extensions_for_type (NAUTILUS_TYPE_INFO_PROVIDER) + 0x555500000000);

Yes I know this is terrible. It did work though, and I needed to stop hitting this strange bug and get on with patching, so that's where it currently sits. I'm still actively looking for a proper solution. If anyone knows why a 64bit pointer might get truncated to a 32bit pointer after a function call in C please do let me know! You can find the full code here and the type method here.

After this was out of the way, and a handful of more methods were implemented, we finally had a working solution between the qvm_trust extension and our patched Nautilus! It was beautiful, really.

Conclusion and going forward

My email to the Nautilus mailing list is still awaiting moderation. GUADEC has now concluded, and with no update on the situation it seems I need to do some more active poking of people to get moderation approval. We'll see where that lands, the code still needs to be cleaned up a bit before sending it upstream anyways, so I'm not too worried.

At this point in the project, we essentially have the following tasks left:

  • Patch KDE Dolphin to provide the same functionality as Nautilus now does
  • Finish up some small tasks with the daemon like logging and some more interfacing with the cli tool
  • Code cleanup, linting, tests
  • Integrate into main QubesOS repos

I'm going to prioritize the latter 3 bullet points at this point to ensure we get a really solid core implementation down of the project. I don't want to spend another couple weeks patching Dolphin only to run out of time for a general cleanup and other polish. I will try to pour through the Dolphin code and note down everything that one would need to do with Dolphin to make it work with file trust attributes, so that in the future either myself or someone else can give it a shot.

It may end up that Dolphin already has most of the functionality and the patch will be very small, we'll see!

Anyways, for now I'm going to start on finishing the remaining small tasks, and will post any questions regarding packaging and/or cleanup to the qubes-devel mailing list.

As always, you can find the code here.

And as of now, you can also find the patched code for Nautilus and NautilusPython on Github as well. Make sure that when you're inspecting the NautilusPython code, you're looking at the nautilus-3.0 branch. That's where all the important commits are!

That's all for now, thanks for reading!

Tags: gsoc-2017, progress-report