Andrew Morgan's Blog

It's a blog

GSoC Weekly Progress Report #9 - Final

August 29, 2017 — Andrew Morgan

Hey everyone, another blog post coming right on the heels of the final week of GSoC. I have to say that it's been an absolute blast participating, and I hope to have the chance to do so again before I graduate.

Barring that, I can say that qubes-MIME-types (now renamed to qubes-file-trust) as a project is now in a complete and working state! Fedora and Debian packages are currently being put together, and will hopefully show up in the repos in the not-too-far future.

And now, on to the final report!

Original goals

The original goals outlined in my proposal consist of the following:

  1. Implement a python context menu in Dolphin and Nautilus as well as a minimal GUI for modifying file trust settings
  2. Create a patch for Nautilus and Dolphin to follow these settings when opening a file
  3. Create a system daemon to watch and enforce file trust settings on files create inside of untrusted folders
  4. Documentation of implementation
  5. Unit testing and integration

In terms of meeting these goals, most were completed on time. The only task yet to be completed is a patch for KDE's Dolphin File Manager. The patch for Nautilus ended up taking at least 3 weeks, and thus towards the end of the project I decided to prioritize polish and bug fixes of the existing code over starting an entirely new patch.

Other than that though, the rest of the goals were met, and the code is now sitting in five separate repositories on my Github. There are, with explanation:

  1. The main repo, containing the python qvm-file-trust command line tool and the C++ system daemon that watches untrusted folders: qubes-file-trust

  2. The patched version of nautilus, patched to allow extensions to react to file-opening events and block them if necessary: qubes-nautilus-trust

  3. The patch version of nautilus-python, patched to allow the same thing as above: qubes-nautilus-trust-python

  4. Our nautilus extension that listens for file-opening events and blocks them if it references an untrusted file (and opens them in a disposable VM instead): qubes-nautilus-trust-extension

  5. Dolphin activities that allow one to change the trusted state of a file or folder from the right-click menu: qubes-dolphin-trust-activities

Recent updates

For those following the series, here are the main updates since the last posting:

Nautilus pointer truncation bug workaround

In the last blog post, I talked about a horribly hacky workaround that I was using to get around the pointer truncation bug that kept causing nautilus to crash. Not only was it hacky, but it also only worked inside of a running gdb instance, which is pretty useless.

Working with a friend, we managed to narrow the bug down to a scope issue. For some reason the GList* was being returned as an integer instead, which explains the 32-bit truncation (integers are 32-bit by default in C). Neither the nautilus maintainer, nor my mentors really had any clue about how to come up with a solution.

To rectify this, instead of grabbing the GList* from nautilus-mime-actions.c (which had scope issues), we sourced the pointer from the same file as the method that generated it, then transferred it to nautilus-mime-actions.c through a double-pointer. You can see the code here and here. Admittedly, this is still a workaround for the scope issue, but it's MUCH cleaner than before, and actually works outside of gdb, so that's handy.

Nautilus patch now supports opening multiple files at once

Yes, originally you could only open one file at a time with the patch (lame!). But, after learning a bit more about how extensions in nautilus are handled, this limitation has now been removed, and things are opened as you'd expect them to be.

Added new commands to qvm-file-trust such as --check-multiple and --check-multiple-all-untrusted

In earlier iterations, you were only able to check the trust state of a single file or folder at a time. There wasn't really a need to check multiple at once. That was until I tested opening multiple files from nautilus. We would have to make a separate call to qvm-file-trust for every single file, and spinning up and down the Python runtime, combined with extensions running in the same thread as nautilus itself, made everything way too slow.

These commands were added in order to check a large list of files all in one go. Both commands tell you which files are untrusted and which aren't, however they each return different error codes based on different situations. More explanation can be found in the updated man page (now written in RestructuredText!)

RPM packages and repo separation

As mentioned above, the qubes-mime-types repo has not only been renamed, but also broken up into three separate repos! This was to keep things clean, as well as to follow packaging, as it was decided to have one package per repo, instead of multiple stemming from a single codebase. Each repo now has its own rpm_spec folder, containing the SPEC files used to build RPM packages. These packages build and install successfully on my machine (minus some minor packaging issues, this is my first time packaging for Linux, so go easy :), and they will hopefully be available in Qubes Fedora (and debian!) repos in the not-too-distant future.

The were lots of little bug fixes across all the repos, and likely more to come once people start getting their hands on the packages. I'll make another post when they're available to download. I'll also continue to maintain the code even after my GSoC period to ensure there aren't any issues for anyone, as well as extend in little ways, such as adding support for upcoming Qubes R4.0 features (multiple DispVM templates, anyone?).

Thanks and closing thoughts

That's all for this week (and Summer). Thank you for taking the time to read my ramblings over the past few months. QubesOS is currently my all-time favorite operating system, and I can't wait to see how it flourishes in the seasons and releases to come.

I'd also like to thank my mentors, Marek Marczykowski-Górecki and Jean-Philippe Ouellet, as well as the rest of the Qubes team, for putting up with my questions and issues along the way.

It was both fun and an honor to work with you, and I hope to continue to do so in the future.

Tags: gsoc-2017, progress-report

GSoC Weekly Progress Report #8

August 08, 2017 — Andrew Morgan

Hey there, welcome to another report! Since the last one, you may remember that I had finished setting up the build environment for Nautilus inside of GNOME Builder and was just getting started on properly doing the patch for supporting file trust attributes. Well, I'm happy to report that the patch is now finished! Screenshots and details below.

Working Demo

As screen recording in dom0 still isn't really doable yet, you'll have to settle for some screenshots of it in action :)

Double-clicking a trusted file opens normally... Double-clicking a trusted file opens normally...

Everyone knows gedit is the ultimate text editor. Everyone knows gedit is the ultimate text editor.

Whereas an untrusted file summons a DispVM! Whereas an untrusted file summons a DispVM!

Virus free, yes-sir-ee! Virus free, yes-sir-ee!

This short demo makes use of a majority of all the different components that have been worked on throughout the project. The now-patched Nautilus makes a call to the patched NautilusPython which asks our qvm_trust python extension to open a file, which it gives a response based on what our cli tool, qvm-file-trust, says about it, and if it is determined an untrusted file, open-file-trust-based will open it securely in a DisposableVM. The contents of the file were never parsed on the local machine!

The low-level part of this process was already working, but it took the patch to Nautilus allowing us to set off the chain from the GUI to really bring it all together. Let's talk about how that was done and how it works.

Nautilus and NautilusPython

By default, Nautilus does not support extensions written in Python, but rather only those written in C. To get around this, a Nautilus C extension called NautilusPython was created. This extension has the ability to provide a bridging interface from Nautilus' extension infastructure to Python, handling any calls Python programs make to Nautilus and calling them from C. Arguments and return variables are all passed along seamlessly.

This is a really nice feature, as writing extensions in Python are about 90% less work than writing them in C, due to the large amount of boiler-plate associated with setting up and tearing down an extension. If you recall, the point of patching Nautilus was to allow extensions to be able to be notified of when a file is being opened, and block that request if necessary. I didn't want to subject myself or anyone else to only being able to write an extension that made use of this new functionality in C, thus patching NautilusPython as well as Nautilus itself to add our new file_open function was necessary. So how was each one done then?

Patching NautilusPython to add a new method was relatively simple, you only need look at the existing methods that an extension can subclass, find one that is functionally similar your own (I used update_file_info as my guide), then insert your own method (file_open in this case) at all the relevant points as well as create the appropriate function bodies and set argument/return variables appropriately.

So overall NautilusPython wasn't too bad really. The real time sink was Nautilus itself.

Patching Nautilus

First of all, I almost immediately ditched GNOME Builder after my last blog post. It's not that it was a bad IDE, far from it actually, I quite liked the interface and the various tools it contained. The ability to download a project from git, package and run it inside a Flatpak, and then send patches upstream all from within the editor was quite appealing! The first major showstopper however, was that the version of Nautilus that Qubes makes use of (v3.22.3), doesn't support Flatpak. While building and running from GNOME Builder without Flatpak seemed possible, no matter what I tried I could not get extensions to function properly with the GNOME Builder version.

After a lot of fiddling about, I settled on what I probably should have been using since the very beginning: qubes-builder!

Not only does it build the correct version (and supports extensions), but I knew that whatever I did with this build system would be exactly what users would be using when they built Qubes' templates or got them from the build servers. So, unfortunately no Flatpak'd builds, but I do have pre-packaged RPMs with debug-symbols! That's probably even better, right?

I'll post them in a separate blog post with instructions on how to set everything up soon.

qubes-builder worked very well and was quite easy to drill into and mess about with, as it is mostly just a collection of various scripts. I eventually built a script that ran in dom0 that would build Nautilus in my development VM, transfer it to my testing VM, delete the old version and install the new version.

The only downside to this process was that building Nautilus took quite a while each time (2-3m) which made quick changes quite an agonizing process. I believe this was due to both having to rebuild every file each time (as they were packaged in a .tar.xz before being built, meaning all files were considered to be modified) as well as a call to dnf to fetch any updates was made, every time. I found out the latter when working without internet and realizing my could would not build! Each of these things probably could have been worked out or changed if I had felt the need to, but I just stuck with the loading times and usually had internet, so it was not too terrible.

Was was terrible and time-consuming were a couple of very strange issues I had during the creation of the patch.

The first was the dreaded provider issue. You see, every time Nautilus calls a method that is meant to be subclassed by a Nautilus extension, you must provide a provider. This can be an INFO provider, MENU provider, and so on. Each method only accepts one of each type, typically corresponding to the category of method you were calling. For the file_open method I decided to stick it under the INFO provider category, as that had to do with information about files, which I felt 'when they were opened' best fell into.

Well, these providers don't just pop out of nowhere. You have to create them! You can do that by calling the intuitively-named nautilus_module_get_extensions_for_type method, providing it the NAUTILUS_TYPE_INFO_PROVIDER enum. It is then supposed to return a provider with a type that matches the one you gave it, but no matter what I did it would always just return a NULL pointer! I spent days on this problem, trying loads of different things. Every other call of nautilus_module_get_extensions_for_type had no special setup or initialization, leading me confused why my own calling of it brought such different results.

After picking through many lines of code, talking with GNOME people over IRC who were currently GUADEC'ing, and trying to understand the whole GNOME development ecosystem, I eventually traced it all back to NautilusPython not finding an extension that made use of a subclassed method from the INFO method type. But wait, we do have an extension that does that with qvm_trust, so why doesn't it work??

Well I hit my head after this one folks. The version of NautilusPython that Nautilus was loading in my test VM was the original, from-repo version, not our modified version. D'oh!

"Well that's just a simple fix, I just have to build it and import it into Nautilus in the testing VM!", I assumed, before realizing that while my modified NautilusPython built, Nautilus would not accept it. Nautilus would just hang with little to no output.

After a few emails, Marek pointed me in the correct direction with the nautilus-3.0 branch of NautilusPython, which, while 6 years since the last commit, worked with the latest version. Alright no problem. Rebase my commits on that branch, rebuild, transfer the newly-built shared-library over and bam, Nautilus runs again! And we have a provider that's not NULL, hooray!

And then, ladies and gentlemen, problem #2 reared its ugly head.

Here's the code to get a list (GList*) of providers to use with a Nautilus extension:

providers = nautilus_module_get_extensions_for_type (NAUTILUS_TYPE_INFO_PROVIDER);

While we were getting a non-null provider value, the value we received was different from the one returned by the method we called. "Wat?" you say, and I too said the same. For some reason we were returning a 64-bit GList pointer from nautilus_module_get_extensions_for_type, but storing a 32-bit GList pointer in providers.

I tried all sorts of weird tricks to get this fixed. Nobody I talked to seemed to have a solution, and after wrestling with it for a few days I begrudingly settled on this very elegant workaround:

// TODO: Providers ends up as 0x55f65ac0
// when method is returning 0x555555f65ac0, very odd...
providers = (GList*) (nautilus_module_get_extensions_for_type (NAUTILUS_TYPE_INFO_PROVIDER) + 0x555500000000);

Yes I know this is terrible. It did work though, and I needed to stop hitting this strange bug and get on with patching, so that's where it currently sits. I'm still actively looking for a proper solution. If anyone knows why a 64bit pointer might get truncated to a 32bit pointer after a function call in C please do let me know! You can find the full code here and the type method here.

After this was out of the way, and a handful of more methods were implemented, we finally had a working solution between the qvm_trust extension and our patched Nautilus! It was beautiful, really.

Conclusion and going forward

My email to the Nautilus mailing list is still awaiting moderation. GUADEC has now concluded, and with no update on the situation it seems I need to do some more active poking of people to get moderation approval. We'll see where that lands, the code still needs to be cleaned up a bit before sending it upstream anyways, so I'm not too worried.

At this point in the project, we essentially have the following tasks left:

  • Patch KDE Dolphin to provide the same functionality as Nautilus now does
  • Finish up some small tasks with the daemon like logging and some more interfacing with the cli tool
  • Code cleanup, linting, tests
  • Integrate into main QubesOS repos

I'm going to prioritize the latter 3 bullet points at this point to ensure we get a really solid core implementation down of the project. I don't want to spend another couple weeks patching Dolphin only to run out of time for a general cleanup and other polish. I will try to pour through the Dolphin code and note down everything that one would need to do with Dolphin to make it work with file trust attributes, so that in the future either myself or someone else can give it a shot.

It may end up that Dolphin already has most of the functionality and the patch will be very small, we'll see!

Anyways, for now I'm going to start on finishing the remaining small tasks, and will post any questions regarding packaging and/or cleanup to the qubes-devel mailing list.

As always, you can find the code here.

And as of now, you can also find the patched code for Nautilus and NautilusPython on Github as well. Make sure that when you're inspecting the NautilusPython code, you're looking at the nautilus-3.0 branch. That's where all the important commits are!

That's all for now, thanks for reading!

Tags: gsoc-2017, progress-report

GSoC Weekly Progress Report #7

July 24, 2017 — Andrew Morgan

Hello and welcome to another weekly report! Well, this one may have been bi-weekly, but at least we're set to be back on schedule now.

Since the last report a few exciting things have been completed. This includes packaging of qvm-file-trust as an installable python package, a functioning qubes-trust-daemon, and some strong headway on patching Nautilus!

So without further ado, let's get into it:

Pretty Python Packaging

qvm-file-trust has been moved from the project's root directory into its own directory and its own python package. This has the benefit of cleaning up the root folder a bit, but more importantly allowing for proper importing into other python scripts.

Tests have been refactored to no longer use the hacky method of importing qvm-file-trust and instead import the new qubesfiletrust package directly.

Finally, installing the python package and all the other scripts can now be achieved through the use of a Makefile. Simply sudo make install and you're done!

Daemon Challenges

Coming from python back to C++ took a moment of re-adjustment as I had started to get used to the conveniences that python provides. There is however no doubt that the tradeoff of what C and C++ offer in terms of performance are worth the drop in comfort.

What was interesting was that even after deciding to switch qubes-trust-daemon from C to C++, a majority of the code that went into programming the daemon was C compatible. Low-level libraries like inotify are written for C, and one must interact with them with C-like components such as null-terminated character arrays as opposed to C++'s std strings. In this sense, C++'s ability to handle nearly all C code is thus incredibly useful.

After a few days I became comfortable working within these two languages. C offers necessary power and speed, while C++ offers many convenient classes and functions that I was able to make use of.

So after all that, here's the story on what is completed with qubes-trust-daemon and what still needs to be done:

What's done:

  • Grab a list of untrusted folders from the global and local list
  • Place an inotify directory watch on those folders and subfolders and listen to various file change events
  • Act on events such as items being created or moved inside of untrusted folders. Upon catching one of these events, we either watch the new folder for further events, or if it is a file, mark that file as untrusted
  • Handle files being deleted or moved out of watched directories

What's still to do:

  • Proper error reporting
  • Testing

Currently, the list of untrusted folders are grabbed and sorted by the daemon directly and then passed to other functions. I am considering adding the ability to export the list that qvm-file-trust creates as a line-by-line list to stdout and having the daemon capturing that instead.

The upside to this is that there is only one implementation of parsing and sorting the lists and it is written in a language where there are less likely to be parsing errors by the programmer. The downside will be that you must start up an instance of qvm-file-trust to perform this action, but as it will only be done once per initialization of the daemon, the cost is mostly negligible.

As it stands on my i7-4810MQ, only 6.7% of the VM's CPU (2 out of my 4 cores) is used as a couple tens of thousand files are being marked as untrusted on startup and on moving a large folder in. That CPU usage is well within reasonable levels, especially on such a large operation. That is a win in my book.

Nautilus

The patch for Nautilus is finally getting going! After some hiccups with Nautilus' build process, I've now switched to GNOME Builder for the compilation and run management of Nautilus. Builder provides a much cleaner experience when building and compiling, and after only hearing about it here and there, it was exciting to finally download and try it out.

After a couple hiccups with importing the Nautilus code and checking out the correct branch (gnome-3-22 in this case) with the help of some fine folks on #gnome-builder, Nautilus was building and launching as expected:

Nautilus built and running from Builder Nautilus built and running from Builder

The other advantage GNOME Builder holds is its ability to build all projects as flatpaks. This makes it much easier to share testing builds of Nautilus without asking people to install Nautilus' horrible list of dependencies, and instead just run a single command to start it up!

In addition to Nautilus, GNOME Builder is also helping me with patching nautilus-python, which are the python bindings that allow extensions written in python to interact with Nautilus' C API.

This folder houses the definitions of functions that can be referenced in nautilus-python extensions. Nautilus' maintainer, Carlos Soriano, also kindly pointed out where these methods connect to Nautilus itself, which turns out to be here.

The typical workflow in a nautilus-python extension is to define a function that corresponds to the data or action you desire, then return some data based on some logic you have in your extension. My plan is to create another function here that will be called when a file is determined to be opened. The file will then be opened depending on whether the extension returns a True or a False.

A new DisposableVM with the file in question can be launched from within the extension, so there is no need for any Qubes-specific code to go into this patch.

User UX

In other news, in addition to the Dolphin Folder Color that allows you to set the color of a folder from an extension, I have now come across a similar offering for Nautilus, titled simply Folder Color (code here).

With the ability to now mark untrusted folders with a color (likely red), and with Nautilus and Dolphin already adding an icon for untrusted files (due to their unreadable nature), I no longer see any reason to add additional icons on top of this. After a few rounds of testing, seeing many 'warning' and 'lock' icons on files got a bit repetitive anyways :)

Conclusion

Phew, that was quite a few things to talk about. At this point I'm going to be devoting the majority of my time to get Nautilus' patch functional. Once we have a complete GUI + cli tool + daemon working together we can more easily get a picture of how to test and tweak the overall flow of the tool.

I'll also be providing flatpak's of Nautilus for easier testing, which will most certainly be appreciated once they are available!

Anyways, that is all for now. As always, you can find the code here.

Have a wonderful day!

Tags: gsoc-2017, progress-report

GSoC Weekly Progress Report #6

July 11, 2017 — Andrew Morgan

Hello again! Few different things going on this week so let's just jump right in.

Tests

First off a basic set of tests have been written for the qvm-file-trust tool. So far these are all unit tests, and mainly test sending input to each method in qvm-file-trust, ensuring that certain methods are called with the correct parameters, and anything that is returned contains the correct data.

In the process of writing these tests I uncovered a few bugs and quirks here and there in the code, which is awesome because that's exactly what tests are designed to do. Currently I only test each method directly, however in the long run there will be tests that send varied input externally to the program. These will check the output as well as the trust status of any files that may be involved.

As was evident from my last post, I had a bit of trouble getting used to some 'pythonic' quirks, as well as the whole concept of mocking. Nevertheless, after getting a few fundamentals down on how to feed in fake inputs and data, writing the majority of the tests was mostly just time-consuming.

I also ran a few passes of pylint over qvm-file-trust to really make sure it was clean and following python's syntactical standards.

I still plan to add a few more tests for the cli tool to the suite, as well as those for the upcoming untrusted folder watching daemon.

Speaking of which...

Look out! It's a daemon!

I took a little time this week to start work on the C daemon that will watch all untrusted folders and mark any files created inside them as untrusted as well. For starters, I've switched to making it a C++ deamon instead. As far as I can tell there isn't any noticable performance difference between the two languages, and C++ comes with a few abstractions for various low-level things like file handling, meaning less room for errors and bugs. I'm also much more experienced in C++, so it seemed to the correct choice.

While following the Qubes Code Style Guidelines, I was able to implement parsing of the global and local rule lists (including functionality such as comment ignoring and override rules) as well as being able to call on qvm-file-trust and set any file as untrusted through it. I'm currently working on iterating through the rule list and setting up an inotify watch on each folder (and its subfolders) for any new files placed within them.

In a very scientific test of setting up a basic C++ program to watch /tmp and holding down tab after ls /tmp/, generating many simulataneous inotify events, the CPU usage of the program was 0%. Whether it will spike up when creating thousands of files within untrusted folders remains to be seen, but I'm trying to keep the code as efficient as possible.

One thing I wasn't entirely sure about was where to send logs. I'm not sure if I should just log to syslog (and thus journald), or /var/log/qubes/... (or both). It's simple enough to do either one.

Water-based File Managers

I've started with modifying some of Dolphin's source code. Unfortunately I've been having a little bit of trouble with their build tool, kdesrc-build. The amount of source code necessary to build Dolphin is apparently quite large. Every time I try to download the full dependencies of Dolphin it seems to stall halfway through.

This may not be the only way to work on Dolphin, but I haven't seen an alternative so far...

The other matter is that Whonix's current Dolphin version is 4.14.2, while an updated Arch system's Dolphin version is 17.04.2 (and source is 17.04.70).

According to this bug report, Whonix should be using KDE Frameworks 5 in v14 (next release), which should be included in Qubes soon following release. Due to this, I'm wondering whether I should base my patches on top of Dolphin's latest code or the legacy v4.14. If the latest, then it may be best to develop on Arch instead of Qubes, or maybe I could find a pre-release build of Whonix 14...

For Nautilus I've just been using the latest code and it's going well. Update on that in a bit!

In any case, I've outlined the required changes that need to be made to each file manager below:

Nautilus:

1. Expose file open events to extensions

2. Respect not opening file if extension requests not to (upstream may not like this part :)

Dolphin:

1. Work around the complete prevention of attempting to open files for which we have no read permissions for 
(instead check with our cli tool and decide what to do based on its output)

2. If the file is deemed an untrusted file, attempt to open it with our dvm-desktop handler

3. Failing that, display an error

That should about cover the functionality. Hopefully it'll be at least partially upstream-able. I'm open to suggestions if anyone has any :)

That about wraps it up for this week! As always the code is available here.

See you next time!

Tags: gsoc-2017, progress-report

GSoC Weekly Progress Report #5

July 04, 2017 — Andrew Morgan

Hello again! Quick report here due to only a few days since the last one.

This period has been mostly focused on writing tests for qvm-file-trust. I got a bit hung up on some of the python-specific stuff but am now plugging away at covering all the methods.

The main difficulty was that the majority of the methods deal with reading from and manipulating the content and attributes of files on the file-system. Ideally the tests shouldn't actually create files on the system to be read, but instead provide testing data to the open() calls of the program.

Functionality to substitute certain python methods with our own are provided by the unittest.mock library, and it's a incredibly useful tool once you wrap your head around it.

The big issues I had while writing tests were having our test data being correctly read by the program, and supporting returning different sets of data for different open() requests within the same method.

The first issue comprised of the unittest.mock.mock_open function not supporting iterators. The qvm-file-trust tool uses the following code to read from the global and local untrusted folder lists:

    with open(GLOBAL_FOLDER_LOC) as global_list:
        for line in global_list:
            ...
            untrusted_paths.add(line)

To dynamically inject the returned content with unittest.mock, we can write the following:

with unittest.mock.patch('qvm-file-trust.open', unittest.mock.mock_open(
    read_data='Return me!')):
    untrusted_folder_paths = qvm_file_trust.retrieve_untrusted_folders()
    ...

Any file then read in the retrieve_untrusted_folders() will now return 'Return me!', instead of the actual file contents.

While simple enough in this case, the contents of our global list file as returned by our test data was always empty, and the inner for loop would never run.

I mulled over this issue for a few hours, when Marek pointed out what was actually happening. It turns out that mock_open does not mock the __iter__ function on the object, and thus our attempts to read line-by-line by iterating over the file fail miserably. This is a recognized bug in python.

To alleviate this, it turns out that we can simply manually override the iteration function and tell it to return our desired lines:

mock_object.return_value.__iter__ = lambda self : iter(
    'home/user/Downloads', '')

We are now able to substitute file content dynamically in our test cases. There was still one problem though. Using the above code, we end up replacing all open() calls with our substituted content. But this method makes multiple calls, to multiple files, which for proper testing requires multiple different returned values.

We need to return different content based on the path we are asked to read.

Figuring this out took another long period of time, but eventually I was able to return multiple values at different calls using mock's side_effect attribute.

This approach was slightly limited in that we can only return data based on when open() was called, rather than what path open() was called with, but as our methods are rather static in this sense there was no trouble.

The resulting code looked like the following:

def test_000_retrieve_override(self, list_mock):
    """Create a mock global and local list and check resulting rules.

    Are global rules are properly overridden by '-' prepended local rules?
    """

    handlers = (unittest.mock.mock_open(
            read_data="/home/user/Downloads\n/home/user/QubesIncoming"
            ).return_value,
            unittest.mock.mock_open(
            read_data="/home/user/Downloads\n-/home/user/QubesIncoming"
            ).return_value)
    list_mock.side_effect = handlers

    untrusted_folder_paths = qvm_file_trust.retrieve_untrusted_folders()

self.assertEqual(untrusted_folder_paths, {'/home/user/Downloads'})

Here we subtitute the global lists rules as

/home/user/Downloads
/home/user/QubesIncoming

and the local lists as

/home/user/Downloads
-/home/user/QubesIncoming

The '-' in the second file should override and negate the rule in the first file. We then check that we end up with only the Downloads folder being left as untrusted, and if so mark the test as passed.

While this took quite a while to figure out, injecting return code from multiple files was necessary for nearly all the tests here, so with that out of the way writing the rest of them should be fairly straight-forward. With a good understanding of how unittest.mock() works, even testing chmod and xattrs should be a breeze.

That about wraps it up for this blog post. Happy America day to anyone celebrating that tomorrow. See you all in a week.

As always, the code is available here.

Tags: gsoc-2017, progress-report

GSoC Weekly Progress Report #4

June 30, 2017 — Andrew Morgan

Hello again! A week has gone by and I've gotten some feedback on the qvm-trust tool introduced in the last post. First off, it's been renamed to qvm-file-trust! It's more descriptive, as the primary application of the tool is manipulation of trust of files, more than anything else. Thanks to Joanna for the suggestion!

In addition to the name change, a number of other changes have been added, as well as a bit of a code refactoring from when I first wrote it. The main bits are:

  • A phrase can now be specified in /etc/qubes/always-open-in-dispvm.phrase, if a file's path contains that phrase, the file is considered untrusted.
  • Now pull and combine results from /etc/qubes/always-open-in-dispvm.list and ~/.config/qubes/always-open-in-dispvm.list to determine list of untrusted folders.
  • Ability to place a '-' in front of a path in the local untrusted folders list in order to override the global's untrusted definition.
  • Add scoped global variables to improve readability.
  • Return codes now reflect the POSIX standard.

The reasoning behind the phrase was the prevent total reliance on xattrs, which isn't supported on all filesystems, such as FAT32, which is still a popular filesystem. Note that the definition of an untrusted file is slightly altered by this. I believe Joanna put it best in the mailing list:

Ideally, I think the evaluation of trustworthiness should be like this:

1. All files are considered [_trusted_] unless we check all the rules below, and
none of them identifies the file as _untrusted_,

2. If the file is under a path which contains pre-defined untrusted string (such
as...  "untrusted", ALWAYS case _insensitive_ match), the file is untrusted, no
matter what xattrs might be saying. E.g. the file might be from some FAT
filesystem attached to the VM via qvm-block or qvm-usb and we still want to
catch it.

3. If the file has xattr saying it's untrusted.

(4. If none of the above, then consider it trusted.)

The separation of the list of untrusted folders paths allows one to define a global list for all AppVMs of a template in the root partition, while simultaneously keeping a local list for each AppVM in the home partition. Additionally, one can override a rule in the global list by placing the same path with a '-' character in front, preventing that path from being considered untrusted.

This is useful in the case of having an untrusted VM that you may be completely fine opening potentially dangerous files in from time to time, as you have nothing of value in it. You can use this with the same template as your personal VM, you need only disable the default rules from the local list in the untrusted VM, and you're set. Props to Marek for this idea!

The error codes have switched around a few times in the last few days. Happy to report they now follow the POSIX standard, and will hopefully be stable now. Thanks for Jean-Philippe for pointing this out! Updated error codes can be found in the documentation. Speaking of which...

Man page

Yes, qvm-file-trust now has a man page!

qvm-file-trust man page

Also viewable in HTML form here.

That's the first man page I've ever written. Shout out to this great resource that made the whole process really painless.

File Managers

We're not finished with them yet. Remember last week where it was determined that no patches of them would be needed? Yeah... turns out that's probably not the case. The new definition of untrusted files above adds some complexity that the current workaround in File Managers does not meet.

Additionally, this definition may change in the future, and instead of trying to find new workarounds for how to emulate that in the file manager, I figure it'd just be simpler for Dolphin and Nautilus to both use qvm-file-trust directly. That way any update to the tool will automatically work in all supported file managers.

By the way, anyone else notice both names 'Dolphin' and 'Nautilus' are water related? Coincidence...?

Testing 1, 2, 3...

I feel that I've hit a pretty solid point in qvm-file-trust in where the purpose of the program and every method is clear, and thus it is a good time to start writing some automated tests to ensure new code doesn't break anything.

As always Marek had some great pointers on how to get started with this.

I'll initially be writing unit tests for qvm-file-trust and some of the other utilities in the repo to ensure everything is working correctly individually. Then, after the file managers have been patched to integrate with qvm-file-trust, and if I have time, I'll do some integration tests that'll run through the GUI and ensure everything is working from both ends.

Conclusion

As always, the code is available here.

That about wraps it up for this week, til next time!

Tags: gsoc-2017, progress-report

GSoC Weekly Progress Report #3

June 22, 2017 — Andrew Morgan

Good news everyone! The code base has been completely refactored from multiple scripts with duplicated code into one lean python cli tool. All the other scripts simply make use of this now.

Additionally, 90% of the bash in the project has now been replaced with python, which should allow more portability and more maintainable code.

New CLI Tool

This new utility is called qvm-trust, and can be used in the following manner:

$ qvm-trust -h
usage: qvm-trust [-h] [-c] [-t] [-u] [-q] path [path ...]

Set or check file/folder trust levels.

positional arguments:
  path             a folder or file path

optional arguments:
  -h, --help       show this help message and exit
  -c, --check      Check whether a file or folder is trusted
  -t, --trusted    Set files or folders as trusted
  -u, --untrusted  Set files or folders as untrusted
  -q, --quiet      Do not print to stdout

Some example usage includes: setting a file as untrusted:

$ qvm-trust --untrusted ./leaked-documents.pdf

Checking the trusted state of a file:

$ qvm-trust --check ./leaked-documents.pdf
File is untrusted

Setting multiple files and folders as untrusted:

$ qvm-trust --untrusted ~/Downloads ./my-sketchy-file.docx ~/QubesIncoming

Checking trusted state through the return code (useful for scripting):

$ qvm-trust --check ~/Downloads --quiet
$ echo $?
1

A reference for the return codes is as follows. Additionally I plan to write up a man page with all usage information soon:

Error codes:
    Unable to read extended attributes: -2
    Unable to read an input file: -1
    No errors: 0
    File/Folder is untrusted: 1
    File/Folder is trusted: 2

Nautilus and Dolphin integration

The Nautilus extension and Dolphin service files now make use of this script. This cut down the file size of the Nautilus extension considerably, as well as removed the need for the large bash script that was dedicated specifically to checking trust in Nautilus, as well as the two extensions for Nautilus. Now there is simply one extension that handles both files and folders.

xattr is also no longer a dependency as we now do all the extended file attribute checking and setting through python.

Additionally the ability to mark file-types as untrusted has been removed, as there really wasn't a considerable use case for it and it made implementation more complex than necessary. In the future it could be added, but at the moment there's no real need for it.

However one benefit to its removal is no more complicated check-box UI to navigate through! It's all simply there in the right-click menu:

Nautilus context menu with right click for folders Nautilus context menu with right click for files

There is now a single context menu item for both files and folders.

On the Dolphin side, the check-boxes have similarly been deprecated, however I was unable to find a way to add the sleek icon next to the service menu item as done in Nautilus. Unfortunately there doesn't seem to be a way to change a service's icon based on a script's output, so for now I've settled on a sub-menu which the user can choose whether to mark a file/folder as trusted or untrusted:

Dolphin with right-click sub-menu

Separate menus since Dolphin can't switch icons on the fly.

Also if you didn't already notice, 'Open in DisposableVM' now shows up with a little Qubes logo in the right-click menu of untrusted files!

Qubes logo now shows up for untrusted files

Qubes logo now shows up for untrusted files.

This was achieved with a .desktop file that is called upon when a certain MIME-type is clicked. This was originally the plan for recognizing untrusted files in the file-manager, however it was not considered a viable option as an untrusted file could have any file-type (i.e PDF, TXT, DOCX, etc). It wouldn't make sense to register all of these types to be opened in DisposableVMs, so it was dropped.

That was until we started 'locking' untrusted files to disallow reading by local applications. This had the unintended but incredibly useful side-effect of having file managers set the MIME-type to application/octet-stream. At that point we simply had to register that MIME-type for our custom .desktop file and we were good!

Small bugs/issues

There are still some small roadblocks that still need to be cleared up:

  1. While the functionality in Nautilus is all there, Nautilus doesn't always seem to re-run the file-checking code in our extension. This can cause the wrong icon to appear when right-clicking a file, which is certainly confusing. The correct icon is shown if you click some empty space, then right-click the file again. I feel this is more a Nautilus bug than an issue with our file-checking code. I'll hit up the devs on #nautilus soon and see if I can get some answers.

  2. KDE's Dolphin handles MIME-type checking a bit differently than Nautilus does. For instance, instead of solely checking for MIME-types by reading the contents of the file, it gives the greatest priority to the file extension.

    Dolphin thinks a PDF is a text file, due to its extension

    This is a PDF that has been renamed with the '.txt' extension. Dolphin thinks it is a text file.

    KWrite being suggested instead of qvm-trust

    KWrite is suggested to edit it.

  3. Dolphin doesn't mark untrusted files as a specific file-type, instead it simply says 'unknown':

    Dolphin is confused What could it be?!

    Clicking on 'Create New File Type' does allow us to make new ones called application/x-kdeuser, however even after doing so the file-type is still unknown.

    Actually while writing this I just thought... we could combine 2 and 3 together and just append '.untrusted' to untrusted file types, then register the '.untrusted' file extension with our desktop handler. Hm... will try that right after I post this :)

  4. Dolphin does not allow unreadable files to be opened. It cancels the application launch and presents an 'Access Denied' message. Not entirely sure if there is a way around this outside of a patch. It would be great to find a workaround.

    Dolphin won't let us open unreadable files This file is unreadable, dummy!

    Update: this may not be an issue with the '.untrusted' workaround above. Apparently if the file-type is recognizable it will send it to the application that's registered

  5. The location of the untrusted-folders file. This file keeps track of the folder paths that are untrusted, and is used by the daemon. Even after reading the Qubes Contribution Guidelines I'm still not sure where to put this. /usr/share/qubes seems to be the apt location but there's not much in there already. Marek?

Going Forward

We're currently heading into week 2 of 4 of what was supposed to be patching the file managers, however now that it seems that's no longer necessary, we're pretty much ahead of schedule! After Dolphin is fixed up, I'll create a daemon to monitor untrusted folders and their contents. If a new file is placed or created inside them, they'll instantly be marked as untrusted.

After that is unit tests and documentation cleanup. Pretty standard stuff.

As always the code is available here.

Anyways, that's about it for this week. Til next time!

Tags: gsoc-2017, progress-report

GSoC Weekly Progress Report #2

June 14, 2017 — Andrew Morgan

The work this week consisted of finishing off the context menus within Nautilus and Dolphin. I'm happy to report that they've both been finished off and accompanied by some icons from GNOME's Adwaita icon set.

They actually work now too :)

Some screenshots below:

Icons appear in menu items now in Nautilus Icons appear in menu items now in Nautilus

We also have a checkmark icon to indicate to the user that a folder is marked as untrusted We also have a checkmark icon to indicate to the user that a folder is marked as untrusted

The popup menu now includes the name of the file that is being marked as well as the file type The popup menu now includes the name of the file that is being marked as well as the file type

Handy icons now show up on untrusted files! Handy icons now show up on untrusted files!

Extended File Attribute Troubles

As discovered earlier in the week, applied Extended File Attributes can get lost after some programs (i.e vim) edit them. This is due to the editor's nature of updating the file by first destroying it, then recreating it later from their temporarily modified buffer. This method is efficient, but unfortunately any file attributes that may be attached to the file that the editor doesn't know about will be lost after the original file is deleted.

You may think that this is a total show-stopper for Extended File Attributes all together, but they actually still work in our use case, as the goal is to prevent local modification of marked files, while sending them to a separate VM for editing.

Because of this, the only program we have to make aware of our special Extended File Attribute is the program that handles the transfer between the two VMs. In our case, this program is qvm-open-in-(d)vm. By simply reading the Extended File Attributes upon sending the file, and reapplying them once it gets the file back, we retain our mark, regardless of what happens to the file in the destination VM.

Denying Local Read Permissions on Untrusted Files

To prevent this mark otherwise being accidentally destroyed on the originating VM, we can simply deny all users permission to read or write from it (through a chmod 0). Props to my mentor Marek for the suggestion.

This has the one hiccup of which we can no longer read a file's Extended File Attributes, however our code can simply 'unlock' the file before processing it by chmod'ing the file back to 0644 before processing, and 'locking' it again afterwards.

Conclusion

Now that the GUI is all finished, it's time to work on making the File Managers (Nautilus and Dolphin) aware of untrusted files. While it's easy enough to check for untrusted files on a right-click basis, we also need to check their status on a single or double left-click (i.e when a file is opened).

Originally I planned to patch the File Managers to allow for running code on a left-click, however after creating the Nautilus extension, it seems to already do this by default. Coupled with the fact that files are no longer locally editable and thus cannot be opened automatically, we may not actually need to patch Nautilus at all!

Dolphin may still require a patch, but I'll be looking for ways to possibly get away with not needing to while working on the Nautilus version first.

As always, the code is available here.

Any and all feedback is appreciated, see you in a week!

Tags: gsoc-2017, progress-report

GSoC 2017 Weekly Progress Report #1

June 06, 2017 — Andrew Morgan

This is the first of many progress reports I will be writing up for my time as a Qubes GSoC student. My project deals with adding a configurable trust-level to all files and folders that specifies whether files should be automatically opened in a DisposableVM when chosen through one of our supported file managers.

The current goal for the first two weeks is to implement context menus for both Dolphin and Nautilus that appear when right-clicking a file or a folder. Clicking on them will result in a simple GUI to allow the user to choose whether that file, folder or file type should always be opened in a disposableVM aka marked as 'untrusted'.

My current progress has resulted in a context menu item in both Nautilus and Dolphin context menus:

Nautilus Context Menu

Dolphin Context Menu

In addition, I also have a simple GUI written with Zenity that is launched when clicking on the context menu item:

Zenity File Trust Dialog GUI

There is also a specific menu item for folders which will only appear when right-clicking a folder:

Nautilus Folder Context Menu

Clicking this will result in a folder being marked as untrusted, meaning any item opened within it will be opened in a disposableVM. You may want to do this to your Downloads folder for instance!

Currently none of these items actually do anything once you've finished clicking through them, but that's the goal for next week.

Source code can be found here.

See you then!

Tags: gsoc-2017, progress-report