This page describes a walk-through testing exercise that was performed by Allan Day on 13 December 2017.

The goal of the exercise was to test the Flatpak user experience on several popular Linux distributions. The test assumed that the user was primarily interested in using Flatpak to install and use applications, rather than wanting to learn technical details. It assumed little to no prior knowledge of Flatpak and how it works.

Test was conducted with a new installation of Fedora 27, running in GNOME Boxes.

• Launched Firefox and searched for "flatpak". Top search results were:
  • flatpak.org
  • flatpak.org/apps.html
  • flatpak.org/getting.html
  • github.com/flatpak/flatpak
  • flathub.org
• Clicked on the top result and the flatpak.org homepage is opened.
  • I see "get Flatpak" and "applications" in the header, but which do I want? I don't know whether I have flatpak already or not. There's also "hello world", which could be confusing if I didn't know any better.
  • The main line on the web page says "the future of application distribution". As a potential user this isn't particularly interesting and makes me wonder if Flatpak is intended for me. I'm basically looking for something that says "here are the cool apps you can install", but I don't see it.
• I click on the applications link. This takes me to a black and white page with lots of text on it. It's explaining technical stuff. This is immediately off-putting. I want to skip the wall of text and get to the apps, so I scroll down, but there's not many apps there. One of the subheadings says "stable applications outside flathub" - what does that mean?
  • A lot of users would give up at this point, but I persevere and go back to the top to read the text. The second paragraph says that "most applications are available in a repository known as flathub". It's not a particularly inviting statement, but sounds like what I want. I click the link, hoping that it will take me to a new site with lots of apps. Instead it takes me to the bottom of a page of text, to a short section called "Using Flathub on the Command Line". It's basically just a single command for adding the remote and has no supporting information. There's also a section on reporting security or legal issues. If I didn't know what Flathub was, I'd be baffled at this point.
• Again, a lot of users would give up at this point, but I'm persevering. I click on "applications" at the top, since that's what I'm interested in. At last! I see a grid of things that look like apps. Great!
  • It's quite a lot of apps. They're just listed alphabetically and they're not easy to browse. There's no way to browse by category, and there's no recommendations. No alternative to using this web UI has been suggested.
• I click on an app at random - HomeBank. Firefox pops up a dialog that has "Software Install" as the default option - so far so good. I click OK. Software pops up with details about HomeBank and an install button - good. Then a message pops up "Sorry, something went wrong" - disconcerting and there's no information about what actually went wrong. It looks broken.
• I dismiss the "something went wrong" banner and click install. A password dialog pops up, which I enter my password into and then I get a progress bar for installation. It feels incredibly slow - I'm sitting watching it stuck at 3% for a substantial period of time. I'm asking myself whether it's broken, then it thankfully jumps to 5%, then 7%, then 9%, 14%... it takes forever. A lot of people would give up at this point. I check my connection - it's not blazing fast, but it feels like the HomeBank install is achingly slow in comparison.
• I go away and when I come back, the app seems to have been installed, although there's another one of those "Sorry, something went wrong" messages, with no detail. I dismiss it and click "Launch". There's quite a long delay - I start thinking that it's broken. But then - ta da! - the app launches.
• I try another app - Teeworlds. Installation feels a bit faster this time - not fast, but I can at least see the progress moving up. It installs and there's no "sorry, something went wrong" (yay). I launch the app, but it doesn't look very good. My cursor disappears and I can't find a way to quit. Eventually I figure out how to get to the overview and close the window from there. Not getting a good impression of what Flatpak has to offer here.
• I try another app - Nautilus. Software opens it fine, there's a bit of a delay but then it starts to install. It installs and I click Launch. Again there's a long delay before the app window appears, but it does. It doesn't look quite like the Nautilus I know though - a lot of the folders are missing from the sidebar on the left. I try to trash a file but it tells me that that won't work and I can only delete it. Is that because it's a Flatpak?

Website issues:

• The flatpak.org home page doesn't offer a clear path for new users
• flatpak.org advertises "get flatpak", but users won't know whether they need this or not
• The apps page on flatpak.org is a dead end with too much text
• The link to Flathub from flatpak.org isn't obvious and the location it takes you to is surprising and disorientating
• Following the path from flatpak.org, there's no simple explanation of what Flathub is or why it's good.
• The grid of applications on Flathub isn't easy to use - there's no way to browse by category, and there aren't recommendations. It isn't clear whether Flathub provides access to apps I couldn't get elsewhere

Other issues:

• Software repeatedly shows banners that say "Sorry, something went wrong". This is disconcerting and confusing.
• Initial installation of the first application feels incredibly slow, to the point of wanting to give up
• Launching the app is very slow
• Some of the apps on Flathub are poor and/or buggy

Test performed with a new install of Ubuntu 17.10, running in GNOME Boxes. Software updates were run before the test was performed.

• I launch Firefox, search for "flatpak", which gives the same results as in test 1
• I follow the same navigation path as in test one - flatpak.org > apps > flathub.
• I click on an app - Audacity. Firefox shows a dialog that has "open with software install" as the default.
  • Software launches but shows the following error: "Don't know how to handle 'file:https://tmp/mozilla_aday0/org.audacityteam.Audacity.flatpakref'
  • I go back to the Flathub apps page. There's no mention of why this might not work.
• I persevere and try the command line options on the applications page. These say "These command line instructions require Flatpak 0.6.13". I don't know whether I have it installed, or which version I do have, so this is confusing.
  • I browse down to the command line instructions for Audacity and copy and paste the command into the terminal. In response, the terminal says "The program 'flatpak' is currently not installed. You can install it by typing sudo apt install flatpak".
  • I follow the instruction to install flatpak and re-run the installation command I copied from the Flathub web page. The response is: . "This application depends on runtimes from: https://flathub.org/repo/ . Configure this new remote 'flathub' [y/n]"
  • To an uninformed user this would be really confusing and potentially scary. Is flathub.org/repo/ something I want? Is it trustworthy? What's a runtime?
  • Anyway, I write y and put in my password when the dialog pops up.
  • Then the terminal asks me another question: . "Required runtime for org.audacityteam.Audacity/x86_64/stable (org.gnome.Platform/x86_64/3.26) is not installed, searching... . Found in remote flathub, do you want to install it? [y/n]:"
  • All of this appeared at once - there was no distinct searching phase, so it took some reading to understand. There's also lots of technical stuff in here and it's not very human-friendly. But I enter y and it downloads something (a runtime, whatever that is), which is probably a good thing?
  • The progress that is shown in the terminal is pretty noisy and technical. When it exits, it's hard to figure out that audacity has been successfully installed.
• I take a leap of faith and go to try and run the flatpak that I've installed. However, it's not listed with my other applications. I search for audacity from the activities overview and it comes up as a result, when I click on it, it's shown in Software but isn't shown as being installed.
• At this point, someone might just install the version from Software. I go back to the the web page where I got the installation instruction, and there's no information about how to run the app. I try the command line tutorial link, but it returns a 404.

In addition to the issues encountered in test 1, test 2 also identified the following:

• If you try to install a flatpakref on Ubuntu just using the GUI, it fails. The error message is technical and unhelpful, and the website doesn't warn why it would happen.
• The command line instructions on Flathub state that they require flatpak 0.6.13 or newer, but most people won't know whether flatpak is installed, or what version they have.
• CLI issues:
  • It asks a lot of questions, is noisy, and uses a lot of technical terminology which users won't necessarily understand.
  • It asks whether you want to add the flathub repo, but it's not clear what this is or what the consequences of saying yes/no are. Also, it asks at the wrong time, before the installation.
  • It isn't clear when an app has been successfully installed
  • Reverse DNS IDs contribute to the feeling of unfriendliness
• I was unable to find how to launch the app I'd installed (this was because you have to restart in order for flatpak apps to show up in the correct places, and I hadn't encountered a guide that told me to do this)
• The link to the command line tutorial on Flathub is dead