Provide MRtrix3 as a Snap package for Linux

tpeeters 2017-01-31 14:49:24 UTC #1

I was reading the installation instructions of MRtrix3 for Linux on http://mrtrix.readthedocs.io/en/latest/installation/linux_install.html and because I have been working on Snap packaging recently (see https://insights.ubuntu.com/2016/06/14/universal-snap-packages-launch-on-multiple-linux-distros/ if you are not familiar with Snap packaging), I realized that for the end-users there is probably an easier way to get the software if the MRtrix3 project were to provide Snap packages. I work for Canonical, and some of my time I spend working on Snap, but MRtrix I am using for a personal project, but because of my experience with Snap I was thinking of whether creating Snap packages of MRtrix would be useful.

If you are using Linux, you can see on https://tutorials.ubuntu.com/tutorial/basic-snap-usage a tutorial on how Snap packages would be used. Snap support comes with the latest versions of Ubuntu, but on other Linux distributions you probably have to install it, see http://snapcraft.io/docs/core/install . With an MRtrix snap package, users could simply type something like 'snap install mrtrix' or 'snap install mrtrix --beta' (depending on the version the users want), and the MRtrix snap package, which includes all of the dependencies would be installed.

So is there interest for this from the users?
And from the developers? Basically what would be needed is to create a relatively simple snapcraft.yaml file (see www.snapcraft.io for more information) that tells the snapcraft tool how to build a snap package, and to upload the snap packages to the snap store. If users and developers are interested in snap packages, I can help out with the creation of the Snap packages.

So, let me know if you are interested or (and why) not.

jdtournier 2017-01-31 23:30:16 UTC #2

Sounds interesting! I hadn't heard about this till now, so I've no idea how it works. It sounds similar to Docker (?), and this is certainly something that we've discussed in the past. There is nothing stopping anyone from doing this, and I'd be all for it, but there's a few reasons why it might be better to hold off for just a little longer (along with a few other concerns):

we're still really in beta, there's a lot of changes being committed relatively frequently, Hopefully we'll release our first official stable release soon, at which point we plan on providing more official pre-built packages for the major OSs. Sounds like snap would also be a good distribution channel.
we're about to merge a pretty big branch with a lot of changes in it, some of which will affect operation and how the software is organised. So again, it would definitely be worth waiting until that happens, or working from the tag_0.3.16 branch while setting this up if you're impatient...
Another more minor concern I have is about providing optimised, high performance binaries. At the moment, users are required to compile locally, and we get the compiler to generate code suitable for running on the user's specific CPU (via the -march=native flag). This means that the code will use vectorised instructions and more advanced features if available, which is particularly beneficial for performance with Eigen (which is designed to make full use of those instructions when available). If we start shipping pre-compiled packages, I'm assuming we'll need to decide on a minimum CPU spec that will run with decent performance on most systems out there. Is there a mechanism within snap to help with this? Is it possible to have the framework compile various versions of the executables suitable for running on different CPU generations, and snap would then install the most appropriate automatically? Now that would be a great feature...

So in short, I'm pretty sure we'd be interested, but maybe we can discuss this when we enter a more stable release cycle? No doubt the other devs will have an opinion on this too...?

tpeeters 2017-02-01 11:16:01 UTC #3

Thanks for your quick and very informational reply :).

Yes, there are certainly similarities with Docker; Snap and Docker both provide packaging for software that run in an isolated (containerized) environment. However, I think Snap is more of an application packaging format while a Docker package will provide you with a service that has an IP-address. But my experiences with Docker are too limited to make a real comparison.

Being in beta does not have to be an objection for trying Snap at all. In the snap Store there are different channels (stable, beta and edge are frequently used), and beta packages are only installed when providing the --beta parameter, and don't show up in a store search. This is one of my favorite features of snap: you can provide multiple versions of the packages, including potentially unstable test versions for adventurous users to try out.

About the optimized binaries, I asked about this in the #snappy channel on Freenode IRC, and I was referred to this compiler feature (thanks zyga): https://gcc.gnu.org/wiki/FunctionMultiVersioning where a single binary automatically uses the best version of a function depending on the CPU. However the '-march=native' flag apparently is so optimized for the specific CPU that it is probably not useful for pre-compiled binaries.

I don't know how much performance you gain with the optimizations (did you do measurements?), but if it is a small gain, I would personally prefer the convenience of a binary package if, for example, I can install MRtrix on multiple machines/VMs/containers within a few minutes instead of having to compile it myself (that of course depends a lot on the specific use case. I don't think binary packages will replace the distribution from source, but it gives extra convenience for users who prefer that).

I am also very interested in what other devs think of this. As a new user of MRtrix being able to install it with 'snap install mrtrix' would be great (I don't know what the equivalent for Docker would be), but of course it is very interesting to see what experienced users and developers think about packaging MRtrix.

jdtournier 2017-02-02 16:53:01 UTC #4

Good to hear. That sounds like a useful feature. Is there a way to integrate this with GitHub? It would be great if we could set it up to automatically build specific branches as soon as commits are added to them...

Good question, no I've never really gone and measured these explicitly. But I have now, and it's instructive... For a standard run of dwi2fod csd, or of dwidenoise, there was essentially no difference in runtime between a -march=native build (the default), an explicit -march=x86-64 build, or a build with no -march flag. Not what I was expecting...

That said, this will change when we merge the imminent tag_0.3.16 branch, since that will allow Eigen 3.3 to make use of the newer AVX instructions, and I did benchmark that as part of trying to get MRtrix3 to work with Eigen 3.3. The speedup was of the order of maybe 20% - so still not enough to warrant the inconvenience of having to compile natively for regular users...

But in any case, the plan was always to provide pre-compiled builds when we hit proper release, so this would be one way to distribute on Linux that might solve a lot of problems (I wasn't looking forward to figuring out how to build different packages for all the different versions of the various Linux distros out there). So I'd be quite keen to get this going, especially if you're willing to help... (?)

tpeeters 2017-02-07 14:42:19 UTC #5

Yes I can have a look to see how much work it would be to create the snap packages. This week I probably won't have time, so maybe afterwards. I don't want to make a promise now because I don't know yet when I will have time. But when I have results, I will report it here.

tpeeters 2017-04-02 13:09:24 UTC #6

As you may have noticed I did not report any results so far. I am currently switching jobs so I did not have much time, but luckily @elopio picked this up and provided a PR here https://github.com/MRtrix3/mrtrix3/pull/935 Thanks a lot Leo!