Off-line visualization

Development
1

Coming from a visualization background, MRtrix tool mrview caught my eye as example of great visualization. So much so that I got people interested in renderings (screenshots really) in a more clinical setting. This is just fine, except that I’d like to automate our processing from start to finish on a cluster. However, mrview uses OpenGL for rendering and is unsuited for “headless” generation of images (see this discussion).

I’ve worked with ParaView (http://www.paraview.org/) extensively. Recently, OSPRay (http://www.ospray.org/) was added to ParaView for high-performance, non-OpenGL rendering. The results are nearly indistinguishable from the VTK renderings. Now, there are some caveats! VTK uses OpenGL 2, so no shaders. OSPRay has a “shader language” in the form of ray-tracing kernels (http://embree.github.io/).

So… the idea would be to make a counterpart tool to mrview that uses OSPRay (or something similar) for rendering. OSPRay implements everything needed to do high quality renderings, can render to a framebuffer for side-by-side display with OpenGL, etc.

There are some “gotchas” with OSPRay, for one, it’s Intel only, and looks like a very difficult beast to compile! However, it comes pre-compiled for the mainstream platforms (Windows, Mac, Linux) as shared libraries, to take away compiler miseries. Development looks very low-level so will take some patience to get it right.

This post is to float the idea past the community, see if there was support for OSPRay or other libraries, and go from there.

My 2¢,
-dan

2

Had a quick look into how much work it might take to enable offscreen rendering, but as it turns out, I think it can already be done :grin:. Skip to the end if you want to find out how.

First off though, just a couple of comments on your previous post:

Well, technically there’s no reason why OpenGL can’t be used for off-screen (headless) rendering: it’s perfectly fine to draw to a framebuffer object, and never actually update the screen. And this will use the graphics hardware if available. And if not, we can always default back to software rendering: mesa provides a high performance driver for that, which would be available on basically any modern GNU/Linux system.

The reason why MRView can’t do it currently is because all of the rendering is tied very closely to the user interface: many of the parameters of the render are taken directly from the various widgets shown to the user. So we’d need to separate the rendering itself from the UI to allow off-screen rendering. As things stand, MRView needs to create an actual window to display all the widgets, so we’d need to bypass all of that, and allow the rendering code to work in an environment where none of the widgets exists, and then find a way to replicate the rendering pipeline so that it renders to an off-screen framebuffer object in headless mode. That strikes me as a lot of work… but not necessarily all that difficult - certainly less work that essentially re-writing everything from scratch, which switching to OSPRay would undoubtedly require.

However: it looks like we won’t have to go this far, because instead we can…

Use Qt5’s offscreen rendering plugin

I’ve just been looking into was whether Qt5 has any options to run in headless mode directly. And lo and behold, it turns out that it does: Qt5 uses platform plugins to manage the various rendering backends that are supported on various OS’s. One of these is the offscreen plugin. It’s very hard to find any documentation about this, but I gave it a shot anyway, and it seems to work:

$ mrview -platform offscreen anat.mif -mode 3 -plane 1 -intensity 0,400 -noannot -capture.grab -exit

produces the expected screenshot0000.png mugshot of yours truly:

So the only issue to sort out at this point is making sure you have an OpenGL 3.3 compliant graphics driver on your target system/server. And that should just be a matter of installing a recent version of mesa, and either using the default hardware driver if it’s good enough, or defaulting to software rendering if the hardware isn’t present or too old. Not sure how to do that last bit, but I’m sure it’ll be documented somewhere on the mesa website…

3

Hi @jdtournier,

You are quite right, but my experience has been that it’s usually more complicated than it’s worth, and usually fragile too. The school of hard knocks can be a strong deterrent! Which is why I brought up rendering apart from OpenGL. We’ve done VTK rendering “headless” using the Mesa driver, which works fairly well, but again, is fragile and hard to support longer term. My artist son did this for me in Blender, which looks stellar!

tractography.png1024×576 605 KB

And ParaView also does a nice job (please excuse the tractography…):

But I still like mrview:

image.jpg755×682 135 KB

But Blender is too complicated to support effectively, ParaView needs OpenGL (perhaps not with OSPRay, didn’t look into it yet), and so far…

Problems on the Mac

Was very excited about the -platform offscreen option, but my Mac doesn’t like it:

30: $HOME/Source/mrtrix3/release/bin/mrview -platform offscreen dti.mif -tractography.load probabalistic-tractography-400k.tck -tractography.thickness 0.5 -tractography.opacity 0.3 -voxel 0,0,22 -focus false -capture.folder . -capture.prefix screen-grab -capture.grab -exit
QOpenGLWidget is not supported on this platform.
This plugin does not support propagateSizeHints()
This plugin does not support createPlatformOpenGLContext!
This plugin does not support createPlatformOpenGLContext!
QOpenGLWidget: Failed to create context
Segmentation fault: 11

I’m guessing might be better on Ubuntu?

I’ll keep thinking about the problem, and perhaps can figure out something simple apart from the packages above. Thanks for the help.

4

No, unfortunately that’s actually not unexpected. If you look hard enough in the documentation, you’ll find that:

Currently the offscreen platform plugin is only fully supported on X11.

So that kinda rules out macOS… Unless you have an X server running…? Might be worth trying.

5

@jdtournier : That’s awesome! I would expect most headless machines to be running Linux anyway…

6

I never knew about this too; it’s really cool (and in many scenarios mighty useful!). We should eventually maybe advertise this possibility better. I know at least a few people who would be interested using this in a clinical environment (of course only in addition to a majority of established and safe practices).

7

Hi

Sound great !
Although I dit not manage to compile mrview on cluster node (where we only have basic integrated graphic card)
So I won’t be able to run it on the cluster …

Any way I tested on my workstation (with a git pull before to get the last version)
I get :
mrview: [ERROR] unknown option “-platform”

I probably miss something

Many thanks
Romain

8

Was this with Qt5? It won’t work with Qt4, as far as I can tell…

9

Sorry, I read to quickly and miss the point. It works as you said with Qt5
Nice feature !
many thanks
romain

10

Just as a note in case someone has the same issue:

If you get an error “QFontDatabase: Cannot find font directory”, people seem to get it working by creating that directory. In my case that is mkdir /usr/lib/x86_64-linux-gnu/fonts. Here is a bug report about this issue.

Despite this, off-screen rendering does not work on my Ubuntu 14.04 with Qt 5.2.1. The screenshots get only partially rendered, contain GUI elements of other Qt programs running and are mostly black. But that might be a problem with my window manager.

11

Gave this a try on a headless machine with Ubuntu 16.04.2 LTS and Qt 5.5.1. Unfortunately, it did not work :frowning:

QOpenGLWidget is not supported on this platform.
QFontDatabase: Cannot find font directory /usr/lib/x86_64-linux-gnu/fonts - is Qt installed correctly?
QFontDatabase: Cannot find font directory /usr/lib/x86_64-linux-gnu/fonts - is Qt installed correctly?
QFontDatabase: Cannot find font directory /usr/lib/x86_64-linux-gnu/fonts - is Qt installed correctly?
QFontDatabase: Cannot find font directory /usr/lib/x86_64-linux-gnu/fonts - is Qt installed correctly?
This plugin does not support propagateSizeHints()

mrview: [SYSTEM FATAL CODE: SIGSEGV (11)] Segmentation fault: Invalid memory access

What QT version are you using, @jdtournier ?

12

OK, that’s not cool… I’m running Arch Linux, Qt 5.8.0. I just tried it again, and weirdly enough, I get the same error as you now… :-1:

No idea why it no longer works - last time I tried was only a couple of months ago, and it was fine then, most likely using Qt 5.7. I can’t see the recent update being the cause of this - I’ll try reverting to 0.3.16, see if it works on that version, just in case we did something funny… I’ll report back in a minute.

13

No go with the code at version 0.3.16 either… Not sure what’s happened with Qt here. I’ll have a look into it when I have a minute.

14

Could it be related to the introduction of the new QOpenGLWidget infrastructure in Qt?

15

Nope, we’ve been using it for almost two years now…