Error population_template > undefined symbol: omp_get_max_threads

Hi all,

I’ve been struggling the whole day to run the command population_template, but I cant figure out why the command is failing. Any suggestions?

Command:  mraverageheader -fill [/scratch/p270430/GGB/template/fod_input/*_wmfod_norm.mif (12 items)] - | mrgrid - regrid -voxel 1.25,1.25,1.25 average_header.mif

population_template: [ERROR] mraverageheader -fill [/scratch/p270430/GGB/template/fod_input/*_wmfod_norm.mif (12 items)] - | mrgrid - regrid -voxel 1.25,1.25,1.25 average_header.mif (population_template:945)
population_template: [ERROR] Information from failed command:


population_template:
                     /apps/haswell/software/MRtrix/3.0.3-foss-2021a/bin/mraverageheader: symbol lookup error: /software/software/MRtrix/3.0.3-foss-2021a/lib/libmrtrix.so: undefined symbol: omp_get_max_threads
                     mrgrid: [ERROR] no filename supplied to standard input (broken pipe?)
                     mrgrid: [ERROR] error opening image "-"
population_template:

I’ve not come across this one before… But it looks to me like you might have compiled a version of MRtrix with OpenMP support (by providing the configure script with the -openmp flag), but the system you’re running on doesn’t have the OpenMP library available – maybe you’ve deployed across to a different system? Or removed the OpenMP package? Or need to module load it…?

If that’s not it, then I’m not sure, we’d need to dig a bit deeper …

Thanks a lot for getting back to me…

Hmm this could all very well be. I’m running the analysis on the university its cluster. They have installed the latest version of MRTrix for me a whole ago. Not sure how they did it…

I cannot module load it, the OpenMP library is not available. Below the list of modules that are loaded. All other steps run perfectly fine on the cluster.

Dear @jdtournier,

I’ve figured out that this issue has nothing to do with the installation or dependencies: the script runs fine as long as the command population_template has to deal with 11 inputs (of course in both the mask and fod folder). As soon as there are 12 or more, it fails. It doesn’t matter which subjects this 12th input refers to.

Any ideas?

Not really… :sweat_smile:

Maybe just to get to the root cause, can you post the output of this command:

ldd $(which mraverageheader)

If we see the OpenMP library in there, that’ll at least confirm the problem.

Assuming that’s the issue, there are two options:

  • contact your HPC support team to ask them where (and whether!) the OpenMP library is located, and how to make sure the dynamic linker can load it when it launches mraverageheader (probably the easiest option)

  • contact your HPC support team to ask them to recompile MRtrix without the -openmp flag

Hope this helps!

THe output you’re requesting

linux-vdso.so.1 => (0x00007fffa8f93000)
libmrtrix.so => /software/software/MRtrix/3.0.3-foss-2021a/lib/libmrtrix.so (0x00007f58de361000)
libstdc++.so.6 => /software/software/GCCcore/10.3.0/lib64/libstdc++.so.6 (0x00007f58de697000)
libm.so.6 => /lib64/libm.so.6 (0x00007f58de043000)
libgcc_s.so.1 => /software/software/GCCcore/10.3.0/lib64/libgcc_s.so.1 (0x00007f58de02a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f58dde0e000)
libc.so.6 => /lib64/libc.so.6 (0x00007f58dda40000)
/lib64/ld-linux-x86-64.so.2 (0x00007f58de661000)
libz.so.1 => /software/software/zlib/1.2.11-GCCcore-10.3.0/lib/libz.so.1 (0x00007f58dda27000)
libtiff.so.5 => /software/software/LibTIFF/4.2.0-GCCcore-10.3.0/lib/libtiff.so.5 (0x00007f58dd99c000)
libpng16.so.16 => /software/software/libpng/1.6.37-GCCcore-10.3.0/lib/libpng16.so.16 (0x00007f58dd960000)
liblzma.so.5 => /software/software/XZ/5.2.5-GCCcore-10.3.0/lib/liblzma.so.5 (0x00007f58dd938000)
libjpeg.so.8 => /software/software/libjpeg-turbo/2.0.6-GCCcore-10.3.0/lib/libjpeg.so.8 (0x00007f58dd897000)

No OpenMP library :sweat_smile:
I already contacted our HPC support team and they figured out it doesn’t have a problem with a small number of inputs, and then I found out that the critical number for the command to fail is 12…

OK, I’m really stumped. I’ve just tried to compile a version here with OpenMP support, and sure enough, libomp.so shows up in the list:

$ ldd bin/mraverageheader 
	linux-vdso.so.1 (0x00007ffc459e5000)
	libmrtrix.so => /home/donald/exp/mrtrix3/bin/../lib/libmrtrix.so (0x00007f3dbe200000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f3dbde00000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f3dbe118000)
	libomp.so => /usr/lib/libomp.so (0x00007f3dbdcef000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f3dbe569000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f3dbdb08000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f3dbe661000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f3dbe54d000)
	libtiff.so.5 => /usr/lib/libtiff.so.5 (0x00007f3dbe083000)
	libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007f3dbe04a000)
	libzstd.so.1 => /usr/lib/libzstd.so.1 (0x00007f3dbda5f000)
	liblzma.so.5 => /usr/lib/liblzma.so.5 (0x00007f3dbda35000)
	libjpeg.so.8 => /usr/lib/libjpeg.so.8 (0x00007f3dbd9b2000)

If I uninstall OpenMP and try to run that command, it immediately fails to run no matter what:

$ bin/mraverageheader 
bin/mraverageheader: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory

I’ve tried to run a version without OpenMP, and that seems to work fine no matter how many inputs I give it.


I can’t find any direct use of or reference to the omp_get_max_threads function in our code. So the best bet is it might have come from a dependency, with Eigen being the only realistic candidate here.

So I had a look through the code in Eigen, and the only reference I can find to that particular function (omp_get_max_threads) is in eigen/Eigen/src/Core/products/Parallelizer.h, line 35:

   31     #ifdef EIGEN_HAS_OPENMP                                                                                             
   32     if(m_maxThreads>0)                                                                                                  
   33       *v = m_maxThreads;                                                                                                
   34     else                                                                                                                
   35       *v = omp_get_max_threads();                                                                                       
   36     #else                                                                                                               
   37     *v = 1;                                                                                                             
   38     #endif                                                                                                              

which wouldn’t be included unless EIGEN_HAS_OPENMP is defined, and that is only defined if OpenMP is present and EIGEN_DONT_PARALLELIZE is not defined (eigen/Eigen/Core, line 262):

261 #if (defined _OPENMP) && (!defined EIGEN_DONT_PARALLELIZE)                                                                
262   #define EIGEN_HAS_OPENMP                                                                                                
263 #endif                                                                                                                    

and EIGEN_DONT_PARALLELIZE is always set if configure runs without -openmp (configure, line 969):

968 if not openmp:                                                                                                           
 969   eigen_cflags += [ '-DEIGEN_DONT_PARALLELIZE' ]    

What’s also weird is that the message you get suggests that the omp_get_max_threads is referenced in the MRtrix library (lib/libmrtrix.so). There’s a (faint) chance the library was compiled with OpenMP, and the rest of the executables weren’t (it would be very weird though). In any case, you can double-check whether that function is indeed referenced in the library with this command:

nm /software/software/MRtrix/3.0.3-foss-2021a/lib/libmrtrix.so | grep omp_get_max_threads

If that shows anything, that might point to some strange mismatch in there. Maybe if you also run:

ldd /software/software/MRtrix/3.0.3-foss-2021a/lib/libmrtrix.so 

and spot libomp.so in the output, that might reinforce that conclusion.

If this was built from source, you might also still have the build config file available, it would be useful to inspect that. It it’s still there, I suspect this might dump its contents (long shot though):

cat /software/software/MRtrix/3.0.3-foss-2021a/config

If you do find traces of OpenMP in the library, I think my best advice would be to clean build the lot from scratch…

Thanks a lot for digging into this so deeply.

I’ve forwarded your reply to our HPC support team. What I can already tell you is that:

nm /software/software/MRtrix/3.0.3-foss-2021a/lib/libmrtrix.so | grep omp_get_max_threads

gives (nothing):

             U omp_get_max_threads

and; ldd /software/software/MRtrix/3.0.3-foss-2021a/lib/libmrtrix.so
gives no libomp.so:

linux-vdso.so.1 =>  (0x00007ffc3ad91000)
libz.so.1 => /software/software/zlib/1.2.11-GCCcore-10.3.0/lib/libz.so.1 (0x00007eff4485e000)
libtiff.so.5 => /software/software/LibTIFF/4.2.0-GCCcore-10.3.0/lib/libtiff.so.5 (0x00007eff447d2000)
libpng16.so.16 => /software/software/libpng/1.6.37-GCCcore-10.3.0/lib/libpng16.so.16 (0x00007eff44796000)
libstdc++.so.6 => /software/software/GCCcore/10.3.0/lib64/libstdc++.so.6 (0x00007eff44172000)
libm.so.6 => /lib64/libm.so.6 (0x00007eff43e70000)
libgcc_s.so.1 => /software/software/GCCcore/10.3.0/lib64/libgcc_s.so.1 (0x00007eff44760000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007eff43c54000)
libc.so.6 => /lib64/libc.so.6 (0x00007eff43886000)
lib64/ld-linux-x86-64.so.2 (0x00007eff44659000)
liblzma.so.5 => /software/software/XZ/5.2.5-GCCcore-10.3.0/lib/liblzma.so.5 (0x00007eff44737000)
libjpeg.so.8 => /software/software/libjpeg-turbo/2.0.6-GCCcore-10.3.0/lib/libjpeg.so.8 (0x00007eff44696000)

… I’ll keep you updated in case the support team finds something …

Hinke asked me to look into the issue. MRtrix has been compiled with openmp support, which is one of the configure options.
But I discovered that there are no OpenMP pragmas in the code itself. This means that the flag does not do anything besides introducing the -fopenmp flag during compile and link time.
But since no OpenMP is actually used libgomp is not actually linked by the compiler. I also found in the logs that libmrtrix3.so is linked without the -fopenmp flag.

I still don’t have a clue why the symbol error pops up when using 12 input files, but not when using 11. But I think that removing the -openmp option from the configure step is probably better. This should completely prevent openmp functions to be introduced, which is fine for code that does not use it.

I will also create a PR for the EasyBuild system easyblock file for mrtrix. We are using EasyBuild for installing software on our cluster.
The relevant files for MRtrix are:

Actually, this does indicate that the library refers to an omp_get_max_threads symbol (presumably a function), though it is undefined in this library. In other words, it expects it to be defined in some other library. I don’t get how this can happen, as in that case I would have expected the library or the command to have been linked with the OpenMP library, which should show up in the ldd output, but that’s not the case. Yet if I try to compile an OpenMP version on my side, I do see the OpenMP library in the ldd output, as expected. Not sure that’s going on…

Indeed, our codebase makes no reference to OpenMP, it would all be entirely through Eigen – but since it’s a header-only library, that will bring it within our own executables.

My guess is that Eigen uses some simple heuristic to determine when it might be worth the overhead of starting other threads rather than simply running within a single thread, and in our use case, 12 happens to be when this gets triggered. But that’s pure speculation…

I agree. This option was introduced to support workflows that aren’t actually merged into the master branch yet. So there’s no need for it at all – I just hadn’t anticipated that it might cause problems…

I hadn’t come across that one, thanks for pointing this out!

Cheers,
Donald

Dear Donald,

Thanks again for the extensive reply.

Fokke rebuilt MRTrix while removing the OpenMP option. The code runs fine now! Finger crossed for a successful completion.

1 Like