Compiling with SMP and GotoBLAS2

Compiling with SMP and GotoBLAS2

Fabio Fumi 16 Aug 2012 10:53

Following suggestions here, I tried compiling using the SMP option (I'm running the package on a 2-CPU Intel i5, Ubuntu system), so I downloaded GotoBLAS2 and compiled it succesfully :

 ls -l /media/u02/GotoBLAS2/*.so
-rwxr-xr-x 1 ffxx68 ffxx68 10108718 2012-08-16 03:09 /media/u02/GotoBLAS2/libgoto2_nehalemp-r1.13.so
lrwxrwxrwx 1 ffxx68 ffxx68       26 2012-08-16 03:09 /media/u02/GotoBLAS2/libgoto2.so -> libgoto2_nehalemp-r1.13.so

Then I changed make.inc, in order to link this and enable SMP:

F77=$(F90)
F90_OPTS = -O3 -march=native # -flto
F77_OPTS =  $(F90_OPTS)
CPP_ON_OPT = -cpp -DXS -DISO -DTETRA -DLIBXC
LIB_ARP =libarpack.a
#LIB_LPK =  -L./ -llapack -lblas
LIB_LPK = -L/media/u02/GotoBLAS2 -lgoto2 -lpthread 
LIB_LPKSMP = -L/media/u02/GotoBLAS2 -lgoto2 -lpthread
LIB_FFT = fftlib.a
LIB_BZINT= libbzint.a
LIBS= $(LIB_ARP) $(LIB_LPK) $(LIB_FFT) $(LIB_BZINT) # -flto -fwhole-program
F90_DEBUGOPTS=-g -fbounds-check -fbacktrace -Wall
F77_DEBUGOPTS= $(F90_DEBUGOPTS)
#Ignore if you don't have MPI or smplibs
MPIF90=mpif90
MPIF90_OPTS=$(F90_OPTS) $(CPP_ON_OPT) -DMPI -DMPIRHO -DMPISEC
F77MT=$(F77)
F90MT=$(F90)
#SMP_LIBS=$(LIBS)
SMP_LIBS= -mkl=parallel   $(LIB_ARP) $(LIB_LPK) $(LIB_FFT) $(LIB_BZINT)
SMPF90_OPTS=-fopenmp $(F90_OPTS) 
SMPF77_OPTS=$(SMPF90_OPTS) 
BUILDMPI=false
BUILDSMP=true

New exciting (lithium) make was completed succesfully:

  ls -l /media/u02/exciting/bin
total 18136
lrwxrwxrwx 1 ffxx68 ffxx68      11 2012-08-16 03:37 exciting -> excitingsmp
-rwxr-xr-x 1 ffxx68 ffxx68 6378628 2012-08-16 03:26 excitingser
-rwxr-xr-x 1 ffxx68 ffxx68 6392140 2012-08-16 03:34 excitingsmp
-rwxr-xr-x 1 ffxx68 ffxx68 1934687 2012-08-16 03:34 spacegroup
-rwxr-xr-x 1 ffxx68 ffxx68 3390566 2012-08-16 03:34 species
-rwxr-xr-x 1 ffxx68 ffxx68  230352 2012-08-16 03:34 stateconvert
-rwxr-xr-x 1 ffxx68 ffxx68  230184 2012-08-16 03:34 stateinfo//

But, execution fails for a missing library, which I don't understand:

 exciting
exciting: error while loading shared libraries: libgoto2.so: cannot open shared object file: No such file or directory

Re: Compiling with SMP and GotoBLAS2

Fabio Fumi 16 Aug 2012 11:35

Ok, solved by myself. Just a basic Linux issue. I was missing this:

export LD_LIBRARY_PATH=/media/u02/GotoBLAS2:$LD_LIBRARY_PATH

Re: Compiling with SMP and GotoBLAS2

Fabio Fumi 16 Aug 2012 13:55

Ok, I've managed to make it work… but SMP is slower than serial!

This are timings of the following, relatively simple, calculation:

<structure speciespath="../../species/">
        <crystal scale="8.2384">
            <basevect>0.00 0.50 0.50</basevect>
            <basevect>0.50 0.00 0.50</basevect>
            <basevect>0.50 0.50 0.00</basevect>
        </crystal>
        <species speciesfile="C.xml">
            <atom coord="0.00 0.00 0.00" />
        </species>
        <species speciesfile="Si.xml">
            <atom coord="0.25 0.25 0.25" />
        </species>
    </structure>

    <groundstate ngridk="8 8 8" 
                     rgkmax="7.0"
                     maxscl="20">
        </groundstate>

SMP:

Timings (CPU seconds) :
 initialisation                        :         1.04
 Hamiltonian and overlap matrix set up :        40.41
 first-variational secular equation    :        50.42
 charge density calculation            :         6.01
 potential calculation                 :         4.95
 total                                 :       102.83

SER:

Timings (CPU seconds) :
 initialisation                        :         0.79
 Hamiltonian and overlap matrix set up :        15.39
 first-variational secular equation    :        24.70
 charge density calculation            :         5.08
 potential calculation                 :         2.92
 total                                 :        48.88

Any hint?

Re: Compiling with SMP and GotoBLAS2

Andris Gulans 27 Aug 2012 07:21

Most likely, the problem is that GotoBLAS is outdated. It is not supported by its developers anymore, but GotoBLAS has a successor - OpenBLAS. You can download it from http://xianyi.github.com/OpenBLAS/. I have tried to calculate your example (SiC) with exciting linked with OpenBLAS, and there was a modest speedup when I increased the number of cores.

I suspect OpenBLAS should solve your issue, but there are more things to watch out for.
1) The speedup is expected to become better, once you try to calculate something bigger than a unit cell of SiC.
2) It is not clear (at least to me) how efficient the multi-threading is on a laptop. But I can't promise that there is a problem, I am simply suspicious.
3) If you don't need all Kohn-Sham states that can be acquired in a complete diagonalisation, consider using ARPACK instead of LAPACK for the eigenvalue problem. ARPACK can be chosen in the element solver. ARPACK can make your calculation at least a couple of times faster, and it is also SMP-friendly. In any case, it should be worth of testing.

Re: Compiling with SMP and GotoBLAS2

Fabio Fumi 27 Aug 2012 10:03

Hi Andris, thanks for the suggestion.

Unfortunately, using OpenBLAS didn't improve things:

Timings (CPU seconds) :
initialisation : 1.01
Hamiltonian and overlap matrix set up : 40.86
first-variational secular equation : 50.29
charge density calculation : 7.94
potential calculation : 4.78
total : 104.88

Anyhow… I'll give it a try with larger systems, maybe.

Re: Compiling with SMP and GotoBLAS2

Andris Gulans 27 Aug 2012 13:06

Are you running this test on a laptop? If you are, could you try the same on a desktop computer or a cluster? If even that does not help, may be we should compare our compilation settings. My timings with exciting compiled using ifort and OpenBLAS are as follows.

$ export OMP_NUM_THREADS=1
$ time ../exciting/bin/excitingsmp

real    0m33.948s
user    0m45.619s
sys     0m16.001s

$ export OMP_NUM_THREADS=2
$ time ../exciting/bin/excitingsmp

real    0m30.120s
user    0m42.399s
sys     0m11.977s

$ export OMP_NUM_THREADS=4
$ time ../exciting/bin/excitingsmp

real    0m25.941s
user    0m51.023s
sys     0m35.182s

$ export OMP_NUM_THREADS=8
$ time ../exciting/bin/excitingsmp

real    0m26.205s
user    0m50.859s
sys     0m36.514s

Re: Compiling with SMP and GotoBLAS2

Fabio Fumi 29 Aug 2012 08:05

I'm using a 2-CPU Ubuntu Virtual Machine, on a Lenovo T410, intel i5, WIn7 laptop.
Host CPU goes to 50% (the i5 is a 4-core cpu). I don't have another host at hand… I should find one.
What is the difference with respect to a desktop?
What system are you running those timings on?

Re: Compiling with SMP and GotoBLAS2

Andris Gulans 29 Aug 2012 09:22

I have run these tests on 8-core Intel(R) Xeon(R) CPU E31245 @ 3.30GHz with 8Mb cache. By the way, I have repeated these tests also on a bit older CPUs (Intel(R) Core(TM)2 Quad CPU Q9450 @ 2.66GHz with 6Mb cache), and the trend was similar. I mean, if more cores in an SMP run were used, the execution times became faster.

The important difference between desktop PCs and laptops is normally in the bandwidth for a communication between CPU and RAM and the cache size. However, I am not an expert in using laptops for first principles calculations, and I can't tell you for sure whether these two differences are the decisive ones. Who knows, maybe the problem is in the Virtual Machine environment…

If you intend to use a laptop for calculations, you will probably be restricted to small system sizes. This is perfectly fine if this is what you are after, but for small systems the gain in SMP parallelisation is not that impressive even on a modern desktop PC or even a typical cluster. My tests above confirm it. If you really would like to accelerate ground-state calculations of small systems, you probably should consider the MPI parallelisation. But it depends on what kind of calculations are you interested in. I guess this brings me to the point that we should write an SMP/MPI tutorial.

Re: Compiling with SMP and GotoBLAS2

Fabio Fumi 31 Aug 2012 08:44

Thanks Andris for these clarifications. I'm not really sure what kind of calculations I intend to do… I'm not in the research and I'm interested in this more like a "hobby"… for the moment :) Indeed, it's nonsense to use laptops for serious calculations and much more powerful hardware is usually needed. But… why not… for educational purposes, maybe.
Btw - what are the CPU time basic scaling rules, on number of atoms and k-grid points?