Selecting appropriate options in "Make"

Selecting appropriate options in "Make"

Robert (guest) 09 Jul 2013 21:57

Hello,

I had a few questions on options in the make file

When you execute the make file you get these options in Lithium.

[osu6811@oakley02 exciting]$ make
Makefile:13: build/make.inc: No such file or directory
perl ./setup.pl

---

1 AIX.xlf.poe.essl
2 CrayXT6
3 g95
4 g95.mac
5 gfortran
6 ifort
7 ifort.XE
8 ifort.mpi.mkl
9 ifort64.mkl

Some are intuitive others not.

For instance. I know the HPC I am logged into has composer XE 2011 (as well as other versions) installed. Included is fortran 12.x.x. I'm also aware that mkl and lapack are installed.

So 6-9 seem to be correct, but 7 and 8 more so. What are the differences? The reference page on compiling Exciting does not discuss the differences.

Here is my library path

[osu6811@oakley02 ~]$ echo $LD_LIBRARY_PATH
/usr/local/mvapich2/1.7-intel/lib:/usr/local/torque/2.5.12/lib:/usr/local/intel/composer_xe_2011_sp1.10.319/compiler/lib/intel64:/usr/local/intel/composer_xe_2011_sp1.10.319/mpirt/lib/intel64:/usr/local/intel/composer_xe_2011_sp1.10.319/ipp/lib/intel64:/usr/local/intel/composer_xe_2011_sp1.10.319/mkl/lib/intel64:/usr/local/intel/composer_xe_2011_sp1.10.319/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21:/usr/local/intel/composer_xe_2011_sp1.10.319/debugger/lib/intel64

Re: Selecting appropriate options in "Make"

Andris Gulans 10 Jul 2013 09:14

Dear Robert,
the main difference between those options is how libraries are supplied and how compilers are referenced. For instance, linking with MKL becomes extremely easy. The option -mkl=sequential (-mkl=parallel) replaces a long list of libraries at the linking stage. If you have the composer XE, then "ifort.XE" is the right one for you. However, it may get a little bit more tedious if you wish to use some mpi other than intel-mpi.

I agree that having so many options is confusing, and the obsolete ones will be removed in the new release. In any case, I would consider those provided platform-dependent make.inc file rather as templates or examples that most likely need to be adjusted. Did you manage to compile exciting?

Robert N (guest) 10 Jul 2013 14:21

Andris,

I decided to compile will all the relevent options and analyze the results.

results:

ifort - no errors -

[osu6811@oakley01 ifort_exciting]$ ls bin
excitingmpi* excitingser* excitingsmp* spacegroup* species* stateconvert* stateinfo*

ifort.xe - errors-
………………………………………………………………… Makefile.libbzint is ready .
make[3]: Entering directory ‘/nfs/12/osu6811/downloads/exciting/build/mpi’
mpif90-ifort -O3 -ip -unroll -scalar_rep -cpp -DXS -DISO -DTETRA -DLIBXC -DMPI -DMPIRHO -DMPISEC -I finclude -c ../../src/libbzint/tetra_internal.f90
make[3]: mpif90-ifort: Command not found
make[3]: * [tetra_internal.o] Error 127
make[3]: Leaving directory ‘/nfs/12/osu6811/downloads/exciting/build/mpi’
make[2]: * [libbzint.a] Error 2
make[2]: Leaving directory ‘/nfs/12/osu6811/downloads/exciting/build/mpi’
make[1]: *** [all] Error 2
make[1]: Leaving directory ‘/nfs/12/osu6811/downloads/exciting/build/mpi’

[osu6811@oakley01 downloads]$ ls xe_exciting/bin
excitingser*

ifort.mpi.mkl - errors -
ld: cannot find -lmkl_lapack
make[3]: * [exciting] Error 1
make[3]: Leaving directory ‘/nfs/12/osu6811/downloads/exciting/build/serial’
make[2]: * [bin] Error 2
make[2]: Leaving directory ‘/nfs/12/osu6811/downloads/exciting/build/serial’
make[1]: * [all] Error 2
make[1]: Leaving directory ‘/nfs/12/osu6811/downloads/exciting/build/serial’
make: * [serial] Error 2

/bin is empty

smpat 10 Jul 2013 14:31

Ahhh yes, the old "cannot find -lmkl_lapack" problem! I've suffered this countless times. Have you tried changing lmkl to mkl? I think that worked for me, at least one of the 10k times I've seen this.

Re: Selecting appropriate options in "Make"

Andris Gulans 10 Jul 2013 14:56

Concerning ifort.xe, it seems you have to edit the build/make.inc manually. mpif90-ifort has to be replaced by whatever relevant mpif90 name. If you have openmpi, it has to be mpif90. If you have intel-mpi, it should be mpiifort or mpif90 -fc=ifort. It may have yet a different name depending on which mpi you are using.

Concerning ifort.mpi.mkl or other options that do not work, it is all about prescribing paths and library files correctly. But, in the end of the day, if you use the same compiler, the same libraries and the same optimization flags, the binaries will not differ.

-lmkl_lapack is not necessary if you have the composer XE. As mentioned before, you can use the shortcuts like -mkl=sequential.

Robert (guest) 10 Jul 2013 16:30

Andris,

Now say I make the changes in "make.inc" How will this correct the situation when no binaries were produced? Should I manually make these changes in make.inc and then rerun "make?" Should I declare these variables prior to running "make"?

$export MPI=mpifort
$make

?

Thank you,

Robert

P.S. I forgot the addition of the last option, ifort64.mkl

ld: cannot find -lmkl_lapack
make[3]: * [exciting] Error 1
make[3]: Leaving directory ‘/nfs/12/osu6811/downloads/exciting/build/serial’
make[2]: * [bin] Error 2
make[2]: Leaving directory ‘/nfs/12/osu6811/downloads/exciting/build/serial’
make[1]: * [all] Error 2
make[1]: Leaving directory ‘/nfs/12/osu6811/downloads/exciting/build/serial’
make: * [serial] Error 2

The old "cannot find -lmkl_lapack" problem.

The results of the above compilations are:

ifort option generates all binaries except the smpmpi hybrid. ifort64 and ifort.mpi.mkl have empty bin folders. ifort.xe has only the serial binary in the bin folder.

Re: Selecting appropriate options in "Make"

Andris Gulans 11 Jul 2013 07:59

Robert,
yes, you make the changes in make.inc and then rerun make. If you have made changes that affect exclusively the linking stage, you do not even have to do "make clean". If you have added/removed some optimization optimisation or debugging flags, you have to do the "cleaning" first.
In your case, if you want to make the "ifort.xe" option work, you have to specify MPIF90=mpiifort (or whatever is the mpi that you are using) in make.inc and then recompile the mpi binary from scratch.

The "old -lmkl_lapack problem" is just a result of not supplying all necessary libraries and paths properly. The make.inc templates provided in exciting cannot be universal and may require some tuning. But, as I said, linking with MKL is easy these days.

Robert N (guest) 11 Jul 2013 16:33

Update.

I spoke with the OSC help desk and was directed to make changes to both the make.inc file and the /src/species/template file.

Here are my files:

*
osu6811@oakley01 exciting]$ cat build/make.inc
F90=ifort
F90_OPTS = -O3 -ip -unroll -scalar_rep
CPP_ON_OPT = -cpp -DXS -DISO -DTETRA -DLIBXC
F77=$(F90)
F77_OPTS = -O3
LIB_ARP =libarpack.a
export USE_SYS_LAPACK=true
LIB_LPK = -mkl
LIB_FFT = fftlib.a
LIB_BZINT=libbzint.a

LIBS= $(LIB_ARP) $(LIB_LPK) $(LIB_FFT) $(LIB_BZINT)

F90_DEBUGOPTS=-g -O0 -warn all -check all -traceback -ftrapuv
F77_DEBUGOPTS=-g -O0 -warn all -check all -traceback -ftrapuv

#Ignore if you don't have MPI or smplibs

MPIF90=mpif90
MPIF90_OPTS=$(F90_OPTS) $(CPP_ON_OPT) -DMPI -DMPIRHO -DMPISEC

F77MT=$(F77)
F90MT=$(F90)
SMP_LIBS= $(LIB_ARP) $(LIB_LPK) $(LIB_FFT) $(LIB_BZINT)
SMPF90_OPTS=$(F90_OPTS)
SMPF77_OPTS=$(SMPF90_OPTS)
MPISMPF90_OPTS=$(SMPF90_OPTS) -DMPI -DMPIRHO -DMPISECBUILDMPI=false
MPIF90MT=$(MPIF90)
BUILDMPI=true
BUILDSMP=true

[osu6811@oakley01 exciting]$ cat src/species/template
include ../../build/make.inc
include ../../build/libraries.inc
FC = $(F90) -g -I../../build/serial/finclude -I../../build/serial
FFLAGS = $(F90_OPTS) $(CPP_ON_OPT)
LD = $(FC)
LDFLAGS = $(F90_OPTS) -L../../build/serial $(LIB_LPK) ../../build/serial/fftlib.a -L../../build/serial/lib $(INTLIBS)
AR = ar
ARFLAGS = -rc
F77=$(F77) $(F77_OPTS)
TMPFILES = *.mod
SUFFIX=ser
[osu6811@oakley01 exciting]$

*

I think it may be worth posting the explanation for these changes as well.

begin email:
//MKL is the Intel Math Kernel Library. It's highly optimized for our processors and includes BLAS and LAPACK along with a lot of other routines. Your software comes with a copy of these libraries that are by default built along with the code; they won't be as fast as the system libraries.

The USE_SYS_LAPACK flag just says don't bother building BLAS and LAPACK because you're using the system version. LIB_LPK is set to the libraries and path for BLAS and LAPACK. Since you're using the Intel complers you just need the -mkl flag. You could have loaded the mkl module (module load mkl) and used the appropriate environment variable. That works with all the compilers, but it's not necessary with Intel.

I consider the change I made to template to be a bug fix. It should work on other systems as well. We don't have separate blas and lapack libraries; they're just part of the mkl libraries. LDFLAGS represents the linker flags, which mostly consists of libraries. You'll note that it's the concatenation of the library flags, except they ignored LIB_LPK and hardcoded that one. //

Robert N (guest) 11 Jul 2013 16:40

Apologies, I didn't post the results.

Compiled with no errors and all binaries were generated except excitingmpismp. I changed into the /bin/mpiandsmp directory and issued "make" again and that generated the excitingmpismp binary.

No errors. I will run some calculations today.

Best,

Robert

Andris Gulans 12 Jul 2013 09:34

You are right about the benefits of using tuned BLAS and LAPACK libraries such as MKL or ACML. This is something that a user has worry about. Actually if you examine the ifort.XE template, you will find there the following lines:
export USE_SYS_LAPACK=true
LIB_LPK = -mkl=sequential
SMP_LIBS=$(LIB_ARP) -mkl=parallel $(LIB_FFT) $(LIB_BZINT)
These lines clearly specify that MKL should be used instead of LAPACK and BLAS that come along with exciting. I think that there was nothing to fix in this template apart from the name of the mpi compiler/wrapper that is system dependent. Or did I miss anything?

What concerns the other template which is called simply ifort, the line with USE_SYS_LAPACK is commented and MKL is not used. Somehow I haven't thought about it. This is the answer to your original question, what is the difference between the different ifort templates. Strictly speaking, make.inc.ifort does not advertise the presence of MKL, hence there is nothing wrong in it.

Furthermore, when you specify -mkl flag as you did in your make.inc, how do you know which part of MKL are you liniking against? Is it the single-threaded or multi-threaded part? If it is the single-threaded part, you will not benefit from the multi-threading when linear algebra is done. Actually I don't know which one is chosen if one specifies -mkl alone, but this is why I would prefer to state it explicitly. As I mentioned earlier, ifort.XE is the right template for you. The only adjustment you had to make was the name of the mpi wrapper in MPIF90.

Robert N (guest) 12 Jul 2013 13:14

Andris,

First let me apologize for not being explicit, the email I posted was from OSC help, it was not my words. I was not critiquing the make.inc templates at all. I was just posting the template that was provided to me from the Ohio Super Computing help center in hopes that you would critique it for me, as you have. I was in the process of inquiring with the person handling my help ticket when they went on vacation so that email was the only explanation I received when I asked for an explanation. I posted that as well also for your critique.

Especially thanks for explaining the difference between the mlk flag I was given and the one you suggested.

Also, when the OSC help desk made the comment:

"I consider the change I made to template to be a bug fix. It should work on other systems as well. We don't have separate blas and lapack libraries; they're just part of the mkl libraries. LDFLAGS represents the linker flags, which mostly consists of libraries. You'll note that it's the concatenation of the library flags, except they ignored LIB_LPK and hardcoded that one."

It was a little unclear to me what they meant, but I know that the only change she made to src>species>template was the addition of $LIB_LPK and the removal of the blas and lapack flags. In regards to the "bug fix," perhaps she meant that this template should also be a function of setup.py or make.inc? That file will most likely still include the blas and lapack flags regardless of which template you select, as they are there before you even run make.

Best,

Robert

One more question? What is LIB_LPK? What is that library responsible for, serial calculations? I'm assuming that is what mkl-sequential means, serial.

Andris Gulans 15 Jul 2013 07:37

Robert,
I am glad this discussion was helpful to you.

The variable LIB_LPK is used for linking non-threaded versions. For the threaded ones, SMP_LIBS is used. But if you wish to monitor whether it goes as intended or not, you can always follow the standard output. There you can find which options and libraries were actually used.