Ground state calculations for larger systems/organic systems

Summary:

Looking for suggestions on how to approach DFT calculations for larger systems (48-150 atoms/unit cell) and if any additional considerations are needed when dealing with organics

Ground state calculations for larger systems/organic systems

Robert (guest) 09 Jul 2013 15:46

Hello,

I am looking for suggestions on how to approach DFT calculations for larger systems (48-150 atoms/unit cell) and if any additional considerations are needed when dealing with organics.

The overall aim is to model the energy-loss function of the materials, but first, a ground state calculation is of course needed.

The simplest system I am working with has 48 atoms per unit cell (4 species) and is a form of Hydroxylapatite ( structure was found on the Crystallography Open Database and converted into an xml input via CIF2Cell. COD ID: 1010647 )

The closest I am getting is with a 1x1x1 k-mesh. In 18 hours it has completed 3 iterations and the data suggests it is in fact converging, but may take several (maybe more than 20) more iterations to actually satisfy the default convergence criteria.

The GSC info is here:
<groundstate swidth="0.01" nempty="10" xctype="GGAPBEsol" ngridk="1 1 1" lradstep="4" gmaxvr="14.0" lmaxvr="8" rgkmax="7.0" maxscl="30"/>

I was considering increasing nempty to something closer to 50 or so since the number of valence states is 155.

I did get a warning for SCL1 and SCL2
—-
Warning(occupy): smallest valence eigenvalue less thanminimum linearization energy : -5.164780185 -1.645800000
for s.c. loop 1

Warning(occupy): smallest valence eigenvalue less thanminimum linearization energy : -3.935421531 -1.645800000
for s.c. loop 2
—-

Since loop 4 is in progress, that suggests loop 3 did not meet this condition.

I hope I have provided adequate information here.

Thank you,

Robert

Re: Ground state calculations for larger systems/organic systems

smpat 09 Jul 2013 16:18

These larger (>48 atoms) systems can be a nightmare. I was suffering from the same painfully slow computation time as you are, and only recently parallized my Elk build, so now I can get an SCL for one of these systems with 4 x 4 x 2 mesh in under an hour. I'd suggest the same, but I have yet to build Exciting successfully on any computer but one, and it doesn't have a parallel structure.

I agree on nempty. I know the rule of thumb is to multiply the value of nempty you would use in a unit cell by the new structure size. In other words, if you go to a 2x1x1 system, and you used nempty of 25 in your 1x1x1 system, use nempty of 50 in the new one.

When I have linearization issues on Elk (which is fundamentally related to Exciting), it's usually because gmaxvr or rgkmax were too low. Have you converged rgkmax?

Robert (guest) 09 Jul 2013 17:03

I believe I mentioned that I have not yet converged my system successfully, so no I have not tested different rgkmax values for convergence if that is what you are asking.

Also, let me clarify that this is not a supercell, this is a unit cell with 48 atoms. It may be possible to reduce it further? If so, I am not aware how to attempt this outside specifying "primcell="true"" in the structure section.

As far as parallelizing, I have mpi parallelization set up and working in exciting; however, exciting parallelizes over k-points so with only 1 kpoint, I do not believe I can use mpi parallelization.

I have not attempted excitingsmp. If I am honest, I do not know how smp parallelization works. If I took a guess, I would say it uses OpenMP which parallelizes over the cores available on a single processor? I am out of my expertise in this field.

I was going to make a separate post inquiring on excitingsmp.

smpat 09 Jul 2013 18:47

Yeah, I've had nothing but trouble so far with parallelized exciting. Even with a large number of k-points, it always seems sluggish.

I don't know of any other method outside of primcell either. That is one massive unit cell. Elk parallelizes over k-points, so I assume Exciting does as well. You could probably have a larger mesh if you're running in parallel, but I'm not an expert in this either.

Re: Ground state calculations for larger systems/organic systems

Andris Gulans 10 Jul 2013 12:37

There may be a couple of ways to speed up larger calculations.
1. Use multithreading (mentioned above). To do it, you have to compile exciting with the smp option, and when you run exciting do not forget to set the environment variable OMP_NUM_THREADS to an appropriate value. Even if does not scale very well, multithreading speeds up calculations. If you decide to make a comparison, please, do not pay any attention to the timings printed in the end of INFO.OUT.
2. If the number of states is small compared to the number of basis functions, it pays off to use arpack instead of lapack for the diagonalization. Just add the following to the groundstate section.
<solver type="Arpack" epsarpack="1d-12" />
It may happen, however, that arpack exceeds the maximum number of iterations which is hardcoded. If this occurs, you will have to make a small change in src/src_iterative_solver/iterativearpacksequn.F90 . For instance, the line "maxitr = 40 * nstfv" can be changed to "maxitr = 400 * nstfv".

Apart from what I have described, it may pay off to run the calculation first with a smaller rgkmax value.

Re: Ground state calculations for larger systems/organic systems

smpat 10 Jul 2013 13:14

Stupid question related to the environmental variable: can we just type "export OMP_NUM_THREADS=x" in the command line? Also, wouldn't the times for each loop reflect the speed of calculations, or am I wrong?

Re: Ground state calculations for larger systems/organic systems

Andris Gulans 10 Jul 2013 13:55

If you run your calculations interactively "export OMP_NUM_THREADS=x" will do the job. If you run them through a queueing system, it may be necessary to include the export line in a job-submit script or even to define it by other means supported by the queueing system. Sometimes it also happens that people forget to set OMP_NUM_THREADS to 1, when they do not intend a multithreaded run, but neverless use a binary with the multithreading support.

"the times for each loop reflect the speed of calculations" - do you refer to the timing information in exciting? If so, the problem is that the time counting is not accurate in the lithium release. I would not rely on it. But the future release should not have this problem.

Re: Ground state calculations for larger systems/organic systems

smpat 10 Jul 2013 14:33

My submission script for Elk (which I imagine will be similar to what I'll use once I figure out how to get Exciting online), has the form

#$ -N BC2P2
#$ -cwd
#$ -pe openmpi* 16
#$ -q medium*
#$ -l dedicated=4

export OMP_NUM_THREADS=4
export OMP_NESTED=true
mpirun ./elk

Is this along the lines of what you were saying? OMP_NESTED=true supposedly parallelizes k-points, but I haven't noticed any difference with/without it.

Yes, I'm referring to the timing information, but now that I know it's not accurate in this version I'll just time it myself.

Robert N (guest) 10 Jul 2013 14:41

What type of scheduler are you submitting your jobs to smpat?

smpat 10 Jul 2013 14:51

I'm not sure if this answers your question:

"We use a program (often known as a batch-scheduler or batch-queue) to coordinate this process of allocating resources. On Newton systems, the batch-scheduler is the Grid Engine."

There's a link to the grid engine, but I'm not allowed to post it (not enough karma). It's the Newton HPC at Univ Tenn.

Robert N (guest) 10 Jul 2013 14:37

Andris,

my version of exciting compiled with both the excitingmpi and excitingsmp binaries but not the hybrid.

I have been executing the mpi version with the command "mpiexec -np $NP excitingmpi"

this is of course through a PBS queue.

I do not really understand how smp works, is that OpenMP? If so, it can only be executed on one node (shared memory) for a max of 12 processors in my case. Or is smp something different?

How would you execute the excitingsmp binary? The same as the serial?

Andris Gulans 10 Jul 2013 15:11

Yes, you got it right, it is basically openmp, and you cannot run it across different nodes. In your case, the maximum number of threads that would make sense is 12.
To compile an mpi-smp hybrid, try to execute "make mpiandsmp".

Re: Ground state calculations for larger systems/organic systems

Andris Gulans 10 Jul 2013 15:15

Sean,
the example you have shown is exactly what I had in mind.

Re: Ground state calculations for larger systems/organic systems

smpat 10 Jul 2013 15:45

You might find this interesting: When I include export OMP_NUM_THREADS=4 with or without export OMP_NESTED=true, my computation time roughly doubles when compared to the general

#$ -N BC2P2
#$ -cwd
#$ -pe openmpi* 16
#$ -q medium*
#$ -l dedicated=4

mpirun ./elk

I haven't seen any improvement (in the computation times provided) with either of these lines, separately or together. According to the INFO.OUT file, the example directly above seems to run the fastest. Unless, of course, the computation is inaccurate in elk output for OMP_NUM_THREADS=x as well.

Robert (guest) 11 Jul 2013 00:58

I believe, if you want to run a hybrid mpi and openmp the typical execution for say 4 nodes with 12 processors per node would be….

OMP_NUM_THREADS=12

mpirun -pernode [options] [program]

in this case [program] would be excitingmpismp I would guess.

can you post a link to the scheduling syntax Grid Engine uses? I realize it won't let you post a link so just "sensor" the obvious parts that identify it as a web url.

I'm trying to understand what you are telling the scheduler. It seems like you are specifying OMP_NUM_THREADS to be both 4 and 16.

Also, I asked you this before. What type of mpi/mpich are you using? mpich2? mvapich?

That is important to know. You may ONLY have OpenMP.

smpat 11 Jul 2013 05:12

Thanks for looking into this, Robert. Here's what I'm able to post:

newton.utk.edu#bin#view#Main#ParallelJobs

Robert (guest) 11 Jul 2013 22:29

Ok. So here is what I think. Your scheduler is basically setting the field for 16 mpi processes to open and for each to utilize 4 processors and communicate via smp (or openMP.)

That said, I see no reason for you to have to specify OMP as well, so I'm not surprised that muddled things up.

Also, Since you have now set the stage for an mpi+smp/openMP hybrid job, you need to run a hybrid application. So what you would need to run with this type of script is excitingmpismp.

If you were to remove the "dedicated=4" part, then you could execute excitingmpi.

If you used :
#$ -pe threads 8
#$ -N test
/path/to/my/program

then you could use excitingsmp

I don't understand what you mean by "I haven't seen any improvement (in the computation times provided) with either of these lines, separately or together."

smpat 12 Jul 2013 10:31

I meant to post my results on this, but was away from the office yesterday afternoon.

I found substantial improvement (this is, of course, case specific) with the submission

#$ -N MYFILE
#$ -cwd
#$ -pe openmpi* 40
#$ -q medium*
export OMP_THREAD_LIMIT=30
export OMP_NESTED=true
mpirun /path/to/my/program

My impression is that the number of threads created was counterproductive (although I am a novice in this area), and using OMP_THREAD_LIMIT with this specific value (in my case) yields the fastest computation time.