I found substantial improvement (this is, of course, case specific) with the submission

#$ -N MYFILE
#$ -cwd
#$ -pe openmpi* 40
#$ -q medium*
export OMP_THREAD_LIMIT=30
export OMP_NESTED=true
mpirun /path/to/my/program

My impression is that the number of threads created was counterproductive (although I am a novice in this area), and using OMP_THREAD_LIMIT with this specific value (in my case) yields the fastest computation time.

That said, I see no reason for you to have to specify OMP as well, so I'm not surprised that muddled things up.

Also, Since you have now set the stage for an mpi+smp/openMP hybrid job, you need to run a hybrid application. So what you would need to run with this type of script is excitingmpismp.

If you were to remove the "dedicated=4" part, then you could execute excitingmpi.

If you used :
#$ -pe threads 8
#$ -N test
/path/to/my/program

then you could use excitingsmp

I don't understand what you mean by "I haven't seen any improvement (in the computation times provided) with either of these lines, separately or together."

newton.utk.edu#bin#view#Main#ParallelJobs

OMP_NUM_THREADS=12

mpirun -pernode [options] [program]

in this case [program] would be excitingmpismp I would guess.

can you post a link to the scheduling syntax Grid Engine uses? I realize it won't let you post a link so just "sensor" the obvious parts that identify it as a web url.

I'm trying to understand what you are telling the scheduler. It seems like you are specifying OMP_NUM_THREADS to be both 4 and 16.

Also, I asked you this before. What type of mpi/mpich are you using? mpich2? mvapich?

That is important to know. You may ONLY have OpenMP.

#$ -N BC2P2
#$ -cwd
#$ -pe openmpi* 16
#$ -q medium*
#$ -l dedicated=4

mpirun ./elk

I haven't seen any improvement (in the computation times provided) with either of these lines, separately or together. According to the INFO.OUT file, the example directly above seems to run the fastest. Unless, of course, the computation is inaccurate in elk output for OMP_NUM_THREADS=x as well.

"We use a program (often known as a batch-scheduler or batch-queue) to coordinate this process of allocating resources. On Newton systems, the batch-scheduler is the Grid Engine."

There's a link to the grid engine, but I'm not allowed to post it (not enough karma). It's the Newton HPC at Univ Tenn.

my version of exciting compiled with both the excitingmpi and excitingsmp binaries but not the hybrid.

I have been executing the mpi version with the command "mpiexec -np $NP excitingmpi"

this is of course through a PBS queue.

I do not really understand how smp works, is that OpenMP? If so, it can only be executed on one node (shared memory) for a max of 12 processors in my case. Or is smp something different?

How would you execute the excitingsmp binary? The same as the serial?

#$ -N BC2P2
#$ -cwd
#$ -pe openmpi* 16
#$ -q medium*
#$ -l dedicated=4

export OMP_NUM_THREADS=4
export OMP_NESTED=true
mpirun ./elk

Is this along the lines of what you were saying? OMP_NESTED=true supposedly parallelizes k-points, but I haven't noticed any difference with/without it.

Yes, I'm referring to the timing information, but now that I know it's not accurate in this version I'll just time it myself.

"the times for each loop reflect the speed of calculations" - do you refer to the timing information in exciting? If so, the problem is that the time counting is not accurate in the lithium release. I would not rely on it. But the future release should not have this problem.

Apart from what I have described, it may pay off to run the calculation first with a smaller rgkmax value.

I don't know of any other method outside of primcell either. That is one massive unit cell. Elk parallelizes over k-points, so I assume Exciting does as well. You could probably have a larger mesh if you're running in parallel, but I'm not an expert in this either.

Also, let me clarify that this is not a supercell, this is a unit cell with 48 atoms. It may be possible to reduce it further? If so, I am not aware how to attempt this outside specifying "primcell="true"" in the structure section.

As far as parallelizing, I have mpi parallelization set up and working in exciting; however, exciting parallelizes over k-points so with only 1 kpoint, I do not believe I can use mpi parallelization.

I have not attempted excitingsmp. If I am honest, I do not know how smp parallelization works. If I took a guess, I would say it uses OpenMP which parallelizes over the cores available on a single processor? I am out of my expertise in this field.

I was going to make a separate post inquiring on excitingsmp.

I agree on nempty. I know the rule of thumb is to multiply the value of nempty you would use in a unit cell by the new structure size. In other words, if you go to a 2x1x1 system, and you used nempty of 25 in your 1x1x1 system, use nempty of 50 in the new one.

When I have linearization issues on Elk (which is fundamentally related to Exciting), it's usually because gmaxvr or rgkmax were too low. Have you converged rgkmax?

I am looking for suggestions on how to approach DFT calculations for larger systems (48-150 atoms/unit cell) and if any additional considerations are needed when dealing with organics.

The overall aim is to model the energy-loss function of the materials, but first, a ground state calculation is of course needed.

The simplest system I am working with has 48 atoms per unit cell (4 species) and is a form of Hydroxylapatite ( structure was found on the Crystallography Open Database and converted into an xml input via CIF2Cell. COD ID: 1010647 )

The closest I am getting is with a 1x1x1 k-mesh. In 18 hours it has completed 3 iterations and the data suggests it is in fact converging, but may take several (maybe more than 20) more iterations to actually satisfy the default convergence criteria.

The GSC info is here:
<groundstate swidth="0.01" nempty="10" xctype="GGAPBEsol" ngridk="1 1 1" lradstep="4" gmaxvr="14.0" lmaxvr="8" rgkmax="7.0" maxscl="30"/>

I was considering increasing nempty to something closer to 50 or so since the number of valence states is 155.

I did get a warning for SCL1 and SCL2
—-
Warning(occupy): smallest valence eigenvalue less thanminimum linearization energy : -5.164780185 -1.645800000
for s.c. loop 1

Warning(occupy): smallest valence eigenvalue less thanminimum linearization energy : -3.935421531 -1.645800000
for s.c. loop 2
—-

Since loop 4 is in progress, that suggests loop 3 did not meet this condition.

I hope I have provided adequate information here.

Thank you,

Robert