[CPMD-list] Wrong kinetic energy when running with LAM (but not for serial exec.)

[Henryk Wicke]

Dear Axel,

first of all, thank you very much for your help. It's very much appreciated!

> i found the vectorization options like -xT pretty dangerous
> on many machines. the compilers seem much more likely to
> miscompile parts of the code. 

I recompiled both my serial as well as my LAM/MPI version of CPMD 
without the -xT compiler flag, but it didn't change the kinetic energy 
in the final results section of the parallel run's output.

> despite the claims
> of compiler vendors SSE (or SIMD) vectorization does not bring
> larger improvements (if any) for quantum chemistry packages

Thanks for pointing that out, I didn't even consider that before. Using 
the "63 Si-Atoms, PBC, 70Ryd."-benchmark from your Bochum website I'm 
seeing only a <2% wall time increase when not using the -xT flag, so I 
guess it's in general better not to use the flag.

> i would recommend switching to OpenMPI. it is pretty mature
> these days and has better (=faster) implementations collective 
> operations than LAM/MPI.

Following your recommendation I've downloaded the recent version of 
OpenMPI (1.2.4) and compiled it with my set of Intel compilers for Linux 
(see previous mail). Then I rebuilt CPMD (without -xT) using OpenMPI.
Using the above mentioned benchmark it seems to be almost exactly as 
fast as LAM/MPI.
I get another conspicuous result for the water molecule calculation 
again though - see below.

> this is the kinetic energy of the DFT part (not the MD).
> how does the _total_ energy behave? this is the parameter
> that matters. the individual components can fluctuate a bit.
> (although half is not exactly a bit). 

The total energy is fine in all cases, but I also thought that 50 % 
really stands out.
Maybe 25 % stands out even more...here are the "FINAL RESULTS" for my 
OpenMPI calculation with 4 CPUs:

  (K+E1+L+N+X)           TOTAL ENERGY =          -17.14953700 A.U.
  (K)                  KINETIC ENERGY =            3.27325768 A.U.
  (E1=A-S+R)     ELECTROSTATIC ENERGY =          -11.42484340 A.U.
  (S)                           ESELF =           12.63317221 A.U.
  (R)                             ESR =            0.81497664 A.U.
  (L)    LOCAL PSEUDOPOTENTIAL ENERGY =          -16.08838781 A.U.
  (N)      N-L PSEUDOPOTENTIAL ENERGY =            1.73074622 A.U.
  (X)     EXCHANGE-CORRELATION ENERGY =           -4.19008185 A.U.
          GRADIENT CORRECTION ENERGY =            -0.24919014 A.U.

So the components of the total energy don't add up to -17... A.U.
The reference from your website is:

  (K+E1+L+N+X)           TOTAL ENERGY =          -17.14953703 A.U.
  (K)                  KINETIC ENERGY =           12.82302629 A.U.
  (E1=A-S+R)     ELECTROSTATIC ENERGY =          -11.42484300 A.U.
  (S)                           ESELF =           12.63317221 A.U.
  (R)                             ESR =            0.81497834 A.U.
  (L)    LOCAL PSEUDOPOTENTIAL ENERGY =          -16.08838626 A.U.
  (N)      N-L PSEUDOPOTENTIAL ENERGY =            1.73075070 A.U.
  (X)     EXCHANGE-CORRELATION ENERGY =           -4.19008181 A.U.
           GRADIENT CORRECTION ENERGY =           -0.24919014 A.U.

It seems as if the kinetic energy is actually correlated with the amount 
of CPUs I'm using, giving me roughly the serial result divided by the 
number of CPUs. To make sure I repeated the calculation with OpenMPI and 
2 CPUs and got 6.36582575 A.U. for the kinetic energy. With LAM/MPI and 
2 CPUs I got 6.36582683 A.U.
Furthermore my serial binary as well as serial calculations using my 
LAM/MPI or OpenMPI CPMD binaries yield a good agreement with your 
kinetic energy.
All other energies besides the kinetic energy are virtually identical in 
all cases.

Further comments would be very much appreciated.

Best regards,
Henryk