DDotMPI and DGEMVMPI implementation #45

CYChenFudan · 2020-06-05T11:49:02Z

MPI versions for the blas subroutines ddot and dgemv are presented. They are included in lib/mpif90/mpiu.f90, then are called by dvdson module.

00. src/tool: Makefile, add "chmod a+x $(GRASP)/bin/rsave" 01. jj2lsj90: In jj2lsj_code.f90, lines 220 and 222 are changed to deal with both parities present. 03. rcsfzerofirst90: In RCSFzerofirst.f90, the number of CSFs in zero-order space are output. The informations could be used conveniently in RCSF and RCI calculations. 04. rmcdhf90: In newco.f90, weighted average energy is output to monitor the energy convergence. 05. rangular90_mpi: getinf.90 modified for iccut=1 06. rci90_mpi: setham_gg.f90, only myid=0 reports how far the calculation has proceeded. 07. libdvd90, dvdson.f90: make lines 1119 and 1120 work for convergence monitor. The output messages are very useful for extremely large-scaled calculations. 08. libdvd90, dvdson.f90: function TSTSEL is modified, so that dvdson performs at least TWO iteration calculations. Or, error results could be obtained in some cases (in O-like isoelectronic sequence?). 09. coming dvdson--mpi

jongrumer · 2020-09-22T08:41:40Z

Ok, I'm slowly getting up to speed after summer vacation. Sorry for the delay!

You have done many things here Chen, very nice! Just summarizing the changes you have made here, so it's easy for everyone to see:

Note: [output] means that the edit only affects output from the code to screen, and is therefore a less sensitive modification.

src/tool: Makefile, add "chmod a+x $(GRASP)/bin/rsave" [already implemented]
jj2lsj90: In jj2lsj_code.f90, lines 220 and 222 are changed to deal with both parities present. [rejected]
rcsfzerofirst90: In RCSFzerofirst.f90, the number of CSFs in zero-order space
are output. The informations could be used conveniently in RCSF and RCI calculations. [output, ok]
rmcdhf90: In newco.f90, weighted average energy is output to monitor the energy convergence. [output, ok]
rangular90_mpi: getinf.90 modified for iccut=1 [output, ok]
rci90_mpi: setham_gg.f90, only myid=0 reports how far the calculation has proceeded. [output, ok]
libdvd90, dvdson.f90: make lines 1119 and 1120 work for convergence
monitor. The output messages are very useful for extremely large-scaled calculations. [output, ok]
libdvd90, dvdson.f90: function TSTSEL is modified, so that dvdson performs at least TWO iteration calculations. Or, error results could be obtained in some cases (in O-like isoelectronic sequence?). [check, cff]
The rscfgenerate bug that was discussed during the spring (issue bug in the generation of csf's using rcsfgenerate #26) [already merged through separate PR]
And finally the MPI versions of DDOT and DGEMV - leading to a parallell version of DVDSON

I need help with testing these changes out properly before we add them to master - any of the more senior members perhaps? @tspejo @gaigalas @cffischer or others?

No 1 (jj2lsj) [rejected], 4 (rangular) and 6-7 (dvdson) are the critical ones of the first group.
Then we have the rcsfgenerate fix in no 8, and the larger update with new parallell versions of DDOT and DGEMV in No 9 which will require some testing.

@gaigalas - could you perhaps verify that update no 1 in jj2lsj on both parities is ok? [done] And maybe also the other updates you feel you can verify easily.
@cffischer - could you have a look at update 7 perhaps?

mortenpi

@CYChenFudan if you'd like, I think me and/or @jongrumer can also make the necessary changes for that, since they're pretty straightforward and do not affect the core PR.

mortenpi · 2020-09-23T22:33:15Z

src/appl/rangular90_mpi/getinf.f90

@@ -60,8 +60,9 @@ SUBROUTINE GETINF
            do i = 1,nblock
              write(istde,*) 'Give ICCUT for block',i
    1         READ *, ICCUT(i)
-              IF ((ICCUT(i) <= 1).OR.(ICCUT(i) >= ncfblk(i))) THEN
-                WRITE (istde,*) 'GETINF: ICCUT must be greater than 1',&
+!cychen              IF ((ICCUT(i) <= 1).OR.(ICCUT(i) >= ncfblk(i))) THEN


I would suggest removing these commented out lines with the old code before merging.

mortenpi · 2020-09-23T22:35:48Z

make_environment_gfortran

@@ -26,4 +33,4 @@ export   FC_MPI="mpifort"                                      # MPI
 export   FC_MPIFLAGS="${FC_FLAGS}"                             # Parallel code compiler flags
 export   FC_MPILD=${FC_LD}                                     # Serial linker flags
 # -------------------------------------------------------------------------------------------------------------------
-export   MPI_TMP="${HOME}/grasp_mpi_tmp"                       # Location for temporary files
+export   MPI_TMP="${HOME}/tmp"                       # Location for temporary files


The PR needs to be brought up to speed with the latest master before merging. E.g. make_environment_gfortran is not a thing anymore. I don't think any of the substantial parts of the PR have conflicts though, just the build files.

Yes, and the changes with copying blas and lapack libraries from a given folder are way too case specific. Better to set e.g. the LD_LIBRARY_PATH flag or let CMake do the job.

CYChenFudan · 2020-09-24T08:18:47Z

Dear Morten, Of course, please make the necessary changes. We are somewhat busy in these days. But Ran and Kai, together with the students in our group could join the codes testing program. Please tell us if any modifications should be tested (Though the changes suggested have been extensively tested in F77-version). With best wishes, Chongyang

jongrumer · 2020-09-24T09:48:00Z

@CYChenFudan Me and @mortenpi realized that you have submitted this pull request from your forked master branch of grasp. This is not ideal perhaps since any edits we will do now on the patch, will also directly affect your personal main version of the code.

For this time around to get this done as soon as possible, is it ok with you that we go ahead and do the necessary edits and deletes on your master branch? We will try to do as few edits/deletes as possible.

For the next time, first create a separate dev branch on your fork that you then submit a pull request from. We will probably squash-merge this PR, which will create a single commit on master that is different from the numerous ones that exist on your forked master branch. This means that from Git's point of view, the CompAS master and you master will diverge once we merge. So you will have to reset you master to the actual master from that on. For this reason, usually, pull requests are preferably made from separate branches to avoid that.

Oh, and try to reply on the github online chat instead of by email to make the thread less cluttered: here #45

CYChenFudan · 2020-09-24T13:50:03Z

@jongrumer @mortenpi
It is OK that you go ahead and do the necessary edits and deletes on my master branch.

jongrumer · 2020-09-24T14:52:00Z

@CYChenFudan Great, we'll get started with the stuff we actually can deal with now then, like sorting out the conflicting files. We need to wait for input from the other CompAS members on the larger changes like the one related to DVDSON.

gaigalas · 2020-09-30T12:03:37Z

@gaigalas - could you perhaps verify that update no 1 in jj2lsj on both parities is ok? And maybe also the other updates you feel you can verify easily.
This is not good idea for many reasons. So I am not planning to check what is not efficient.

jongrumer · 2020-09-30T14:31:10Z

@gaigalas Ok great - I trust your judgement of course, so we shall remove the edits to jj2lsj (part 1 in the list above) from this patch. Let us know if you have opinions on any of the other edits. But I think we can handle them now, this was the most critical to get info on. Thanks a lot!

gaigalas · 2020-09-30T14:41:57Z

OK. I will look other issue, too. But, there are the files *.f90 for 2, 3, 5 ... ?

jongrumer · 2020-09-30T15:02:30Z

@gaigalas Great thanks GG 👍 I think I have most of them under control, I updated the list above now with "status comments", but if you could have a look at the edits to no 7 libdvd90, dvdson.f90: function TSTSEL and if possible also the MPI-libraries in no 9, that would be very helpful. My feeling is that we can merge 7 but have to wait with 9 until tested more.

You can see the changes to the files directly here on github quite easily by clicking on the "Files changed" tab up on the top of this page. This takes you here: https://github.com/compas/grasp/pull/45/files where you can scroll through the files. The files related to dvdson and mpi-blas are a bit down. Red lines are those that are removed, and green the new ones. If you have any direct comments on a certain line of code, you can press the blue "plus"-sign in the source code. This opens upp a message box where you can write your comment related to that particular line.

If you want to have a look at the files locally on your own computer, you have to download Chens version of the codes, either by pressing the green button "Code" --> "Download ZIP" at his page https://github.com/CYChenFudan/grasp or by cloning the repository directly to some folder at your computer with:

git clone https://github.com/CYChenFudan/grasp.git

Let me know if you have any further questions, and thanks again for the help!

Revert changes, this is move to a separate PR (jg/chen_rcsfgenerate) to get this into master as soon as possible.

jongrumer · 2020-10-06T12:59:05Z

I've now split out the rcsfgenerate fix to a separate PR (#55) that I will merge now directly since this is the most crucial fix to get into master as soon as possible as far as I can tell.

I will edit my first summarizing comment accordingly.

…elized.

…tcomment-46147246)

CYChenFudan added 3 commits June 2, 2020 17:47

Fix the bug in the generation of csf's using rcsfgenerate

4b0aea0

DDOTMPI and DGEMVMPI added in mpiu.f90; Then, DVDSON calls them

f7968ac

This was referenced Sep 23, 2020

bug in the generation of csf's using rcsfgenerate #26

Open

jj2lsj's parity limitation #17

Open

mortenpi reviewed Sep 23, 2020

View reviewed changes

This was linked to issues Sep 30, 2020

bug in the generation of csf's using rcsfgenerate #26

Open

jj2lsj's parity limitation #17

Open

Update matain.f90

e8efff4

Revert changes, this is move to a separate PR (jg/chen_rcsfgenerate) to get this into master as soon as possible.

jongrumer mentioned this pull request Oct 6, 2020

rcsfgenerate -- bug fix by C. Y. Chen #55

Merged

CYChenFudan added 5 commits January 16, 2021 07:06

Modifcations for searching the targeted eigenpairs one by one

76eb371

Minor modifications for rmcdhf90/maneig.f90, according to those for gdvd

14ba948

Minor modification for searching the target eigenpairs one by one.

ddad394

Comment off the serial module rci90 and rmcdhf90, as dvdson is parall…

892c594

…elized.

Modifications for Librangular code as suggested by G.G (76eb371#commi…

7e2a0ab

…tcomment-46147246)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DDotMPI and DGEMVMPI implementation #45

DDotMPI and DGEMVMPI implementation #45

CYChenFudan commented Jun 5, 2020

jongrumer commented Sep 22, 2020 •

edited

Loading

mortenpi left a comment

mortenpi Sep 23, 2020

mortenpi Sep 23, 2020

jongrumer Sep 30, 2020 •

edited

Loading

CYChenFudan commented Sep 24, 2020 via email •

edited by jongrumer

Loading

jongrumer commented Sep 24, 2020

CYChenFudan commented Sep 24, 2020

jongrumer commented Sep 24, 2020

gaigalas commented Sep 30, 2020

jongrumer commented Sep 30, 2020 •

edited

Loading

gaigalas commented Sep 30, 2020

jongrumer commented Sep 30, 2020 •

edited

Loading

jongrumer commented Oct 6, 2020

DDotMPI and DGEMVMPI implementation #45

Are you sure you want to change the base?

DDotMPI and DGEMVMPI implementation #45

Conversation

CYChenFudan commented Jun 5, 2020

jongrumer commented Sep 22, 2020 • edited Loading

mortenpi left a comment

Choose a reason for hiding this comment

mortenpi Sep 23, 2020

Choose a reason for hiding this comment

mortenpi Sep 23, 2020

Choose a reason for hiding this comment

jongrumer Sep 30, 2020 • edited Loading

Choose a reason for hiding this comment

CYChenFudan commented Sep 24, 2020 via email • edited by jongrumer Loading

jongrumer commented Sep 24, 2020

CYChenFudan commented Sep 24, 2020

jongrumer commented Sep 24, 2020

gaigalas commented Sep 30, 2020

jongrumer commented Sep 30, 2020 • edited Loading

gaigalas commented Sep 30, 2020

jongrumer commented Sep 30, 2020 • edited Loading

jongrumer commented Oct 6, 2020

jongrumer commented Sep 22, 2020 •

edited

Loading

jongrumer Sep 30, 2020 •

edited

Loading

CYChenFudan commented Sep 24, 2020 via email •

edited by jongrumer

Loading

jongrumer commented Sep 30, 2020 •

edited

Loading

jongrumer commented Sep 30, 2020 •

edited

Loading