SourceForge

Pre-Fetching in MPI

If you define NO_PERMANENT_GUARDCELLS in PARAMESH_PREPROCESSOR.FH then you need to read this page. If not you can skip it since all these details will be handled automatically for you.

When messaging is done with MPI, we assume that the machine meassage startup costs and bandwidth are such that we need to pack as many messages as we can into as few messages as possible. To do this we pre-fetch data.

Suppose we have a section of code which will loop over all the local grid blocks and update their solution in some fashion, and let us also suppose that this update will require that the block guarcells are filled. Therefore we will have a code section that looks like this



        do lb = 1,lnblocks

           .
           .
           .

           call amr_1blk_guardcell(......,lb,.....)

           .
           .
           .

         enddo

To minimize communication costs, we communicate all the off-processor information needed by all local blocks for guardcell filling by calling the routine MPI_AMR_COMM_SETUP before entering the loop over LB.



        lcc       = .true.
        lfc       = .false.
        lec       = .false.
        lnc       = .false.

        lguard    = .true.
        lprolong  = .false.
        lflux     = .false.
        ledge     = .false.
        lrestrict = .false.

        tag_offset = 100

        call mpi_amr_comm_setup(mype,nprocs,
     .                          lguard,lprolong,
     .                          lflux,ledge,lrestrict,
     .                          iopt,lcc,lfc,lec,lnc,tag_offset)


        do lb = 1,lnblocks

           .
           .
           .

           call amr_1blk_guardcell(......,lb,.....)

           .
           .
           .

         enddo

MPI_AMR_COMM_SETUP is a general pre-fetching routine which supports the four basic communication dependent tasks in amr, guardcell filling, prolongation, restriction and conservation support. Any time we modify the grid, either by calling AMR_REFINE_DEREFINE or when restarting a run by reading a checkpoint file, we analyse the control information needed to manage these different communication tasks. This control information is stored, and then retrieved by MPI_AMR_COMM_SETUP when it is needed. The arguments to MPI_AMR_COMM_SETUP can be used to select the appropriate control information for each type of communication operation.

In the example above, the logical LGUARD is set to be true and so data is prefetched to support guardcell filling. Also LCC is true so UNKtype data is pre-fetched.

The arguments LPROLONG, LFLUX, LEDGE, LRESTRICT configure pre-fetching for prolongation, for conservation control, for circulation integral management and restriction respectively. However, it is not expected that the user should need to call MPI_AMR_COMM_SETUP for anything other than guardcell filling.

The arguments LFC, LEC, LNC select cell-face-centrered, cell-edge-centered and cell-corner data respectively. You can choose any or all of these data types to be true.

The argument TAG_OFFSET is used to set initial MPI message tags. It can have any non-negative value.

Return to Main Page.