ScalES-PPM
Loading...
Searching...
No Matches
dist_array_c.c File Reference

Distributed data structure of multiple global arrays. More...

#include <assert.h>
#include <limits.h>
#include <math.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
#include <mpi.h>
#include "core/core.h"
#include "core/ppm_extents.h"
#include "core/ppm_extents_mp.h"
#include "core/ppm_xfuncs.h"
#include "ppm/dist_array.h"

Enumerations

enum  { exposed_flag = 1 , transfer_mode_mask = 1 << 1 }
 

Functions

struct PPM_dist_mult_array * PPM_dist_mult_array_new (size_t num_sub_arrays, const struct PPM_global_array_desc *sub_arrays, const struct PPM_extent *local_chunk, MPI_Comm comm, size_t cache_size, enum PPM_dma_sync_mode sync_mode)
 create distributed multi-array data structure
 
struct PPM_dist_mult_array * PPM_dist_mult_array_copy (struct PPM_dist_mult_array *dm_array)
 destroy dist_mult_array data type
 
void PPM_dist_mult_array_expose (struct PPM_dist_mult_array *dm_array)
 make local data available to other ranks
 
void PPM_dist_mult_array_unexpose (struct PPM_dist_mult_array *dm_array)
 wait for all ranks to finish queries in current exposure epoch
 
void PPM_dist_mult_array_delete (struct PPM_dist_mult_array *dm_array)
 destroy dist_mult_array data type
 
void * PPM_dist_mult_array_local_ptr (struct PPM_dist_mult_array *dm_array, size_t sub_array_idx)
 
void PPM_dist_mult_array_rma_sync (struct PPM_dist_mult_array *dm_array)
 synchronize RMA updates only, ignore local updates
 
int PPM_dist_mult_array_get_sync_mode (struct PPM_dist_mult_array *dm_array)
 query distributed array synchronization protocol
 
void PPM_dist_mult_array_set_sync_mode (struct PPM_dist_mult_array *dm_array, enum PPM_dma_sync_mode sync_mode, size_t cache_size)
 switch distributed array to different synchronization protocol
 
void PPM_dist_mult_array_get (struct PPM_dist_mult_array *dm_array, size_t sub_array, int32_t coord[], void *v_out)
 Get value out of distributed multi-array, independent of rank the data resides on.
 
unsigned PPM_dist_mult_array_a_rank (struct PPM_dist_mult_array *dm_array, size_t sub_array_idx)
 
void PPM_dist_mult_array_rank_rect (struct PPM_dist_mult_array *dm_array, size_t sub_array_idx, int rank, struct PPM_extent *rect)
 
MPI_Comm PPM_dist_mult_array_comm (struct PPM_dist_mult_array *dm_array)
 
void PPM_dist_mult_array_set_transfer_mode (struct PPM_dist_mult_array *dm_array, int mode)
 
int PPM_dist_mult_array_get_transfer_mode (struct PPM_dist_mult_array *dm_array)
 

Detailed Description

Distributed data structure of multiple global arrays.

Version
1.0
Author
Thomas Jahns jahns.nosp@m.@dkr.nosp@m.z.de Moritz Hanke hanke.nosp@m.@dkr.nosp@m.z.de

Function Documentation

◆ PPM_dist_mult_array_a_rank()

unsigned PPM_dist_mult_array_a_rank ( struct PPM_dist_mult_array * dm_array,
size_t sub_array_idx )

Query array rank for sub-array of distributed multi-array.

Parameters
[in]dm_arraydistributed multiple arrays to query
sub_array_idxindex of sub-array to query
Returns
array rank in 1 to PPM_dma_max_rank

◆ PPM_dist_mult_array_copy()

struct PPM_dist_mult_array * PPM_dist_mult_array_copy ( struct PPM_dist_mult_array * dm_array)

destroy dist_mult_array data type

Remarks
since an MPI window is freed, this routine is collective for all processes which participated in the corresponding call to PPM_dist_mult_array_new
Parameters
[in]dm_arraydistributed multi-array to delete

◆ PPM_dist_mult_array_delete()

void PPM_dist_mult_array_delete ( struct PPM_dist_mult_array * dm_array)

destroy dist_mult_array data type

Remarks
since an MPI window is freed, this routine is collective for all processes which participated in the corresponding call to PPM_dist_mult_array_new
Parameters
[in,out]dm_arraydistributed multi-array to delete

◆ PPM_dist_mult_array_expose()

void PPM_dist_mult_array_expose ( struct PPM_dist_mult_array * dm_array)

make local data available to other ranks

This call is collective for all ranks in the communicator used to create dm_array note: this enters an exposure epoch on the data shared via RMA local data must not be changed while the array is in exposed state

◆ PPM_dist_mult_array_get()

void PPM_dist_mult_array_get ( struct PPM_dist_mult_array * dm_array,
size_t sub_array,
int32_t coord[],
void * v_out )

Get value out of distributed multi-array, independent of rank the data resides on.

For example, to query the data assigned in the example for dist_mult_array_local_ptr, use a query like this:

struct PPMdist_mult_array *dma;
int i5;
PPM_dist_mult_array_get(dma, 5, (/ i_l, j_l /), i5)
void PPM_dist_mult_array_get(struct PPM_dist_mult_array *dm_array, size_t sub_array, int32_t coord[], void *v_out)
Get value out of distributed multi-array, independent of rank the data resides on.
Definition dist_array_c.c:957
void PPM_dist_mult_array_expose(struct PPM_dist_mult_array *dm_array)
make local data available to other ranks
Definition dist_array_c.c:670

In case dma was initialized with sync_mode=sync_mode_active_target, the value of i5 must not be accessed before calling a synchronizing routine, either

void PPM_dist_mult_array_unexpose(struct PPM_dist_mult_array *dm_array)
wait for all ranks to finish queries in current exposure epoch
Definition dist_array_c.c:688

or

void PPM_dist_mult_array_rma_sync(struct PPM_dist_mult_array *dm_array)
synchronize RMA updates only, ignore local updates
Definition dist_array_c.c:706

◆ PPM_dist_mult_array_get_sync_mode()

int PPM_dist_mult_array_get_sync_mode ( struct PPM_dist_mult_array * dm_array)

query distributed array synchronization protocol

Parameters
dm_arraydistributed multiple arrays to query sync mode for
Returns
sync mode set during initialization or the last call to PPM_dist_mult_array_set_sync_mode for dm_array

◆ PPM_dist_mult_array_get_transfer_mode()

int PPM_dist_mult_array_get_transfer_mode ( struct PPM_dist_mult_array * dm_array)

get mode to transfer data for a distributed multi-array using synchronization mode PPM_dma_sync_mode_passive_target.

Parameters
[in]dm_arraydistributed multiple arrays to get mode for
Returns
currently active mode

◆ PPM_dist_mult_array_new()

struct PPM_dist_mult_array * PPM_dist_mult_array_new ( size_t num_sub_arrays,
const struct PPM_global_array_desc * sub_arrays,
const struct PPM_extent * local_chunk,
MPI_Comm comm,
size_t cache_size,
enum PPM_dma_sync_mode sync_mode )

create distributed multi-array data structure

The resulting data type represents a number of arrays distributed over the ranks of the communicator passed to this function.

Parameters
num_sub_arraysnumber of distributed arrays
sub_arrayscontains array ranks, data types and bounds of each distributed global array. The bounds are represented by an extent type that stores start and size.
local_chunkpoints to array of extents with dimensions [num_sub_arrays][MAX(sub_arrays[].a_rank)] where local_chunk[j][i] describes for dimension i of sub_array j of the global arrays the local part available on this MPI rank.
Only contiguous local parts are possible.
commMPI communicator for which this data structure is collectively created
cache_sizenumber of ranks to cache remote local parts of or automatically determined if 0
sync_modeswitch synchronization modes, either PPM_sync_mode_passive_target or sync_mode_active_target_mode. For sync_mode==PPM_dma_sync_mode_active_target_mode, execution of RMA is deferred until the next synchronizing call (dist_mult_array_unexpose or dist_mult_array_rma_sync). (see PPM_dist_mult_array_get for example)
Returns
initialized dist_mult_array structure
Remarks
This operation is collective for all MPI ranks in comm

◆ PPM_dist_mult_array_rank_rect()

void PPM_dist_mult_array_rank_rect ( struct PPM_dist_mult_array * dm_array,
size_t sub_array_idx,
int rank,
struct PPM_extent * rect )

Query bounds of sub-array in distributed multi-array.

Parameters
[in]dm_arraydistributed multiple arrays to query
sub_array_idxindex of sub-array to query
rankMPI rank on communicator of dm_array to query chunk for, own rank if less than 0
[out]rectthis array of size PPM_dist_mult_array_a_rank(dm_array, sub_array_idx) is set to the local chunk size of rank

◆ PPM_dist_mult_array_rma_sync()

void PPM_dist_mult_array_rma_sync ( struct PPM_dist_mult_array * dm_array)

synchronize RMA updates only, ignore local updates

This call is collective for all ranks in the communicator used to create dm_array. Also dm_array must be in exposed state.

Parameters
dm_arraydistributed multiple arrays to synchronize all RMA for

◆ PPM_dist_mult_array_set_sync_mode()

void PPM_dist_mult_array_set_sync_mode ( struct PPM_dist_mult_array * dm_array,
enum PPM_dma_sync_mode sync_mode,
size_t cache_size )

switch distributed array to different synchronization protocol

This call is collective for all ranks in the communicator used to create dm_array

note: this ends an exposure epoch on the data shared via RMA note: in case the current sync mode is PPM_dma_sync_mode_passive_target, the cache_size can still be changed without actually switching the mode

Parameters
dm_arraydistributed multiple arrays to change access/sync mode for
sync_modenew sync mode to set
cache_sizein case sync_mode is PPM_dma_sync_mode_passive_target, the number of ranks to cache data for is set to this value (or determined automatically if 0)

◆ PPM_dist_mult_array_set_transfer_mode()

void PPM_dist_mult_array_set_transfer_mode ( struct PPM_dist_mult_array * dm_array,
int mode )

set mode to transfer data for a distributed multi-array using synchronization mode PPM_dma_sync_mode_passive_target.

Parameters
[in,out]dm_arraydistributed multiple arrays to set mode for
[in]modetransfer data as struct or contiguous array of bytes (the latter usually only works for MPI implementations on homogeneous systems which don't feature different endianness or alignments but is also often faster and this type of platform is also the most common).

◆ PPM_dist_mult_array_unexpose()

void PPM_dist_mult_array_unexpose ( struct PPM_dist_mult_array * dm_array)

wait for all ranks to finish queries in current exposure epoch

This call is collective for all ranks in the communicator used to create dm_array

note: local data can only be changed after the distributed array was initially created or dist_mult_array_unexpose has been called

Das diesem Bericht zugrundeliegende Vorhaben wurde mit Mitteln des Bundesministeriums für Bildung, und Forschung unter dem Förderkennzeichen 01IH08004E gefördert. Die Verantwortung für den Inhalt dieser Veröffentlichung liegt beim Autor.