6 Distributed array creating and deleting

6.1 Creating distributed array

The creation of a distributed array using function crtda_ means only an initialization of the internal system structures describing the array. The memory for the distributed array elements is allocated only after mapping this array on an abstract machine by the function align_. (In another words, memory is allocated after evaluating for each processor abstract machine it is mapped onto, a local part of distributed array according to its distribution, see section 7.1). An access to array elements is performed through Run-Time System functions (see section 13).

A header of the created array is an array of *RankPtr+1 (if *ExtHdrSignPtr = 0) or *RankPtr*2+2 (if *ExtHdrSignPtr = 1) elements of "long" type. The user program has to allocate this header in memory (whether statically or dynamically). Run-Time System only initializes this header. The main purpose of this header is to provide an effective access to the elements of the local part of the distributed array.

The base pointer BasePtr has to point to any variable of the same type as that of distributed array element. The value of this variable is not essential because the base pointer is only used to access to the elements from the local part of the distributed array.

The structure and usage of distributed array header (jointly with base pointer) to access to the array local part are considered in section 13.4.

If the flag *StaticSignPtr of the static array creation is not equal to zero, then the created distributed array is not deleted automatically when the control exits from the current program block (see section 8). Such array has to be deleted explicitly using the function delda_.

The parameter *ReDistrParPtr may have the following values:

The number of the parameters defined in the arrays LowShdWidthArray and HiShdWidthArray (that is the width of low and high shadow edges) has to be equal to the created array rank. See section 12 for more detail description of the shadow edges of the distributed array local parts. Note, that the shadow edge width can be equal to zero.

The function returns zero.

6.2 Deleting distributed array

long delda_(long ArrayHeader[]);

ArrayHeader – the header of the array to be deleted.

The function deletes the distributed array created by the function crtda_. After array deleting the user program can use distributed array header for any goal.

The distributed array can be deleted by delda_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10). All edge groups, in which the array was included, (see section 12) will be deleted also.

A distributed array can be deleted also by the function delobj_ (see section 17.5).

The function returns zero.

6.3 Creating additional header of distributed array

The function creates the distributed array header, which further will be used equitably with existing headers. In particular it can be used to create other additional headers.

When the distributed array is aligned by functions align_ and malign_ (see section 7) all its headers are initialized. The array realigning by the functions realn_, mrealn_, redis_ and mredis_ (see sections 5 and 7) also causes all its headers updating.

The function returns zero.

6.4 Deleting distributed array header

long delhdr_ (long ArrayHeader[]);

ArrayHeader - header to be deleted.

The function deletes the header, created by addhdr_ function or specified in crtda_ funcôion call. The only header of distributed array can't be deleted.

The function returns zero.

7 Mapping distributed array

7.1 Aligning distributed array

The function align_ defines the location of the specified distributed array in the assigned abstract machine representation space (in other words the function aligns the distributed array). Besides, the elements of the array are distributed among the child abstract machines incorporating in the representation. The abstract machine representation is defined by means of the pattern, which can be as the representation itself, or a distributed array (this is called "indirect mapping"). In the last case the reference to the pattern is a first word of distributed array header.

Using the distributed array as a pattern is allowed only if this array has already been mapped on an abstract machine representation. Moreover, using an abstract machine representation as a pattern is allowed only if this representation has been mapped on a processor system. So the function align_ only distributes elements of the aligned distributed array among processors from this processor system.

Aligning pattern can be mapped (as final result) on the current processor system or its subsystem only.

It is not allowed to map already mapped distributed array.

Consider distributed array alignment in detail. Let F be a multifunctional with the domain of definition in a space of indexes of the aligned array and with the image in the space of indexes of the pattern:

where:

An alignment of the distributed array with the pattern is a defining the F multifunction. This defining designates that an element (I₁, ... , I_n) of an aligned array has to be located on the processor if and only if some pattern elements from F((I₁, ... , I_n)) set are mapped on this processor. The set F((I₁, ... , I_n)) is an image-set, and components of this set are vectors of space of indexes of the pattern.

The F₁, ... , F_m functions are named "coordinate alignment rules". Run-Time System provides the following set of alignment rules.

1. F_j(I₁, ... , I_n) = {A_j*I_k + B_j} , where:

This alignment rule means that for each element (I₁, … , I_n) of index space of the aligned array the corresponding image-set contains one element – A_j*I_k + B_j. This element belongs to the value range of the index variable of the pattern j-th dimension.

Note, that A_j and B_j have to meet the following requirements:

0 £ B_j £ MAX_j; and 0 £ A_j*MAX_k + B_j £ MAX_j , where:

2. F_j(I₁, ... , I_n) = {q Î M_j: 0 £ q £ MAX_j} , where:

This mapping rule means that for each element (I₁, … , I_n) of index space of the aligned array a corresponding image-set, consists of whole range of values of index variable of the pattern j-th dimension. In such a case, the symbol "*" ("any of the admissible") is usually used.

Examples.

1. Alignment rule F( (I₁,I₂) ) = {I₁} ´ {I₂} means that element of two-dimensional array has to be mapped onto any processor, an element of two-dimensional pattern is mapped on, if the indexes of both elements are equal.
2. Alignment rule F( (I₁,I₂) ) = {*} ´ {I₁+5} ´ {*} means that the element of two-dimensional array has to be mapped onto any processor, an element of three-dimensional pattern is mapped on, if the index of the pattern second dimension is equal to the index of the aligned array first dimension plus 5. The index of the aligned array second dimension and the indexes of the pattern first and third dimension are not considered.
3. Alignment rule F( (I₁,I₂,I₃) ) = {*} ´ {*} means, that each element of three-dimensional array has to be mapped onto each processor, any element of two-dimension pattern is mapped on. The indexes of the aligned array and of the pattern are not considered.
4. Alignment rule F( (I₁,I₂) ) = {0} ´ {1} ´ {2} means, that an element of two-dimensional array has to be mapped onto each processor, the element (0,1,2) of three-dimensional pattern is mapped on. The indexes of aligned array are not considered.

Defining the aligned array distribution among pattern space (that is the defining F₁, ... , F_m functions) it is necessary to take into account that all elements of the distributed array have to be inside the pattern space. Observance of correct mapping on each stage of a chain of alignments guarantees the correct final distribution of array elements among the processors.

Note that if an array is used as a pattern, then after realignment of this array by the function realn_ (see section 7.3), the array aligned with this pattern keeps its location.

When the function align_ is called the parameters of the alignment rule F_j(I₁, ... , I_n)= {A_j*I_k + B_j} for the j-th pattern dimension have to be defined as follows:

To define alignment rule F_j(I₁, ... , I_n) with image in set of all values of the index variable of j-th dimension of template for any I₁, ... , I_n, the value AxisArray[j-1] (k value) has to be set to -1. The values CoeffArray[j-1] and ConstArray[j-1] are irrelevant in this case.

The number of the alignment rules has to be equal to the rank of the pattern, when the function align_ is called.

The function returns non-zero value, if mapped array has a local part on the current processor, and zero otherwise.

Note 1. The distributed array mapping, specified for function align_, may be implemented only in the case, if for all the processors, at least one array element is mapped onto, a local size of each array dimension, obtained as a result of mapping, is not less than widths of high and low shadow edges of the dimension.

Note 2. If DisArrayFill parameter of Run-Time system startup is non-zero (sysdebug.* files of base parameter sets), mapped distributed array will be cleared by "zero" byte, i.e. by the byte with all zero bits). If DisArrayFill parameter is zero, the mapped array contents is undefined.

7.2 Alignments superposition

Let the function G defines the location of the n-dimensional array in the space of the r-dimensional pattern; and let the function P defines the location of the r-dimensional array in the space of the m-dimensional pattern:

Then the result of alignment P (by function align_) of r-dimensional array PatternDA in m-dimensional pattern Pattern space; and of next alignment G of n-dimensional array DA in r-dimensional array PatternDA space is alignment F of n-dimensional array DA in m-dimensional pattern Pattern space. F is a product (superposition) of P and G:

The coordinate alignment rules F_s (0 £ s £ m) from F mapping are a superposition of the coordinate alignment rules for P and G mappings:

1. Let P_s((J₁, ... , J_r)) = { q Î M_s: 0 £ q £ MAX_s } , where:

Then F_s(I₁, ... , I_n) ) = M_s, that is the full replication of the array DA along the s-th dimension of the pattern Pattern.

where:

Then F_s((I₁, ... , I_n)) = A_s*I_l + B_s , where A_s = C_s*Q_k and B_s = C_s*R_k + D_s.

where:

Then

that is the partial replication (stretched by the linear function P_s) of the array DA along the s-th dimension of the pattern Pattern.

The described alignment F is equivalent to the sequence of the alignments P and G. Consider the case, when the pattern Pattern is not an abstract machine representation. Then it is aligned by some function Q with another pattern. In this case, the alignment functions F and Q may be substituted by their superposition Q*F. So, the resulting alignment of the array DA with the abstract machine representation can be built substituting sequential alignment functions by their superposition (Run-Time System does it to align the array DA by the pattern PatternDA when the function align_ is called).

7.3 Realigning distributed array

The function realn_ cancels the allocation of the distributed array with header *ArrayHeaderPtr, defined previously by the function align_(or malign_, see section 7.5). The function sets a new allocation for this array according to the parameters AxisArray, CoefArray and ConstArray. If this array has been used as pattern of alignment of some arrays, nevertheless the allocations of these arrays are not changed.

As remapped distributed array, as alignment pattern must be mapped on the current processor system or its subsystem (perhaps, each one on its own).

The contents of redistributed arrays, depending on *NewSignPtr value, will be

• kept (if *NewSignPtr = 0);
• undefined (if *NewSignPtr = 1);
• cleared by "zero" byte, i.e. by the byte with all zero bits (if *NewSignPtr = 2).

If NewSignPtr = 1 redistributed array will be nevertheless cleared by "zero" byte, if Run-Time system was started with non-zero DisArrayFill parameter (sysdebug.* files of base parameter sets).

When the array is remapped, the values of its shadow edges (see section 12) are not kept.

The array can be realigned by function realn_ only, if it was created by the function crtda_ with parameter *ReDistrParPtr equal to 2 or 3 (see section 6).

The distributed array, specified in the function call, can be not mapped earlier. In this case realn_ function is executed as align_ function, but if NewSignPtr = 2 firstly distributed array will be cleared by "zero" byte. *ReDistrParPtr parameter value of the distributed array creation is irrelevant.

The function returns non-zero, if realigned array has a local part on the current processor, and zero in other case.

7.4 Requesting map

The function arrmap_ creates an object (a map), describing current mapping of the distributed array onto the processor system representation, and returns a reference to the created object.

The map contains in particular the following information:

• a reference to the abstract machine representation that distributed array is mapped on;
• aligning rules for all dimensions of the representation;
• the ranks of the distributed array and the abstract machine representation (for checking).

If flag of static map *StaticSignPtr is not equal to zero, then the map will not be deleted when control exits a program block (see section 8). Such map can be deleted only explicitly by function delarm_ considered in section 7.7.

7.5 Specifying distributed array mapping according to map

The function malign_ performs distributed array mapping onto abstract machine representation according to map.

The ranks of the distributed array and the abstract machine representation must be equal to corresponding ranks in map.

If AMViewRefPtr is equal to NULL or *AMViewRefPtr is equal to zero, then abstract machine representation, whose reference is in the map, will be used as alignment pattern.

Before malign_ function call the abstract machine representation (specified by *AMViewRefPtr reference or in the map) must be mapped on the current processor system or its subsystem.

It is not allowed to map already mapped distributed array.

The function returns non-zero, if mapped array has a local part on the current processor, and zero in the other case.

Note. If DisArrayFill parameter of Run-Time system startup is non-zero (sysdebug.* files of base parameter sets), mapped distributed array will be cleared by "zero" byte, i.e. by the byte with all zero bits). If DisArrayFill parameter is zero, the contents of mapped array is undefined.

7.6 Realigning distributed array according to map

Function mrealn_ cancels allocation defined earlier by the function. malign_ (or align_ , see section 7.1) for the distributed array with header *ArrayHeaderPtr, and defines for the array new allocation according to specified map. The allocation of all distributed arrays, for whose alignment considered array was used as pattern, will not be changed.

The ranks of the distributed array and the abstract machine representation must be equal to corresponding ranks in the map.

If AMViewRefPtr is NULL or *AMViewRefPtr is zero, then abstract machine representation, whose reference is in the map, will be used as alignment pattern.

As remapped distributed array, as the abstract machine representation (specified by *AMViewRefPtr reference or in the map) must be mapped on the current processor subsystem or its subsystem (perhaps, each one on its own) when the function is called.

The contents of redistributed arrays, depending on *NewSignPtr value, will be

• kept (if *NewSignPtr = 0);
• undefined (if *NewSignPtr = 1);
• cleared by "zero" byte, i.e. by the byte with all zero bits (if *NewSignPtr = 2).

If NewSignPtr = 1 redistributed array will be nevertheless cleared by "zero" byte, if Run-Time system was started with non-zero DisArrayFill parameter (sysdebug.* files of base parameter sets).

When the array is remapped, the contents of its shadow edges (see section 12) are not kept.

The array can be realigned by function mrealn_ only in the case, if it was created by the function crtda_ with parameter *ReDistrParPtr equal to 2 or 3 (see section 6).

The distributed array, specified in the function call, can be not mapped earlier. In this case mrealn_ function is executed as malign_ function, but if NewSignPtr = 2 firstly distributed array will be cleared by "zero" byte. *ReDistrParPtr parameter value of the distributed array creation is irrelevant.

The function returns non-zero, if remapped array has a local part on the current processor, and zero in other case.

7.7 Deleting map

long delarm_ (ArrayMapRef *ArrayMapRefPtr);

*ArrayMapRefPtr - reference to the map of the distributed array.

The function deletes the map of the distributed array created by function arrmap_. After deleting the map the reference can be used by user program for its own goals.

The map can be deleted by delarm_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10).

To delete map the function delobj_ can also be used (see section 17.5).

The function returns zero.

Note. One of the possible usages of the functions, aligning the distributed arrays according to the map is keeping and restoring the array allocation in procedures, dealing with external distributed arrays with their redistribution.

Note, that at the moment of using the map for mapping and remapping arrays, the array, being source for the map, may not exist.

7.8 Requesting reference to abstract machine representation, specified distributed array is mapped on

AMViewRef getamv_ (long ArrayHeader[]);

ArrayHeader - distributed array header.

The function getamv_ returns the reference to abstract machine representation, the distributed array, specified by ArrayHeader header, was aligned with. NULL value of returned reference means that given array is not mapped.