`exarl.network.data_structures`

Module Contents

Classes

`ExaMPIConstant`	This class is built to maintain a single value using mpi rdma.
`ExaMPIBuffUnchecked`	This class is creates an RMA buffer of a fixed size on each rank.
`ExaMPIBuffChecked`	This class is creates an RMA buffer of a fixed size on each rank.
`ExaMPIDistributedQueue`	This class creates a circular buffer in an RMA window across nodes in a communicator.
`ExaMPIDistributedStack`	This class creates a stack in an RMA window across nodes in a communicator.
`ExaMPICentralizedStack`	This class creates a stack in RMA windows across nodes in a communicator.
`ExaMPICentralizedQueue`	This class creates circular buffers in RMA windows across nodes in a communicator.

Attributes

MPI

exarl.network.data_structures.MPI

class exarl.network.data_structures.ExaMPIConstant(comm, rank_mask, the_type, name=None)

This class is built to maintain a single value using mpi rdma. Each rank will have a window the size of the type.

Variables

comm (mpi4py.MPI.Comm) – raw MPI communicator
npType (type) – numpy type of constant
mpiType (type) – mpi type of the constant
rank (int) – rank that hosts the data
win (MPI.win) – MPI window for constant
sum (int) – internal constant numpy 1 for incrementing
buff (numpy array) – internal numpy buffer used for RMA ops
name (string) – name of the constant for debugging

Parameters

comm (mpi4py.MPI.Comm) – Communicator for all ranks involved
rank_mask (int, optional) – host of the window
the_type (int, optional) – python type (int, float)
name (string, optional) – name of constant for debbuging

put(self, value, rank)

Places a constant on a given rank

Parameters

value (int) – Number to send to all ranks
rank (integer) – Host rank of the actual number

get(self, rank)

Gets a constant from a given rank

Parameters: rank (integer) – Host rank of the actual number
Returns: int – Constant from host rank

inc(self, rank)

Increments a constant on host rank

Parameters: rank (integer) – Host rank of the actual number
Returns: int – Constant from host rank before the increment

min(self, value, rank)

Takes the min of new value and constant on host rank

Parameters

value (integer) – To value to compare constant with
rank (integer) – Host rank of the actual number

Returns

int – Minimum of the new value and constant

class exarl.network.data_structures.ExaMPIBuffUnchecked(comm, data, rank_mask=None, length=1, max_model_lag=None, failPush=False, name=None)

Bases: exarl.base.ExaData

This class is creates an RMA buffer of a fixed size on each rank. The buffer is used to send and recieve data across all participating ranks. This buffer does not check to see if it is overwriting data or if there is valid data from a get. This class always succeds a pop.

Variables

comm (mpi4py.MPI.Comm) – raw MPI communicator
win (MPI.win) – MPI window for buffer
buff (bytearray) – internal buffer used for RMA ops

Intializer

Parameters

comm (MPI Comm) – Communicator for all ranks involved
data (list) – Example data used to create buffer
rank_mask (int, optional) – host of the window
length (int, optional) – Not used
max_model_lag (int, optional) – Not used
failPush (bool, optional) – Not used
name (string, optional) – name of constant for debbuging

:param : :type : param comm: Communicator for all ranks involved :param : :type : type comm: MPI Comm :param : :type : param data: Example data used to create buffer :param : :type : type data: list :param : :type : param rank_mask: host of the window :param : :type : type rank_mask: int, optional :param : :type : param length: Not used :param : :type : type length: int, optional :param : :type : param max_model_lag: Not used :param : :type : type max_model_lag: int, optional :param : :type : param failPush: Not used :param : :type : type failPush: bool, optional :param : :type : param name: name of constant for debbuging :param : :type : type name: string, optional

__del__(self)

pop(self, rank, count=1)

Returns value of buffer at given rank. There is no check done to see if the data is valid.

Parameters

rank (integer) – Host rank where to take data from
count (integer) – How many pops to perform

Returns

list – Buffer at given rank

push(self, data, rank=None)

Pushes data to a rank’s buffer.

Parameters

data (list) – Data to be pushed to rank’s buffer
rank (integer) – Host rank of the actual number

Returns

list – Returns a capacity of 1 and loss of 1

class exarl.network.data_structures.ExaMPIBuffChecked(comm, data, rank_mask=None, length=1, max_model_lag=None, failPush=False, name=None)

Bases: exarl.base.ExaData

This class is creates an RMA buffer of a fixed size on each rank. The buffer is used to send and recieve data across all participating ranks. On pop, checks to see if the data is first valid.

Variables

comm (mpi4py.MPI.Comm) – raw MPI communicator
win (MPI.win) – MPI window for buffer
buff (bytearray) – internal buffer used for RMA ops

pop(value, rank, count): Returns value stored in buffer at rank

push(self, data, rank): Pushes data to buffer at rank

Parameters

comm (mpi4py.MPI.Comm) – Communicator for all ranks involved
data (list) – Example data used to create buffer
rank_mask (int, optional) – host of the window
length (int) – Not used
max_model_lag (int) – Not used
failPush (bool) – Not used
name (string, optional) – name of constant for debbuging

__del__(self)

pop(self, rank, count=1)

Returns value of buffer at given rank. Checks to see if the data is valid first.

Parameters

rank (integer) – Host rank where to take data from
count (integer, optional) – How many pops to perform

Returns

list – Buffer at given rank if valid

push(self, data, rank=None)

Pushes data to a rank’s buffer.

Parameters

data (list) – Data to be pushed to rank’s buffer
rank (integer, optional) – Host rank of the actual number

Returns

list – Returns a capacity of 1 and loss if data is overwritten

class exarl.network.data_structures.ExaMPIDistributedQueue(comm, data=None, rank_mask=None, length=32, max_model_lag=None, failPush=False, name=None)

Bases: exarl.base.ExaData

This class creates a circular buffer in an RMA window across nodes in a communicator. Only one RMA window is made of length entries, thus there is only one host.

Variables

comm (mpi4py.MPI.Comm) – raw MPI communicator
length (int) – capacity of the queue
failPush (bool) – flag setting if push can overwrite data
buff (bytearray) – internal buffer for queue used for RMA ops
plus (np.array) – numpy constant for adding
minus (np.array) – numpy constant for subtracting
headBuffer (np.array) – buffer containing head counter
tailBuffer (np.array) – buffer containing tail counter
head (MPI.win) – RMA window based on headBuffer
tail (MPI.win) – RMA window based on tailBuffer
win (MPI.win) – MPI window based on buffer for queue

Parameters

comm (mpi4py.MPI.Comm) – Communicator for all ranks involved
data (list, optional) – Example data used to create buffer
rank_mask (int, optional) – host of the window
length (int, optional) – capacity of queue
max_model_lag (int, optional) – Will not consider data past given model valide
failPush (bool, optional) – Fail to overwrite data if queue is full
name (string, optional) – name of constant for debbuging

__del__(self)

pop(self, rank, count=1)

Returns data from head of queue if there is data.

Parameters

rank (integer) – Host rank where to take data from
count (integer, optional) – How many pops to perform

Returns

list – Data from queue if there is any.

push(self, data, rank=None)

Pushes data to a rank’s queue.

Parameters

data (list) – Data to be pushed to rank’s queue
rank (integer, optional) – Rank to push data to

Returns

list – Returns a capacity of queue and loss if data is overwritten

class exarl.network.data_structures.ExaMPIDistributedStack(comm, data, rank_mask=None, length=32, max_model_lag=None, failPush=False, name=None)

Bases: exarl.base.ExaData

This class creates a stack in an RMA window across nodes in a communicator. Only one window is made, thus there is only one host.

Variables

comm (mpi4py.MPI.Comm) – raw MPI communicator
length (int) – capacity of the stack
failPush (bool) – flag setting if push can overwrite data
buff (bytearray) – internal numpy buffer for stack used for RMA ops
plus (np.array) – numpy constant for adding
minus (np.array) – numpy constant for subtracting
headBuffer (np.array) – buffer containing head counter
tailBuffer (np.array) – buffer containing tail counter
head (MPI.win) – window based on headBuffer
tail (MPI.win) – window based on tailBuffer
win (MPI.win) – MPI window based on buffer for stack

Parameters

comm (mpi4py.MPI.Comm) – Communicator for all ranks involved
data (list) – Example data used to create buffer
rank_mask (int, optional) – host of the window
length (int, optional) – capacity of stack
max_model_lag (int) – Will not consider data past given model valide
failPush (bool, optional) – Fail to overwrite data if queue is full
name (string, optional) – name of constant for debbuging

__del__(self)

pop(self, rank, count=1)

Returns data from head of stack if there is data.

Parameters

rank (integer) – Host rank where to take data from
count (integer, optional) – How many pops to perform

Returns

list – Data from stack if there is any.

push(self, data, rank=None)

Pushes data to a rank’s stack.

Parameters

data (list) – Data to be pushed to rank’s stack
rank (integer, optional) – Host to push data to

Returns

list – Returns a capacity of stack and loss if data is overwritten

class exarl.network.data_structures.ExaMPICentralizedStack(comm, data, rank_mask=None, length=32, max_model_lag=None, failPush=False, name=None)

Bases: exarl.base.ExaData

This class creates a stack in RMA windows across nodes in a communicator. There is a stack per rank. Each rank acts as a host.

Variables

comm (mpi4py.MPI.Comm) – raw MPI communicator
length (int) – capacity of the stack
failPush (bool) – flag setting if push can overwrite data
buff (bytearray) – internal buffer for stack used for RMA ops
plus (np.array) – numpy constant for adding
minus (np.array) – numpy constant for subtracting
headBuffer (np.array) – buffer containing head counter
tailBuffer (np.array) – buffer containing tail counter
head (MPI.win) – window based on headBuffer
tail (MPI.win) – window based on tailBuffer
win (MPI.win) – MPI window based on buffer for stack

Parameters

comm (mpi4py.MPI.Comm) – Communicator for all ranks involved
data (list) – Example data used to create buffer
rank_mask (int, optional) – host of the window
length (int, optional) – capacity of stack
max_model_lag (int, optional) – Will not consider data past given model valide
failPush (bool, optional) – Fail to overwrite data if queue is full
name (string, optional) – name of constant for debbuging

__del__(self)

pop(self, rank, count=1)

Returns data from head of stack if there is data.

Parameters

rank (integer) – Host rank where to take data from
count (integer) – How many pops to perform

Returns

list – Data from stack if there is any.

push(self, data, rank=None)

Pushes data to a rank’s stack.

Parameters

data (list) – Data to be pushed to rank’s stack
rank (integer, optional) – Rank to push data to

Returns

list – Returns a capacity of stack and loss if data is overwritten

class exarl.network.data_structures.ExaMPICentralizedQueue(comm, data, rank_mask=None, length=32, max_model_lag=None, failPush=False, name=None)

Bases: exarl.base.ExaData

This class creates circular buffers in RMA windows across nodes in a communicator. There is a queue per rank. Each rank acts as a host.

Variables

comm (mpi4py.MPI.Comm) – raw MPI communicator
length (int) – capacity of the queue
failPush (bool) – flag setting if push can overwrite data
buff (bytearray) – internal buffer for queue used for RMA ops
plus (np.array) – numpy constant for adding
minus (np.array) – numpy constant for subtracting
headBuffer (np.array) – buffer containing head counter
tailBuffer (np.array) – buffer containing tail counter
head (MPI.win) – window based on headBuffer
tail (MPI.win) – window based on tailBuffer
win (MPI.win) – MPI window based on buffer for queue

Parameters

comm (mpi4py.MPI.Comm) – Communicator for all ranks involved
data (list) – Example data used to create buffer
rank_mask (int, optional) – host of the window
length (int, optional) – capacity of queue
max_model_lag (int, optional) – Will not consider data past given model valide
failPush (bool, optional) – Fail to overwrite data if queue is full
name (string, optional) – name of constant for debbuging

__del__(self)

pop(self, rank, count=1)

Returns data from head of queue if there is data.

Parameters

rank (integer) – Host rank where to take data from
count (integer, optional) – How many pops to perform

Returns

list – Data from queue if there is any.

push(self, data, rank=None)

Pushes data to a rank’s queue.

Parameters

data (list) – Data to be pushed to rank’s queue
rank (integer, optional) – Rank to push data to

Returns

list – Returns a capacity of queue and loss if data is overwritten

exarl.network.data_structures

Module Contents

Classes

Attributes

`exarl.network.data_structures`