bagua.torch_api.algorithms.bytegrad

Module Contents

class bagua.torch_api.algorithms.bytegrad.ByteGradAlgorithm(hierarchical=True, average=True)

Bases: bagua.torch_api.algorithms.Algorithm

This is the base class that all Bagua algorithms inherit.

Create an instance of the ByteGrad algorithm.

Parameters

• hierarchical (bool) – Enable hierarchical communication.

• average (bool) – If True, the gradients on each worker are averaged. Otherwise, they are summed.

class bagua.torch_api.algorithms.bytegrad.ByteGradAlgorithmImpl(process_group, hierarchical=True, average=True)

Bases: bagua.torch_api.algorithms.AlgorithmImpl

This is the base class that all Bagua algorithm implementations inherit.

It provides methods that can be override to implement different kinds of distributed algorithms.

Parameters

• process_group (bagua.torch_api.communication.BaguaProcessGroup) – The process group to work on.

• hierarchical (bool) –

• average (bool) –

Implementation of the ByteGrad algorithm.

Parameters

• process_group (BaguaProcessGroup) – The process group to work on.

• hierarchical (bool) – Enable hierarchical communication.

• average (bool) – If True, the gradients on each worker are averaged. Otherwise, they are summed.