bagua.torch_api.algorithms.bytegrad¶
Module Contents¶
- class bagua.torch_api.algorithms.bytegrad.ByteGradAlgorithm(hierarchical=True, average=True)¶
Bases:
bagua.torch_api.algorithms.Algorithm
This is the base class that all Bagua algorithms inherit.
Create an instance of the ByteGrad algorithm.
- Parameters:
hierarchical (bool) – Enable hierarchical communication.
average (bool) – If
True
, the gradients on each worker are averaged. Otherwise, they are summed.
- class bagua.torch_api.algorithms.bytegrad.ByteGradAlgorithmImpl(process_group, hierarchical=True, average=True)¶
Bases:
bagua.torch_api.algorithms.AlgorithmImpl
This is the base class that all Bagua algorithm implementations inherit.
It provides methods that can be override to implement different kinds of distributed algorithms.
- Parameters:
process_group (bagua.torch_api.communication.BaguaProcessGroup) – The process group to work on.
hierarchical (bool) –
average (bool) –
Implementation of the ByteGrad algorithm.
- Parameters:
process_group (BaguaProcessGroup) – The process group to work on.
hierarchical (bool) – Enable hierarchical communication.
average (bool) – If
True
, the gradients on each worker are averaged. Otherwise, they are summed.