bagua.torch_api.algorithms.decentralized¶
Module Contents¶
- class bagua.torch_api.algorithms.decentralized.DecentralizedAlgorithm(hierarchical=True, peer_selection_mode='all', communication_interval=1)¶
Bases:
bagua.torch_api.algorithms.Algorithm
Create an instance of the Decentralized SGD algorithm.
- Parameters
hierarchical (bool) – Enable hierarchical communication.
peer_selection_mode (str) – Can be
"all"
or"shift_one"
."all"
means all workers’ weights are averaged in each communication step."shift_one"
means each worker selects a different peer to do weights average in each communication step.communication_interval (int) – Number of iterations between two communication steps.
- class bagua.torch_api.algorithms.decentralized.DecentralizedAlgorithmImpl(process_group, hierarchical=True, peer_selection_mode='all', communication_interval=1)¶
Bases:
bagua.torch_api.algorithms.AlgorithmImpl
Implementation of the Decentralized SGD algorithm.
- Parameters
process_group (BaguaProcessGroup) – The process group to work on.
hierarchical (bool) – Enable hierarchical communication.
peer_selection_mode (str) – Can be
"all"
or"shift_one"
."all"
means all workers’ weights are averaged in each communication step."shift_one"
means each worker selects a different peer to do weights average in each communication step.communication_interval (int) – Number of iterations between two communication steps.
- class bagua.torch_api.algorithms.decentralized.LowPrecisionDecentralizedAlgorithm(hierarchical=True, communication_interval=1)¶
Bases:
bagua.torch_api.algorithms.Algorithm
Create an instance of the Low Precision Decentralized SGD algorithm.
- Parameters
hierarchical (bool) – Enable hierarchical communication.
communication_interval (int) – Number of iterations between two communication steps.
- class bagua.torch_api.algorithms.decentralized.LowPrecisionDecentralizedAlgorithmImpl(process_group, hierarchical=True, communication_interval=1)¶
Bases:
bagua.torch_api.algorithms.AlgorithmImpl
Implementation of the Low Precision Decentralized SGD algorithm.
- Parameters
process_group (BaguaProcessGroup) – The process group to work on.
hierarchical (bool) – Enable hierarchical communication.
communication_interval (int) – Number of iterations between two communication steps.