bagua.torch_api.algorithms.decentralized

Module Contents

class bagua.torch_api.algorithms.decentralized.DecentralizedAlgorithm(hierarchical=True, peer_selection_mode='all', communication_interval=1)

Bases: bagua.torch_api.algorithms.Algorithm

Create an instance of the Decentralized SGD algorithm.

Parameters:

  • hierarchical (bool) – Enable hierarchical communication.

  • peer_selection_mode (str) – Can be "all" or "shift_one". "all" means all workers’ weights are averaged in each communication step. "shift_one" means each worker selects a different peer to do weights average in each communication step.

  • communication_interval (int) – Number of iterations between two communication steps.

class bagua.torch_api.algorithms.decentralized.DecentralizedAlgorithmImpl(process_group, hierarchical=True, peer_selection_mode='all', communication_interval=1)

Bases: bagua.torch_api.algorithms.AlgorithmImpl

Implementation of the Decentralized SGD algorithm.

Parameters:

  • process_group (BaguaProcessGroup) – The process group to work on.

  • hierarchical (bool) – Enable hierarchical communication.

  • peer_selection_mode (str) – Can be "all" or "shift_one". "all" means all workers’ weights are averaged in each communication step. "shift_one" means each worker selects a different peer to do weights average in each communication step.

  • communication_interval (int) – Number of iterations between two communication steps.

class bagua.torch_api.algorithms.decentralized.LowPrecisionDecentralizedAlgorithm(hierarchical=True, communication_interval=1)

Bases: bagua.torch_api.algorithms.Algorithm

Create an instance of the Low Precision Decentralized SGD algorithm.

Parameters:

  • hierarchical (bool) – Enable hierarchical communication.

  • communication_interval (int) – Number of iterations between two communication steps.

class bagua.torch_api.algorithms.decentralized.LowPrecisionDecentralizedAlgorithmImpl(process_group, hierarchical=True, communication_interval=1)

Bases: bagua.torch_api.algorithms.AlgorithmImpl

Implementation of the Low Precision Decentralized SGD algorithm.

Parameters:

  • process_group (BaguaProcessGroup) – The process group to work on.

  • hierarchical (bool) – Enable hierarchical communication.

  • communication_interval (int) – Number of iterations between two communication steps.