bagua.torch_api.algorithms.decentralized¶
Module Contents¶
- class bagua.torch_api.algorithms.decentralized.DecentralizedAlgorithm(peer_selection_mode='all', compression=None, communication_interval=1)¶
Bases:
bagua.torch_api.algorithms.Algorithm
Create an instance of the Decentralized algorithm.
- Parameters
peer_selection_mode (str) – Can be “all” or “shift_one”. “all” means all workers’ weights are averaged in each communication step. “shift_one” means each worker selects a different peer to do weights average in each communication step.
compression (str) – Not supported yet.
communication_interval (int) – Number of iterations between two communication steps.
- init_backward_hook(self, bagua_module)¶
- Parameters
bagua_module (bagua.torch_api.distributed.BaguaModule) –
- init_forward_pre_hook(self, bagua_module)¶
- Parameters
bagua_module (bagua.torch_api.distributed.BaguaModule) –
- init_operations(self, bagua_module, bucket)¶
- Parameters
bagua_module (bagua.torch_api.distributed.BaguaModule) –
bucket (bagua.torch_api.bucket.BaguaBucket) –
- init_post_backward_hook(self, bagua_module)¶
- Parameters
bagua_module (bagua.torch_api.distributed.BaguaModule) –
- init_tensors(self, bagua_module)¶
- Parameters
bagua_module (bagua.torch_api.distributed.BaguaModule) –
- Return type