Total: 1
DNN training is extremely time-consuming, necessitating efficient multi-accelerator parallelization,where a single iteration of training is split over the accelerators.Current approaches often use intra-batch parallelization.Combining inter-batch pipeline parallelism with intra-batch parallelism is a common approach to further improve parallel training throughput.Here, we develop a system, called TiMePReSt, that adds both of them, but in a way which helps to better overlap computation and communication within a mini-batch, and limits the amount of communication.The traditional pipeline parallel training maintains similar working principle as conventional training.Thus, it suffers from high GPU memory usage during training to maintain consistent version of weights in forward and backward passes of a mini-batch.Here, it has been shown experimentally that violating consistency of weight versions does not necessarily reduce prediction capability of a parallely trained DNN.TiMePReSt helps to overcome GPU memory overhead and achieve zero degree of staleness of weights, but not effecting prediction capability.Existing techniques often become costly in terms of training time.Thus, TiMePReSt introduces a variant of intra-batch parallelism that parallelizes the forward pass of each mini-batch by decomposing it into smaller micro-batches.Synchronization between backward and forward passes are performed in a novel way reduce training time in TiMePReSt.The chances of occurring multiple sequence problem and its relation with version difference have been observed in TiMePReSt.A mathematical relationship between the number of micro-batches and worker machines has been formulated.A mathematical expression of version difference has also been devised so that the version difference for different combination of these two can be computed mathematically without preparing diagrams for all the combinations.