# Batch Vs Mini Batch Gradient Descend

Deep leaning works best in the regime of Big data . Algorithms tends to work slow on large data sets. thatâ€™s why we need fast optimization algorithms to cope well with the large data sets. One little example of optimization is when we use vectorization instead of loops . So whenever we find a loop that can be implemented using vectorization we do it without giving it a second thought.

### Batch vs Mini batch gradient descend :

Mini batch gradient descend is also used to get an optimized algorithm.

**We want our gradient descend(backward propagation ) to not sit idle until we have gone through whole training data (forward propagation).**

Unlike batch descend , mini batch descend use mini batches of training data . when we use whole training data to get the output from forward propagation .Gradient descend step will be at rest during this whole time . If we could somehow manage to get this gradient descend working before going through whole training data we could get a better optimized algorithm. So we divide training data into mini batches and we do our forward and back propagation for each mini batch .

**Batch gradient descend :** If batch size is equal to # of training examples(m)

**Disadvantage:** It takes very long time per iteration if training set is very large. Gradient descend sits idle.

**Stochastic gradient descend:** If batch size is 1 then we will do forward and backward propagation for 1 training example at a time.

**Disadvantage :** We lose speed up from the vectorization because we are using only a single training example at a time.

**What should be the batch size ?**
We want to keep vectorization and at the same time we want to make progress without going through entire training set. so we have to use batch size between 1 and m . typical mini batch size are 64,128,256 . These sizes are usually written as power of 2.

**Note:** If training set is small ( m < 2000) then we can use batch gradient descend because in case of small training set gradient decent will not have to wait long enough.

## Leave a comment