Handwriting recognition is having high demand in commercial & academics. In recent years lots of good work has been done on hand written digit recognition to improve accuracy. Handwritten digit recognition system needs larger dataset and long training time to improve accuracy & reduce error rate. Training of Neural Networks for large data sets is very time consuming task on traditional serial computing on CPU. Hence, in this paper parallel learning method is presented for handwritten digit recognition scenario to reduce training time. Standard back propagation (BP) learning algorithm with multilayer perceptron (MLP) classification is chosen for this task & implemented on GPU for parallel training. GPU training is speedup by the factor 2 compare to CPU training and achieved 97.7% accuracy on MNIST dataset. This paper focused on specific parallelization environment Compute Unified Device Architecture (CUDA) on a GPU hence effectively speedup learning & reduce training time.