Convolutional neural network (CNN) based methods have been very successful for handwritten numeral recognition (HNR) applications. However, CNN seems to misclassify similar shaped numerals (i.e., the silhouette of the numerals that look same). This paper presents an enhanced HNR system to improve the classification accuracy of the similar shaped hand written numerals incorporating the terminals points with CNN’s recognition. In hand written numerals, the terminal points (i.e. the start and end positions) are considered as additional property to discriminate between similar shaped numerals. Start-End Writing Measure (SEWM) and its integration with CNN is the main contribution of this research. There are three major functional steps in the proposed SEWM-CNN: classification of a numeral image using a standard CNN; identification of start and end writing points from the silhouette of the numerals; and finally, system output integrating SEWM with the CNN decision. The proposed method is tested on rich benchmark numeral datasets of Bengali and Devanagari numerals. SEWM-CNN reveals itself as a suitable HNR method for Bengali and Devanagari numerals while compared with other existing methods.