Considering that Pakistan's economy depends heavily on agriculture more than 21-22% of GDP is generated by the sector-it is critical to maintain high productivity and reduce disease-related losses. Early diagnosis of plant diseases is crucial because they represent serious hazards to crop quality and productivity, such as early and late blight. In the past, disease identification was done by manual observation, which takes a lot of time and labor. However, disease detection in agriculture has completely changed as a result of recent developments in machine learning, computer vision, and deep learning algorithms. Unfortunately, memory consumption, accuracy and computation time are still major challenges. In recognition of these difficulties, we modified a convolutional neural network (CNN) to reduce the number of trainable parameters while maintaining accuracy in order to speed up computation. We evaluated the suggested model's performance against a range of deep learning and machine learning techniques for classifying potato early blight and late blight. The suggested model, which used 116,202 trainable parameters, surpassed the others with an overall accuracy of 99%.