This study uses image processing and machine learning techniques to provide a comprehensive system for grading mango fruits according to their physical and visual attributes. More than 400 photos of mangos make up the dataset, which is divided into four grades (A, B, C, and D) based on characteristics like size, weight, shape, texture, and spots as well as color components (red, green, and blue). The method involves a lot of image preparation, including scaling and rotation of the dataset, as well as standardizing dimensions and pixel values. To prepare the datasets and reduce processing, feature extraction and reduction are used. Because recurrent neural networks (RNNs) and convolutional neural networks (CNNs) are better at picture identification and weight-sharing, they are used in the model design. Furthermore, training efficiency and performance are balanced through classification using CNN and Transfer Learning. Improved preprocessing and treatment of outliers lead to improved dataset results. A confusion matrix is used to assess the 92% accuracy rate classifier's performance. There are 200 samples in the dataset for training, 100 samples for testing, and 100 samples for validation. The suggested method is a potential way to grade mangoes, according to the results.