In this study, three dataset of yellow tea sample data were collected from a total of 675 Bixiangzao variety tea images of yellowing leaves at different stage. Position transformation, random brightness, increasing contrast, adding noise, and random scaling operations were used to enhance the data. The pre-training model of MobileNetV3-Large was obtained from the ImageNet data sets using transfer learning method, and ownership weight information of the network was transferred for training, then the lightweight convolutional neural network MobileNetV3-Large recognition model for yellowing degree of yellow tea was established. Grad-CAM(Gradient-weighted class activation mapping) heat map visualization and confidence scores were used to monitor the changes in yellow tea quality. The results of using the model to estimate the teas showed that the trained MobileNetV3-Large model achieves recognition accuracy of 98.51%, precision of 99.10%, recall of 98.93% and F1-score of 98.20%. The recognition accuracy of MobileNetV3-Large model was higher than the traditional machine learning models SVM(Support vector machine), XGBoost(eXtreme gradient boosting) and KNN(K-Nearest neighbors). As visualized by Grad-CAM heat map, the MobileNetV3-Large model was able to accurately locate and extract yellowing leaf features in different recognition scenarios, and accurately identify the yellowing degree. In conclustion, the MobileNetV3-Large model had better generalization and could rapidly and nondestructively identify the yellowing degree of yellow tea.