Aiming at the challenge of identifying weeds and crop seedlings in agricultural environment, this study proposed a lightweight method based on optimized MobileViT model to improve the accuracy and real-time recognition.This study conducts an in-depth analysis of the MobileViT model, introducing the SimAM attention mechanism. Its parameter-free and energy function-based design enhances the model's feature attention while maintaining a lightweight.Secondly, SCConv convolutional module is used to reduce the space and channel redundancy of features in the convolutional neural network to reduce the computational cost and model storage, and improve the performance of the convolutional module. A Loss function strategy combining Label Smoothing Loss and Cross Entropy Loss is adopted to improve the generalization performance of the model, reduce the risk of overfitting, and accelerate the convergence process of the model. In order to evaluate the performance of the proposed model MobileViT-SS, 12 common crop seedling and weed images were selected as the training data set, and the model was trained on these images. The experimental results show that the method is effective. The average recognition accuracy is 95.91%, the precision is 95.97%, the recall rate is 95.46%, and the F1 score is 95.69%. These indicators are superior to current widely used deep neural network models such as VGG-16, ResNet-18, and MobileNetv3. The method proposed in this study can accurately and quickly distinguish a variety of morphologically similar weeds and crop seedlings, and provide technical reference for weed identification in similar crops.