A fusion framework of CNNs and vision transformers for high-accuracy image recognition

Abstract:

Recent progress in image classification has shown that Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) each offer unique strengths. CNNs are highly effective at learning local spatial patterns through layered convolution operations, whereas Transformers excel at capturing global relationships within an image using self-attention. This paper introduces a hybrid CNN–Transformer model that combines these advantages to deliver high-accuracy and scalable image recognition. In the proposed approach, CNN layers are used at the initial stages to extract fine-detail features and preserve local information. These features are then processed by Transformer encoder blocks, which model long-range dependencies and deeper semantic relationships across different image regions. A dedicated fusion module brings together the local and global representations, enabling more powerful and discriminative feature learning. Comprehensive experiments on standard datasets—including CIFAR-10, CIFAR-100, and ImageNet—show that the hybrid system consistently outperforms standalone CNN and Transformer models. The combined architecture demonstrates faster and more stable convergence, reduced redundancy in parameters, and improved robustness against noise, distortion, and occlusion. Findings from this study suggest that integrating CNNs with Vision Transformers is a promising direction for future computer vision research, offering an effective balance between performance, efficiency, and generalization.