Abstract: This paper uses the “AM-GoogLeNet + BP” combined data-driven methods to predict the uniaxial compression stress-strain curve of the concrete meso-model (including mortar, aggregates, porosity) effectively by combining the good performance of GoogLeNet convolutional neural network and BP neural network in image data mining and data analysis, respectively. The physical interpretability of the neural network is improved by introducing the mechanical parameters to optimize the image data-driven training results. The automated modeling of the concrete meso-model in Abaqus and microscopic image generation process are realized by Python language, and the generated mesoscopic image database and the corresponding compression stress–strain curves are used as the training dataset. Three typical attention mechanisms, SENet, ECANet and CBAM, are introduced into GoogLeNet respectively to enhance the analysis ability of neural network for each phase of concrete in an adaptive manner and the performance of the three attention mechanisms is compared and analyzed. The initial stress-strain prediction curves of the concrete meso-model are obtained with this method; In order to improve the prediction accuracy of the peak stress, the physical parameters such as aggregate volume fraction, porosity and initial peak stress are introduced into BP neural network as inputs. It is also compared with the method of introducing the physical parameters directly into the convolutional neural network input layer. At the same time, the weight of influence of aggregate volume fraction and porosity on peak stress is given quantitatively. The results show that this method has high prediction accuracy for the concrete meso-model with different aggregate volume fraction and porosity. In this paper, the “AM-GoogLeNet + BP” combined data-driven prediction model is used to solve the complexity of the traditional method in the analysis of mesoscale parameters from the statistical point of view, which provides a new idea for the study of the cross-scale mechanical behavior of composite materials.