Abstract: As we all know, surface strengthening or damage and localized damage zone (otherwise known as fracture process zone) can cause the size-dependent nominal strength of quasi-brittle materials. However, developing a model that can encompass these two factors remains one of the challenging issues in size effect research. In this paper, a size effect model, which can reflect the surface strengthening or damage of the specimen, is firstly established by systematically and in depth analyzing the factors that induce surface strengthening or damage on the specimen. On this basis, the size effects caused by the specimen's surface layer and the interior zone are analyzed, and then energy balance size effect model considering the surface strengthening or damage (EBM-SSD) is proposed by integrating the size effect model of surface strengthening or damage and the energy balance size effect model previously proposed by the author. Subsequently, a parametric analysis of the EBM-SSD is conducted. Finally, the EBM-SSD is applied to predict experimental and simulated data for three types of size effects and compared with predictions from the energy balance size effect model considering variations in the local damage zone (EBM-LDZ) previously proposed by the author. The results indicate that the ability of the EBM-SSD to capture no, ascending, descending, and ascending-then-descending size effects stems from the superposition or competition between the strengthened or damaged surface layer and the internal region as the specimen size varies. The comparison results show that EBM-SSD can accurately describe the ascending, descending, and initial rise and subsequent decline size effects. Except for sample No. 5 Pilton sandstone, the determination coefficients R² of EBM-SSD for the remaining experimental and simulated size effect data is higher than that of EBM-LDZ, and its mean determination coefficient M-R² is 0.9729.