Abstract: Thanks to the great advances in fast boundary element method (BEM) achieved in the last two decades, the BEM has been increasingly used in the dynamic design of engineering structures, the analysis of noise and vibration. Consequently, solving large-scale eigenvalue problems and performing modal analysis for complicated structures and acoustic fields using the BEM becomes an very important but challenging task; so far there are no effective numerical methods for this purpose. This paper aims to extend the application of the newly-developed resolvent sampling based Rayleigh-Ritz projection method (RSRR) to the solution of the general nonlinear eigenvalue problems (NEP) in BEM. First, in order to generate reliable eigenvector search spaces, a series of BEM linear systerms in frequency domain are solved. Then the original NEP can be transformed to a reduced NEP based the classical Rayleigh-Ritz procedure, and the reduced NEP could be solved by those exiting NEP solvers easily. Second, to reduce the prohibitive computational burden involved in solving the projected NEP by the Rayleigh-Ritz procedure, a BEM matrix interpolation technique and a fast computation method for reduced NEP systerm matrix are proposed based on the discretized Cauchy integral formula of analytic functions. Then a simple rule for estimating the number of terms in the interpolation is proposed as well. Finally, the RSRR method is used to solve large-scale practical acoustic modal analysis problems using fast BEM with complicated sound absorbing boundary conditions. Numerical results indicate that the method can robustly dig out all the interested eigenvalues and the corresponding eigenvectors with good accuracy and high computational efficiency.