Abstract: To solve near-field wave problems by finite element method, absorbing boundary conditions were needed to simulate the unbound field robustly and efficiently. Taking numerical simulation of wave motion in SH waveguide as an example, high-frequency instability caused by transmitting boundary were discussed. Directly from discrete model, mechanism of instability was found by analyzing the matching relationship of motion equations of interior nodes and nodes on absorbing boundary. Results showed that instability was caused by high frequency wave, which was allowed to enter continuously from the absorbing boundary into the computational region without excitation. Thus an approach of eliminating such instability was proposed by adjusting the motion equations of interior nodes to change the matching relationship. Theoretical and numerical results showed that the present approach was efficient.