Abstract: The Birkhoff system is a more extensive constrained mechanical system than Hamilton system, which can be applied to atomic and molecular physics, and hadron physics.It is an important and difficult project to study the stability of non-steady mechanical system, and it is very difficult to study the stability by using the direct method of constructing Lyapunov function, here how to construct the Lyapunov function is always an open question.This paper gives an indirect method which is called the gradient system method.A kind of gradient systems with non-symmetrical negative-definite matrix is proposed, and the solution of the gradient system can be stable or asymptotic stable.The study of the gradient system is particularly suitable by using the method of Lyapunov functions, in which the function V is usually taken as the Lyapunov function.Firstly the equations of motion of the generalized Birkhoff system are listed.The generalized Birkhoff system is a kind of extensive constrained mechanical system, holonomic and nonholonomic constraint systems can be incorporated into the system.When the additional terms of the system are equal to zero, it becomes the Birkhoff system.Then the conditions under which the solutions of the generalized Birkhoff system can be stable or asymptotically stable are given.Further the generalized Birkhoff systems whose solution is stable are constructed by using the gradient system with non-symmetrical negative-definite matrix.The method is also suitable for the study of other constrained mechanical systems.Lastly some examples are given to illustrate the application of the results.