Abstract: Variational principle has great generality, which can be divided into differential and integral, Gauss principle is the variational principle with differential form. Among the existing differential variational principles, only the Gauss principle has the extreme value characteristics, which can be expressed as the Gauss variation of the compulsion function equals to zero. Gauss principle can be used to obtain the motion law of a particle system directly by finding the extreme value of function. Therefore, Gauss principle plays a unique role in the dynamics modeling and approximate calculation of complex systems, such as the design and analysis of robots, approximate solutions of nonlinear vibration equations and dynamics of multi-body systems. This paper deals with the generalized Gauss principle for mechanical systems with variable mass and its extension to higher order nonholonomic mechanics. Firstly, Gauss’s principle of least compulsion for mechanical system with variable mass is established, and extended to second order linear nonholonomic constrained systems by constructing modified compulsion function. Secondly, the generalized Gauss principle of mechanical system with variable mass for arbitrary order cases is proposed, and generalized Gauss’s principle of least compulsion is established, and the generalized compulsion function is constructed to extend the principle to high order nonholonomic constrained systems with variable mass. It is shown that for variable-mass mechanical system with bilateral ideal high-order nonholonomic constraints, the acceleration of real motion minimizes the generalized compulsion function under the k\text-th Gauss variation in every instant among all the possible accelerations compatible with the constraints in the k\text-th acceleration space. At the end of this paper, the differential equations of motion of a burning uniform sphere moving along a rough horizontal plane and the variable-mass Hamel problem are derived by applying the generalized Gauss’s principle of least compulsion.