Abstract: As an active cooling technique, the regenerative cooling is widely applied in the thermal protection system of hypersonic vehicle engines. To further improve the heat transfer performance of regenerative cooling engines, the topology optimization of coupled thermal-fluid problem considering variable thermodynamic and transport properties was investigated. First, a coupled thermo-fluid topology optimization model was established, the adjoint equation and sensitivity considering variable thermodynamic and transport properties were derived based on the continuous adjoint method. The solver is built on the open source platform OpenFOAM, the filtering and projection techniques are adopted for alleviating numerical issues during the optimization process. A look-up table method along with an interpolation module are included to calculate the thermophysical properties and corresponding partial derivatives. Then the topology optimization of the coupled thermal-fluid system was conducted. The results show that as the power dissipation increases, the topology layouts becomes more complex, and the flow separation and remixing of coolant are stimulated. By further extracting five optimized topology layouts of the coupled thermal-fluid structures, i.e. Cases 1 ~ 5, the flow and heat transfer properties of the three-dimensional cooling channel are numerically evaluated. It is found that the complex flow separation and remixing facilitate the formation of vortices and stimulate the turbulence energy and thus improve the local heat transfer performance of the channels. The heat transfer performance of Cases 3 ~ 5 are finally enhanced compared to the conventional layout, and the average Nusselt numbers are increased by 12.6%, 16.0%, and 23.4%, respectively.