The truss structure of header in the caterpillar type rapeseed windrower is complex and changeable，which leads to the problem of large vibration of the header.Based on the variable density method，the multi-objective topology optimization of frame structure of the header was carried out to reduce the vibration of the header under different working conditions. The topology optimization model of header was established by using Hypermesh software. The virtual prototype model of rapeseed windrower based on RecurDyn was established by using multi-body dynamics method. The dynamic loads of the moving pair of the header under parts debugging conditions，transport conditions and field operation conditions were obtained. The comprehensive objective function of static stiffness and dynamic frequency of the header was constructed with the compromise programming method. The weight of each sub- object was determined according to the analytic hierarchy process （AHP）. Taking the volume fraction and maximum load of moving pair as constraints，the ideal material density distribution of the header spatial structure was obtained. Considering the practical engineering application，the optimized frame of header was normalized. The finite element model of the optimized frame of header was established and the finite element static analysis and modal analysis were carried out. The vibration amplitudes at the measuring points of the header before and after optimization were compared and analyzed. The results showed that the maximum stress value decreased from 107.99 MPa to 65.45 MPa after optimization. The maximum deformation decreased from 0.82 mm to 0.36 mm. The natural frequencies of the first three orders were increased to varying degrees，and the natural frequency of the first order was increased to 24.187 Hz. The results of actual vibration test showed that the vibration amplitude of each measuring point of the header decreases after optimization. The overall vibration of the header decreased，in which the amplitude of the longitudinal cutter support decreased from 4.83 m/s² to 1.49 m/s². It will provide reference for the structure improvement and optimization of rapeseed windrower.