In the processing of infrared small target image which has low signal-To-noise ratio and complex background, the target detection and recognition are very hard. So, how to suppress infrared complex background in low signal-To-clutter addition becomes the key problem in the detection of infrared small target image. The topological derivative can quantify the sensitivity of a problem when the domain under consideration is perturbed by changing its topology. Considering the idea of topology optimization, a modified topological derivative based background suppression method for infrared dim small target detection was proposed. An appropriate functional and variational problem is related to the cost function. Thus, the corresponding topological derivative can be used as an indicator function leads to the processed image through a minimization process. Firstly, introduce perturbations to each pixel of the infrared image. Secondly, calculate the corresponding topological derivative. These pixels also have the least cost function. Finally, using the modified optimal diffusion coefficient to diffuse the pixels where the topological derivative is negative to make its background smooth and achieve the purpose of removing the background clutter while enhancing the small target. Compared with other several experiment results of existing background suppressing methods in indexes, the method the paper proposed has innovative ideas and gets well effects of background suppressing and are practical methods. All of above have the important research value for the related work in future. © 2017 SPIE.