The infrared polarization degree of an object is affected by its own characteristics along with surroundings. This paper proposes an infrared polarization computational model to analyze the depolarization effect of surroundings. We prove that the sum of emissivity and reflectivity is equal to one in the s-polarized or p-polarized direction based on a coherent matrix of polarization, and this proof lays a theoretical basis for the specular emission polarization. Additionally we propose an emission Stokes vector for rough surface based on the blackbody emissivity vector, which is derived from polarization bi-directional reflectance distribution function (pBRDF). Finally, we combine the reflection and the emission together to present a complete computational model of degree of linear polarization (DoLP). The presented model implies the depolarization impact of temperature difference. An infrared polarization measurement environment is constructed to validate the model. Experimental results show the depolarization effect is in agreement with the prediction of the model. In addition to the known factors, such as observation angle, roughness, and material, the radiation from surroundings would alter the ratio of emission to incidence and affects the measured DoLP of infrared polarization. When the emitted radiation is close to incident radiation, the polarization degrades and even disappears.