Flame retardants (FRs) improve the intrinsic flammability of epoxy resins (EPs). However, the fundamental properties of EPs are inevitably compromised with the improvement of flame retardancy due to addition of FRs. Here, an ionic liquid (IL) octyltriphenylphosphonium-chelated orthoborate ([OTP][BScB]) has been synthesized and used as a flame-retarding additive for EP (hereafter (EP/[OTP][BScB])). Themogravimetry analysis (TGA) result indicated that the addition of [OTP][BScB] improved the residue yield during thermal degradation. Flammability and combustion tests showed the superior flame-retarding efficiency of the IL. More specifically, the addition of 3 wt% IL made EP/[OTP][BScB] pass the V-0 grade in the vertical burning test (UL-94) and achieve 32.5% value in the limited oxygen index (LOI) test. Cone calorimeter (CC) test showed that, compared with neat EP, EP/10[OTP][BScB] showed a reduction in the peak heat release rate (PHRR) and total heat release (THR) of by 39.7% and 21.8%, respectively, during combustion. Accordingly, the flame-retarding mechanisms in the gas and condensed phases were investigated in detail. In addition, tensile tests showed that the incorporation of the IL enhanced the mechanical property of EP/[OTP][BScB] to some extent. Moreover, UV–vis spectroscopy showed that the transparency of EP was basically maintained on the addition of [OTP][BScB]. Therefore, the IL [OTP][BScB] is a promising flame retardant to improve the properties of EP comprehensively. © 2020 John Wiley & Sons Ltd