Fire and smoke object detection is of great significance due to the extreme destructive power of fire disasters. Most of the existing methods, whether traditional computer vision-based models with sensors or deep learning-based models have circumscribed application scenes with relatively poor detection speed and accuracy. This means seldom taking smoke into consideration and always focusing on classification tasks. To advance object detection research in fire and smoke detection, we introduce a dataset called DFS (Dataset for Fire and Smoke detection), which is of high quality, constructed by collecting from real scenes and annotated by strict and reasonable rules. To reduce the possibility of erroneous judgments caused by objects that are similar to fires in color and brightness, apart from annotating ‘fire’ and ‘smoke’, we annotate these objects as a new class ‘other’. There are a total of 9462 images named by the fire size, which can benefit different detection tasks. Furthermore, by carrying out extensive and abundant experiments on Various object detection models, we provide a comprehensive benchmark on our dataset. Experimental results show that DFS well represents real applications in fire and smoke detection and is quite challenging. We also test models with different training and testing proportions on our dataset to find the optimal split ratio in real situations. The dataset is released at https://github.com/siyuanwu/DFS-FIRE-SMOKE-Dataset. © 2022, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.