Breast cancer, as one of the most leading cancers for women, has attached more and more attention. Early image-based detection of masses for mammogram screening plays a crucial role for radiological diagnosis. In this paper, we propose to incorporate global and local information for accurate masses detection. Specifically, we improve a local ROI-based CNN framework which is named as YOLO for coarse mass localization, followed by an improved U-net structure to incorporate global information for fine mass detection. Experimental results on benchmark dataset of INbreast show that our proposed method can achieve preferable results.