The breast cancer is one of the most common diseases in women. This paper proposed a breast tumor computer aided diagnosis (CAD) system utilized the Breast Imaging Reporting and Data System (BI-RADS) features. The BI-RADS feature scoring scheme is designed to transform the BI-RADS report to a vector. And the decision tree algorithm is adopted to classify the vector. Compared with previous CAD system, the proposed system is easier to be understood by the clinician. Without the image preprocessing, the proposed system can be applied in different ultrasound machines. There are 440 samples collected from the Cancer Center of Sun Yat-sen University. In the experiment, the five-fold cross validation is employed to evaluate the proposed system. The result shows that the performance of the proposed system is better than the CAD method which takes the BI-RADS feature as a guide to extract features from images. The average accuracy achieves 89.38%, specificity is 90.74%, sensitivity is 86.18%, positive predictive value (PPV) reaches 93.57% and negative predictive value (NVP) is 79.82%. © 2017 IEEE.