Taxi demand prediction is valuable for the decision-making of online taxi-hailing platforms. Data-driven deep learning approaches have been widely utilized in this area, and many complex spatiotemporal characteristics of taxi demand have been studied. However, the heterogeneity of demand patterns among different taxi zones has not been taken into account. To this end, this paper explores zone clustering and how to utilize the inter-zone heterogeneity to improve the prediction. First, based on the pairwise clustering theory, a taxi zone clustering algorithm is designed by considering the correlations among different taxi zones. Then, both the cluster-level and the global-level prediction modules are developed to extract intra- and inter-cluster characteristics, respectively. Finally, a Multi-Level Recurrent Neural Networks (MLRNN) model is proposed by combining the two modules. Experiments on two taxi trip records datasets from New York City demonstrate that our model improves the prediction accuracy compared with other state-of-the-art methods.