Skeleton-based action recognition has drawn much attention recently. Previous methods mainly focus on using RNNs or CNNs to process skeletons. But they ignore the topological structure of the skeleton which is very important for action recognition. Recently, Graph Convolutional Networks (GCNs) achieve remarkable performance in modeling non-Euclidean structures. However, current graph convolutional networks lack the capacity of modeling hierarchical information, which may be sub-optimal for classifying actions which are performed in a hierarchical way. In this work, a novel Hierarchical Graph Convolutional Network (HiGCN) is proposed to deal with these problems. The proposed model includes several Hierarchical Graph Convolutional Layers (HiGCLs). Each layer consists of an attention block and a hierarchical graph convolutional block, which are used for salient feature enhancement and hierarchical representation learning, respectively. To represent hierarchical information of human actions, we propose a graph pooling method, which is differentiable and can be plugged into GCN in an end-to-end manner. Extensive experiments on two benchmark datasets show the state-of-the-art performance of our method.