Gait recognition is one of the most important techniques for human identification at a distance. Most current gait recognition frameworks consist of several separate steps: silhouette segmentation, feature extraction, feature learning, and similarity measurement. These modules are mutually independent with each part fixed, resulting in a suboptimal performance in challenging conditions. In this paper, we integrate those steps into one framework, i.e., an end-to-end network for gait recognition, named GaitNet. It is composed of two convolutional neural networks: one corresponds to gait segmentation, and the other corresponds to classification. The two networks are modeled in one joint learning procedure which can be trained jointly. This strategy greatly simplifies the traditional step-by-step manner and is thus much more efficient for practical applications. Moreover, joint learning can automatically adjust each part to fit the global optimal objective, leading to obvious performance improvement over separate learning. We evaluate our method on three large scale gait datasets, including CASIA-B, SZU RGB-D Gait and a newly built database with complex dynamic outdoor backgrounds. Extensive experimental results show that the proposed method is effective and achieves the state-of-the-art results. The code and data will be released upon request. (C) 2019 Elsevier Ltd. All rights reserved.