Hyperspectral image (HSI) classification aims to identify categories of hyperspectral pixels. Recently, many convolutional neural networks (CNNs) have been designed to explore the spectrums and spatial information of HSI for classification. In recent CNN-based methods, 2-D or 3-D convolutions are inevitably utilized as basic operations to extract the spatial or spectral-spatial features. However, 2-D and 3-D convolutions are sensitive to the image rotation, which may result in that recent CNN-based methods are not robust to the HSI rotation. In this paper, a cross-attention spectral-spatial network (CASSN) is proposed to alleviate the problem of HSI rotation. First, a cross spectral attention component is proposed to exploit the local and global spectrums of the pixel to generate band weight for suppress redundant bands. Second, a spectral feature extraction component is utilized to capture spectral features. Then, a cross spatial attention component is proposed to generate spectral-spatial features from the HSI patch under the guidance of the pixel to be classified. Finally, the spectral-spatial feature is fed to a softmax classifier to obtain the category. The effectiveness of CASSN is demonstrated on three public databases. IEEE