This paper proposes a novel deep learning based approach for monocular visual odometry (VO) called FlowVO-Net. Our approach utilizes CNN to extract motion information between two consecutive frames and employs Bi-directional convolution LSTM (Bi-ConvLSTM) for temporal modelling. ConvLSTM can encode not only temporal information but also spatial correlation, and the bidirectional architecture enables it to learn the geometric relationship from image sequences pre and post. Besides, our approach jointly predicts optical flow as an auxiliary task in a self-supervised way by measuring photometric consistency. Experiment results indicate competitive performance of the proposed FlowVO-Net to the state-of-art methods.