Traffic light recognition plays a crucial role in intelligent transport systems. In traffic scene images, traffic light instances usually occupy a small region. Thus, recent state-of-the-art object detectors such as Faster RCNN and SSD obtain low accuracy on traffic light recognition in traffic scene images. This paper presents a deep learning framework for traffic light recognition in traffic scene image. Considering that feature maps at shallow layers have higher resolution which will improve small traffic light detection, and feature maps at deep layers contain more discriminative representation which will improve traffic light classification task, this paper designs a feature fusion subnet for feature extraction to solve the problem of small traffic light detection. The feature fusion subnet fuses feature maps at different layers. Thus, the feature fusion subnet not only can preserve the information of small traffic lights but also enhance the semantic information. Furthermore, a detection subnet is designed at the detection prediction stage. The detection subnet includes multiple detection layers, and each layer performs detection predictions with a coarse-to-fine detection strategy. The coarse-to-fine detection strategy is applied to improve the classification performance of the detection network. The proposed approach is evaluated on Bosch Small Traffic Lights dataset. Experimental results show that the proposed approach obtains higher accuracy compared with recent state-of-the-art detectors such as Faster R-CNN and SSD.