Recently, many object detectors based on deep convolutional neural networks such as Faster R-CNN, SSD, RetinaNet, and so on have been proposed and showed significant improvements over traditional object detectors. However, these deep learning-based object detectors usually focus on detection accuracy. This paper proposes a one-stage deep learning-based object detection framework to improve the inference speed and achieve real-time object detection in outdoor scene images without compromising on accuracy. To improve the inference speed, this paper adopts MobileNet v2 architecture at first to generate the base convolution feature maps. MobileNet v2 achieved comparable performance compared with other state-ofthe-art networks while being simpler and faster. To improve the detection accuracy, an enhanced feature pyramid generation module is used to construct rich and multi-scale feature maps from a single resolution input image. Each feature level is a high-level semantic feature map and can be used for detecting objects at a different scale. Finally, a detection network which includes a classification subnet to predict the probability of object presence and a box regression subnet to regress the offset from each anchor box to a nearby groundtruth object is attached to each feature level in the enhanced feature pyramid to locate objects at different scales. In addition, focal loss function is used in the classification branch of the detection network to dedicate the class imbalance problems for the one-stage object detector. Experimental results on public datasets show that the proposed approach achieves nearly as performance as other state-of-the-art approaches, while the inference speed is significant improved.