Deep convolutional neural networks (CNNs) have achieved significant improvements in different vision tasks, including classification, detection and segmentation. However, the increasing model size and computation makes it difficult to implement DNNs on embedded systems with limited hardware resources. Many approaches proposed to build a lightweight network and have achieved comparable performance, such as MobileNets, ShuffleNet, and ESPNet. This paper proposes a lightweight and efficient network based on depthwise dilated separable convolution and MobileNetv2 architecture. Depthwise dilated convolution in depthwise dilated separable convolution module effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Furthermore, instead of using a convolution with 3×3 kernel size for each depthwise separable convolution block in MobileNetv2, this paper uses dilated convolutions with different dilation rates to learn the representations in parallel. The proposed model is evaluated on two public datasets. The results show that the proposed model achieves better classification accuracy compared with MobileNetv2. In addition, a simple object detection framework based on the proposed model is designed and conducted on an embedded system. Experiment results show the effectiveness of the proposed model in different vision tasks.