The number of road accidents caused by non-vigilant drivers is increasing rapidly all over the world. Therefore, driver's vigilance detection is an important area of research in intelligent transportation systems. This paper presents an image processing based approach using minimum hardware to detect a driver's fatigue and generate an alarm if the driver is inattentive to driving. The system works in separate modules on each video frame from a smart, lowpower, Near Infra-Red (NIR) camera installed at the vehicle's dash board. Our proposed system is based on two trained Haar classifiers to detect driver's face and eyes and then tracking the iris with the method of template matching. Our algorithm updates statistics of the iris's tracking and the direction of gaze in successive video frames. An alarm is generated if the frequency of iris detection in a specific time period goes below a certain threshold or the driver does not concentrate straight to the road. The search area of every module is restricted to a particular region of interest (user-selectable) which increases the detection speed. Fatigue detection through iris tracking by template matching provides a fast means of detecting driver's fatigue. Another module, running in parallel with the face, eyes and iris detection modules tracks the head of the driver. In the situations where the driver is wearing sun glasses, eyes cannot be detected and hence iris cannot be tracked. In this case, head tracking helps to analyze driver's fatigue. We tested our algorithm in the scenarios very close to the real cases (different lighting conditions, subjects, etc). The results presented in this paper show that our algorithm is able to maintain an adequate level of detection accuracy and its minimum architecture makes it best suited for implementation in real scenarios.