Emotions are an essential part of immaculate communication. The purpose of this research work is to classify six basic emotions of humans namely anger, disgust, fear, happiness, sadness and surprise. In proposed method a sequential deep convolutional neural network is proposed for audio and visual modality. Audio classification is performed via fine-tuning of a pretrained AlexNet model whereas, visual classification is performed with a hybrid deep network containing CNN and LSTM. Decision level and score level fusion have been implemented for multimodalities. SVM, random forest, K-NN, and logistic regression classifiers were being used for classifying emotion for fused audio-visual data. Experiments have been performed on RML and BAUM-1s dataset with LOSO and LOSGO cross validation techniques respectively. Recognition rates were extremely positive which shows the validity of the proposed methodology.