Understanding human emotion means communicating with our fellow humans on a deeper level. Unfortunately, understanding human emotion is not as easy as it sounds. As humans, we express our emotions in different ways, be it using our tone of speech, words of choosing, or facial expressions. To just pick one over the many ways we express emotion and draw a conclusion would mean that we lose out on information causing us to arrive at the wrong conclusion. This research shows that fusing two modalities, text and speech, with a stacking ensemble method, shows leap and bounds improvements in its accuracy when compared to the unimodal approach. Tested on the IEMOCAP dataset with a 4 emotions subset of anger, happy, sad, and neutral, the text model of BERT (Bidirectional Encoder Representations from Transformer) managed to achieve an accuracy score of 65.4%, and the audio model of CNN (Convolutional Neural Network) + Bi-LSTM (Bidirectional Long Short-Term Memory) with the implementation of LFLB (Local Feature Learning Block) managed to achieve an accuracy score of 60.6%. These results were then combined into one with a stacking ensemble method and achieved an accuracy of 75.181%.