Facial Emotion Recognition is a part of human-computer interaction research whose results can be applied to society, ranging from social robots to accompany elderly to video games that adjust the level of difficulty based on the player's facial expressions. Many researchers attempt to create models for better accuracy with deep learning techniques, especially using Convolutional Neural Networks (CNN) that perform well in pattern recognition and image processing. This research adopts the state-of-the-art models in ImageNet and modifies the classification layer with SpinalNet and ProgressiveSpinalNet architecture to improve the accuracy. The classification is performed on the dataset FER2013, which is openly shared to the public in Kaggle contains over 35.000 face image datasets for seven distinct emotions. After finishing the training process and finetuning its hyperparameter, the resulting model with SpinalNet and ProgressiveSpinalNet outperformed all the existing single standalone model research on FER2013. One of the proposed architectures, VGGSpinalNet, has achieved the highest single network accuracy of 74.45%.