Classifying brain tumours is an exclusive and difficult task in clinical image analysis. Radiologists could reliably detect tumours using machine learning algorithms without extensive surgery. However, several difficulties arise, including difficulty locating a competent specialist in identifying brain malignancies using images using deep learning models and the main issue in the erection of the most effective deep learning system for diagnosing tumour cells. We used deep learning and adaptive algorithms to build a sophisticated and incredibly accurate system with feature fusion to automatically categorize brain tumors. The proposed framework extracts deep features from CNN architectures with varying depths and designs. The highest-performing CNN architectures' features are then fused to form a single vector classified using SVM and KNN. The novel vector obtained the highest accuracy of 92% via the feature fusion method. Therefore, the suggested framework can be successfully used in clinical settings to categorize three different forms of tumors, namely gliomas, meningiomas, and pituitary tumors, from medical imaging.