In every software project, a software effort estimation process is not only vital but also extremely critical. Project success or failure depends massively on, the concise knowledge of effort and schedule estimates. The development of agile techniques in the field of software development has presented researchers and practitioners with many opportunities and challenges. An estimated effort for agile software development is one of the main challenges. Although traditional estimates of effort are used to estimate effort for agile software projects, most of them lead to inaccurate estimates. This paper focuses on the development of the agile effort estimation model. A machine learning classifier that uses information contained within an issue report is proposed to classify the difficulty or the weight of a given task according to a range of story point scales. The model has two levels of attention mechanisms implemented at the word and sentence levels, allowing it to pay distinguished attention to more and less relevant semantic features when constructing the document representation. The proposed model has achieved 87% classification accuracy. An empirical evaluation demonstrates that our approach has a greater or at least equivalent F score, Precision, and Recall when compared to classical classifiers.