Boundary elements method is a novel numerical technique to solve the boundary integration equation, and it has already been applied to initial value problems. This method is applied to electromagnetic eigenvalue problem and a numerical technique is presented that can be used to analyze the guided modes of optical waveguides with arbitrary structure. The numerical results for an elliptical core optical fiber are presented and the cutoff frequencies for the first order and second order higher modes are calculated. The maximum anisotropy of the elliptical core fiber is found for the normalized frequency of 1.85 and ellipse factor of the core equal to 0.8. From the results, it is concluded that the boundary element method has high accuracy of calculations, easy to apply and highly stable solution.