Artificial Neural Networks (ANNs) have undergone a profound transformation from the early single-layer perceptron to sophisticated deep learning architectures capable of modeling complex, high-dimensional data distributions. This paper presents a systematic review of the evolutionary trajectory of neural network models, emphasizing the interplay between architectural innovations, optimization strategies, and emerging computational challenges. The review traces foundational developments beginning with the linear decision function of the perceptron, typically expressed as $y=\mathrm{sign}(\mathbf{w}^{T}\mathbf{x}+b)$, and extends to multilayer and deep networks trained through gradient-based optimization, where parameter updates follow $\theta_{t+1}=\theta_{t}-\eta\nabla_{\theta}\mathcal{L}(\theta)$. Particular attention is given to the role of loss minimization, regularization mechanisms such as dropout and weight decay, and adaptive learning algorithms including Adam and RMSProp in stabilizing convergence across large-scale datasets. Drawing upon benchmark datasets such as MNIST, CIFAR-10, and ImageNet, the study synthesizes empirical findings from diverse experimental settings involving convolutional, recurrent, and transformer-based architectures. The analysis further examines computational trade-offs related to model depth, parameter efficiency, and generalization performance, highlighting how advances in parallel computing frameworks and distributed training pipelines have enabled networks with millions of parameters to achieve state-of-the-art predictive accuracy. Despite these achievements, persistent research challenges remain, including interpretability limitations, data dependency, energy consumption, and robustness to distributional shifts. This review contributes a structured and technically grounded synthesis of ANN evolution by integrating architectural, algorithmic, and practical perspectives, thereby providing a consolidated reference framework for researchers investigating next-generation neural network design and deployment.

Artificial Neural Networks; Deep Learning Architectures; Gradient-Based Optimization; Model Generalization; Neural Network Evolution; Training Algorithms; Computational Efficiency