Evaluating the Efficiency and Performance of Minimalist Neural Network Architectures with Hybrid Activation Functions

Abstract

This research introduces an Adaptive Hybrid Activation Function (AHAF) for minimalist neural networks, addressing efficiency and performance challenges in resource-constrained applications.Unlike existing hybrid activations (e.g., Swish) or minimalist approaches (e.g., Lottery Ticket Hypothesis (LTH)), AHAF incorporates dynamic α-adaptation (0.50-2.13 range) with theoretical guarantees, achieving O(1/√T) convergence (Theorem 1) and gradient saturation mitigation (Lemma 2).Evaluated on Iris and Modified National Institute of Standards and Technology (MNIST) datasets, 8-unit AHAF networks matched 128-unit Rectified Linear Unit (ReLU) models in accuracy (96.67%) while reducing parameters by 84% and improving convergence speed by 18-22%.AHAF demonstrated superior adversarial robustness under Projected Gradient Descent (PGD) attacks (ϵ=0.1),showing only 1.18% accuracy drop versus ReLU's 3.42% (p 1.0).These results challenge conventional assumptions about hybrid activation overhead, establishing AHAF as a framework for sustainable Artificial Intelligence (AI) deployment in edge computing and adversarial-prone environments.The study bridges two key paradigms -dynamic activation adaptation and minimalist architecture design -offering both theoretical advances (formalized convergence proofs) and empirical validation (energy efficiency metrics, adversarial testing).By demonstrating that small networks with meta-learned activations can outperform traditional architectures, this work provides actionable insights for developing efficient, robust AI systems without sacrificing accuracy.The replicable methodology and open benchmarks further support adoption in real-world applications where computational resources and model stability are critical constraints.