Superior Alternatives to MLPs? Kolmogorov-Arnold Networks Eclipse MLPs in Accuracy and Efficiency

Multi-layer perceptrons (MLPs) stand as the bedrock of contemporary deep learning architectures, serving as indispensable components in various machine learning applications. Leveraging the expressive power conferred by the universal approximation theorem, MLPs excel in approximating nonlinear functions, embodying a default choice for many tasks.

However, despite their widespread adoption, MLPs harbor notable limitations. They often exhaust a significant portion of non-embedding parameters and frequently lack interpretability without supplementary post-analysis techniques.

In a new paper KAN: Kolmogorov-Arnold Networks, a research team from Massachusetts Institute of Technology, California Institute of Technology, Northeastern University and The NSF Institute for Artificial Intelligence and Fundamental Interactions introduces Kolmogorov-Arnold Networks (KANs) as promising alternatives to MLPs. These networks showcase superior performance in both accuracy and interpretability domains.

While MLPs draw inspiration from the universal approximation theorem, KANs take cues from the Kolmogorov-Arnold representation theorem. Unlike MLPs, which rely on fixed activation functions at individual nodes, KANs employ learnable activation functions along edges, effectively replacing traditional linear weight matrices with adaptable 1D functions parameterized as splines. In KANs, nodes simply aggregate incoming signals without applying nonlinear transformations.

KANs exhibit remarkable efficiency. In the context of solving partial differential equations (PDEs), a 2-Layer width-10 KAN achieves 100-fold greater accuracy compared to a 4-Layer width-100 MLP, yielding mean squared errors (MSEs) of 10^-7 and 10^-5, respectively. Remarkably, KANs also demonstrate superior parameter efficiency, with the width-10 KAN requiring only 10² parameters, whereas the width-100 MLP demands 10⁴ parameters, making KANs a hundred times more parameter efficient.

Overall, extensive empirical evaluations underscore the remarkable efficacy of KANs over MLPs, emphasizing substantial improvements in both accuracy and interpretability metrics. These findings substantiate KANs as compelling alternatives to MLPs, thereby presenting exciting avenues for enhancing contemporary deep learning models, which heavily rely on MLP architectures.

The paper KAN: Kolmogorov-Arnold Networks is on arXiv.

Author: Hecate He | Editor: Chain Zhang

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.