Read the original document by opening this link in a new tab.
Table of Contents
1 Introduction
2 Alternatives to softmax
2.1 Softmax
2.2 Taylor softmax
2.3 Soft-margin softmax
2.4 SM-Taylor softmax
3 Experimental Results
4 Conclusion and Future Work
Summary
Softmax function is widely used in artificial neural networks for multiclass classification, multilabel classification, attention mechanisms, etc. This paper explores alternatives to the softmax function such as Taylor softmax, soft-margin softmax, and SM-Taylor softmax. Experimental results on image classification tasks reveal that SM-Taylor softmax outperforms the original softmax function and other alternatives. Future work includes exploring larger models and datasets to find a universal replacement for softmax.