
Sophia is a second-order optimizer that achieves a 2x speed-up compared to Adam in training language models, reducing time, cost, and compute resources.


coremsg

sophia-a-scalable-stochastic-second-order-optimizer-for-language-model-pre-training


Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training


title_rewrite


Sophia is a second-order optimizer that achieves a 2x speed-up compared to Adam in training language models, reducing time and cost significantly.



The authors introduce Sophia, a second-order optimizer for language model pre-training that achieves significant speed-ups compared to Adam. Sophia adapts efficiently to heterogeneous curvatures and uses a light-weight diagonal Hessian estimate.


Sophia-A-Scalable-Stochastic-Second-order-Optimizer-for-Language-Model-Pre-training-at-Stanford-University


Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training at Stanford University
