The study investigates the presence of 1/f noise, also known as pink noise, in deep neural networks, particularly Long Short-Term Memory (LSTM) networks trained on a natural language processing task. The key findings are:
LSTM networks trained on the IMDb movie review dataset exhibit clear 1/f noise patterns in the time series of their neuron activations, similar to the 1/f noise observed in biological neural networks like the human brain.
This 1/f noise is not present in the input data itself, indicating that the networks are self-organizing to generate these patterns.
As the capacity of the LSTM networks increases beyond what is needed for the task, the 1/f noise pattern starts to break down, with many neurons becoming inactive and the overall activation patterns shifting towards white noise.
The study also finds a distinction between the 1/f noise patterns in the "internal" activations (used for regulating the network) versus the "external" activations (the final output), mirroring the differences observed between surface EEG and deep fMRI measurements in the human brain.
These findings suggest that the emergence of 1/f noise in deep neural networks is a signature of optimal, self-organized information processing, akin to what is observed in biological neural networks. The authors propose that the transparency and controllability of artificial neural networks make them valuable tools for further investigating the fundamental origins of 1/f noise and its relationship to healthy neural function.
다른 언어로
소스 콘텐츠 기반
arxiv.org
더 깊은 질문