🔬 Neural Network · Recurrent Architecture

Elman Network Simulation

The Elman network (Simple Recurrent Network, SRN) is a classic recurrent neural network proposed by Jeffrey Elman in 1990. By introducing a context layer that stores the historical hidden state, it gives the network a "memory" capability, enabling it to process temporal data and sequence-prediction tasks.

Core Formulas and Parameter Details
Hidden State Update
$$h(t) = f\left(W_h \cdot x(t) + U_h \cdot c(t) + b_h\right)$$
The hidden state h(t) is driven by two parts: the current input x(t) via the weight Wh, and the previous time step's context c(t) via the recurrent weight Uh. f is the nonlinear activation function.
  • h(t): hidden-state vector at time step t (4-dimensional)
  • Wh: input-to-hidden weight matrix (3×4)
  • Uh: context-to-hidden recurrent weight matrix (4×4)
  • bh: hidden-layer bias vector (4-dimensional)
Context Layer Update
$$c(t) = h(t-1)$$
The context layer c(t) directly copies the previous time step's hidden state h(t-1). This is the core mechanism of the Elman network—the delay line—which lets the network "remember" the previous step's information and thereby handle temporal dependencies.
  • c(t): context-layer state at time step t (same dimension as h)
  • h(t-1): hidden state at time step t-1
  • Initial condition: c(0) = 0 (zero vector)
Output Layer Computation
$$y(t) = g\left(W_y \cdot h(t) + b_y\right)$$
The output y(t) is obtained by linearly transforming the hidden state h(t) via the weight Wy, then passing it through the output activation function g. In this demo the output is a 1-dimensional scalar used for sequence prediction.
  • y(t): network output at time step t (1-dimensional)
  • Wy: hidden-to-output weight matrix (4×1)
  • by: output bias (scalar)
  • g: output activation function (this demo uses identity / linear)
Activation Function
$$f(z) = \sigma(z) = \frac{1}{1+e^{-z}}, \quad f(z) = \tanh(z) = \frac{e^z-e^{-z}}{e^z+e^{-z}}$$
The Elman network hidden layer commonly uses the Sigmoid or Tanh activation function. Sigmoid outputs in the range (0,1) and Tanh in (-1,1). Tanh is zero-centered and usually converges faster in practice.
  • σ(z): Sigmoid function, output ∈ (0,1), max gradient 0.25
  • tanh(z): hyperbolic tangent, output ∈ (-1,1), max gradient 1
  • z: pre-activation value (weighted sum)
Loss Function (MSE)
$$L = \frac{1}{T}\sum_{t=1}^{T}\left(y(t) - \hat{y}(t)\right)^2$$
Training an Elman network typically uses Mean Squared Error (MSE) as the loss function, measuring the average squared difference between the network's predicted output y(t) and the true target ŷ(t). Parameters are optimized via the BPTT (Backpropagation Through Time) algorithm.
  • L: loss value (scalar)
  • T: total number of time steps in the sequence
  • ŷ(t): true target value at time step t
  • BPTT: computes gradients by unrolling the time dimension
Temporal Unfolding
$$h(t) = f\left(W_h x(t) + U_h f\left(W_h x(t-1) + U_h h(t-2) + b_h\right) + b_h\right)$$
Unrolling the recurrent formula over time reveals that the hidden state at each time step depends on all historical inputs. This unfolding is the basis of the BPTT training algorithm and an intuitive way to understand the Elman network's "memory" capability.
  • Nested recursion: h(t) depends on h(t-1), h(t-1) depends on h(t-2)...
  • Long-term dependency: the chain of dependencies can cause gradients to vanish or explode
Interactive Forward Propagation Simulation

Parameter Settings

steps

📥 Input Sequence Settings

Weight Parameters

Weight matrices will be displayed after running
Click "Run" after adjusting parameters to update the results

Elman Network Structure (3-4-4-1)

Hidden State Heatmap (activation values per time step)

Output Sequence y(t)

Forward Propagation State Details

Click "Run Simulation" to view detailed state

Deep Dive into the Elman Network

📖Standard Introduction

The Elman network (Elman Network), also known as the Simple Recurrent Network (SRN), is a classic recurrent neural network architecture proposed by American cognitive scientist Jeffrey L. Elman in his 1990 paper "Finding Structure in Time".

Building on the standard three-layer feedforward neural network (input-hidden-output), the Elman network adds a context layer (Context Layer). The number of neurons in the context layer equals that of the hidden layer; each context neuron receives the activation of its corresponding hidden neuron from the previous time step and feeds it back to all hidden neurons. This design lets the network "remember" past states, giving it the ability to process time-series data and dynamic system modeling.

Mathematically, the Elman network's core innovation is the introduction of a recurrent dependency on the previous time step's hidden state in the hidden-state update formula: h(t) = f(W·x(t) + U·h(t-1) + b). This recurrent connection embeds the time dimension into the network structure, so the network output depends not only on the current input but also on the complete history of input sequences.

💡Plain-Language Introduction

Imagine you are reading an article. When you come across the word "apple", you need to judge from the preceding context whether it means fruit or a tech company. A standard neural network is like a person who only looks at the current word and cannot use contextual information; the Elman network, by contrast, is like a reader who "remembers" what came before—it saves the "state of mind" after processing each piece of information and consults that state when processing the next.

Concretely, the Elman network has a "memo notebook" (the context layer). Each time it finishes processing an input, it copies the hidden state (the equivalent of its "current understanding") onto this notebook. When processing the next input, it looks at both the new input and the old state on the notebook, combining the two to decide the current output. This is why the Elman network can understand sequential data—because it "remembers" what happened before.

In technical terms: the Elman network's core mechanism is to delay the hidden state by one step and feed it back to the input, forming a kind of "short-term memory". Simple as it is, this memory is enough to capture short-term dependencies in a sequence.

🎯What Problem Does It Solve

Before the Elman network, standard feedforward neural networks (such as the multilayer perceptron, MLP) had a fundamental limitation: they assume inputs are mutually independent. When processing each input sample, they completely ignore the relationship between consecutive samples, and therefore cannot handle the following kinds of problems:

  • Time-series prediction: predicting tomorrow's stock price from the past few days' prices—feedforward networks cannot exploit the temporal correlations in historical data.
  • Natural language processing: understanding the meaning of "apple" in "I like apples" requires combining the preceding "like" with the context; feedforward networks cannot handle the sequential dependencies between words.
  • Speech recognition: speech signals are continuous time series, and feedforward networks cannot model the transition relations between phonemes.
  • Dynamic system modeling: state evolution in control systems depends on historical states, requiring a memory mechanism.

The Elman network was the first to solve this problem through recurrent connections, enabling neural networks to naturally process data with temporal structure and laying the theoretical foundation for later, more powerful recurrent architectures such as LSTM and GRU.

⭐Why It Matters

The Elman network holds milestone significance in the history of neural network development, and its importance is reflected in the following aspects:

  • Pioneering contribution: the Elman network was one of the first recurrent architectures to be widely recognized and applied, proving that time-series modeling can be achieved by simply extending a feedforward network.
  • Theoretical cornerstone: its idea of "recurrent connection + context layer" profoundly influenced the subsequent development of RNNs. Designs such as LSTM's gating mechanism and GRU's update gate can be viewed as enhancements of the Elman network's basic recurrent idea.
  • Cognitive-science significance: Elman originally designed this network to study the temporal structure of human language processing. His experiments showed that even a simple recurrent network can learn grammatical rules and word-order patterns, providing strong support for the cognitive modeling of neural networks.
  • Educational value: thanks to its simple, easy-to-understand architecture, the Elman network remains the best introductory model for learning the principles of recurrent neural networks, helping learners build an intuitive understanding of "memory" and "temporal dependency".

🌐Application Scenarios

The Elman network and its variants have found practical applications in many fields:

  • Time-series prediction: stock-price trend forecasting, power-load forecasting, traffic-flow prediction, weather trend analysis, etc.
  • Natural language processing: part-of-speech tagging, named-entity recognition, language modeling, simple machine-translation tasks, etc.
  • Speech processing: phoneme classification, acoustic modeling in speech recognition, speaker recognition, etc.
  • Control systems: nonlinear dynamic-system modeling, controller design, fault detection and diagnosis, etc.
  • Biosignal processing: electroencephalogram (EEG) signal classification, electrocardiogram (ECG) analysis, electromyographic (EMG) signal pattern recognition, etc.
  • Financial risk control: credit-score sequence analysis, behavioral-sequence modeling in fraud detection, etc.

⚠️Limitations and Drawbacks

Despite its important historical status, the Elman network also has clear limitations:

  • Gradient vanishing/exploding problem: during BPTT training, the error gradient is repeatedly multiplied by the recurrent weight matrix along the time dimension, causing gradients to decay rapidly (vanish) or grow sharply (explode), making it hard for the network to learn long-term dependencies. Experiments show the Elman network typically remembers only about 5–10 time steps of information.
  • Limited long-term memory: the context layer's "memory" is a direct copy of the hidden state and lacks a gating mechanism to selectively retain or forget information. For tasks requiring long-term memory (such as long-text understanding), performance drops sharply.
  • Lower training efficiency: BPTT must unroll the network over time before computing gradients, and computational complexity grows linearly with sequence length. For long sequences, training time and memory consumption increase significantly.
  • Sensitivity to hyperparameters: hyperparameters such as learning rate, weight initialization, and activation-function choice strongly affect training results, making tuning difficult.
  • Limited expressiveness: compared with modern architectures such as LSTM and GRU, the Elman network's hidden-state expressiveness is weaker, making it hard to model complex nonlinear temporal dynamics.

These limitations are precisely why LSTM (1997) and GRU (2014) were proposed. But understanding the Elman network is the foundation for understanding these advanced architectures.