Boltzmann Machine ยท Interactive Simulation

The restricted Boltzmann machine (RBM) โ€” an energy-based probabilistic generative model that learns a data distribution through Gibbs sampling and contrastive divergence.

Core Formulas and Parameter Details

Energy function

Defines a scalar energy for each system state. The lower the energy, the more stable and more probable the state. In an RBM the energy has three contributions: visible biases, hidden biases and the bidirectional connection weights.

  • v_i โ€” state (0 or 1) of the i-th visible neuron, m in total
  • h_j โ€” state (0 or 1) of the j-th hidden neuron, n in total
  • a_i โ€” bias of the i-th visible neuron
  • b_j โ€” bias of the j-th hidden neuron
  • w_{ij} โ€” connection weight between visible unit i and hidden unit j
Boltzmann distribution

The probability of a particular state (v,h) follows the Boltzmann distribution: lower-energy states are more probable. This is the central idea of statistical mechanics applied directly to machine learning.

  • Z โ€” the partition function (normalizing constant)
  • E โ€” the value of the energy function
  • e โ€” Euler number โ‰ˆ 2.71828
Partition function

Sums over every possible combination of visible and hidden states so that the distribution is normalized (total probability 1). Its cost grows exponentially with the number of neurons, which is the main computational bottleneck of Boltzmann machines.

  • Summation range โ€” covers all 2^(m+n) state combinations
Sigmoid activation function

Maps any real number into (0, 1), turning a neuron total input into an activation probability. In an RBM the sigmoid is the core of every conditional probability computation.

  • x โ€” the neuron total input (bias + weighted sum)
  • Output range โ€” (0, 1), interpretable as a probability
Hidden-layer conditional probability

The probability that hidden unit j is active given the visible state v. Because an RBM has no intra-layer connections, the hidden units are conditionally independent and can be sampled in parallel.

  • b_j โ€” hidden-layer bias
  • w_{ij} โ€” connection weight
  • ฯƒ โ€” the sigmoid function
Visible-layer conditional probability

The probability that visible unit i is active given the hidden state h. It reconstructs the visible data from the hidden representation and is the reverse step of Gibbs sampling.

  • a_i โ€” visible-layer bias
  • h_j โ€” hidden neuron state
Temperature-parametrized distribution

The temperature T controls the sharpness of the distribution. A high temperature flattens it (more random), a low temperature concentrates it on low-energy states (more deterministic). Simulated annealing exploits exactly this property.

  • T โ€” temperature parameter, T > 0
  • T โ†’ 0 โ€” the distribution collapses onto the lowest-energy state
  • T โ†’ โˆž โ€” the distribution approaches uniform
Contrastive divergence update rule

The CD algorithm proposed by Hinton approximates the gradient by the difference between the data-driven positive-phase statistics and the negative-phase statistics from a single Gibbs reconstruction, avoiding the expensive partition function.

  • ฮท โ€” learning rate
  • โŸจยทโŸฉ_data โ€” expectation under the data distribution
  • โŸจยทโŸฉ_recon โ€” expectation under the reconstruction distribution
Log-likelihood gradient

The exact gradient of the training-data log-likelihood. The first term comes from the data distribution and the second from the model distribution; CD approximates the model distribution with the reconstruction distribution.

  • P(v) โ€” marginal probability of the visible layer
  • model โ€” the distribution obtained by sampling from the model itself
Free energy

The effective energy after summing out the hidden layer; it measures the cost of a visible state v. The lower the free energy, the better the state fits the learned distribution, which makes it useful for anomaly detection.

  • F(v) โ€” free energy of the visible state v
  • Summation โ€” over all possible hidden-layer states
KL divergence

Measures the difference between the model distribution and the data distribution. Training an RBM means minimizing this divergence so that the model distribution approaches the true data distribution.

  • P_data โ€” the empirical distribution of the training data
  • P_model โ€” the distribution learned by the RBM
  • D_KL โ‰ฅ 0 โ€” zero if and only if the two distributions are identical

Interactive RBM Training Simulation

Ready ยท click "Start Training"
Network Structure
Training Energy / Reconstruction Error
Weight Matrix Heatmap
Current Input vs Reconstruction

A Deeper Look at Boltzmann Machines

Standard Introduction

The Boltzmann machine (BM) is a stochastic energy-based neural network model introduced by Geoffrey Hinton and Terry Sejnowski in 1985; its name comes from the Boltzmann distribution in statistical mechanics. A BM consists of symmetrically connected binary neurons, each taking the state 0 or 1, and the joint distribution over states is defined by an energy function through the Boltzmann distribution. The restricted Boltzmann machine (RBM) is an important variant whose units are split into a visible layer and a hidden layer, with no intra-layer connections and full inter-layer connections. This structure makes conditional probabilities and Gibbs sampling efficient. RBMs are trained with contrastive divergence (CD), can learn the underlying probability distribution of the data, and serve as the basic building block of deep belief networks (DBN) and deep Boltzmann machines (DBM).

Plain-Language Introduction

Imagine a system built from many little switches (neurons), each either on (1) or off (0). The system has a notion of energy: some switch combinations give low energy (stable, comfortable) and others give high energy (unstable). What a Boltzmann machine does is keep adjusting the connection strengths (weights) between the switches until the combinations the system likes to settle into correspond exactly to the patterns we want it to learn. Once training is done, if you give it an incomplete pattern it will automatically complete it into the closest full pattern โ€” much as you can guess a whole face from half of one, because the machine has learned the probability distribution over the pattern โ€œfaceโ€.

What Problem Does It Solve

  • Probabilistic modelling:it learns the full data distribution rather than only a discriminative prediction.
  • Feature extraction:the hidden layer learns high-level abstract features automatically, with no manual labels.
  • Missing-data completion:Gibbs sampling can infer missing values from partial observations.
  • Generative modelling:it can generate new samples that resemble the training distribution.
  • Dimensionality reduction and representation learning:high-dimensional data is compressed into a low-dimensional hidden representation.

Why It Matters

Boltzmann machines are one of the cornerstones of the deep learning revival. In 2006, Hinton and colleagues stacked RBMs into deep belief networks (DBN), solving the problem of training deep networks and triggering the deep learning revolution. The contrastive divergence algorithm gave RBMs an efficient approximate training method that avoids the exponential cost of computing the partition function. Boltzmann machines also fused statistical mechanics with machine learning, laying the theoretical groundwork for later energy-based models, generative adversarial networks (GANs) and variational autoencoders (VAEs). They remain a key bridge for understanding the connection between probabilistic graphical models and neural networks.

Application Scenarios

  • Collaborative filtering and recommender systems:RBMs were used for movie-rating prediction in the Netflix Prize.
  • Image denoising and inpainting:recovering the original signal from a noisy image.
  • Handwritten digit recognition:a classic benchmark model on MNIST.
  • Speech recognition:learning hierarchical feature representations of speech signals.
  • Bioinformatics:gene-expression analysis and protein structure prediction.
  • Anomaly detection:using the free energy to flag samples that do not fit the training distribution.
  • Pre-training deep belief networks:providing deep networks with a good weight initialization.

Limitations and Drawbacks

  • The partition function is intractable:exact computation requires enumerating every state at O(2^(m+n)), so only approximations are possible.
  • Slow training:CD needs many Gibbs sampling steps and converges more slowly than modern optimizers.
  • Binary restriction:standard BM/RBM handle binary data only; extending them to continuous data needs special design.
  • Superseded by newer models:for generative modelling, GANs and VAEs usually produce better image quality.
  • Hyperparameter sensitivity:learning rate, temperature and hidden-layer size strongly affect performance and need careful tuning.
  • Poor sampler mixing:in high-dimensional spaces Gibbs sampling can get stuck in local modes and mixes slowly.