Implicit neural
representations

How to store a picture, a sound, a shape or a physical field inside a neural network

Simon Nyholm + Claude Opus 5.5
\( f_\theta(x, y) = (r, g, b) \)
1Background

Neural networks can approximate any function

Universal approximation theorem
For every continuous \( f : [0,1]^n \to \R \) and every \( \varepsilon > 0 \) there is a network \[ \mathcal{N}(x) = \sum_{j=1}^{N} a_j\, \sigma(w_j \cdot x + b_j) \] such that \( |f(x) - \mathcal{N}(x)| < \varepsilon \) for all \( x \).
target \( f \) ReLU network \( \mathcal{N} \)
In reality, \( N \) needs to be very large.
1Background

A function can be much more than a curve

Audio
\( t \mapsto p(t) \)
\( \R \to \R \)
Image red, green, blue
\( (x, y) \mapsto (r, g, b) \)
\( \R^2 \to \R^3 \)
Video
\( (x, y, t) \mapsto (r, g, b) \)
\( \R^3 \to \R^3 \)
3D shape signed distance
\( (x, y, z) \mapsto d \)
\( \R^3 \to \R \)
Left: the surface, where \( d = 0 \). Right: a slice through \( d \), blue inside (\( d<0 \)), orange outside.
Radiance field NeRF
\( (x, y, z, \theta, \phi) \mapsto (r, g, b, \sigma) \)
\( \R^5 \to \R^4 \)
Colour and density \( \sigma \) at every point, seen from direction \( (\theta, \phi) \). Integrating along camera rays renders any new viewpoint.
Physical field wind, temperature
\( (x, y, z, t) \mapsto (\mathbf{u}, T) \)
\( \R^4 \to \R^4 \)
Any information can be encoded by a function \( f \). We are interested in mappings between low-dimensional spaces.
1Background

Implicit neural representation

Explicit: a grid of numbers
128 × 128 × 3 = 49,152 numbers
Implicit: a function \( f_\theta \)
Fit with gradient descent: \( \displaystyle \min_\theta \sum_{i} \big\| f_\theta(\mathbf{x}_i) - \mathbf{y}_i \big\|^2 \) (pixel positions \( \mathbf{x}_i \), colors \( \mathbf{y}_i \))
The weights \( \theta \) are the image: query it anywhere, at any resolution.
2Naive learning

Attempt 1: a plain ReLU multilayer perceptron (MLP)

Target128 × 128 · 49k values
step 0
The largest network has 16× more parameters than the image has values — and it is still blurry.
2Naive learning

Learning a 1D signal: the chirp

\( y(t) = \sin\!\big(2\pi(t + 20\,t^2)\big) \)
target ReLU MLP 3 × 256 · 132k ReLU MLP 4 × 512 · 790k
Loss
Spectrum \( |\hat{y}(\xi)| \)
Spectral bias

Networks learn low frequencies first. High frequencies take forever.

3Fourier features

Recall: Fourier series

\( \displaystyle f(x) \approx a_0 + \sum_{k=1}^{K} \big( a_k \cos(2\pi k x) + b_k \sin(2\pi k x) \big) \)
\( \displaystyle \mathbf{z}(t) = \sum_{k} r_k \begin{pmatrix} \cos(kt + \varphi_k) \\ \sin(kt + \varphi_k) \end{pmatrix} \in \R^2 \)
Sinusoids are a very efficient vocabulary. Idea: let the MLP build its function out of sinusoids.
3Fourier features

Fourier features 2020

\( \gamma(\mathbf{x}) = \big[\cos(2\pi B\mathbf{x}),\; \sin(2\pi B\mathbf{x})\big] \in \R^{2m}, \qquad B \in \R^{m \times 2},\; B_{ij} \sim \mathcal{N}(0, \sigma^2) \)
\( f_\theta(\mathbf{x}) = \mathrm{MLP}_\theta\big(\gamma(\mathbf{x})\big) \)
\( (x, y) \)
2 numbers
→
\( \gamma(\mathbf{x}) \): 2m numbers (m = 128 → 256), 8 shown
→
MLP with 2m inputs
→
\( (r, g, b) \)
# the entire trick (PyTorch), x: (N, 2) coordinates
B = sigma * torch.randn(m, 2)       # m × 2, fixed
Bx = 2 * pi * x @ B.T               # (N, m)
gamma = torch.cat([cos(Bx), sin(Bx)], -1)   # (N, 2m)
Fourier series: linear combination of sinusoids
Fourier features: nonlinear combination (an MLP) of random sinusoids
3Fourier features

The chirp again, with Fourier features

\( y(t) = \sin\!\big(2\pi(t + 20\,t^2)\big) \)
target ReLU MLP 132k Fourier features + MLP 3 × 64 · 17k
Loss
Spectrum \( |\hat{y}(\xi)| \)
Fewer parameters, faster learning

…although a chirp is, admittedly, tailor-made for sinusoidal features.

3Fourier features

Attempt 2: Fourier features

Target128 × 128 px
ReLU MLP791k params · 5000 steps
Fourier features + MLP50k params
PSNR—
training steps0
wall-clock time0.0 s
4Applications

Why are INRs cool?

Continuous

Evaluate anywhere, good interpolation.

Differentiable

Exact derivatives \( \nabla_{\mathbf{x}} f_\theta \) by autodiff: normals, PDE residuals (PINNs).

Plugs into physics

Fit through a forward model: \( \min_\theta \| A(f_\theta) - y \|^2 \).

Compact

Store \( \theta \), not samples. Compress images, textures, simulation and weather data.

4Applications

Example: Computed tomography (CT)

X-ray source detector measured projection
Sinogram \( y(\varphi, s) \)
angle \( \varphi \) →
detector \( s \) →
\( y(\varphi, s) = \displaystyle\int \mu\big(s\,\mathbf{n}_\varphi + t\,\mathbf{n}_\varphi^{\perp}\big)\,dt \;=\; (A\mu)(\varphi, s) \)
\( \mu(x, y) \)attenuation: how strongly the material absorbs X-rays at \( (x, y) \)
\( \varphi \)rotation angle of source and detector
\( s \)position along the detector
\( t \)position along the ray, \( \mathbf{n}_\varphi = (\cos\varphi, \sin\varphi) \)
We never observe \( \mu \) itself, only its line integrals. Reconstruction means inverting \( A \).
4Applications

Solving the inverse problem

Inverse problem: we measure \( y = A(\mu) \) for a known forward model \( A \) and want \( \mu \) back.
Fitting an image\( \displaystyle \min_\theta \; \big\| {\color{#2a78d6} f_\theta} - y \big\|^2 \)
Inverse problem\( \displaystyle \min_\theta \; \big\| {\color{#b07800} A}\big({\color{#2a78d6} f_\theta}\big) - y \big\|^2 \)
\( f_\theta \) is our current best guess of the object.
We have a differentiable model for \( A \).
Gradients flow through \( A \) back to \( \theta \).
4Applications

Sparse-view CT from 30 projections

Ground truth (unknown in practice)
Measurements \( y \)
Filtered back-projection classical: sharpen each projection, then smear it back along its rays
INR \( f_\theta \)
step 0
115 views
230 views
390 views
4Applications

Other problems

\( \displaystyle \min_\theta \; \big\| {\color{#b07800} A}\big({\color{#2a78d6} f_\theta}\big) - y \big\|^2 \)
\( f_\theta \) is…forward model \( A \)data \( y \)
CTattenuation \( \mu(x, y, z) \)line integralsX-ray projections
NeRFcolor & density \( (x, y, z, \theta, \phi) \)volume rendering along camera raysphotos
PINNsPDE solution \( u(x, t) \)PDE residual, boundary conditionszero residual, sensor data
Phase retrievalaudio signal \( s(t) \)spectrogram \( |\mathrm{STFT}| \)magnitude spectrogram
Super-resolutionimage \( I(x, y) \)blur + downsamplelow-res image
4Applications

Neural texture compression

A material on a 3D model is many textures stacked: 9+ channels at 4096 × 4096.

  • Albedo (3) base colour
  • Normal (3) surface direction: fake bumps
  • Roughness (1) matte vs. shiny
  • Ambient occlusion (1) shadow in crevices
  • Height (1) displacement
\( f_\theta(u, v, \text{lod}) \)

One small network per material (latent grid + tiny MLP) decodes all channels at once, per pixel, inside the shader.

5Summary

Takeaways

  1. 1
    Images, sound, video, shapes and physical fields are all functions \( \R^n \to \R^m \).
  2. 2
    An INR stores such a function as a small neural network \( f_\theta \).
  3. 3
    Plain MLPs are biased towards low frequencies. Fourier features fix this in three lines of code.
  4. 4
    INRs are compact, continuous, differentiable — and can be fitted through physics: \( \min_\theta \|A(f_\theta) - y\|^2 \).

That's it!