Your Brain's Rendering Pipeline,
in 2000+ Lines of GLSL⚙

The Pipeline: Biology → GLSL

The visual system is vastly more complex than any current simulation captures — dozens of cortical areas, parallel processing streams, recurrent feedback loops. The shader models four stages chosen for their outsized impact on peripheral appearance: retinal spatial filtering, LGN attentional gating, V1 feature integration, and V4 color processing. Eccentricity from the gaze point drives every parameter.

Biological Primer

The retina is not a passive sensor — it's a layered neural circuit.

The sections below summarize the minimum context for following the simulation. The key structures — retina, LGN, V1, V4 — have been well characterized since Kuffler (1953) and Hubel & Wiesel (1962).

The Retina: Where Vision Begins

Light hits photoreceptors (rods and cones), which convert photons to electrical signals. These signals pass through bipolar cells to retinal ganglion cells, whose axons form the optic nerve.

In the fovea — a 1.5mm pit at the retina's center, densely packed with cones — each cone connects to its own ganglion cell (1:1 wiring). In the periphery, 100+ rods converge onto a single ganglion cell, trading spatial resolution for sensitivity. This convergence ratio is the fundamental reason peripheral vision loses detail — signals are pooled, not lost. You can detect that something is there, but you can't tell what it is.

Receptive Fields and Center-Surround🔬

A neuron's receptive field is the region of visual space that drives its response. Retinal ganglion cells have center-surround receptive fields (Kuffler, 1953): an excitatory center ringed by an inhibitory surround, or vice versa. The Difference-of-Gaussians (DoG) is a mathematical model of this response profile.

Center-surround organization makes these neurons bandpass filters — they respond to spatial frequencies matching their field size, not to uniform illumination. Receptive field size scales with eccentricity: larger fields in the periphery means coarser frequency tuning further from fixation.

The Visual Pathway

Signals leave the retina via the optic nerve and pass through a series of processing stages before reaching cortex.

Lateral Geniculate Nucleus (LGN)

A six-layered thalamic relay. Magnocellular layers (fed by parasol ganglion cells) carry fast, achromatic, motion-sensitive signals. Parvocellular layers (fed by midget ganglion cells) carry slower, color-sensitive, high spatial frequency signals.

V1 (primary visual cortex)

Orientation-selective neurons (Hubel & Wiesel, 1962) decompose the image into local features — edges, bars, gratings at specific angles.

V4

Processes increasingly complex properties: contours, color constancy, object recognition.

Eccentricity and the Three Zones

Eccentricity is the angular distance from fixation, measured in degrees of visual angle. The cortex devotes disproportionate surface area to the fovea — a property called cortical magnification (M-scaling). This is why a letter at 10° eccentricity must be ~5x larger than a foveal letter to be equally legible.

Stage 1: The LGN Gate

"The LGN is not a relay station. It's a gate."
— Sherman & Guillery, The role of the thalamus in the flow of information to the cortex (2002)

The Lateral Geniculate Nucleus sits between retina and cortex. For decades, neuroscientists treated it as a simple relay — signals pass through unchanged. Sherman and Guillery showed it's actually an attentional gatekeeper: top-down signals from cortex modulate what gets through. If you've built information retrieval systems, this is a familiar pattern — a filter stage that decides what enters the processing pipeline at all, before any expensive downstream computation. In the shader, this gating is instantiated as two binary switches and a multiplication. The signal either passes or it doesn't.

Structure Masking⚙

The LGN gate needs to know what kind of content occupies each region of the page — and how that knowledge is represented constrains what operations are natural downstream. A content analysis pass (content-analysis.js) scans the DOM and encodes three signals into a structure map:

• Rhythm — text cadence and repetition (are there regular line breaks, list items, tabular patterns?)
• Density — visual weight per region (dense paragraph vs. whitespace vs. sparse navigation)
• Type — element classification (text, image, interactive control, empty)

When the LGN stage encounters low density — empty whitespace between content — it suppresses the downstream simulation entirely. The biological LGN doesn't gate blank space to cortex; the simulation mirrors this by zeroing the suppression factor for regions with no visual content to process.

// LGN Structure Masking — lines 319–323 of peripheral.frag
if (config.lgn_use_structure_mask) {
    if (u_has_structure > 0.5 && signal.density < 0.1) {
        signal.suppressionFactor = 0.0;  // Skip peripheral simulation for empty space
    }
}

Saliency Gating

The saliency map (edge detection + color contrast, temporally smoothed; pop-out reference) tells the LGN what deserves processing bandwidth. This is the same problem biology solves: the retina captures ~10⁷ bits/sec but the optic nerve transmits ~10⁶. The visual system doesn't "degrade then un-degrade" — it selectively allocates limited processing resources to high-value input. Our simulation mirrors this: salient regions receive reduced peripheral filtering (the simulation's analogue of preferential processing), while low-saliency regions are processed cheaply.

fovea para periph

Dashboard — center focus. The fovea reveals full detail at gaze point. Peripheral regions are filtered through the full LGN→V1→V4 pipeline.

Saliency map. Bright = high saliency = more processing bandwidth allocated. Text blocks, faces, and colored status badges demand the most resources.

Stage 2: V1 Feature Extraction

After the LGN gate, the signal that survives enters V1 — and here the nature of the representation changes fundamentally. Primary visual cortex decomposes the signal into features: orientation, spatial frequency, motion. In the periphery, these features are extracted at progressively lower resolution — receptive fields grow larger, features crowd into each other, and positional certainty degrades.

Scrutinizer's V1 stage implements three key peripheral phenomena: positional uncertainty (domain warping), crowding (feature scrambling), and spatial pooling (resolution loss). The LGN's suppression factor modulates all of them — up to 25% attenuation for salient content.

Domain Warping: Positional Uncertainty

In the periphery, you can detect that something is there but not where exactly it is. Scrutinizer simulates this with fractal noise displacement — two octaves of Simplex noise at 150x and 300x frequency, with an anisotropic 2x radial bias (crowding is more severe along the radial axis toward/away from fixation than tangentially; Toet & Levi 1992).

Crowding: The Fundamental Limit

"Crowding is not blur." — Denis Pelli (2008). You can detect a letter in your periphery. You cannot identify it when flanked by other letters. The bottleneck isn't resolution — it's obligatory feature integration across the receptive field.

This is arguably the most important phenomenon in peripheral vision — and it's an information-theoretic one (crowding reference stimuli, Bouma spacing). Crowding determines not what you can't see, but what you can't separate. Adjacent features are compulsorily pooled — averaged, texture-ified — once they share a receptive field. The information isn't gone; it's been irreversibly mixed. The individual signals are still in there, but the representation can no longer distinguish them. Bouma's law quantifies the critical spacing:

The shader implements two crowding modes:

• Fractal Noise — Continuous warp via Simplex noise, increasing amplitude with eccentricity. Preserves overall structure but scrambles local features.
• Discrete Scramble — Grid-based cell shredding. Each cell gets a random displacement vector, creating a "shattered glass" effect. Saliency guides where the grid breaks.

// V1 Discrete Scramble — lines 403–433 of peripheral.frag
float scrambleZone = smoothstep(parafovea * 1.0, parafovea * 1.5, dist);
vec2 cellID = floor(uv * cellFreq);
vec2 jitter = hash22(cellID) - 0.5;
vec2 throwDist = vec2(0.008, 0.0016);  // 5:1 H:V ratio
discreteScramble = jitter * throwDist * scrambleZone²;

fovea para periph

Article — center focus. V1 crowding filters paragraph text in the periphery while headlines receive more bandwidth (larger receptive fields).

fovea para periph

E-commerce — product focus. When you fixate on the product image, the price tag, CTA button, and reviews are filtered through V1 scrambling — a visualization of the feature pooling that constrains peripheral object recognition.

Block Pooling: Making Receptive Fields Visible

Two presentation modes make the pooling geometry tangible. Minecraft quantizes the image into blocks sized by cortical magnification — 4px at the fovea, 64px in the far periphery. Each block shows the mean color of its receptive field. MC Eyeball does the same with polar sectors radiating from the gaze point, matching the radial elongation of TTM pooling regions (Toet & Levi 1992).

Minecraft (Block Pooling). Blocks grow with eccentricity, showing the cortical magnification function as visible geometry. Fine blocks at the fovea, coarse blocks in the periphery.

MC Eyeball (Polar Pooling). Wedge-shaped sectors emanate from the gaze point, radially elongated ~2:1. This is what TTM pooling regions look like — the geometry the summary statistics are computed within.

Toward Mongrel Textures: DoG Band Decomposition

By the time the signal reaches the periphery, what does the representation actually contain? Rosenholtz et al. (2012) answered this precisely: summary statistics — local texture-like representations that preserve aggregate properties (mean color, dominant orientation, spatial frequency distribution) while discarding individual feature identity.

A lossy compression where the encoder is the visual system itself.

Rosenholtz's mongrel images, synthesized to match these statistics, are perceptually indistinguishable from the original in the periphery. The catch: synthesizing mongrels takes ~30 minutes per image on current hardware (M3 MacBook with MPS acceleration, 300 optimization iterations).

Real-Time Approximation

The shader's DoG band decomposition approximates this at real-time rates. Rather than computing full texture statistics, it decomposes the image into spatial frequency bands and attenuates them independently based on eccentricity. Fine detail drops out before coarse structure — matching the biological attenuation profile where retinal ganglion cells act as bandpass filters whose size scales with eccentricity (Curcio et al., 1990).

The Approximate Laplacian Pyramid⚙

The shader decomposes the hardware MIP chain into an approximate Laplacian pyramid — each band captures a spatial frequency range:

// 8 Half-Octave DoG Bands — peripheral.frag
// 9 MIP levels at LOD 0.0, 0.5, 1.0, ... 4.0 (half-integer = hardware trilinear)
vec4 mip[9];
mip[0] = textureLod(tex, uv, 0.0);
mip[1] = textureLod(tex, uv, 0.5);
// ... mip[2]–mip[7] at LOD 1.0, 1.5, 2.0, 2.5, 3.0, 3.5
mip[8] = textureLod(tex, uv, 4.0);

vec4 band[8];
band[0] = mip[0] - mip[1];  // ~5.66 cpd: serifs, fine detail
band[1] = mip[1] - mip[2];  // ~4.0 cpd:  thin strokes
// ... band[2]–band[7]: √2-spaced down to ~0.5 cpd (layout blocks)
// residual = mip[8]         // ~0.35 cpd: DC, always preserved

M-Scaling: Cortical Magnification per Band

Each band has its own cutoff eccentricity — the distance from the fovea where that frequency band is fully attenuated. With 8 half-octave bands, the cutoffs follow $\sqrt{2}$ spacing, matching the $E_2$ scaling of cortical magnification (validated against published contrast sensitivity data — see report):

Each band's weight is computed via smoothstep rolloff, with a transition width controlled by u_dog_sharpness. At sharpness=0, the transitions are wide and gradual (biological). At sharpness=1, they're narrow and crisp (engineering utility mode).

// Per-band weights via smoothstep rolloff
float w[8];
for (int k = 0; k < 8; k++) {
    w[k] = 1.0 - smoothstep(c[k] - c[k]*trans, c[k] + c[k]*trans, normEcc);
}

// Reconstruction: residual + weighted bands
result = mip[8];
for (int k = 0; k < 8; k++) { result += band[k] * w[k]; }
result = clamp(result, 0.0, 1.0);

What This Means in Practice

Close to the fovea, all 8 bands survive — full detail. In the parafovea, the finest bands (serifs, thin strokes) drop out first. You can still read words but individual letterforms lose their character. Further out, letter-body bands fade — shapes become blobs. In the far periphery, only the coarsest bands and the DC residual remain: you see buttons and layout blocks but not their labels.

Structure map — article. The content analysis feeds the LGN. Red channel = text rhythm, Green = density, Blue = element type. DoG band weights are modulated by this structure.

fovea para periph

Techmeme — center focus. Dense news layout. DoG preserves headline structure while filtering individual link text. The data table density is exactly the kind of content where DoG outperforms uniform blur.

Fidelity Gap: What DoG Misses

The DoG decomposition captures frequency-selective attenuation but not the full mongrel texture model. True summary statistics include orientation distributions, phase correlations, and cross-scale feature conjunctions that our isotropic bands discard. Adding oriented DoG filters — even just horizontal/vertical — would double the texture lookups but could capture the anisotropy of real V1 receptive fields (Gabor-like, phase-sensitive). The gap between our real-time approximation and a full Rosenholtz-style synthesis is where future work lives.

Stage 3: V4 Color & Aesthetics

Visual area V4 processes color, shape, and object recognition. In the periphery, color perception shifts dramatically — the transition from cone-dominated (foveal) to rod-dominated (peripheral) processing alters hue sensitivity, saturation, and spectral response (per-channel chromatic decay validated — see report). The shader's V4 stage simulates these shifts using Oklab color space, rod-spectrum tinting, and selectable aesthetic rendering modes.

Oklab: Perceptually Uniform Desaturation⚙

A critical design choice. If you desaturate in RGB or HSL, hue shifts as saturation drops — warm colors skew yellow, cool colors skew cyan. Oklab (Ottosson, 2020) separates lightness (L) from chrominance (a, b) in a perceptually uniform space. Killing a and b channels produces clean grayscale without hue contamination.

Chromatic Pooling: Why Blue Outlasts Red

The Oklab desaturation above treats both chrominance channels equally. But biology doesn't. Your visual system has two independent color pathways with very different eccentricity profiles:

• Red-Green (L−M) — a foveal specialization. Sharp at the center, degrades fast. At 15° eccentricity, sensitivity has dropped 71%.
• Blue-Yellow (S−(L+M)) — retina-wide. Works almost everywhere. At 15°, only 21% lost (Bowers, Gegenfurtner & Goettker 2025).

The shader models this asymmetry with independent decay rates per channel. The practical consequence for web design:

Blue and yellow signals persist in the periphery. Red and green degrade early for small elements. Always provide a luminance difference alongside color — the achromatic channel carries the signal regardless of chromatic decay.

Magnocellular Pathway: Luminance Contrast Preservation🔬

The magnocellular (M) pathway — fast, achromatic, motion-sensitive — preserves luminance contrast even as the parvocellular (P) pathway loses color and detail. The shader samples the clean (undistorted) image, computes a luminance ratio, and blends it back into the simulated output. This is why you can still detect edges and motion in your far periphery even when you can't see color or detail.

// Magnocellular Pathway — lines 555–570 of peripheral.frag
float cleanLuma = dot(cleanSample, vec3(0.299, 0.587, 0.114));
float distortedLuma = dot(col, vec3(0.299, 0.587, 0.114));
float lumaRatio = cleanLuma / max(distortedLuma, 0.01);
float preservation = mix(0.6, 0.1,
    smoothstep(0.0, parafovea - fovea, eccentricity));
col *= mix(1.0, lumaRatio, preservation * contrastRamp);

Structure map — Techmeme. The structure analysis that feeds the LGN gate. Dense news sites reveal the rhythm/density/type decomposition that drives the entire pipeline.

Saliency map — Techmeme. Edge detection + color contrast in Oklab space. Bright regions receive more processing bandwidth through the LGN gate.

What the Pipeline Models

Each stage of the rendering pipeline corresponds to a biological mechanism in the early visual system. For implementation details and full citations, see the scientific literature review.

Mobile: The Smaller Fovea

On mobile, the fovea subtends the same visual angle, but the screen is held closer and the viewport is denser. The simulation reveals something counterintuitive: much of the content on a phone screen sits in the parafoveal zone — not the far periphery, but the transitional region where feature integration begins to degrade. This is where crowding first bites.

fovea para periph

iPhone 14 — article. The same article page at mobile scale. Much of the content sits in the parafoveal zone, where feature integration starts to pool individual elements into texture.

fovea para periph

iPhone 14 — dashboard. Data-dense UIs on mobile. The fovea reveals one card; most content falls into parafoveal territory.

fovea para periph

iPad Air — article. Tablet's larger viewport provides a middle ground. More foveal real estate, but still significant parafoveal load.

fovea para periph

iPad Air — dashboard. The data table density makes this an ideal test case for the DoG band decomposition — rows fade by spatial frequency, not uniformly.

Select References

• Bouma, H. (1970). Interaction effects in parafoveal letter recognition. Nature, 226, 177–178.
• Curcio, C. A., et al. (1990). Human photoreceptor topography. J. Comparative Neurology, 292(4), 497–523.
• Hubel, D. H. & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiology, 160(1), 106–154.
• Ottosson, B. (2020). A perceptual color space for image processing. (Oklab specification).
• Pelli, D. G. (2008). Crowding: a cortical constraint on object recognition. Current Opinion in Neurobiology, 18(4), 445–451.
• Rosenholtz, R., et al. (2012). A summary statistic representation in peripheral vision explains visual search. Journal of Vision, 12(4):14.
• Sherman, S. M. & Guillery, R. W. (2002). The role of the thalamus in the flow of information to the cortex. Phil. Trans. R. Soc. B, 357(1428), 1695–1708.

Further Reading

• Annotated Bibliography — each cited paper mapped to its role in the shader implementation
• Foveated Vision Model — detailed theory document for the simulation
• Project Trajectory — 98-day development arc, commit analytics, the revert day
• Visual Cognition — curated talks on peripheral vision, crowding, and visual attention
• Vision Demonstrations — demos, walkthroughs, and design rationale

About the Author

Andy Edmonds studied cognitive science at Carnegie Mellon, Georgia Tech, and Clemson, then left academia in 1995 for what became two decades of machine learning at scale — shipping recommendation systems, search relevance, and information retrieval products at eBay, Adobe, Microsoft, and Meta.

The through-line: how does information flow through a system, and what happens to the signal at each stage? Scrutinizer applies that question to the visual system. The retina decomposes. The LGN gates. V1 pools. V4 recolors — each stage mapping cleanly onto GPU operations.