Your Brain's Rendering Pipeline,
in 2000+ Lines of GLSL⚙

The Pipeline: Biology → GLSL

The visual system is vastly more complex than any current simulation captures: dozens of cortical areas, parallel processing streams, recurrent feedback loops. The shader models four stages chosen for their outsized impact on peripheral appearance: retinal spatial filtering, LGN attentional gating, V1 feature integration, and V4 color processing. Every parameter depends on eccentricity from the gaze point.

Biological Primer

The retina is a layered neural circuit.

The sections below summarize the minimum context for following the simulation. The structures involved (retina, LGN, V1, V4) have been well characterized since Kuffler (1953) and Hubel & Wiesel (1962).

The Retina: Where Vision Begins

Light hits photoreceptors (rods and cones), which convert photons to electrical signals. These signals pass through bipolar cells to retinal ganglion cells, whose axons form the optic nerve.

In the fovea, a 1.5mm pit at the retina's center densely packed with cones, each cone connects to its own ganglion cell (1:1 wiring). In the periphery, each ganglion cell pools many cones (and 100+ rods, whose signals dominate only in dim light), trading spatial resolution for sensitivity. This convergence ratio is the fundamental reason peripheral vision loses detail. Because signals are pooled, you can detect that something is there but can't tell what it is.

Receptive Fields and Center-Surround🔬

A neuron's receptive field is the region of visual space that drives its response. Retinal ganglion cells have center-surround receptive fields (Kuffler, 1953): an excitatory center ringed by an inhibitory surround, or vice versa. The Difference-of-Gaussians (DoG) is a mathematical model of this response profile.

Center-surround organization makes these neurons bandpass filters. They respond to spatial frequencies matching their field size and respond only weakly to uniform illumination. Receptive field size scales with eccentricity: larger fields in the periphery means coarser frequency tuning further from fixation.

The Visual Pathway

Signals leave the retina via the optic nerve and pass through a series of processing stages before reaching cortex.

Lateral Geniculate Nucleus (LGN)

A six-layered thalamic relay. Magnocellular layers (fed by parasol ganglion cells) carry fast, achromatic, motion-sensitive signals. Parvocellular layers (fed by midget ganglion cells) carry slower, color-sensitive, high spatial frequency signals.

V1 (primary visual cortex)

Orientation-selective neurons (Hubel & Wiesel, 1962) decompose the image into local features: edges, bars, gratings at specific angles.

V4

Processes increasingly complex properties: contours, color constancy, object recognition.

Eccentricity and the Three Zones

Eccentricity is the angular distance from fixation, measured in degrees of visual angle. The cortex devotes disproportionate surface area to the fovea, a property called cortical magnification (M-scaling). This is why a letter at 10° eccentricity must be ~5x larger than a foveal letter to be equally legible.

Stage 1: The LGN Gate

The LGN is a relay with a gate: only 5–10% of the input to its relay cells comes from the retina, and the rest is modulatory.
Paraphrased from Sherman & Guillery, The role of the thalamus in the flow of information to the cortex (2002)

The Lateral Geniculate Nucleus sits between retina and cortex. For decades, neuroscientists treated it as a simple relay that passes signals through unchanged. Sherman and Guillery showed that only 5–10% of the input to LGN relay cells comes from the retina; the rest is modulatory, including feedback from layer 6 of visual cortex, and it controls what gets through. If you've built information retrieval systems, this is a familiar pattern: a filter stage that selects what enters the processing pipeline before any expensive downstream computation. In the shader, this gating is instantiated as two binary switches and a multiplication. The structure mask is all-or-nothing; the saliency gate is graded.

Structure Masking⚙

The LGN stage needs to know what kind of content occupies each region of the page, and how that knowledge is represented constrains which operations are natural downstream. A content analysis pass (content-analysis.js) scans the DOM and encodes three signals into a structure map:

• Rhythm: text cadence and repetition (are there regular line breaks, list items, tabular patterns?)
• Density: visual weight per region (dense paragraph vs. whitespace vs. sparse navigation)
• Type: element classification (text, image, interactive control, empty)

When the LGN stage encounters low density (empty whitespace between content), it suppresses the downstream simulation entirely. Center-surround cells in the retina and LGN respond weakly to uniform regions, so blank space carries little signal; the simulation skips it by zeroing the suppression factor for regions with no visual content to process.

// LGN Structure Masking — processLGN() in peripheral.frag
if (config.lgn_use_structure_mask) {
    if (u_has_structure > 0.5 && signal.density < 0.1) {
        signal.suppressionFactor = 0.0;  // Skip peripheral simulation for empty space
    }
}

Saliency Gating

The LGN stage reads the saliency map (edge detection + color contrast, temporally smoothed; pop-out reference) to decide what deserves processing bandwidth. The visual system faces the same constraint. About 10¹⁰ bits/sec reach the retina, but only ~10⁶–10⁷ leave it through the optic nerve (Raichle 2015; Koch et al. 2006). The visual system allocates limited processing resources selectively to high-value input. In the simulation, salient regions receive less peripheral filtering (its analogue of preferential processing), and low-saliency regions are processed cheaply.

fovea para periph

Dashboard, center focus. Full detail is visible at the gaze point. Peripheral regions are filtered through the full LGN→V1→V4 pipeline.

Saliency map. Bright = high saliency = more processing bandwidth allocated. Text blocks, faces, and colored status badges get the most resources.

Stage 2: V1 Feature Extraction

After the LGN gate, the signal that survives enters V1, where the representation changes in kind. Primary visual cortex decomposes the signal into features: orientation, spatial frequency, motion. In the periphery, these features are extracted at progressively lower resolution: receptive fields grow larger, features crowd into each other, and positional certainty degrades.

Scrutinizer's V1 stage implements three peripheral phenomena: positional uncertainty (domain warping), crowding (feature scrambling), and spatial pooling (resolution loss). The LGN stage's suppression factor modulates all of them, with up to a 70% reduction for the most salient content.

Domain Warping: Positional Uncertainty

In the periphery, you can detect that something is there but not where exactly it is. Scrutinizer simulates this with fractal noise displacement: two octaves of Simplex noise at 150x and 300x frequency, with a 2x horizontal bias that approximates the radial anisotropy of crowding (crowding is more severe along the radial axis toward/away from fixation than tangentially; Toet & Levi 1992).

Crowding: The Fundamental Limit

Crowding is not blur (Pelli, 2008). You can detect a letter in your periphery. You cannot identify it when flanked by other letters. The bottleneck is obligatory feature integration across the receptive field.

Crowding is arguably the most important phenomenon in peripheral vision, and it is an information-theoretic one (crowding reference stimuli, Bouma spacing). It limits what you can separate. Adjacent features are compulsorily pooled (averaged, texture-ified) once they share a receptive field. The individual signals are irreversibly mixed, and the representation can no longer distinguish them. Bouma's law quantifies the critical spacing:

The shader implements two crowding modes:

• Fractal Noise: Continuous warp via Simplex noise, increasing amplitude with eccentricity. Preserves overall structure but scrambles local features.
• Discrete Scramble: Grid-based cell shredding. Each cell gets a random displacement vector, creating a "shattered glass" effect. High-saliency content is displaced less.

// V1 Discrete Scramble — processV1() and applyCutter() in peripheral.frag
float scrambleZone = smoothstep(0.02, 0.15, corticalStrength);
vec2 cellID = floor(uv * cellFreq);
vec2 jitter = hash22(cellID) - 0.5;
vec2 throwDist = vec2(0.008, 0.0016);  // 5:1 H:V ratio
discreteScramble = jitter * throwDist * scrambleZone²;

fovea para periph

Article, center focus. The V1 crowding stage filters paragraph text in the periphery while headlines receive more bandwidth (larger receptive fields).

fovea para periph

E-commerce, product focus. When you fixate on the product image, the price tag, CTA button, and reviews are filtered through the V1 scrambling stage, which visualizes the feature pooling that limits peripheral object recognition.

Block Pooling: Making Receptive Fields Visible

Two presentation modes make the pooling geometry tangible. Minecraft quantizes the image into blocks sized by cortical magnification, from 4px at the fovea to 64px in the far periphery. Each block shows the mean color of its receptive field. MC Eyeball does the same with polar sectors radiating from the gaze point, matching the radial elongation of TTM pooling regions (Toet & Levi 1992).

Minecraft (Block Pooling). Blocks grow with eccentricity, showing the cortical magnification function as visible geometry. Fine blocks at the fovea, coarse blocks in the periphery.

MC Eyeball (Polar Pooling). Wedge-shaped sectors emanate from the gaze point, radially elongated ~2:1. This is the shape of TTM pooling regions, the geometry within which the summary statistics are computed.

Toward Mongrel Textures: DoG Band Decomposition

What does the peripheral representation contain? Rosenholtz et al. (2012) describe it as summary statistics: local texture-like representations that preserve aggregate properties (mean color, dominant orientation, spatial frequency distribution) while discarding individual feature identity.

A lossy compression where the encoder is the visual system itself.

Rosenholtz's mongrel images, synthesized to match these statistics, show what that representation preserves, and observers' performance on mongrels predicts their peripheral performance in crowding tasks (Balas, Nakano & Rosenholtz, 2009). Synthesizing mongrels takes ~30 minutes per image on current hardware (M3 MacBook with MPS acceleration, 300 optimization iterations).

Real-Time Approximation

The shader's DoG band decomposition approximates this at real-time rates. It decomposes the image into spatial frequency bands and attenuates them independently based on eccentricity, without computing full texture statistics. Fine detail drops out before coarse structure, matching the biological attenuation profile in which retinal ganglion cells act as bandpass filters whose size scales with eccentricity (Dacey & Petersen, 1992).

The Approximate Laplacian Pyramid⚙

The shader decomposes the hardware MIP chain into an approximate Laplacian pyramid of 12 half-octave bands, two per octave. Each captures a spatial frequency range (the four finest octaves are shown):

// 12 Half-Octave DoG Bands — peripheral.frag
// 13 MIP levels at LOD 0.0, 0.5, 1.0, ... 6.0 (half-integer = hardware trilinear)
vec4 mip[13];
mip[0] = textureLod(tex, uv, 0.0);
mip[1] = textureLod(tex, uv, 0.5);
// ... mip[2]–mip[11] at LOD 1.0, 1.5, ... 5.5
mip[12] = textureLod(tex, uv, 6.0);

vec4 band[12];
band[0] = mip[0] - mip[1];  // ~5.66 cpd: serifs, fine detail
band[1] = mip[1] - mip[2];  // ~4.0 cpd:  thin strokes
// ... band[2]–band[11]: √2-spaced down to ~0.125 cpd (full-width color fields)
// residual = mip[12]        // ~0.088 cpd: DC, always preserved

M-Scaling: Cortical Magnification per Band

Each band has its own cutoff eccentricity, the distance from the fovea where that frequency band is fully attenuated. With 12 half-octave bands, the cutoffs follow $\sqrt{2}$ spacing, matching the $E_2$ scaling of cortical magnification (tested against Rovamo & Virsu's 1979 contrast sensitivity data: frequency ordering holds, but the fit is partial, r = 0.60 against a 0.9 target; see report):

Each band's weight is computed via smoothstep rolloff, with a transition width controlled by u_dog_sharpness. At sharpness=0, the transitions are wide and gradual (biological). At sharpness=1, they're narrow and crisp (engineering utility mode).

// Per-band weights via smoothstep rolloff
float w[12];
for (int k = 0; k < 12; k++) {
    w[k] = 1.0 - smoothstep(c[k] - c[k]*trans, c[k] + c[k]*trans, normEcc);
}

// Reconstruction: residual + weighted bands
result = mip[12];
for (int k = 0; k < 12; k++) { result += band[k] * w[k]; }
result = clamp(result, 0.0, 1.0);

What This Means in Practice

Close to the fovea, all 12 bands survive, so detail is complete. In the parafovea, the finest bands (serifs, thin strokes) drop out first. You can still read words but individual letterforms lose their character. Further out, letter-body bands fade and shapes become blobs. In the far periphery, only the coarsest bands and the DC residual remain: you see buttons and layout blocks but not their labels.

Structure map, article. The LGN stage reads this content analysis. Red channel = text rhythm, Green = density, Blue = element type. DoG band weights are modulated by this structure.

fovea para periph

Techmeme, center focus. Dense news layout. The DoG stage preserves headline structure while filtering individual link text. The data table density is exactly the kind of content where DoG outperforms uniform blur.

Fidelity Gap: What DoG Misses

The DoG decomposition models frequency-selective attenuation. It does not implement the full mongrel texture model. True summary statistics include orientation distributions, phase correlations, and cross-scale feature conjunctions that our isotropic bands discard. The shader does measure local orientation energy (horizontal, vertical, 45°, 135°) and extends band cutoffs for cardinal edges to model the oblique effect, but the bands themselves are not oriented filters and cannot capture the Gabor-like, phase-sensitive receptive fields of real V1. Closing the gap between our real-time approximation and a full Rosenholtz-style synthesis is future work.

Stage 3: V4 Color & Aesthetics

Visual area V4 processes color, shape, and object recognition. In the periphery, color perception shifts: cone-opponent sensitivity falls with eccentricity, red-green (L−M) much faster than blue-yellow, which alters hue and saturation (per-channel chromatic decay validated; see report). The shader's V4 stage simulates these shifts using Oklab color space, rod-spectrum tinting, and selectable aesthetic rendering modes.

Oklab: Perceptually Uniform Desaturation⚙

If you desaturate in RGB or HSL, hue shifts as saturation drops: warm colors skew yellow, cool colors skew cyan. Oklab (Ottosson, 2020) separates lightness (L) from chrominance (a, b) in a perceptually uniform space. Killing a and b channels produces clean grayscale without hue contamination.

Chromatic Pooling: Why Blue Outlasts Red

The Oklab desaturation above treats both chrominance channels equally. The visual system has two independent color pathways with very different eccentricity profiles:

• Red-Green (L−M): a foveal specialization. Sharp at the center, degrades fast. From 5° to 15° eccentricity, detection sensitivity drops 71%.
• Blue-Yellow (S−(L+M)): retina-wide. Works almost everywhere. Over the same range, it drops only 21% (Bowers, Gegenfurtner & Goettker 2025).

The shader models this asymmetry with independent decay rates per channel. The table below lists practical consequences for web design.

Blue and yellow signals persist in the periphery. Red and green degrade early for small elements. Always provide a luminance difference alongside color, because luminance contrast survives chromatic decay.

Magnocellular Pathway: Luminance Contrast Preservation🔬

The magnocellular (M) pathway is fast, achromatic, and motion-sensitive, and it preserves luminance contrast even as the parvocellular (P) pathway loses color and detail. The shader simulates this by sampling the clean (undistorted) image, computing a luminance ratio, and blending it back into the simulated output.

// Magnocellular Pathway — processV4() in peripheral.frag
float cleanLuma = dot(cleanSample, vec3(0.299, 0.587, 0.114));
float distortedLuma = dot(col, vec3(0.299, 0.587, 0.114));
float lumaRatio = cleanLuma / max(distortedLuma, 0.01);
float contrastPreservation = mix(0.6, 0.1, t);  // t = smoothstep(0, 4 × fovea, eccentricity)
col *= mix(1.0, lumaRatio, contrastPreservation * t);

Structure map, Techmeme. The structure analysis read by the LGN gate stage. Dense news sites show the rhythm/density/type decomposition that the rest of the pipeline uses.

Saliency map, Techmeme. Edge detection + color contrast in Oklab space. Bright regions receive more processing bandwidth through the simulated LGN gate.

What the Pipeline Models

Each stage of the rendering pipeline corresponds to a biological mechanism in the early visual system. For implementation details and full citations, see the scientific literature review.

Mobile: The Smaller Fovea

On mobile, the fovea subtends the same visual angle, but the screen is held closer and the viewport is denser. In the simulation, much of the content on a phone screen sits in the parafoveal zone, the transitional region where feature integration begins to degrade and crowding begins.

fovea para periph

iPhone 14, article. The same article page at mobile scale. Much of the content sits in the parafoveal zone, where feature integration starts to pool individual elements into texture.

fovea para periph

iPhone 14, dashboard. Data-dense UIs on mobile. Only one card falls in the fovea; most content is in parafoveal territory.

fovea para periph

iPad Air, article. Tablet's larger viewport provides a middle ground. More foveal real estate, but still significant parafoveal load.

fovea para periph

iPad Air, dashboard. The dense data table is an ideal test case for the DoG band decomposition. Rows fade by spatial frequency.

Select References

• Bouma, H. (1970). Interaction effects in parafoveal letter recognition. Nature, 226, 177–178.
• Curcio, C. A., et al. (1990). Human photoreceptor topography. J. Comparative Neurology, 292(4), 497–523.
• Hubel, D. H. & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiology, 160(1), 106–154.
• Ottosson, B. (2020). A perceptual color space for image processing. (Oklab specification).
• Pelli, D. G. (2008). Crowding: a cortical constraint on object recognition. Current Opinion in Neurobiology, 18(4), 445–451.
• Rosenholtz, R., et al. (2012). A summary statistic representation in peripheral vision explains visual search. Journal of Vision, 12(4):14.
• Sherman, S. M. & Guillery, R. W. (2002). The role of the thalamus in the flow of information to the cortex. Phil. Trans. R. Soc. B, 357(1428), 1695–1708.

Further Reading

• Annotated Bibliography: each cited paper mapped to its role in the shader implementation
• Foveated Vision Model: detailed theory document for the simulation
• Project Trajectory: 98-day development arc, commit analytics, the revert day
• Visual Cognition: curated talks on peripheral vision, crowding, and visual attention
• Vision Demonstrations: demos, walkthroughs, and design rationale

About the Author

Andy Edmonds studied cognitive science at Carnegie Mellon, Georgia Tech, and Clemson, then left academia in 1995 for what became two decades of machine learning at scale, shipping recommendation systems, search relevance, and information retrieval products at eBay, Adobe, Microsoft, and Meta.

Scrutinizer applies a question from that work to the visual system: how does information flow through a system, and what happens to the signal at each stage? The retina decomposes, the LGN gates, V1 pools, and V4 recolors, and each stage maps onto GPU operations.