Are LLMs perception-based modal or non-perception-based amodal conceptual systems?

Large Language Models (LLMs) are fundamentally non-perception-based, **amodal conceptual systems** in their core architecture and training mechanics, though they frequently learn statistical approximations of modal structure. In cognitive science, modal (or grounded) systems posit that concepts are intrinsically tied to sensory-motor simulations of the physical world. Conversely, amodal systems treat concepts as abstract, arbitrary symbols processed through formal relations, detached from any specific sensory modality. Because standard LLMs are trained exclusively on next-token prediction over disembodied text corpora, they lack sensory organs and direct physical interaction. However, because human language itself encodes rich statistical footprints of perceptual experience, LLMs construct high-dimensional vector spaces that capture relational properties resembling modal cognition, blurring the historical boundary between pure amodal symbol manipulation and grounded representation. ## Theoretical Framework: Modal versus Amodal Systems To evaluate where LLMs fit, one must distinguish the two primary paradigms of conceptual representation in cognitive psychology: * **Modal (Grounded) Conceptual Systems:** Championed by theorists like Lawrence Barsalou, this framework argues that semantic knowledge is reenacted across the brain's sensory-motor modalities. Understanding the concept of a "kick," for example, involves neural simulations in motor and premotor areas. * **Amodal Conceptual Systems:** Tracing back to classical computationalism and Jerry Fodor’s Language of Thought, this approach posits that concepts are represented in an abstract, language-like format distinct from sensory inputs. Symbols derive their function from formal syntactic rules rather than experiential grounding. When applied to artificial intelligence, these frameworks intersect with Stevan Harnad’s classical "symbol grounding problem"—the dilemma of how arbitrary computational symbols acquire meaning without external referents. ## The Architecture and Status of Large Language Models Unimodal LLMs operate entirely within a linguistic manifold derived from the distributional hypothesis—the linguistic principle that words occurring in similar contexts tend to have similar meanings. Structurally, this aligns them with amodal systems. They do not possess a body, navigate physical space, or experience interoception; their tokens are indices within a vast mathematical matrix rather than reactivations of sensory states. Yet, treating LLMs as *purely* amodal overlooks how thoroughly human language mirrors perception. Language does not float entirely free of reality; it systematically records perceptual regularities. Consequently, an LLM’s hidden layers develop geometric topologies where semantic distances mirror physical and perceptual realities. While the model lacks visual qualia, its latent space organizes concepts such that "hot" and "cold," or "loud" and "quiet," occupy structured relational coordinates derived from textual co-occurrence. ## Empirical Illustration: Representing Color Space Consider how an LLM handles abstract and sensory concepts like color. In a modal human mind, the concept of "yellow" activates visual simulations of wavelengths associated with lemons or the sun. In a traditional amodal symbol system, "yellow" is a categorical marker linked to logical propositions. An LLM approaches "yellow" through neither direct perception nor arbitrary logic, but via distributional proximity. In the model’s high-dimensional embedding space, "yellow" is clustered closely with "lemon," "banana," and "sunshine," while maintaining a quantifiable vector distance from "midnight" or "navy." Although the system cannot "see" yellow, the geometry of its parameter space reconstructs a functional analogue of human color categorization based entirely on text-based statistical regularities. ## Epistemic Limits and the Multimodal Turn The status of LLMs as amodal systems remains a subject of intense debate among cognitive scientists and computer scientists. Well-supported evidence shows that text-only LLMs suffer from systematic failures in physical reasoning—such as naive physics or spatial navigation—because text is a compressed, symbolic projection of reality rather than reality itself. However, the rapid proliferation of Multimodal Large Language Models (MLLMs) complicates this classification. By integrating vision encoders, audio processors, and action loops with a core transformer architecture, MLLMs ingest pixel and waveform data directly. In these hybrid systems, the LLM acts as an amodal semantic glue that coordinates multiple modality-specific encoders. Whether this functional integration achieves genuine grounding or merely sophisticated data fusion remains the central unresolved question in contemporary cognitive AI. ## Follow-up questions 1. How do distributional statistics in text-only models simulate properties of modal conceptual spaces without direct sensory input? 2. In what ways do Multimodal Large Language Models (MLLMs) alter the traditional symbol grounding problem compared to unimodal text transformers? 3. What are the primary epistemological limits of treating statistical co-occurrence patterns in language as a valid substitute for embodied conceptual grounding?