Artificial Intelligence is often marketed as a sudden breakthrough of the 2020s. But for those of us engineering LLM applications or deploying neural architectures in production, treating AI as a “magic black box” is a liability. To truly understand why modern Transformers hallucinate, why they are bad at math, or why “Prompt Engineering” even exists, we must look at the architectural lineage of machine thought.

We are dissecting the evolution of engineering constraints. From the deterministic rigidness of expert systems to the probabilistic fluidness of modern weights, every shift was driven by a limitation in either compute, data, or mathematical optimization.

1. The Mechanization of Reason (Combinatorial Search)

Before we had binary code, we had the concept of the Symbol. The earliest “AI” wasn’t about numbers; it was about mechanizing logic. In the 13th century, Ramon Llull proposed a radical idea: if human thought consists of combining basic concepts (like “Truth,” “Power,” or “Wisdom”), then a machine could generate all possible truths by mechanically combining these symbols. This is the great-grandfather of Knowledge Graphs and generative combinatorics.

Llull’s Ars Magna used rotating paper discs to synthesize concepts. While seemingly primitive, this established the core component of modern AI: Combinatorial Search. Llull believed that the universe followed a logical grammar that could be mapped. This paved the way for Gottfried Leibniz, who dreamed of a Characteristica Universalis—a universal language of thought where disputes could be settled by “let us calculate” (Calculemus) rather than arguing.

The Engineering Trade-off

The limitation of Llullian logic was Explosion. The number of possible combinations grows factorially. Without a way to “prune” the search tree, a machine would spend eternity generating nonsense mixed with truth. This is a problem we still face today with “sampling” in LLMs—if we don’t guide the output, the model drifts into incoherence.

2. The Universal Machine & The Software Abstraction

Fast forward to 1936. Alan Turing formalized the “Algorithm.” Before Turing, a computing machine was physically built for one task (like a loom or a calculator). Turing proved that you could build a Universal Machine that could simulate any other machine if given the right instructions (software).

This “Universal Turing Machine” (UTM) introduced the concept of the Stored Program. This is the theoretical basis for why your GPU can run a video game, then train a neural network, then mine crypto. Hardware is the substrate; the intelligence is stored on the instruction tape (which, in a modern sense, are the weights and biases of a model).

From an engineering perspective, Turing’s work meant that intelligence could be abstracted from biology. If thought is a series of state changes on a tape, then intelligence is “platform-independent.” This spawned the Physical Symbol System Hypothesis: the belief that any system capable of manipulating symbols can, in theory, achieve human-level intelligence.

3. Why Symbolic AI Failed in Real-World Systems

From the 1950s to the 1980s, AI was dominated by Symbolic AI (often called GOFAI – Good Old Fashioned AI). The premise was simple: “Intelligence is just rule-following.” If we could just write down all the rules of reality in a logic-based language like LISP or Prolog, we would have a brain.

Engineers built Expert Systems using rigid IF-THEN logic. In the 1980s, companies like Digital Equipment Corporation (DEC) used systems like XCON to configure complex computer orders. It saved them millions, but it hit a wall.

The Collapse: The “Brittleness” Factor

These systems failed because of the Frame Problem and the Knowledge Acquisition Bottleneck. In the real world, the number of rules is infinite. If you program a robot to “make coffee,” you also have to tell it “don’t burn the house down,” “don’t use toilet water,” and “gravity exists.”

Symbolic AI was brittle. If it encountered a scenario 1% outside its programmed rules, it didn’t fail gracefully—it crashed or outputted absolute nonsense. There was no “common sense” to fill the gaps between the logical nodes. By the late 80s, the “AI Winter” set in as investors realized that hand-coding the world was an impossible engineering task.

4. The First AI Winter: The XOR Problem

Parallel to the symbolic era, some researchers were looking at biology. The Perceptron, created by Frank Rosenblatt in 1958, was the first implementation of a “neural network.” It learned by adjusting weights based on error—the ancestor of modern training.

In 1969, Marvin Minsky and Seymour Papert published Perceptrons. They mathematically proved that a single-layer perceptron could not solve the XOR (Exclusive OR) problem. It could only classify data that was “linearly separable” (data you can divide with a straight line).

This proof effectively killed the “Connectionist” (neural) movement for two decades. The community concluded that neural networks were a mathematical dead end. They were wrong—they just didn’t have Backpropagation or the compute to prove it.

5. Why Neural Networks Needed GPUs to Succeed

The comeback started in 1986 when Geoffrey Hinton, David Rumelhart, and Ronald Williams popularized Backpropagation. This allowed us to train “Hidden Layers”—the layers between input and output. Backprop uses the Chain Rule of calculus to calculate how much each individual weight contributed to an error at the end of the network, allowing for precise adjustments.

The Hardware Lottery

Even with Backpropagation, Deep Learning was slow. For decades, it was a niche academic interest. Then came 2012. A team used NVIDIA GPUs to train a network called AlexNet for the ImageNet competition. GPUs, designed for the parallel math of rendering 3D video games, were perfect for the matrix multiplications required by neural networks.

Engineering Pivot: We stopped trying to be “smart” with rules and started being “massive” with compute. This is the Bitter Lesson (coined by Rich Sutton): over the long term, general methods that leverage compute always outperform methods that leverage human-coded expertise.

6. Why Transformers Scale Where RNNs Failed

Until 2017, the state of the art was Recurrent Neural Networks (RNNs) and LSTMs. They processed data sequentially (one word at a time). This created two massive engineering bottlenecks:

1. Sequential Bottleneck: You couldn’t parallelize training across thousands of GPU cores because step $N$ depended on step $N-1$.
2. Vanishing Gradients: By the time the model got to the end of a long sentence, it “forgot” the beginning because the mathematical signal (the gradient) got smaller and smaller as it was multiplied through time steps.

The Solution: Attention is All You Need

The Transformer architecture changed everything by using Self-Attention. Instead of reading sequentially, it looks at the entire sequence at once. Every word (token) creates three vectors: Query (Q), Key (K), and Value (V).

The word “Bank” sends out a Query: “I need context.” It checks its Query against the Keys of all other words. If it sees the Key for “River,” the Attention score is high. If it sees “Money,” the score is high. It then pulls in the corresponding Values to build a context-rich representation. This is why LLMs are so good at picking up nuances in long-form text.

7. Why LLMs Hallucinate (Architectural Reason)

Hallucination isn’t a “glitch”—it’s a feature of Probabilistic Computing. Symbolic AI was “Closed-World”; if a fact wasn’t in the database, it didn’t exist. Neural Networks are “Open-World” and continuous. They live in a space of Vector Embeddings where every concept is a point in a high-dimensional cloud.

When you ask an LLM a question, it is traversing this cloud and predicting the Next Most Likely Token. It doesn’t have a “Fact-Checker” module. If the statistically most probable next word is wrong (because of training data bias or “noise” in the weights), the model will say it with total confidence. It is a “stochastic parrot” that prioritizes plausibility over veracity.