5 core pillars of Large Language Model (LLM) operations

The below outlines the 5 main pillars of using an AI LLM. The terminology is the basis of discussing ‘AI’ within an enterprise.

---

1. Tokenization: Unit of Computation

To understand AI performance and cost, one must first recognize that models do not process raw text in the form of words or characters. Instead, they operate on tokens which are atomic units of data representing discrete sequences of characters.

• Definition: A token can encompass a single word, a common prefix (e.g., “un-“), a suffix (e.g., “-tion”), or even individual punctuation marks.

• Cost Impact: Cloud service providers and API architectures measure consumption in tokens. Every input sent and every output generated incurs a cost based on these counts.

• Operational Limit: Tokens are the primary constraint of a model’s memory. When a conversation exceeds a specific token threshold, the model must evict the oldest data to make room for new inputs, similar to how a computer manages RAM.

2. The Context Window: Active Memory Capacity

The Context Window represents the maximum quantity of tokens a model can maintain in its active memory at any given moment. This serves as the model’s immediate workspace.

• Scope: This window must contain the user’s initial instructions (system prompts), the ongoing conversation history, any uploaded documents, and the model’s pending response.

• Evolution of Scale: Early iterations of LLMs were restricted to small windows (roughly 4,000 tokens), limiting them to short interactions. Modern frontier models now support windows exceeding one million tokens, enabling the analysis of entire codebases or multi-volume manuscripts in a single session.

• Strategic Limitation: If an AI seems to forget a detail from earlier in a session, it is usually because that information has shifted outside the context window’s boundaries.

3. Temperature: Stochasticity vs. Determinism

Temperature is a configuration parameter that governs the degree of randomness or creativity in a model’s response. It determines the probability distribution used when the AI selects the next token.

• Low Temperature (Approaching 0): The model becomes deterministic, consistently selecting the most statistically probable token. This is ideal for technical documentation, code generation, and factual summarization where precision is paramount.

• High Temperature (Approaching 1.0+): The model takes statistical risks, choosing less likely tokens to produce more diverse and creative prose. This is highly effective for brainstorming, marketing copy, and creative writing.

• Application: Most consumer interfaces use a balanced default (approx. 0.7), but professional developers manually tune this to align the output with the specific business requirement.

4. Hallucination: Probabilistic Prediction Risks

A Hallucination occurs when an LLM generates factually incorrect information while maintaining a tone of absolute certainty. To mitigate this risk, one must understand that LLMs are not traditional databases.

• Mechanism: LLMs function as sophisticated next-token predictors. They do not look up facts; they calculate the most likely sequence of language based on patterns identified during training.

• The Confidence Gap: Because the model is trained to provide fluid, helpful responses, it may prioritize linguistic coherence over factual accuracy if it lacks specific data. It will complete the pattern even if the resulting facts are non-existent.

• Verification Requirement: In high-stakes environments legal, medical, or financial AI output should be treated as a draft that requires human-in-the-loop validation.

5. RAG: Retrieval-Augmented Generation

RAG is an architectural framework designed to solve two primary LLM limitations: knowledge cutoff (the model only knows what it was trained on) and a lack of access to private, proprietary data.

• The Process: Rather than retraining a massive model on new data, RAG allows the system to search a specific set of documents (such as your company’s internal PDFs) in real-time.

• Vector Databases: When a query is submitted, the system retrieves the most relevant chunks of information from a specialized database.

• Grounding: These retrieved facts are fed into the model alongside the user’s question. The AI is then instructed to answer only using that provided context, effectively grounding the response in verified facts and significantly reducing the likelihood of hallucination.