AI Publications

🔗 https://www.factory.ai/context-window-problem?utm_source=substack&utm_medium=email

by Varin Nair

1. Critical Context for Effective Agents

Large language models (LLMs) are limited to around 1 million tokens in their context window. In contrast, enterprise monorepos and the surrounding knowledge ecosystem span millions of tokens across code, documentation, logs, and conversations—a massive gulf that hampers agentic workflows at scale.

Effective agents require various layers of context—not just code, but also task descriptions, developer persona, system architecture, historical decisions, and team conventions. Without these, LLMs produce misaligned outputs, misunderstand requirements, or violate team norms.

2. Why Existing Approaches Fail

• Naive vector retrieval: Chunking files into vectors and retrieving similar ones flattens code’s rich structure, breaks dependency chains, and interrupts multi-hop reasoning—leading agents astray with irrelevant fragments.

• Scaling context windows: Even frontier models with 1–2M token windows fall short of encompassing full codebases. Big context doesn’t guarantee better outcomes—attention tends to fade mid-prompt (a phenomenon known as “context rot”), and costs skyrocket.

3. Factory’s Context Stack

Factory addresses these constraints with a multi-layered context scaffolding that distills “all the company knows” into “exactly what the AI needs now”:

1. Repository Overviews: Summarizes structure, build setup, key files—delivered upfront to give the LLM a lightweight architectural map.

2. Semantic Search: Uses code-tuned embeddings to rank relevant files and folders, kicking off reasoning with a precise subset.

3. Targeted File System Commands: Fetches specific file sections, diffs, or outputs on demand—staying within token budgets yet highly focused.

4. Enterprise Context Integrations: Incorporates observability data (like Sentry logs), design docs, architecture guides, and tribal knowledge from Notion or Google Docs—providing depth beyond code.

5. Hierarchical Memory:
   

• User Memory tracks individual preferences and past project work (e.g., OS, dev tools, style preferences).

• Org Memory encodes team-wide norms like coding standards, onboarding templates, and documentation styles.

Together, these layers allow Factory’s “Droids” (AI agents) to work with precise, relevant context—no more noise, no more guesswork.

4. Key Metrics Improved

Adopting the Context Stack brings measurable impact:

• Reduced onboarding and dev cycle time—because context is automatically curated throughout sessions.

• Higher code acceptance rates—inputs align with team standards, minimizing review friction.

• Improved user satisfaction and retention—as memory personalization grows richer with each session, making the tool feel ever-smarter.

5. Future Directions

Looking ahead, advancements will include:

• Larger LLM context windows and enhanced reasoning capabilities.

• Smarter agents capable of long-horizon planning and multi-step task execution.Yet, challenges remain—agents still get distracted by irrelevant context, require the right tools to act, and need durable external memory for state continuity. Multi-agent orchestration and robust tooling will be essential. Factory’s Context Stack serves as a blueprint for building reliable, scalable, and cost‑effective agentic systems in real-world engineering environments.