Why Most AI Security Tools Break in Production

At Wild West Hackin' Fest in Denver, LimaCharlie Sr. Solutions Engineer Chris Botelho gave a talk on a problem most vendors avoid discussing: AI security integrations that work great in demos and fall apart in production. Chris walked through what broke when LimaCharlie first connected an LLM to its platform, why it broke, and the architecture that fixed it. Everything covered is open source, and the full talk is embedded below.

The 50,000 Token Startup Tax

LimaCharlie's first attempt was the obvious one. Take 140+ API endpoints, wrap them in a single MCP server, and hand the whole thing to the model. The result was 50,000 tokens burned at startup before the analyst typed a single prompt. That matters because the context window, the working memory of an LLM, is a finite and expensive resource. Anthropic's own research found MCP servers consuming a third of developer context windows before any real work began. Context windows have since grown, with Claude and other frontier models now offering 1M tokens, but bigger windows do not solve the problem. Model performance degrades well before the window fills. Past roughly 50% utilization, output gets slower, less accurate, and less consistent, and models reliably lose track of information buried in the middle of a large context.

Progressive Disclosure Instead of Everything at Once

The fix borrows a pattern every developer knows: lazy loading. Instead of front-loading the full API surface, LimaCharlie built a single meta skill. On startup, the model knows one thing, how to ask the MCP server what is available. It gets a brief overview and requests detail only when a task requires it. That took the model from tracking 140 tools down to three at any given time, and cut startup context usage from 50,000 tokens to around 1,000.

Large data got the same treatment. A 10 MB log file translates to millions of tokens, more than most models can hold. Rather than streaming results into context, the MCP server writes large outputs to ephemeral cloud storage and returns a URL. The model downloads the file, parses it on disk with standard tools, and only relevant findings enter the context window.

The Right Model for the Right Job

The final piece is delegation. Opus handles high-level reasoning, detection logic, and analysis. Specialized subagents running Sonnet do the grunt work: fetching files, writing parsing scripts, processing data. Each subagent gets its own context window, so the main model stays clean. The team tested Haiku for subagent work and abandoned it after persistent hallucinations, including invented organizations and operating systems that did not exist.

The combined result is 80 to 90% lower context usage, faster responses, and deterministic behavior at scale.

From Malware Sample to Verified Detections in 23 Minutes

From a single prompt, the system reverse engineered a malware sample with Ghidra, decrypted its configuration, wrote detection rules in LimaCharlie, spun up an Azure VM, deployed a sensor, detonated the sample, verified every rule fired, and cleaned up after itself. Total time: 23 minutes, hands off.

AI doesn’t replace analysts. It automates the repetitive steps analysts already perform so they can spend time on the work that actually requires judgment.

All of the tooling from the talk is open source, including the MCP server, the skills, and LCRE, LimaCharlie's reverse engineering CLI. Explore the repositories at https://github.com/refractionPOINT/lc-ai/tree/master or try the platform free at https://app.limacharlie.io/signup.