Tokens, caching and the cost model
The cost of an agent is not the price of one question — it is the price of the loop. Every turn resends the whole conversation history to the model, so in a long session cost grows quadratically, not linearly: on turn 20 you are paying not for question 20 but for the previous 19 turns as well.
Four mechanisms hold that in check: compression, prompt caching, memory limits and progressive disclosure of skills.
| Mechanism | What it does | Config key |
|---|---|---|
| Compression | Summarises the older part lossily as the context limit approaches | `compression.threshold`, `compression.target_ratio` |
| Prompt caching | Caches the system prompt, skills and long context files across sessions | `prompt_caching.cache_ttl` (5m or 1h) |
| Memory limits | Keeps MEMORY.md and USER.md within a character budget | `memory.memory_char_limit`, `memory.user_char_limit` |
| Progressive disclosure | Loads only skill names and descriptions, the full text only on demand | The skills system (automatic) |
| Tool output limits | Stops a huge terminal output from flooding the context | `tool_output.max_bytes`, `max_lines` |
Prompt caching gives the biggest win for the least effort: it turns on automatically on providers that support it and saves you from paying repeatedly for the stable part of the system prompt. That is also why churning the stable part (the skill list, context files) hurts — each change invalidates the cache.
Compression comes at a price: it is a lossy summary. When an agent in a long session forgets "the rule we agreed at the start", compression is usually the reason. The fix is not to stretch the session artificially — it is to move the important fact into memory (MEMORY.md) or a context file.
The three most effective steps to run cheaply: (1) route auxiliary work to a cheap model, (2) switch off toolsets you do not need — every tool schema eats tokens on every request, (3) run short, focused sessions. And a fourth: measure what is actually expensive with hermes insights instead of guessing.
Practice. Run the same task twice: once with every toolset on, once with --toolsets "terminal,file". After each, compare the output of hermes prompt-size and hermes insights. Done means: you can state, as a number, how much the system prompt shrank and how much the cost dropped.
📚 Sources and documentation
- Configuration: compression, caching, limitsofficialhermes-agent.nousresearch.com
The exact names and default values of every key.
- Provider routingofficialhermes-agent.nousresearch.com
Controlling exactly which provider handles which work.
- Skills and progressive disclosureofficialhermes-agent.nousresearch.com