Failure Taxonomy

System returned no relevant context for the query.

Example: Query asks about a meeting date; system returns unrelated content.

System returned context, but none of it is relevant to the question.

Example: Query asks about dietary preferences; system returns work schedule.

System returned too much context, diluting the relevant information.

Example: System dumps entire conversation history instead of targeted recall.

System returned outdated information that was later corrected.

Example: User updated their address; system returns the old one.

System confused the temporal ordering of events.

Example: Events A then B occurred; system says B happened before A.

System cannot recall when something happened.

Example: Query asks 'when did X happen?'; system has no temporal context.

System confused two different entities mentioned in the conversation.

Example: Alice likes coffee, Bob likes tea; system says Alice likes tea.

System returned a fact that contradicts stored information.

Example: Stored: budget is $5000; returned: budget is $3000.

System returned some but not all of the expected information.

Example: Expected 3 items; system returned 2.

System generated information not grounded in any stored memory.

Example: System invents a meeting that never occurred.

System made a claim that cannot be traced to stored context.

Example: System says 'as we discussed last week' when no such discussion exists.

System did not honor an explicit delete or forget request.

Example: User says 'forget my phone number'; system still returns it.

System cannot identify the source of a stored fact.

Example: System returns a fact but cannot say when or how it was learned.

Information from one context leaked into another.

Example: User A's data appears in User B's memory space.

Information leaked between agents that should have isolated memory.

Example: Agent 1's private context appears in Agent 2's responses.

Information was lost during agent-to-agent handoff.

Example: User provides info to Agent 1; Agent 2 has no knowledge of it.

Conflicting information from multiple sources was not resolved.

Example: Two agents provide different answers; system doesn't choose.

System returned an empty response.

Example: Recall returns '' or null.

System raised an error during recall.

Example: ConnectionError, TimeoutError, or internal exception.

System did not respond within the allowed time.

Example: Recall took >30s and was killed.

System response was in an unexpected format.

Example: Expected text; got JSON or binary data.

The LLM judge scored differently than the deterministic scorer.

Example: Deterministic says FAIL; LLM judge says correct (or vice versa).

System abstained but the answer was retrievable.

Example: System says 'I don't know' when the answer is in its memory.