A nine-person trip plans to leave on May 1, then quietly moves the departure to April 30. Days later someone asks the group assistant to post a ticket reminder, and it announces the old date. The assistant remembered the history and still failed the group. That gap - between recall and service - is what Tencent's Yuanbao team and the NLP group at King's College London set out to measure, and it is the reason they built GroupAssistBench from real chat logs rather than synthetic dialogue.
The benchmark draws on 50 real group chats containing 80,299 historical messages, and turns them into 2,395 evaluation items. It splits the problem in two. Level 1 measures memory recall, with 1,374 questions that have a defined factual answer, such as which date was originally agreed. Level 2 measures memory use, with 1,021 tasks that require completing a service - post the reminder, check the fare, settle the cost - where no fixed wording exists but the answer must respect decisions that are still in force and constraints set by individual members. Within each level, items are tagged M1 for a single memory, M2 for integrating several, and M3 for updates and conflicts, which is where a departed decision has to be recognised as superseded while other members' conditions still apply.
Building the set was mostly an exercise in excluding shortcuts. The team first compared model answers with and without the chat history and kept only requests that genuinely depended on it, so that common sense could not carry a question. Each required fact was then tied to a memory anchor pointing at the source message and evidence span, with a per-question cutoff so the model could not read past the moment of asking. The public release is anonymised and carries no original identities. Sampling checks judged 95.8% of items valid by majority vote.
Level 2 also needed a scoring scheme, since an open-ended reply cannot be graded by string match. Each task carries a custom rubric bound to its evidence, with three parts: grounding, which checks that the necessary historical facts appear; service, which checks the task was actually completed; and penalty, which deducts for violating a standing decision, ignoring a member's constraint or acting on an expired arrangement - using a dead date as the final plan, for instance. On 1,152 rubric judgements across 60 tasks and 180 responses, the automatic judge agreed with the human majority 89.3% of the time.
The results are the point. Across 14 configurations - full context, several retrieval-augmented stacks and external memory systems - one system, EverMemOS, scored 0.621 on Level 1 recall, beating the 0.570 of full context running the same task model. On Level 2 service the same system scored 0.247, well below full context at 0.431. Full context itself is not cheap: it averaged 52,435 tokens per prompt. The strongest Level 2 score in the table came from DeepSeek-v4-flash on full context at 0.469, ahead of Claude Sonnet 5 at 0.428 and GPT-5.6-terra at 0.431.
To explain the inversion, the authors traced 497 paired memories - spans that a Level 1 question asks about directly and that some Level 2 task also depends on. Among memories the system had already recalled correctly but that were not counted as used, about 36% never entered the Level 2 model input at all, usually because a request like "help everyone arrange it" gives no retrieval cue naming which preferences matter. The remaining 64% did reach the model and were still not applied. Handing evidence to a model does not teach it that the evidence constrains the reply.
Two light interventions narrowed the gap. Relational Evidence Refinement (RER) reorganises retrieved evidence by person, event, time and decision and pulls in adjacent candidates; Assistance Constraint Binding (ACB) converts recovered facts into explicit reply constraints with a subject, a scope and a current validity state. RER lifted LightMem's Level 2 average by 6.28 percentage points; ACB lifted xMemory by 3.94 and EverMemOS by 3.16; combined, the three systems gained 7.03, 4.64 and 4.43 points over their own baselines. The authors caution that the two mechanisms' independent contributions still need separating.
The paper, released on arXiv as 2611.04867 with code and dataset public, matters less for its leaderboard than for what it invalidates. A memory benchmark that grades recall rewards storing and retrieving more, and retrieval systems duly optimise for it; in a multi-user setting the binding constraint is deciding whose memory governs the current answer and whether it is still valid. Personal assistants and group agents will be judged on that second skill, and it is currently worth roughly half a point of headroom even for the best configuration tested.
Comments (0)
Log in to join the discussion
Log InNo comments yet