GitHub - AML-memory/agent-memory-leaderboard · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
AML-memory
agent-memory-leaderboard
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>1 Commit<br>1 Commit
beam
beam
clbench
clbench
locomo-refined
locomo-refined
longmemeval-s
longmemeval-s
personamem
personamem
scriptmem
scriptmem
README.md
README.md
api_config.py
api_config.py
requirements.txt
requirements.txt
View all files
Repository files navigation
Public Evaluation Code
This directory publishes the answer-generation and evaluation contracts used by<br>the Agent Memory Leaderboard for its public datasets.
The scripts require Python 3.10 or newer and the dependency listed in<br>requirements.txt. Service URLs, model names, and credentials must be supplied<br>through CLI arguments or environment variables; no credential is bundled here.
Provenance and integrity
Internal shorthand labels<br>are normalized to descriptive public names; answer construction and scoring<br>behavior are unchanged.
The disclosed runtime behavior covers Add/Search/Answer concurrency, timeouts,<br>retries, key rotation, checkpoints, and aggregation. Internal deployment code,<br>service addresses, credentials,<br>databases, datasets, and run artifacts are intentionally outside this public<br>evaluation release.
Disclosed production parameters
Full task timeout: 72 hours; retrieval top_k: 100.
Add: 64 global workers by default, 48-record per-dataset hard cap, 20-message<br>chunks, 1,200-second HTTP timeout, and global-largest-next-record-v1<br>scheduling using estimated chunk counts.
Search: 32 workers, 1,200-second HTTP timeout, six request attempts, up to<br>three failed-record requeues, and adaptive concurrency reduction after five<br>retryable 5xx responses.
Answer: 32 workers, gpt-4o-mini, temperature 0, 180-second HTTP timeout,<br>and six attempts. The configured checkpoint value is 128 tokens, but the<br>outbound provider request intentionally does not send max_tokens.
Judge: a configured evaluator, 360-second HTTP timeout, and a global cap of<br>15 requests. Worker phase limits are 8 for binary evaluation, 15 for rubric<br>evaluation, and 16 for BEAM event alignment; the global cap means effective<br>Judge request concurrency never exceeds 15.
Judge failures: HTTP 429, timeout, and transport failures rotate the key.<br>Pool-wide 429 exhaustion uses a 30-second cooldown and at most three cooldown<br>rounds. Qwen JSON requests set enable_thinking=false.
Scores are computed on [0, 1] internally and published on [0, 100].
About<br>No description, website, or topics provided.<br>Resources<br>Readme<br>Activity<br>Custom properties<br>Stars<br>1 star<br>Watchers<br>0 watching<br>Forks<br>0 forks<br>Report repository
Releases
Packages
Contributors
Languages
You can’t perform that action at this time.