Back to portfolio

AI Research · University of Toronto · Advanced Data Systems · 2026

Episodic Memory for an LLM Agent

Code

A Minecraft agent that remembers past episodes as video and text, and retrieves the most similar ones to guide its next action.

Episodic Memory for an LLM Agent - 1

Overview

LLM agents forget what happened in earlier attempts and repeat the same mistakes. This project gives a Minecraft agent an episodic memory. Each past episode is stored as CLIP video embeddings and a text description from a vision-language model. At decision time, the agent retrieves the most similar episodes and adds them to its prompt before it chooses an action. The task is collecting wood in MineRL.

What I built

  • Built multimodal retrieval that fuses video and text similarity over stored episodes.
  • Encoded episodes with MineCLIP video embeddings and Qwen2.5-VL descriptions, stored in a vector database.
  • Built a model-inference server for the vision-language and CLIP models, with batched action generation.
  • Ran long agent rollouts as fault-tolerant Slurm jobs that restart on failure.

Tech stack

PythonPyTorchMineCLIPQwen2.5-VLChromaDBMineRLSlurm