Open Deep Research
lastmod 2025-04-04created 2025-02-053 edits
About
- DeepResearch, announced by OpenAI, summarizes content and answers questions through web search
- Drew attention by achieving a high score on the GAIA benchmark
- Combines a powerful LLM with an internal agent framework to use various tools, such as web browsing, step by step
What is an agent framework and why does it matter?
- An agent framework is a structure that adds a layer on top of an LLM so that it can perform various actions such as browsing and reading PDFs
- Combining an LLM with an agent system makes it far more powerful than using it only as a simple chat
- Even applying a simple agent framework through a library like smolagents improves performance significantly
- OpenAI DeepResearch also uses this approach to achieve excellent performance
GAIA benchmark
- GAIA is a very difficult benchmark for evaluating agent performance
- For example, it poses a complex question asking to link the fruits shown in “Embroidery from Uzbekistan” to the old breakfast menu of a particular ship and list them in order
- A single LLM alone only reaches about 7%, while DeepResearch achieves over 67%, showing a large gap
- GAIA questions require multi-step reasoning, information retrieval, multimodal processing, and so on, making them well suited to testing the true value of the agent approach
Building Open Deep Research
- Ran an experiment combining an open-source LLM with an agent framework to reproduce the DeepResearch approach
- The goal is to raise GAIA performance with tools such as a simple text-based web browser and a file viewing tool
- Using CodeAgent
- The CodeAgent approach expresses actions as code instead of JSON
- According to Wang et al. (2024), code representation has the advantages of being compact, intuitive, and well suited to LLMs
- Fewer steps reduce cost, and it is also better for managing multimodal state
- First tool: text-based web browser
- Rich features like Operator are not implemented yet; as an early stage, it provides only simple browsing
- Second tool: inspector for viewing text file formats
- Supports reading content through a simple document processing tool
- Future plans include more fine-grained file format support, integrating a vision model for web browsing, and introducing a GUI agent
Results
- The 24-hour reproduction experiment reached about 54% on the GAIA benchmark
- Using code representation instead of JSON raised the score from 33% to 54%
- Anyone can reproduce it using the open smolagents framework and tools
- There is much room for improvement by combining an Operator-level browser, a powerful local model, and so on
Community reimplementations
- Various implementations have appeared in the community, including dzhng, assafelovic, nickscamara, jina-ai, and mshumer
- Each uses different libraries or tries different search/indexing approaches
- The aim is to share and build on reproduction results using open LLMs, vision models, code-based action representations, and so on
dzhng/deep-research
An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction overtime and deep dive into a topic.
Most important next steps
- Building a GUI agent to support advanced web browser features like OpenAI's Operator is important
- Aims to provide, as open source, the ability to view the screen and operate it with a mouse and keyboard
- Plans to improve completeness by integrating with smolagents, OpenAI Operator, and others
- Key tasks include improving GAIA scores, using open LLMs, and implementing a visual web browser