Zhu, Deyao 朱德尧

Welcome! I am a Research Scientist at ByteDance Seed Edge. I am a core contributor to BAGEL , and the project lead of MiniGPT-4 . I received my PhD from KAUST, where I was advised by Mohamed Elhoseiny. My doctoral research focused on model-based RL, with an emphasis on learning sample efficiency. My research interests are learning from experience, multimodal LLMs, and reinforcement learning.

Email  /  Google Scholar  /  GitHub  /  Linkedin

EdgeBench studies how agents learn from real-world environments across 134 long-horizon tasks, revealing log-sigmoid scaling laws from large-scale environment interaction.

BAGEL is an open-source unified multimodal model for understanding and generation, pretrained on large-scale interleaved text, image, video, and web data.

Causal Diffusion introduces a next-token forecasting view of diffusion models and proposes CausalFusion, a decoder-only transformer that factorizes data across sequence positions and diffusion noise levels.

MiniGPT-4 shows that the secret behind the next-level vision-language-ability of GPT-4 can be simply a more powerful LLM. By aligning open-sourced vision and advanced language models together, MiniGPT-4 reproduces many GPT-4's vision-related demo.

We discover the powerful questioning ability of modern LLMs. We use it to enrich the image caption of BLIP-2 by prompting ChatGPT to keep asking informative questions to BLIP-2 and summarize the conversation at the end as the final caption.

Extract knowledge from datasets without action labels to help online reinforcement learning by pretraining an Action-Free Decision Transformer to form intrinsic rewards.

Applying RL methods on a graph world model instead of the original complex environment simplifies the policy learning.

A better metric for trajectory prediction that consider the whole prediction distribution.

Reducing the likelihood of the context-violating predictions directly in the predicted distribution improves the prediction quality.

Template