Zhu, Deyao 朱德尧

Welcome! I am a Research Scientist at ByteDance Seed Edge. I am a core contributor to BAGEL , and the project lead of MiniGPT-4 . I received my PhD from KAUST, where I was advised by Mohamed Elhoseiny. My doctoral research focused on model-based RL, with an emphasis on sample-efficient learning. My research interests are learning from experience, multimodal LLMs, and reinforcement learning.

Email  /  Google Scholar  /  GitHub  /  Linkedin

BAGEL is an open-source unified multimodal model for understanding and generation, pretrained on large-scale interleaved text, image, video, and web data.

Causal Diffusion introduces a next-token forecasting view of diffusion models and proposes CausalFusion, a decoder-only transformer that factorizes data across sequence positions and diffusion noise levels.

MiniGPT-4 shows that the secret behind the next-level vision-language-ability of GPT-4 can be simply a more powerful LLM. By aligning open-sourced vision and advanced language models together, MiniGPT-4 reproduces many GPT-4's vision-related demo.

Video ChatCaptioner creates comprehensive spatiotemporal video descriptions by letting ChatGPT to select the video frame it want to know and ask questions to BLIP-2. ChatGPT at the end summarizes all the information from BLIP-2 as the final video description.

We discover the powerful questioning ability of modern LLMs. We use it to enrich the image caption of BLIP-2 by prompting ChatGPT to keep asking informative questions to BLIP-2 and summarize the conversation at the end as the final caption.

ZeroSeg,a novel method that leverages the existing pretrained vision-language(VL) model to train open-vocabulary zero-shot semantic segmentation models

Extract knowledge from datasets without action labels to help online reinforcement learning by pretraining an Action-Free Decision Transformer to form intrinsic rewards.

Applying RL methods on a graph world model instead of the original complex environment simplifies the policy learning.

A better metric for trajectory prediction that consider the whole prediction distribution.

Modeling an effective message-passing flow through an attention mechanism can be critical to tackling the compositionality and long-tail challenges in visual relationship recognition.

Reducing the likelihood of the context-violating predictions directly in the predicted distribution improves the prediction quality.

Hallucinating surrounding vehicles' driving intents helps model predict better.

Reducing the total correlation of the latent feature's dimensions to learn a physically disentangled representation of blocks.

Template