多模态赛道¶

学习路径¶

flowchart TD
    L01["01 CLIP\nRetrieval"] --> L02["02 BLIP\nCaptioning + ITM"]
    L02 --> L03["03 LLaVA\nInstruction VLM"]
    L01 --> L04["04 Grounding\nReferring"]
    L04 --> L05["05 Mask Grounding\nMask 预测"]
    L03 --> L06["06 Flamingo\nInterleaved VLM"]
    L02 --> L07["07 Q-Former\nBridge VLM"]
    L06 --> L08["08 Perceiver\nResampler VLM"]
    L07 --> L09["09 PaliGemma\nMultitask VLM"]
    L01 --> L10["10 OWL-ViT\nOpen-Vocab Detection"]
    L05 --> L11["11 Grounded-SAM\nOpen-Vocab Segmentation"]
    L09 --> L12["12 KV-OCR\nDocument VLM"]
    L03 --> L13["13 Video VLM\nTemporal QA"]
    L13 --> L14["14 BMN\nTemporal Grounding"]
    L14 --> L15["15 2D-TAN\nTemporal Grounding"]
    L15 --> L16["16 Multi-Scale\n2D-TAN"]

    style L01 fill:#2563eb,color:#fff
    style L02 fill:#2563eb,color:#fff
    style L03 fill:#7c3aed,color:#fff
    style L04 fill:#dc2626,color:#fff
    style L05 fill:#dc2626,color:#fff
    style L06 fill:#7c3aed,color:#fff
    style L07 fill:#7c3aed,color:#fff
    style L08 fill:#7c3aed,color:#fff
    style L09 fill:#059669,color:#fff
    style L10 fill:#dc2626,color:#fff
    style L11 fill:#dc2626,color:#fff
    style L12 fill:#059669,color:#fff
    style L13 fill:#d97706,color:#fff
    style L14 fill:#d97706,color:#fff
    style L15 fill:#d97706,color:#fff
    style L16 fill:#d97706,color:#fff

先修知识¶

课程列表¶

运行示例¶

python -m tracks.multimodal.lesson_01_clip_toy_retrieval.train \
  --device cpu --epochs 1 \
  --max-train-batches 2 --max-eval-batches 1

python -m tracks.multimodal.lesson_03_llava_toy_instruction_vlm.train \
  --device cpu --epochs 1 \
  --max-train-batches 2 --max-eval-batches 1

python -m tracks.multimodal.lesson_10_owlvit_toy_open_vocab_detection.train \
  --device cpu --epochs 1 \
  --max-train-batches 2 --max-eval-batches 1

python -m tracks.multimodal.lesson_16_multiscale_2dtan_toy_temporal_grounding.train \
  --device cpu --epochs 1 \
  --max-train-batches 2 --max-eval-batches 1

VLM 技术演进脉络¶

flowchart LR
    subgraph 2021["2021 — 对比对齐"]
        CLIP["CLIP"]
        ALIGN["ALIGN"]
        ViLT["ViLT"]
    end
    subgraph 2022["2022 — 融合与生成"]
        BLIP["BLIP"]
        CoCa["CoCa"]
        Flamingo["Flamingo"]
    end
    subgraph 2023["2023 — 指令跟随"]
        BLIP2["BLIP-2"]
        LLaVA["LLaVA"]
        CogVLM["CogVLM"]
        QwenVL["Qwen-VL"]
    end

    CLIP --> BLIP --> BLIP2
    CLIP --> Flamingo --> BLIP2
    BLIP2 --> LLaVA
    BLIP2 --> CogVLM
    LLaVA --> QwenVL

    style CLIP fill:#2563eb,color:#fff
    style BLIP fill:#059669,color:#fff
    style BLIP2 fill:#7c3aed,color:#fff
    style LLaVA fill:#7c3aed,color:#fff

VLM Zoo¶

# 列出所有 VLM 架构
python scripts/vlm_zoo.py --list

# 搜索特定架构
python scripts/vlm_zoo.py --search llava

# 查看时间线
python scripts/vlm_zoo.py --timeline

# 冒烟测试
python scripts/vlm_zoo.py --smoke dlvlm:clip_tiny

多模态任务分类¶

下一步¶