DeepSeek Compresses the Million-Token Context
DeepSeek introduced DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts model built around deployment cost reduction. It combines cross-layer KV cache reuse, FP4 KV caching, and SWA Bounded Replay to shrink cache footprints while supporting contexts up to one million tokens. The release is positioned for input-heavy agentic workloads, with checkpoints available on Hugging Face.
- The architecture uses a 40-layer Causal Encoder-Decoder split into a 20-layer causal encoder and a 20-layer decoder.
- Sparse attention assigns layers to Full, Reindex, or Reuse modes so later layers can share KV data and sparse-attention indices.
- The model includes 196B parameters of Engram memory accessed sparsely through token-based lookup.
