New coding model DeepScale

https://huggingface.co/agentica-org/DeepScaleR-1.5B-Preview

DeepScaleR Overview

DeepScaleR-1.5B-Preview is a language model fine-tuned from DeepSeek-R1-Distilled-Qwen-1.5B using distributed reinforcement learning (RL) to scale up to long context lengths. The model achieves 43.1% Pass@1 accuracy on AIME 2024, representing a 15% improvement over the base model (28.8%) and surpassing OpenAI's O1-Preview performance with just 1.5B parameters.

Data

Our training dataset consists of approximately 40,000 unique problem-answer pairs compiled from:

• AIME problems (1984-2023)

• AMC problems (prior to 2023)

• Omni-MATH dataset

• Still dataset

Training Recipe

We employ Deepseek's Group Relative Policy Optimization (GRPO), a simplified RL algorithm that extends PPO by:

• Normalizing advantage function over all samples generated from the same prompt.

• Applying KL divergence regularization on top of PPO's surrogate loss to prevent significant policy drift.

Reward Function: Our reward function is simple but effective:

• 1 for correct answers passing LaTeX/Sympy checks

• 0 for incorrect or improperly formatted answers

• Note: No partial rewards (such as PRMs) or intermediate feedback.

Iterative Context Lengthening: A key challenge in scaling RL for reasoning is compute cost. Our approach trains models with progressively longer contexts as the model improves, thus saving monetary costs and end2end training time:

• Initial 8K Context (0-1040 steps):

  • 22.9% -> 33% Pass@1 on AIME 2024

  • Trained on 8 A100-80GB GPUs, BS= (Prompts) (Samples/Prompt) = 128 8 = 1024

• Extended to 16K (steps 1040-1520):

  • 33% -> 43% Pass@1 on AIME 2024

  • Trained on 32 A100-80GB GPUs, BS= (Prompts) (Samples/Prompt) = 128 16 = 2048

• Further extended to 24K (step 1520+):

  • 38% -> 43% Pass@1 on AIME 2024

  • Trained on 32 A100-80GB GPUs, BS= (Prompts) (Samples/Prompt) = 128 16 = 2048

  • Significant improvements within <200 steps

A more detailed description of the training recipe can be found in our blog post.