From Zero to 250M: How I Built the STE Dataset to Train a Tiny LLM from Scratch

Overview of STE dataset

This blog assumes you have a basic understanding of LLMs and how they're trained. If not, I would recommend you to first go through this llm-course by Maxime Labonne.

I will release the source code and dataset on GitHub and HuggingFace soon. Stay tuned for more updates in my future blogs.

What is happening in the world of LLMs?

Sam Altman's first tweet on X about release of ChatGPT

Since November 2022, when OpenAI released ChatGPT, the world has really seen a lot of advancements in the way we use AI in our day to day lives. LLMs has really taken over the world. Everyday we see new models, techniques, applications, of these models. People are finding new ways to use, train and make these models better. As I write this blog, people on X are awestruck about moltbook where the agents of openclaw hangout and talk just like people talk on X, Reddit or any social media platform. Yes, you heard it right, they talk just like humans!!!!!. Just see the screenshot below for some sample posts in there.

Source: Twitter, Some of the posts by OpenClaw agents on Moltbook

These models has become part of every human life who has access to internet in some or the other way. It's crazy how fast things are moving. This is really the crazy sci-fi stuff we see in movies, books becoming a reality. Personally I have been using LLMs everyday. I use it for coding, writing, making illustrations, brainstorming ideas, asking questions, giving life to my ideas... It's just amazing how these models are evolving day by day.

A few random notes from claude coding quite a bit last few weeks.

Coding workflow. Given the latest lift in LLM coding capability, like many others I rapidly went from about 80% manual+autocomplete coding and 20% agents in November to 80% agent coding and 20% edits+touchups in…

— Andrej Karpathy (@karpathy) January 26, 2026

Why did I even build this dataset?

I started my journey in AI in 2020 with Andrew Ng's Deep Learning Specialization. Since then, there hasn't been a day where I haven't stopped thinking about AI, how it works, how to use it... With so much happening in the world of LLMs, I became curious about how these models are even built: what data they are trained on, how to train them, how to make them conversational, how to make them reason, how to make them understand context, how to make them generate human-like text...

In the AI era, you’re either an API consumer or a model architect. I decided I didn’t just want to use the models—I wanted to understand how they work from the inside out and I believe this is first tiny step in that direction.

I decided to train a small LLM from scratch. Instead of using existing datasets, I wanted to create my own dataset. So I started researching about the topics that would make a good dataset for training an LLM. I came across this TinyStories, a dataset with synthetically generated tiny stories. This motivated me to create such dataset for my own LLM. I decided to focus more on the reasoning capabilities of the model, where the model knows about the world, can reason real world problems and can generate human-like text. I took this idea to Gemini and asked it to suggest some ideas. It came-up with this idea of Simple Thoughts Experiments (STE) dataset.

Geimini's Deep Research response about ideas for dataset

How did I build STE dataset?

Once I decided the idea of STE dataset, I started researching about the topics that would make a good dataset for training an LLM, what models to use for different training stages (pretraining, instruction tuning, alignment, reasoning), how much data to generate and it's costs, cleaning data, using existing datasets.

Listing down all the topics

To give the LLM a good foundation, I decided to include topics from various domains like Physics, Chemistry, Biology, Mathematics, Computer Science, General Knowledge, etc. For every topics I (well I asked Gemini to do so) listed down the potential sub-topics. For every sub-topics different areas and aspects were considered. Finally these areas were used to generate prompts for data generation (more on how the prompts were generated in the next sections).

A mermaid diagram showing the topics and sub-topics covered in STE dataset

What models to use for different training stages

Though the models are becoming pretty large in terms of parameters (billions to trillions), every model has it's own style of generating text. Using a single model to generate data for all the stages would definitely introduce bias in the dataset and thereby the model trained on it. So, I decided to diversify the models used for data generation. Initially I started with smaller set of models but through the course of data geenration, I added more models to the list (especially to pretraining stage). The figure below lists the models I used for data generation.

Models used for different stages of STE data generation

While some models are selected based on their capabilities (like o4-mini being good at judging the quality of data, DeepSeek-R1 and Claude 3.5 being good at reasoning and generating human-like text, etc.), others are selected based on their cost, speed, and availability.

Estimating data size and costs

Samples

The amount of data required depends on the size of model that we want to train. The famous Chinchilla scaling laws Hoffmann et al., 2022 suggest that for optimal training, the amount of pretraining data should be around 20 times the number of parameters in the model. While this is for pretraining data, the amount of data required for SFT, alignment and reasoning stages depends on the model architecture and training strategy. LLMs often requires very less data for SFT, alignment and reasoning stages compared to pretraining data (assuming the pretraining is done on high quality data). Below is the table with estimated data size for different model sizes.

Model Size	Pretraining Samples	Pretraining Tokens	SFT Samples	Alignment Pairs	Reasoning Samples
1M	50k – 150k	20M – 60M	2k – 5k	1k – 2k	500 – 1k
10M	300k – 1M	150M – 500M	5k – 20k	3k – 10k	2k – 5k
50M	2M – 5M	1.5B – 3B	25k – 80k	10k – 40k	10k – 30k
100M	5M – 15M	5B – 10B	50k – 200k	20k – 100k	20k – 80k

Data size estimation for different model sizes

Tokens

Number of tokens per sample for different stages of training is as important as the number of samples. Below is the table with estimated number of tokens per sample for different stages of training.

Stage	Component	Tokens (Typical Range)
Pretraining	Raw text	512-700
SFT	System prompt	10–25
	User instruction	50–150
	Assistant response	150–300
Alignment	System prompt	5–20
	User prompt	50–150
	Preferred response	100–250
	Rejected response	100–250
Reasoning	Problem statement	50–150
	Reasoning steps	200–500
	Final response	30–80

Number of tokens per sample components for different stages of training for a 10M parameter model

Costs

I used OpenAI API for OpenAI models, DeepInfra and OpenRouter for other models. I couldn't use Mistral or Claude because my credit card didn't work with them and DeepSeek API is not available in India (it said so when I tried to add credits). I spent around 170$ for data generation.

Bills from OpenAI, DeepInfra and OpenRouter

FYI, this is what ChatGPT has estimated for the cost of data generation.

ChatGPT's response on cost of data generation

Developing prompts for data generation

Pretraining

Though there are so many topics, sub-topics and areas, it's really difficult to achieve the target of 250M tokens by just varying these components in the prompt. So, I added different variations like opening styles, contexts, and focus angles to the prompts. For example,

prompt_variations = {
    "opening_styles": [
        "Explain clearly...", 
        "Imagine you are...", 
        "In a professional tone..."
    ],
    "contexts": [
        "research laboratory", 
        "startup environment", 
        "educational setting"
    ],
    "focus_angles": [
        "computational efficiency", 
        "user experience", 
        "scalability"
    ]
}

List of sample opening styles, contexts and focus angles

SFT

For SFT data generation, variations like question styles, context and complexity variations were added to prompt

prompt_variations = {
    "question_styles": [
        "Ask as a curious student who just encountered this",
        "Ask as someone who noticed something unusual",
        "Ask as a person trying to understand the basics"
    ],
    "contexts": [
        "in everyday life",
        "while cooking",
        "during outdoor activities"
    ],
    "complexity": [
        "Keep the question simple and direct.",
        "Make it a thoughtful question.",
        "Frame it as a 'what if' scenario."
    ]
}

List of sample question styles context and complexity variations

Alignment

For Alignment data, we need to generate pairs of preferred and rejected responses for a given prompt. So, I added variationslike question type and perspectives.

prompt_variations = {
    "question_types": [
        "Ask as a curious student who just encountered this",
        "Ask as someone who noticed something unusual",
        "Ask as a person trying to understand the basics"
    ],
    "perspectives": [
        "from a curious beginner's perspective",
        "as someone who just observed something puzzling",
        "from a practical problem-solving angle"
    ]
}

List of sample question types and perspectives

Reasoning

For generating different styles of reasoning samples, we need to vary aspects like the way question is being asked and the scenario in which the response should be.

prompt_variations = {
    "question_styles": [
        "Focus on the 'why' behind the phenomenon",
        "Ask about what would happen in an extreme case",
        "Question a common assumption people make",
        "Ask about the mechanism or process"
    ],
    "scenarios": [
        "during a thought experiment",
        "in a practical problem-solving situation",
        "when explaining to someone else",
    ]
}

List of sample question styles and scenarios

Utilizing existing datasets

After generating almost 65% (228k samples) of pretraining data, I decided to include some samples from existing datasets that macthes the style and requirements of STE dataset. I did some research and found Fineweb, wikipedia, cosmopedia to be most suitable for this purpose. I added following filters and samples rest 35% of pretraining data:

Explanatory patterns: words like "explain", "describe", "illustrate", "demonstrate", "show", "clarify", "detail", "elaborate", "outline", "summarize" Indicates educational content.
Negative patterns: Ignored samples with URLs from social media and other non-educational sources.
Negative Keywords: Ignored samples with keywords like "free shipping", "upvote", "gossip", "entertainment", "reply", "posted" to filter content from social media and other non-educational sources.
Topic matching: Used topic, sub-topic and area keywords to filter relevant samples.

Data Statistics

Sample & Token Scale

Overall distribution of samples and tokens across training stages

Average Tokens per Component

Average tokens per component in each training stage

If you notice that the average tokens for reasoning thought traces and final responses is too high. This is because of the model's tendency to generate longer responses to justify its reasoning. To address this, the thought traces and final responses are summarized using gpt-4o-mini to bring down the average tokens.

Preprocessing of thought traces and final responses

1. Pretraining Stage

Data Sources

Data sources contributing to the pretraining corpus

Model Mixture

Models utilized in pretraining data generation

Leading Topics

Distribution of topics in pretraining data

2. Supervised Fine-Tuning (SFT)

Knowledge Coverage

Distribution of topics in SFT data

Source Models

Models responsible for generating supervised training data

3. Alignment

Model Roles (Chosen vs Rejected vs Judge)

Comparison of Chosen vs Rejected vs Judge models

Alignment Topics

Topics covered in alignment data

4. Reasoning

The "Thinker" Models

Models used for generating reasoning data

Reasoning Topics

Topics covered in reasoning data

What did I learn? Few key insights

Make sure to monitor the generated samples and check the duplication of samples
Try to use wide variety/family of models for generating different components of the dataset
Don't have a notion that the larger the model, the better the dataset. Even smaller models can generate high quality data. This helps if there are budget constraints
Try to use existing datasets to supplement the generated data (if generated data is not enough)
Before geenrating the dataset at full scale, try to generate a small sample and update the prompts and parameters
Some models are good at generating certain types of data, so research and use the best model for each type of data
Make sure to mention the tokens count per component, per training stage in the prompt. Number of tokens has a direct impact on model quality. For example, models lie DeepSeek-R1 generates thought traces with more than 800 tokens which is not suitable for a smaller model. Ensure to preprocess the generated data to avoid such larger traces and responses.

What are the next steps?

Cleaning and Preprocessing
Deduplication
Tokenization
Architecture Selection
Pretraining
Supervised Fine-Tuning
Alignment
Reasoning
Evaluation on benchmarks

References

FineWeb-Edu: Piktus, A., et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. Hugging Face Dataset.
Cosmopedia: Ben Allal, L., et al. (2024). Cosmopedia: How to Create a Large-Scale Synthetic Dataset for Pre-training. Hugging Face Dataset.
Wikipedia: Foundation, W. Wikipedia Dataset (20231101.en). Hugging Face Dataset.
Smol Training Playbook: Hugging Face. Smol Training Playbook. Hugging Face Spaces.
LLM Course: Labonne, M. Large Language Model Course. GitHub Repository.
Chinchilla Paper: Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.

Tools

HuggingFace Datasets
Image Generation: Gemini, ChatGPT
Editor: Antigravity
History Diagram: Mermaid