Why am I building this?

I started learning ML in 2021 through Andrew Ng's Deep learning Specialization course on Coursera. Through this course I came across a documentatry by Deepmind Alpha Go Documentary where they beat a Go champion using their algorithm Alpha Go. This was the first time I heard of the field "Reinforcement Learning". It was fascinating to me how a Algorthm was able to beat great champions who has been mastering the game for years. That's when I decided to learn RL once I am good at Deep Learning. I got the Reinforcement Learning book (Sutto & Barto) in 2022 and finally started reading it this year :p.

Like my DotLM project, I wanted to build something from scratch to showcase my RL learnings. In Decmber 2025, when LLMs have become very trendy, I wanted to test how good it is at developing UI and created this 2D grid-based simulation environment replicating how Mars environment would looklike. It has done a pretty decent job at designing the UI.

AresSim Old Demo

My main objective of creating this environment is to test-out different RL algorithms, LLMs as Agents and compare their exploration capabilities. Once I became comfortable with RL (notations, maths and few algorithms), I started working on making the UI better. I fixed the final version of UI in July 2026.

The entire project is heavily inspired from NeuralMMO by Joseph Suarez. I recommend follwing him on X and watching his live-sessions. He has been working in RL before it was cool.

What is AresSim?

AresSim is a grid-based Mars survival Simulation and RL Environment. The main objective of Rover, acting as Agent is to develop and service the base for Humans and help with extracting Mineral and Ice by exploring the Grid-based Mars Surface. "Ares" means "Mars" in Roman mythology. The environment is designed by considering relastic situations on Mars and also not complex enough to make it a use-case for RL algorithms. In this blog, I will explain the details of the environment and how to formulate this as a RL problem. The upcoming blogs will be showcasing the results of different approaches I tried on this environment.

If you really want to visualize actually Mars surface, Checkout Mars 2020 Images captured by Perseverance Rover from 2020 to this day.

AresSim World

The environment is a 32x32 grid where each cell belongs to a family of terrain types.

Terrain types

Every cell on the 32×32 Mars grid has a terrain type that controls movement cost, which actions are legal, and where resources can be found. Alongside the label, each cell also carries roughness and dust channels (0–100%) — roughness drives battery drain and stress, while dust feeds into solar output and pad maintenance over time.

Terrain types

Build Pad

Build Pad is the 5x5 colony’s home. The environment generator picks a flat, safe site from the seed (retries with offsets if needed), places the rover there (initial position), and lays down four starter systems — habitat, solar, charger, and storage — shown in the UI as one integrated site.

Most pad actions require the rover on the pad: Unload (ice → water/O₂), Build (habitat progress), and Service (dust/damage). Entire 5x5 build pad zone is treated identically while formulating as RL problem.

Explore → Extract → Return to pad → Unload → Build / Service

Build Pad

Weather

Weather in AresSim cycles through four states. It changes how much power the colony generates, how hard each rover action is on the battery, and how fast dust accumulates on the build pad. Every simulation step advances the clock and the weather state. Weather is chosen from this cycle as the episode progresses.

Clear → Dusty → Dust Front → Cold Night → (repeat)

What each state does

Seed-based worlds

Every episode starts from an integer seed (default range 0–99,999). That single number drives the full Mars layout: terrain height and roughness, ice and ore clusters, crater and ridge placement, a validated 5×5 build pad, rover spawn, starting battery and colony reserves, and initial weather. Generation uses a fixed deterministic PRNG (no wall-clock randomness), the same seed always rebuilds the same map before a single action is taken. After each transition the engine emits a SHA-256 state checksum so that identical seed + identical action sequence can produce identical world state.

AresSim Environment for different seeds

Workflow

  seed  ──►  procedural map  ──►  reset(world)
                │                    │
                │                    ▼
                │              action sequence
                │                    │
                ▼                    ▼
         same layout every time   same checksums every time

The Agent

In AresSim, the Agent is whoever is driving the rover. It can be Human (Manual mode), an Learned Policy (Algorithm mode), LLM Agent or from a Replay.

Actions

Cargo limit: 12 kg total (Ice + Ore). Agent has to use Unload action to unload all the contents from it;s Cargo into the Build Pad. Rover cannot enter creater and also moves cardinally.

Objective

The goal is to survive as long as possible while keeping the colony viable. An episode ends when rover battery, rover health, or colony livability hits zero (more on this below). Ideally the agent is expected to:

• Explore the 32×32 map
• Scan and Collect Ore
• Extract Ice
• Unload Ice and Ore at Build Pad.
• Build habitat progress toward 100%
• Service the pad when dust, damage, or power margin is bad
• Recharge in Build Pad when battery is low

Battery Drainage

Battery drain stacks terrain + weather + cargo + action type. Rougher ground and heavier payload cost more per step.

Terrain

Weather

Other factors

UI Walkthrough

Manual, Algorithm, and Replay control bars

Camera viewpoints: perspective, 3D survey, and top-down

Data layers: ice, ore, elevation, and weather

Lighting across the map and build pad

Run analytics, mission rewards, guide, and timeline

Manual, Algorithm, and Replay control bars

1 / 5

Formulating the RL problem

AresSim is a single-agent, discrete-time, partially observable MDP on a 32×32 Mars grid. The hidden full world state $s_t$ lives in the simulator; the policy only sees observation $o_t$ and mask $m_t$ (Agent can only see 8x8 grid region around it with it present at middle of the region).

Horizon: episodes end on terminal survival failure or external truncation at $T_{\max}$ (e.g. 1200 steps in rollouts).

Environment

• World: 32×32 cells, one rover (rover_0), one 5×5 build pad, procedural terrain from seed.
• Task: phase1_open_exploration_v1 — survive and explore; no win condition.
• Clock: weather cycles each step; sol/time advances deterministically.
• Truth vs view: simulator owns rules; the RL layer only projects observation, mask, and reward.

Weather (cycles: Clear → Dusty → Dust Front → Cold Night):

Solar generation (drives colony power margin):

Power margin (negative margin drains rover battery even on Wait):

Battery drain (per step, simplified):

$r_{\text{cell}}$, $d_{\text{cell}}$ = local roughness and dust (0–1). Recharge applies on Wait when $\Delta P > 0$.

Agent

The agent is the rover decision-maker: pick $a_t \in \mathcal{A}$ each step from $(o_t, m_t)$.

Policy input:

$m_t[i]=1$ iff action $i$ passes simulator validation; Wait is always legal. Masks come from rules, not from the network.

Phase 1: one agent rover_0. Human play uses the same actions under actor Player; algorithms use actor Agent.

Terminal conditions

State

Hidden state $s_t$: full 32×32 map, all channels, colony reserves, weather, dust, structures.

Observation $o_t$ — schema aresim.obs.local.v1 (rover fixed at local $[3,3]$, window $8×8$):

Partial observability: cells outside the window are unknown padding, so the policy must move and scan to learn the map.

Actions & rewards

Action effects:

Training reward $r_t^{\text{RL}}$ (profile shaped_train) is weighted sum of terms, then clipped to $[-2, 2]$ on non-terminal steps:

Try it yourself

git clone https://github.com/shanmukh05/AresSim.git && cd AresSim
python3 -m venv engine/.venv
engine/.venv/bin/pip install -e './engine[dev,env]'

Inspect the Environment through API

from aresim.factory import make_gym_env

env = make_gym_env(max_episode_steps=200)
obs, info = env.reset(seed=1447)
print("schema:", info["observation_schema"])
print("checksum:", info["state_checksum"])
print("mask:", obs["action_mask"])

obs, reward, term, trunc, info = env.step(5)  # Scan
print("reward:", reward, "scanned flags changed:")
print("terminal:", term, "reason:", info.get("terminal_reason"))

Look at:

• obs["observation"]["terrain_type"] — local terrain IDs
• obs["action_mask"] — which buttons would be green
• info["reward_breakdown"] — why reward moved
• info["engine_reward"] vs reward — two scoring systems
• info["state_checksum"] — reproducibility spot-check

Play - Export - Replay in UI

# terminal 1
engine/.venv/bin/python -m aresim.api
# terminal 2 
npm install && npm run dev

1. Start Frontend + Backend (commands above).
2. Manual mode — set seed, move/scan/extract, watch battery and weather in the HUD.
3. Export — save the run as a trajectory JSON from the action bar.
4. Replay — switch to Replay mode, load the file, step through the same episode without re-simulating decisions. Check Reun Analytics, Rewards, Timeline, Guide, Settings Panel.

What's next?

My next focus is on running few RL Algorithms such as PPO (& it's variants), DQN (& it's variants) and see how they're perform on AresSim. Once I cover enough RL algorithms, I will trying-out different LLMs as Agents. Using LLMs requires us to design the state, action and other environment information very efficienlty for long-range exploration.

As of the day of publishing this blog, I added Baselines (Random, Random-valid, Wait and Scripted) and PPO (Masked) initial version. Check teh repo for more details about these.

References

1. Suárez, J. et al. “Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning.” arXiv:2311.03736 (2023).
2. Neural MMO
3. Mars 2020 Images — Perseverance rover imagery from PDS Geosciences.
4. The Beginner's RL Playground — Arthur Juliani
5. Awesome Deep RL
6. Suarez, Joseph et al. “Neural MMO v1.3: A Massively Multiagent Game Environment for Training and Evaluating Neural Networks.” Adaptive Agents and Multi-Agent Systems (2020).

Tools

• UI Design/Development: ChatGPT/Codex
• Editor: Cursor
• Flowchart and Diagrams: Excalidraw
• Screen Recording: OpenScreen