Comparing GDPO and GRPO
by marimo team
A comparison of the GDPO and GRPO reinforcment learning algorithms.
About this notebook
This notebook compares how GRPO (Group Relative Policy Optimization) and GDPO (Group reward-Decoupled Policy Optimization) compute advantages from the same rewards. It follows a paper on alphaXiv. The input is a small table of binary rewards, one row per response, with three reward dimensions: correctness, style, and conciseness. The notebook builds the table as a custom anywidget and computes the advantages with NumPy.
The reward table is interactive. Click a reward cell to toggle it between 0 and 1, and the widget recomputes the advantages under both methods. A Difference column marks the rows where GDPO keeps information that GRPO loses. A switch below the widget selects the training data for a policy-gradient experiment: fresh Bernoulli samples each epoch, or the fixed rewards from the widget. When you flip the switch or edit the table, marimo re-runs the training cells and redraws the plot.
GRPO sums the rewards of each response and then normalizes the totals. GDPO normalizes each reward dimension separately and then sums the results. So when two responses reach the same total from different rewards, GRPO gives them equal advantages and GDPO does not. The notebook trains a three-dimensional Bernoulli policy for 150 epochs with each method and plots the per-dimension probabilities over training with matplotlib.
