Exploring Historical Self Play for Autocurricula Generation

Contributions and acknowledgments. Anand and Sandy devised the project and provided direction throughout. Anand developed the implementation and obtained the initial results. Sandy reran experiments to check the results and provided feedback for improvements. We thank Logan Cross and Kevin Xu for discussions. These credits are preserved from the original manuscript. Code is linked at https://github.com/AnandS29/AsymmPlay.

Editorial status. The original PDF contains a one-page overview followed by the full report. This revision consolidates repeated material while preserving its research question, reward construction, sampling distributions, experimental settings, all ten main-report figure comparisons, discussion, future directions, and references. It preserves the historical study rather than reporting new experiments. The original figures remain in the original PDF; their evidence is described here without inventing numerical traces. Raw data, exact seed counts, and implementation details not given by the report remain unresolved. Review notes distinguish corrections to unsupported conclusions from the original empirical observations.

1 Introduction

An agent in a sparse-reward environment may observe little feedback until it reaches a small set of rewarding states. Exploration methods can help the learner encounter those states [1], [2], [3]. Another approach is to organize experience into a curriculum [4], [5]: a sequence of tasks that gives the learner useful intermediate problems before asking it to solve a difficult final task.

Curriculum design has a familiar supervised-learning analogue. Examples can be ordered by a known difficulty measure, such as noise level, or chosen by an expert to introduce more complex distinctions gradually [4]. But difficulty is not always easy to define, especially for images, geometric structure, or a large state-action space. An automatic curriculum attempts to generate this sequence without requiring an expert to specify every intermediate task.

This project studies asymmetric self-play [6]. A teaching agent learns to propose tasks; a student learns to complete them. The teacher’s objective is designed to favor tasks that reveal what the student cannot yet do. As the student improves, the teacher faces a changing problem and can propose new tasks. The student likewise learns against a changing curriculum.

Two changing neural policies can produce unstable training. Adversarial neural-network training provides a related example [7]: progress in one network changes the objective faced by the other, potentially contributing to oscillations or weak gradients. The analogy motivates testing whether exposure to historical policies can make the teacher-student interaction more stable [8].

The game-theoretic motivation comes from fictitious play, in which a player responds to an empirical distribution of previous opponent behavior. Historical self-play (HSP) uses stored neural policies to approximate an aspect of that idea. The question investigated here is empirical: how does this additional sampling mechanism affect training and evaluation compared with ordinary asymmetric self-play?

An equilibrium interpretation requires care. A Nash equilibrium is a profile from which no player can profitably deviate unilaterally. A network that stops changing, or a reward curve that becomes flat, need not satisfy that property. The original report often moves directly from observed plateaus to equilibrium language; this revision retains the plateaus as observations and treats equilibrium as a separate, untested claim.

2 Related work

2.1 Adversarial learning

Generative adversarial networks train a generator and discriminator through opposing objectives [7]. In our teacher-student setting, one network generates tasks and the other attempts them. The construction is analogous in its coupled training, but the roles and objectives differ: the student solves tasks rather than discriminating data, and the eventual objective is a capable student rather than a generator. The analogy does not by itself transfer a convergence result from one system to the other.

2.2 Curriculum learning

Curriculum learning organizes training examples or tasks into a useful sequence [4]. Automatic goal and reverse-curriculum methods reduce the need for manual task selection [9], [10]; multi-agent interactions can also generate curricula [5]. Our approach uses a learning teacher to choose tasks for a student. Calling the curriculum unsupervised refers to the absence of manually labeled task sequences, not the absence of reward functions or environment design.

2.3 Self-play

Self-play has a long history in reinforcement learning, including checkers, backgammon, and Go [11], [12], [13]. Sukhbaatar et al. [6] introduce an asymmetric task-setting approach. Our teacher has a similar task-setting role. We use the student’s expected reward in the teacher’s objective and study resettable goal-reaching tasks, instead of requiring the student to reverse every action the teacher performed.

2.4 Fictitious play

In fictitious play, a player best-responds to an empirical distribution of the opponent’s previous strategies [14]. Extensive-form and deep-learning adaptations include full-width extensive-form fictitious play and sampled approximations [15], [16]. The original report contrasts its discrete-time procedure with these approaches as though they were uniformly continuous-time methods; that broad distinction is not established by the report. The relevant distinction here is its implementation: storing historical networks and sampling them, following the practical motivation in [8].

Classical fictitious-play analysis, including the historical discussion by Berger [17], provides motivation for studying past-policy mixtures. It does not prove that every procedure described as historical averaging converges. Assumptions about the game, best-response accuracy, update rule, and empirical distribution matter.

3 Asymmetric self-play

The student and teacher have separate policies, $\pi_{\mathrm{student}}$ and $\pi_{\mathrm{teacher}}$. During a teaching iteration, the teacher proposes a task and the student trains to accomplish it. The original report defines the teacher’s shaped reward by Equation (1)$$\begin{equation} \label{eq:teacher} \widetilde R_{\mathrm{teacher}} =\gamma\max\left\{0, R_{\mathrm{teacher}}-\mathbb{E}_{\tau\sim\pi_{\mathrm{student}}}[R_{\mathrm{student}}(\tau)]\right\}. \end{equation}$$ Here the student expectation is over trajectories induced by its policy on the proposed task. This writes explicitly the trajectory interpretation given in the original prose, where the expectation’s subscript alternates between a state and a policy. $R_{\mathrm{agent}}$ denotes the task return used for each agent, and $\gamma$ scales the teacher’s shaped reward. In this equation $\gamma$ is a reward-scale parameter, not necessarily the discount factor of the underlying learning algorithm.

The intended incentive is to reward a teacher when its proposed task exposes a gap between its own return and the student’s expected return. Estimating the expectation by averaging sampled student trajectories may reduce sensitivity to an individual stochastic rollout. It also adds sampling cost, and the report does not specify the number of samples or whether the student is evaluated before or after updating on the proposed task.

The original text states that the positive-part operation prevents the teacher from choosing tasks that are too difficult. Equation equation (1) alone does not establish that conclusion. Its behavior depends on the definitions and signs of the returns, feasibility of the task, and what the teacher must demonstrate. A complete difficulty-control claim needs those conditions stated explicitly.

3.1 Resettable and reversible environments

A resettable environment allows the teacher to act, then restores the initial conditions so that the student can attempt the task. A reversible environment allows the student to undo changes and return to the teacher’s initial state. These create different task families.

The experiments use a resettable environment. The teacher proposes an end state or goal, and the student starts from the same initial conditions. The student need not reproduce the teacher’s route in reverse. This leaves flexibility in how it reaches the goal and may support alternative routes or intermediate subgoals. Those possible benefits motivate the setup; they are not separately measured by the reported experiments.

3.2 Historical averaging and fictitious-play motivation

In a two-player discrete-time fictitious-play model, each player responds to the other player’s past empirical strategy distribution. The original report refers to Brown’s and Robinson’s analyses [17], [14] and notes both alternating and simultaneous update conventions. Their equivalence or qualitative similarity requires a specified setting; this report does not prove it for the learning system implemented here.

The source states that convergence of strategies can imply a Nash equilibrium under classical fictitious-play assumptions. The conditional statement is different from proving convergence, and its assumptions are not shown to hold for our neural teacher and student. In particular, approximate policy-gradient updates need not be exact best responses, and an exponentially weighted mixture differs from the ordinary empirical average.

3.2.1 Application to reinforcement learning

The implementation stores past network parameters as historical policies. At a sampling opportunity, an agent either uses a historical network or continues with its current network. The original report describes rolling out and updating the selected model with the chosen reinforcement-learning algorithm. It leaves the precise bookkeeping open: whether old checkpoints remain fixed, whether an update overwrites a stored checkpoint, and how the updated model becomes the current policy all require confirmation from code.

Three choices specify the historical-sampling scheme: the probability of using history, the distribution over stored policies, and whether the selection occurs once per trajectory or at individual time steps. They need to be specified for both agents if the method is to be compared with a two-sided fictitious-play procedure.

3.2.2 Sampling distributions

Let $\mathcal P=\{P_1,\ldots,P_n\}$ be the ordered set of stored policies, with larger indices denoting more recent policies. A uniform choice has probability mass function $$\Pr(P_i)=\begin{cases}1/n,&P_i\in\mathcal P,\\0,&P_i\notin\mathcal P.\end{cases}$$ An exponentially weighted choice has probability mass function $$\Pr(P_i)=\begin{cases} \displaystyle\frac{\alpha^i}{\sum_{j=1}^n\alpha^j},&P_i\in\mathcal P,\\[0.6em] 0,&P_i\notin\mathcal P. \end{cases}$$ For $\alpha>1$, recent policies receive more weight. Uniform sampling treats early and late checkpoints equally. Exponential weighting may reduce exposure to little-trained policies, but whether that improves learning depends on the task and history; the distribution alone does not guarantee better performance.

3.2.3 Inter-trajectory sampling

An agent samples a historical model with the specified probability before a rollout and uses that model throughout the trajectory. The resulting behavior corresponds to one selected historical policy for that episode. The source then applies the learning update to the selected model.

3.2.4 Intra-trajectory sampling

An agent has an opportunity to sample from history at each step. Several policies can therefore contribute actions to a single trajectory. Such a trajectory generally differs from one generated by first sampling a complete policy and keeping it fixed. This distinction can affect credit assignment and the match between the collected behavior and the learner’s update assumptions.

4 Experiments

4.1 Setup

The environment is a partially observable grid world. A key and a locked door create a dependency: the agent must obtain the key and use the door to access another part of the space. The limited field of view adds uncertainty. This is similar in spirit to the environment used by Sukhbaatar et al. [6].

4.1.0.1 Original Figure 1: grid-world environment.

The original image depicts a walled grid divided by an interior barrier, with a key on one side, a door in the barrier, and a goal beyond it. An agent and its shaded field of view illustrate partial observability. This description preserves the environmental structure visible in the original figure; it does not supply an unreported environment specification or an exact map file.

Teaching and historical-teaching conditions first pretrain with teacher-proposed goals, then train on a fixed goal using ordinary reinforcement learning. The nonteaching condition uses only the latter phase. Following the accounting convention described in [6], the report treats pretraining interactions as “free” in the downstream comparison. Therefore, better downstream performance should not be interpreted as an equal-total-interaction sample-efficiency comparison.

Evaluation uses three goal settings: the training goal, a different hand-selected goal, and multiple randomly chosen goals. These test different forms of transfer within the environment. The report states that results are averaged over multiple random seeds, but gives neither the seed count nor uncertainty intervals.

The student uses PPO [19]. The teacher is described in the source as “Asynchronous Actor Critic (A2C)” with citation [18]. The name and abbreviation are inconsistent: the exact asynchronous or synchronous implementation should be checked in the code before assigning an algorithm label more precisely. For exponential historical sampling, the report sets $\alpha=1.2$, following the choice attributed to [8].

4.2 Ordinary training, teaching, and historical teaching

The main comparison uses these labels:

• nt: ordinary reinforcement learning, with no teacher pretraining;

• t: teacher-based pretraining followed by ordinary training;

• th: teacher-based pretraining with historical sampling, followed by ordinary training.

The teaching setup uses ten teaching iterations with ten student iterations each. In the historical condition, the probability of sampling a past policy is $0.2$, selection is exponentially weighted, and sampling occurs between trajectories. All conditions have fifty nonteaching iterations. The report does not translate these iterations into a complete accounting of environment transitions.

4.2.0.1 Original Figure 2: evaluation on the training goal.

The original figure plots average training return and average evaluation return against iteration for all three conditions. Both teaching conditions begin the downstream phase above the nonteaching condition. The teaching and historical-teaching curves are broadly close, while the nonteaching evaluation curve improves from a lower level.

4.2.0.2 Original Figure 3: evaluation on a different goal.

The original training and evaluation panels again compare nt, t, and th. The two pretrained conditions show similar evaluation performance over the displayed iterations, while the nonteaching condition improves from a lower starting point. This is evidence about the selected alternate goal, not a guarantee of transfer to arbitrary goals.

4.2.0.3 Original Figure 4: evaluation on multiple random goals.

The original panels use the same condition labels. The ordinary-teaching evaluation curve rises above the historical-teaching curve later in training, while nonteaching remains lower over the displayed range. The report interprets this divergence as a possible exploration cost of historical sampling. That explanation remains a hypothesis: the plotted returns do not directly measure the explored state space or identify the causal mechanism.

Across these comparisons, teacher pretraining is associated with better initial downstream performance. The report also notes limited further improvement in some pretrained conditions. One proposed explanation is that the learner has settled into a policy difficult to change during ordinary training. It is not justified to identify that state as a Nash equilibrium from the return curves alone.

The original text further suggests that the initial goal is difficult and that success on other goals may reflect learning easier alternatives while adapting to the target. This interpretation is retained as a possible explanation, not a measured ordering of goal difficulty. Comparing goal difficulty would require the goal definitions and an explicit difficulty criterion.

4.3 Historical-sampling choices

The remaining experiments compare sampling distributions, historical-use probabilities, and sampling times. The original figures plot training and evaluation return for the training goal (same), an alternate goal (diff), and random goals (rand). Their numerical trajectories are not recreated here because the underlying data are not included in the manuscript. The original PDF retains the full plots.

4.3.1 Uniform versus exponential sampling

4.3.1.1 Original Figure 5: exponential sampling.

The figure reports training and evaluation curves under the exponential distribution. Its evaluation curves for the alternate and random goals are above the same-goal curve in the displayed run aggregation. The original discussion describes stronger performance and relatively limited further change compared with uniform sampling.

4.3.1.2 Original Figure 6: uniform sampling.

The alternate-goal and random-goal evaluation curves improve during the displayed iterations. The original report compares their lower initial values and continued improvement with the exponential condition and proposes that favoring recent policies can be useful. The comparisons support a qualitative observation for these experiments, not a general theorem about sampling distributions.

The source interprets the more stable exponential curve as convergence to an equilibrium and the changing uniform curve as failure to converge. This revision does not adopt that inference. Stationarity of average return is not a test of unilateral exploitability, and a policy can change without a large change in its average return.

4.3.2 Probability of historical sampling

4.3.2.1 Original Figure 7: historical probability $0.05$.

The source uses exponential weighting and a $0.05$ probability of historical selection. It describes the curves as relatively smooth compared with the higher-probability condition, with the alternate and random goals performing better than the same-goal evaluation in the displayed comparison.

4.3.2.2 Original Figure 8: historical probability $0.4$.

The source uses the same distribution family with probability $0.4$. Its discussion reports somewhat greater variation, but broadly similar qualitative behavior. More frequent historical selection changes exposure to past policies. Under exponential weighting it still favors recent checkpoints; increasing the selection probability does not itself alter the conditional distribution over historical indices.

The original interpretation is that exponential weighting may be relatively robust to the tested probabilities. Only two values are compared here, and neither uncertainty estimates nor a formal sensitivity analysis is provided. The result therefore does not establish robustness over a larger range or relative to untested uniform-sampling schedules.

4.3.3 Between-trajectory versus within-trajectory sampling

4.3.3.1 Original Figure 9: inter-trajectory historical averaging.

The original training and evaluation panels show a condition using one selected historical policy for a trajectory. In its evaluation panel, the same-goal curve is higher than the alternate- and random-goal curves. The report describes stronger performance than the intra-trajectory condition over the displayed comparison.

4.3.3.2 Original Figure 10: intra-trajectory historical averaging.

The corresponding panels show continued improvement from lower returns and lower transfer to the alternate and random goals. The source proposes that changing policies during a trajectory makes learning more difficult and may require additional teaching iterations. This remains an explanation to test, rather than a demonstrated failure of equilibrium convergence.

The two sampling schemes change the behavior distribution in distinct ways. Sampling a whole policy once per trajectory produces a mixture over policy-generated trajectories. Resampling inside a trajectory can create action sequences no stored policy would produce by itself. That distinction motivates checking the update rule and credit assignment before attributing the performance difference solely to the frequency of policy changes.

5 Discussion

5.1 Effectiveness of historical averaging

The experiments report that some historical self-play configurations attain performance similar to ordinary teacher-based pretraining. Both can improve downstream performance relative to the nonteaching baseline under the report’s pretraining-accounting convention. Exponential weighting and inter-trajectory selection are simple mechanisms for exposing learners to historical behavior while preserving recent experience.

The original discussion presents historical averaging as adding a convergence guarantee without harming performance. Neither part is established in full generality here. Performance differs across evaluation goals and sampling choices, and classical fictitious-play guarantees have not been proved for the particular network updates, task game, or historical distribution. The defensible contribution is a practical comparison and a reason to investigate the mechanism further.

5.2 Choosing a historical-sampling scheme

Within the reported experiments, inter-trajectory sampling and exponential weighting provide a plausible starting configuration. The random-goal comparison suggests a possible cost in exploration or transfer relative to ordinary teaching. Lower historical-use probability appears smoother in one comparison, while the broader effect of that probability remains limited by the tested settings.

These observations can guide a subsequent experiment, but they do not determine a generally optimal schedule. A stronger comparison would hold total environment interactions fixed, specify checkpoint management, report variability across seeds, and measure both task performance and an explicit stability criterion.

5.3 Future work

5.3.1 Stochastic fictitious play

The original report proposes adding randomness to payoffs or smoothing responses as a route toward stochastic fictitious play. It motivates this direction by distinguishing pure and mixed strategies. A single neural policy can already be stochastic over actions; sampling a policy from history also defines a mixture at another level. Those meanings must be kept separate.

The future question is which smoothed response rule and game class yield a useful convergence statement, and how accurately the implemented learner approximates that rule. Adding arbitrary reward noise is not, by itself, a proof of convergence to a mixed Nash equilibrium.

5.3.2 Continuous environments

The experiments use a discrete grid world with a key and a door. Continuous state or action spaces would test whether teacher-generated tasks remain useful under more demanding control requirements. The original discussion gives a robotics example: a teacher might generate subtasks corresponding to picking up a key, placing it in a lock, and turning it. These are suggested applications, not experiments performed in the report. They also raise questions about reset costs, feasible task generation, and transfer between subtasks.

5.3.3 Goal-conditioned learning

Another extension gives the proposed goal as an explicit input to the student. A goal-conditioned policy could then distinguish tasks that would otherwise have similar observations but different objectives. The original report suggests applications to hierarchical reinforcement learning and exploration. The question for future evaluation is whether goal conditioning improves transfer and learning efficiency beyond the teacher-pretraining effect already observed.

6 Conclusion

Historical self-play augments a teacher-student curriculum with sampling from past policies. The reported grid-world comparisons examine where that mechanism performs similarly to ordinary self-play and where choices of history distribution or sampling time affect the result. Teacher pretraining is useful under the study’s downstream evaluation convention, while transfer to different goals reveals differences between configurations.

The connection to fictitious play supplies a research direction rather than a completed convergence proof. A next step is to specify the game and learning procedure precisely enough to test that connection, while evaluating the practical method with equal interaction budgets and uncertainty estimates. Which historical mixtures retain useful training diversity without making the student’s task unnecessarily difficult remains open.

Items requiring author confirmation or further evidence

1. Confirm the teacher’s A2C/A3C implementation, network architectures, update schedules, and all omitted hyperparameters.

2. Define the task returns in equation (1), reward scale, trajectory averaging count, and timing of student evaluation relative to training.

3. Specify which agent samples history, when checkpoints are saved, whether they are frozen, and which parameters are updated after historical rollouts.

4. Supply seed counts, goal definitions, environment-step counts, raw curves, and uncertainty estimates. The qualitative results here preserve the original figures; they do not constitute a rerun.

5. Resolve the original convergence claims. Proving a Nash result requires assumptions and a proof beyond citing classical fictitious play or observing a flat return curve.

6. Verify whether comparisons between figures use identical underlying task and training settings; the report does not provide a complete configuration table.

References

[1]

Justin Fu, John D. Co-Reyes, and Sergey Levine. EX2: Exploration with Exemplar Models for Deep Reinforcement Learning. arXiv:1703.01260, 2017.

[2]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level Control through Deep Reinforcement Learning. Nature, 518(7540):529–533, 2015. doi:10.1038/nature14236.

[3]

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous Control with Deep Reinforcement Learning. ICLR, 2016.

[4]

Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum Learning. Proceedings of ICML, pp. 41–48, 2009. doi:10.1145/1553374.1553380.

[5]

Joel Z. Leibo, Edward Hughes, Marc Lanctot, and Thore Graepel. Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research. arXiv:1903.00742, 2019.

[6]

Sainbayar Sukhbaatar, Ilya Kostrikov, Arthur Szlam, and Rob Fergus. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-play. arXiv:1703.05407, 2017.

[7]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. Advances in Neural Information Processing Systems, 27:2672–2680, 2014.

[8]

Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient Exploration via State Marginal Matching. arXiv:1906.05274, 2019.

[9]

Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic Goal Generation for Reinforcement Learning Agents. arXiv:1705.06366, 2017.

[10]

Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse Curriculum Generation for Reinforcement Learning. arXiv:1707.05300, 2017.

[11]

A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development, 3(3):210–229, 1959. doi:10.1147/rd.33.0210.

[12]

Gerald Tesauro. Temporal Difference Learning and TD-Gammon. Communications of the ACM, 38(3):58–68, 1995. doi:10.1145/203330.203343.

[13]

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, 529(7587):484–489, 2016. doi:10.1038/nature16961.

[14]

Julia Robinson. An Iterative Method of Solving a Game. Annals of Mathematics, 54(2):296–301, 1951.

[15]

Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious Self-play in Extensive-form Games. Proceedings of ICML, 37:805–813, 2015.

[16]

Johannes Heinrich and David Silver. Deep Reinforcement Learning from Self-play in Imperfect-information Games. arXiv:1603.01121, 2016.

[17]

Ulrich Berger. Brown’s Original Fictitious Play. Journal of Economic Theory, 135:572–578, 2007. doi:10.1016/j.jet.2005.12.010.

[18]

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. arXiv:1602.01783, 2016.

[19]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017.