In this project, I extended LLNL’s ProteinTuneRL to study how reinforcement learning can optimise an antibody sequence while retaining the constraints learned from natural antibodies. The task was CDR-H3 sequence infilling, where the surrounding heavy-chain framework stays fixed and IgLM, an antibody language model, generates a variable-length replacement for the missing loop.
My main contribution was a token-level A2C critic with Generalised Advantage Estimation (GAE), alongside correctness improvements to the online RL pipeline and tools for independent evaluation. These experiments analyse how the optimisation behaves, but they do not establish improved experimental binding or validated antibody function.
The design problem
Antibody binding involves complementarity-determining regions (CDRs), and CDR-H3 is a particularly variable loop within these regions. IgLM uses the unmodified sequence as context and generates the missing region one token at a time, after which the completed sequence receives a scalar reward. A KL penalty measures divergence from the pretrained policy and discourages the optimiser from moving too far away from it while improving that reward.
The example reward, ss_perc_sheet, uses Biopython’s composition-based secondary-structure fraction, which is an inexpensive amino-acid statistic rather than a prediction of the folded structure or a measurement of binding affinity. By default, it scores the combined heavy and light chains, while an optional region-only mode scores the infilled span and requires separately calibrated normalisation constants.
How the training workflow operates
- Mask the annotated heavy-chain CDR-H3 while retaining the framework and paired light chain.
- Sample a replacement from IgLM, restricting control tokens to the permitted generation support.
- Reconstruct the full sequence and compute the proxy reward.
- Apply an optional KL penalty against the frozen reference policy.
- Update the policy using REINFORCE, PPO, or A2C; record rewards, divergence, and checkpoints.
The implementation uses configuration-driven factories for datasets, tokenisers, models, metrics, trainers, and optimisers, so the components can be selected without rewriting the training workflow. Offline DPO and DRO are also available, although the figures on this page concern the online methods.
Implementation contributions
| Area | Implemented change | Why it matters |
|---|---|---|
| A2C + GAE | Token-level value head and masked advantage estimation | Propagates a terminal reward through the generated span |
| Generation support | Consistent constrained-token log probabilities in PPO, A2C, and reference KL scoring | Keeps sampling and recomputed probabilities on the same support |
| Episode boundaries | Bootstrap the final value only for maximum-length truncation | Distinguishes cut-off episodes from stop-token termination |
| PPO baseline | Learned value baseline conditioned on the prompt only | Avoids conditioning the baseline on the sampled action |
| Adaptive KL | Scalar, cross-process controller update using the target divergence | Keeps the regularisation coefficient consistent across ranks |
| Device placement | Auxiliary tensors and critics follow the supplied policy device | Supports deliberately CPU-placed policies |
A2C changes how credit is assigned across the generated tokens, but its critic adds computational cost, so it should not be assumed to be lighter or more stable than the other methods. Its usefulness depends on the task, hyperparameters, and sequence length, and stop-token credit assignment remains a known limitation of the current implementation.
Training behaviour
I examined four 2,000-step, batch-size-eight runs and plotted reward and KL against both update steps and sampled sequences, because the number of updates alone does not describe how much sequence sampling a method has used. The learning rates differ across methods, so the plots describe these runs rather than providing a controlled, multi-seed ranking. Where the KL penalty is enabled, it is included in the reward, which means that the plotted value cannot be interpreted as raw biological quality.
The following figures come from the existing project repository and show smoothed training traces; they are not newly generated experiments or held-out biological validation.
Training traces for PPO, A2C, and the REINFORCE variants
PPO

PPO shows relatively smooth progress before reaching a reward plateau, and its clipped update constrains policy movement, although this curve alone cannot demonstrate that antibody function has improved.
A2C with GAE

The token-level critic produces different early learning dynamics from PPO, which makes it useful to investigate whether assigning credit at this level can improve the balance between reward and divergence for short, variable-length loops. A single training trace is not sufficient to establish that one algorithm is better.
REINFORCE

REINFORCE provides a sequence-level policy-gradient baseline, and its noisier updates help show why variance reduction and divergence control matter in this setting.
Explicit-KL REINFORCE

Stronger KL regularisation changes the balance between improving the proxy and retaining the pretrained policy, so a lower regularised reward may reflect the stronger penalty rather than a worse raw proxy score.
| Method | Credit assignment | Main control | Interpretation |
|---|---|---|---|
| REINFORCE | Sequence-level reward and baseline | Entropy bonus; optional reference KL | Simple baseline with potentially noisy updates |
| PPO | Sequence-level clipped likelihood ratio | Clipping; mean or prompt-only learned baseline | Conservative policy updates |
| A2C + GAE | Token-level values and advantages | GAE, entropy, optional gradient clipping | Tests finer credit assignment for infilling |
| Explicit-KL REINFORCE | Sequence-level policy gradient | Stronger reference-policy penalty | Exposes the reward–drift trade-off |
Evaluating the generated sequences independently
An optimiser can improve a compositional statistic without improving the folded antibody, so I added two evaluation tools that check the generated sequences independently of the training reward.
| Evaluator | Reports | Does not establish |
|---|---|---|
| ESM-2 masked pseudo-log-likelihood | Whole-heavy-chain or generated-region scores; mean or sum reduction; batched residue masking | Experimental affinity, developability, or function |
| Reference-PDB comparison | Framework-aligned backbone RMSD, equal-length region RMSD, loop-length difference, SASA, and region mean B-factor | Docking accuracy without a native complex; pLDDT unless stored as B-factor by the producer |
The documented ESM-2 smoke test used cached facebook/esm2_t6_8M_UR50D weights on five held-out sequences and took 9.5 seconds, producing a mean whole-chain log probability of −2.367 and a mean HCDR-H3 log probability of −3.051. These values show that the scoring workflow runs, but they do not compare trained policies. The project status records 17 passing tests covering the RL fixes, optimiser integration, ESM masking, and synthetic PDB alignment cases.
Reference-structure evaluation still requires a matched pair of external experimental and predicted structures. The bundled data provide paired sequences, CDR annotations, and computed proxy scores, but they do not include antigen structures, native complexes, epitopes, or experimental affinity labels, so they cannot be used as docking ground truth.
Scope and next questions
-
The online trainer currently supports one reward metric per run, while multi-objective reward composition, experimental biological validation, and a reference-complex docking benchmark remain future work.
-
Imperial HPC access has ended, so its scaffolding has been removed, although the existing research data and training outputs are retained.
-
The next experiments should compare token-level and sequence-level credit assignment under matched settings and examine whether increasing the proxy reward coincides with declining independent ESM or structural scores across different KL budgets.
Working on this project helped me examine how constrained sequence optimisation behaves and how it can be evaluated, while also making clear why a higher proxy reward is insufficient evidence of a better antibody.
Tools and methods
| Area | Tools and methods |
|---|---|
| Sequence modelling | IgLM antibody infilling, constrained-token generation |
| Online optimisation | PyTorch, REINFORCE, PPO, token-level A2C, GAE, KL regularisation |
| Training reward | Biopython composition-based secondary-structure fraction |
| Independent evaluation | ESM-2 masked pseudo-log-likelihood, reference-PDB alignment and comparison |
| Validation | RL and optimiser tests, masking checks, synthetic structure-alignment cases |
Code and attribution
- Project fork on GitHub
- Original LLNL ProteinTuneRL framework
- Project article: a reinforcement learning perspective on antibody design
The framework and infilling illustration originate from LLNL and contributors. The implementation extensions and analysis described here build on that work; upstream attribution and the repository licence are retained.