Work

ProteinTuneRL: A Reinforcement Learning Perspective on Antibody Design

Reinforcement LearningAntibody DesignPyTorchScientific Machine LearningProtein Language Models

I extended IgLM antibody infilling with token-level A2C and GAE, studied how reward changes with policy divergence, and added ESM-2 and reference-structure tools to evaluate sequences independently of the training reward.

ProteinTuneRL diagram showing antibody sequence infilling within a masked region

In this project, I extended LLNL’s ProteinTuneRL to study how reinforcement learning can optimise an antibody sequence while retaining the constraints learned from natural antibodies. The task was CDR-H3 sequence infilling, where the surrounding heavy-chain framework stays fixed and IgLM, an antibody language model, generates a variable-length replacement for the missing loop.

My main contribution was a token-level A2C critic with Generalised Advantage Estimation (GAE), alongside correctness improvements to the online RL pipeline and tools for independent evaluation. These experiments analyse how the optimisation behaves, but they do not establish improved experimental binding or validated antibody function.

The design problem

Antibody binding involves complementarity-determining regions (CDRs), and CDR-H3 is a particularly variable loop within these regions. IgLM uses the unmodified sequence as context and generates the missing region one token at a time, after which the completed sequence receives a scalar reward. A KL penalty measures divergence from the pretrained policy and discourages the optimiser from moving too far away from it while improving that reward.

The example reward, ss_perc_sheet, uses Biopython’s composition-based secondary-structure fraction, which is an inexpensive amino-acid statistic rather than a prediction of the folded structure or a measurement of binding affinity. By default, it scores the combined heavy and light chains, while an optional region-only mode scores the infilled span and requires separately calibrated normalisation constants.

How the training workflow operates

  1. Mask the annotated heavy-chain CDR-H3 while retaining the framework and paired light chain.
  2. Sample a replacement from IgLM, restricting control tokens to the permitted generation support.
  3. Reconstruct the full sequence and compute the proxy reward.
  4. Apply an optional KL penalty against the frozen reference policy.
  5. Update the policy using REINFORCE, PPO, or A2C; record rewards, divergence, and checkpoints.

The implementation uses configuration-driven factories for datasets, tokenisers, models, metrics, trainers, and optimisers, so the components can be selected without rewriting the training workflow. Offline DPO and DRO are also available, although the figures on this page concern the online methods.

Implementation contributions

AreaImplemented changeWhy it matters
A2C + GAEToken-level value head and masked advantage estimationPropagates a terminal reward through the generated span
Generation supportConsistent constrained-token log probabilities in PPO, A2C, and reference KL scoringKeeps sampling and recomputed probabilities on the same support
Episode boundariesBootstrap the final value only for maximum-length truncationDistinguishes cut-off episodes from stop-token termination
PPO baselineLearned value baseline conditioned on the prompt onlyAvoids conditioning the baseline on the sampled action
Adaptive KLScalar, cross-process controller update using the target divergenceKeeps the regularisation coefficient consistent across ranks
Device placementAuxiliary tensors and critics follow the supplied policy deviceSupports deliberately CPU-placed policies

A2C changes how credit is assigned across the generated tokens, but its critic adds computational cost, so it should not be assumed to be lighter or more stable than the other methods. Its usefulness depends on the task, hyperparameters, and sequence length, and stop-token credit assignment remains a known limitation of the current implementation.

Training behaviour

I examined four 2,000-step, batch-size-eight runs and plotted reward and KL against both update steps and sampled sequences, because the number of updates alone does not describe how much sequence sampling a method has used. The learning rates differ across methods, so the plots describe these runs rather than providing a controlled, multi-seed ranking. Where the KL penalty is enabled, it is included in the reward, which means that the plotted value cannot be interpreted as raw biological quality.

The following figures come from the existing project repository and show smoothed training traces; they are not newly generated experiments or held-out biological validation.

Training traces for PPO, A2C, and the REINFORCE variants

PPO

PPO training traces showing reward and KL divergence against update steps and training samples

PPO shows relatively smooth progress before reaching a reward plateau, and its clipped update constrains policy movement, although this curve alone cannot demonstrate that antibody function has improved.

A2C with GAE

A2C training traces showing reward and KL divergence against update steps and training samples

The token-level critic produces different early learning dynamics from PPO, which makes it useful to investigate whether assigning credit at this level can improve the balance between reward and divergence for short, variable-length loops. A single training trace is not sufficient to establish that one algorithm is better.

REINFORCE

REINFORCE training traces showing reward and KL divergence against update steps and training samples

REINFORCE provides a sequence-level policy-gradient baseline, and its noisier updates help show why variance reduction and divergence control matter in this setting.

Explicit-KL REINFORCE

Explicit-KL REINFORCE training traces showing reward and KL divergence against update steps and training samples

Stronger KL regularisation changes the balance between improving the proxy and retaining the pretrained policy, so a lower regularised reward may reflect the stronger penalty rather than a worse raw proxy score.

MethodCredit assignmentMain controlInterpretation
REINFORCESequence-level reward and baselineEntropy bonus; optional reference KLSimple baseline with potentially noisy updates
PPOSequence-level clipped likelihood ratioClipping; mean or prompt-only learned baselineConservative policy updates
A2C + GAEToken-level values and advantagesGAE, entropy, optional gradient clippingTests finer credit assignment for infilling
Explicit-KL REINFORCESequence-level policy gradientStronger reference-policy penaltyExposes the reward–drift trade-off

Evaluating the generated sequences independently

An optimiser can improve a compositional statistic without improving the folded antibody, so I added two evaluation tools that check the generated sequences independently of the training reward.

EvaluatorReportsDoes not establish
ESM-2 masked pseudo-log-likelihoodWhole-heavy-chain or generated-region scores; mean or sum reduction; batched residue maskingExperimental affinity, developability, or function
Reference-PDB comparisonFramework-aligned backbone RMSD, equal-length region RMSD, loop-length difference, SASA, and region mean B-factorDocking accuracy without a native complex; pLDDT unless stored as B-factor by the producer

The documented ESM-2 smoke test used cached facebook/esm2_t6_8M_UR50D weights on five held-out sequences and took 9.5 seconds, producing a mean whole-chain log probability of −2.367 and a mean HCDR-H3 log probability of −3.051. These values show that the scoring workflow runs, but they do not compare trained policies. The project status records 17 passing tests covering the RL fixes, optimiser integration, ESM masking, and synthetic PDB alignment cases.

Reference-structure evaluation still requires a matched pair of external experimental and predicted structures. The bundled data provide paired sequences, CDR annotations, and computed proxy scores, but they do not include antigen structures, native complexes, epitopes, or experimental affinity labels, so they cannot be used as docking ground truth.

Scope and next questions

  • The online trainer currently supports one reward metric per run, while multi-objective reward composition, experimental biological validation, and a reference-complex docking benchmark remain future work.

  • Imperial HPC access has ended, so its scaffolding has been removed, although the existing research data and training outputs are retained.

  • The next experiments should compare token-level and sequence-level credit assignment under matched settings and examine whether increasing the proxy reward coincides with declining independent ESM or structural scores across different KL budgets.

Working on this project helped me examine how constrained sequence optimisation behaves and how it can be evaluated, while also making clear why a higher proxy reward is insufficient evidence of a better antibody.

Tools and methods

AreaTools and methods
Sequence modellingIgLM antibody infilling, constrained-token generation
Online optimisationPyTorch, REINFORCE, PPO, token-level A2C, GAE, KL regularisation
Training rewardBiopython composition-based secondary-structure fraction
Independent evaluationESM-2 masked pseudo-log-likelihood, reference-PDB alignment and comparison
ValidationRL and optimiser tests, masking checks, synthetic structure-alignment cases

Code and attribution

The framework and infilling illustration originate from LLNL and contributors. The implementation extensions and analysis described here build on that work; upstream attribution and the repository licence are retained.

Start a conversation

Have something
in mind?

If you have a question about my work or something you would like to work on together, I would be glad to hear about it.

Get in touch