Biological Connectivity Patterns as a Blueprint for Efficient Neural Architectures in Reinforcement Learning

I worked on this collaborative study through the Neuromatch Impact Scholars Program, where we investigated whether connectivity patterns inspired by biological neural circuits could help a small reinforcement-learning model perform as well as larger networks.

Authors: Muratore, P., Moradimanesh, Z., Shaikh, R.A., Zargarnezhad, N., Meek, A., Kuzovkin, I.

Affiliation: Neuromatch Academy, Neuromatch, Inc., USA

This page is a portfolio account of our shared work; the original report is available on Zenodo.

The problem we studied

• Larger neural networks can improve performance, but they also increase computational requirements, so we wanted to examine whether choosing a useful connectivity pattern could reduce the number of trainable parameters needed for a task.
• We used Neural Circuit Architectural Priors (NCAP), whose controller is based on neural circuits from C. elegans, and compared it with multilayer perceptrons (MLPs) trained from scratch or through knowledge distillation.
• The task was swimming in MuJoCo, controlling six joints across five links without a designated target. NCAP uses only four trainable parameters, while a naive MLP in our experiments needed at least 100 parameters to achieve comparable performance.

How we designed the comparison

We ran 250 experimental trials with randomly generated MLP architectures, so we could examine connectivity alongside parameter count rather than comparing NCAP with only one alternative network.

• For training from scratch, we used PPO because its stability and relative insensitivity to hyperparameters helped us concentrate on architectural differences. SAC and DDPG were alternatives, but sensitivity to training instability in sparse or noisy reward settings was a concern in our setup.
• For distillation, we ran the highest-reward model from previous runs as a teacher, recorded its observations and actions, and trained each student to predict those actions with a mean squared error loss before evaluating the student in the environment.
• We recorded maximum reward and reward standard deviation, ranked models by parameter count, and bootstrapped the maximum-reward distributions with 1000 samples per model, using NCAP’s Z-score within those distributions for comparison.

What the experiments showed

The figure compares the reward achieved by each model with its parameter count, while separating the four network depths, so we can see whether a larger model consistently produces a better controller.

• NCAP achieved comparable performance with substantially fewer parameters, placing within the top 1% among models with up to 39 parameters (p < 0.01), the top 5% with up to 45 parameters (p < 0.05), and the top 10% with up to 74 parameters (p < 0.1). It remained within the top 31% of all evaluated models.
• Some smaller, better-connected MLPs outperformed networks that were orders of magnitude larger but poorly connected, which showed why parameter count alone was insufficient to explain the results.
• As depth increased from one to four layers, the minimum number of parameters needed to reach maximal reward also increased, suggesting that greater depth made this particular task harder to learn.
• Distillation helped models reach good performance within the available training budget of 5 million iterations for training and 2.5 million for distillation. At depth four, only distilled networks reached maximum performance, apart from one model trained from scratch whose reward was close to 1000.

Interpreting the result

The comparison supports using task-relevant connectivity priors when designing small controllers, because networks with similar parameter counts could perform differently depending on how they were connected. However, this was a swimming-task study, so it does not establish that a smaller or biologically inspired network will be better in every setting.

• Learning-rate schedules, initialisation, and random-seed effects remain relevant to the robustness of the comparison, even though the study controlled these factors.
• NCAP remained distinct from the randomly sampled configurations, so the experiments did not show that pruning or distillation alone reproduced its efficiency.
• The useful next question is how much of this advantage transfers to other control tasks and architectures, rather than treating the result as a general argument against larger networks.

Tools and methods

Acknowledgements

The Impact Scholars Program supported this work, which originated in the Neuromatch Academy NeuroAI course in July 2024. Kaitlyn Hoesterey and Ravikrishnan Jayakumar were members of the original team and contributed to its initial conceptualisation.