Research

Hierarchical reinforcement learning (HRL) remains limited by sparse rewards, long-horizon credit assignment, and catastrophic interference across behaviours, yet the role of neural network architecture in mitigating these challenges remains underexplored. This work systematically evaluates how architectural design influences stability and multi-behaviour learning in deep reinforcement learning. We first isolate the effects of critic depth, residual shortcuts, and normalisation within the Soft Actor-Critic (SAC) framework across a variety of ubiquitous tasks selected from the DeepMind Control Suite, demonstrating that increased depth improves representational capacity but induces optimisation instability unless supported by Residual Layers and Layer Normalisation. Batch Normalisation, in contrast, consistently degrades performance under non-stationary RL training dynamics. Guided by these controlled experiments, we investigate architectural conditioning in the long-horizon, partially observable environment of Pokémon Red using SAC-Discrete. Utilising an expert system for selecting tasks, we compare naïve task-vector concatenation with Featurewise Linear Modulation (FiLM) across both vector and pixel-based state representations, and we show that FiLM enables more stable multi-stage behavioural learning and reduces interference between sequential tasks. Deeper critics augmented with residual connections and Layer Normalisation further enhance stability and behavioural retention. Collectively, these results demonstrate that architectural capacity and conditioning mechanisms substantially influence hierarchical learning dynamics, highlighting network design as a useful and underemphasised component in scaling reinforcement learning to complex multi-stage domains.