AG-SE 3D U-Net: A Dual-Attention Deep Learning Framework for Automated Whole Prostate Gland Segmentation in Volumetric MRI

Segmentation of the prostate gland from three dimensional (3D) magnetic resonance imaging (MRI) is an essential step in prostate cancer diagnosis and treatment planning. Nevertheless, whole-gland segmentation accurately continues to be a challenge due to the high degree of inter-patient anatomical variability, low soft tissue contrast, and variation in signal intensities across different MRI protocols. This is particularly true at the apex, base, and peripheral zones as boundary error could impact clinical decision making. In this study, we propose the use of a volumetric deep learning architecture called AG-SE 3D U-Net that incorporates attention gates (AG) in all encoder-decoder skip connections and squeeze-and-excitation (SE) blocks in each convolutional block to improve the ability of the model to represent features both spatially and by channels. To ensure stable training of the network, group normalization and a composite loss function that combines binary cross-entropy, dice and focal Tversky losses have been used.
We performed a four variant ablation study (baseline, +SE, +AG, +AG+SE) on 66 paired 3D T2-weighted MRI volumes from the ProstateX database using fivefold stratified cross validation with test time augmentation (TTA). We observed a mean DSC of 0.8697±0.0143, IoU of 0.7730±0.0210, and HD95 of 3.46±0.69mm, with the highest DSC value of 0.8891 in Fold 4.When compared to the baseline 3D U-Net (DSC: 0.8553±0.0192; IoU: 0.7517±0.0273; HD95: 3.37±0.28mm), our proposed model had better segmentation performance based on overlap metrics. Additionally, qualitative evaluation of the images revealed sharper borders and less leakage into adjacent structures. Therefore, our results demonstrate that AG-SE 3D U-Net represents a viable strategy for automatic prostate segmentation.