A Deep Attention-Multiple Instance Learning Framework to Predict Survival of Soft-Tissue Sarcoma from Whole Slide Images

Soft-tissue sarcomas are heterogeneous cancers of the mesenchymal lineage that can develop anywhere in the body. A precise prediction of sarcomas patients’ prognosis is critical for clinicians to define an adequate treatment plan. In this paper, we proposed an end-to-end Deep learning framework via Multiple Instance Learning (MIL), Deep Attention-MIL framework, for the survival predictions: Overall survival (OS), Metastasis-free survival (MFS), and Local-recurrence free survival (LRFS) of sarcomas patients, by studying the features from Whole Slide Images (WSIs) of their tumors. The Deep Attention-MIL framework consists of three steps: tiles selection from the WSIs to choose the relevant tiles for the study; tiles feature extraction by using a pre-trained deep learning model; and a Deep Attention-MIL model to predict the risk score for each patient via MIL approach. The risk scores outputted from the Deep Attention-MIL model are used to divide the patients into low/high-risk groups and predict survival time. The framework was trained and validated on a local dataset including 220 patients, then it was used to predict the survival for 48 patients in an external validation dataset. The experiments showed the proposed framework yielded satisfactory and promising results and contributed to accurate cancer survival predictions on both the validation and external testing datasets: By using the WSIs feature only, we obtained an average C-index (of 5-fold cross-validation) of 0.6901, 0.7179, and 0.6211 for OS, MFS, and LRFS tasks on the validation dataset, respectively. On the external testing dataset, these scores are 0.6294, 0.682, and 0.76 for the three tasks (OS, MFS, LRFS), respectively. By adding the clinical features, these scores have been improved both on validation and external testing datasets. We obtained an average C-index of 0.7835/0.6378, 0.7389/0.6885, and 0.6883/0.7272 for the three tasks (OS, MFS, LRFS) on validation/external testing datasets.