Earth Observation (EO) systems combined with Artificial Intelligence (AI) techniques have significantly advanced in recent years. The emergence and success of foundational models (FMs), such as ChatGPT and DALL-E 2, based on Large Language Models (LLMs), have influenced the development of the so-called “Geo” Foundational Models (GFMs). These models are trained using Self-Supervised Learning (SSL) strategy on broad datasets and can be adapted to a wide variety of downstream tasks, such as flood mapping, Building Damage Classification (BDC), and Land Cover (LC) classification. This versatility addresses complex issues in Earth science, offering a significant advantage compared to traditional Deep Learning (DL) or Machine Learning (ML) models, which are typically restricted to one or several data types and optimized for a single task, such as classification or change detection. With the launch of GFM Clay, which was initially pre-trained for segmentation, classification, and biomass information extraction using a variety of sensors such as Sentinel-1, Sentinel-2, and Landsat, we fine-tune the model for the task of building damage assessment (BDA) in Beira, Mozambique. Therefore, in this study, the model is fine-tuned to the EDDA dataset, a high-resolution drone dataset released in October 2023, which captures building damage levels after Tropical Cyclone (TC) Idai devastated nearly 90% of Beira, the fourth-largest city in Mozambique. The fine-tuned Clay model achieved a validation mean intersection over union (mIoU) score of 0.829 and an F1 score of 0.869, demonstrating strong performance even with imbalanced data, compared to a fully supervised U-Net model, that we trained using the same dataset, and attained an mIoU of 0.567 and F1-score of 0.717. The qualitative results observed are highly acceptable, underscoring the transferability of the Clay Model to different tasks, datasets, and regions. These capabilities are crucial assets for its potential globalization.