When the Agartha singularity manifested, Fergus disappeared from Chaldea, ending up inside the Singularity at a younger age. Fergus is unable to fight most of the enemies they encounter throughout Agartha, as the fighters are all women and he is reluctant to raise a sword against them. As such, he is delegated to protecting Ritsuka Fujimaru.

(Source: TYPE-MOON Wiki)