Preheat the oven to 175°C with top and bottom heat. Grease a 20 × 20 cm baking tin and line the base with baking paper.
Stir the xanthan gum into the flour. Add the softened butter, light brown sugar, speculaas spice mix, whole egg and salt. Mix until a thick dough forms. If it still crumbles when pressed together, mix in 1 tsp milk.
Press the dough evenly into the tin and smooth the top. Whisk the egg yolk with a splash of milk, brush it over the dough and scatter the flaked almonds on top.
Bake for about 30 minutes, until the edges are set and the top is lightly browned. Let the butter cake cool completely in the tin before slicing.