Preheat the oven to 350˚F. Lightly spray a 9 inch round pan and set aside.
In a small bowl, combine the raspberries and the brown sugar. Set aside.
In a large bowl, combine the flour, white sugar, baking powder, baking soda and salt. Add the buttermilk, butter, vanilla and egg. Mix until fully incorporated.
Pour batter into loaf tin. Top with the raspberries, then sprinkle the slivered almonds on top.
Bake for 40 minutes, or until a toothpick inserted comes out clean. Let cool on wire rack.
Combine the confectioners sugar, milk and vanilla extract until well mixed. Drizzle over cake.