Preheat oven to 350°. Butter and lightly flour the sides and bottoms of two 8" round cake pans; knock out excess flour.
Using an electric mixer on medium speed, cream the butter and sugar together in a large bowl for at least 10 minutes, or until very light and fluffy. Beat in the eggs, mashed banana, and vanilla. Sift the flour, baking soda, salt and spices together onto a piece of waxed paper. Stir into the banana mixture alternately with the buttermilk.
Spoon into the prepared pans and smooth the tops with a rubber spatula.
Bake for 25 to 30 minutes, or until a cake tester inserted in the center comes out clean. Transfer the pans to wire racks and let cool for 10 minutes. Unmold onto the racks and let cool completely.
Meanwhile, make the filling: in a small, heavy saucepan, heat the milk until small bubbles form around the edges of the pan. Meanwhile, beat the sugar and egg yolks together in a small bowl. Beat in the flour and salt. Gradually beat in the hot milk.
Return to the saucepan and cook over medium heat, stirring constantly, until the mixture begins to thicken and comes to a simmer. Lower the heat to low and continue stirring for 2 minutes. Remove from heat and beat in the butter and vanilla. Pour into a bowl, cover with a piece of plastic wrap pressed onto the surface of the custard, let cool slightly, and refrigerate.
To assemble the cake: Split each cake layer in half. If the custard is too thick to spread, thin it with a little cream. Spread 1/3 of the filling and arrange 1/3 of the banana slices evenly on 3 of the layers. Stack the layers and top with the fourth layer. Cover with plastic wrap and refrigerate for 2 to 3 hours.
Just before serving, make the topping: in a deep bowl, whip the cream until very soft peaks form. Beat in the sugar and vanilla. Spread the whipped cream frosting over the top of the cake, allowing some to drip down the sides.