Heat oven to 350°F Grease bottom only of 2 (9-inch) round cake pans with shortening or cooking spray. Line bottom with cooking parchment paper; grease and flour paper and sides of pans. In medium bowl, stir together flour, baking powder, baking soda, ginger, nutmeg and salt.
In large bowl, beat granulated sugar and 1/2 cup butter with electric mixer on medium speed 3 to 4 minutes or until fluffy. On low speed, beat in eggs, one at a time, beating well after each addition. Beat in bananas. (Mixture will look curdled.) Alternately beat in flour mixture and buttermilk,beginning and ending with flour mixture. Stir in 1/2 cup nuts. Spread batter evenly in pans.
Bake 30 to 35 minutes or until dark golden brown and cakes begin to pull away from sides of pans. Cool 5 minutes; remove from pans to cooling racks. Cool completely, about 1 hour.
In large bowl, beat frosting ingredients with electric mixer on medium speed until smooth.
On serving plate, place 1 cake layer, rounded side down. Spread with half of frosting. Place remaining cake layer, rounded side up, on frosting, pressing gently to secure (frosting should show around edge). Spread frosting on top of cake, leaving side unfrosted. Sprinkle 1/4 cup nuts around top edge of cake.