Combine the first 5 ingredients in a large mixing bowl; stir well. Add the banana, 1/3 cup buttermilk, oil, and vanilla. Beat at a low speed of an electric mixer until blended. Beat an additional minute at high speed. Add remaining 1/3 cup buttermilk and egg yolks, and beat at a high speed 1 minute or until blended.
Beat the egg whites at high speed until foamy. Gradually add 1/3 cup sugar, 1 tbsp at a time, beating until stiff peaks form and the sugar dissolves (2 to 4 minutes). Fold the beaten egg whites and the pecans into the banana mixture.
Pour the batter into 2 greased and floured 9-inch round cakepans. Bake at 350 degrees for 30 to 35 minutes or until a wooden pick inserted in the center comes out clean. Cool in the pans on wire racks for 10 minutes; remove from pans, and let cool completely on wire racks.
For frosting: Beat the whipping cream in a large mixing bowl until foamy; gradually add the powdered sugar and nutmeg, beating until soft peaks form.
Spread 3 1/2 cups Nutmeg Whipped Cream between the layers and on top and sides of cake. Spoon remaining Nutmeg Whipped Cream into a decorating bag fitted with a large star tip; decorate cake. (Or you can just put as much whipped cream as you'd like between the layers and on top, and forget about decorating!) Cover and chill. Prep time does not include the time for cooling the cake layers before frosting.