In a mixing bowl, cream butter and sugars until fluffy. Add eggs; beat for 2 minutes. Add banana and vanilla; beat for 2 minutes. Combine the flour, baking powder, baking soda, and salt; add to creamed mixture alternately with buttermilk. Pour into two greased and floured 9 inch round baking pans. Bake at 375 for 25-30 minutes or until a toothpick inserted in the center comes out clean. Cool in pans 10 minutes; remove to wire racks and cool completely.
For filling, combine half-and-half cream, sugar, butter, flour and salt in a saucepan. Bring to a boil; cook and stir for 2 minutes. Remove from the heat; stir in vanilla and pecans. Cool. Spread between cake layers.
For frosting, beat whipping cream until soft peaks form. Gradually beat in the confectioners sugar; beat until stiff peaks form. Spread over top and sides of cake. Store in the refrigerator.