Preheat oven to 350. Grease a 15x10x1-inch pan; line bottom with waxed or parchment paper, grease paper, and set aside.
In a large mixing bowl, cream butter and sugar; beat in yolks.
Combine crumbs, flour, and baking powder. Stir in milk and vanilla.
Beat crumb mixture into butter mixture, 1/3 at a time, beating on low speed after each addition. Stir in walnuts; set aside.
Beat egg whites on medium speed until stiff peaks form. Gently stir about half of the egg whites into graham cracker mixture to lighten; fold in remaining whites.
Spread batter into prepared pan. Bake for 25-30 minutes. Cool in pan for 10 minutes. Invert onto large cooling rack; gently peel paper off cake. Cool completely.
Make frosting. Cut cake crosswise into 3 equal parts. Place one piece on serving plate; frost top with one-third of the frosting. Repeat, stacking layers with remaining cake and frosting. Top with crystallized ginger and walnuts, if desired. Serve at once or cover and store in refrigerator up to 8 hours.
Frosting: Beat butter until fluffy. Gradually add 2 cups of the powdered sugar, beating well. Slowly beat in milk and vanilla, then remaining sugar. Beat in additional milk, if needed, to acheive spreading consistency.