Custard Sauce: In a heavy saucepan, combine 1/2 cup milk & cornstarch. Stir with wire whisk until cornstarch dissolved. Add remaining 2.5 cups of milk and sugar. Cover over moderate heat. Stir until sauce thickens and comes to a boil.
In small bowl, stir egg yolks with a fork and stir in 4-5 TBSP of custard sauce. Whisk mixture back into remaining sauce.
Return to boil and boil for 1 minute while stirring constantly. Remove from heat and add vanilla.
Trifle: Cut pound cake into 1 inch thick slices. Spread jam on slices. Place slices of cake, jam side up, in glass trifle bowl creating a first layer. Combine sherry and brandy. Sprinkle first layer with sherry & brandy combination. Repeat with rest of cake & alcohol mixture.
Let rest for 30 minutes at room temperature. While cake is resting, beat whipping cream in a cold bowl until soft peaks form. Add 2 TBSP of sugar to whipping cream and stir.
Pour custard sauce over top of cake. Then top with whipped cream and garnish with fresh fruit.