Prepare and bake gingerbread according to package directions using two greased 9 inch round cake pans. Cool completely on wire racks.
Meanwhile, for pudding, in a large saucepan combine pudding mix and milk. Stir until smooth. Cook and stir over medium heat until mixture comes to a boil. Cook and stir 1-2 minutes longer, or until thickened. Remove from heat and cool to room temperature.
Combine pumpkin and brown sugar and stir into pudding.
In a 4-quart glass serving bowl or trifle dish crumble one gingerbread cake and gently press down. Top with half of pudding mixture and half of whipped topping. Repeat layers.