In a saucepan over medium heat bring milk, ginger and lavender to a boil. Remover from heat and cover and let steep for 10 minutes. Pour mixture through a sieve over a bowl, pressing on the solids. Discard solids.
In a bowl whisk together 1/3 cup sugar and egg and yolks for 1 minute.
Slowly whisk in hot milk mixture until blended.
Divide mixture among 6 custard cups or ramekins and place them in a shallow roasting pan. Add enough boiling water to come halfway up sides of custard cups.
Bake for 35-45 minutes.
Transfer custard cups to rack and cool completely.
Chill covered until set (at least 4 hours but can make these ahead and refrigerate for up to 3 days).
To serve and make topping; preheat broiler and srpinkle custards evenly with 4 teaspoons sugar and place on bottom of broiler pan. Broil for 2-5 mintues (5 inches from heat). Remove from oven and let stand a few minutes before serving. Sprinkle with fresh lavender if desired.