Prepare a 8 1/2- x 4 1/2-inch loaf pan (or Nordicware gingerbread loaf pan) with a non-stick baking spray.
Using some light brown sugar (or plain granulated sugar), dust the bottom and all sides of the pan, leaving a thin coating of the sugar.
In a large mixing bowl, combine flour, pepper, cloves, fennel, nutmeg, star anise, cinnamon, baking soda, and salt.
Mix in 3 tablespoons of crystallized ginger; set aside.
Using an electric mixer with a large bowl, beat the butter, brown sugar, and granulated sugar together until light and creamy.
Beat in the eggs, one at a time, until well mixed.
Add the vanilla extract and the fresh ginger.
Slowly mix in 1/2 of the flour/spice mixture; then add milk and yogurt.
Add the remaining 1/2 of the flour/spice mixture; blend just until smooth. Pour batter into the prepared loaf pan.
Sprinkle the remaining 3 tablespoons of crystallized ginger over the top surface of the batter, pressing the ginger lightly into the batter.
Bake approximately 50 to 60 minutes or until a toothpick inserted into the center comes out clean.
Remove from oven, let pan cool on a wire rack for ten minutes, and then turn cake out out of the loaf pan onto the wire rack to cool completely. When completely cool, using a fine mesh strainer, dust the top lightly with confectioners (powdered) sugar.
If desired, garnish with whipped cream and seasonal berries.