Preheat oven to 350 degrees. Butter an 8x4-inch loaf pan and line it with parchment or waxed (greaseproof) paper.
Cream the butter and sugar until light and fluffy. Add the egg yolks, one at a time, beating well after each addition. Sift together the flour and baking powder and fold into the mixture alternately with the milk, lemon zest, caraway seeds, and vanilla. Beat the egg whites until stiff, then fold into the mixture.
Pour the mixture into the prepared pan. Bake for 1-1/4 hours, or until a skewer inserted in the middle of the cake comes out clean. Allow cake to cool in the pan on a rack for 5 minutes before turning out onto a wire rack. Serve warm or cold.