Mix together the milk, creme fraiche, eggs, sugar and vanilla until well blended, then stir in the raisins and orange peel. Pull off smallish hunks of brioche and stir into the milk mixture. Really mash the brioche around. Let it stand for about ten minutes, soaking up liquid, then smoosh it around some more with a wooden spoon. It can't be too smooshed up.
Spoon it into a loaf mold. I use the silicone pans because I find them so easy to use. If you DO use them, remember to let the bread pudding cool in the pan before trying to unmold it. Then bake at 200 degrees C for about 35 minutes, or until a knife comes out clean.