Trim crusts from the bread and cut it into roughly 1/3 inch slices if not already sliced. Stack and cut them in half diagonally to make triangles.
Line a 6 cup mold or bowl with plastic wrap, fitting the wrap tightly into the corners. Make sure you have a plate small enough to fit just inside the rim of the bowl or, if not, cut a stiff cardboard circle to fit and wrap in aluminum foil.
Piece together the bread slices in a single layer so they completely line the bowl. There should be no gaps. Whereever the bread overlaps, press the edges together tightly. Trim bread if it reaches over the top of the bowl.
Cut a shallow "X" in the base of each peach and dunk in rapidly boiling water until the peel starts to come loose - 20 seconds to 1 minute - depending on ripeness of peach. Remove and rinse under cold water. If the peel doesn't come off easily, return to boiling water briefly. Pit and cut into chunks.
Combine the fruit and sugar in a saucepan and warm over medium heat. Cook just until the fruit begins to soften and relase its juice, about 3 or 4 minutes. Stir in brandy.
Spoon the fruit into the bowl; there should be enough that it mounds a little above the lip. Place a sheet of plastic wrap loosely across the top, then place the plate or cardboard circle on top. Press gently and put a heavy can on top. Refrigerate at least 12 hours.
When ready to serve, remove plate and top layer of plastic wrap. Place a serving plate on top of the bowl and invert. If the pudding resists unmolding, hold the plastic wrap and jiggle gently until it unmolds cleanly. Remove the plastic wrap lining and serve with yogurt or whipped cream.