Warm the oil in a saucepan over medium heat with the garlic, peppercorns, rosemary, salt and red pepper flakes. Remove from heat and let cool to room temperature.
While the oil is cooling, cut the mozzarella into 1-inch cubes.
Remove the rosemary sprigs and peppercorns from the oil and pour over the mozzarella. Let stand at room temperature for several hours or refrigerate for up to 4 days. (Bring to room temperature before serving.).
Cut the tomatoes into fork-sized chunks. Drain excess oil from the mozzarella and fold in the tomato chunks.