Preheat an oven to 325°F.
Place the garlic cloves in a small baking dish. Drizzle with the olive oil and sprinkle with the rosemary. Cover with aluminum foil and bake until very soft, 35 to 40 minutes. Remove from the oven and, when cool enough to handle, squeeze the garlic from the skins into a small bowl. Mash with a fork. Strain the oil through a fine-mesh sieve held over the garlic and mix well.
Meanwhile, place the potatoes in a saucepan and add water to cover by 1 inch. Bring to a boil and cook, uncovered, until tender, about 20 minutes.
Just before the potatoes are ready, in a small saucepan over low heat, combine the butter and milk and heat until the butter is melted and the mixture is hot. Drain the potatoes, transfer to a warmed bowl, and mash well with a fork or potato masher.
Add the butter mixture to the potatoes and stir until smooth. Mix in the garlic and season with salt and pepper. Stir in the chives and serve immediately.