Cook potatoes in large pot of boiling salted water until tender, about 40 minutes. Drain. Return potatoes to pot; mash.
Combine half and half and rosemary in small saucepan; bring to simmer. Remove from heat; stir into mashed potatoes. Add butter; stir to melt. Season to taste with salt and pepper. (Can be made 2 hours ahead. Cover; let stand at room temperature. Before serving, stir over low heat until warm.).