Heat the oven to 350°F and arrange a rack in the middle. Coat a 9-by-5-by-3-inch loaf pan with butter and flour; tap out excess.
Combine flour, salt, baking soda, baking powder, cinnamon, and nutmeg in a medium bowl and whisk to aerate and break up any lumps; set aside.
In the bowl of a stand mixer fitted with a paddle attachment, mix sweet potato flesh, granulated sugar, and brown sugar on medium speed until well combined, about 1 minute. Add butter and mix on low speed until smooth. Add eggs one at a time, mixing until fully incorporated, then mix in vanilla.
Stop the mixer and scrape down the sides of the bowl with a rubber spatula. On low speed add 1/2 of the flour mixture, then 1/4 cup of the milk. Repeat with remaining flour mixture and milk, mixing until just combined, about 1 minute. Remove the bowl from the mixer and fold in nuts.
Pour the batter into the prepared pan and bake until a cake tester inserted in the center comes out clean, about 55 to 60 minutes. Let cool in the pan for 15 minutes, then turn out on a wire rack to cool completely. The bread will last covered for up to 5 days.