Grease a 9-inch square pan. In a saucepan, melt the butter and add the brown sugar. Cook over low heat for 2 minutes, stirring constantly. Add the milk and continue cooking, stirring until the mixture boils. Remove from the heat and allow to cool.
Gradually add the confectioners' sugar until the mixture is of fudge consistency. Add the nuts. Spread in the prepared pan and cool.