Line an ungreased 13"x9" baking pan with 1 layer of crackers; cutting to fit if neccessary; set aside.
In a large saucepan, melt butter over medium heat. Add cracker crumbs, brown sugar, milk and sugar. Bring to a boil and boil for 5 minutes, stirring constantly. Remove from heat, and pour half of the butter mixture evenly over crackers. Place another layer of crackers evenly over butter mixture; cutting to fit if necessary. Pour remaining butter mixture evenly over crackers. Top evenly with remaining crackers.
In a small saucepan combine peanut butter, chocolate morsels and butterscotch morsels. Melt over medium low heat, stirring constantly. Spread evenly over top of crackers. Cover and chill for at least 1 hour. Cut into bars and store in refrigerator.