note: the crust is optional. Line a square 8"x8 baking pan with foil or parchment paper, extending it up and over pan (this helps you get it out of the pan later). Butter sides and bottom of the pan, especially the corners. Set aside.
If you're making the crust: Preheat oven to 375. In a medium sized bowl, melt butter in them microwave. Stir in sugar with a fork, braking up any lumps. Add crushed graham crackers and toss to mix well. Spread evenly into the baking pan and bake for 4 to 5 minutes or until edge is lightly browned. Remove from the oven and let cool on a wire rack.
Stir white chocolate and sweetened condensed milk in a large heavy saucepan over low heat just until chocolate is melted and all the lumps are gone. Remove from heat and stir in zest and juice. Spread mixture evenly into the prepared pan.
Cover and chill for two hours or overnight. Lift the fudge from pan using edges of foil. Peel off foil and use a heavy non-serrated knife to cut the fudge into one-inch pieces. (Tip: Keep a damp kitchen towel handy to clean off the knife after each cut. This will keep give your fudge a cleaner appearance.).
Store in an airtight container at room temperature for up to one week.
* Zesting the limes - Be sure to wash the limes in hot, soapy water before zesting them. I like to scrub mine with a wet terry cloth towel to get all the wax off. Be sure they are totally dry before zesting. Also, Key limes are really small and their zest is thin. Go very slowly when zesting each lime as you'll be at the pith (white part) after just one or two passes.