Thinly slice two limes; set aside. Zest remaining limes to measure about 1 tablespoon zest; juice limes to measure 1/2 cup juice.
Whisk cream cheese, condensed milk, lime juice and zest. Gently fold in all but 1/2 cup of the whipped topping until smooth.
Cut pound cake into 1-inch cubes.
To assemble trifle, place half the cake cubes into bottom of trifle bowl. Top with half of the sherbet; spread evenly. Top sherbet with half of the berries and half of the cream cheese mixture. Arrange lime slices in a circular pattern against inside of bowl to garnish. Repeat layers one time with remaining pound cake cubes, sherbet, berries and cream cheese mixture.
Decorate the top of the trifle with the remaining whipped topping, almonds and additional berries, if desired.
Note: Any flavor of sherbet can be used in this recipe.