In a saucepan, off the heat, combine the sweetened condensed milk, egg yolks, butter and brown sugar. Cook over medium heat, stirring until it thickens, about 5 minutes. Stir in the vanilla, coconut and pecans. Pour into a bowl and let cool. Refrigerate until the mixture is completely chilled.