This paper proposes a deep learning and set based approach to the face recognition subject to aging. One possible solution is to apply a probabilistic model to represent a face simultaneously with its identity variable, which is stable through time, and its aging variable, which changes with time. Face recognition across age progression is remains one of the areas most challenging tasks now a days, as the aging process affects both the shape and texture of a face.