Disadvantaged circumstances in youth tend to translate into poor health development. However, the fact that this is not always the case has been seen as indicative of differential resilience. The current study highlights factors outside the context of the family with the potential to counteract the long-term negative influences of social and material adversity in adolescence on general health status. This study was based on two waves of questionnaire data from the Northern Swedish Cohort. From the wave in 1981 (age 16), indicators of social and material conditions as well as factors related to school, peers, and spare time were derived. From the wave in 2008 (age 43), information about self-rated health was used. Ordinal logistic regression models (n = 908) showed that adversity in youth was associated with poorer self-rated health in midlife among men and women alike, net of health status at baseline. However, having an advantaged situation with regard to school, peers, or spare time appeared to protect against the detrimental influences of disadvantaged circumstances in the family context on subsequent health. This suggests that health-promoting interventions may benefit from focusing on contexts outside the family in their effort to strengthen processes of resilience among disadvantaged youths.