Urinary stone disease is a highly prevalent condition affecting approximately 10% of the population, and has increased in incidence significantly over the past 20 years. Along with this, the rate of stone disease among women and children is also on the rise. The management of stone disease in specific populations, such as in children and during pregnancy can present unique challenges to the urologist. In both populations, a multi-disciplinary approach is strongly recommended given the complexities of the patients. Prompt and accurate diagnosis requires a high degree of suspicion and judicious use of diagnostic imaging given the higher risks of radiation exposure. In general, management proceeds from conservative to more invasive approaches and must be individualized to the patient with careful consideration of the potential adverse effects. However, innovations in endourologic equipment and techniques have allowed for the wider application of surgical stone treatment in these patients, and significant advancement in the field. This review covers the history and current advances in the diagnosis and management of stone disease in pregnant and pediatric populations. It is paramount for the urologist to understand the complexities of properly managing stones in these patients in order to maximize treatment efficacy, while minimizing complications and morbidity.