Generalist foundation models (GFMs) are renowned for their exceptional capability and flexibility in diverse tasks. In the field of medicine, while GFMs exhibit superior generalizability, specialist models excel in precision because of their domain-specific knowledge. Here we show a cooperative framework, Generalist-Specialist Collaboration (GSCo), that synergistically combines a powerful generalist model with lightweight specialists. In this framework, specialists provide expert guidance, such as diagnostic predictions and visually similar clinical cases, as contextual information to the generalist, which then makes a final diagnosis. We developed MedDr, an open-source GFM tailored for medicine, as well as a suite of lightweight specialist models crafted for specific downstream tasks. A comprehensive evaluation on 32 datasets across diverse medical modalities shows that MedDr outperforms state-of-the-art GFMs on downstream datasets. Furthermore, GSCo exceeds GFMs and specialists in medical image diagnosis and report generation. This approach offers an effective and computationally efficient paradigm for deploying GFMs in clinical settings, enhancing scalability and enabling precise analysis across a wide range of scenarios.