Medical imaging has been a common examination in daily clinical routine for screening and diagnosis of a variety of diseases. Although hospitals have accumulated a large number of image exams and associated reports, it is not yet challenging to effectively use them to build high precision computer-aided diagnosis systems. In this chapter, we first present an overview of cutting-edge techniques for mining existing images and free-text report data for the machine learning purpose and then demonstrate two case studies in radiological and pathological images: (1) we present a method to text-mine disease image labels (where each image can have multilabels) from the associated radiological reports using natural language processing, and (2) we introduce the semantic knowledge of medical images from their diagnostic reports to provide an inspirational network training and an interpretable prediction mechanism with our proposed novel multimodal neural network.