Matrix-Assisted Laser Desorption Ionization mass spectrometry imaging (MALDI-MSI) is a mass spectrometry ionization technique that can be used to directly analyze tissues and has led the way in the development of biological and clinical applications for imaging mass spectrometry. One of its advantages is measuring the distribution of a large number of analytes at one time without destroying the sample, making it a useful method in tissue-based studies. However, analysis of the MALDI-MSI images from tissue microarrays (TMAs) remains less studied. While several automated systems have been developed for tissue classification (e.g., cancer vs non-cancer), they process the MALDI data at the measuring point level, which ignores spatial relationships among individual points within the tissue sample. In this work, we propose mNet, a new deep learning framework to analyze MALDI-MSI data of TMAs at the tissue-needle-core level to ensure that the samples maintain their original spatial context. In addition, we introduced data augmentation techniques to increase data size which is often limited in biomedical data. We applied our framework to analyzing TMAs from breast and lung cancer. We found that our framework outperforms conventional machine learning methods in the challenging race detection task. The results highlight the potential of deep learning to assist pathologists in analyzing tissue specimens in a label-free, high-throughput manner.