[ICML25]MATS: An Audio Language Model under Text-only Supervision

Wen Wang¹, Ruibing Hou^{1, 📧}, Hong Chang^1,2, Shiguang Shan^1,2, Xilin Chen^1,2

¹ Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China ² University of Chinese Academy of Sciences, China

^📧 corresponding author, houruibing@ict.ac.cn

 

 

News

• Feb. 19th, 2025: We released our paper on Arxiv.
• May. 1, 2025: MATS was accepted by ICML2025
• We released the weight checkpoint on Huggingface.

Table of Contents

• News
• Table of Contents
• Abstract
• Install
• Evaluation
• Acknowledgement
• Citation

Abstract

Large audio-language models (LALMs), built upon powerful Large Language Models (LLMs), have exhibited remarkable audio comprehension and reasoning capabilities. However, the training of LALMs demands a large corpus of audio-language pairs, which requires substantial costs in both data collection and training resources. In this paper, we propose MATS, an audio-language multimodal LLM designed to handle Multiple Audio task using solely Text-only Supervision. By leveraging pre-trained audio-language alignment models such as CLAP, we develop a text-only training strategy that projects the shared audio-language latent space into LLM latent space, endowing the LLM with audio comprehension capabilities without relying on audio data during training. To further bridge the modality gap between audio and language embeddings within CLAP, we propose the Strongly-related noisy text with audio (Santa) mechanism. Santa maps audio embeddings into CLAP language embedding space while preserving essential information from the audio input. Extensive experiments demonstrate that MATS, despite being trained exclusively on text data, achieves competitive performance compared to recent LALMs trained on large-scale audio-language pairs.

The architecture of proposed MATS.

Install

conda create -n mats python==3.9
conda activate mats
pip install -r requirements.txt

And you can download the CLAP and LLaMA checkpoint.

Evaluation

1. Please modify the datasets in the file configs/audiofree_config_open.yaml according to the given example. Note that only the following tasks are supported: [caption, cla_label, music_caption, QA].

    test_file: "path"
    task: "caption"
    question: None

    test_file: "path"
    task: "QA"
    question: "What xxx"

2. run

   python test.py

Acknowledgement

• SALMONN: MATS-LLaMA is built upon the SALMONN codebases.

Citation

If you find MATS is useful in your research or applications, please consider giving us a star 🌟 and citing it by the following BibTeX entry.

 @article{wang2025matsaudiolanguagemodel,
  title={MATS: An Audio Language Model under Text-only Supervision},
  author={Wen Wang and Ruibing Hou and Hong Chang and Shiguang Shan and Xilin Chen},
  journal={arXiv preprint arXiv:2502.13433},
  year={2025}
}