Audio classification guide for Python

The MediaPipe Audio Classifier task lets you perform classification on audio data. You can use this task to identify sound events from a set of trained categories. These instructions show you how to use the Audio Classifier with Python.

For more information about the capabilities, models, and configuration options of this task, see the Overview.

Code example

The example code for Audio Classifier provides a complete implementation of this task in Python for your reference. This code helps you test this task and get started on building your own audio classifier. You can view, run, and edit the Audio Classifier example code using just your web browser with Google Colab. You can view the source code for this example on GitHub.

If you are implementing the Audio Classifier for Raspberry Pi, refer to the Raspberry Pi example app.

Setup

This section describes key steps for setting up your development environment and code projects specifically to use Audio Classifier. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for Python.

Packages

The Audio Classifier task the mediapipe pip package. You can install the dependency with the following:

$ python -m pip install mediapipe

Imports

Import the following classes to access the Audio Classifier task functions:

import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import audio

Model

The MediaPipe Audio Classifier task requires a trained model that is compatible with this task. For more information on available trained models for Audio Classifier, see the task overview Models section.

Select and download a model, and then store it in a local directory. You can use the recommended Yamnet model.

model_path = '/absolute/path/to/lite-model_yamnet_classification_tflite_1.tflite'

Specify the path of the model within the Model Name parameter, as shown below:

base_options = BaseOptions(model_asset_path=model_path)

Create the task

Use the create_from_options function to create the task. The create_from_options function accepts configuration options including running mode, display names locale, max number of results, confidence threshold, category allow list, and deny list. For more information on configuration options, see Configuration Overview.

The Audio Classifier task supports audio clips and audio streams as input. You must specify the running mode corresponding to your input data type when creating the task. Choose the tab corresponding to your input data type to see how to create the task and run inference.

AudioClassifier = mp.tasks.audio.AudioClassifier
AudioClassifierOptions = mp.tasks.audio.AudioClassifierOptions
AudioRunningMode = mp.tasks.audio.RunningMode
BaseOptions = mp.tasks.BaseOptions

options = AudioClassifierOptions(
    base_options=BaseOptions(model_asset_path='/path/to/model.tflite'),
    max_results=5,
    running_mode=AudioRunningMode.AUDIO_CLIPS)

with AudioClassifier.create_from_options(options) as classifier:
  # The classifier is initialized. Use it here.
  # ...
    

AudioClassifier = mp.tasks.audio.AudioClassifier
AudioClassifierOptions = mp.tasks.audio.AudioClassifierOptions
AudioClassifierResult = mp.tasks.audio.AudioClassifierResult
AudioRunningMode = mp.tasks.audio.RunningMode
BaseOptions = mp.tasks.BaseOptions

def print_result(result: AudioClassifierResult, timestamp_ms: int):
    print(AudioClassifierResult result: {}’.format(result))

options = AudioClassifierOptions(
    base_options=BaseOptions(model_asset_path='/path/to/model.tflite'),
    running_mode=AudioRunningMode.AUDIO_STREAM,
    max_results=5,
    result_callback=print_result)

with AudioClassifier.create_from_options(options) as classifier:
  # The classifier is initialized. Use it here.
  # ...
  

For a complete example of creating a Audio Classifier for use with audio, see the code example.

Configuration options

This task has the following configuration options for Python applications:

Prepare data

Audio Classifier works with audio clips and audio streams. The task handles the data input preprocessing, including resampling, buffering, and framing.

Prepare your input as an audio file or a numpy array, then convert it to a MediaPipe AudioData object. You can use an external library such as SciPy to load your input audios as numpy arrays.

The following examples explain and show how to prepare data for processing for each of the available data types:

import numpy as np
from scipy.io import wavfile

AudioData = mp.tasks.components.containers.AudioData

sample_rate, buffer = wavfile.read('/path/to/audio.wav')
audio_data = AudioData.create_from_array(
    buffer.astype(float) / np.iinfo(np.int16).max, sample_rate)
    

import numpy as np

AudioData = mp.tasks.components.containers.AudioData

# Read microphone data as np arrays, then call

audio_data = AudioData.create_from_array(
    buffer.astype(float) / np.iinfo(np.int16).max, sample_rate)
    

Run the task

You call the classify function corresponding to your running mode to trigger inferences. The Audio Classifier API returns the possible categories for the audio events within the input audio buffer.

# Perform audio classification on the provided audio clip.
audio_classifier_result_list = classifier.classify(audio_data)
    

# Send live audio data to perform audio classification.
# Results are sent to the `result_callback` provided in the `AudioClassifierOptions`
classifier.classify_async(audio_data, timestamp_ms)
    

Note the following:

• When running in the audio stream mode, you must also provide the Audio Classifier task the timestamp of the input audio data.
• When running in the audio clips model, the Audio Classifier task will block the current thread until it finishes processing the input audio.

For a more complete example of running Audio Classifier with audio clips, see the code example.

Handle and display results

Upon running inference, the Audio Classifier task returns an AudioClassifierResult object which contains the list of possible categories for the audio events within the input audio.

The following shows an example of the output data from this task:

AudioClassifierResult:
  Timestamp in microseconds: 100
  ClassificationResult #0:
    Timestamp in microseconds: 100  
    Classifications #0 (single classification head):
      head index: 0
      category #0:
        category name: "Speech"
        score: 0.6
        index: 0
      category #1:
        category name: "Music"
        score: 0.2
        index: 1

The Audio Classifier example code demonstrates how to display the classification results returned from this task, see the code example for details.