Tensorflow.js tf.data.microphone() Function

Tensorflow.js is an open-source library developed by Google for running machine learning models and deep learning neural networks in the browser or node environment.

The tf.data.microphone() function is used to produce an iterator that creates frequency-domain spectrogram Tensors from microphone audio stream with browser’s native FFT. 

Note: 

• This code is effective only when the device is having a microphone. While running this API, it will ask to permit to open the microphone.
• This API only works through the browser environment.

Syntax:

tf.data.microphone (microphoneConfig)

Parameters: This function accepts a parameter which is illustrated below:

• microphoneConfig: A MicrophoneConfig object contains configurations of reading audio data from the microphone. It is an optional parameter.

This object contains some configuration which is specified below:

1. sampleRateHz: Its range lies between 44100 and 48000.
2. fftSize: It is a number value must be a power of 2 between 2 to 4 and 2 to 14.
3. columnTruncateLength: It is a number value.
4. numFramesPerSpectrogram: It is a number value.
5. audioTrackConstraints: It is MediaTrackConstraints.
6. smoothingTimeConstant: It is a number value.
7. includeSpectrogram: It is a boolean value.
8. includeWaveform: It is a boolean value.

Return Value: It returns a MicrophoneIterator.

Example 1: After running the below code, it will ask for permission to start the microphone. After giving permission below code will return and will give output.

Javascript

// Importing the tensorflow.js library
import * as tf from "@tensorflow/tfjs"

// Calling the .data.microphone() function
// with its parameters
const mic = await tf.data.microphone({
   fftSize: 1024,
   columnTruncateLength: 32,
   numFramesPerSpectrogram: 10,
   sampleRateHz:43000,
   includeSpectrogram: true,
   includeWaveform: false
});

// Capturing the data recorded by microphone
const audioData = await mic.capture();
const spectrogramTensor = audioData.spectrogram;

// Printing the data like sampling rate
// expected and actual
spectrogramTensor.print();
const waveformTensor = audioData.waveform;
waveformTensor.print();

// Stopping the microphone
mic.stop();

Output: It gives an error because here the sampling rate expected is 43000 but the actual recorded value is 48000.

An error occurred
Mismatch in sampling rate: Expected: 43000; Actual: 48000

Example 2: After running the below code, it will ask for permission to start the microphone. After giving permission below code will return and will give output.

Javascript

// Importing the tensorflow.js library
import * as tf from "@tensorflow/tfjs"

// Initializing the configurations of
// reading audio data from microphone
const x = {
   fftSize: 1024,
   columnTruncateLength: 32,
   numFramesPerSpectrogram: 10,
   sampleRateHz:48000,
   includeSpectrogram: true,
   includeWaveform: false
};

// Calling the .data.microphone() function
// with the parameter specified above
const mic = await tf.data.microphone(x);

// Capturing the data recorded by microphone
const audioData = await mic.capture();
const spectrogramTensor = audioData.spectrogram;

// Creating an iterator that generate frequency-domain
// spectrogram Tensors from the microphone
spectrogramTensor.print();
const waveformTensor = audioData.waveform;

// Stopping the microphone
mic.stop();

Output:

  Tensor
    [[[0        ],
      [0        ],
      [0        ],
      ...,
      [0        ],
      [0        ],
      [0        ]],

     [[0        ],
      [0        ],
      [0        ],
      ...,
      [0        ],
      [0        ],
      [0        ]],

     [[0        ],
      [0        ],
      [0        ],
      ...,
      [0        ],
      [0        ],
      [0        ]],

     [[0        ],
      [0        ],
      [0        ],
      ...,
      [0        ],
      [0        ],
      [0        ]],

     [[0        ],
      [0        ],
      [0        ],
      ...,
      [0        ],
      [0        ],
      [0        ]],

     [[0        ],
      [0        ],
      [0        ],
      ...,
      [0        ],
      [0        ],
      [0        ]],

     [[0        ],
      [0        ],
      [0        ],
      ...,
      [0        ],
      [0        ],
      [0        ]],

     [[0        ],
      [0        ],
      [0        ],
      ...,
      [0        ],
      [0        ],
      [0        ]],

     [[0        ],
      [0        ],
      [0        ],
      ...,
      [0        ],
      [0        ],
      [0        ]],

     [[-Infinity],
      [-Infinity],
      [-Infinity],
      ...,
      [-Infinity],
      [-Infinity],
      [-Infinity]]]

Reference:https://js.tensorflow.org/api/latest/#data.microphone