wav2vec2

mindnlp.transformers.models.wav2vec2.configuration_wav2vec2

Wav2Vec2 model configuration

mindnlp.transformers.models.wav2vec2.configuration_wav2vec2.Wav2Vec2Config

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [Wav2Vec2Model]. It is used to instantiate an Wav2Vec2 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Wav2Vec2 facebook/wav2vec2-base-960h architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
vocab_size	Vocabulary size of the Wav2Vec2 model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling [Wav2Vec2Model] or [TFWav2Vec2Model]. Vocabulary size of the model. Defines the different tokens that can be represented by the inputs_ids passed to the forward method of [Wav2Vec2Model]. TYPE: `int`, *optional*, defaults to 32 DEFAULT: 32
hidden_size	Dimensionality of the encoder layers and the pooler layer. TYPE: `int`, *optional*, defaults to 768 DEFAULT: 768
num_hidden_layers	Number of hidden layers in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
num_attention_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
intermediate_size	Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 3072 DEFAULT: 3072
hidden_act	The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" are supported. TYPE: `str` or `function`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
hidden_dropout	The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
activation_dropout	The dropout ratio for activations inside the fully connected layer. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
attention_dropout	The dropout ratio for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
final_dropout	The dropout probability for the final projection layer of [Wav2Vec2ForCTC]. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
layerdrop	The LayerDrop probability. See the LayerDrop paper for more details. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
initializer_range	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 0.02
layer_norm_eps	The epsilon used by the layer normalization layers. TYPE: `float`, *optional*, defaults to 1e-12 DEFAULT: 1e-05
feat_extract_norm	The norm to be applied to 1D convolutional layers in feature encoder. One of "group" for group normalization of only the first 1D convolutional layer or "layer" for layer normalization of all 1D convolutional layers. TYPE: `str`, *optional*, defaults to `"group"` DEFAULT: 'group'
feat_proj_dropout	The dropout probability for output of the feature encoder. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
feat_extract_activation	The non-linear activation function (function or string) in the 1D convolutional layers of the feature extractor. If string, "gelu", "relu", "selu" and "gelu_new" are supported. TYPE: `str, `optional`, defaults to `"gelu"` DEFAULT: 'gelu'
feat_quantizer_dropout	The dropout probability for quantized feature encoder states. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
conv_dim	A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the feature encoder. The length of conv_dim defines the number of 1D convolutional layers. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(512, 512, 512, 512, 512, 512, 512)` DEFAULT: (512, 512, 512, 512, 512, 512, 512)
conv_stride	A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length of conv_stride defines the number of convolutional layers and has to match the length of conv_dim. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(5, 2, 2, 2, 2, 2, 2)` DEFAULT: (5, 2, 2, 2, 2, 2, 2)
conv_kernel	A tuple of integers defining the kernel size of each 1D convolutional layer in the feature encoder. The length of conv_kernel defines the number of convolutional layers and has to match the length of conv_dim. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(10, 3, 3, 3, 3, 3, 3)` DEFAULT: (10, 3, 3, 3, 3, 2, 2)
conv_bias	Whether the 1D convolutional layers have a bias. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
num_conv_pos_embeddings	Number of convolutional positional embeddings. Defines the kernel size of 1D convolutional positional embeddings layer. TYPE: `int`, *optional*, defaults to 128 DEFAULT: 128
num_conv_pos_embedding_groups	Number of groups of 1D convolutional positional embeddings layer. TYPE: `int`, *optional*, defaults to 16 DEFAULT: 16
do_stable_layer_norm	Whether to apply stable layer norm architecture of the Transformer encoder. do_stable_layer_norm is True corresponds to applying layer norm before the attention layer, whereas do_stable_layer_norm is False corresponds to applying layer norm after the attention layer. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
apply_spec_augment	Whether to apply SpecAugment data augmentation to the outputs of the feature encoder. For reference see SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
mask_time_prob	Percentage (between 0 and 1) of all feature vectors along the time axis which will be masked. The masking procecure generates ''mask_time_prob*len(time_axis)/mask_time_length'' independent masks over the axis. If reasoning from the propability of each feature vector to be chosen as the start of the vector span to be masked, mask_time_prob should be prob_vector_start*mask_time_length. Note that overlap may decrease the actual percentage of masked vectors. This is only relevant if apply_spec_augment is True. TYPE: `float`, *optional*, defaults to 0.05 DEFAULT: 0.05
mask_time_length	Length of vector span along the time axis. TYPE: `int`, *optional*, defaults to 10 DEFAULT: 10
mask_time_min_masks	The minimum number of masks of length mask_feature_length generated along the time axis, each time step, irrespectively of mask_feature_prob. Only relevant if ''mask_time_prob*len(time_axis)/mask_time_length < mask_time_min_masks'' TYPE: `int`, *optional*, defaults to 2), DEFAULT: 2
mask_feature_prob	Percentage (between 0 and 1) of all feature vectors along the feature axis which will be masked. The masking procecure generates ''mask_feature_prob*len(feature_axis)/mask_time_length'' independent masks over the axis. If reasoning from the propability of each feature vector to be chosen as the start of the vector span to be masked, mask_feature_prob should be prob_vector_start*mask_feature_length. Note that overlap may decrease the actual percentage of masked vectors. This is only relevant if apply_spec_augment is True. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
mask_feature_length	Length of vector span along the feature axis. TYPE: `int`, *optional*, defaults to 10 DEFAULT: 10
mask_feature_min_masks	The minimum number of masks of length mask_feature_length generated along the feature axis, each time step, irrespectively of mask_feature_prob. Only relevant if ''mask_feature_prob*len(feature_axis)/mask_feature_length < mask_feature_min_masks'' TYPE: `int`, *optional*, defaults to 0), DEFAULT: 0
num_codevectors_per_group	Number of entries in each quantization codebook (group). TYPE: `int`, *optional*, defaults to 320 DEFAULT: 320
num_codevector_groups	Number of codevector groups for product codevector quantization. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
contrastive_logits_temperature	The temperature kappa in the contrastive loss. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
feat_quantizer_dropout	The dropout probability for the output of the feature encoder that's used by the quantizer. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
num_negatives	Number of negative samples for the contrastive loss. TYPE: `int`, *optional*, defaults to 100 DEFAULT: 100
codevector_dim	Dimensionality of the quantized feature vectors. TYPE: `int`, *optional*, defaults to 256 DEFAULT: 256
proj_codevector_dim	Dimensionality of the final projection of both the quantized and the transformer features. TYPE: `int`, *optional*, defaults to 256 DEFAULT: 256
diversity_loss_weight	The weight of the codebook diversity loss component. TYPE: `int`, *optional*, defaults to 0.1 DEFAULT: 0.1
ctc_loss_reduction	Specifies the reduction to apply to the output of torch.nn.CTCLoss. Only relevant when training an instance of [Wav2Vec2ForCTC]. TYPE: `str`, *optional*, defaults to `"sum"` DEFAULT: 'sum'
ctc_zero_infinity	Whether to zero infinite losses and the associated gradients of torch.nn.CTCLoss. Infinite losses mainly occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance of [Wav2Vec2ForCTC]. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
use_weighted_layer_sum	Whether to use a weighted average of layer outputs with learned weights. Only relevant when using an instance of [Wav2Vec2ForSequenceClassification]. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
classifier_proj_size	Dimensionality of the projection before token mean-pooling for classification. TYPE: `int`, *optional*, defaults to 256 DEFAULT: 256
tdnn_dim	A tuple of integers defining the number of output channels of each 1D convolutional layer in the TDNN module of the XVector model. The length of tdnn_dim defines the number of TDNN layers. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(512, 512, 512, 512, 1500)` DEFAULT: (512, 512, 512, 512, 1500)
tdnn_kernel	A tuple of integers defining the kernel size of each 1D convolutional layer in the TDNN module of the XVector model. The length of tdnn_kernel has to match the length of tdnn_dim. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(5, 3, 3, 1, 1)` DEFAULT: (5, 3, 3, 1, 1)
tdnn_dilation	A tuple of integers defining the dilation factor of each 1D convolutional layer in TDNN module of the XVector model. The length of tdnn_dilation has to match the length of tdnn_dim. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(1, 2, 3, 1, 1)` DEFAULT: (1, 2, 3, 1, 1)
xvector_output_dim	Dimensionality of the XVector embedding vectors. TYPE: `int`, *optional*, defaults to 512 DEFAULT: 512
add_adapter	Whether a convolutional network should be stacked on top of the Wav2Vec2 Encoder. Can be very useful for warm-starting Wav2Vec2 for SpeechEncoderDecoder models. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
adapter_kernel_size	Kernel size of the convolutional layers in the adapter network. Only relevant if add_adapter is True. TYPE: `int`, *optional*, defaults to 3 DEFAULT: 3
adapter_stride	Stride of the convolutional layers in the adapter network. Only relevant if add_adapter is True. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
num_adapter_layers	Number of convolutional layers that should be used in the adapter network. Only relevant if add_adapter is True. TYPE: `int`, *optional*, defaults to 3 DEFAULT: 3
adapter_attn_dim	Dimension of the attention adapter weights to be used in each attention block. An example of a model using attention adapters is facebook/mms-1b-all. TYPE: `int`, *optional* DEFAULT: None
output_hidden_size	Dimensionality of the encoder output layer. If not defined, this defaults to hidden-size. Only relevant if add_adapter is True. TYPE: `int`, *optional* DEFAULT: None

Example

>>> from transformers import Wav2Vec2Config, Wav2Vec2Model
...
>>> # Initializing a Wav2Vec2 facebook/wav2vec2-base-960h style configuration
>>> configuration = Wav2Vec2Config()
...
>>> # Initializing a model (with random weights) from the facebook/wav2vec2-base-960h style configuration
>>> model = Wav2Vec2Model(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/wav2vec2/configuration_wav2vec2.py

class Wav2Vec2Config(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`Wav2Vec2Model`]. It is used to instantiate an
    Wav2Vec2 model according to the specified arguments, defining the model architecture. Instantiating a configuration
    with the defaults will yield a similar configuration to that of the Wav2Vec2
    [facebook/wav2vec2-base-960h](https://hf-mirror.com/facebook/wav2vec2-base-960h) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.


    Args:
        vocab_size (`int`, *optional*, defaults to 32):
            Vocabulary size of the Wav2Vec2 model. Defines the number of different tokens that can be represented by
            the `inputs_ids` passed when calling [`Wav2Vec2Model`] or [`TFWav2Vec2Model`]. Vocabulary size of the
            model. Defines the different tokens that can be represented by the *inputs_ids* passed to the forward
            method of [`Wav2Vec2Model`].
        hidden_size (`int`, *optional*, defaults to 768):
            Dimensionality of the encoder layers and the pooler layer.
        num_hidden_layers (`int`, *optional*, defaults to 12):
            Number of hidden layers in the Transformer encoder.
        num_attention_heads (`int`, *optional*, defaults to 12):
            Number of attention heads for each attention layer in the Transformer encoder.
        intermediate_size (`int`, *optional*, defaults to 3072):
            Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
        hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
            `"relu"`, `"selu"` and `"gelu_new"` are supported.
        hidden_dropout (`float`, *optional*, defaults to 0.1):
            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
        activation_dropout (`float`, *optional*, defaults to 0.1):
            The dropout ratio for activations inside the fully connected layer.
        attention_dropout (`float`, *optional*, defaults to 0.1):
            The dropout ratio for the attention probabilities.
        final_dropout (`float`, *optional*, defaults to 0.1):
            The dropout probability for the final projection layer of [`Wav2Vec2ForCTC`].
        layerdrop (`float`, *optional*, defaults to 0.1):
            The LayerDrop probability. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556) for more
            details.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        layer_norm_eps (`float`, *optional*, defaults to 1e-12):
            The epsilon used by the layer normalization layers.
        feat_extract_norm (`str`, *optional*, defaults to `"group"`):
            The norm to be applied to 1D convolutional layers in feature encoder. One of `"group"` for group
            normalization of only the first 1D convolutional layer or `"layer"` for layer normalization of all 1D
            convolutional layers.
        feat_proj_dropout (`float`, *optional*, defaults to 0.0):
            The dropout probability for output of the feature encoder.
        feat_extract_activation (`str, `optional`, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the 1D convolutional layers of the feature
            extractor. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported.
        feat_quantizer_dropout (`float`, *optional*, defaults to 0.0):
            The dropout probability for quantized feature encoder states.
        conv_dim (`Tuple[int]` or `List[int]`, *optional*, defaults to `(512, 512, 512, 512, 512, 512, 512)`):
            A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the
            feature encoder. The length of *conv_dim* defines the number of 1D convolutional layers.
        conv_stride (`Tuple[int]` or `List[int]`, *optional*, defaults to `(5, 2, 2, 2, 2, 2, 2)`):
            A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length
            of *conv_stride* defines the number of convolutional layers and has to match the length of *conv_dim*.
        conv_kernel (`Tuple[int]` or `List[int]`, *optional*, defaults to `(10, 3, 3, 3, 3, 3, 3)`):
            A tuple of integers defining the kernel size of each 1D convolutional layer in the feature encoder. The
            length of *conv_kernel* defines the number of convolutional layers and has to match the length of
            *conv_dim*.
        conv_bias (`bool`, *optional*, defaults to `False`):
            Whether the 1D convolutional layers have a bias.
        num_conv_pos_embeddings (`int`, *optional*, defaults to 128):
            Number of convolutional positional embeddings. Defines the kernel size of 1D convolutional positional
            embeddings layer.
        num_conv_pos_embedding_groups (`int`, *optional*, defaults to 16):
            Number of groups of 1D convolutional positional embeddings layer.
        do_stable_layer_norm (`bool`, *optional*, defaults to `False`):
            Whether to apply *stable* layer norm architecture of the Transformer encoder. `do_stable_layer_norm is
            True` corresponds to applying layer norm before the attention layer, whereas `do_stable_layer_norm is
            False` corresponds to applying layer norm after the attention layer.
        apply_spec_augment (`bool`, *optional*, defaults to `True`):
            Whether to apply *SpecAugment* data augmentation to the outputs of the feature encoder. For reference see
            [SpecAugment: A Simple Data Augmentation Method for Automatic Speech
            Recognition](https://arxiv.org/abs/1904.08779).
        mask_time_prob (`float`, *optional*, defaults to 0.05):
            Percentage (between 0 and 1) of all feature vectors along the time axis which will be masked. The masking
            procecure generates ''mask_time_prob*len(time_axis)/mask_time_length'' independent masks over the axis. If
            reasoning from the propability of each feature vector to be chosen as the start of the vector span to be
            masked, *mask_time_prob* should be `prob_vector_start*mask_time_length`. Note that overlap may decrease the
            actual percentage of masked vectors. This is only relevant if `apply_spec_augment is True`.
        mask_time_length (`int`, *optional*, defaults to 10):
            Length of vector span along the time axis.
        mask_time_min_masks (`int`, *optional*, defaults to 2),:
            The minimum number of masks of length `mask_feature_length` generated along the time axis, each time step,
            irrespectively of `mask_feature_prob`. Only relevant if ''mask_time_prob*len(time_axis)/mask_time_length <
            mask_time_min_masks''
        mask_feature_prob (`float`, *optional*, defaults to 0.0):
            Percentage (between 0 and 1) of all feature vectors along the feature axis which will be masked. The
            masking procecure generates ''mask_feature_prob*len(feature_axis)/mask_time_length'' independent masks over
            the axis. If reasoning from the propability of each feature vector to be chosen as the start of the vector
            span to be masked, *mask_feature_prob* should be `prob_vector_start*mask_feature_length`. Note that overlap
            may decrease the actual percentage of masked vectors. This is only relevant if `apply_spec_augment is
            True`.
        mask_feature_length (`int`, *optional*, defaults to 10):
            Length of vector span along the feature axis.
        mask_feature_min_masks (`int`, *optional*, defaults to 0),:
            The minimum number of masks of length `mask_feature_length` generated along the feature axis, each time
            step, irrespectively of `mask_feature_prob`. Only relevant if
            ''mask_feature_prob*len(feature_axis)/mask_feature_length < mask_feature_min_masks''
        num_codevectors_per_group (`int`, *optional*, defaults to 320):
            Number of entries in each quantization codebook (group).
        num_codevector_groups (`int`, *optional*, defaults to 2):
            Number of codevector groups for product codevector quantization.
        contrastive_logits_temperature (`float`, *optional*, defaults to 0.1):
            The temperature *kappa* in the contrastive loss.
        feat_quantizer_dropout (`float`, *optional*, defaults to 0.0):
            The dropout probability for the output of the feature encoder that's used by the quantizer.
        num_negatives (`int`, *optional*, defaults to 100):
            Number of negative samples for the contrastive loss.
        codevector_dim (`int`, *optional*, defaults to 256):
            Dimensionality of the quantized feature vectors.
        proj_codevector_dim (`int`, *optional*, defaults to 256):
            Dimensionality of the final projection of both the quantized and the transformer features.
        diversity_loss_weight (`int`, *optional*, defaults to 0.1):
            The weight of the codebook diversity loss component.
        ctc_loss_reduction (`str`, *optional*, defaults to `"sum"`):
            Specifies the reduction to apply to the output of `torch.nn.CTCLoss`. Only relevant when training an
            instance of [`Wav2Vec2ForCTC`].
        ctc_zero_infinity (`bool`, *optional*, defaults to `False`):
            Whether to zero infinite losses and the associated gradients of `torch.nn.CTCLoss`. Infinite losses mainly
            occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance
            of [`Wav2Vec2ForCTC`].
        use_weighted_layer_sum (`bool`, *optional*, defaults to `False`):
            Whether to use a weighted average of layer outputs with learned weights. Only relevant when using an
            instance of [`Wav2Vec2ForSequenceClassification`].
        classifier_proj_size (`int`, *optional*, defaults to 256):
            Dimensionality of the projection before token mean-pooling for classification.
        tdnn_dim (`Tuple[int]` or `List[int]`, *optional*, defaults to `(512, 512, 512, 512, 1500)`):
            A tuple of integers defining the number of output channels of each 1D convolutional layer in the *TDNN*
            module of the *XVector* model. The length of *tdnn_dim* defines the number of *TDNN* layers.
        tdnn_kernel (`Tuple[int]` or `List[int]`, *optional*, defaults to `(5, 3, 3, 1, 1)`):
            A tuple of integers defining the kernel size of each 1D convolutional layer in the *TDNN* module of the
            *XVector* model. The length of *tdnn_kernel* has to match the length of *tdnn_dim*.
        tdnn_dilation (`Tuple[int]` or `List[int]`, *optional*, defaults to `(1, 2, 3, 1, 1)`):
            A tuple of integers defining the dilation factor of each 1D convolutional layer in *TDNN* module of the
            *XVector* model. The length of *tdnn_dilation* has to match the length of *tdnn_dim*.
        xvector_output_dim (`int`, *optional*, defaults to 512):
            Dimensionality of the *XVector* embedding vectors.
        add_adapter (`bool`, *optional*, defaults to `False`):
            Whether a convolutional network should be stacked on top of the Wav2Vec2 Encoder. Can be very useful for
            warm-starting Wav2Vec2 for SpeechEncoderDecoder models.
        adapter_kernel_size (`int`, *optional*, defaults to 3):
            Kernel size of the convolutional layers in the adapter network. Only relevant if `add_adapter is True`.
        adapter_stride (`int`, *optional*, defaults to 2):
            Stride of the convolutional layers in the adapter network. Only relevant if `add_adapter is True`.
        num_adapter_layers (`int`, *optional*, defaults to 3):
            Number of convolutional layers that should be used in the adapter network. Only relevant if `add_adapter is
            True`.
        adapter_attn_dim (`int`, *optional*):
            Dimension of the attention adapter weights to be used in each attention block. An example of a model using
            attention adapters is [facebook/mms-1b-all](https://hf-mirror.com/facebook/mms-1b-all).
        output_hidden_size (`int`, *optional*):
            Dimensionality of the encoder output layer. If not defined, this defaults to *hidden-size*. Only relevant
            if `add_adapter is True`.

    Example:
        ```python
        >>> from transformers import Wav2Vec2Config, Wav2Vec2Model
        ...
        >>> # Initializing a Wav2Vec2 facebook/wav2vec2-base-960h style configuration
        >>> configuration = Wav2Vec2Config()
        ...
        >>> # Initializing a model (with random weights) from the facebook/wav2vec2-base-960h style configuration
        >>> model = Wav2Vec2Model(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """
    model_type = "wav2vec2"

    def __init__(
        self,
        vocab_size=32,
        hidden_size=768,
        num_hidden_layers=12,
        num_attention_heads=12,
        intermediate_size=3072,
        hidden_act="gelu",
        hidden_dropout=0.1,
        activation_dropout=0.1,
        attention_dropout=0.1,
        feat_proj_dropout=0.0,
        feat_quantizer_dropout=0.0,
        final_dropout=0.1,
        layerdrop=0.1,
        initializer_range=0.02,
        layer_norm_eps=1e-5,
        feat_extract_norm="group",
        feat_extract_activation="gelu",
        conv_dim=(512, 512, 512, 512, 512, 512, 512),
        conv_stride=(5, 2, 2, 2, 2, 2, 2),
        conv_kernel=(10, 3, 3, 3, 3, 2, 2),
        conv_bias=False,
        num_conv_pos_embeddings=128,
        num_conv_pos_embedding_groups=16,
        do_stable_layer_norm=False,
        apply_spec_augment=True,
        mask_time_prob=0.05,
        mask_time_length=10,
        mask_time_min_masks=2,
        mask_feature_prob=0.0,
        mask_feature_length=10,
        mask_feature_min_masks=0,
        num_codevectors_per_group=320,
        num_codevector_groups=2,
        contrastive_logits_temperature=0.1,
        num_negatives=100,
        codevector_dim=256,
        proj_codevector_dim=256,
        diversity_loss_weight=0.1,
        ctc_loss_reduction="sum",
        ctc_zero_infinity=False,
        use_weighted_layer_sum=False,
        classifier_proj_size=256,
        tdnn_dim=(512, 512, 512, 512, 1500),
        tdnn_kernel=(5, 3, 3, 1, 1),
        tdnn_dilation=(1, 2, 3, 1, 1),
        xvector_output_dim=512,
        pad_token_id=0,
        bos_token_id=1,
        eos_token_id=2,
        add_adapter=False,
        adapter_kernel_size=3,
        adapter_stride=2,
        num_adapter_layers=3,
        output_hidden_size=None,
        adapter_attn_dim=None,
        **kwargs,
    ):
        """
        Initializes a new instance of the Wav2Vec2Config class.

        Args:
            self: The class instance.
            vocab_size (int, optional): The size of the vocabulary. Defaults to 32.
            hidden_size (int, optional): The size of the hidden layers. Defaults to 768.
            num_hidden_layers (int, optional): The number of hidden layers. Defaults to 12.
            num_attention_heads (int, optional): The number of attention heads. Defaults to 12.
            intermediate_size (int, optional): The size of the intermediate layers. Defaults to 3072.
            hidden_act (str, optional): The activation function for the hidden layers. Defaults to 'gelu'.
            hidden_dropout (float, optional): The dropout rate for the hidden layers. Defaults to 0.1.
            activation_dropout (float, optional): The dropout rate for the activation function. Defaults to 0.1.
            attention_dropout (float, optional): The dropout rate for the attention mechanism. Defaults to 0.1.
            feat_proj_dropout (float, optional): The dropout rate for the feature projection. Defaults to 0.0.
            feat_quantizer_dropout (float, optional): The dropout rate for the feature quantizer. Defaults to 0.0.
            final_dropout (float, optional): The final dropout rate. Defaults to 0.1.
            layerdrop (float, optional): The layer dropout rate. Defaults to 0.1.
            initializer_range (float, optional): The range for weight initialization. Defaults to 0.02.
            layer_norm_eps (float, optional): The epsilon value for layer normalization. Defaults to 1e-05.
            feat_extract_norm (str, optional): The normalization method for feature extraction. Defaults to 'group'.
            feat_extract_activation (str, optional): The activation function for feature extraction. Defaults to 'gelu'.
            conv_dim (tuple, optional): The dimensions for convolutional layers. Defaults to (512, 512, 512, 512, 512, 512, 512).
            conv_stride (tuple, optional): The stride for convolutional layers. Defaults to (5, 2, 2, 2, 2, 2, 2).
            conv_kernel (tuple, optional): The kernel size for convolutional layers. Defaults to (10, 3, 3, 3, 3, 2, 2).
            conv_bias (bool, optional): Whether to include bias in convolutional layers. Defaults to False.
            num_conv_pos_embeddings (int, optional): The number of positional embeddings for convolutional layers. Defaults to 128.
            num_conv_pos_embedding_groups (int, optional): The number of groups for positional embeddings. Defaults to 16.
            do_stable_layer_norm (bool, optional): Whether to use stable layer normalization. Defaults to False.
            apply_spec_augment (bool, optional): Whether to apply SpecAugment during training. Defaults to True.
            mask_time_prob (float, optional): The probability of masking time steps during SpecAugment. Defaults to 0.05.
            mask_time_length (int, optional): The maximum length of time masking during SpecAugment. Defaults to 10.
            mask_time_min_masks (int, optional): The minimum number of time masks during SpecAugment. Defaults to 2.
            mask_feature_prob (float, optional): The probability of masking features during SpecAugment. Defaults to 0.0.
            mask_feature_length (int, optional): The maximum length of feature masking during SpecAugment. Defaults to 10.
            mask_feature_min_masks (int, optional): The minimum number of feature masks during SpecAugment. Defaults to 0.
            num_codevectors_per_group (int, optional): The number of codevectors per group for quantization. Defaults to 320.
            num_codevector_groups (int, optional): The number of codevector groups for quantization. Defaults to 2.
            contrastive_logits_temperature (float, optional): The temperature for contrastive loss. Defaults to 0.1.
            num_negatives (int, optional): The number of negative samples for contrastive loss. Defaults to 100.
            codevector_dim (int, optional): The dimension of the codevectors. Defaults to 256.
            proj_codevector_dim (int, optional): The dimension of projected codevectors. Defaults to 256.
            diversity_loss_weight (float, optional): The weight for diversity loss. Defaults to 0.1.
            ctc_loss_reduction (str, optional): The reduction method for CTC loss. Defaults to 'sum'.
            ctc_zero_infinity (bool, optional): Whether to zero out infinity in CTC loss. Defaults to False.
            use_weighted_layer_sum (bool, optional): Whether to use weighted layer sum. Defaults to False.
            classifier_proj_size (int, optional): The size of the projection for the classifier. Defaults to 256.
            tdnn_dim (tuple, optional): The dimensions for time-delay neural network layers. Defaults to (512, 512, 512, 512, 1500).
            tdnn_kernel (tuple, optional): The kernel size for time-delay neural network layers. Defaults to (5, 3, 3, 1, 1).
            tdnn_dilation (tuple, optional): The dilation for time-delay neural network layers. Defaults to (1, 2, 3, 1, 1).
            xvector_output_dim (int, optional): The output dimension for x-vector representation. Defaults to 512.
            pad_token_id (int, optional): The token ID for padding. Defaults to 0.
            bos_token_id (int, optional): The token ID for the beginning of sentence. Defaults to 1.
            eos_token_id (int, optional): The token ID for the end of sentence. Defaults to 2.
            add_adapter (bool, optional): Whether to add adapter layers. Defaults to False.
            adapter_kernel_size (int, optional): The kernel size for adapter layers. Defaults to 3.
            adapter_stride (int, optional): The stride for adapter layers. Defaults to 2.
            num_adapter_layers (int, optional): The number of adapter layers. Defaults to 3.
            output_hidden_size (int, optional): The size of the output hidden layers. Defaults to None.
            adapter_attn_dim (int, optional): The attention dimension for adapter layers. Defaults to None.

        Returns:
            None.

        Raises:
            ValueError: If the configuration for convolutional layers is incorrect,
                i.e., if the dimensions, strides, or kernel sizes are not of the same length.

        """
        super().__init__(**kwargs, pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id)
        self.hidden_size = hidden_size
        self.feat_extract_norm = feat_extract_norm
        self.feat_extract_activation = feat_extract_activation
        self.conv_dim = list(conv_dim)
        self.conv_stride = list(conv_stride)
        self.conv_kernel = list(conv_kernel)
        self.conv_bias = conv_bias
        self.num_conv_pos_embeddings = num_conv_pos_embeddings
        self.num_conv_pos_embedding_groups = num_conv_pos_embedding_groups
        self.num_feat_extract_layers = len(self.conv_dim)
        self.num_hidden_layers = num_hidden_layers
        self.intermediate_size = intermediate_size
        self.hidden_act = hidden_act
        self.num_attention_heads = num_attention_heads
        self.hidden_dropout = hidden_dropout
        self.attention_dropout = attention_dropout
        self.activation_dropout = activation_dropout
        self.feat_proj_dropout = feat_proj_dropout
        self.final_dropout = final_dropout
        self.layerdrop = layerdrop
        self.layer_norm_eps = layer_norm_eps
        self.initializer_range = initializer_range
        self.vocab_size = vocab_size
        self.do_stable_layer_norm = do_stable_layer_norm
        self.use_weighted_layer_sum = use_weighted_layer_sum

        if (
            (len(self.conv_stride) != self.num_feat_extract_layers)
            or (len(self.conv_kernel) != self.num_feat_extract_layers)
            or (len(self.conv_dim) != self.num_feat_extract_layers)
        ):
            raise ValueError(
                "Configuration for convolutional layers is incorrect. It is required that `len(config.conv_dim)` =="
                " `len(config.conv_stride)` == `len(config.conv_kernel)`, but is `len(config.conv_dim) ="
                f" {len(self.conv_dim)}`, `len(config.conv_stride) = {len(self.conv_stride)}`,"
                f" `len(config.conv_kernel) = {len(self.conv_kernel)}`."
            )

        # fine-tuning config parameters for SpecAugment: https://arxiv.org/abs/1904.08779
        self.apply_spec_augment = apply_spec_augment
        self.mask_time_prob = mask_time_prob
        self.mask_time_length = mask_time_length
        self.mask_time_min_masks = mask_time_min_masks
        self.mask_feature_prob = mask_feature_prob
        self.mask_feature_length = mask_feature_length
        self.mask_feature_min_masks = mask_feature_min_masks

        # parameters for pretraining with codevector quantized representations
        self.num_codevectors_per_group = num_codevectors_per_group
        self.num_codevector_groups = num_codevector_groups
        self.contrastive_logits_temperature = contrastive_logits_temperature
        self.feat_quantizer_dropout = feat_quantizer_dropout
        self.num_negatives = num_negatives
        self.codevector_dim = codevector_dim
        self.proj_codevector_dim = proj_codevector_dim
        self.diversity_loss_weight = diversity_loss_weight

        # ctc loss
        self.ctc_loss_reduction = ctc_loss_reduction
        self.ctc_zero_infinity = ctc_zero_infinity

        # adapter
        self.add_adapter = add_adapter
        self.adapter_kernel_size = adapter_kernel_size
        self.adapter_stride = adapter_stride
        self.num_adapter_layers = num_adapter_layers
        self.output_hidden_size = output_hidden_size or hidden_size
        self.adapter_attn_dim = adapter_attn_dim

        # SequenceClassification-specific parameter. Feel free to ignore for other classes.
        self.classifier_proj_size = classifier_proj_size

        # XVector-specific parameters. Feel free to ignore for other classes.
        self.tdnn_dim = list(tdnn_dim)
        self.tdnn_kernel = list(tdnn_kernel)
        self.tdnn_dilation = list(tdnn_dilation)
        self.xvector_output_dim = xvector_output_dim

    @property
    def inputs_to_logits_ratio(self):
        """
        Calculates the ratio of inputs to logits for the Wav2Vec2Config class.

        Args:
            self (Wav2Vec2Config): The instance of the Wav2Vec2Config class.

        Returns:
            None: This method does not return any value.

        Raises:
            None.

        This method calculates the ratio of inputs to logits by multiplying the convolution stride values.
        The convolution stride values are accessed using the self.conv_stride attribute. The functools.reduce() function
        is used to multiply all the stride values together. If there are no stride values, the ratio is assumed to be
        1. The calculated ratio is then returned as the output of this method.
        """
        return functools.reduce(operator.mul, self.conv_stride, 1)

mindnlp.transformers.models.wav2vec2.configuration_wav2vec2.Wav2Vec2Config.inputs_to_logits_ratio property

Calculates the ratio of inputs to logits for the Wav2Vec2Config class.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2Config class. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
None	This method does not return any value.

This method calculates the ratio of inputs to logits by multiplying the convolution stride values. The convolution stride values are accessed using the self.conv_stride attribute. The functools.reduce() function is used to multiply all the stride values together. If there are no stride values, the ratio is assumed to be 1. The calculated ratio is then returned as the output of this method.

mindnlp.transformers.models.wav2vec2.configuration_wav2vec2.Wav2Vec2Config.__init__(vocab_size=32, hidden_size=768, num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, hidden_act='gelu', hidden_dropout=0.1, activation_dropout=0.1, attention_dropout=0.1, feat_proj_dropout=0.0, feat_quantizer_dropout=0.0, final_dropout=0.1, layerdrop=0.1, initializer_range=0.02, layer_norm_eps=1e-05, feat_extract_norm='group', feat_extract_activation='gelu', conv_dim=(512, 512, 512, 512, 512, 512, 512), conv_stride=(5, 2, 2, 2, 2, 2, 2), conv_kernel=(10, 3, 3, 3, 3, 2, 2), conv_bias=False, num_conv_pos_embeddings=128, num_conv_pos_embedding_groups=16, do_stable_layer_norm=False, apply_spec_augment=True, mask_time_prob=0.05, mask_time_length=10, mask_time_min_masks=2, mask_feature_prob=0.0, mask_feature_length=10, mask_feature_min_masks=0, num_codevectors_per_group=320, num_codevector_groups=2, contrastive_logits_temperature=0.1, num_negatives=100, codevector_dim=256, proj_codevector_dim=256, diversity_loss_weight=0.1, ctc_loss_reduction='sum', ctc_zero_infinity=False, use_weighted_layer_sum=False, classifier_proj_size=256, tdnn_dim=(512, 512, 512, 512, 1500), tdnn_kernel=(5, 3, 3, 1, 1), tdnn_dilation=(1, 2, 3, 1, 1), xvector_output_dim=512, pad_token_id=0, bos_token_id=1, eos_token_id=2, add_adapter=False, adapter_kernel_size=3, adapter_stride=2, num_adapter_layers=3, output_hidden_size=None, adapter_attn_dim=None, **kwargs)

Initializes a new instance of the Wav2Vec2Config class.

PARAMETER	DESCRIPTION
self	The class instance.
vocab_size	The size of the vocabulary. Defaults to 32. TYPE: int DEFAULT: 32
hidden_size	The size of the hidden layers. Defaults to 768. TYPE: int DEFAULT: 768
num_hidden_layers	The number of hidden layers. Defaults to 12. TYPE: int DEFAULT: 12
num_attention_heads	The number of attention heads. Defaults to 12. TYPE: int DEFAULT: 12
intermediate_size	The size of the intermediate layers. Defaults to 3072. TYPE: int DEFAULT: 3072
hidden_act	The activation function for the hidden layers. Defaults to 'gelu'. TYPE: str DEFAULT: 'gelu'
hidden_dropout	The dropout rate for the hidden layers. Defaults to 0.1. TYPE: float DEFAULT: 0.1
activation_dropout	The dropout rate for the activation function. Defaults to 0.1. TYPE: float DEFAULT: 0.1
attention_dropout	The dropout rate for the attention mechanism. Defaults to 0.1. TYPE: float DEFAULT: 0.1
feat_proj_dropout	The dropout rate for the feature projection. Defaults to 0.0. TYPE: float DEFAULT: 0.0
feat_quantizer_dropout	The dropout rate for the feature quantizer. Defaults to 0.0. TYPE: float DEFAULT: 0.0
final_dropout	The final dropout rate. Defaults to 0.1. TYPE: float DEFAULT: 0.1
layerdrop	The layer dropout rate. Defaults to 0.1. TYPE: float DEFAULT: 0.1
initializer_range	The range for weight initialization. Defaults to 0.02. TYPE: float DEFAULT: 0.02
layer_norm_eps	The epsilon value for layer normalization. Defaults to 1e-05. TYPE: float DEFAULT: 1e-05
feat_extract_norm	The normalization method for feature extraction. Defaults to 'group'. TYPE: str DEFAULT: 'group'
feat_extract_activation	The activation function for feature extraction. Defaults to 'gelu'. TYPE: str DEFAULT: 'gelu'
conv_dim	The dimensions for convolutional layers. Defaults to (512, 512, 512, 512, 512, 512, 512). TYPE: tuple DEFAULT: (512, 512, 512, 512, 512, 512, 512)
conv_stride	The stride for convolutional layers. Defaults to (5, 2, 2, 2, 2, 2, 2). TYPE: tuple DEFAULT: (5, 2, 2, 2, 2, 2, 2)
conv_kernel	The kernel size for convolutional layers. Defaults to (10, 3, 3, 3, 3, 2, 2). TYPE: tuple DEFAULT: (10, 3, 3, 3, 3, 2, 2)
conv_bias	Whether to include bias in convolutional layers. Defaults to False. TYPE: bool DEFAULT: False
num_conv_pos_embeddings	The number of positional embeddings for convolutional layers. Defaults to 128. TYPE: int DEFAULT: 128
num_conv_pos_embedding_groups	The number of groups for positional embeddings. Defaults to 16. TYPE: int DEFAULT: 16
do_stable_layer_norm	Whether to use stable layer normalization. Defaults to False. TYPE: bool DEFAULT: False
apply_spec_augment	Whether to apply SpecAugment during training. Defaults to True. TYPE: bool DEFAULT: True
mask_time_prob	The probability of masking time steps during SpecAugment. Defaults to 0.05. TYPE: float DEFAULT: 0.05
mask_time_length	The maximum length of time masking during SpecAugment. Defaults to 10. TYPE: int DEFAULT: 10
mask_time_min_masks	The minimum number of time masks during SpecAugment. Defaults to 2. TYPE: int DEFAULT: 2
mask_feature_prob	The probability of masking features during SpecAugment. Defaults to 0.0. TYPE: float DEFAULT: 0.0
mask_feature_length	The maximum length of feature masking during SpecAugment. Defaults to 10. TYPE: int DEFAULT: 10
mask_feature_min_masks	The minimum number of feature masks during SpecAugment. Defaults to 0. TYPE: int DEFAULT: 0
num_codevectors_per_group	The number of codevectors per group for quantization. Defaults to 320. TYPE: int DEFAULT: 320
num_codevector_groups	The number of codevector groups for quantization. Defaults to 2. TYPE: int DEFAULT: 2
contrastive_logits_temperature	The temperature for contrastive loss. Defaults to 0.1. TYPE: float DEFAULT: 0.1
num_negatives	The number of negative samples for contrastive loss. Defaults to 100. TYPE: int DEFAULT: 100
codevector_dim	The dimension of the codevectors. Defaults to 256. TYPE: int DEFAULT: 256
proj_codevector_dim	The dimension of projected codevectors. Defaults to 256. TYPE: int DEFAULT: 256
diversity_loss_weight	The weight for diversity loss. Defaults to 0.1. TYPE: float DEFAULT: 0.1
ctc_loss_reduction	The reduction method for CTC loss. Defaults to 'sum'. TYPE: str DEFAULT: 'sum'
ctc_zero_infinity	Whether to zero out infinity in CTC loss. Defaults to False. TYPE: bool DEFAULT: False
use_weighted_layer_sum	Whether to use weighted layer sum. Defaults to False. TYPE: bool DEFAULT: False
classifier_proj_size	The size of the projection for the classifier. Defaults to 256. TYPE: int DEFAULT: 256
tdnn_dim	The dimensions for time-delay neural network layers. Defaults to (512, 512, 512, 512, 1500). TYPE: tuple DEFAULT: (512, 512, 512, 512, 1500)
tdnn_kernel	The kernel size for time-delay neural network layers. Defaults to (5, 3, 3, 1, 1). TYPE: tuple DEFAULT: (5, 3, 3, 1, 1)
tdnn_dilation	The dilation for time-delay neural network layers. Defaults to (1, 2, 3, 1, 1). TYPE: tuple DEFAULT: (1, 2, 3, 1, 1)
xvector_output_dim	The output dimension for x-vector representation. Defaults to 512. TYPE: int DEFAULT: 512
pad_token_id	The token ID for padding. Defaults to 0. TYPE: int DEFAULT: 0
bos_token_id	The token ID for the beginning of sentence. Defaults to 1. TYPE: int DEFAULT: 1
eos_token_id	The token ID for the end of sentence. Defaults to 2. TYPE: int DEFAULT: 2
add_adapter	Whether to add adapter layers. Defaults to False. TYPE: bool DEFAULT: False
adapter_kernel_size	The kernel size for adapter layers. Defaults to 3. TYPE: int DEFAULT: 3
adapter_stride	The stride for adapter layers. Defaults to 2. TYPE: int DEFAULT: 2
num_adapter_layers	The number of adapter layers. Defaults to 3. TYPE: int DEFAULT: 3
output_hidden_size	The size of the output hidden layers. Defaults to None. TYPE: int DEFAULT: None
adapter_attn_dim	The attention dimension for adapter layers. Defaults to None. TYPE: int DEFAULT: None

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the configuration for convolutional layers is incorrect, i.e., if the dimensions, strides, or kernel sizes are not of the same length.

Source code in mindnlp/transformers/models/wav2vec2/configuration_wav2vec2.py

def __init__(
    self,
    vocab_size=32,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    hidden_act="gelu",
    hidden_dropout=0.1,
    activation_dropout=0.1,
    attention_dropout=0.1,
    feat_proj_dropout=0.0,
    feat_quantizer_dropout=0.0,
    final_dropout=0.1,
    layerdrop=0.1,
    initializer_range=0.02,
    layer_norm_eps=1e-5,
    feat_extract_norm="group",
    feat_extract_activation="gelu",
    conv_dim=(512, 512, 512, 512, 512, 512, 512),
    conv_stride=(5, 2, 2, 2, 2, 2, 2),
    conv_kernel=(10, 3, 3, 3, 3, 2, 2),
    conv_bias=False,
    num_conv_pos_embeddings=128,
    num_conv_pos_embedding_groups=16,
    do_stable_layer_norm=False,
    apply_spec_augment=True,
    mask_time_prob=0.05,
    mask_time_length=10,
    mask_time_min_masks=2,
    mask_feature_prob=0.0,
    mask_feature_length=10,
    mask_feature_min_masks=0,
    num_codevectors_per_group=320,
    num_codevector_groups=2,
    contrastive_logits_temperature=0.1,
    num_negatives=100,
    codevector_dim=256,
    proj_codevector_dim=256,
    diversity_loss_weight=0.1,
    ctc_loss_reduction="sum",
    ctc_zero_infinity=False,
    use_weighted_layer_sum=False,
    classifier_proj_size=256,
    tdnn_dim=(512, 512, 512, 512, 1500),
    tdnn_kernel=(5, 3, 3, 1, 1),
    tdnn_dilation=(1, 2, 3, 1, 1),
    xvector_output_dim=512,
    pad_token_id=0,
    bos_token_id=1,
    eos_token_id=2,
    add_adapter=False,
    adapter_kernel_size=3,
    adapter_stride=2,
    num_adapter_layers=3,
    output_hidden_size=None,
    adapter_attn_dim=None,
    **kwargs,
):
    """
    Initializes a new instance of the Wav2Vec2Config class.

    Args:
        self: The class instance.
        vocab_size (int, optional): The size of the vocabulary. Defaults to 32.
        hidden_size (int, optional): The size of the hidden layers. Defaults to 768.
        num_hidden_layers (int, optional): The number of hidden layers. Defaults to 12.
        num_attention_heads (int, optional): The number of attention heads. Defaults to 12.
        intermediate_size (int, optional): The size of the intermediate layers. Defaults to 3072.
        hidden_act (str, optional): The activation function for the hidden layers. Defaults to 'gelu'.
        hidden_dropout (float, optional): The dropout rate for the hidden layers. Defaults to 0.1.
        activation_dropout (float, optional): The dropout rate for the activation function. Defaults to 0.1.
        attention_dropout (float, optional): The dropout rate for the attention mechanism. Defaults to 0.1.
        feat_proj_dropout (float, optional): The dropout rate for the feature projection. Defaults to 0.0.
        feat_quantizer_dropout (float, optional): The dropout rate for the feature quantizer. Defaults to 0.0.
        final_dropout (float, optional): The final dropout rate. Defaults to 0.1.
        layerdrop (float, optional): The layer dropout rate. Defaults to 0.1.
        initializer_range (float, optional): The range for weight initialization. Defaults to 0.02.
        layer_norm_eps (float, optional): The epsilon value for layer normalization. Defaults to 1e-05.
        feat_extract_norm (str, optional): The normalization method for feature extraction. Defaults to 'group'.
        feat_extract_activation (str, optional): The activation function for feature extraction. Defaults to 'gelu'.
        conv_dim (tuple, optional): The dimensions for convolutional layers. Defaults to (512, 512, 512, 512, 512, 512, 512).
        conv_stride (tuple, optional): The stride for convolutional layers. Defaults to (5, 2, 2, 2, 2, 2, 2).
        conv_kernel (tuple, optional): The kernel size for convolutional layers. Defaults to (10, 3, 3, 3, 3, 2, 2).
        conv_bias (bool, optional): Whether to include bias in convolutional layers. Defaults to False.
        num_conv_pos_embeddings (int, optional): The number of positional embeddings for convolutional layers. Defaults to 128.
        num_conv_pos_embedding_groups (int, optional): The number of groups for positional embeddings. Defaults to 16.
        do_stable_layer_norm (bool, optional): Whether to use stable layer normalization. Defaults to False.
        apply_spec_augment (bool, optional): Whether to apply SpecAugment during training. Defaults to True.
        mask_time_prob (float, optional): The probability of masking time steps during SpecAugment. Defaults to 0.05.
        mask_time_length (int, optional): The maximum length of time masking during SpecAugment. Defaults to 10.
        mask_time_min_masks (int, optional): The minimum number of time masks during SpecAugment. Defaults to 2.
        mask_feature_prob (float, optional): The probability of masking features during SpecAugment. Defaults to 0.0.
        mask_feature_length (int, optional): The maximum length of feature masking during SpecAugment. Defaults to 10.
        mask_feature_min_masks (int, optional): The minimum number of feature masks during SpecAugment. Defaults to 0.
        num_codevectors_per_group (int, optional): The number of codevectors per group for quantization. Defaults to 320.
        num_codevector_groups (int, optional): The number of codevector groups for quantization. Defaults to 2.
        contrastive_logits_temperature (float, optional): The temperature for contrastive loss. Defaults to 0.1.
        num_negatives (int, optional): The number of negative samples for contrastive loss. Defaults to 100.
        codevector_dim (int, optional): The dimension of the codevectors. Defaults to 256.
        proj_codevector_dim (int, optional): The dimension of projected codevectors. Defaults to 256.
        diversity_loss_weight (float, optional): The weight for diversity loss. Defaults to 0.1.
        ctc_loss_reduction (str, optional): The reduction method for CTC loss. Defaults to 'sum'.
        ctc_zero_infinity (bool, optional): Whether to zero out infinity in CTC loss. Defaults to False.
        use_weighted_layer_sum (bool, optional): Whether to use weighted layer sum. Defaults to False.
        classifier_proj_size (int, optional): The size of the projection for the classifier. Defaults to 256.
        tdnn_dim (tuple, optional): The dimensions for time-delay neural network layers. Defaults to (512, 512, 512, 512, 1500).
        tdnn_kernel (tuple, optional): The kernel size for time-delay neural network layers. Defaults to (5, 3, 3, 1, 1).
        tdnn_dilation (tuple, optional): The dilation for time-delay neural network layers. Defaults to (1, 2, 3, 1, 1).
        xvector_output_dim (int, optional): The output dimension for x-vector representation. Defaults to 512.
        pad_token_id (int, optional): The token ID for padding. Defaults to 0.
        bos_token_id (int, optional): The token ID for the beginning of sentence. Defaults to 1.
        eos_token_id (int, optional): The token ID for the end of sentence. Defaults to 2.
        add_adapter (bool, optional): Whether to add adapter layers. Defaults to False.
        adapter_kernel_size (int, optional): The kernel size for adapter layers. Defaults to 3.
        adapter_stride (int, optional): The stride for adapter layers. Defaults to 2.
        num_adapter_layers (int, optional): The number of adapter layers. Defaults to 3.
        output_hidden_size (int, optional): The size of the output hidden layers. Defaults to None.
        adapter_attn_dim (int, optional): The attention dimension for adapter layers. Defaults to None.

    Returns:
        None.

    Raises:
        ValueError: If the configuration for convolutional layers is incorrect,
            i.e., if the dimensions, strides, or kernel sizes are not of the same length.

    """
    super().__init__(**kwargs, pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id)
    self.hidden_size = hidden_size
    self.feat_extract_norm = feat_extract_norm
    self.feat_extract_activation = feat_extract_activation
    self.conv_dim = list(conv_dim)
    self.conv_stride = list(conv_stride)
    self.conv_kernel = list(conv_kernel)
    self.conv_bias = conv_bias
    self.num_conv_pos_embeddings = num_conv_pos_embeddings
    self.num_conv_pos_embedding_groups = num_conv_pos_embedding_groups
    self.num_feat_extract_layers = len(self.conv_dim)
    self.num_hidden_layers = num_hidden_layers
    self.intermediate_size = intermediate_size
    self.hidden_act = hidden_act
    self.num_attention_heads = num_attention_heads
    self.hidden_dropout = hidden_dropout
    self.attention_dropout = attention_dropout
    self.activation_dropout = activation_dropout
    self.feat_proj_dropout = feat_proj_dropout
    self.final_dropout = final_dropout
    self.layerdrop = layerdrop
    self.layer_norm_eps = layer_norm_eps
    self.initializer_range = initializer_range
    self.vocab_size = vocab_size
    self.do_stable_layer_norm = do_stable_layer_norm
    self.use_weighted_layer_sum = use_weighted_layer_sum

    if (
        (len(self.conv_stride) != self.num_feat_extract_layers)
        or (len(self.conv_kernel) != self.num_feat_extract_layers)
        or (len(self.conv_dim) != self.num_feat_extract_layers)
    ):
        raise ValueError(
            "Configuration for convolutional layers is incorrect. It is required that `len(config.conv_dim)` =="
            " `len(config.conv_stride)` == `len(config.conv_kernel)`, but is `len(config.conv_dim) ="
            f" {len(self.conv_dim)}`, `len(config.conv_stride) = {len(self.conv_stride)}`,"
            f" `len(config.conv_kernel) = {len(self.conv_kernel)}`."
        )

    # fine-tuning config parameters for SpecAugment: https://arxiv.org/abs/1904.08779
    self.apply_spec_augment = apply_spec_augment
    self.mask_time_prob = mask_time_prob
    self.mask_time_length = mask_time_length
    self.mask_time_min_masks = mask_time_min_masks
    self.mask_feature_prob = mask_feature_prob
    self.mask_feature_length = mask_feature_length
    self.mask_feature_min_masks = mask_feature_min_masks

    # parameters for pretraining with codevector quantized representations
    self.num_codevectors_per_group = num_codevectors_per_group
    self.num_codevector_groups = num_codevector_groups
    self.contrastive_logits_temperature = contrastive_logits_temperature
    self.feat_quantizer_dropout = feat_quantizer_dropout
    self.num_negatives = num_negatives
    self.codevector_dim = codevector_dim
    self.proj_codevector_dim = proj_codevector_dim
    self.diversity_loss_weight = diversity_loss_weight

    # ctc loss
    self.ctc_loss_reduction = ctc_loss_reduction
    self.ctc_zero_infinity = ctc_zero_infinity

    # adapter
    self.add_adapter = add_adapter
    self.adapter_kernel_size = adapter_kernel_size
    self.adapter_stride = adapter_stride
    self.num_adapter_layers = num_adapter_layers
    self.output_hidden_size = output_hidden_size or hidden_size
    self.adapter_attn_dim = adapter_attn_dim

    # SequenceClassification-specific parameter. Feel free to ignore for other classes.
    self.classifier_proj_size = classifier_proj_size

    # XVector-specific parameters. Feel free to ignore for other classes.
    self.tdnn_dim = list(tdnn_dim)
    self.tdnn_kernel = list(tdnn_kernel)
    self.tdnn_dilation = list(tdnn_dilation)
    self.xvector_output_dim = xvector_output_dim

mindnlp.transformers.models.wav2vec2.feature_extraction_wav2vec2

Feature extractor class for Wav2Vec2

mindnlp.transformers.models.wav2vec2.feature_extraction_wav2vec2.Wav2Vec2FeatureExtractor

Bases: SequenceFeatureExtractor

Constructs a Wav2Vec2 feature extractor.

This feature extractor inherits from [~feature_extraction_sequence_utils.SequenceFeatureExtractor] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

PARAMETER	DESCRIPTION
feature_size	The feature dimension of the extracted features. TYPE: `int`, defaults to 1 DEFAULT: 1
sampling_rate	The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). TYPE: `int`, defaults to 16000 DEFAULT: 16000
padding_value	The value that is used to fill the padding values. TYPE: `float`, defaults to 0.0 DEFAULT: 0.0
do_normalize	Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly improve the performance for some models, e.g., wav2vec2-lv60. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
return_attention_mask	Whether or not [~Wav2Vec2FeatureExtractor.__call__] should return attention_mask. Wav2Vec2 models that have set config.feat_extract_norm == "group", such as wav2vec2-base, have not been trained using attention_mask. For such models, input_values should simply be padded with 0 and no attention_mask should be passed. For Wav2Vec2 models that have set config.feat_extract_norm == "layer", such as wav2vec2-lv60, attention_mask should be passed for batched inference. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False

Source code in mindnlp/transformers/models/wav2vec2/feature_extraction_wav2vec2.py

class Wav2Vec2FeatureExtractor(SequenceFeatureExtractor):
    r"""
    Constructs a Wav2Vec2 feature extractor.

    This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains
    most of the main methods. Users should refer to this superclass for more information regarding those methods.

    Args:
        feature_size (`int`, defaults to 1):
            The feature dimension of the extracted features.
        sampling_rate (`int`, defaults to 16000):
            The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
        padding_value (`float`, defaults to 0.0):
            The value that is used to fill the padding values.
        do_normalize (`bool`, *optional*, defaults to `True`):
            Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly
            improve the performance for some models, *e.g.*,
            [wav2vec2-lv60](https://hf-mirror.com/models?search=lv60).
        return_attention_mask (`bool`, *optional*, defaults to `False`):
            Whether or not [`~Wav2Vec2FeatureExtractor.__call__`] should return `attention_mask`.

            <Tip>

            Wav2Vec2 models that have set `config.feat_extract_norm == "group"`, such as
            [wav2vec2-base](https://hf-mirror.com/facebook/wav2vec2-base-960h), have **not** been trained using
            `attention_mask`. For such models, `input_values` should simply be padded with 0 and no `attention_mask`
            should be passed.

            For Wav2Vec2 models that have set `config.feat_extract_norm == "layer"`, such as
            [wav2vec2-lv60](https://hf-mirror.com/facebook/wav2vec2-large-960h-lv60-self), `attention_mask` should be
            passed for batched inference.

            </Tip>"""
    model_input_names = ["input_values", "attention_mask"]

    def __init__(
        self,
        feature_size=1,
        sampling_rate=16000,
        padding_value=0.0,
        return_attention_mask=False,
        do_normalize=True,
        **kwargs,
    ):
        """
        Initialize the Wav2Vec2FeatureExtractor class.

        Args:
            self (object): The instance of the class.
            feature_size (int, optional): The size of the input features. Defaults to 1.
            sampling_rate (int, optional): The sampling rate of the audio data. Defaults to 16000.
            padding_value (float, optional): The value used for padding sequences. Defaults to 0.0.
            return_attention_mask (bool, optional): Whether to return the attention mask. Defaults to False.
            do_normalize (bool, optional): Whether to normalize the input features. Defaults to True.
            **kwargs: Additional keyword arguments.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(feature_size=feature_size, sampling_rate=sampling_rate, padding_value=padding_value, **kwargs)
        self.return_attention_mask = return_attention_mask
        self.do_normalize = do_normalize

    @staticmethod
    def zero_mean_unit_var_norm(
        input_values: List[np.ndarray], attention_mask: List[np.ndarray], padding_value: float = 0.0
    ) -> List[np.ndarray]:
        """
        Every array in the list is normalized to have zero mean and unit variance
        """
        if attention_mask is not None:
            attention_mask = np.array(attention_mask, np.int32)
            normed_input_values = []

            for vector, length in zip(input_values, attention_mask.sum(-1)):
                normed_slice = (vector - vector[:length].mean()) / np.sqrt(vector[:length].var() + 1e-7)
                if length < normed_slice.shape[0]:
                    normed_slice[length:] = padding_value

                normed_input_values.append(normed_slice)
        else:
            normed_input_values = [(x - x.mean()) / np.sqrt(x.var() + 1e-7) for x in input_values]

        return normed_input_values

    def __call__(
        self,
        raw_speech: Union[np.ndarray, List[float], List[np.ndarray], List[List[float]]],
        padding: Union[bool, str, PaddingStrategy] = False,
        max_length: Optional[int] = None,
        truncation: bool = False,
        pad_to_multiple_of: Optional[int] = None,
        return_attention_mask: Optional[bool] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        sampling_rate: Optional[int] = None,
        **kwargs,
    ) -> BatchFeature:
        """
        Main method to featurize and prepare for the model one or several sequence(s).

        Args:
            raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`):
                The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float
                values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not
                stereo, i.e. single float per timestep.
            padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`):
                Select a strategy to pad the returned sequences (according to the model's padding side and padding
                index) among:

                - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
                sequence if provided).
                - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
                acceptable input length for the model if that argument is not provided.
                - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
                lengths).
            max_length (`int`, *optional*):
                Maximum length of the returned list and optionally padding length (see above).
            truncation (`bool`):
                Activates truncation to cut input sequences longer than *max_length* to *max_length*.
            pad_to_multiple_of (`int`, *optional*):
                If set will pad the sequence to a multiple of the provided value.

                This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
                `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128.
            return_attention_mask (`bool`, *optional*):
                Whether to return the attention mask. If left to the default, will return the attention mask according
                to the specific feature_extractor's default.

                [What are attention masks?](../glossary#attention-mask)

                <Tip>

                Wav2Vec2 models that have set `config.feat_extract_norm == "group"`, such as
                [wav2vec2-base](https://hf-mirror.com/facebook/wav2vec2-base-960h), have **not** been trained using
                `attention_mask`. For such models, `input_values` should simply be padded with 0 and no
                `attention_mask` should be passed.

                For Wav2Vec2 models that have set `config.feat_extract_norm == "layer"`, such as
                [wav2vec2-lv60](https://hf-mirror.com/facebook/wav2vec2-large-960h-lv60-self), `attention_mask` should
                be passed for batched inference.

                </Tip>

            return_tensors (`str` or [`~utils.TensorType`], *optional*):
                If set, will return tensors instead of list of python integers. Acceptable values are:

                - `'tf'`: Return TensorFlow `tf.constant` objects.
                - `'pt'`: Return PyTorch `torch.Tensor` objects.
                - `'np'`: Return Numpy `np.ndarray` objects.
            sampling_rate (`int`, *optional*):
                The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass
                `sampling_rate` at the forward call to prevent silent errors.
            padding_value (`float`, defaults to 0.0):
        """
        if sampling_rate is not None:
            if sampling_rate != self.sampling_rate:
                raise ValueError(
                    f"The model corresponding to this feature extractor: {self} was trained using a sampling rate of"
                    f" {self.sampling_rate}. Please make sure that the provided `raw_speech` input was sampled with"
                    f" {self.sampling_rate} and not {sampling_rate}."
                )
        else:
            logger.warning(
                "It is strongly recommended to pass the ``sampling_rate`` argument to this function. "
                "Failing to do so can result in silent errors that might be hard to debug."
            )

        is_batched_numpy = isinstance(raw_speech, np.ndarray) and len(raw_speech.shape) > 1
        if is_batched_numpy and len(raw_speech.shape) > 2:
            raise ValueError(f"Only mono-channel audio is supported for input to {self}")
        is_batched = is_batched_numpy or (
            isinstance(raw_speech, (list, tuple)) and (isinstance(raw_speech[0], (np.ndarray, tuple, list)))
        )

        # always return batch
        if not is_batched:
            raw_speech = [raw_speech]

        # convert into correct format for padding
        encoded_inputs = BatchFeature({"input_values": raw_speech})

        padded_inputs = self.pad(
            encoded_inputs,
            padding=padding,
            max_length=max_length,
            truncation=truncation,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=return_attention_mask,
        )

        # convert input values to correct format
        input_values = padded_inputs["input_values"]
        if not isinstance(input_values[0], np.ndarray):
            padded_inputs["input_values"] = [np.asarray(array, dtype=np.float32) for array in input_values]
        elif (
            not isinstance(input_values, np.ndarray)
            and isinstance(input_values[0], np.ndarray)
            and input_values[0].dtype is np.dtype(np.float64)
        ):
            padded_inputs["input_values"] = [array.astype(np.float32) for array in input_values]
        elif isinstance(input_values, np.ndarray) and input_values.dtype is np.dtype(np.float64):
            padded_inputs["input_values"] = input_values.astype(np.float32)

        # convert attention_mask to correct format
        attention_mask = padded_inputs.get("attention_mask")
        if attention_mask is not None:
            padded_inputs["attention_mask"] = [np.asarray(array, dtype=np.int32) for array in attention_mask]

        # zero-mean and unit-variance normalization
        if self.do_normalize:
            attention_mask = (
                attention_mask
                if self._get_padding_strategies(padding, max_length=max_length) is not PaddingStrategy.DO_NOT_PAD
                else None
            )
            padded_inputs["input_values"] = self.zero_mean_unit_var_norm(
                padded_inputs["input_values"], attention_mask=attention_mask, padding_value=self.padding_value
            )

        if return_tensors is not None:
            padded_inputs = padded_inputs.convert_to_tensors(return_tensors)

        return padded_inputs

mindnlp.transformers.models.wav2vec2.feature_extraction_wav2vec2.Wav2Vec2FeatureExtractor.__call__(raw_speech, padding=False, max_length=None, truncation=False, pad_to_multiple_of=None, return_attention_mask=None, return_tensors=None, sampling_rate=None, **kwargs)

Main method to featurize and prepare for the model one or several sequence(s).

PARAMETER	DESCRIPTION
raw_speech	The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. TYPE: `np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`
padding	Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: True or 'longest': Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). 'max_length': Pad to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided. False or 'do_not_pad' (default): No padding (i.e., can output a batch with sequences of different lengths). TYPE: `bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False` DEFAULT: False
max_length	Maximum length of the returned list and optionally padding length (see above). TYPE: `int`, *optional* DEFAULT: None
truncation	Activates truncation to cut input sequences longer than max_length to max_length. TYPE: `bool` DEFAULT: False
pad_to_multiple_of	If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. TYPE: `int`, *optional* DEFAULT: None
return_attention_mask	Whether to return the attention mask. If left to the default, will return the attention mask according to the specific feature_extractor's default. What are attention masks? Wav2Vec2 models that have set config.feat_extract_norm == "group", such as wav2vec2-base, have not been trained using attention_mask. For such models, input_values should simply be padded with 0 and no attention_mask should be passed. For Wav2Vec2 models that have set config.feat_extract_norm == "layer", such as wav2vec2-lv60, attention_mask should be passed for batched inference. TYPE: `bool`, *optional* DEFAULT: None
return_tensors	If set, will return tensors instead of list of python integers. Acceptable values are: 'tf': Return TensorFlow tf.constant objects. 'pt': Return PyTorch torch.Tensor objects. 'np': Return Numpy np.ndarray objects. TYPE: `str` or [`~utils.TensorType`], *optional* DEFAULT: None
sampling_rate	The sampling rate at which the raw_speech input was sampled. It is strongly recommended to pass sampling_rate at the forward call to prevent silent errors. TYPE: `int`, *optional* DEFAULT: None
padding_value	TYPE: `float`, defaults to 0.0

Source code in mindnlp/transformers/models/wav2vec2/feature_extraction_wav2vec2.py

def __call__(
    self,
    raw_speech: Union[np.ndarray, List[float], List[np.ndarray], List[List[float]]],
    padding: Union[bool, str, PaddingStrategy] = False,
    max_length: Optional[int] = None,
    truncation: bool = False,
    pad_to_multiple_of: Optional[int] = None,
    return_attention_mask: Optional[bool] = None,
    return_tensors: Optional[Union[str, TensorType]] = None,
    sampling_rate: Optional[int] = None,
    **kwargs,
) -> BatchFeature:
    """
    Main method to featurize and prepare for the model one or several sequence(s).

    Args:
        raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`):
            The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float
            values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not
            stereo, i.e. single float per timestep.
        padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`):
            Select a strategy to pad the returned sequences (according to the model's padding side and padding
            index) among:

            - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
            sequence if provided).
            - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
            acceptable input length for the model if that argument is not provided.
            - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
            lengths).
        max_length (`int`, *optional*):
            Maximum length of the returned list and optionally padding length (see above).
        truncation (`bool`):
            Activates truncation to cut input sequences longer than *max_length* to *max_length*.
        pad_to_multiple_of (`int`, *optional*):
            If set will pad the sequence to a multiple of the provided value.

            This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
            `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128.
        return_attention_mask (`bool`, *optional*):
            Whether to return the attention mask. If left to the default, will return the attention mask according
            to the specific feature_extractor's default.

            [What are attention masks?](../glossary#attention-mask)

            <Tip>

            Wav2Vec2 models that have set `config.feat_extract_norm == "group"`, such as
            [wav2vec2-base](https://hf-mirror.com/facebook/wav2vec2-base-960h), have **not** been trained using
            `attention_mask`. For such models, `input_values` should simply be padded with 0 and no
            `attention_mask` should be passed.

            For Wav2Vec2 models that have set `config.feat_extract_norm == "layer"`, such as
            [wav2vec2-lv60](https://hf-mirror.com/facebook/wav2vec2-large-960h-lv60-self), `attention_mask` should
            be passed for batched inference.

            </Tip>

        return_tensors (`str` or [`~utils.TensorType`], *optional*):
            If set, will return tensors instead of list of python integers. Acceptable values are:

            - `'tf'`: Return TensorFlow `tf.constant` objects.
            - `'pt'`: Return PyTorch `torch.Tensor` objects.
            - `'np'`: Return Numpy `np.ndarray` objects.
        sampling_rate (`int`, *optional*):
            The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass
            `sampling_rate` at the forward call to prevent silent errors.
        padding_value (`float`, defaults to 0.0):
    """
    if sampling_rate is not None:
        if sampling_rate != self.sampling_rate:
            raise ValueError(
                f"The model corresponding to this feature extractor: {self} was trained using a sampling rate of"
                f" {self.sampling_rate}. Please make sure that the provided `raw_speech` input was sampled with"
                f" {self.sampling_rate} and not {sampling_rate}."
            )
    else:
        logger.warning(
            "It is strongly recommended to pass the ``sampling_rate`` argument to this function. "
            "Failing to do so can result in silent errors that might be hard to debug."
        )

    is_batched_numpy = isinstance(raw_speech, np.ndarray) and len(raw_speech.shape) > 1
    if is_batched_numpy and len(raw_speech.shape) > 2:
        raise ValueError(f"Only mono-channel audio is supported for input to {self}")
    is_batched = is_batched_numpy or (
        isinstance(raw_speech, (list, tuple)) and (isinstance(raw_speech[0], (np.ndarray, tuple, list)))
    )

    # always return batch
    if not is_batched:
        raw_speech = [raw_speech]

    # convert into correct format for padding
    encoded_inputs = BatchFeature({"input_values": raw_speech})

    padded_inputs = self.pad(
        encoded_inputs,
        padding=padding,
        max_length=max_length,
        truncation=truncation,
        pad_to_multiple_of=pad_to_multiple_of,
        return_attention_mask=return_attention_mask,
    )

    # convert input values to correct format
    input_values = padded_inputs["input_values"]
    if not isinstance(input_values[0], np.ndarray):
        padded_inputs["input_values"] = [np.asarray(array, dtype=np.float32) for array in input_values]
    elif (
        not isinstance(input_values, np.ndarray)
        and isinstance(input_values[0], np.ndarray)
        and input_values[0].dtype is np.dtype(np.float64)
    ):
        padded_inputs["input_values"] = [array.astype(np.float32) for array in input_values]
    elif isinstance(input_values, np.ndarray) and input_values.dtype is np.dtype(np.float64):
        padded_inputs["input_values"] = input_values.astype(np.float32)

    # convert attention_mask to correct format
    attention_mask = padded_inputs.get("attention_mask")
    if attention_mask is not None:
        padded_inputs["attention_mask"] = [np.asarray(array, dtype=np.int32) for array in attention_mask]

    # zero-mean and unit-variance normalization
    if self.do_normalize:
        attention_mask = (
            attention_mask
            if self._get_padding_strategies(padding, max_length=max_length) is not PaddingStrategy.DO_NOT_PAD
            else None
        )
        padded_inputs["input_values"] = self.zero_mean_unit_var_norm(
            padded_inputs["input_values"], attention_mask=attention_mask, padding_value=self.padding_value
        )

    if return_tensors is not None:
        padded_inputs = padded_inputs.convert_to_tensors(return_tensors)

    return padded_inputs

mindnlp.transformers.models.wav2vec2.feature_extraction_wav2vec2.Wav2Vec2FeatureExtractor.__init__(feature_size=1, sampling_rate=16000, padding_value=0.0, return_attention_mask=False, do_normalize=True, **kwargs)

Initialize the Wav2Vec2FeatureExtractor class.

PARAMETER	DESCRIPTION
self	The instance of the class. TYPE: object
feature_size	The size of the input features. Defaults to 1. TYPE: int DEFAULT: 1
sampling_rate	The sampling rate of the audio data. Defaults to 16000. TYPE: int DEFAULT: 16000
padding_value	The value used for padding sequences. Defaults to 0.0. TYPE: float DEFAULT: 0.0
return_attention_mask	Whether to return the attention mask. Defaults to False. TYPE: bool DEFAULT: False
do_normalize	Whether to normalize the input features. Defaults to True. TYPE: bool DEFAULT: True
**kwargs	Additional keyword arguments. DEFAULT: {}

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/feature_extraction_wav2vec2.py

def __init__(
    self,
    feature_size=1,
    sampling_rate=16000,
    padding_value=0.0,
    return_attention_mask=False,
    do_normalize=True,
    **kwargs,
):
    """
    Initialize the Wav2Vec2FeatureExtractor class.

    Args:
        self (object): The instance of the class.
        feature_size (int, optional): The size of the input features. Defaults to 1.
        sampling_rate (int, optional): The sampling rate of the audio data. Defaults to 16000.
        padding_value (float, optional): The value used for padding sequences. Defaults to 0.0.
        return_attention_mask (bool, optional): Whether to return the attention mask. Defaults to False.
        do_normalize (bool, optional): Whether to normalize the input features. Defaults to True.
        **kwargs: Additional keyword arguments.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(feature_size=feature_size, sampling_rate=sampling_rate, padding_value=padding_value, **kwargs)
    self.return_attention_mask = return_attention_mask
    self.do_normalize = do_normalize

mindnlp.transformers.models.wav2vec2.feature_extraction_wav2vec2.Wav2Vec2FeatureExtractor.zero_mean_unit_var_norm(input_values, attention_mask, padding_value=0.0) staticmethod

Every array in the list is normalized to have zero mean and unit variance

Source code in mindnlp/transformers/models/wav2vec2/feature_extraction_wav2vec2.py

@staticmethod
def zero_mean_unit_var_norm(
    input_values: List[np.ndarray], attention_mask: List[np.ndarray], padding_value: float = 0.0
) -> List[np.ndarray]:
    """
    Every array in the list is normalized to have zero mean and unit variance
    """
    if attention_mask is not None:
        attention_mask = np.array(attention_mask, np.int32)
        normed_input_values = []

        for vector, length in zip(input_values, attention_mask.sum(-1)):
            normed_slice = (vector - vector[:length].mean()) / np.sqrt(vector[:length].var() + 1e-7)
            if length < normed_slice.shape[0]:
                normed_slice[length:] = padding_value

            normed_input_values.append(normed_slice)
    else:
        normed_input_values = [(x - x.mean()) / np.sqrt(x.var() + 1e-7) for x in input_values]

    return normed_input_values

mindnlp.transformers.models.wav2vec2.processing_wav2vec2

Speech processor class for Wav2Vec2

mindnlp.transformers.models.wav2vec2.processing_wav2vec2.Wav2Vec2Processor

Bases: ProcessorMixin

Constructs a Wav2Vec2 processor which wraps a Wav2Vec2 feature extractor and a Wav2Vec2 CTC tokenizer into a single processor.

[Wav2Vec2Processor] offers all the functionalities of [Wav2Vec2FeatureExtractor] and [PreTrainedTokenizer]. See the docstring of [~Wav2Vec2Processor.__call__] and [~Wav2Vec2Processor.decode] for more information.

PARAMETER	DESCRIPTION
feature_extractor	An instance of [Wav2Vec2FeatureExtractor]. The feature extractor is a required input. TYPE: `Wav2Vec2FeatureExtractor`
tokenizer	An instance of [PreTrainedTokenizer]. The tokenizer is a required input. TYPE: [`PreTrainedTokenizer`]

Source code in mindnlp/transformers/models/wav2vec2/processing_wav2vec2.py

class Wav2Vec2Processor(ProcessorMixin):
    r"""
    Constructs a Wav2Vec2 processor which wraps a Wav2Vec2 feature extractor and a Wav2Vec2 CTC tokenizer into a single
    processor.

    [`Wav2Vec2Processor`] offers all the functionalities of [`Wav2Vec2FeatureExtractor`] and [`PreTrainedTokenizer`].
    See the docstring of [`~Wav2Vec2Processor.__call__`] and [`~Wav2Vec2Processor.decode`] for more information.

    Args:
        feature_extractor (`Wav2Vec2FeatureExtractor`):
            An instance of [`Wav2Vec2FeatureExtractor`]. The feature extractor is a required input.
        tokenizer ([`PreTrainedTokenizer`]):
            An instance of [`PreTrainedTokenizer`]. The tokenizer is a required input.
    """
    feature_extractor_class = "Wav2Vec2FeatureExtractor"
    tokenizer_class = "AutoTokenizer"

    def __init__(self, feature_extractor, tokenizer):
        """
        Initializes a new instance of the Wav2Vec2Processor class.

        Args:
            self (Wav2Vec2Processor): The current instance of the Wav2Vec2Processor class.
            feature_extractor (object): The feature extractor used for processing input data.
                It should be an instance of a feature extraction class.
            tokenizer (object): The tokenizer used for tokenizing input data.
                It should be an instance of a tokenizer class.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(feature_extractor, tokenizer)
        self.current_processor = self.feature_extractor
        self._in_target_context_manager = False

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
        """
        This method creates an instance of the Wav2Vec2Processor class from a pre-trained model.

        Args:
            cls (class): The class itself.
            pretrained_model_name_or_path (str): The name or path of the pre-trained model to load.

        Returns:
            None.

        Raises:
            OSError: If an OSError occurs during the loading process.
            FutureWarning: If the tokenizer is being loaded from a config that does not include a `tokenizer_class`
                attribute, a FutureWarning is issued. It advises adding a `'tokenizer_class': 'Wav2Vec2CTCTokenizer'`
                attribute to either the `config.json` or `tokenizer_config.json` file to suppress the warning.
        """
        try:
            return super().from_pretrained(pretrained_model_name_or_path, **kwargs)
        except OSError:
            warnings.warn(
                f"Loading a tokenizer inside {cls.__name__} from a config that does not"
                " include a `tokenizer_class` attribute is deprecated and will be "
                "removed in v5. Please add `'tokenizer_class': 'Wav2Vec2CTCTokenizer'`"
                " attribute to either your `config.json` or `tokenizer_config.json` "
                "file to suppress this warning: ",
                FutureWarning,
            )

            feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(pretrained_model_name_or_path, **kwargs)
            tokenizer = Wav2Vec2CTCTokenizer.from_pretrained(pretrained_model_name_or_path, **kwargs)

            return cls(feature_extractor=feature_extractor, tokenizer=tokenizer)

    def __call__(self, *args, **kwargs):
        """
        When used in normal mode, this method forwards all its arguments to Wav2Vec2FeatureExtractor's
        [`~Wav2Vec2FeatureExtractor.__call__`] and returns its output. If used in the context
        [`~Wav2Vec2Processor.as_target_processor`] this method forwards all its arguments to PreTrainedTokenizer's
        [`~PreTrainedTokenizer.__call__`]. Please refer to the docstring of the above two methods for more information.
        """
        # For backward compatibility
        if self._in_target_context_manager:
            return self.current_processor(*args, **kwargs)

        if "raw_speech" in kwargs:
            warnings.warn("Using `raw_speech` as a keyword argument is deprecated. Use `audio` instead.")
            audio = kwargs.pop("raw_speech")
        else:
            audio = kwargs.pop("audio", None)
        sampling_rate = kwargs.pop("sampling_rate", None)
        text = kwargs.pop("text", None)
        if len(args) > 0:
            audio = args[0]
            args = args[1:]

        if audio is None and text is None:
            raise ValueError("You need to specify either an `audio` or `text` input to process.")

        if audio is not None:
            inputs = self.feature_extractor(audio, *args, sampling_rate=sampling_rate, **kwargs)
        if text is not None:
            encodings = self.tokenizer(text, **kwargs)

        if text is None:
            return inputs
        elif audio is None:
            return encodings
        else:
            inputs["labels"] = encodings["input_ids"]
            return inputs

    def pad(self, *args, **kwargs):
        """
        When used in normal mode, this method forwards all its arguments to Wav2Vec2FeatureExtractor's
        [`~Wav2Vec2FeatureExtractor.pad`] and returns its output. If used in the context
        [`~Wav2Vec2Processor.as_target_processor`] this method forwards all its arguments to PreTrainedTokenizer's
        [`~PreTrainedTokenizer.pad`]. Please refer to the docstring of the above two methods for more information.
        """
        # For backward compatibility
        if self._in_target_context_manager:
            return self.current_processor.pad(*args, **kwargs)

        input_features = kwargs.pop("input_features", None)
        labels = kwargs.pop("labels", None)
        if len(args) > 0:
            input_features = args[0]
            args = args[1:]

        if input_features is not None:
            input_features = self.feature_extractor.pad(input_features, *args, **kwargs)
        if labels is not None:
            labels = self.tokenizer.pad(labels, **kwargs)

        if labels is None:
            return input_features
        elif input_features is None:
            return labels
        else:
            input_features["labels"] = labels["input_ids"]
            return input_features

    def batch_decode(self, *args, **kwargs):
        """
        This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
        refer to the docstring of this method for more information.
        """
        return self.tokenizer.batch_decode(*args, **kwargs)

    def decode(self, *args, **kwargs):
        """
        This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer
        to the docstring of this method for more information.
        """
        return self.tokenizer.decode(*args, **kwargs)

    @contextmanager
    def as_target_processor(self):
        """
        Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning
        Wav2Vec2.
        """
        warnings.warn(
            "`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your "
            "labels by using the argument `text` of the regular `__call__` method (either in the same call as "
            "your audio inputs, or in a separate call."
        )
        self._in_target_context_manager = True
        self.current_processor = self.tokenizer
        yield
        self.current_processor = self.feature_extractor
        self._in_target_context_manager = False

mindnlp.transformers.models.wav2vec2.processing_wav2vec2.Wav2Vec2Processor.__call__(*args, **kwargs)

When used in normal mode, this method forwards all its arguments to Wav2Vec2FeatureExtractor's [~Wav2Vec2FeatureExtractor.__call__] and returns its output. If used in the context [~Wav2Vec2Processor.as_target_processor] this method forwards all its arguments to PreTrainedTokenizer's [~PreTrainedTokenizer.__call__]. Please refer to the docstring of the above two methods for more information.

Source code in mindnlp/transformers/models/wav2vec2/processing_wav2vec2.py

def __call__(self, *args, **kwargs):
    """
    When used in normal mode, this method forwards all its arguments to Wav2Vec2FeatureExtractor's
    [`~Wav2Vec2FeatureExtractor.__call__`] and returns its output. If used in the context
    [`~Wav2Vec2Processor.as_target_processor`] this method forwards all its arguments to PreTrainedTokenizer's
    [`~PreTrainedTokenizer.__call__`]. Please refer to the docstring of the above two methods for more information.
    """
    # For backward compatibility
    if self._in_target_context_manager:
        return self.current_processor(*args, **kwargs)

    if "raw_speech" in kwargs:
        warnings.warn("Using `raw_speech` as a keyword argument is deprecated. Use `audio` instead.")
        audio = kwargs.pop("raw_speech")
    else:
        audio = kwargs.pop("audio", None)
    sampling_rate = kwargs.pop("sampling_rate", None)
    text = kwargs.pop("text", None)
    if len(args) > 0:
        audio = args[0]
        args = args[1:]

    if audio is None and text is None:
        raise ValueError("You need to specify either an `audio` or `text` input to process.")

    if audio is not None:
        inputs = self.feature_extractor(audio, *args, sampling_rate=sampling_rate, **kwargs)
    if text is not None:
        encodings = self.tokenizer(text, **kwargs)

    if text is None:
        return inputs
    elif audio is None:
        return encodings
    else:
        inputs["labels"] = encodings["input_ids"]
        return inputs

mindnlp.transformers.models.wav2vec2.processing_wav2vec2.Wav2Vec2Processor.__init__(feature_extractor, tokenizer)

Initializes a new instance of the Wav2Vec2Processor class.

PARAMETER	DESCRIPTION
self	The current instance of the Wav2Vec2Processor class. TYPE: Wav2Vec2Processor
feature_extractor	The feature extractor used for processing input data. It should be an instance of a feature extraction class. TYPE: object
tokenizer	The tokenizer used for tokenizing input data. It should be an instance of a tokenizer class. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/processing_wav2vec2.py

def __init__(self, feature_extractor, tokenizer):
    """
    Initializes a new instance of the Wav2Vec2Processor class.

    Args:
        self (Wav2Vec2Processor): The current instance of the Wav2Vec2Processor class.
        feature_extractor (object): The feature extractor used for processing input data.
            It should be an instance of a feature extraction class.
        tokenizer (object): The tokenizer used for tokenizing input data.
            It should be an instance of a tokenizer class.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(feature_extractor, tokenizer)
    self.current_processor = self.feature_extractor
    self._in_target_context_manager = False

mindnlp.transformers.models.wav2vec2.processing_wav2vec2.Wav2Vec2Processor.as_target_processor()

Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning Wav2Vec2.

Source code in mindnlp/transformers/models/wav2vec2/processing_wav2vec2.py

@contextmanager
def as_target_processor(self):
    """
    Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning
    Wav2Vec2.
    """
    warnings.warn(
        "`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your "
        "labels by using the argument `text` of the regular `__call__` method (either in the same call as "
        "your audio inputs, or in a separate call."
    )
    self._in_target_context_manager = True
    self.current_processor = self.tokenizer
    yield
    self.current_processor = self.feature_extractor
    self._in_target_context_manager = False

mindnlp.transformers.models.wav2vec2.processing_wav2vec2.Wav2Vec2Processor.batch_decode(*args, **kwargs)

This method forwards all its arguments to PreTrainedTokenizer's [~PreTrainedTokenizer.batch_decode]. Please refer to the docstring of this method for more information.

Source code in mindnlp/transformers/models/wav2vec2/processing_wav2vec2.py

def batch_decode(self, *args, **kwargs):
    """
    This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
    refer to the docstring of this method for more information.
    """
    return self.tokenizer.batch_decode(*args, **kwargs)

mindnlp.transformers.models.wav2vec2.processing_wav2vec2.Wav2Vec2Processor.decode(*args, **kwargs)

This method forwards all its arguments to PreTrainedTokenizer's [~PreTrainedTokenizer.decode]. Please refer to the docstring of this method for more information.

Source code in mindnlp/transformers/models/wav2vec2/processing_wav2vec2.py

def decode(self, *args, **kwargs):
    """
    This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer
    to the docstring of this method for more information.
    """
    return self.tokenizer.decode(*args, **kwargs)

mindnlp.transformers.models.wav2vec2.processing_wav2vec2.Wav2Vec2Processor.from_pretrained(pretrained_model_name_or_path, **kwargs) classmethod

This method creates an instance of the Wav2Vec2Processor class from a pre-trained model.

PARAMETER	DESCRIPTION
cls	The class itself. TYPE: class
pretrained_model_name_or_path	The name or path of the pre-trained model to load. TYPE: str

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
OSError	If an OSError occurs during the loading process.
FutureWarning	If the tokenizer is being loaded from a config that does not include a tokenizer_class attribute, a FutureWarning is issued. It advises adding a 'tokenizer_class': 'Wav2Vec2CTCTokenizer' attribute to either the config.json or tokenizer_config.json file to suppress the warning.

Source code in mindnlp/transformers/models/wav2vec2/processing_wav2vec2.py

@classmethod
def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
    """
    This method creates an instance of the Wav2Vec2Processor class from a pre-trained model.

    Args:
        cls (class): The class itself.
        pretrained_model_name_or_path (str): The name or path of the pre-trained model to load.

    Returns:
        None.

    Raises:
        OSError: If an OSError occurs during the loading process.
        FutureWarning: If the tokenizer is being loaded from a config that does not include a `tokenizer_class`
            attribute, a FutureWarning is issued. It advises adding a `'tokenizer_class': 'Wav2Vec2CTCTokenizer'`
            attribute to either the `config.json` or `tokenizer_config.json` file to suppress the warning.
    """
    try:
        return super().from_pretrained(pretrained_model_name_or_path, **kwargs)
    except OSError:
        warnings.warn(
            f"Loading a tokenizer inside {cls.__name__} from a config that does not"
            " include a `tokenizer_class` attribute is deprecated and will be "
            "removed in v5. Please add `'tokenizer_class': 'Wav2Vec2CTCTokenizer'`"
            " attribute to either your `config.json` or `tokenizer_config.json` "
            "file to suppress this warning: ",
            FutureWarning,
        )

        feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(pretrained_model_name_or_path, **kwargs)
        tokenizer = Wav2Vec2CTCTokenizer.from_pretrained(pretrained_model_name_or_path, **kwargs)

        return cls(feature_extractor=feature_extractor, tokenizer=tokenizer)

mindnlp.transformers.models.wav2vec2.processing_wav2vec2.Wav2Vec2Processor.pad(*args, **kwargs)

When used in normal mode, this method forwards all its arguments to Wav2Vec2FeatureExtractor's [~Wav2Vec2FeatureExtractor.pad] and returns its output. If used in the context [~Wav2Vec2Processor.as_target_processor] this method forwards all its arguments to PreTrainedTokenizer's [~PreTrainedTokenizer.pad]. Please refer to the docstring of the above two methods for more information.

Source code in mindnlp/transformers/models/wav2vec2/processing_wav2vec2.py

def pad(self, *args, **kwargs):
    """
    When used in normal mode, this method forwards all its arguments to Wav2Vec2FeatureExtractor's
    [`~Wav2Vec2FeatureExtractor.pad`] and returns its output. If used in the context
    [`~Wav2Vec2Processor.as_target_processor`] this method forwards all its arguments to PreTrainedTokenizer's
    [`~PreTrainedTokenizer.pad`]. Please refer to the docstring of the above two methods for more information.
    """
    # For backward compatibility
    if self._in_target_context_manager:
        return self.current_processor.pad(*args, **kwargs)

    input_features = kwargs.pop("input_features", None)
    labels = kwargs.pop("labels", None)
    if len(args) > 0:
        input_features = args[0]
        args = args[1:]

    if input_features is not None:
        input_features = self.feature_extractor.pad(input_features, *args, **kwargs)
    if labels is not None:
        labels = self.tokenizer.pad(labels, **kwargs)

    if labels is None:
        return input_features
    elif input_features is None:
        return labels
    else:
        input_features["labels"] = labels["input_ids"]
        return input_features

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2

Tokenization class for Wav2Vec2.

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer

Bases: PreTrainedTokenizer

Constructs a Wav2Vec2CTC tokenizer.

This tokenizer inherits from [PreTrainedTokenizer] which contains some of the main methods. Users should refer to the superclass for more information regarding such methods.

PARAMETER	DESCRIPTION
vocab_file	File containing the vocabulary. TYPE: `str`
bos_token	The beginning of sentence token. TYPE: `str`, *optional*, defaults to `"<s>"` DEFAULT: '<s>'
eos_token	The end of sentence token. TYPE: `str`, *optional*, defaults to `"</s>"` DEFAULT: '</s>'
unk_token	The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. TYPE: `str`, *optional*, defaults to `"<unk>"` DEFAULT: '<unk>'
pad_token	The token used for padding, for example when batching sequences of different lengths. TYPE: `str`, *optional*, defaults to `"<pad>"` DEFAULT: '<pad>'
word_delimiter_token	The token used for defining the end of a word. TYPE: `str`, *optional*, defaults to `"|"` DEFAULT: '|'
do_lower_case	Whether or not to accept lowercase input and lowercase the output when decoding. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
target_lang	A target language the tokenizer should set by default. target_lang has to be defined for multi-lingual, nested vocabulary such as facebook/mms-1b-all. TYPE: `str`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

class Wav2Vec2CTCTokenizer(PreTrainedTokenizer):

    """
    Constructs a Wav2Vec2CTC tokenizer.

    This tokenizer inherits from [`PreTrainedTokenizer`] which contains some of the main methods. Users should refer to
    the superclass for more information regarding such methods.

    Args:
        vocab_file (`str`):
            File containing the vocabulary.
        bos_token (`str`, *optional*, defaults to `"<s>"`):
            The beginning of sentence token.
        eos_token (`str`, *optional*, defaults to `"</s>"`):
            The end of sentence token.
        unk_token (`str`, *optional*, defaults to `"<unk>"`):
            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
            token instead.
        pad_token (`str`, *optional*, defaults to `"<pad>"`):
            The token used for padding, for example when batching sequences of different lengths.
        word_delimiter_token (`str`, *optional*, defaults to `"|"`):
            The token used for defining the end of a word.
        do_lower_case (`bool`, *optional*, defaults to `False`):
            Whether or not to accept lowercase input and lowercase the output when decoding.
        target_lang (`str`, *optional*):
            A target language the tokenizer should set by default. `target_lang` has to be defined for multi-lingual,
            nested vocabulary such as [facebook/mms-1b-all](https://hf-mirror.com/facebook/mms-1b-all).

        **kwargs
            Additional keyword arguments passed along to [`PreTrainedTokenizer`]
    """
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    model_input_names = ["input_ids", "attention_mask"]

    def __init__(
        self,
        vocab_file,
        bos_token="<s>",
        eos_token="</s>",
        unk_token="<unk>",
        pad_token="<pad>",
        word_delimiter_token="|",
        replace_word_delimiter_char=" ",
        do_lower_case=False,
        target_lang=None,
        **kwargs,
    ):
        """
        Initializes a new instance of the Wav2Vec2CTCTokenizer class.

        Args:
            self (Wav2Vec2CTCTokenizer): The instance of the Wav2Vec2CTCTokenizer class.
            vocab_file (str): The path to the vocabulary file.
            bos_token (str, optional): The beginning of sentence token. Default is '<s>'.
            eos_token (str, optional): The end of sentence token. Default is '</s>'.
            unk_token (str, optional): The unknown token. Default is '<unk>'.
            pad_token (str, optional): The padding token. Default is '<pad>'.
            word_delimiter_token (str, optional): The word delimiter token. Default is '|'.
            replace_word_delimiter_char (str, optional): The character used to replace the word delimiter. Default is ' '.
            do_lower_case (bool, optional): Whether to convert all tokens to lowercase. Default is False.
            target_lang (str, optional): The target language for encoding. Default is None.
            **kwargs: Additional keyword arguments.

        Returns:
            None

        Raises:
            None
        """
        self._word_delimiter_token = word_delimiter_token

        self.do_lower_case = do_lower_case
        self.replace_word_delimiter_char = replace_word_delimiter_char
        self.target_lang = target_lang

        with open(vocab_file, encoding="utf-8") as vocab_handle:
            self.vocab = json.load(vocab_handle)

        # if target lang is defined vocab must be a nested dict
        # with each target lang being one vocabulary
        if target_lang is not None:
            self.encoder = self.vocab[target_lang]
        else:
            self.encoder = self.vocab

        self.decoder = {v: k for k, v in self.encoder.items()}

        super().__init__(
            unk_token=unk_token,
            bos_token=bos_token,
            eos_token=eos_token,
            pad_token=pad_token,
            do_lower_case=do_lower_case,
            word_delimiter_token=word_delimiter_token,
            replace_word_delimiter_char=replace_word_delimiter_char,
            target_lang=target_lang,
            **kwargs,
        )

        # make sure that tokens made of several
        # characters are not split at tokenization
        for token in self.encoder.keys():
            if len(token) > 1:
                self.add_tokens(AddedToken(token, rstrip=True, lstrip=True, normalized=False))

    def set_target_lang(self, target_lang: str):
        """
        Set the target language of a nested multi-lingual dictionary
        """
        if self.vocab == self.encoder:
            raise ValueError(f"{self.vocab} is not a multi-lingual, nested tokenizer. Cannot set target language.")

        if target_lang not in self.vocab:
            raise ValueError(f"{target_lang} does not exist. Choose one of {', '.join(self.vocab.keys())}.")

        self.target_lang = target_lang
        self.init_kwargs["target_lang"] = target_lang
        self.encoder = self.vocab[target_lang]
        self.decoder = {v: k for k, v in self.encoder.items()}

        # make sure that tokens made of several
        # characters are not split at tokenization
        for token in self.encoder.keys():
            if len(token) > 1:
                self.add_tokens(AddedToken(token, rstrip=True, lstrip=True, normalized=False))

    @property
    def word_delimiter_token(self) -> str:
        """
        `str`: Word delimiter token. Log an error if used while not having been set.
        """
        if self._word_delimiter_token is None and self.verbose:
            logger.error("Using word_delimiter_token, but it is not set yet.")
            return None
        return str(self._word_delimiter_token)

    @property
    def word_delimiter_token_id(self) -> Optional[int]:
        """
        `Optional[int]`: Id of the word_delimiter_token in the vocabulary. Returns `None` if the token has not been
        set.
        """
        if self._word_delimiter_token is None:
            return None
        return self.convert_tokens_to_ids(self.word_delimiter_token)

    @word_delimiter_token.setter
    def word_delimiter_token(self, value):
        """
        Sets the word delimiter token for the Wav2Vec2CTCTokenizer.

        Args:
            self (Wav2Vec2CTCTokenizer): The instance of the Wav2Vec2CTCTokenizer class.
            value (str): The word delimiter token to be set.

        Returns:
            None.

        Raises:
            None.
        """
        self._word_delimiter_token = value

    @word_delimiter_token_id.setter
    def word_delimiter_token_id(self, value):
        """
        Sets the word delimiter token ID for the Wav2Vec2CTCTokenizer.

        Args:
            self (Wav2Vec2CTCTokenizer): The Wav2Vec2CTCTokenizer instance.
            value (list[int]): A list of integers representing the token IDs for word delimiters.

        Returns:
            None.

        Raises:
            TypeError: If the provided value is not a list of integers.
            ValueError: If the provided value contains invalid token IDs.
        """
        self._word_delimiter_token = self.convert_tokens_to_ids(value)

    @property
    def vocab_size(self) -> int:
        """
        Returns the size of the vocabulary used by the Wav2Vec2CTCTokenizer.

        Args:
            self: An instance of the Wav2Vec2CTCTokenizer class.

        Returns:
            int: The size of the vocabulary, which represents the total number of unique tokens in the decoder.

        Raises:
            None.

        Example:
            ```python
            >>> tokenizer = Wav2Vec2CTCTokenizer()
            >>> tokenizer.vocab_size()
            50000
            ```
        """
        return len(self.decoder)

    def get_vocab(self) -> Dict:
        """
        Returns the vocabulary used by the Wav2Vec2CTCTokenizer.

        Args:
            self (Wav2Vec2CTCTokenizer): An instance of the Wav2Vec2CTCTokenizer class.

        Returns:
            Dict: A dictionary representing the vocabulary used by the tokenizer.
                The keys are integers representing the token IDs, and the values are the corresponding tokens.

        Raises:
            None.

        This method retrieves the vocabulary used by the Wav2Vec2CTCTokenizer instance. The vocabulary is a dictionary
        that combines the encoder and added_tokens_encoder dictionaries. The encoder dictionary maps tokens to unique
        integer IDs, while the added_tokens_encoder dictionary contains additional tokens added by the user.
        The resulting vocabulary dictionary is returned.
        """
        vocab = dict(self.encoder)
        vocab.update(self.added_tokens_encoder)
        return vocab

    def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int:
        """
        Add tokens to the Wav2Vec2CTCTokenizer's vocabulary.

        Args:
            self (Wav2Vec2CTCTokenizer): The instance of the Wav2Vec2CTCTokenizer class.
            new_tokens (Union[List[str], List[AddedToken]]): A list of new tokens to be added to the vocabulary.
                Each token can be either a string or an instance of AddedToken.
            special_tokens (bool, optional): A flag indicating whether the new tokens are special tokens.
                Defaults to False.

        Returns:
            int: The number of tokens added to the vocabulary.

        Raises:
            None

        This method takes a list of new tokens and adds them to the vocabulary of the Wav2Vec2CTCTokenizer.
        The new tokens can be either strings or instances of AddedToken. If a token is a string, a default AddedToken
        object will be created with the token as its text and the following default values for its attributes:
        rstrip=False, lstrip=False, normalized=False. If a token is already an instance of AddedToken,
        it will be added as is. The method then calls the super()._add_tokens() method to add the tokens to the
        vocabulary. The special_tokens flag can be used to indicate whether the new tokens are special tokens.
        """
        # Overwritten to never strip!
        to_add = []
        for token in new_tokens:
            if isinstance(token, str):
                to_add.append(AddedToken(token, rstrip=False, lstrip=False, normalized=False))
            else:
                to_add.append(token)

        return super()._add_tokens(to_add, special_tokens)

    def _tokenize(self, text, **kwargs):
        """
        Converts a string into a sequence of tokens (string), using the tokenizer.
        """
        if self.do_lower_case:
            text = text.upper()

        return list(text.replace(" ", self.word_delimiter_token))

    def _convert_token_to_id(self, token: str) -> int:
        """Converts a token (str) in an index (integer) using the vocab."""
        return self.encoder.get(token, self.encoder.get(self.unk_token))

    def _convert_id_to_token(self, index: int) -> str:
        """Converts an index (integer) in a token (str) using the vocab."""
        result = self.decoder.get(index, self.unk_token)
        return result

    def convert_tokens_to_string(
        self,
        tokens: List[str],
        group_tokens: bool = True,
        spaces_between_special_tokens: bool = False,
        output_char_offsets: bool = False,
        output_word_offsets: bool = False,
    ) -> Dict[str, Union[str, float]]:
        """
        Converts a connectionist-temporal-classification (CTC) output tokens into a single string.
        """
        if len(tokens) == 0:
            return {"text": "", "char_offsets": [], "word_offsets": []}
        # group same tokens into non-repeating tokens in CTC style decoding
        if group_tokens:
            chars, char_repetitions = zip(*((token, len(list(group_iter))) for token, group_iter in groupby(tokens)))
        else:
            chars = tokens
            char_repetitions = len(tokens) * [1]

        # filter self.pad_token which is used as CTC-blank token
        processed_chars = list(filter(lambda char: char != self.pad_token, chars))

        # replace delimiter token
        processed_chars = [
            self.replace_word_delimiter_char if char == self.word_delimiter_token else char for char in processed_chars
        ]

        # retrieve offsets
        char_offsets = word_offsets = None
        if output_char_offsets or output_word_offsets:
            char_offsets = self._compute_offsets(char_repetitions, chars, self.pad_token)

            if len(char_offsets) != len(processed_chars):
                raise ValueError(
                    f"`char_offsets`: {char_offsets} and `processed_tokens`: {processed_chars}"
                    " have to be of the same length, but are: "
                    f"`len(offsets)`: {len(char_offsets)} and `len(processed_tokens)`:"
                    f" {len(processed_chars)}"
                )

            # set tokens to correct processed token
            for i, char in enumerate(processed_chars):
                char_offsets[i]["char"] = char

            # retrieve word offsets from character offsets
            word_offsets = None
            if output_word_offsets:
                word_offsets = self._get_word_offsets(char_offsets, self.replace_word_delimiter_char)

            # don't output chars if not set to True
            if not output_char_offsets:
                char_offsets = None

        # join to string
        join_char = " " if spaces_between_special_tokens else ""
        string = join_char.join(processed_chars).strip()

        if self.do_lower_case:
            string = string.lower()

        return {"text": string, "char_offsets": char_offsets, "word_offsets": word_offsets}

    @staticmethod
    def _compute_offsets(
        char_repetitions: List[int], chars: List[str], ctc_token: int
    ) -> List[Dict[str, Union[str, int]]]:
        """
        Compute offsets for characters based on char repetitions and tokens.

        Args:
            char_repetitions (List[int]): A list of integers representing the number of repetitions for each character.
            chars (List[str]): A list of characters.
            ctc_token (int): The CTC token to be filtered out from the offsets.

        Returns:
            List[Dict[str, Union[str, int]]]: A list of dictionaries where each dictionary contains the character,
                start offset, and end offset.

        Raises:
            None
        """
        end_indices = np.asarray(char_repetitions).cumsum()
        start_indices = np.concatenate(([0], end_indices[:-1]))

        offsets = [
            {"char": t, "start_offset": s, "end_offset": e} for t, s, e in zip(chars, start_indices, end_indices)
        ]

        # filter out CTC token
        offsets = list(filter(lambda offsets: offsets["char"] != ctc_token, offsets))
        return offsets

    @staticmethod
    def _get_word_offsets(
        offsets: Dict[str, Union[str, float]], word_delimiter_char: str = " "
    ) -> Dict[str, Union[str, float]]:
        """
        Method to extract word offsets from a given set of character offsets.

        Args:
            offsets (Dict[str, Union[str, float]]): A dictionary containing character offsets with keys 'char',
                'start_offset', and 'end_offset'. The 'char' key represents the character, 'start_offset' represents
                the start offset, and 'end_offset' represents the end offset.
            word_delimiter_char (str, optional): The character used as a word delimiter. Defaults to a space character.

        Returns:
            Dict[str, Union[str, float]]: A dictionary containing word offsets with keys 'word', 'start_offset',
                and 'end_offset'. The 'word' key represents the extracted word, 'start_offset' represents the start
                offset, and 'end_offset' represents the end offset.

        Raises:
            None
        """
        word_offsets = []

        last_state = "SPACE"
        word = ""
        start_offset = 0
        end_offset = 0
        for offset in offsets:
            char = offset["char"]
            state = "SPACE" if char == word_delimiter_char else "WORD"

            if state == last_state:
                # If we are in the same state as before, we simply repeat what we've done before
                end_offset = offset["end_offset"]
                word += char
            else:
                # Switching state
                if state == "SPACE":
                    # Finishing a word
                    word_offsets.append({"word": word, "start_offset": start_offset, "end_offset": end_offset})
                else:
                    # Starting a new word
                    start_offset = offset["start_offset"]
                    end_offset = offset["end_offset"]
                    word = char

            last_state = state
        if last_state == "WORD":
            word_offsets.append({"word": word, "start_offset": start_offset, "end_offset": end_offset})

        return word_offsets

    def prepare_for_tokenization(self, text, is_split_into_words=False, **kwargs):
        """
        Prepare the input text for tokenization.

        Args:
            self (Wav2Vec2CTCTokenizer): The instance of the Wav2Vec2CTCTokenizer class.
            text (str): The input text to be prepared for tokenization.
            is_split_into_words (bool): A flag indicating whether the input text is already split into words.
                If True, the input text is expected to be split into words;
                otherwise, the input text is treated as a continuous string.
                Defaults to False.

        Returns:
            tuple: A tuple containing the prepared text and optional keyword arguments.

        Raises:
            None
        """
        if is_split_into_words:
            text = " " + text
        return (text, kwargs)

    def _decode(
        self,
        token_ids: List[int],
        skip_special_tokens: bool = False,
        clean_up_tokenization_spaces: bool = None,
        group_tokens: bool = True,
        spaces_between_special_tokens: bool = False,
        output_word_offsets: Optional[bool] = False,
        output_char_offsets: Optional[bool] = False,
    ) -> str:
        """
        special _decode function is needed for Wav2Vec2Tokenizer because added tokens should be treated exactly the
        same as tokens of the base vocabulary and therefore the function `convert_tokens_to_string` has to be called on
        the whole token list and not individually on added tokens
        """
        filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)

        result = []
        for token in filtered_tokens:
            if skip_special_tokens and token in self.all_special_ids:
                continue
            result.append(token)

        string_output = self.convert_tokens_to_string(
            result,
            group_tokens=group_tokens,
            spaces_between_special_tokens=spaces_between_special_tokens,
            output_word_offsets=output_word_offsets,
            output_char_offsets=output_char_offsets,
        )

        text = string_output["text"]

        clean_up_tokenization_spaces = (
            clean_up_tokenization_spaces
            if clean_up_tokenization_spaces is not None
            else self.clean_up_tokenization_spaces
        )
        if clean_up_tokenization_spaces:
            text = self.clean_up_tokenization(text)

        if output_word_offsets or output_char_offsets:
            return Wav2Vec2CTCTokenizerOutput(
                text=text,
                char_offsets=string_output["char_offsets"],
                word_offsets=string_output["word_offsets"],
            )
        else:
            return text

    # overwritten from `tokenization_utils_base.py` because tokenizer can output
    # `ModelOutput` which should not be a list for batched output and
    # because we need docs for `output_char_offsets` here
    def batch_decode(
        self,
        sequences: Union[List[int], List[List[int]], "np.ndarray", "Tensor"],
        skip_special_tokens: bool = False,
        clean_up_tokenization_spaces: bool = None,
        output_char_offsets: bool = False,
        output_word_offsets: bool = False,
        **kwargs,
    ) -> List[str]:
        """
        Convert a list of lists of token ids into a list of strings by calling decode.

        Args:
            sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`):
                List of tokenized input ids. Can be obtained using the `__call__` method.
            skip_special_tokens (`bool`, *optional*, defaults to `False`):
                Whether or not to remove special tokens in the decoding.
            clean_up_tokenization_spaces (`bool`, *optional*):
                Whether or not to clean up the tokenization spaces.
            output_char_offsets (`bool`, *optional*, defaults to `False`):
                Whether or not to output character offsets. Character offsets can be used in combination with the
                sampling rate and model downsampling rate to compute the time-stamps of transcribed characters.

                <Tip>

                Please take a look at the Example of [`~Wav2Vec2CTCTokenizer.decode`] to better understand how to make
                use of `output_char_offsets`. [`~Wav2Vec2CTCTokenizer.batch_decode`] works the same way with batched
                output.

                </Tip>

            output_word_offsets (`bool`, *optional*, defaults to `False`):
                Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate
                and model downsampling rate to compute the time-stamps of transcribed words.

                <Tip>

                Please take a look at the Example of [`~Wav2Vec2CTCTokenizer.decode`] to better understand how to make
                use of `output_word_offsets`. [`~Wav2Vec2CTCTokenizer.batch_decode`] works the same way with batched
                output.

                </Tip>

            kwargs (additional keyword arguments, *optional*):
                Will be passed to the underlying model specific decode method.

        Returns:
            `List[str]` or [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`]: The list of decoded
                sentences. Will be a [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`] when
                `output_char_offsets == True` or `output_word_offsets == True`.
        """
        batch_decoded = [
            self.decode(
                seq,
                skip_special_tokens=skip_special_tokens,
                clean_up_tokenization_spaces=clean_up_tokenization_spaces,
                output_char_offsets=output_char_offsets,
                output_word_offsets=output_word_offsets,
                **kwargs,
            )
            for seq in sequences
        ]
        if output_char_offsets or output_word_offsets:
            # transform list of dicts to dict of lists
            return Wav2Vec2CTCTokenizerOutput({k: [d[k] for d in batch_decoded] for k in batch_decoded[0]})

        return batch_decoded

    # overwritten from `tokenization_utils_base.py` because we need docs for `output_char_offsets`
    # and `output_word_offsets` here
    def decode(
        self,
        token_ids: Union[int, List[int], "np.ndarray", "Tensor"],
        skip_special_tokens: bool = False,
        clean_up_tokenization_spaces: bool = None,
        output_char_offsets: bool = False,
        output_word_offsets: bool = False,
        **kwargs,
    ) -> str:
        """
        Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special
        tokens and clean up tokenization spaces.

        Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.

        Args:
            token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`):
                List of tokenized input ids. Can be obtained using the `__call__` method.
            skip_special_tokens (`bool`, *optional*, defaults to `False`):
                Whether or not to remove special tokens in the decoding.
            clean_up_tokenization_spaces (`bool`, *optional*):
                Whether or not to clean up the tokenization spaces.
            output_char_offsets (`bool`, *optional*, defaults to `False`):
                Whether or not to output character offsets. Character offsets can be used in combination with the
                sampling rate and model downsampling rate to compute the time-stamps of transcribed characters.

                <Tip>

                Please take a look at the example below to better understand how to make use of `output_char_offsets`.

                </Tip>

            output_word_offsets (`bool`, *optional*, defaults to `False`):
                Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate
                and model downsampling rate to compute the time-stamps of transcribed words.

                <Tip>

                Please take a look at the example below to better understand how to make use of `output_word_offsets`.

                </Tip>

            kwargs (additional keyword arguments, *optional*):
                Will be passed to the underlying model specific decode method.

        Returns:
            `str` or [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`]: The list of decoded
                sentences. Will be a [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`] when
                `output_char_offsets == True` or `output_word_offsets == True`.

        Example:
            ```python
            >>> # Let's see how to retrieve time steps for a model
            >>> from transformers import AutoTokenizer, AutoFeatureExtractor, AutoModelForCTC
            >>> from datasets import load_dataset
            >>> import datasets
            >>> import torch
            ...
            >>> # import model, feature extractor, tokenizer
            >>> model = AutoModelForCTC.from_pretrained("facebook/wav2vec2-base-960h")
            >>> tokenizer = AutoTokenizer.from_pretrained("facebook/wav2vec2-base-960h")
            >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")
            ...
            >>> # load first sample of English common_voice
            >>> dataset = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="train", streaming=True)
            >>> dataset = dataset.cast_column("audio", datasets.Audio(sampling_rate=16_000))
            >>> dataset_iter = iter(dataset)
            >>> sample = next(dataset_iter)
            ...
            >>> # forward sample through model to get greedily predicted transcription ids
            >>> input_values = feature_extractor(sample["audio"]["array"], return_tensors="pt").input_values
            >>> logits = model(input_values).logits[0]
            >>> pred_ids = torch.argmax(logits, axis=-1)
            ...
            >>> # retrieve word stamps (analogous commands for `output_char_offsets`)
            >>> outputs = tokenizer.decode(pred_ids, output_word_offsets=True)
            >>> # compute `time_offset` in seconds as product of downsampling ratio and sampling_rate
            >>> time_offset = model.config.inputs_to_logits_ratio / feature_extractor.sampling_rate
            ...
            >>> word_offsets = [
            ...     {
            ...         "word": d["word"],
            ...         "start_time": round(d["start_offset"] * time_offset, 2),
            ...         "end_time": round(d["end_offset"] * time_offset, 2),
            ...     }
            ...     for d in outputs.word_offsets
            ... ]
            >>> # compare word offsets with audio `en_train_0/common_voice_en_19121553.mp3` online on the dataset viewer:
            >>> # https://hf-mirror.com/datasets/mozilla-foundation/common_voice_11_0/viewer/en
            >>> word_offsets[:3]
            [{'word': 'THE', 'start_time': 0.7, 'end_time': 0.78}, {'word': 'TRICK', 'start_time': 0.88, 'end_time': 1.08}, {'word': 'APPEARS', 'start_time': 1.2, 'end_time': 1.64}]
            ```
        """
        # Convert inputs to python lists
        token_ids = to_py_obj(token_ids)

        return self._decode(
            token_ids=token_ids,
            skip_special_tokens=skip_special_tokens,
            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
            output_char_offsets=output_char_offsets,
            output_word_offsets=output_word_offsets,
            **kwargs,
        )

    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        """
        Save the vocabulary to a specified directory.

        Args:
            self: The instance of the Wav2Vec2CTCTokenizer class.
            save_directory (str): The directory where the vocabulary will be saved.
            filename_prefix (Optional[str]): An optional prefix to be added to the filename. Defaults to None.

        Returns:
            Tuple[str]: A tuple containing the file path of the saved vocabulary.

        Raises:
            OSError: If the save_directory is not a valid directory.
        """
        if not os.path.isdir(save_directory):
            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
            return
        vocab_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
        )

        with open(vocab_file, "w", encoding="utf-8") as f:
            f.write(json.dumps(self.vocab, indent=2, sort_keys=True, ensure_ascii=False) + "\n")

        return (vocab_file,)

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.vocab_size: int property

Returns the size of the vocabulary used by the Wav2Vec2CTCTokenizer.

PARAMETER	DESCRIPTION
self	An instance of the Wav2Vec2CTCTokenizer class.

RETURNS	DESCRIPTION
int	The size of the vocabulary, which represents the total number of unique tokens in the decoder. TYPE: int

Example

>>> tokenizer = Wav2Vec2CTCTokenizer()
>>> tokenizer.vocab_size()
50000

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.word_delimiter_token: str property writable

str: Word delimiter token. Log an error if used while not having been set.

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.word_delimiter_token_id: Optional[int] property writable

Optional[int]: Id of the word_delimiter_token in the vocabulary. Returns None if the token has not been set.

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.__init__(vocab_file, bos_token='<s>', eos_token='</s>', unk_token='<unk>', pad_token='<pad>', word_delimiter_token='|', replace_word_delimiter_char=' ', do_lower_case=False, target_lang=None, **kwargs)

Initializes a new instance of the Wav2Vec2CTCTokenizer class.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2CTCTokenizer class. TYPE: Wav2Vec2CTCTokenizer
vocab_file	The path to the vocabulary file. TYPE: str
bos_token	The beginning of sentence token. Default is ''. TYPE: str DEFAULT: '<s>'
eos_token	The end of sentence token. Default is ''. TYPE: str DEFAULT: '</s>'
unk_token	The unknown token. Default is ''. TYPE: str DEFAULT: '<unk>'
pad_token	The padding token. Default is ''. TYPE: str DEFAULT: '<pad>'
word_delimiter_token	The word delimiter token. Default is '|'. TYPE: str DEFAULT: '|'
replace_word_delimiter_char	The character used to replace the word delimiter. Default is ' '. TYPE: str DEFAULT: ' '
do_lower_case	Whether to convert all tokens to lowercase. Default is False. TYPE: bool DEFAULT: False
target_lang	The target language for encoding. Default is None. TYPE: str DEFAULT: None
**kwargs	Additional keyword arguments. DEFAULT: {}

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def __init__(
    self,
    vocab_file,
    bos_token="<s>",
    eos_token="</s>",
    unk_token="<unk>",
    pad_token="<pad>",
    word_delimiter_token="|",
    replace_word_delimiter_char=" ",
    do_lower_case=False,
    target_lang=None,
    **kwargs,
):
    """
    Initializes a new instance of the Wav2Vec2CTCTokenizer class.

    Args:
        self (Wav2Vec2CTCTokenizer): The instance of the Wav2Vec2CTCTokenizer class.
        vocab_file (str): The path to the vocabulary file.
        bos_token (str, optional): The beginning of sentence token. Default is '<s>'.
        eos_token (str, optional): The end of sentence token. Default is '</s>'.
        unk_token (str, optional): The unknown token. Default is '<unk>'.
        pad_token (str, optional): The padding token. Default is '<pad>'.
        word_delimiter_token (str, optional): The word delimiter token. Default is '|'.
        replace_word_delimiter_char (str, optional): The character used to replace the word delimiter. Default is ' '.
        do_lower_case (bool, optional): Whether to convert all tokens to lowercase. Default is False.
        target_lang (str, optional): The target language for encoding. Default is None.
        **kwargs: Additional keyword arguments.

    Returns:
        None

    Raises:
        None
    """
    self._word_delimiter_token = word_delimiter_token

    self.do_lower_case = do_lower_case
    self.replace_word_delimiter_char = replace_word_delimiter_char
    self.target_lang = target_lang

    with open(vocab_file, encoding="utf-8") as vocab_handle:
        self.vocab = json.load(vocab_handle)

    # if target lang is defined vocab must be a nested dict
    # with each target lang being one vocabulary
    if target_lang is not None:
        self.encoder = self.vocab[target_lang]
    else:
        self.encoder = self.vocab

    self.decoder = {v: k for k, v in self.encoder.items()}

    super().__init__(
        unk_token=unk_token,
        bos_token=bos_token,
        eos_token=eos_token,
        pad_token=pad_token,
        do_lower_case=do_lower_case,
        word_delimiter_token=word_delimiter_token,
        replace_word_delimiter_char=replace_word_delimiter_char,
        target_lang=target_lang,
        **kwargs,
    )

    # make sure that tokens made of several
    # characters are not split at tokenization
    for token in self.encoder.keys():
        if len(token) > 1:
            self.add_tokens(AddedToken(token, rstrip=True, lstrip=True, normalized=False))

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.batch_decode(sequences, skip_special_tokens=False, clean_up_tokenization_spaces=None, output_char_offsets=False, output_word_offsets=False, **kwargs)

Convert a list of lists of token ids into a list of strings by calling decode.

PARAMETER	DESCRIPTION
sequences	List of tokenized input ids. Can be obtained using the __call__ method. TYPE: `Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`
skip_special_tokens	Whether or not to remove special tokens in the decoding. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
clean_up_tokenization_spaces	Whether or not to clean up the tokenization spaces. TYPE: `bool`, *optional* DEFAULT: None
output_char_offsets	Whether or not to output character offsets. Character offsets can be used in combination with the sampling rate and model downsampling rate to compute the time-stamps of transcribed characters. Please take a look at the Example of [~Wav2Vec2CTCTokenizer.decode] to better understand how to make use of output_char_offsets. [~Wav2Vec2CTCTokenizer.batch_decode] works the same way with batched output. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
output_word_offsets	Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate and model downsampling rate to compute the time-stamps of transcribed words. Please take a look at the Example of [~Wav2Vec2CTCTokenizer.decode] to better understand how to make use of output_word_offsets. [~Wav2Vec2CTCTokenizer.batch_decode] works the same way with batched output. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
kwargs	Will be passed to the underlying model specific decode method. TYPE: additional keyword arguments, *optional* DEFAULT: {}

RETURNS	DESCRIPTION
List[str]	List[str] or [~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput]: The list of decoded sentences. Will be a [~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput] when output_char_offsets == True or output_word_offsets == True.

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def batch_decode(
    self,
    sequences: Union[List[int], List[List[int]], "np.ndarray", "Tensor"],
    skip_special_tokens: bool = False,
    clean_up_tokenization_spaces: bool = None,
    output_char_offsets: bool = False,
    output_word_offsets: bool = False,
    **kwargs,
) -> List[str]:
    """
    Convert a list of lists of token ids into a list of strings by calling decode.

    Args:
        sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`):
            List of tokenized input ids. Can be obtained using the `__call__` method.
        skip_special_tokens (`bool`, *optional*, defaults to `False`):
            Whether or not to remove special tokens in the decoding.
        clean_up_tokenization_spaces (`bool`, *optional*):
            Whether or not to clean up the tokenization spaces.
        output_char_offsets (`bool`, *optional*, defaults to `False`):
            Whether or not to output character offsets. Character offsets can be used in combination with the
            sampling rate and model downsampling rate to compute the time-stamps of transcribed characters.

            <Tip>

            Please take a look at the Example of [`~Wav2Vec2CTCTokenizer.decode`] to better understand how to make
            use of `output_char_offsets`. [`~Wav2Vec2CTCTokenizer.batch_decode`] works the same way with batched
            output.

            </Tip>

        output_word_offsets (`bool`, *optional*, defaults to `False`):
            Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate
            and model downsampling rate to compute the time-stamps of transcribed words.

            <Tip>

            Please take a look at the Example of [`~Wav2Vec2CTCTokenizer.decode`] to better understand how to make
            use of `output_word_offsets`. [`~Wav2Vec2CTCTokenizer.batch_decode`] works the same way with batched
            output.

            </Tip>

        kwargs (additional keyword arguments, *optional*):
            Will be passed to the underlying model specific decode method.

    Returns:
        `List[str]` or [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`]: The list of decoded
            sentences. Will be a [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`] when
            `output_char_offsets == True` or `output_word_offsets == True`.
    """
    batch_decoded = [
        self.decode(
            seq,
            skip_special_tokens=skip_special_tokens,
            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
            output_char_offsets=output_char_offsets,
            output_word_offsets=output_word_offsets,
            **kwargs,
        )
        for seq in sequences
    ]
    if output_char_offsets or output_word_offsets:
        # transform list of dicts to dict of lists
        return Wav2Vec2CTCTokenizerOutput({k: [d[k] for d in batch_decoded] for k in batch_decoded[0]})

    return batch_decoded

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.convert_tokens_to_string(tokens, group_tokens=True, spaces_between_special_tokens=False, output_char_offsets=False, output_word_offsets=False)

Converts a connectionist-temporal-classification (CTC) output tokens into a single string.

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def convert_tokens_to_string(
    self,
    tokens: List[str],
    group_tokens: bool = True,
    spaces_between_special_tokens: bool = False,
    output_char_offsets: bool = False,
    output_word_offsets: bool = False,
) -> Dict[str, Union[str, float]]:
    """
    Converts a connectionist-temporal-classification (CTC) output tokens into a single string.
    """
    if len(tokens) == 0:
        return {"text": "", "char_offsets": [], "word_offsets": []}
    # group same tokens into non-repeating tokens in CTC style decoding
    if group_tokens:
        chars, char_repetitions = zip(*((token, len(list(group_iter))) for token, group_iter in groupby(tokens)))
    else:
        chars = tokens
        char_repetitions = len(tokens) * [1]

    # filter self.pad_token which is used as CTC-blank token
    processed_chars = list(filter(lambda char: char != self.pad_token, chars))

    # replace delimiter token
    processed_chars = [
        self.replace_word_delimiter_char if char == self.word_delimiter_token else char for char in processed_chars
    ]

    # retrieve offsets
    char_offsets = word_offsets = None
    if output_char_offsets or output_word_offsets:
        char_offsets = self._compute_offsets(char_repetitions, chars, self.pad_token)

        if len(char_offsets) != len(processed_chars):
            raise ValueError(
                f"`char_offsets`: {char_offsets} and `processed_tokens`: {processed_chars}"
                " have to be of the same length, but are: "
                f"`len(offsets)`: {len(char_offsets)} and `len(processed_tokens)`:"
                f" {len(processed_chars)}"
            )

        # set tokens to correct processed token
        for i, char in enumerate(processed_chars):
            char_offsets[i]["char"] = char

        # retrieve word offsets from character offsets
        word_offsets = None
        if output_word_offsets:
            word_offsets = self._get_word_offsets(char_offsets, self.replace_word_delimiter_char)

        # don't output chars if not set to True
        if not output_char_offsets:
            char_offsets = None

    # join to string
    join_char = " " if spaces_between_special_tokens else ""
    string = join_char.join(processed_chars).strip()

    if self.do_lower_case:
        string = string.lower()

    return {"text": string, "char_offsets": char_offsets, "word_offsets": word_offsets}

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.decode(token_ids, skip_special_tokens=False, clean_up_tokenization_spaces=None, output_char_offsets=False, output_word_offsets=False, **kwargs)

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces.

Similar to doing self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids)).

PARAMETER	DESCRIPTION
token_ids	List of tokenized input ids. Can be obtained using the __call__ method. TYPE: `Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`
skip_special_tokens	Whether or not to remove special tokens in the decoding. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
clean_up_tokenization_spaces	Whether or not to clean up the tokenization spaces. TYPE: `bool`, *optional* DEFAULT: None
output_char_offsets	Whether or not to output character offsets. Character offsets can be used in combination with the sampling rate and model downsampling rate to compute the time-stamps of transcribed characters. Please take a look at the example below to better understand how to make use of output_char_offsets. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
output_word_offsets	Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate and model downsampling rate to compute the time-stamps of transcribed words. Please take a look at the example below to better understand how to make use of output_word_offsets. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
kwargs	Will be passed to the underlying model specific decode method. TYPE: additional keyword arguments, *optional* DEFAULT: {}

RETURNS	DESCRIPTION
str	str or [~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput]: The list of decoded sentences. Will be a [~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput] when output_char_offsets == True or output_word_offsets == True.

Example

>>> # Let's see how to retrieve time steps for a model
>>> from transformers import AutoTokenizer, AutoFeatureExtractor, AutoModelForCTC
>>> from datasets import load_dataset
>>> import datasets
>>> import torch
...
>>> # import model, feature extractor, tokenizer
>>> model = AutoModelForCTC.from_pretrained("facebook/wav2vec2-base-960h")
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/wav2vec2-base-960h")
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")
...
>>> # load first sample of English common_voice
>>> dataset = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="train", streaming=True)
>>> dataset = dataset.cast_column("audio", datasets.Audio(sampling_rate=16_000))
>>> dataset_iter = iter(dataset)
>>> sample = next(dataset_iter)
...
>>> # forward sample through model to get greedily predicted transcription ids
>>> input_values = feature_extractor(sample["audio"]["array"], return_tensors="pt").input_values
>>> logits = model(input_values).logits[0]
>>> pred_ids = torch.argmax(logits, axis=-1)
...
>>> # retrieve word stamps (analogous commands for `output_char_offsets`)
>>> outputs = tokenizer.decode(pred_ids, output_word_offsets=True)
>>> # compute `time_offset` in seconds as product of downsampling ratio and sampling_rate
>>> time_offset = model.config.inputs_to_logits_ratio / feature_extractor.sampling_rate
...
>>> word_offsets = [
...     {
...         "word": d["word"],
...         "start_time": round(d["start_offset"] * time_offset, 2),
...         "end_time": round(d["end_offset"] * time_offset, 2),
...     }
...     for d in outputs.word_offsets
... ]
>>> # compare word offsets with audio `en_train_0/common_voice_en_19121553.mp3` online on the dataset viewer:
>>> # https://hf-mirror.com/datasets/mozilla-foundation/common_voice_11_0/viewer/en
>>> word_offsets[:3]
[{'word': 'THE', 'start_time': 0.7, 'end_time': 0.78}, {'word': 'TRICK', 'start_time': 0.88, 'end_time': 1.08}, {'word': 'APPEARS', 'start_time': 1.2, 'end_time': 1.64}]

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def decode(
    self,
    token_ids: Union[int, List[int], "np.ndarray", "Tensor"],
    skip_special_tokens: bool = False,
    clean_up_tokenization_spaces: bool = None,
    output_char_offsets: bool = False,
    output_word_offsets: bool = False,
    **kwargs,
) -> str:
    """
    Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special
    tokens and clean up tokenization spaces.

    Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.

    Args:
        token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`):
            List of tokenized input ids. Can be obtained using the `__call__` method.
        skip_special_tokens (`bool`, *optional*, defaults to `False`):
            Whether or not to remove special tokens in the decoding.
        clean_up_tokenization_spaces (`bool`, *optional*):
            Whether or not to clean up the tokenization spaces.
        output_char_offsets (`bool`, *optional*, defaults to `False`):
            Whether or not to output character offsets. Character offsets can be used in combination with the
            sampling rate and model downsampling rate to compute the time-stamps of transcribed characters.

            <Tip>

            Please take a look at the example below to better understand how to make use of `output_char_offsets`.

            </Tip>

        output_word_offsets (`bool`, *optional*, defaults to `False`):
            Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate
            and model downsampling rate to compute the time-stamps of transcribed words.

            <Tip>

            Please take a look at the example below to better understand how to make use of `output_word_offsets`.

            </Tip>

        kwargs (additional keyword arguments, *optional*):
            Will be passed to the underlying model specific decode method.

    Returns:
        `str` or [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`]: The list of decoded
            sentences. Will be a [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`] when
            `output_char_offsets == True` or `output_word_offsets == True`.

    Example:
        ```python
        >>> # Let's see how to retrieve time steps for a model
        >>> from transformers import AutoTokenizer, AutoFeatureExtractor, AutoModelForCTC
        >>> from datasets import load_dataset
        >>> import datasets
        >>> import torch
        ...
        >>> # import model, feature extractor, tokenizer
        >>> model = AutoModelForCTC.from_pretrained("facebook/wav2vec2-base-960h")
        >>> tokenizer = AutoTokenizer.from_pretrained("facebook/wav2vec2-base-960h")
        >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")
        ...
        >>> # load first sample of English common_voice
        >>> dataset = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="train", streaming=True)
        >>> dataset = dataset.cast_column("audio", datasets.Audio(sampling_rate=16_000))
        >>> dataset_iter = iter(dataset)
        >>> sample = next(dataset_iter)
        ...
        >>> # forward sample through model to get greedily predicted transcription ids
        >>> input_values = feature_extractor(sample["audio"]["array"], return_tensors="pt").input_values
        >>> logits = model(input_values).logits[0]
        >>> pred_ids = torch.argmax(logits, axis=-1)
        ...
        >>> # retrieve word stamps (analogous commands for `output_char_offsets`)
        >>> outputs = tokenizer.decode(pred_ids, output_word_offsets=True)
        >>> # compute `time_offset` in seconds as product of downsampling ratio and sampling_rate
        >>> time_offset = model.config.inputs_to_logits_ratio / feature_extractor.sampling_rate
        ...
        >>> word_offsets = [
        ...     {
        ...         "word": d["word"],
        ...         "start_time": round(d["start_offset"] * time_offset, 2),
        ...         "end_time": round(d["end_offset"] * time_offset, 2),
        ...     }
        ...     for d in outputs.word_offsets
        ... ]
        >>> # compare word offsets with audio `en_train_0/common_voice_en_19121553.mp3` online on the dataset viewer:
        >>> # https://hf-mirror.com/datasets/mozilla-foundation/common_voice_11_0/viewer/en
        >>> word_offsets[:3]
        [{'word': 'THE', 'start_time': 0.7, 'end_time': 0.78}, {'word': 'TRICK', 'start_time': 0.88, 'end_time': 1.08}, {'word': 'APPEARS', 'start_time': 1.2, 'end_time': 1.64}]
        ```
    """
    # Convert inputs to python lists
    token_ids = to_py_obj(token_ids)

    return self._decode(
        token_ids=token_ids,
        skip_special_tokens=skip_special_tokens,
        clean_up_tokenization_spaces=clean_up_tokenization_spaces,
        output_char_offsets=output_char_offsets,
        output_word_offsets=output_word_offsets,
        **kwargs,
    )

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.get_vocab()

Returns the vocabulary used by the Wav2Vec2CTCTokenizer.

PARAMETER	DESCRIPTION
self	An instance of the Wav2Vec2CTCTokenizer class. TYPE: Wav2Vec2CTCTokenizer

RETURNS	DESCRIPTION
Dict	A dictionary representing the vocabulary used by the tokenizer. The keys are integers representing the token IDs, and the values are the corresponding tokens. TYPE: Dict

This method retrieves the vocabulary used by the Wav2Vec2CTCTokenizer instance. The vocabulary is a dictionary that combines the encoder and added_tokens_encoder dictionaries. The encoder dictionary maps tokens to unique integer IDs, while the added_tokens_encoder dictionary contains additional tokens added by the user. The resulting vocabulary dictionary is returned.

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def get_vocab(self) -> Dict:
    """
    Returns the vocabulary used by the Wav2Vec2CTCTokenizer.

    Args:
        self (Wav2Vec2CTCTokenizer): An instance of the Wav2Vec2CTCTokenizer class.

    Returns:
        Dict: A dictionary representing the vocabulary used by the tokenizer.
            The keys are integers representing the token IDs, and the values are the corresponding tokens.

    Raises:
        None.

    This method retrieves the vocabulary used by the Wav2Vec2CTCTokenizer instance. The vocabulary is a dictionary
    that combines the encoder and added_tokens_encoder dictionaries. The encoder dictionary maps tokens to unique
    integer IDs, while the added_tokens_encoder dictionary contains additional tokens added by the user.
    The resulting vocabulary dictionary is returned.
    """
    vocab = dict(self.encoder)
    vocab.update(self.added_tokens_encoder)
    return vocab

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.prepare_for_tokenization(text, is_split_into_words=False, **kwargs)

Prepare the input text for tokenization.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2CTCTokenizer class. TYPE: Wav2Vec2CTCTokenizer
text	The input text to be prepared for tokenization. TYPE: str
is_split_into_words	A flag indicating whether the input text is already split into words. If True, the input text is expected to be split into words; otherwise, the input text is treated as a continuous string. Defaults to False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
tuple	A tuple containing the prepared text and optional keyword arguments.

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def prepare_for_tokenization(self, text, is_split_into_words=False, **kwargs):
    """
    Prepare the input text for tokenization.

    Args:
        self (Wav2Vec2CTCTokenizer): The instance of the Wav2Vec2CTCTokenizer class.
        text (str): The input text to be prepared for tokenization.
        is_split_into_words (bool): A flag indicating whether the input text is already split into words.
            If True, the input text is expected to be split into words;
            otherwise, the input text is treated as a continuous string.
            Defaults to False.

    Returns:
        tuple: A tuple containing the prepared text and optional keyword arguments.

    Raises:
        None
    """
    if is_split_into_words:
        text = " " + text
    return (text, kwargs)

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.save_vocabulary(save_directory, filename_prefix=None)

Save the vocabulary to a specified directory.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2CTCTokenizer class.
save_directory	The directory where the vocabulary will be saved. TYPE: str
filename_prefix	An optional prefix to be added to the filename. Defaults to None. TYPE: Optional[str] DEFAULT: None

RETURNS	DESCRIPTION
Tuple[str]	Tuple[str]: A tuple containing the file path of the saved vocabulary.

RAISES	DESCRIPTION
OSError	If the save_directory is not a valid directory.

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
    """
    Save the vocabulary to a specified directory.

    Args:
        self: The instance of the Wav2Vec2CTCTokenizer class.
        save_directory (str): The directory where the vocabulary will be saved.
        filename_prefix (Optional[str]): An optional prefix to be added to the filename. Defaults to None.

    Returns:
        Tuple[str]: A tuple containing the file path of the saved vocabulary.

    Raises:
        OSError: If the save_directory is not a valid directory.
    """
    if not os.path.isdir(save_directory):
        logger.error(f"Vocabulary path ({save_directory}) should be a directory")
        return
    vocab_file = os.path.join(
        save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
    )

    with open(vocab_file, "w", encoding="utf-8") as f:
        f.write(json.dumps(self.vocab, indent=2, sort_keys=True, ensure_ascii=False) + "\n")

    return (vocab_file,)

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizer.set_target_lang(target_lang)

Set the target language of a nested multi-lingual dictionary

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def set_target_lang(self, target_lang: str):
    """
    Set the target language of a nested multi-lingual dictionary
    """
    if self.vocab == self.encoder:
        raise ValueError(f"{self.vocab} is not a multi-lingual, nested tokenizer. Cannot set target language.")

    if target_lang not in self.vocab:
        raise ValueError(f"{target_lang} does not exist. Choose one of {', '.join(self.vocab.keys())}.")

    self.target_lang = target_lang
    self.init_kwargs["target_lang"] = target_lang
    self.encoder = self.vocab[target_lang]
    self.decoder = {v: k for k, v in self.encoder.items()}

    # make sure that tokens made of several
    # characters are not split at tokenization
    for token in self.encoder.keys():
        if len(token) > 1:
            self.add_tokens(AddedToken(token, rstrip=True, lstrip=True, normalized=False))

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput dataclass

Bases: ModelOutput

Output type of [Wav2Vec2CTCTokenizer], with transcription.

PARAMETER	DESCRIPTION
text	Decoded logits in text from. Usually the speech transcription. TYPE: list of `str` or `str`
char_offsets	Offsets of the decoded characters. In combination with sampling rate and model downsampling rate char offsets can be used to compute time stamps for each charater. Total logit score of the beam associated with produced text. TYPE: list of `List[Dict[str, Union[int, str]]]` or `List[Dict[str, Union[int, str]]]` DEFAULT: None
word_offsets	Offsets of the decoded words. In combination with sampling rate and model downsampling rate word offsets can be used to compute time stamps for each word. TYPE: list of `List[Dict[str, Union[int, str]]]` or `List[Dict[str, Union[int, str]]]` DEFAULT: None

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

@dataclass
class Wav2Vec2CTCTokenizerOutput(ModelOutput):
    """
    Output type of [` Wav2Vec2CTCTokenizer`], with transcription.

    Args:
        text (list of `str` or `str`):
            Decoded logits in text from. Usually the speech transcription.
        char_offsets (list of `List[Dict[str, Union[int, str]]]` or `List[Dict[str, Union[int, str]]]`):
            Offsets of the decoded characters. In combination with sampling rate and model downsampling rate char
            offsets can be used to compute time stamps for each charater. Total logit score of the beam associated with
            produced text.
        word_offsets (list of `List[Dict[str, Union[int, str]]]` or `List[Dict[str, Union[int, str]]]`):
            Offsets of the decoded words. In combination with sampling rate and model downsampling rate word offsets
            can be used to compute time stamps for each word.
    """
    text: Union[List[str], str]
    char_offsets: Union[List[ListOfDict], ListOfDict] = None
    word_offsets: Union[List[ListOfDict], ListOfDict] = None

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2Tokenizer

Bases: PreTrainedTokenizer

Constructs a Wav2Vec2 tokenizer.

This tokenizer inherits from [PreTrainedTokenizer] which contains some of the main methods. Users should refer to the superclass for more information regarding such methods.

PARAMETER	DESCRIPTION
vocab_file	File containing the vocabulary. TYPE: `str`
bos_token	The beginning of sentence token. TYPE: `str`, *optional*, defaults to `"<s>"` DEFAULT: '<s>'
eos_token	The end of sentence token. TYPE: `str`, *optional*, defaults to `"</s>"` DEFAULT: '</s>'
unk_token	The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. TYPE: `str`, *optional*, defaults to `"<unk>"` DEFAULT: '<unk>'
pad_token	The token used for padding, for example when batching sequences of different lengths. TYPE: `str`, *optional*, defaults to `"<pad>"` DEFAULT: '<pad>'
word_delimiter_token	The token used for defining the end of a word. TYPE: `str`, *optional*, defaults to `"|"` DEFAULT: '|'
do_lower_case	Whether or not to lowercase the output when decoding. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
do_normalize	Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly improve the performance for some models, e.g., wav2vec2-lv60. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
return_attention_mask	Whether or not [~Wav2Vec2Tokenizer.__call__] should return attention_mask. Wav2Vec2 models that have set config.feat_extract_norm == "group", such as wav2vec2-base, have not been trained using attention_mask. For such models, input_values should simply be padded with 0 and no attention_mask should be passed. For Wav2Vec2 models that have set config.feat_extract_norm == "layer", such as wav2vec2-lv60, attention_mask should be passed for batched inference. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

class Wav2Vec2Tokenizer(PreTrainedTokenizer):
    """
    Constructs a Wav2Vec2 tokenizer.

    This tokenizer inherits from [`PreTrainedTokenizer`] which contains some of the main methods. Users should refer to
    the superclass for more information regarding such methods.

    Args:
        vocab_file (`str`):
            File containing the vocabulary.
        bos_token (`str`, *optional*, defaults to `"<s>"`):
            The beginning of sentence token.
        eos_token (`str`, *optional*, defaults to `"</s>"`):
            The end of sentence token.
        unk_token (`str`, *optional*, defaults to `"<unk>"`):
            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
            token instead.
        pad_token (`str`, *optional*, defaults to `"<pad>"`):
            The token used for padding, for example when batching sequences of different lengths.
        word_delimiter_token (`str`, *optional*, defaults to `"|"`):
            The token used for defining the end of a word.
        do_lower_case (`bool`, *optional*, defaults to `False`):
            Whether or not to lowercase the output when decoding.
        do_normalize (`bool`, *optional*, defaults to `False`):
            Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly
            improve the performance for some models, *e.g.*,
            [wav2vec2-lv60](https://hf-mirror.com/models?search=lv60).
        return_attention_mask (`bool`, *optional*, defaults to `False`):
            Whether or not [`~Wav2Vec2Tokenizer.__call__`] should return `attention_mask`.

            <Tip>

            Wav2Vec2 models that have set `config.feat_extract_norm == "group"`, such as
            [wav2vec2-base](https://hf-mirror.com/facebook/wav2vec2-base-960h), have **not** been trained using
            `attention_mask`. For such models, `input_values` should simply be padded with 0 and no `attention_mask`
            should be passed.

            For Wav2Vec2 models that have set `config.feat_extract_norm == "layer"`, such as
            [wav2vec2-lv60](https://hf-mirror.com/facebook/wav2vec2-large-960h-lv60-self), `attention_mask` should be
            passed for batched inference.

            </Tip>

        **kwargs
            Additional keyword arguments passed along to [`PreTrainedTokenizer`]
    """
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = {
        "vocab_file": {
            "facebook/wav2vec2-base-960h": "https://hf-mirror.com/facebook/wav2vec2-base-960h/resolve/main/vocab.json"
        },
        "tokenizer_config_file": {
            "facebook/wav2vec2-base-960h": (
                "https://hf-mirror.com/facebook/wav2vec2-base-960h/resolve/main/tokenizer.json"
            ),
        },
    }
    model_input_names = ["input_values", "attention_mask"]

    def __init__(
        self,
        vocab_file,
        bos_token="<s>",
        eos_token="</s>",
        unk_token="<unk>",
        pad_token="<pad>",
        word_delimiter_token="|",
        do_lower_case=False,
        do_normalize=False,
        return_attention_mask=False,
        **kwargs,
    ):
        """
        Initializes a new instance of the Wav2Vec2Tokenizer class.

        Args:
            self: The instance of the class.
            vocab_file (str): The path to the vocabulary file.
            bos_token (str, optional): The beginning of sentence token. Default is '<s>'.
            eos_token (str, optional): The end of sentence token. Default is '</s>'.
            unk_token (str, optional): The unknown token. Default is '<unk>'.
            pad_token (str, optional): The padding token. Default is '<pad>'.
            word_delimiter_token (str, optional): The word delimiter token. Default is '|'.
            do_lower_case (bool, optional): Whether to convert tokens to lowercase. Default is False.
            do_normalize (bool, optional): Whether to apply text normalization. Default is False.
            return_attention_mask (bool, optional): Whether to return the attention mask. Default is False.

        Returns:
            None

        Raises:
            FutureWarning: This class is deprecated and will be removed in version 5 of Transformers.
                Please use Wav2Vec2Processor or Wav2Vec2CTCTokenizer instead.
        """
        warnings.warn(
            "The class `Wav2Vec2Tokenizer` is deprecated and will be removed in version 5 of Transformers. Please use"
            " `Wav2Vec2Processor` or `Wav2Vec2CTCTokenizer` instead.",
            FutureWarning,
        )

        self._word_delimiter_token = word_delimiter_token

        self.do_lower_case = do_lower_case
        self.return_attention_mask = return_attention_mask
        self.do_normalize = do_normalize

        with open(vocab_file, encoding="utf-8") as vocab_handle:
            self.encoder = json.load(vocab_handle)

        self.decoder = {v: k for k, v in self.encoder.items()}

        super().__init__(
            unk_token=unk_token,
            bos_token=bos_token,
            eos_token=eos_token,
            pad_token=pad_token,
            do_lower_case=do_lower_case,
            do_normalize=do_normalize,
            return_attention_mask=return_attention_mask,
            word_delimiter_token=word_delimiter_token,
            **kwargs,
        )

    @property
    def word_delimiter_token(self) -> str:
        """
        `str`: Padding token. Log an error if used while not having been set.
        """
        if self._word_delimiter_token is None and self.verbose:
            logger.error("Using word_delimiter_token, but it is not set yet.")
            return None
        return str(self._word_delimiter_token)

    @property
    def word_delimiter_token_id(self) -> Optional[int]:
        """
        `Optional[int]`: Id of the word_delimiter_token in the vocabulary. Returns `None` if the token has not been
        set.
        """
        if self._word_delimiter_token is None:
            return None
        return self.convert_tokens_to_ids(self.word_delimiter_token)

    @word_delimiter_token.setter
    def word_delimiter_token(self, value):
        """
        word_delimiter_token

        Setter method for setting the word delimiter token in the Wav2Vec2Tokenizer class.

        Args:
            self (Wav2Vec2Tokenizer): The instance of the Wav2Vec2Tokenizer class.
            value (str): The value to be set as the word delimiter token. Should be a string
                representing the word delimiter token.

        Returns:
            None.

        Raises:
            None.
        """
        self._word_delimiter_token = value

    @word_delimiter_token_id.setter
    def word_delimiter_token_id(self, value):
        """
        Method to set the token ID for word delimiter in the Wav2Vec2Tokenizer class.

        Args:
            self (Wav2Vec2Tokenizer): The instance of the Wav2Vec2Tokenizer class.
                This parameter refers to the tokenizer object itself.
            value (Union[int, List[int]]): The new token ID or list of token IDs for word delimiter.
                The value should be an integer or a list of integers representing token IDs.
                If a list is provided, the tokens will be converted to their corresponding IDs.

        Returns:
            None: This method does not return any value. It sets the word delimiter token ID internally.

        Raises:
            ValueError: If the provided value is not a valid integer or list of integers.
            TypeError: If the provided value is not of type int or list.
        """
        self._word_delimiter_token = self.convert_tokens_to_ids(value)

    def __call__(           self,
        raw_speech: Union[np.ndarray, List[float], List[np.ndarray], List[List[float]]],
        padding: Union[bool, str, PaddingStrategy] = False,
        max_length: Optional[int] = None,
        pad_to_multiple_of: Optional[int] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        verbose: bool = True,
        **kwargs,
    ) -> BatchEncoding:
        """
        Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of
        sequences.

        Args:
            raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`):
                The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float
                values, a list of numpy array or a list of list of float values. Must be mono channel audio, not
                stereo, i.e. single float per timestep.
        """
        is_batched_numpy = isinstance(raw_speech, np.ndarray) and len(raw_speech.shape) > 1
        if is_batched_numpy and len(raw_speech.shape) > 2:
            raise ValueError(f"Only mono-channel audio is supported for input to {self}")
        is_batched = is_batched_numpy or (
            isinstance(raw_speech, (list, tuple)) and (isinstance(raw_speech[0], (np.ndarray, tuple, list)))
        )

        # make sure input is in list format
        if is_batched and not isinstance(raw_speech[0], np.ndarray):
            raw_speech = [np.asarray(speech) for speech in raw_speech]
        elif not is_batched and not isinstance(raw_speech, np.ndarray):
            raw_speech = np.asarray(raw_speech)

        # always return batch
        if not is_batched:
            raw_speech = [raw_speech]

        # zero-mean and unit-variance normalization
        if self.do_normalize:
            raw_speech = [(x - np.mean(x)) / np.sqrt(np.var(x) + 1e-5) for x in raw_speech]

        # convert into correct format for padding
        encoded_inputs = BatchEncoding({"input_values": raw_speech})

        padded_inputs = self.pad(
            encoded_inputs,
            padding=padding,
            max_length=max_length,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=self.return_attention_mask,
            return_tensors=return_tensors,
            verbose=verbose,
        )

        return padded_inputs

    @property
    def vocab_size(self) -> int:
        """
        Method to retrieve the vocabulary size of the Wav2Vec2Tokenizer instance.

        Args:
            self (Wav2Vec2Tokenizer): The instance of the Wav2Vec2Tokenizer class.
                This parameter refers to the current instance of the Wav2Vec2Tokenizer class.
                It is used to access the decoder attribute to calculate the vocabulary size.

        Returns:
            int: An integer representing the size of the vocabulary.
                The return value corresponds to the number of elements in the decoder attribute of the instance.

        Raises:
            None.
        """
        return len(self.decoder)

    def get_vocab(self) -> Dict:
        """
        This method returns a vocabulary dictionary containing the encoder and added tokens encoder.

        Args:
            self (Wav2Vec2Tokenizer): The instance of the Wav2Vec2Tokenizer class.

        Returns:
            Dict: A dictionary containing the combined encoder and added tokens encoder.

        Raises:
            None.
        """
        return dict(self.encoder, **self.added_tokens_encoder)

    def _convert_token_to_id(self, token: str) -> int:
        """Converts a token (str) in an index (integer) using the vocab."""
        return self.encoder.get(token, self.encoder.get(self.unk_token))

    def _convert_id_to_token(self, index: int) -> str:
        """Converts an index (integer) in a token (str) using the vocab."""
        result = self.decoder.get(index, self.unk_token)
        return result

    def convert_tokens_to_string(self, tokens: List[str]) -> str:
        """
        Converts a connectionist-temporal-classification (CTC) output tokens into a single string.
        """
        # group same tokens into non-repeating tokens in CTC style decoding
        grouped_tokens = [token_group[0] for token_group in groupby(tokens)]

        # filter self.pad_token which is used as CTC-blank token
        filtered_tokens = list(filter(lambda token: token != self.pad_token, grouped_tokens))

        # replace delimiter token
        string = "".join([" " if token == self.word_delimiter_token else token for token in filtered_tokens]).strip()

        if self.do_lower_case:
            string = string.lower()

        return string

    def _decode(
        self,
        token_ids: List[int],
        skip_special_tokens: bool = False,
        clean_up_tokenization_spaces: bool = None,
        **kwargs,
    ) -> str:
        """
        special _decode function is needed for Wav2Vec2Tokenizer because added tokens should be treated exactly the
        same as tokens of the base vocabulary and therefore the function `convert_tokens_to_string` has to be called on
        the whole token list and not individually on added tokens
        """
        filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)

        result = []
        for token in filtered_tokens:
            if skip_special_tokens and token in self.all_special_ids:
                continue
            result.append(token)

        text = self.convert_tokens_to_string(result)

        clean_up_tokenization_spaces = (
            clean_up_tokenization_spaces
            if clean_up_tokenization_spaces is not None
            else self.clean_up_tokenization_spaces
        )
        if clean_up_tokenization_spaces:
            clean_text = self.clean_up_tokenization(text)
            return clean_text
        else:
            return text

    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        """
        Saves the vocabulary of the Wav2Vec2Tokenizer to a file.

        Args:
            self (Wav2Vec2Tokenizer): An instance of the Wav2Vec2Tokenizer class.
            save_directory (str): The directory where the vocabulary file will be saved.
            filename_prefix (Optional[str], optional): A prefix to be added to the filename. Defaults to None.

        Returns:
            Tuple[str]: A tuple containing the path to the saved vocabulary file.

        Raises:
            FileNotFoundError: If the specified save_directory does not exist.
            IsADirectoryError: If save_directory is not a directory.
        """
        if not os.path.isdir(save_directory):
            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
            return
        vocab_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
        )

        with open(vocab_file, "w", encoding="utf-8") as f:
            f.write(json.dumps(self.encoder, indent=2, sort_keys=True, ensure_ascii=False) + "\n")

        return (vocab_file,)

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2Tokenizer.vocab_size: int property

Method to retrieve the vocabulary size of the Wav2Vec2Tokenizer instance.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2Tokenizer class. This parameter refers to the current instance of the Wav2Vec2Tokenizer class. It is used to access the decoder attribute to calculate the vocabulary size. TYPE: Wav2Vec2Tokenizer

RETURNS	DESCRIPTION
int	An integer representing the size of the vocabulary. The return value corresponds to the number of elements in the decoder attribute of the instance. TYPE: int

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2Tokenizer.word_delimiter_token: str property writable

str: Padding token. Log an error if used while not having been set.

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2Tokenizer.word_delimiter_token_id: Optional[int] property writable

Optional[int]: Id of the word_delimiter_token in the vocabulary. Returns None if the token has not been set.

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2Tokenizer.__call__(raw_speech, padding=False, max_length=None, pad_to_multiple_of=None, return_tensors=None, verbose=True, **kwargs)

Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences.

PARAMETER	DESCRIPTION
raw_speech	The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy array or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. TYPE: `np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def __call__(           self,
    raw_speech: Union[np.ndarray, List[float], List[np.ndarray], List[List[float]]],
    padding: Union[bool, str, PaddingStrategy] = False,
    max_length: Optional[int] = None,
    pad_to_multiple_of: Optional[int] = None,
    return_tensors: Optional[Union[str, TensorType]] = None,
    verbose: bool = True,
    **kwargs,
) -> BatchEncoding:
    """
    Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of
    sequences.

    Args:
        raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`):
            The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float
            values, a list of numpy array or a list of list of float values. Must be mono channel audio, not
            stereo, i.e. single float per timestep.
    """
    is_batched_numpy = isinstance(raw_speech, np.ndarray) and len(raw_speech.shape) > 1
    if is_batched_numpy and len(raw_speech.shape) > 2:
        raise ValueError(f"Only mono-channel audio is supported for input to {self}")
    is_batched = is_batched_numpy or (
        isinstance(raw_speech, (list, tuple)) and (isinstance(raw_speech[0], (np.ndarray, tuple, list)))
    )

    # make sure input is in list format
    if is_batched and not isinstance(raw_speech[0], np.ndarray):
        raw_speech = [np.asarray(speech) for speech in raw_speech]
    elif not is_batched and not isinstance(raw_speech, np.ndarray):
        raw_speech = np.asarray(raw_speech)

    # always return batch
    if not is_batched:
        raw_speech = [raw_speech]

    # zero-mean and unit-variance normalization
    if self.do_normalize:
        raw_speech = [(x - np.mean(x)) / np.sqrt(np.var(x) + 1e-5) for x in raw_speech]

    # convert into correct format for padding
    encoded_inputs = BatchEncoding({"input_values": raw_speech})

    padded_inputs = self.pad(
        encoded_inputs,
        padding=padding,
        max_length=max_length,
        pad_to_multiple_of=pad_to_multiple_of,
        return_attention_mask=self.return_attention_mask,
        return_tensors=return_tensors,
        verbose=verbose,
    )

    return padded_inputs

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2Tokenizer.__init__(vocab_file, bos_token='<s>', eos_token='</s>', unk_token='<unk>', pad_token='<pad>', word_delimiter_token='|', do_lower_case=False, do_normalize=False, return_attention_mask=False, **kwargs)

Initializes a new instance of the Wav2Vec2Tokenizer class.

PARAMETER	DESCRIPTION
self	The instance of the class.
vocab_file	The path to the vocabulary file. TYPE: str
bos_token	The beginning of sentence token. Default is ''. TYPE: str DEFAULT: '<s>'
eos_token	The end of sentence token. Default is ''. TYPE: str DEFAULT: '</s>'
unk_token	The unknown token. Default is ''. TYPE: str DEFAULT: '<unk>'
pad_token	The padding token. Default is ''. TYPE: str DEFAULT: '<pad>'
word_delimiter_token	The word delimiter token. Default is '|'. TYPE: str DEFAULT: '|'
do_lower_case	Whether to convert tokens to lowercase. Default is False. TYPE: bool DEFAULT: False
do_normalize	Whether to apply text normalization. Default is False. TYPE: bool DEFAULT: False
return_attention_mask	Whether to return the attention mask. Default is False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
FutureWarning	This class is deprecated and will be removed in version 5 of Transformers. Please use Wav2Vec2Processor or Wav2Vec2CTCTokenizer instead.

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def __init__(
    self,
    vocab_file,
    bos_token="<s>",
    eos_token="</s>",
    unk_token="<unk>",
    pad_token="<pad>",
    word_delimiter_token="|",
    do_lower_case=False,
    do_normalize=False,
    return_attention_mask=False,
    **kwargs,
):
    """
    Initializes a new instance of the Wav2Vec2Tokenizer class.

    Args:
        self: The instance of the class.
        vocab_file (str): The path to the vocabulary file.
        bos_token (str, optional): The beginning of sentence token. Default is '<s>'.
        eos_token (str, optional): The end of sentence token. Default is '</s>'.
        unk_token (str, optional): The unknown token. Default is '<unk>'.
        pad_token (str, optional): The padding token. Default is '<pad>'.
        word_delimiter_token (str, optional): The word delimiter token. Default is '|'.
        do_lower_case (bool, optional): Whether to convert tokens to lowercase. Default is False.
        do_normalize (bool, optional): Whether to apply text normalization. Default is False.
        return_attention_mask (bool, optional): Whether to return the attention mask. Default is False.

    Returns:
        None

    Raises:
        FutureWarning: This class is deprecated and will be removed in version 5 of Transformers.
            Please use Wav2Vec2Processor or Wav2Vec2CTCTokenizer instead.
    """
    warnings.warn(
        "The class `Wav2Vec2Tokenizer` is deprecated and will be removed in version 5 of Transformers. Please use"
        " `Wav2Vec2Processor` or `Wav2Vec2CTCTokenizer` instead.",
        FutureWarning,
    )

    self._word_delimiter_token = word_delimiter_token

    self.do_lower_case = do_lower_case
    self.return_attention_mask = return_attention_mask
    self.do_normalize = do_normalize

    with open(vocab_file, encoding="utf-8") as vocab_handle:
        self.encoder = json.load(vocab_handle)

    self.decoder = {v: k for k, v in self.encoder.items()}

    super().__init__(
        unk_token=unk_token,
        bos_token=bos_token,
        eos_token=eos_token,
        pad_token=pad_token,
        do_lower_case=do_lower_case,
        do_normalize=do_normalize,
        return_attention_mask=return_attention_mask,
        word_delimiter_token=word_delimiter_token,
        **kwargs,
    )

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2Tokenizer.convert_tokens_to_string(tokens)

Converts a connectionist-temporal-classification (CTC) output tokens into a single string.

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def convert_tokens_to_string(self, tokens: List[str]) -> str:
    """
    Converts a connectionist-temporal-classification (CTC) output tokens into a single string.
    """
    # group same tokens into non-repeating tokens in CTC style decoding
    grouped_tokens = [token_group[0] for token_group in groupby(tokens)]

    # filter self.pad_token which is used as CTC-blank token
    filtered_tokens = list(filter(lambda token: token != self.pad_token, grouped_tokens))

    # replace delimiter token
    string = "".join([" " if token == self.word_delimiter_token else token for token in filtered_tokens]).strip()

    if self.do_lower_case:
        string = string.lower()

    return string

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2Tokenizer.get_vocab()

This method returns a vocabulary dictionary containing the encoder and added tokens encoder.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2Tokenizer class. TYPE: Wav2Vec2Tokenizer

RETURNS	DESCRIPTION
Dict	A dictionary containing the combined encoder and added tokens encoder. TYPE: Dict

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def get_vocab(self) -> Dict:
    """
    This method returns a vocabulary dictionary containing the encoder and added tokens encoder.

    Args:
        self (Wav2Vec2Tokenizer): The instance of the Wav2Vec2Tokenizer class.

    Returns:
        Dict: A dictionary containing the combined encoder and added tokens encoder.

    Raises:
        None.
    """
    return dict(self.encoder, **self.added_tokens_encoder)

mindnlp.transformers.models.wav2vec2.tokenization_wav2vec2.Wav2Vec2Tokenizer.save_vocabulary(save_directory, filename_prefix=None)

Saves the vocabulary of the Wav2Vec2Tokenizer to a file.

PARAMETER	DESCRIPTION
self	An instance of the Wav2Vec2Tokenizer class. TYPE: Wav2Vec2Tokenizer
save_directory	The directory where the vocabulary file will be saved. TYPE: str
filename_prefix	A prefix to be added to the filename. Defaults to None. TYPE: Optional[str] DEFAULT: None

RETURNS	DESCRIPTION
Tuple[str]	Tuple[str]: A tuple containing the path to the saved vocabulary file.

RAISES	DESCRIPTION
FileNotFoundError	If the specified save_directory does not exist.
IsADirectoryError	If save_directory is not a directory.

Source code in mindnlp/transformers/models/wav2vec2/tokenization_wav2vec2.py

def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
    """
    Saves the vocabulary of the Wav2Vec2Tokenizer to a file.

    Args:
        self (Wav2Vec2Tokenizer): An instance of the Wav2Vec2Tokenizer class.
        save_directory (str): The directory where the vocabulary file will be saved.
        filename_prefix (Optional[str], optional): A prefix to be added to the filename. Defaults to None.

    Returns:
        Tuple[str]: A tuple containing the path to the saved vocabulary file.

    Raises:
        FileNotFoundError: If the specified save_directory does not exist.
        IsADirectoryError: If save_directory is not a directory.
    """
    if not os.path.isdir(save_directory):
        logger.error(f"Vocabulary path ({save_directory}) should be a directory")
        return
    vocab_file = os.path.join(
        save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
    )

    with open(vocab_file, "w", encoding="utf-8") as f:
        f.write(json.dumps(self.encoder, indent=2, sort_keys=True, ensure_ascii=False) + "\n")

    return (vocab_file,)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2

Mindspore Wav2Vec2 model.

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.AMSoftmaxLoss

Bases: Cell

The AMSoftmaxLoss class represents a neural network cell for computing the AM-Softmax loss. This class inherits from nn.Cell and provides methods for initializing the loss function and constructing the computation graph.

ATTRIBUTE	DESCRIPTION
scale	The scale parameter for the AM-Softmax loss function. TYPE: float
margin	The margin parameter for the AM-Softmax loss function. TYPE: float
num_labels	The number of unique labels in the dataset. TYPE: int
weight	The weight parameter for the neural network. TYPE: Parameter

METHOD	DESCRIPTION
__init__	Initializes the AMSoftmaxLoss instance with input dimension, number of labels, scale, and margin.
construct	Constructs the computation graph for the AM-Softmax loss function using the given hidden states and labels.

Note

The AMSoftmaxLoss class is designed for use in neural network training and optimization tasks.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class AMSoftmaxLoss(nn.Cell):

    """
    The AMSoftmaxLoss class represents a neural network cell for computing the AM-Softmax loss. This class inherits
    from nn.Cell and provides methods for initializing the loss function and constructing the computation graph.

    Attributes:
        scale (float): The scale parameter for the AM-Softmax loss function.
        margin (float): The margin parameter for the AM-Softmax loss function.
        num_labels (int): The number of unique labels in the dataset.
        weight (Parameter): The weight parameter for the neural network.

    Methods:
        __init__: Initializes the AMSoftmaxLoss instance with input dimension, number of labels, scale, and margin.

        construct: Constructs the computation graph for the AM-Softmax loss function using the given
            hidden states and labels.

    Note:
        The AMSoftmaxLoss class is designed for use in neural network training and optimization tasks.
    """
    def __init__(self, input_dim, num_labels, scale=30.0, margin=0.4):
        """
        __init__

        Initializes an instance of the AMSoftmaxLoss class.

        Args:
            self (object): The instance of the class.
            input_dim (int): The dimension of the input features.
            num_labels (int): The number of unique labels for classification.
            scale (float, optional): The scale factor for the angular margin. Defaults to 30.0.
            margin (float, optional): The angular margin value. Defaults to 0.4.

        Returns:
            None.

        Raises:
            ValueError: If input_dim or num_labels are not positive integers.
            TypeError: If scale or margin are not of type float.
        """
        super().__init__()
        self.scale = scale
        self.margin = margin
        self.num_labels = num_labels
        self.weight = Parameter(ops.randn(input_dim, num_labels), requires_grad=True)

    def construct(self, hidden_states, labels):
        """
        This method constructs an AMSoftmax loss function.

        Args:
            self (object): The instance of the AMSoftmaxLoss class.
            hidden_states (tensor): A tensor representing the hidden states of the model.
            labels (tensor): A tensor containing the ground truth labels for the corresponding hidden states.
                It is expected that the labels are flattened for processing.

        Returns:
            None.

        Raises:
            ValueError: If the dimensions of the weight tensor and hidden_states tensor are not compatible
                for matrix multiplication.
            RuntimeError: If there is an issue with the normalization operation on the weight or hidden_states tensor.
            ValueError: If the labels tensor does not match the expected shape for one-hot encoding.
            RuntimeError: If there is a problem with the cross-entropy calculation.
        """
        labels = labels.flatten()
        weight = self.weight / ops.norm(self.weight, dim=0, keepdim=True)
        hidden_states = hidden_states / ops.norm(hidden_states, dim=1, keepdim=True)
        cos_theta = ops.mm(hidden_states, weight)
        psi = cos_theta - self.margin

        onehot = ops.one_hot(labels, self.num_labels)
        logits = self.scale * ops.where(onehot.bool(), psi, cos_theta)
        loss = ops.cross_entropy(logits, labels)
        return loss

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.AMSoftmaxLoss.__init__(input_dim, num_labels, scale=30.0, margin=0.4)

init

Initializes an instance of the AMSoftmaxLoss class.

PARAMETER	DESCRIPTION
self	The instance of the class. TYPE: object
input_dim	The dimension of the input features. TYPE: int
num_labels	The number of unique labels for classification. TYPE: int
scale	The scale factor for the angular margin. Defaults to 30.0. TYPE: float DEFAULT: 30.0
margin	The angular margin value. Defaults to 0.4. TYPE: float DEFAULT: 0.4

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If input_dim or num_labels are not positive integers.
TypeError	If scale or margin are not of type float.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, input_dim, num_labels, scale=30.0, margin=0.4):
    """
    __init__

    Initializes an instance of the AMSoftmaxLoss class.

    Args:
        self (object): The instance of the class.
        input_dim (int): The dimension of the input features.
        num_labels (int): The number of unique labels for classification.
        scale (float, optional): The scale factor for the angular margin. Defaults to 30.0.
        margin (float, optional): The angular margin value. Defaults to 0.4.

    Returns:
        None.

    Raises:
        ValueError: If input_dim or num_labels are not positive integers.
        TypeError: If scale or margin are not of type float.
    """
    super().__init__()
    self.scale = scale
    self.margin = margin
    self.num_labels = num_labels
    self.weight = Parameter(ops.randn(input_dim, num_labels), requires_grad=True)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.AMSoftmaxLoss.construct(hidden_states, labels)

This method constructs an AMSoftmax loss function.

PARAMETER	DESCRIPTION
self	The instance of the AMSoftmaxLoss class. TYPE: object
hidden_states	A tensor representing the hidden states of the model. TYPE: tensor
labels	A tensor containing the ground truth labels for the corresponding hidden states. It is expected that the labels are flattened for processing. TYPE: tensor

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the dimensions of the weight tensor and hidden_states tensor are not compatible for matrix multiplication.
RuntimeError	If there is an issue with the normalization operation on the weight or hidden_states tensor.
ValueError	If the labels tensor does not match the expected shape for one-hot encoding.
RuntimeError	If there is a problem with the cross-entropy calculation.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states, labels):
    """
    This method constructs an AMSoftmax loss function.

    Args:
        self (object): The instance of the AMSoftmaxLoss class.
        hidden_states (tensor): A tensor representing the hidden states of the model.
        labels (tensor): A tensor containing the ground truth labels for the corresponding hidden states.
            It is expected that the labels are flattened for processing.

    Returns:
        None.

    Raises:
        ValueError: If the dimensions of the weight tensor and hidden_states tensor are not compatible
            for matrix multiplication.
        RuntimeError: If there is an issue with the normalization operation on the weight or hidden_states tensor.
        ValueError: If the labels tensor does not match the expected shape for one-hot encoding.
        RuntimeError: If there is a problem with the cross-entropy calculation.
    """
    labels = labels.flatten()
    weight = self.weight / ops.norm(self.weight, dim=0, keepdim=True)
    hidden_states = hidden_states / ops.norm(hidden_states, dim=1, keepdim=True)
    cos_theta = ops.mm(hidden_states, weight)
    psi = cos_theta - self.margin

    onehot = ops.one_hot(labels, self.num_labels)
    logits = self.scale * ops.where(onehot.bool(), psi, cos_theta)
    loss = ops.cross_entropy(logits, labels)
    return loss

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.TDNNLayer

Bases: Cell

TDNNLayer represents a time-delay neural network (TDNN) layer for processing sequential data. It inherits from nn.Cell and is initialized with a Wav2Vec2Config and an optional layer_id.

ATTRIBUTE	DESCRIPTION
config	The configuration for the Wav2Vec2 model. TYPE: Wav2Vec2Config
layer_id	The index of the TDNN layer. TYPE: int

METHOD	DESCRIPTION
construct	Applies the TDNN layer operations to the input hidden_states.

The TDNNLayer class applies a convolutional layer with specified kernel size and dilation to the input data. It then applies a ReLU activation function to the output.

Note

This class is part of the Wav2Vec2 model architecture.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class TDNNLayer(nn.Cell):

    """TDNNLayer represents a time-delay neural network (TDNN) layer for processing sequential data.
    It inherits from nn.Cell and is initialized with a Wav2Vec2Config and an optional layer_id.

    Attributes:
        config (Wav2Vec2Config): The configuration for the Wav2Vec2 model.
        layer_id (int): The index of the TDNN layer.

    Methods:
        construct(hidden_states): Applies the TDNN layer operations to the input hidden_states.

    The TDNNLayer class applies a convolutional layer with specified kernel size and dilation to the input data.
    It then applies a ReLU activation function to the output.

    Note:
        This class is part of the Wav2Vec2 model architecture.

    """
    def __init__(self, config: Wav2Vec2Config, layer_id=0):
        """
        Initializes a TDNNLayer object.

        Args:
            self: The instance of the TDNNLayer class.
            config (Wav2Vec2Config): An instance of Wav2Vec2Config that holds configuration parameters for the layer.
            layer_id (int): An integer representing the ID of the layer. Default is 0. Must be within the range of
                available layers in the configuration.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not of type Wav2Vec2Config.
            ValueError: If the layer_id is outside the valid range of available layers in the configuration.
        """
        super().__init__()
        self.in_conv_dim = config.tdnn_dim[layer_id - 1] if layer_id > 0 else config.tdnn_dim[layer_id]
        self.out_conv_dim = config.tdnn_dim[layer_id]
        self.kernel_size = config.tdnn_kernel[layer_id]
        self.dilation = config.tdnn_dilation[layer_id]

        self.kernel = nn.Dense(self.in_conv_dim * self.kernel_size, self.out_conv_dim)
        self.activation = nn.ReLU()

    def construct(self, hidden_states):
        '''
        Constructs the TDNN layer with the input hidden_states.

        Args:
            self (TDNNLayer): The instance of the TDNNLayer class.
            hidden_states (Tensor): The input hidden states to be processed by the TDNN layer.
                It should be a tensor of shape (batch_size, in_channels, sequence_length).

        Returns:
            hidden_states (Tensor): The processed hidden states after applying the TDNN layer operations.
                It will be a tensor of shape (batch_size, out_channels, new_length), where out_channels is the number
                of output channels and new_length is the length of the output sequence.

        Raises:
            TypeError: If the input hidden_states is not a tensor.
            ValueError: If the input hidden_states does not have the expected shape or dimensions.
        '''
        hidden_states = hidden_states.unsqueeze(1)
        hidden_states = ops.unfold(
            hidden_states,
            (self.kernel_size, self.in_conv_dim),
            stride=(1, self.in_conv_dim),
            dilation=(self.dilation, 1),
        )
        hidden_states = hidden_states.swapaxes(1, 2)
        hidden_states = self.kernel(hidden_states)

        hidden_states = self.activation(hidden_states)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.TDNNLayer.__init__(config, layer_id=0)

Initializes a TDNNLayer object.

PARAMETER	DESCRIPTION
self	The instance of the TDNNLayer class.
config	An instance of Wav2Vec2Config that holds configuration parameters for the layer. TYPE: Wav2Vec2Config
layer_id	An integer representing the ID of the layer. Default is 0. Must be within the range of available layers in the configuration. TYPE: int DEFAULT: 0

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not of type Wav2Vec2Config.
ValueError	If the layer_id is outside the valid range of available layers in the configuration.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config, layer_id=0):
    """
    Initializes a TDNNLayer object.

    Args:
        self: The instance of the TDNNLayer class.
        config (Wav2Vec2Config): An instance of Wav2Vec2Config that holds configuration parameters for the layer.
        layer_id (int): An integer representing the ID of the layer. Default is 0. Must be within the range of
            available layers in the configuration.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not of type Wav2Vec2Config.
        ValueError: If the layer_id is outside the valid range of available layers in the configuration.
    """
    super().__init__()
    self.in_conv_dim = config.tdnn_dim[layer_id - 1] if layer_id > 0 else config.tdnn_dim[layer_id]
    self.out_conv_dim = config.tdnn_dim[layer_id]
    self.kernel_size = config.tdnn_kernel[layer_id]
    self.dilation = config.tdnn_dilation[layer_id]

    self.kernel = nn.Dense(self.in_conv_dim * self.kernel_size, self.out_conv_dim)
    self.activation = nn.ReLU()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.TDNNLayer.construct(hidden_states)

Constructs the TDNN layer with the input hidden_states.

PARAMETER	DESCRIPTION
self	The instance of the TDNNLayer class. TYPE: TDNNLayer
hidden_states	The input hidden states to be processed by the TDNN layer. It should be a tensor of shape (batch_size, in_channels, sequence_length). TYPE: Tensor

RETURNS	DESCRIPTION
hidden_states	The processed hidden states after applying the TDNN layer operations. It will be a tensor of shape (batch_size, out_channels, new_length), where out_channels is the number of output channels and new_length is the length of the output sequence. TYPE: Tensor

RAISES	DESCRIPTION
TypeError	If the input hidden_states is not a tensor.
ValueError	If the input hidden_states does not have the expected shape or dimensions.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    '''
    Constructs the TDNN layer with the input hidden_states.

    Args:
        self (TDNNLayer): The instance of the TDNNLayer class.
        hidden_states (Tensor): The input hidden states to be processed by the TDNN layer.
            It should be a tensor of shape (batch_size, in_channels, sequence_length).

    Returns:
        hidden_states (Tensor): The processed hidden states after applying the TDNN layer operations.
            It will be a tensor of shape (batch_size, out_channels, new_length), where out_channels is the number
            of output channels and new_length is the length of the output sequence.

    Raises:
        TypeError: If the input hidden_states is not a tensor.
        ValueError: If the input hidden_states does not have the expected shape or dimensions.
    '''
    hidden_states = hidden_states.unsqueeze(1)
    hidden_states = ops.unfold(
        hidden_states,
        (self.kernel_size, self.in_conv_dim),
        stride=(1, self.in_conv_dim),
        dilation=(self.dilation, 1),
    )
    hidden_states = hidden_states.swapaxes(1, 2)
    hidden_states = self.kernel(hidden_states)

    hidden_states = self.activation(hidden_states)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Adapter

Bases: Cell

Wav2Vec2Adapter is a class that represents an adapter layer for adapting the hidden states of a Wav2Vec2 model. This class inherits from nn.Cell and implements methods for initializing and constructing the adapter layer.

ATTRIBUTE	DESCRIPTION
proj	A dense layer used for projecting hidden states if output_hidden_size is different from hidden_size. TYPE: Dense or None
proj_layer_norm	A layer normalization module applied after projection if needed. TYPE: LayerNorm or None
layers	A list of Wav2Vec2AdapterLayer instances representing adapter layers. TYPE: CellList
layerdrop	The probability of dropping a layer during training. TYPE: float

METHOD	DESCRIPTION
__init__	Initializes the Wav2Vec2Adapter object with the provided configuration.
construct	Applies the adapter layer transformations to the input hidden states.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2Adapter(nn.Cell):

    """
    Wav2Vec2Adapter is a class that represents an adapter layer for adapting the hidden states of a Wav2Vec2 model.
    This class inherits from nn.Cell and implements methods for initializing and constructing the adapter layer.

    Attributes:
        proj (nn.Dense or None): A dense layer used for projecting hidden states if output_hidden_size is
            different from hidden_size.
        proj_layer_norm (nn.LayerNorm or None): A layer normalization module applied after projection if needed.
        layers (nn.CellList): A list of Wav2Vec2AdapterLayer instances representing adapter layers.
        layerdrop (float): The probability of dropping a layer during training.

    Methods:
        __init__: Initializes the Wav2Vec2Adapter object with the provided configuration.
        construct: Applies the adapter layer transformations to the input hidden states.

    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a new instance of the Wav2Vec2Adapter class.

        Args:
            self: The current instance of the class.
            config (Wav2Vec2Config): An instance of Wav2Vec2Config containing configuration parameters for the adapter.
                This parameter is required for initializing the adapter and must be an instance of Wav2Vec2Config.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not of type Wav2Vec2Config.
            ValueError: If the output_hidden_size in the config parameter does not match the hidden_size.
        """
        super().__init__()

        # feature dim might need to be down-projected
        if config.output_hidden_size != config.hidden_size:
            self.proj = nn.Dense(config.hidden_size, config.output_hidden_size)
            self.proj_layer_norm = nn.LayerNorm(config.output_hidden_size)
        else:
            self.proj = self.proj_layer_norm = None

        self.layers = nn.CellList([Wav2Vec2AdapterLayer(config) for _ in range(config.num_adapter_layers)])
        self.layerdrop = config.layerdrop

    def construct(self, hidden_states):
        """
        This method constructs the hidden states by applying transformations and layers.

        Args:
            self (object): The instance of the Wav2Vec2Adapter class.
            hidden_states (numpy.ndarray): The input hidden states to be processed.
                It is expected to be a 3D array with shape (batch_size, sequence_length, hidden_size).

        Returns:
            numpy.ndarray: The processed hidden states with shape (batch_size, sequence_length, hidden_size).

        Raises:
            None
        """
        # down project hidden_states if necessary
        if self.proj is not None and self.proj_layer_norm is not None:
            hidden_states = self.proj(hidden_states)
            hidden_states = self.proj_layer_norm(hidden_states)

        hidden_states = hidden_states.swapaxes(1, 2)

        for layer in self.layers:
            layerdrop_prob = np.random.random()
            if not self.training or (layerdrop_prob > self.layerdrop):
                hidden_states = layer(hidden_states)

        hidden_states = hidden_states.swapaxes(1, 2)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Adapter.__init__(config)

Initializes a new instance of the Wav2Vec2Adapter class.

PARAMETER	DESCRIPTION
self	The current instance of the class.
config	An instance of Wav2Vec2Config containing configuration parameters for the adapter. This parameter is required for initializing the adapter and must be an instance of Wav2Vec2Config. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not of type Wav2Vec2Config.
ValueError	If the output_hidden_size in the config parameter does not match the hidden_size.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a new instance of the Wav2Vec2Adapter class.

    Args:
        self: The current instance of the class.
        config (Wav2Vec2Config): An instance of Wav2Vec2Config containing configuration parameters for the adapter.
            This parameter is required for initializing the adapter and must be an instance of Wav2Vec2Config.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not of type Wav2Vec2Config.
        ValueError: If the output_hidden_size in the config parameter does not match the hidden_size.
    """
    super().__init__()

    # feature dim might need to be down-projected
    if config.output_hidden_size != config.hidden_size:
        self.proj = nn.Dense(config.hidden_size, config.output_hidden_size)
        self.proj_layer_norm = nn.LayerNorm(config.output_hidden_size)
    else:
        self.proj = self.proj_layer_norm = None

    self.layers = nn.CellList([Wav2Vec2AdapterLayer(config) for _ in range(config.num_adapter_layers)])
    self.layerdrop = config.layerdrop

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Adapter.construct(hidden_states)

This method constructs the hidden states by applying transformations and layers.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2Adapter class. TYPE: object
hidden_states	The input hidden states to be processed. It is expected to be a 3D array with shape (batch_size, sequence_length, hidden_size). TYPE: ndarray

RETURNS	DESCRIPTION
	numpy.ndarray: The processed hidden states with shape (batch_size, sequence_length, hidden_size).

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    """
    This method constructs the hidden states by applying transformations and layers.

    Args:
        self (object): The instance of the Wav2Vec2Adapter class.
        hidden_states (numpy.ndarray): The input hidden states to be processed.
            It is expected to be a 3D array with shape (batch_size, sequence_length, hidden_size).

    Returns:
        numpy.ndarray: The processed hidden states with shape (batch_size, sequence_length, hidden_size).

    Raises:
        None
    """
    # down project hidden_states if necessary
    if self.proj is not None and self.proj_layer_norm is not None:
        hidden_states = self.proj(hidden_states)
        hidden_states = self.proj_layer_norm(hidden_states)

    hidden_states = hidden_states.swapaxes(1, 2)

    for layer in self.layers:
        layerdrop_prob = np.random.random()
        if not self.training or (layerdrop_prob > self.layerdrop):
            hidden_states = layer(hidden_states)

    hidden_states = hidden_states.swapaxes(1, 2)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2AdapterLayer

Bases: Cell

Wav2Vec2AdapterLayer is a Python class that represents an adapter layer for the Wav2Vec2 model. This class inherits from nn.Cell.

The adapter layer contains methods for initialization and construction.

The init method initializes the adapter layer with the provided configuration. It sets up a 1D convolutional layer with specified parameters such as kernel size, stride, padding, and bias.

The construct method takes hidden_states as input and applies the convolutional layer followed by the gated linear unit (GLU) activation function. It then returns the processed hidden states.

This class provides functionality for creating and processing adapter layers within the Wav2Vec2 model.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2AdapterLayer(nn.Cell):

    '''
    Wav2Vec2AdapterLayer is a Python class that represents an adapter layer for the Wav2Vec2 model.
    This class inherits from nn.Cell.

    The adapter layer contains methods for initialization and construction.

    The __init__ method initializes the adapter layer with the provided configuration. It sets up a 1D convolutional
    layer with specified parameters such as kernel size, stride, padding, and bias.

    The construct method takes hidden_states as input and applies the convolutional layer followed by the
    gated linear unit (GLU) activation function. It then returns the processed hidden states.

    This class provides functionality for creating and processing adapter layers within the Wav2Vec2 model.
    '''
    def __init__(self, config: Wav2Vec2Config):
        """
        __init__

        Initializes a new instance of the Wav2Vec2AdapterLayer class.

        Args:
            self: The instance of the Wav2Vec2AdapterLayer class.
            config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
                for the adapter layer.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.conv = nn.Conv1d(
            config.output_hidden_size,
            2 * config.output_hidden_size,
            config.adapter_kernel_size,
            stride=config.adapter_stride,
            padding=1,
            pad_mode='pad',
            has_bias=True,
        )

    def construct(self, hidden_states):
        """
        Method to construct the Wav2Vec2AdapterLayer.

        Args:
            self (Wav2Vec2AdapterLayer): The instance of the Wav2Vec2AdapterLayer class.
            hidden_states (Tensor): The input hidden states to be processed. It should be a tensor.

        Returns:
            Tensor: The processed hidden states after applying convolution and gated linear units (GLU) operation.

        Raises:
            None.
        """
        hidden_states = self.conv(hidden_states)
        hidden_states = ops.glu(hidden_states, axis=1)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2AdapterLayer.__init__(config)

init

Initializes a new instance of the Wav2Vec2AdapterLayer class.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2AdapterLayer class.
config	An instance of the Wav2Vec2Config class containing the configuration parameters for the adapter layer. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    __init__

    Initializes a new instance of the Wav2Vec2AdapterLayer class.

    Args:
        self: The instance of the Wav2Vec2AdapterLayer class.
        config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
            for the adapter layer.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.conv = nn.Conv1d(
        config.output_hidden_size,
        2 * config.output_hidden_size,
        config.adapter_kernel_size,
        stride=config.adapter_stride,
        padding=1,
        pad_mode='pad',
        has_bias=True,
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2AdapterLayer.construct(hidden_states)

Method to construct the Wav2Vec2AdapterLayer.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2AdapterLayer class. TYPE: Wav2Vec2AdapterLayer
hidden_states	The input hidden states to be processed. It should be a tensor. TYPE: Tensor

RETURNS	DESCRIPTION
Tensor	The processed hidden states after applying convolution and gated linear units (GLU) operation.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    """
    Method to construct the Wav2Vec2AdapterLayer.

    Args:
        self (Wav2Vec2AdapterLayer): The instance of the Wav2Vec2AdapterLayer class.
        hidden_states (Tensor): The input hidden states to be processed. It should be a tensor.

    Returns:
        Tensor: The processed hidden states after applying convolution and gated linear units (GLU) operation.

    Raises:
        None.
    """
    hidden_states = self.conv(hidden_states)
    hidden_states = ops.glu(hidden_states, axis=1)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Attention

Bases: Cell

Multi-headed attention from 'Attention Is All You Need' paper

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2Attention(nn.Cell):
    """Multi-headed attention from 'Attention Is All You Need' paper"""
    def __init__(
        self,
        embed_dim: int,
        num_heads: int,
        dropout: float = 0.0,
        is_decoder: bool = False,
        bias: bool = True,
        is_causal: bool = False,
        config: Optional[Wav2Vec2Config] = None,
    ):
        """
        Initializes an instance of the Wav2Vec2Attention class.

        Args:
            embed_dim (int): The dimension of the input embeddings.
            num_heads (int): The number of attention heads.
            dropout (float, optional): The dropout probability. Defaults to 0.0.
            is_decoder (bool, optional): Whether the attention module is used as a decoder. Defaults to False.
            bias (bool, optional): Whether to include bias in linear projections. Defaults to True.
            is_causal (bool, optional): Whether the attention is causal. Defaults to False.
            config (Optional[Wav2Vec2Config], optional): The configuration object. Defaults to None.

        Returns:
            None

        Raises:
            ValueError: If embed_dim is not divisible by num_heads.
        """
        super().__init__()
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.dropout = dropout
        self.head_dim = embed_dim // num_heads
        self.config = config

        if (self.head_dim * num_heads) != self.embed_dim:
            raise ValueError(
                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}"
                f" and `num_heads`: {num_heads})."
            )
        self.scaling = self.head_dim**-0.5
        self.is_decoder = is_decoder
        self.is_causal = is_causal

        self.k_proj = nn.Dense(embed_dim, embed_dim, has_bias=bias)
        self.v_proj = nn.Dense(embed_dim, embed_dim, has_bias=bias)
        self.q_proj = nn.Dense(embed_dim, embed_dim, has_bias=bias)
        self.out_proj = nn.Dense(embed_dim, embed_dim, has_bias=bias)

    def _shape(self, tensor: Tensor, seq_len: int, bsz: int):
        """
        This method '_shape' is defined in the class 'Wav2Vec2Attention' and is used to reshape the input tensor to
        the specified shape.

        Args:
            tensor (Tensor): The input tensor to be reshaped. It should be of type Tensor.
            seq_len (int): The length of the sequence. It should be an integer.
            bsz (int): The batch size. It should be an integer.

        Returns:
            None.

        Raises:
            None.
        """
        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).swapaxes(1, 2)

    def construct(
        self,
        hidden_states: Tensor,
        key_value_states: Optional[Tensor] = None,
        past_key_value: Optional[Tuple[Tensor]] = None,
        attention_mask: Optional[Tensor] = None,
        layer_head_mask: Optional[Tensor] = None,
        output_attentions: bool = False,
    ) -> Tuple[Tensor, Optional[Tensor], Optional[Tuple[Tensor]]]:
        """Input shape: Batch x Time x Channel"""
        # if key_value_states are provided this layer is used as a cross-attention layer
        # for the decoder
        is_cross_attention = key_value_states is not None

        bsz, tgt_len, _ = hidden_states.shape

        # get query proj
        query_states = self.q_proj(hidden_states) * self.scaling
        # get key, value proj
        # `past_key_value[0].shape[2] == key_value_states.shape[1]`
        # is checking that the `sequence_length` of the `past_key_value` is the same as
        # the provided `key_value_states` to support prefix tuning
        if (
            is_cross_attention
            and past_key_value is not None
            and past_key_value[0].shape[2] == key_value_states.shape[1]
        ):
            # reuse k,v, cross_attentions
            key_states = past_key_value[0]
            value_states = past_key_value[1]
        elif is_cross_attention:
            # cross_attentions
            key_states = self._shape(self.k_proj(key_value_states), -1, bsz)
            value_states = self._shape(self.v_proj(key_value_states), -1, bsz)
        elif past_key_value is not None:
            # reuse k, v, self_attention
            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)
            key_states = ops.cat([past_key_value[0], key_states], axis=2)
            value_states = ops.cat([past_key_value[1], value_states], axis=2)
        else:
            # self_attention
            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

        if self.is_decoder:
            # if cross_attention save Tuple(Tensor, Tensor) of all cross attention key/value_states.
            # Further calls to cross_attention layer can then reuse all cross-attention
            # key/value_states (first "if" case)
            # if uni-directional self-attention (decoder) save Tuple(Tensor, Tensor) of
            # all previous decoder key/value_states. Further calls to uni-directional self-attention
            # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
            # if encoder bi-directional self-attention `past_key_value` is always `None`
            past_key_value = (key_states, value_states)

        proj_shape = (bsz * self.num_heads, -1, self.head_dim)
        query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
        key_states = key_states.reshape(*proj_shape)
        value_states = value_states.reshape(*proj_shape)

        src_len = key_states.shape[1]
        attn_weights = ops.bmm(query_states, key_states.swapaxes(1, 2))

        if attn_weights.shape != (bsz * self.num_heads, tgt_len, src_len):
            raise ValueError(
                f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
                f" {attn_weights.shape}"
            )

        if attention_mask is not None:
            if attention_mask.shape != (bsz, 1, tgt_len, src_len):
                raise ValueError(
                    f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.shape}"
                )
            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        attn_weights = ops.softmax(attn_weights, axis=-1)

        if layer_head_mask is not None:
            if layer_head_mask.shape != (self.num_heads,):
                raise ValueError(
                    f"Head mask for a single layer should be of size {(self.num_heads,)}, but is"
                    f" {layer_head_mask.shape}"
                )
            attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        if output_attentions:
            # this operation is a bit awkward, but it's required to
            # make sure that attn_weights keeps its gradient.
            # In order to do so, attn_weights have to be reshaped
            # twice and have to be reused in the following
            attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
            attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
        else:
            attn_weights_reshaped = None

        attn_probs = ops.dropout(attn_weights, p=self.dropout, training=self.training)

        attn_output = ops.bmm(attn_probs, value_states)

        if attn_output.shape != (bsz * self.num_heads, tgt_len, self.head_dim):
            raise ValueError(
                f"`attn_output` should be of size {(bsz * self.num_heads, tgt_len, self.head_dim)}, but is"
                f" {attn_output.shape}"
            )

        attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
        attn_output = attn_output.swapaxes(1, 2)

        # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be
        # partitioned across GPUs when using tensor-parallelism.
        attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)

        attn_output = self.out_proj(attn_output)

        return attn_output, attn_weights_reshaped, past_key_value

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Attention.__init__(embed_dim, num_heads, dropout=0.0, is_decoder=False, bias=True, is_causal=False, config=None)

Initializes an instance of the Wav2Vec2Attention class.

PARAMETER	DESCRIPTION
embed_dim	The dimension of the input embeddings. TYPE: int
num_heads	The number of attention heads. TYPE: int
dropout	The dropout probability. Defaults to 0.0. TYPE: float DEFAULT: 0.0
is_decoder	Whether the attention module is used as a decoder. Defaults to False. TYPE: bool DEFAULT: False
bias	Whether to include bias in linear projections. Defaults to True. TYPE: bool DEFAULT: True
is_causal	Whether the attention is causal. Defaults to False. TYPE: bool DEFAULT: False
config	The configuration object. Defaults to None. TYPE: Optional[Wav2Vec2Config] DEFAULT: None

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
ValueError	If embed_dim is not divisible by num_heads.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(
    self,
    embed_dim: int,
    num_heads: int,
    dropout: float = 0.0,
    is_decoder: bool = False,
    bias: bool = True,
    is_causal: bool = False,
    config: Optional[Wav2Vec2Config] = None,
):
    """
    Initializes an instance of the Wav2Vec2Attention class.

    Args:
        embed_dim (int): The dimension of the input embeddings.
        num_heads (int): The number of attention heads.
        dropout (float, optional): The dropout probability. Defaults to 0.0.
        is_decoder (bool, optional): Whether the attention module is used as a decoder. Defaults to False.
        bias (bool, optional): Whether to include bias in linear projections. Defaults to True.
        is_causal (bool, optional): Whether the attention is causal. Defaults to False.
        config (Optional[Wav2Vec2Config], optional): The configuration object. Defaults to None.

    Returns:
        None

    Raises:
        ValueError: If embed_dim is not divisible by num_heads.
    """
    super().__init__()
    self.embed_dim = embed_dim
    self.num_heads = num_heads
    self.dropout = dropout
    self.head_dim = embed_dim // num_heads
    self.config = config

    if (self.head_dim * num_heads) != self.embed_dim:
        raise ValueError(
            f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}"
            f" and `num_heads`: {num_heads})."
        )
    self.scaling = self.head_dim**-0.5
    self.is_decoder = is_decoder
    self.is_causal = is_causal

    self.k_proj = nn.Dense(embed_dim, embed_dim, has_bias=bias)
    self.v_proj = nn.Dense(embed_dim, embed_dim, has_bias=bias)
    self.q_proj = nn.Dense(embed_dim, embed_dim, has_bias=bias)
    self.out_proj = nn.Dense(embed_dim, embed_dim, has_bias=bias)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Attention.construct(hidden_states, key_value_states=None, past_key_value=None, attention_mask=None, layer_head_mask=None, output_attentions=False)

Input shape: Batch x Time x Channel

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    hidden_states: Tensor,
    key_value_states: Optional[Tensor] = None,
    past_key_value: Optional[Tuple[Tensor]] = None,
    attention_mask: Optional[Tensor] = None,
    layer_head_mask: Optional[Tensor] = None,
    output_attentions: bool = False,
) -> Tuple[Tensor, Optional[Tensor], Optional[Tuple[Tensor]]]:
    """Input shape: Batch x Time x Channel"""
    # if key_value_states are provided this layer is used as a cross-attention layer
    # for the decoder
    is_cross_attention = key_value_states is not None

    bsz, tgt_len, _ = hidden_states.shape

    # get query proj
    query_states = self.q_proj(hidden_states) * self.scaling
    # get key, value proj
    # `past_key_value[0].shape[2] == key_value_states.shape[1]`
    # is checking that the `sequence_length` of the `past_key_value` is the same as
    # the provided `key_value_states` to support prefix tuning
    if (
        is_cross_attention
        and past_key_value is not None
        and past_key_value[0].shape[2] == key_value_states.shape[1]
    ):
        # reuse k,v, cross_attentions
        key_states = past_key_value[0]
        value_states = past_key_value[1]
    elif is_cross_attention:
        # cross_attentions
        key_states = self._shape(self.k_proj(key_value_states), -1, bsz)
        value_states = self._shape(self.v_proj(key_value_states), -1, bsz)
    elif past_key_value is not None:
        # reuse k, v, self_attention
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)
        key_states = ops.cat([past_key_value[0], key_states], axis=2)
        value_states = ops.cat([past_key_value[1], value_states], axis=2)
    else:
        # self_attention
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

    if self.is_decoder:
        # if cross_attention save Tuple(Tensor, Tensor) of all cross attention key/value_states.
        # Further calls to cross_attention layer can then reuse all cross-attention
        # key/value_states (first "if" case)
        # if uni-directional self-attention (decoder) save Tuple(Tensor, Tensor) of
        # all previous decoder key/value_states. Further calls to uni-directional self-attention
        # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
        # if encoder bi-directional self-attention `past_key_value` is always `None`
        past_key_value = (key_states, value_states)

    proj_shape = (bsz * self.num_heads, -1, self.head_dim)
    query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
    key_states = key_states.reshape(*proj_shape)
    value_states = value_states.reshape(*proj_shape)

    src_len = key_states.shape[1]
    attn_weights = ops.bmm(query_states, key_states.swapaxes(1, 2))

    if attn_weights.shape != (bsz * self.num_heads, tgt_len, src_len):
        raise ValueError(
            f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
            f" {attn_weights.shape}"
        )

    if attention_mask is not None:
        if attention_mask.shape != (bsz, 1, tgt_len, src_len):
            raise ValueError(
                f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.shape}"
            )
        attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
        attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

    attn_weights = ops.softmax(attn_weights, axis=-1)

    if layer_head_mask is not None:
        if layer_head_mask.shape != (self.num_heads,):
            raise ValueError(
                f"Head mask for a single layer should be of size {(self.num_heads,)}, but is"
                f" {layer_head_mask.shape}"
            )
        attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
        attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

    if output_attentions:
        # this operation is a bit awkward, but it's required to
        # make sure that attn_weights keeps its gradient.
        # In order to do so, attn_weights have to be reshaped
        # twice and have to be reused in the following
        attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
        attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
    else:
        attn_weights_reshaped = None

    attn_probs = ops.dropout(attn_weights, p=self.dropout, training=self.training)

    attn_output = ops.bmm(attn_probs, value_states)

    if attn_output.shape != (bsz * self.num_heads, tgt_len, self.head_dim):
        raise ValueError(
            f"`attn_output` should be of size {(bsz * self.num_heads, tgt_len, self.head_dim)}, but is"
            f" {attn_output.shape}"
        )

    attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
    attn_output = attn_output.swapaxes(1, 2)

    # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be
    # partitioned across GPUs when using tensor-parallelism.
    attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)

    attn_output = self.out_proj(attn_output)

    return attn_output, attn_weights_reshaped, past_key_value

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2AttnAdapterLayer

Bases: Cell

This class represents a single layer of an attention adapter module in the Wav2Vec2 model. The adapter module is designed to enhance the training throughput by directly implementing the adapter modules with 3D tensor weights as parameters, without using ModuleList.

ATTRIBUTE	DESCRIPTION
input_dim	The dimension of the input tensor to the adapter module. TYPE: int
hidden_dim	The hidden dimension of the adapter module. TYPE: int
norm	A layer normalization module to normalize the hidden states. TYPE: LayerNorm
linear_1	A linear transformation module that maps the hidden states to the input dimension. TYPE: Dense
act_fn	An activation function module that applies the ReLU activation to the hidden states. TYPE: ReLU
linear_2	A linear transformation module that maps the hidden states back to the hidden dimension. TYPE: Dense

METHOD	DESCRIPTION
construct	Applies the attention adapter layer operations to the input hidden states tensor. Args: hidden_states (Tensor): The input hidden states tensor. Returns: Tensor: The output hidden states tensor after applying the attention adapter layer operations.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2AttnAdapterLayer(nn.Cell):

    """
    This class represents a single layer of an attention adapter module in the Wav2Vec2 model. The adapter module is
    designed to enhance the training throughput by directly implementing the adapter modules with 3D tensor weights as
    parameters, without using ModuleList.

    Attributes:
        input_dim (int): The dimension of the input tensor to the adapter module.
        hidden_dim (int): The hidden dimension of the adapter module.
        norm (nn.LayerNorm): A layer normalization module to normalize the hidden states.
        linear_1 (nn.Dense): A linear transformation module that maps the hidden states to the input dimension.
        act_fn (nn.ReLU): An activation function module that applies the ReLU activation to the hidden states.
        linear_2 (nn.Dense): A linear transformation module that maps the hidden states back to the hidden dimension.

    Methods:
        construct:
            Applies the attention adapter layer operations to the input hidden states tensor.

            Args:

            -  hidden_states (Tensor): The input hidden states tensor.
            Returns:

            - Tensor: The output hidden states tensor after applying the attention adapter layer operations.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed
        up training throughput.
        """
        super().__init__()
        self.input_dim = config.adapter_attn_dim
        self.hidden_dim = config.hidden_size

        self.norm = nn.LayerNorm(self.hidden_dim)
        self.linear_1 = nn.Dense(self.hidden_dim, self.input_dim)
        self.act_fn = nn.ReLU()
        self.linear_2 = nn.Dense(self.input_dim, self.hidden_dim)

    def construct(self, hidden_states: Tensor):
        """
        Method: construct

        Description:
        Constructs the adaptation layer for the Wav2Vec2AttnAdapterModel.

        Args:
            self: (Wav2Vec2AttnAdapterLayer) The instance of the Wav2Vec2AttnAdapterLayer class.
            hidden_states: (Tensor) The input hidden states to be processed by the adaptation layer.

        Returns:
            None

        Raises:
            ValueError: If the input hidden_states tensor is empty or invalid.
            TypeError: If the input hidden_states is not of type Tensor.
        """
        hidden_states = self.norm(hidden_states)
        hidden_states = self.linear_1(hidden_states)
        hidden_states = self.act_fn(hidden_states)
        hidden_states = self.linear_2(hidden_states)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2AttnAdapterLayer.__init__(config)

Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed
    up training throughput.
    """
    super().__init__()
    self.input_dim = config.adapter_attn_dim
    self.hidden_dim = config.hidden_size

    self.norm = nn.LayerNorm(self.hidden_dim)
    self.linear_1 = nn.Dense(self.hidden_dim, self.input_dim)
    self.act_fn = nn.ReLU()
    self.linear_2 = nn.Dense(self.input_dim, self.hidden_dim)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2AttnAdapterLayer.construct(hidden_states)

Description: Constructs the adaptation layer for the Wav2Vec2AttnAdapterModel.

PARAMETER	DESCRIPTION
self	(Wav2Vec2AttnAdapterLayer) The instance of the Wav2Vec2AttnAdapterLayer class.
hidden_states	(Tensor) The input hidden states to be processed by the adaptation layer. TYPE: Tensor

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
ValueError	If the input hidden_states tensor is empty or invalid.
TypeError	If the input hidden_states is not of type Tensor.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states: Tensor):
    """
    Method: construct

    Description:
    Constructs the adaptation layer for the Wav2Vec2AttnAdapterModel.

    Args:
        self: (Wav2Vec2AttnAdapterLayer) The instance of the Wav2Vec2AttnAdapterLayer class.
        hidden_states: (Tensor) The input hidden states to be processed by the adaptation layer.

    Returns:
        None

    Raises:
        ValueError: If the input hidden_states tensor is empty or invalid.
        TypeError: If the input hidden_states is not of type Tensor.
    """
    hidden_states = self.norm(hidden_states)
    hidden_states = self.linear_1(hidden_states)
    hidden_states = self.act_fn(hidden_states)
    hidden_states = self.linear_2(hidden_states)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Encoder

Bases: Cell

A class representing the Wav2Vec2Encoder in the Wav2Vec2 model architecture.

The Wav2Vec2Encoder is responsible for encoding the input hidden states with positional embeddings and applying a series of Wav2Vec2EncoderLayer for feature extraction.

ATTRIBUTE	DESCRIPTION
config	The configuration for the Wav2Vec2 model. TYPE: Wav2Vec2Config
pos_conv_embed	The positional convolutional embedding layer. TYPE: Wav2Vec2PositionalConvEmbedding
layer_norm	The layer normalization layer. TYPE: LayerNorm
dropout	The dropout layer. TYPE: Dropout
layers	The list of Wav2Vec2EncoderLayer instances. TYPE: CellList

METHOD

DESCRIPTION

construct

Applies the Wav2Vec2Encoder layer-wise to the hidden states. Args: hidden_states (Tensor): The input hidden states. attention_mask (Optional[Tensor], optional): The attention mask tensor. Defaults to None. output_attentions (bool, optional): Whether to output the attentions. Defaults to False. output_hidden_states (bool, optional): Whether to output the hidden states. Defaults to False. return_dict (bool, optional): Whether to return a BaseModelOutput dictionary. Defaults to True. Returns: BaseModelOutput or Tuple[Tensor, Tuple[Tensor], Tuple[Tensor]]: The encoded hidden states, all hidden states (if output_hidden_states=True), and all self-attentions (if output_attentions=True).

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2Encoder(nn.Cell):

    """
    A class representing the Wav2Vec2Encoder in the Wav2Vec2 model architecture.

    The Wav2Vec2Encoder is responsible for encoding the input hidden states with positional embeddings and applying
    a series of Wav2Vec2EncoderLayer for feature extraction.

    Attributes:
        config (Wav2Vec2Config): The configuration for the Wav2Vec2 model.
        pos_conv_embed (Wav2Vec2PositionalConvEmbedding): The positional convolutional embedding layer.
        layer_norm (nn.LayerNorm): The layer normalization layer.
        dropout (nn.Dropout): The dropout layer.
        layers (nn.CellList): The list of Wav2Vec2EncoderLayer instances.

    Methods:
        construct(hidden_states, attention_mask=None, output_attentions=False, output_hidden_states=False, return_dict=True):
            Applies the Wav2Vec2Encoder layer-wise to the hidden states.

            Args:

            - hidden_states (Tensor): The input hidden states.
            - attention_mask (Optional[Tensor], optional): The attention mask tensor. Defaults to None.
            - output_attentions (bool, optional): Whether to output the attentions. Defaults to False.
            - output_hidden_states (bool, optional): Whether to output the hidden states. Defaults to False.
            - return_dict (bool, optional): Whether to return a BaseModelOutput dictionary. Defaults to True.

            Returns:

            - BaseModelOutput or Tuple[Tensor, Tuple[Tensor], Tuple[Tensor]]: The encoded hidden states, all hidden
            states (if output_hidden_states=True), and all self-attentions (if output_attentions=True).
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes the Wav2Vec2Encoder class.

        Args:
            self: The instance of the class.
            config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
                for the encoder. It specifies the configuration for the Wav2Vec2 model, such as hidden size,
                layer normalization epsilon, hidden dropout probability, and the number of hidden layers.

        Returns:
            None.

        Raises:
            None: This method does not raise any exceptions explicitly. However, exceptions may be raised during the
                initialization of the Wav2Vec2PositionalConvEmbedding, nn.LayerNorm, nn.Dropout, and nn.CellList objects.
        """
        super().__init__()
        self.config = config
        self.pos_conv_embed = Wav2Vec2PositionalConvEmbedding(config)
        self.layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)
        self.dropout = nn.Dropout(p=config.hidden_dropout)
        self.layers = nn.CellList([Wav2Vec2EncoderLayer(config) for _ in range(config.num_hidden_layers)])

    def construct(
        self,
        hidden_states: Tensor,
        attention_mask: Optional[Tensor] = None,
        output_attentions: bool = False,
        output_hidden_states: bool = False,
        return_dict: bool = True,
    ):
        """
        Constructs the Wav2Vec2Encoder.

        Args:
            self (Wav2Vec2Encoder): The instance of the Wav2Vec2Encoder class.
            hidden_states (Tensor): The input hidden states. A tensor of shape (batch_size, sequence_length, hidden_size).
            attention_mask (Optional[Tensor]): An optional tensor specifying the attention mask. Defaults to None.
            output_attentions (bool): Whether to output attentions. Defaults to False.
            output_hidden_states (bool): Whether to output hidden states. Defaults to False.
            return_dict (bool): Whether to return a dictionary. Defaults to True.

        Returns:
            None.

        Raises:
            ValueError: If the hidden_states tensor has invalid shape or type.
            ValueError: If the attention_mask tensor has invalid shape or type.
            TypeError: If the output_attentions or output_hidden_states parameters are not of type bool.
            TypeError: If the return_dict parameter is not of type bool.
        """
        all_hidden_states = () if output_hidden_states else None
        all_self_attentions = () if output_attentions else None

        if attention_mask is not None:
            # make sure padded tokens output 0
            expand_attention_mask = attention_mask.unsqueeze(-1).repeat(1, 1, hidden_states.shape[2])
            hidden_states[~expand_attention_mask] = 0

            # extend attention_mask
            attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
            attention_mask = attention_mask * finfo(hidden_states.dtype, 'min')
            attention_mask = attention_mask.expand(
                attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
            )

        position_embeddings = self.pos_conv_embed(hidden_states)
        hidden_states = hidden_states + position_embeddings
        hidden_states = self.layer_norm(hidden_states)
        hidden_states = self.dropout(hidden_states)

        for layer in self.layers:
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
            dropout_probability = ops.rand([])

            skip_the_layer = self.training and (dropout_probability < self.config.layerdrop)
            if not skip_the_layer:
                layer_outputs = layer(
                    hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
                )
                hidden_states = layer_outputs[0]

            if skip_the_layer:
                layer_outputs = (None, None)

            if output_attentions:
                all_self_attentions = all_self_attentions + (layer_outputs[1],)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)
        return BaseModelOutput(
            last_hidden_state=hidden_states,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
        )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Encoder.__init__(config)

Initializes the Wav2Vec2Encoder class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An instance of the Wav2Vec2Config class containing the configuration parameters for the encoder. It specifies the configuration for the Wav2Vec2 model, such as hidden size, layer normalization epsilon, hidden dropout probability, and the number of hidden layers. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
None	This method does not raise any exceptions explicitly. However, exceptions may be raised during the initialization of the Wav2Vec2PositionalConvEmbedding, nn.LayerNorm, nn.Dropout, and nn.CellList objects.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes the Wav2Vec2Encoder class.

    Args:
        self: The instance of the class.
        config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
            for the encoder. It specifies the configuration for the Wav2Vec2 model, such as hidden size,
            layer normalization epsilon, hidden dropout probability, and the number of hidden layers.

    Returns:
        None.

    Raises:
        None: This method does not raise any exceptions explicitly. However, exceptions may be raised during the
            initialization of the Wav2Vec2PositionalConvEmbedding, nn.LayerNorm, nn.Dropout, and nn.CellList objects.
    """
    super().__init__()
    self.config = config
    self.pos_conv_embed = Wav2Vec2PositionalConvEmbedding(config)
    self.layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)
    self.dropout = nn.Dropout(p=config.hidden_dropout)
    self.layers = nn.CellList([Wav2Vec2EncoderLayer(config) for _ in range(config.num_hidden_layers)])

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Encoder.construct(hidden_states, attention_mask=None, output_attentions=False, output_hidden_states=False, return_dict=True)

Constructs the Wav2Vec2Encoder.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2Encoder class. TYPE: Wav2Vec2Encoder
hidden_states	The input hidden states. A tensor of shape (batch_size, sequence_length, hidden_size). TYPE: Tensor
attention_mask	An optional tensor specifying the attention mask. Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
output_attentions	Whether to output attentions. Defaults to False. TYPE: bool DEFAULT: False
output_hidden_states	Whether to output hidden states. Defaults to False. TYPE: bool DEFAULT: False
return_dict	Whether to return a dictionary. Defaults to True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the hidden_states tensor has invalid shape or type.
ValueError	If the attention_mask tensor has invalid shape or type.
TypeError	If the output_attentions or output_hidden_states parameters are not of type bool.
TypeError	If the return_dict parameter is not of type bool.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    hidden_states: Tensor,
    attention_mask: Optional[Tensor] = None,
    output_attentions: bool = False,
    output_hidden_states: bool = False,
    return_dict: bool = True,
):
    """
    Constructs the Wav2Vec2Encoder.

    Args:
        self (Wav2Vec2Encoder): The instance of the Wav2Vec2Encoder class.
        hidden_states (Tensor): The input hidden states. A tensor of shape (batch_size, sequence_length, hidden_size).
        attention_mask (Optional[Tensor]): An optional tensor specifying the attention mask. Defaults to None.
        output_attentions (bool): Whether to output attentions. Defaults to False.
        output_hidden_states (bool): Whether to output hidden states. Defaults to False.
        return_dict (bool): Whether to return a dictionary. Defaults to True.

    Returns:
        None.

    Raises:
        ValueError: If the hidden_states tensor has invalid shape or type.
        ValueError: If the attention_mask tensor has invalid shape or type.
        TypeError: If the output_attentions or output_hidden_states parameters are not of type bool.
        TypeError: If the return_dict parameter is not of type bool.
    """
    all_hidden_states = () if output_hidden_states else None
    all_self_attentions = () if output_attentions else None

    if attention_mask is not None:
        # make sure padded tokens output 0
        expand_attention_mask = attention_mask.unsqueeze(-1).repeat(1, 1, hidden_states.shape[2])
        hidden_states[~expand_attention_mask] = 0

        # extend attention_mask
        attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
        attention_mask = attention_mask * finfo(hidden_states.dtype, 'min')
        attention_mask = attention_mask.expand(
            attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
        )

    position_embeddings = self.pos_conv_embed(hidden_states)
    hidden_states = hidden_states + position_embeddings
    hidden_states = self.layer_norm(hidden_states)
    hidden_states = self.dropout(hidden_states)

    for layer in self.layers:
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
        dropout_probability = ops.rand([])

        skip_the_layer = self.training and (dropout_probability < self.config.layerdrop)
        if not skip_the_layer:
            layer_outputs = layer(
                hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
            )
            hidden_states = layer_outputs[0]

        if skip_the_layer:
            layer_outputs = (None, None)

        if output_attentions:
            all_self_attentions = all_self_attentions + (layer_outputs[1],)

    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    if not return_dict:
        return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)
    return BaseModelOutput(
        last_hidden_state=hidden_states,
        hidden_states=all_hidden_states,
        attentions=all_self_attentions,
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2EncoderLayer

Bases: Cell

A class representing an encoder layer of the Wav2Vec2 model.

The Wav2Vec2EncoderLayer class inherits from the nn.Cell class and implements the functionality of a single encoder layer in the Wav2Vec2 model architecture. It consists of multiple sub-modules, including an attention mechanism, dropout layers, layer normalization, and a feed-forward neural network.

ATTRIBUTE	DESCRIPTION
attention	The attention mechanism used in the layer. TYPE: Wav2Vec2Attention
dropout	The dropout layer applied to the hidden states. TYPE: Dropout
layer_norm	The layer normalization applied to the hidden states. TYPE: LayerNorm
feed_forward	The feed-forward neural network used in the layer. TYPE: Wav2Vec2FeedForward
final_layer_norm	The final layer normalization applied to the hidden states. TYPE: LayerNorm

METHOD

DESCRIPTION

construct

Applies the forward pass of the encoder layer. Args: hidden_states (Tensor): The input hidden states. attention_mask (Tensor, optional): The attention mask to apply to the attention mechanism (default: None). output_attentions (bool, optional): Whether to return the attention weights (default: False). Returns: outputs (tuple): A tuple containing the output hidden states. If output_attentions is True, the tuple also contains the attention weights.

Note

The Wav2Vec2EncoderLayer class is designed to be used within the Wav2Vec2Encoder class, which stacks multiple encoder layers to form the complete Wav2Vec2 model.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2EncoderLayer(nn.Cell):

    """A class representing an encoder layer of the Wav2Vec2 model.

    The Wav2Vec2EncoderLayer class inherits from the nn.Cell class and implements the functionality of a single encoder
    layer in the Wav2Vec2 model architecture. It consists of multiple sub-modules, including an attention mechanism,
    dropout layers, layer normalization, and a feed-forward neural network.

    Attributes:
        attention (Wav2Vec2Attention): The attention mechanism used in the layer.
        dropout (nn.Dropout): The dropout layer applied to the hidden states.
        layer_norm (nn.LayerNorm): The layer normalization applied to the hidden states.
        feed_forward (Wav2Vec2FeedForward): The feed-forward neural network used in the layer.
        final_layer_norm (nn.LayerNorm): The final layer normalization applied to the hidden states.

    Methods:
        construct(hidden_states, attention_mask=None, output_attentions=False):
            Applies the forward pass of the encoder layer.

            Args:

            - hidden_states (Tensor): The input hidden states.
            - attention_mask (Tensor, optional): The attention mask to apply to the attention mechanism (default: None).
            - output_attentions (bool, optional): Whether to return the attention weights (default: False).

            Returns:

            - outputs (tuple): A tuple containing the output hidden states. If output_attentions is True, the tuple
            also contains the attention weights.

    Note:
        The Wav2Vec2EncoderLayer class is designed to be used within the Wav2Vec2Encoder class, which stacks multiple
        encoder layers to form the complete Wav2Vec2 model.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a Wav2Vec2EncoderLayer instance.

        Args:
            self (Wav2Vec2EncoderLayer): The instance of the Wav2Vec2EncoderLayer class.
            config (Wav2Vec2Config):
                An instance of Wav2Vec2Config containing configuration parameters for the encoder layer.

                - Wav2Vec2Config.hidden_size (int): The hidden size for the encoder layer.
                - Wav2Vec2Config.num_attention_heads (int): The number of attention heads in the attention mechanism.
                - Wav2Vec2Config.attention_dropout (float): The dropout probability for the attention mechanism.
                - Wav2Vec2Config.hidden_dropout (float): The dropout probability for the hidden layers.
                - Wav2Vec2Config.layer_norm_eps (float): The epsilon value for layer normalization.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.attention = Wav2Vec2Attention(
            embed_dim=config.hidden_size,
            num_heads=config.num_attention_heads,
            dropout=config.attention_dropout,
            is_decoder=False,
        )
        self.dropout = nn.Dropout(p=config.hidden_dropout)
        self.layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)
        self.feed_forward = Wav2Vec2FeedForward(config)
        self.final_layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)

    def construct(self, hidden_states, attention_mask=None, output_attentions=False):
        """
        Constructs the Wav2Vec2EncoderLayer.

        This method applies the Wav2Vec2EncoderLayer to the input hidden_states. It performs attention, residual
        connections, layer normalization, feed-forward, and final layer normalization.

        Args:
            self (Wav2Vec2EncoderLayer): The instance of the Wav2Vec2EncoderLayer class.
            hidden_states (torch.Tensor): The input hidden states of shape (batch_size, sequence_length, hidden_size).
            attention_mask (torch.Tensor, optional): The attention mask of shape (batch_size, sequence_length).
                Defaults to None.
            output_attentions (bool, optional): Whether to output the attention weights. Defaults to False.

        Returns:
            tuple: A tuple containing the hidden states of shape (batch_size, sequence_length, hidden_size).
                If output_attentions is True, the tuple also contains the attention weights of shape (batch_size,
                num_heads, sequence_length, sequence_length).

        Raises:
            None
        """
        attn_residual = hidden_states
        hidden_states, attn_weights, _ = self.attention(
            hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
        )
        hidden_states = self.dropout(hidden_states)
        hidden_states = attn_residual + hidden_states

        hidden_states = self.layer_norm(hidden_states)
        hidden_states = hidden_states + self.feed_forward(hidden_states)
        hidden_states = self.final_layer_norm(hidden_states)

        outputs = (hidden_states,)

        if output_attentions:
            outputs += (attn_weights,)

        return outputs

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2EncoderLayer.__init__(config)

Initializes a Wav2Vec2EncoderLayer instance.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2EncoderLayer class. TYPE: Wav2Vec2EncoderLayer
config	An instance of Wav2Vec2Config containing configuration parameters for the encoder layer. Wav2Vec2Config.hidden_size (int): The hidden size for the encoder layer. Wav2Vec2Config.num_attention_heads (int): The number of attention heads in the attention mechanism. Wav2Vec2Config.attention_dropout (float): The dropout probability for the attention mechanism. Wav2Vec2Config.hidden_dropout (float): The dropout probability for the hidden layers. Wav2Vec2Config.layer_norm_eps (float): The epsilon value for layer normalization. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a Wav2Vec2EncoderLayer instance.

    Args:
        self (Wav2Vec2EncoderLayer): The instance of the Wav2Vec2EncoderLayer class.
        config (Wav2Vec2Config):
            An instance of Wav2Vec2Config containing configuration parameters for the encoder layer.

            - Wav2Vec2Config.hidden_size (int): The hidden size for the encoder layer.
            - Wav2Vec2Config.num_attention_heads (int): The number of attention heads in the attention mechanism.
            - Wav2Vec2Config.attention_dropout (float): The dropout probability for the attention mechanism.
            - Wav2Vec2Config.hidden_dropout (float): The dropout probability for the hidden layers.
            - Wav2Vec2Config.layer_norm_eps (float): The epsilon value for layer normalization.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.attention = Wav2Vec2Attention(
        embed_dim=config.hidden_size,
        num_heads=config.num_attention_heads,
        dropout=config.attention_dropout,
        is_decoder=False,
    )
    self.dropout = nn.Dropout(p=config.hidden_dropout)
    self.layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)
    self.feed_forward = Wav2Vec2FeedForward(config)
    self.final_layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2EncoderLayer.construct(hidden_states, attention_mask=None, output_attentions=False)

Constructs the Wav2Vec2EncoderLayer.

This method applies the Wav2Vec2EncoderLayer to the input hidden_states. It performs attention, residual connections, layer normalization, feed-forward, and final layer normalization.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2EncoderLayer class. TYPE: Wav2Vec2EncoderLayer
hidden_states	The input hidden states of shape (batch_size, sequence_length, hidden_size). TYPE: Tensor
attention_mask	The attention mask of shape (batch_size, sequence_length). Defaults to None. TYPE: Tensor DEFAULT: None
output_attentions	Whether to output the attention weights. Defaults to False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
tuple	A tuple containing the hidden states of shape (batch_size, sequence_length, hidden_size). If output_attentions is True, the tuple also contains the attention weights of shape (batch_size, num_heads, sequence_length, sequence_length).

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states, attention_mask=None, output_attentions=False):
    """
    Constructs the Wav2Vec2EncoderLayer.

    This method applies the Wav2Vec2EncoderLayer to the input hidden_states. It performs attention, residual
    connections, layer normalization, feed-forward, and final layer normalization.

    Args:
        self (Wav2Vec2EncoderLayer): The instance of the Wav2Vec2EncoderLayer class.
        hidden_states (torch.Tensor): The input hidden states of shape (batch_size, sequence_length, hidden_size).
        attention_mask (torch.Tensor, optional): The attention mask of shape (batch_size, sequence_length).
            Defaults to None.
        output_attentions (bool, optional): Whether to output the attention weights. Defaults to False.

    Returns:
        tuple: A tuple containing the hidden states of shape (batch_size, sequence_length, hidden_size).
            If output_attentions is True, the tuple also contains the attention weights of shape (batch_size,
            num_heads, sequence_length, sequence_length).

    Raises:
        None
    """
    attn_residual = hidden_states
    hidden_states, attn_weights, _ = self.attention(
        hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
    )
    hidden_states = self.dropout(hidden_states)
    hidden_states = attn_residual + hidden_states

    hidden_states = self.layer_norm(hidden_states)
    hidden_states = hidden_states + self.feed_forward(hidden_states)
    hidden_states = self.final_layer_norm(hidden_states)

    outputs = (hidden_states,)

    if output_attentions:
        outputs += (attn_weights,)

    return outputs

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2EncoderLayerStableLayerNorm

Bases: Cell

This class represents an encoder layer in the Wav2Vec2 model with stable layer normalization. It inherits from the nn.Cell class.

ATTRIBUTE	DESCRIPTION
attention	An instance of the Wav2Vec2Attention class for attention mechanism. TYPE: Wav2Vec2Attention
dropout	An instance of the nn.Dropout class for dropout regularization. TYPE: Dropout
layer_norm	An instance of the nn.LayerNorm class for stable layer normalization. TYPE: LayerNorm
feed_forward	An instance of the Wav2Vec2FeedForward class for feed-forward layer. TYPE: Wav2Vec2FeedForward
final_layer_norm	An instance of the nn.LayerNorm class for stable layer normalization of final output. TYPE: LayerNorm
adapter_layer	An instance of the Wav2Vec2AttnAdapterLayer class for adapter layer, if provided. None otherwise. TYPE: Wav2Vec2AttnAdapterLayer or None

METHOD

DESCRIPTION

construct

Applies the encoder layer operations on the input hidden states. Args: hidden_states (Tensor): The input hidden states. attention_mask (Optional[Tensor]): The attention mask tensor, if provided. Defaults to None. output_attentions (bool): Whether to output attention weights. Defaults to False. Returns: Tuple[Tensor, Union[Tensor, None]]: A tuple containing the final hidden states and optionally the attention weights, if output_attentions is True.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2EncoderLayerStableLayerNorm(nn.Cell):

    """
    This class represents an encoder layer in the Wav2Vec2 model with stable layer normalization.
    It inherits from the nn.Cell class.

    Attributes:
        attention (Wav2Vec2Attention): An instance of the Wav2Vec2Attention class for attention mechanism.
        dropout (nn.Dropout): An instance of the nn.Dropout class for dropout regularization.
        layer_norm (nn.LayerNorm): An instance of the nn.LayerNorm class for stable layer normalization.
        feed_forward (Wav2Vec2FeedForward): An instance of the Wav2Vec2FeedForward class for feed-forward layer.
        final_layer_norm (nn.LayerNorm): An instance of the nn.LayerNorm class for stable layer normalization of final
            output.
        adapter_layer (Wav2Vec2AttnAdapterLayer or None): An instance of the Wav2Vec2AttnAdapterLayer class for adapter
            layer, if provided. None otherwise.

    Methods:
        construct:
            Applies the encoder layer operations on the input hidden states.

            Args:

            - hidden_states (Tensor): The input hidden states.
            - attention_mask (Optional[Tensor]): The attention mask tensor, if provided. Defaults to None.
            - output_attentions (bool): Whether to output attention weights. Defaults to False.

            Returns:

            - Tuple[Tensor, Union[Tensor, None]]: A tuple containing the final hidden states and optionally the
            attention weights, if output_attentions is True.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a new instance of the Wav2Vec2EncoderLayerStableLayerNorm class.

        Args:
            self: The instance of the class.
            config (Wav2Vec2Config): The configuration object containing the settings for the encoder layer.
                It should be an instance of the Wav2Vec2Config class.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.attention = Wav2Vec2Attention(
            embed_dim=config.hidden_size,
            num_heads=config.num_attention_heads,
            dropout=config.attention_dropout,
            is_decoder=False,
        )
        self.dropout = nn.Dropout(p=config.hidden_dropout)
        self.layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)
        self.feed_forward = Wav2Vec2FeedForward(config)
        self.final_layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)

        if getattr(config, "adapter_attn_dim", None) is not None:
            self.adapter_layer = Wav2Vec2AttnAdapterLayer(config)
        else:
            self.adapter_layer = None

    def construct(
        self,
        hidden_states: Tensor,
        attention_mask: Optional[Tensor] = None,
        output_attentions: bool = False,
    ):
        """
        Constructs the Wav2Vec2EncoderLayerStableLayerNorm.

        Args:
            self: Instance of the Wav2Vec2EncoderLayerStableLayerNorm class.
            hidden_states (Tensor): The input hidden states to be processed by the encoder layer.
            attention_mask (Optional[Tensor]): Optional tensor representing the attention mask.
                Defaults to None. If provided, masks certain elements in the attention computation.
            output_attentions (bool): Flag indicating whether to output attention weights during computation.
                Defaults to False.

        Returns:
            Tuple: A tuple containing the processed hidden states and optionally the attention weights.

        Raises:
            None.
        """
        attn_residual = hidden_states
        hidden_states = self.layer_norm(hidden_states)
        hidden_states, attn_weights, _ = self.attention(
            hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
        )
        hidden_states = self.dropout(hidden_states)
        hidden_states = attn_residual + hidden_states
        hidden_states = hidden_states + self.feed_forward(self.final_layer_norm(hidden_states))

        if self.adapter_layer is not None:
            hidden_states = hidden_states + self.adapter_layer(hidden_states)

        outputs = (hidden_states,)

        if output_attentions:
            outputs += (attn_weights,)

        return outputs

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2EncoderLayerStableLayerNorm.__init__(config)

Initializes a new instance of the Wav2Vec2EncoderLayerStableLayerNorm class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	The configuration object containing the settings for the encoder layer. It should be an instance of the Wav2Vec2Config class. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a new instance of the Wav2Vec2EncoderLayerStableLayerNorm class.

    Args:
        self: The instance of the class.
        config (Wav2Vec2Config): The configuration object containing the settings for the encoder layer.
            It should be an instance of the Wav2Vec2Config class.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.attention = Wav2Vec2Attention(
        embed_dim=config.hidden_size,
        num_heads=config.num_attention_heads,
        dropout=config.attention_dropout,
        is_decoder=False,
    )
    self.dropout = nn.Dropout(p=config.hidden_dropout)
    self.layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)
    self.feed_forward = Wav2Vec2FeedForward(config)
    self.final_layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)

    if getattr(config, "adapter_attn_dim", None) is not None:
        self.adapter_layer = Wav2Vec2AttnAdapterLayer(config)
    else:
        self.adapter_layer = None

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2EncoderLayerStableLayerNorm.construct(hidden_states, attention_mask=None, output_attentions=False)

Constructs the Wav2Vec2EncoderLayerStableLayerNorm.

PARAMETER	DESCRIPTION
self	Instance of the Wav2Vec2EncoderLayerStableLayerNorm class.
hidden_states	The input hidden states to be processed by the encoder layer. TYPE: Tensor
attention_mask	Optional tensor representing the attention mask. Defaults to None. If provided, masks certain elements in the attention computation. TYPE: Optional[Tensor] DEFAULT: None
output_attentions	Flag indicating whether to output attention weights during computation. Defaults to False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
Tuple	A tuple containing the processed hidden states and optionally the attention weights.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    hidden_states: Tensor,
    attention_mask: Optional[Tensor] = None,
    output_attentions: bool = False,
):
    """
    Constructs the Wav2Vec2EncoderLayerStableLayerNorm.

    Args:
        self: Instance of the Wav2Vec2EncoderLayerStableLayerNorm class.
        hidden_states (Tensor): The input hidden states to be processed by the encoder layer.
        attention_mask (Optional[Tensor]): Optional tensor representing the attention mask.
            Defaults to None. If provided, masks certain elements in the attention computation.
        output_attentions (bool): Flag indicating whether to output attention weights during computation.
            Defaults to False.

    Returns:
        Tuple: A tuple containing the processed hidden states and optionally the attention weights.

    Raises:
        None.
    """
    attn_residual = hidden_states
    hidden_states = self.layer_norm(hidden_states)
    hidden_states, attn_weights, _ = self.attention(
        hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
    )
    hidden_states = self.dropout(hidden_states)
    hidden_states = attn_residual + hidden_states
    hidden_states = hidden_states + self.feed_forward(self.final_layer_norm(hidden_states))

    if self.adapter_layer is not None:
        hidden_states = hidden_states + self.adapter_layer(hidden_states)

    outputs = (hidden_states,)

    if output_attentions:
        outputs += (attn_weights,)

    return outputs

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2EncoderStableLayerNorm

Bases: Cell

Wav2Vec2EncoderStableLayerNorm is a Python class that represents an encoder with stable layer normalization for the Wav2Vec2 model. This class inherits from the nn.Cell module.

This class initializes with a Wav2Vec2Config object and constructs a series of encoder layers with stable layer normalization. The encoder layers operate on the input hidden states and optionally apply attention masks, producing hidden states with added positional embeddings and layer normalization.

The construct method applies the encoder layers to the input hidden states, handling attention masks, outputting hidden states, and attentions based on the specified configurations.

This class provides functionalities for building and using a stable layer normalization encoder for the Wav2Vec2 model, supporting various output options and configurations.

For detailed information on the class methods and usage, please refer to the specific method docstrings within the source code.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2EncoderStableLayerNorm(nn.Cell):

    """
    Wav2Vec2EncoderStableLayerNorm is a Python class that represents an encoder with stable layer normalization for
    the Wav2Vec2 model. This class inherits from the nn.Cell module.

    This class initializes with a Wav2Vec2Config object and constructs a series of encoder layers with stable
    layer normalization. The encoder layers operate on the input hidden states and optionally apply
    attention masks, producing hidden states with added positional embeddings and layer normalization.

    The construct method applies the encoder layers to the input hidden states, handling attention masks,
    outputting hidden states, and attentions based on the specified configurations.

    This class provides functionalities for building and using a stable layer normalization encoder for the Wav2Vec2
    model, supporting various output options and configurations.

    For detailed information on the class methods and usage, please refer to the specific method docstrings within
    the source code.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes an instance of the Wav2Vec2EncoderStableLayerNorm class.

        Args:
            self: The object instance.
            config (Wav2Vec2Config): The configuration object for the Wav2Vec2 model.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.config = config
        self.pos_conv_embed = Wav2Vec2PositionalConvEmbedding(config)
        self.layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)
        self.dropout = nn.Dropout(p=config.hidden_dropout)
        self.layers = nn.CellList(
            [Wav2Vec2EncoderLayerStableLayerNorm(config) for _ in range(config.num_hidden_layers)]
        )

    def construct(
        self,
        hidden_states,
        attention_mask=None,
        output_attentions=False,
        output_hidden_states=False,
        return_dict=True,
    ):
        """
        Constructs the Wav2Vec2EncoderStableLayerNorm.

        Args:

        - hidden_states: The input hidden states of shape (batch_size, sequence_length, hidden_size).
        - attention_mask: Optional attention mask of shape (batch_size, sequence_length).
        It is used to mask the attention scores.
        - output_attentions: Boolean flag indicating whether to output attention weights. Defaults to False.
        - output_hidden_states: Boolean flag indicating whether to output hidden states of all layers. Defaults to False.
        - return_dict: Boolean flag indicating whether to return a dictionary as output. Defaults to True.

        Returns:
            None

        Raises:
            None
        """
        all_hidden_states = () if output_hidden_states else None
        all_self_attentions = () if output_attentions else None

        if attention_mask is not None:
            # make sure padded tokens are not attended to
            expand_attention_mask = attention_mask.unsqueeze(-1).repeat(1, 1, hidden_states.shape[2])
            hidden_states[~expand_attention_mask] = 0

            # extend attention_mask
            attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
            attention_mask = attention_mask * finfo(hidden_states.dtype, 'min')
            attention_mask = attention_mask.expand(
                attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
            )

        position_embeddings = self.pos_conv_embed(hidden_states)
        hidden_states = hidden_states + position_embeddings
        hidden_states = self.dropout(hidden_states)

        for layer in self.layers:
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
            dropout_probability = ops.rand([])

            skip_the_layer = self.training and (dropout_probability < self.config.layerdrop)
            if not skip_the_layer:
                layer_outputs = layer(
                    hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
                )
                hidden_states = layer_outputs[0]

            if skip_the_layer:
                layer_outputs = (None, None)

            if output_attentions:
                all_self_attentions = all_self_attentions + (layer_outputs[1],)

        hidden_states = self.layer_norm(hidden_states)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)
        return BaseModelOutput(
            last_hidden_state=hidden_states,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
        )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2EncoderStableLayerNorm.__init__(config)

Initializes an instance of the Wav2Vec2EncoderStableLayerNorm class.

PARAMETER	DESCRIPTION
self	The object instance.
config	The configuration object for the Wav2Vec2 model. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes an instance of the Wav2Vec2EncoderStableLayerNorm class.

    Args:
        self: The object instance.
        config (Wav2Vec2Config): The configuration object for the Wav2Vec2 model.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.config = config
    self.pos_conv_embed = Wav2Vec2PositionalConvEmbedding(config)
    self.layer_norm = nn.LayerNorm(config.hidden_size, epsilon=config.layer_norm_eps)
    self.dropout = nn.Dropout(p=config.hidden_dropout)
    self.layers = nn.CellList(
        [Wav2Vec2EncoderLayerStableLayerNorm(config) for _ in range(config.num_hidden_layers)]
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2EncoderStableLayerNorm.construct(hidden_states, attention_mask=None, output_attentions=False, output_hidden_states=False, return_dict=True)

Constructs the Wav2Vec2EncoderStableLayerNorm.

Args:

• hidden_states: The input hidden states of shape (batch_size, sequence_length, hidden_size).
• attention_mask: Optional attention mask of shape (batch_size, sequence_length). It is used to mask the attention scores.
• output_attentions: Boolean flag indicating whether to output attention weights. Defaults to False.
• output_hidden_states: Boolean flag indicating whether to output hidden states of all layers. Defaults to False.
• return_dict: Boolean flag indicating whether to return a dictionary as output. Defaults to True.

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    hidden_states,
    attention_mask=None,
    output_attentions=False,
    output_hidden_states=False,
    return_dict=True,
):
    """
    Constructs the Wav2Vec2EncoderStableLayerNorm.

    Args:

    - hidden_states: The input hidden states of shape (batch_size, sequence_length, hidden_size).
    - attention_mask: Optional attention mask of shape (batch_size, sequence_length).
    It is used to mask the attention scores.
    - output_attentions: Boolean flag indicating whether to output attention weights. Defaults to False.
    - output_hidden_states: Boolean flag indicating whether to output hidden states of all layers. Defaults to False.
    - return_dict: Boolean flag indicating whether to return a dictionary as output. Defaults to True.

    Returns:
        None

    Raises:
        None
    """
    all_hidden_states = () if output_hidden_states else None
    all_self_attentions = () if output_attentions else None

    if attention_mask is not None:
        # make sure padded tokens are not attended to
        expand_attention_mask = attention_mask.unsqueeze(-1).repeat(1, 1, hidden_states.shape[2])
        hidden_states[~expand_attention_mask] = 0

        # extend attention_mask
        attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
        attention_mask = attention_mask * finfo(hidden_states.dtype, 'min')
        attention_mask = attention_mask.expand(
            attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
        )

    position_embeddings = self.pos_conv_embed(hidden_states)
    hidden_states = hidden_states + position_embeddings
    hidden_states = self.dropout(hidden_states)

    for layer in self.layers:
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
        dropout_probability = ops.rand([])

        skip_the_layer = self.training and (dropout_probability < self.config.layerdrop)
        if not skip_the_layer:
            layer_outputs = layer(
                hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
            )
            hidden_states = layer_outputs[0]

        if skip_the_layer:
            layer_outputs = (None, None)

        if output_attentions:
            all_self_attentions = all_self_attentions + (layer_outputs[1],)

    hidden_states = self.layer_norm(hidden_states)

    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    if not return_dict:
        return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)
    return BaseModelOutput(
        last_hidden_state=hidden_states,
        hidden_states=all_hidden_states,
        attentions=all_self_attentions,
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeatureEncoder

Bases: Cell

Construct the features from raw audio waveform

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2FeatureEncoder(nn.Cell):
    """Construct the features from raw audio waveform"""
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a new instance of the Wav2Vec2FeatureEncoder class.

        Args:
            self: The object itself.
            config (Wav2Vec2Config):
                The configuration object for the feature encoder.

                - config.feat_extract_norm (str): The type of normalization to be applied during feature extraction.

                    - 'group': Applies group normalization to the convolutional layers.
                    - 'layer': Applies layer normalization to the convolutional layers.

                - config.num_feat_extract_layers (int): The number of feature extraction layers.

        Returns:
            None.

        Raises:
            ValueError: If `config.feat_extract_norm` is not one of ['group', 'layer'].

        """
        super().__init__()

        if config.feat_extract_norm == "group":
            conv_layers = [Wav2Vec2GroupNormConvLayer(config, layer_id=0)] + [
                Wav2Vec2NoLayerNormConvLayer(config, layer_id=i + 1) for i in range(config.num_feat_extract_layers - 1)
            ]
        elif config.feat_extract_norm == "layer":
            conv_layers = [
                Wav2Vec2LayerNormConvLayer(config, layer_id=i) for i in range(config.num_feat_extract_layers)
            ]
        else:
            raise ValueError(
                f"`config.feat_extract_norm` is {config.feat_extract_norm}, but has to be one of ['group', 'layer']"
            )
        self.conv_layers = nn.CellList(conv_layers)
        self._requires_grad = True

    def _freeze_parameters(self):
        """
        Freezes the parameters of the Wav2Vec2FeatureEncoder.

        Args:
            self: An instance of the Wav2Vec2FeatureEncoder class.

        Returns:
            None.

        Raises:
            None.
        """
        for _, param in self.parameters_and_names():
            param.requires_grad = False
        self._requires_grad = False

    def construct(self, input_values):
        """
        Method 'construct' in the class 'Wav2Vec2FeatureEncoder' constructs the hidden states from the input values
        using convolutional layers.

        Args:
            self (object): The instance of the class.
            input_values (tensor): The input values for constructing hidden states. It is expected to be a 2D tensor.

        Returns:
            tensor: The constructed hidden states after passing through the convolutional layers.

        Raises:
            None
        """
        hidden_states = input_values[:, None]
        for conv_layer in self.conv_layers:
            hidden_states = conv_layer(hidden_states)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeatureEncoder.__init__(config)

Initializes a new instance of the Wav2Vec2FeatureEncoder class.

PARAMETER	DESCRIPTION
self	The object itself.
config	The configuration object for the feature encoder. config.feat_extract_norm (str): The type of normalization to be applied during feature extraction. 'group': Applies group normalization to the convolutional layers. 'layer': Applies layer normalization to the convolutional layers. config.num_feat_extract_layers (int): The number of feature extraction layers. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If config.feat_extract_norm is not one of ['group', 'layer'].

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a new instance of the Wav2Vec2FeatureEncoder class.

    Args:
        self: The object itself.
        config (Wav2Vec2Config):
            The configuration object for the feature encoder.

            - config.feat_extract_norm (str): The type of normalization to be applied during feature extraction.

                - 'group': Applies group normalization to the convolutional layers.
                - 'layer': Applies layer normalization to the convolutional layers.

            - config.num_feat_extract_layers (int): The number of feature extraction layers.

    Returns:
        None.

    Raises:
        ValueError: If `config.feat_extract_norm` is not one of ['group', 'layer'].

    """
    super().__init__()

    if config.feat_extract_norm == "group":
        conv_layers = [Wav2Vec2GroupNormConvLayer(config, layer_id=0)] + [
            Wav2Vec2NoLayerNormConvLayer(config, layer_id=i + 1) for i in range(config.num_feat_extract_layers - 1)
        ]
    elif config.feat_extract_norm == "layer":
        conv_layers = [
            Wav2Vec2LayerNormConvLayer(config, layer_id=i) for i in range(config.num_feat_extract_layers)
        ]
    else:
        raise ValueError(
            f"`config.feat_extract_norm` is {config.feat_extract_norm}, but has to be one of ['group', 'layer']"
        )
    self.conv_layers = nn.CellList(conv_layers)
    self._requires_grad = True

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeatureEncoder.construct(input_values)

Method 'construct' in the class 'Wav2Vec2FeatureEncoder' constructs the hidden states from the input values using convolutional layers.

PARAMETER	DESCRIPTION
self	The instance of the class. TYPE: object
input_values	The input values for constructing hidden states. It is expected to be a 2D tensor. TYPE: tensor

RETURNS	DESCRIPTION
tensor	The constructed hidden states after passing through the convolutional layers.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, input_values):
    """
    Method 'construct' in the class 'Wav2Vec2FeatureEncoder' constructs the hidden states from the input values
    using convolutional layers.

    Args:
        self (object): The instance of the class.
        input_values (tensor): The input values for constructing hidden states. It is expected to be a 2D tensor.

    Returns:
        tensor: The constructed hidden states after passing through the convolutional layers.

    Raises:
        None
    """
    hidden_states = input_values[:, None]
    for conv_layer in self.conv_layers:
        hidden_states = conv_layer(hidden_states)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeatureExtractor

Bases: Wav2Vec2FeatureEncoder

Wav2Vec2FeatureExtractor is a class that represents a feature extractor for Wav2Vec2 models. It is designed to extract features from audio data for use in Wav2Vec2 models.

This class inherits from Wav2Vec2FeatureEncoder, and it is recommended to use Wav2Vec2FeatureEncoder instead of this class, as Wav2Vec2FeatureExtractor has been deprecated and will be removed in Transformers v5.

Please refer to the documentation for Wav2Vec2FeatureEncoder for feature extraction and encoding in Wav2Vec2 models.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2FeatureExtractor(Wav2Vec2FeatureEncoder):

    """
    Wav2Vec2FeatureExtractor is a class that represents a feature extractor for Wav2Vec2 models.
    It is designed to extract features from audio data for use in Wav2Vec2 models.

    This class inherits from Wav2Vec2FeatureEncoder, and it is recommended to use Wav2Vec2FeatureEncoder instead of
    this class, as Wav2Vec2FeatureExtractor has been deprecated and will be removed in Transformers v5.

    Please refer to the documentation for Wav2Vec2FeatureEncoder for feature extraction and encoding in Wav2Vec2 models.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        This method initializes an instance of the Wav2Vec2FeatureExtractor class.

        Args:
            self: The instance of the class.
            config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
                for the feature extractor.

        Returns:
            None.

        Raises:
            FutureWarning: If the class Wav2Vec2FeatureExtractor is used, a FutureWarning is raised indicating that
                the class has been depreciated and will be removed in Transformers v5. It is recommended to use the base
                class instead.
        """
        super().__init__(config)
        warnings.warn(
            f"The class `{self.__class__.__name__}` has been depreciated "
            "and will be removed in Transformers v5. "
            f"Use `{self.__class__.__bases__[0].__name__}` instead.",
            FutureWarning,
        )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeatureExtractor.__init__(config)

This method initializes an instance of the Wav2Vec2FeatureExtractor class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An instance of the Wav2Vec2Config class containing the configuration parameters for the feature extractor. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
FutureWarning	If the class Wav2Vec2FeatureExtractor is used, a FutureWarning is raised indicating that the class has been depreciated and will be removed in Transformers v5. It is recommended to use the base class instead.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    This method initializes an instance of the Wav2Vec2FeatureExtractor class.

    Args:
        self: The instance of the class.
        config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
            for the feature extractor.

    Returns:
        None.

    Raises:
        FutureWarning: If the class Wav2Vec2FeatureExtractor is used, a FutureWarning is raised indicating that
            the class has been depreciated and will be removed in Transformers v5. It is recommended to use the base
            class instead.
    """
    super().__init__(config)
    warnings.warn(
        f"The class `{self.__class__.__name__}` has been depreciated "
        "and will be removed in Transformers v5. "
        f"Use `{self.__class__.__bases__[0].__name__}` instead.",
        FutureWarning,
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeatureProjection

Bases: Cell

Wav2Vec2FeatureProjection is a Python class that represents a feature projection module for Wav2Vec2. This class inherits from nn.Cell and contains methods for initializing the feature projection and constructing the hidden states.

The init method initializes the feature projection module by setting up layer normalization, dense projection, and dropout.

The construct method applies layer normalization to the hidden states, projects the normalized states using dense projection, and applies dropout to the projected states before returning the hidden states and the normalized hidden states.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2FeatureProjection(nn.Cell):

    """
    Wav2Vec2FeatureProjection is a Python class that represents a feature projection module for Wav2Vec2.
    This class inherits from nn.Cell and contains methods for initializing the feature projection and constructing the
    hidden states.

    The __init__ method initializes the feature projection module by setting up layer normalization, dense projection,
    and dropout.

    The construct method applies layer normalization to the hidden states, projects the normalized states using dense
    projection, and applies dropout to the projected states before returning the hidden states and the normalized
    hidden states.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes the Wav2Vec2FeatureProjection class.

        Args:
            self: The instance of the Wav2Vec2FeatureProjection class.
            config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
                for the Wav2Vec2 feature projection. It specifies the configuration for the layer
                normalization, projection, and dropout layers.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not of type Wav2Vec2Config.
            ValueError: If the config.conv_dim[-1] is not valid or if the config.hidden_size is not valid.
            RuntimeError: If an error occurs during the initialization of layer normalization, projection,
                or dropout layers.
        """
        super().__init__()
        self.layer_norm = nn.LayerNorm(config.conv_dim[-1], epsilon=config.layer_norm_eps)
        self.projection = nn.Dense(config.conv_dim[-1], config.hidden_size)
        self.dropout = nn.Dropout(p=config.feat_proj_dropout)

    def construct(self, hidden_states):
        """
        This method constructs the hidden states by applying layer normalization, projection, and dropout.

        Args:
            self (Wav2Vec2FeatureProjection): The instance of the Wav2Vec2FeatureProjection class.
            hidden_states (Tensor): The input hidden states to be processed. It should be a tensor of shape
                (batch_size, sequence_length, feature_dim).

        Returns:
            Tuple[Tensor, Tensor]:
                A tuple containing two tensors:

                - hidden_states (Tensor): The processed hidden states after applying layer normalization, projection,
                and dropout.
                - norm_hidden_states (Tensor): The normalized hidden states obtained after applying layer normalization.

        Raises:
            None.
        """
        # non-projected hidden states are needed for quantization
        norm_hidden_states = self.layer_norm(hidden_states)
        hidden_states = self.projection(norm_hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states, norm_hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeatureProjection.__init__(config)

Initializes the Wav2Vec2FeatureProjection class.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2FeatureProjection class.
config	An instance of the Wav2Vec2Config class containing the configuration parameters for the Wav2Vec2 feature projection. It specifies the configuration for the layer normalization, projection, and dropout layers. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not of type Wav2Vec2Config.
ValueError	If the config.conv_dim[-1] is not valid or if the config.hidden_size is not valid.
RuntimeError	If an error occurs during the initialization of layer normalization, projection, or dropout layers.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes the Wav2Vec2FeatureProjection class.

    Args:
        self: The instance of the Wav2Vec2FeatureProjection class.
        config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
            for the Wav2Vec2 feature projection. It specifies the configuration for the layer
            normalization, projection, and dropout layers.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not of type Wav2Vec2Config.
        ValueError: If the config.conv_dim[-1] is not valid or if the config.hidden_size is not valid.
        RuntimeError: If an error occurs during the initialization of layer normalization, projection,
            or dropout layers.
    """
    super().__init__()
    self.layer_norm = nn.LayerNorm(config.conv_dim[-1], epsilon=config.layer_norm_eps)
    self.projection = nn.Dense(config.conv_dim[-1], config.hidden_size)
    self.dropout = nn.Dropout(p=config.feat_proj_dropout)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeatureProjection.construct(hidden_states)

This method constructs the hidden states by applying layer normalization, projection, and dropout.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2FeatureProjection class. TYPE: Wav2Vec2FeatureProjection
hidden_states	The input hidden states to be processed. It should be a tensor of shape (batch_size, sequence_length, feature_dim). TYPE: Tensor

RETURNS	DESCRIPTION
	Tuple[Tensor, Tensor]: A tuple containing two tensors: hidden_states (Tensor): The processed hidden states after applying layer normalization, projection, and dropout. norm_hidden_states (Tensor): The normalized hidden states obtained after applying layer normalization.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    """
    This method constructs the hidden states by applying layer normalization, projection, and dropout.

    Args:
        self (Wav2Vec2FeatureProjection): The instance of the Wav2Vec2FeatureProjection class.
        hidden_states (Tensor): The input hidden states to be processed. It should be a tensor of shape
            (batch_size, sequence_length, feature_dim).

    Returns:
        Tuple[Tensor, Tensor]:
            A tuple containing two tensors:

            - hidden_states (Tensor): The processed hidden states after applying layer normalization, projection,
            and dropout.
            - norm_hidden_states (Tensor): The normalized hidden states obtained after applying layer normalization.

    Raises:
        None.
    """
    # non-projected hidden states are needed for quantization
    norm_hidden_states = self.layer_norm(hidden_states)
    hidden_states = self.projection(norm_hidden_states)
    hidden_states = self.dropout(hidden_states)
    return hidden_states, norm_hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeedForward

Bases: Cell

Wav2Vec2FeedForward is a class representing the feedforward network for the Wav2Vec2 model. This class inherits from nn.Cell and contains methods for initializing the network and constructing the feedforward layers.

The init method initializes the feedforward network with the provided configuration. It sets up the intermediate dropout, intermediate dense, intermediate activation function, output dense, and output dropout layers based on the configuration parameters.

The construct method takes hidden states as input and processes them through the intermediate dense layer, intermediate activation function, intermediate dropout layer, output dense layer, and output dropout layer. It then returns the processed hidden states.

Note

This docstring is based on the provided code snippet and may need to be updated with additional information once the entire class implementation is available.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2FeedForward(nn.Cell):

    """
    Wav2Vec2FeedForward is a class representing the feedforward network for the Wav2Vec2 model.
    This class inherits from nn.Cell and contains methods for initializing the network and constructing the
    feedforward layers.

    The __init__ method initializes the feedforward network with the provided configuration.
    It sets up the intermediate dropout, intermediate dense, intermediate activation function, output dense, and output
    dropout layers based on the configuration parameters.

    The construct method takes hidden states as input and processes them through the intermediate dense layer,
    intermediate activation function, intermediate dropout layer, output dense layer, and output dropout layer.
    It then returns the processed hidden states.

    Note:
        This docstring is based on the provided code snippet and may need to be updated with additional information once
        the entire class implementation is available.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initialize the Wav2Vec2FeedForward class.

        Args:
            self: Instance of the class.
            config (Wav2Vec2Config): Configuration object containing parameters for initialization.
                The config parameter is of type Wav2Vec2Config and holds the configuration settings required for
                initializing the feed-forward module.
                It is expected to contain the following attributes:

                - activation_dropout (float): Dropout probability for intermediate layers.
                - hidden_size (int): Size of the hidden layers.
                - intermediate_size (int): Size of the intermediate layer.
                - hidden_act (str or function): Activation function for the hidden layers.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.intermediate_dropout = nn.Dropout(p=config.activation_dropout)

        self.intermediate_dense = nn.Dense(config.hidden_size, config.intermediate_size)
        if isinstance(config.hidden_act, str):
            self.intermediate_act_fn = ACT2FN[config.hidden_act]
        else:
            self.intermediate_act_fn = config.hidden_act

        self.output_dense = nn.Dense(config.intermediate_size, config.hidden_size)
        self.output_dropout = nn.Dropout(p=config.hidden_dropout)

    def construct(self, hidden_states):
        """
        Constructs the feed-forward network for the Wav2Vec2 model.

        Args:
            self (Wav2Vec2FeedForward): An instance of the Wav2Vec2FeedForward class.
            hidden_states (torch.Tensor): The input hidden states to be passed through the feed-forward network.

        Returns:
            torch.Tensor: The output hidden states after passing through the feed-forward network.

        Raises:
            TypeError: If the input hidden_states is not of type torch.Tensor.
            ValueError: If the input hidden_states does not have a rank of 2.

        This method takes the input hidden states and passes them through a feed-forward network consisting of several
        layers. The feed-forward network is constructed using intermediate dense layers, activation functions,
        and dropout layers. The hidden_states are first passed through the intermediate dense layer, followed by the
        intermediate activation function and dropout layer. The resulting hidden_states are then passed through the
        output dense layer and another dropout layer. The final output hidden_states are returned.
        Note that the input hidden_states must be a tensor of rank 2, representing a batch of hidden states.
        """
        hidden_states = self.intermediate_dense(hidden_states)
        hidden_states = self.intermediate_act_fn(hidden_states)
        hidden_states = self.intermediate_dropout(hidden_states)

        hidden_states = self.output_dense(hidden_states)
        hidden_states = self.output_dropout(hidden_states)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeedForward.__init__(config)

Initialize the Wav2Vec2FeedForward class.

PARAMETER	DESCRIPTION
self	Instance of the class.
config	Configuration object containing parameters for initialization. The config parameter is of type Wav2Vec2Config and holds the configuration settings required for initializing the feed-forward module. It is expected to contain the following attributes: activation_dropout (float): Dropout probability for intermediate layers. hidden_size (int): Size of the hidden layers. intermediate_size (int): Size of the intermediate layer. hidden_act (str or function): Activation function for the hidden layers. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initialize the Wav2Vec2FeedForward class.

    Args:
        self: Instance of the class.
        config (Wav2Vec2Config): Configuration object containing parameters for initialization.
            The config parameter is of type Wav2Vec2Config and holds the configuration settings required for
            initializing the feed-forward module.
            It is expected to contain the following attributes:

            - activation_dropout (float): Dropout probability for intermediate layers.
            - hidden_size (int): Size of the hidden layers.
            - intermediate_size (int): Size of the intermediate layer.
            - hidden_act (str or function): Activation function for the hidden layers.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.intermediate_dropout = nn.Dropout(p=config.activation_dropout)

    self.intermediate_dense = nn.Dense(config.hidden_size, config.intermediate_size)
    if isinstance(config.hidden_act, str):
        self.intermediate_act_fn = ACT2FN[config.hidden_act]
    else:
        self.intermediate_act_fn = config.hidden_act

    self.output_dense = nn.Dense(config.intermediate_size, config.hidden_size)
    self.output_dropout = nn.Dropout(p=config.hidden_dropout)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2FeedForward.construct(hidden_states)

Constructs the feed-forward network for the Wav2Vec2 model.

PARAMETER	DESCRIPTION
self	An instance of the Wav2Vec2FeedForward class. TYPE: Wav2Vec2FeedForward
hidden_states	The input hidden states to be passed through the feed-forward network. TYPE: Tensor

RETURNS	DESCRIPTION
	torch.Tensor: The output hidden states after passing through the feed-forward network.

RAISES	DESCRIPTION
TypeError	If the input hidden_states is not of type torch.Tensor.
ValueError	If the input hidden_states does not have a rank of 2.

This method takes the input hidden states and passes them through a feed-forward network consisting of several layers. The feed-forward network is constructed using intermediate dense layers, activation functions, and dropout layers. The hidden_states are first passed through the intermediate dense layer, followed by the intermediate activation function and dropout layer. The resulting hidden_states are then passed through the output dense layer and another dropout layer. The final output hidden_states are returned. Note that the input hidden_states must be a tensor of rank 2, representing a batch of hidden states.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    """
    Constructs the feed-forward network for the Wav2Vec2 model.

    Args:
        self (Wav2Vec2FeedForward): An instance of the Wav2Vec2FeedForward class.
        hidden_states (torch.Tensor): The input hidden states to be passed through the feed-forward network.

    Returns:
        torch.Tensor: The output hidden states after passing through the feed-forward network.

    Raises:
        TypeError: If the input hidden_states is not of type torch.Tensor.
        ValueError: If the input hidden_states does not have a rank of 2.

    This method takes the input hidden states and passes them through a feed-forward network consisting of several
    layers. The feed-forward network is constructed using intermediate dense layers, activation functions,
    and dropout layers. The hidden_states are first passed through the intermediate dense layer, followed by the
    intermediate activation function and dropout layer. The resulting hidden_states are then passed through the
    output dense layer and another dropout layer. The final output hidden_states are returned.
    Note that the input hidden_states must be a tensor of rank 2, representing a batch of hidden states.
    """
    hidden_states = self.intermediate_dense(hidden_states)
    hidden_states = self.intermediate_act_fn(hidden_states)
    hidden_states = self.intermediate_dropout(hidden_states)

    hidden_states = self.output_dense(hidden_states)
    hidden_states = self.output_dropout(hidden_states)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForAudioFrameClassification

Bases: Wav2Vec2PreTrainedModel

This class represents a Wav2Vec2 model for audio frame classification. It inherits from the Wav2Vec2PreTrainedModel and includes methods for initializing the model, freezing the feature encoder and base model, as well as constructing the model for inference and training.

ATTRIBUTE	DESCRIPTION
wav2vec2	The Wav2Vec2Model used for audio frame classification. TYPE: Wav2Vec2Model
classifier	The classification head for the model. TYPE: Dense
num_labels	The number of labels for classification. TYPE: int
layer_weights	The weights for weighted layer sum if configured. TYPE: Parameter

METHOD	DESCRIPTION
__init__	Initializes the Wav2Vec2ForAudioFrameClassification model with the provided configuration.
freeze_feature_encoder	Disables the gradient computation for the feature encoder, preventing its parameters from being updated during training.
freeze_base_model	Disables the gradient computation for the base model, preventing its parameters from being updated during training while allowing the classification head to be updated.
construct	Constructs the model for inference and training, handling input values, attention masks, labels, and other optional parameters. Returns TokenClassifierOutput containing loss, logits, hidden states, and attentions.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2ForAudioFrameClassification(Wav2Vec2PreTrainedModel):

    """
    This class represents a Wav2Vec2 model for audio frame classification. It inherits from the Wav2Vec2PreTrainedModel
    and includes methods for initializing the model, freezing the feature encoder and base model, as well as
    constructing the model for inference and training.

    Attributes:
        wav2vec2 (Wav2Vec2Model): The Wav2Vec2Model used for audio frame classification.
        classifier (nn.Dense): The classification head for the model.
        num_labels (int): The number of labels for classification.
        layer_weights (Parameter, optional): The weights for weighted layer sum if configured.

    Methods:
        __init__:
            Initializes the Wav2Vec2ForAudioFrameClassification model with the provided configuration.

        freeze_feature_encoder:
            Disables the gradient computation for the feature encoder, preventing its parameters from being updated
            during training.

        freeze_base_model:
            Disables the gradient computation for the base model, preventing its parameters from being updated during
            training while allowing the classification head to be updated.

        construct:
            Constructs the model for inference and training, handling input values, attention masks, labels, and other
            optional parameters. Returns TokenClassifierOutput containing loss, logits, hidden states, and attentions.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a new instance of the Wav2Vec2ForAudioFrameClassification class.

        Args:
            self: The instance of the class.
            config (Wav2Vec2Config): The configuration object for the Wav2Vec2 model.
                It specifies the parameters and settings for the model initialization.
                Must be an instance of Wav2Vec2Config.

        Returns:
            None.

        Raises:
            ValueError: If the 'config' object has the attribute 'add_adapter' set to True,
                which is not supported for audio frame classification with Wav2Vec2.
        """
        super().__init__(config)

        if hasattr(config, "add_adapter") and config.add_adapter:
            raise ValueError(
                "Audio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)"
            )
        self.wav2vec2 = Wav2Vec2Model(config)
        num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
        if config.use_weighted_layer_sum:
            self.layer_weights = Parameter(ops.ones(num_layers) / num_layers)
        self.classifier = nn.Dense(config.hidden_size, config.num_labels)
        self.num_labels = config.num_labels

        self.init_weights()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.wav2vec2.feature_extractor._freeze_parameters()

    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for _, param in self.wav2vec2.parameters_and_names():
            param.requires_grad = False

    def construct(
        self,
        input_values: Optional[Tensor],
        attention_mask: Optional[Tensor] = None,
        labels: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, TokenClassifierOutput]:
        r"""
        Args:
            labels (`Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

        outputs = self.wav2vec2(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        if self.config.use_weighted_layer_sum:
            hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
            hidden_states = ops.stack(hidden_states, axis=1)
            norm_weights = ops.softmax(self.layer_weights, axis=-1)
            hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
        else:
            hidden_states = outputs[0]

        logits = self.classifier(hidden_states)

        loss = None
        if labels is not None:
            labels = labels.astype(mindspore.int32)
            loss = ops.cross_entropy(logits.view(-1, self.num_labels), ops.argmax(labels.view(-1, self.num_labels), dim=1))

        if not return_dict:
            output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
            return output

        return TokenClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForAudioFrameClassification.__init__(config)

Initializes a new instance of the Wav2Vec2ForAudioFrameClassification class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	The configuration object for the Wav2Vec2 model. It specifies the parameters and settings for the model initialization. Must be an instance of Wav2Vec2Config. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the 'config' object has the attribute 'add_adapter' set to True, which is not supported for audio frame classification with Wav2Vec2.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a new instance of the Wav2Vec2ForAudioFrameClassification class.

    Args:
        self: The instance of the class.
        config (Wav2Vec2Config): The configuration object for the Wav2Vec2 model.
            It specifies the parameters and settings for the model initialization.
            Must be an instance of Wav2Vec2Config.

    Returns:
        None.

    Raises:
        ValueError: If the 'config' object has the attribute 'add_adapter' set to True,
            which is not supported for audio frame classification with Wav2Vec2.
    """
    super().__init__(config)

    if hasattr(config, "add_adapter") and config.add_adapter:
        raise ValueError(
            "Audio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)"
        )
    self.wav2vec2 = Wav2Vec2Model(config)
    num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
    if config.use_weighted_layer_sum:
        self.layer_weights = Parameter(ops.ones(num_layers) / num_layers)
    self.classifier = nn.Dense(config.hidden_size, config.num_labels)
    self.num_labels = config.num_labels

    self.init_weights()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForAudioFrameClassification.construct(input_values, attention_mask=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). TYPE: `Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    input_values: Optional[Tensor],
    attention_mask: Optional[Tensor] = None,
    labels: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, TokenClassifierOutput]:
    r"""
    Args:
        labels (`Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

    outputs = self.wav2vec2(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    if self.config.use_weighted_layer_sum:
        hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
        hidden_states = ops.stack(hidden_states, axis=1)
        norm_weights = ops.softmax(self.layer_weights, axis=-1)
        hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
    else:
        hidden_states = outputs[0]

    logits = self.classifier(hidden_states)

    loss = None
    if labels is not None:
        labels = labels.astype(mindspore.int32)
        loss = ops.cross_entropy(logits.view(-1, self.num_labels), ops.argmax(labels.view(-1, self.num_labels), dim=1))

    if not return_dict:
        output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
        return output

    return TokenClassifierOutput(
        loss=loss,
        logits=logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForAudioFrameClassification.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for _, param in self.wav2vec2.parameters_and_names():
        param.requires_grad = False

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForAudioFrameClassification.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.wav2vec2.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForCTC

Bases: Wav2Vec2PreTrainedModel

This class represents a Wav2Vec2 model fine-tuned for Connectionist Temporal Classification (CTC) tasks. It inherits from the Wav2Vec2PreTrainedModel, providing methods for initializing the model, tying weights, freezing the feature extractor, feature encoder, and base model, as well as constructing the model for inference and training.

The Wav2Vec2ForCTC class encapsulates the Wav2Vec2 model with additional methods for CTC-specific functionality, such as handling labels for CTC, computing CTC loss, and processing input values for CTC tasks.

The class provides methods for fine-tuning the Wav2Vec2 model for CTC tasks, including freezing specific components of the model, as well as constructing the model for CTC inference and training.

Additionally, the class provides methods for tying weights and freezing specific components of the model to ensure compatibility with adapter weights and to control parameter updates during training.

This class is designed for fine-tuning the Wav2Vec2 model for CTC tasks, providing a comprehensive set of methods for customizing the model's behavior and supporting CTC-specific functionality.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2ForCTC(Wav2Vec2PreTrainedModel):

    """
    This class represents a Wav2Vec2 model fine-tuned for Connectionist Temporal Classification (CTC) tasks.
    It inherits from the Wav2Vec2PreTrainedModel, providing methods for initializing the model, tying weights,
    freezing the feature extractor, feature encoder, and base model, as well as constructing the model
    for inference and training.

    The Wav2Vec2ForCTC class encapsulates the Wav2Vec2 model with additional methods for CTC-specific functionality,
    such as handling labels for CTC, computing CTC loss, and processing input values for CTC tasks.

    The class provides methods for fine-tuning the Wav2Vec2 model for CTC tasks, including freezing specific components
    of the model, as well as constructing the model for CTC inference and training.

    Additionally, the class provides methods for tying weights and freezing specific components of the model to ensure
    compatibility with adapter weights and to control parameter updates during training.

    This class is designed for fine-tuning the Wav2Vec2 model for CTC tasks, providing a comprehensive set of methods
    for customizing the model's behavior and supporting CTC-specific functionality.
    """
    def __init__(self, config: Wav2Vec2Config, target_lang: Optional[str] = None):
        """
        Initializes a new instance of the Wav2Vec2ForCTC class.

        Args:
            self: The object itself.
            config (Wav2Vec2Config): The configuration for the Wav2Vec2Model.
            target_lang (Optional[str], optional): The target language. Defaults to None.

        Returns:
            None

        Raises:
            ValueError: If the configuration does not define the vocabulary size of the language model head.

        Note:
            The vocabulary size of the language model head must be defined either by instantiating the model
            with `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)` or by explicitly defining the
            `vocab_size` in the model's configuration.

        """
        super().__init__(config)

        self.wav2vec2 = Wav2Vec2Model(config)
        self.dropout = nn.Dropout(p=config.final_dropout)

        self.target_lang = target_lang

        if config.vocab_size is None:
            raise ValueError(
                f"You are trying to instantiate {self.__class__} with a configuration that "
                "does not define the vocabulary size of the language model head. Please "
                "instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. "
                "or define `vocab_size` of your model's configuration."
            )
        output_hidden_size = (
            config.output_hidden_size if hasattr(config, "add_adapter") and config.add_adapter else config.hidden_size
        )
        self.lm_head = nn.Dense(output_hidden_size, config.vocab_size)

        # Initialize weights and apply final processing
        self.post_init()

    def tie_weights(self):
        """
        This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when
        passing `target_lang=...` to `from_pretrained(...)`.

        This method is **not** supposed to be called by the user and is prone to be changed in the future.
        """
        # Note that `tie_weights` is usually used to tie input and output embedding weights. The method is re-purposed to
        # correctly load adapter layers for Wav2Vec2 so that we do not have to introduce a new API to
        # [`PreTrainedModel`]. While slightly hacky, Wav2Vec2 never has to tie input and output embeddings, so that it is
        # ok to repurpose this function here.
        target_lang = self.target_lang

        if target_lang is not None and getattr(self.config, "adapter_attn_dim", None) is None:
            raise ValueError(f"Cannot pass `target_lang`: {target_lang} if `config.adapter_attn_dim` is not defined.")
        elif target_lang is None and getattr(self.config, "adapter_attn_dim", None) is not None:
            logger.info("By default `target_lang` is set to 'eng'.")
        elif target_lang is not None:
            self.load_adapter(target_lang, force_load=True)

    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.wav2vec2.feature_extractor._freeze_parameters()

    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for _, param in self.wav2vec2.parameters_and_names():
            param.requires_grad = False

    def construct(
        self,
        input_values: Optional[Tensor],
        attention_mask: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[Tensor] = None,
    ) -> Union[Tuple, CausalLMOutput]:
        r"""
        Args:
            labels (`Tensor` of shape `(batch_size, target_length)`, *optional*):
                Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to
                the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`.
                All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ...,
                config.vocab_size - 1]`.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.wav2vec2(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = outputs[0]
        hidden_states = self.dropout(hidden_states)

        logits = self.lm_head(hidden_states)

        loss = None
        if labels is not None:
            labels = labels.astype(mindspore.int32)
            if labels.max() >= self.config.vocab_size:
                raise ValueError(f"Label values must be <= vocab_size: {self.config.vocab_size}")

            # retrieve loss input_lengths from attention_mask
            attention_mask = (
                attention_mask if attention_mask is not None else ops.ones_like(input_values, dtype=mindspore.int64)
            )
            input_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(-1)).to(mindspore.int64)

            # assuming that padded tokens are filled with -100
            # when not being attended to
            labels_mask = labels >= 0
            target_lengths = labels_mask.sum(-1)
            flattened_targets = labels.masked_select(labels_mask)

            # ctc_loss doesn't support fp16
            log_probs = ops.log_softmax(logits, axis=-1).swapaxes(0, 1)

            loss, log_alpha = ops.ctc_loss(
                log_probs,
                labels,     # flattened_targets
                input_lengths,
                target_lengths,
                blank=self.config.pad_token_id,
                reduction=self.config.ctc_loss_reduction,
                zero_infinity=self.config.ctc_zero_infinity,
            )

        if not return_dict:
            output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
            return ((loss,) + output) if loss is not None else output

        return CausalLMOutput(
            loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
        )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForCTC.__init__(config, target_lang=None)

Initializes a new instance of the Wav2Vec2ForCTC class.

PARAMETER	DESCRIPTION
self	The object itself.
config	The configuration for the Wav2Vec2Model. TYPE: Wav2Vec2Config
target_lang	The target language. Defaults to None. TYPE: Optional[str] DEFAULT: None

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
ValueError	If the configuration does not define the vocabulary size of the language model head.

Note

The vocabulary size of the language model head must be defined either by instantiating the model with Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size) or by explicitly defining the vocab_size in the model's configuration.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config, target_lang: Optional[str] = None):
    """
    Initializes a new instance of the Wav2Vec2ForCTC class.

    Args:
        self: The object itself.
        config (Wav2Vec2Config): The configuration for the Wav2Vec2Model.
        target_lang (Optional[str], optional): The target language. Defaults to None.

    Returns:
        None

    Raises:
        ValueError: If the configuration does not define the vocabulary size of the language model head.

    Note:
        The vocabulary size of the language model head must be defined either by instantiating the model
        with `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)` or by explicitly defining the
        `vocab_size` in the model's configuration.

    """
    super().__init__(config)

    self.wav2vec2 = Wav2Vec2Model(config)
    self.dropout = nn.Dropout(p=config.final_dropout)

    self.target_lang = target_lang

    if config.vocab_size is None:
        raise ValueError(
            f"You are trying to instantiate {self.__class__} with a configuration that "
            "does not define the vocabulary size of the language model head. Please "
            "instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. "
            "or define `vocab_size` of your model's configuration."
        )
    output_hidden_size = (
        config.output_hidden_size if hasattr(config, "add_adapter") and config.add_adapter else config.hidden_size
    )
    self.lm_head = nn.Dense(output_hidden_size, config.vocab_size)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForCTC.construct(input_values, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

PARAMETER

DESCRIPTION

labels

Labels for connectionist temporal classification. Note that target_length has to be smaller or equal to the sequence length of the output logits. Indices are selected in [-100, 0, ..., config.vocab_size - 1]. All labels set to -100 are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size - 1]. TYPE: `Tensor` of shape `(batch_size, target_length)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    input_values: Optional[Tensor],
    attention_mask: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[Tensor] = None,
) -> Union[Tuple, CausalLMOutput]:
    r"""
    Args:
        labels (`Tensor` of shape `(batch_size, target_length)`, *optional*):
            Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to
            the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`.
            All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ...,
            config.vocab_size - 1]`.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.wav2vec2(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    hidden_states = outputs[0]
    hidden_states = self.dropout(hidden_states)

    logits = self.lm_head(hidden_states)

    loss = None
    if labels is not None:
        labels = labels.astype(mindspore.int32)
        if labels.max() >= self.config.vocab_size:
            raise ValueError(f"Label values must be <= vocab_size: {self.config.vocab_size}")

        # retrieve loss input_lengths from attention_mask
        attention_mask = (
            attention_mask if attention_mask is not None else ops.ones_like(input_values, dtype=mindspore.int64)
        )
        input_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(-1)).to(mindspore.int64)

        # assuming that padded tokens are filled with -100
        # when not being attended to
        labels_mask = labels >= 0
        target_lengths = labels_mask.sum(-1)
        flattened_targets = labels.masked_select(labels_mask)

        # ctc_loss doesn't support fp16
        log_probs = ops.log_softmax(logits, axis=-1).swapaxes(0, 1)

        loss, log_alpha = ops.ctc_loss(
            log_probs,
            labels,     # flattened_targets
            input_lengths,
            target_lengths,
            blank=self.config.pad_token_id,
            reduction=self.config.ctc_loss_reduction,
            zero_infinity=self.config.ctc_zero_infinity,
        )

    if not return_dict:
        output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
        return ((loss,) + output) if loss is not None else output

    return CausalLMOutput(
        loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForCTC.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for _, param in self.wav2vec2.parameters_and_names():
        param.requires_grad = False

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForCTC.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.wav2vec2.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForCTC.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForCTC.tie_weights()

This method overwrites [~PreTrainedModel.tie_weights] so that adapter weights can be correctly loaded when passing target_lang=... to from_pretrained(...).

This method is not supposed to be called by the user and is prone to be changed in the future.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def tie_weights(self):
    """
    This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when
    passing `target_lang=...` to `from_pretrained(...)`.

    This method is **not** supposed to be called by the user and is prone to be changed in the future.
    """
    # Note that `tie_weights` is usually used to tie input and output embedding weights. The method is re-purposed to
    # correctly load adapter layers for Wav2Vec2 so that we do not have to introduce a new API to
    # [`PreTrainedModel`]. While slightly hacky, Wav2Vec2 never has to tie input and output embeddings, so that it is
    # ok to repurpose this function here.
    target_lang = self.target_lang

    if target_lang is not None and getattr(self.config, "adapter_attn_dim", None) is None:
        raise ValueError(f"Cannot pass `target_lang`: {target_lang} if `config.adapter_attn_dim` is not defined.")
    elif target_lang is None and getattr(self.config, "adapter_attn_dim", None) is not None:
        logger.info("By default `target_lang` is set to 'eng'.")
    elif target_lang is not None:
        self.load_adapter(target_lang, force_load=True)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForMaskedLM

Bases: Wav2Vec2PreTrainedModel

This class represents a Wav2Vec2 model for Masked Language Modeling (MLM). It is deprecated and should be replaced with Wav2Vec2ForCTC.

The Wav2Vec2ForMaskedLM class inherits from the Wav2Vec2PreTrainedModel class.

ATTRIBUTE	DESCRIPTION
`wav2vec2`	The underlying Wav2Vec2Model.
`dropout`	A dropout layer for regularization.
`lm_head`	A dense layer for language modeling prediction.

METHOD	DESCRIPTION
`__init__`	Initializes a new instance of the Wav2Vec2ForMaskedLM class.
`construct`	Constructs the model for masked language modeling.

Note

This class is deprecated and should be replaced with Wav2Vec2ForCTC.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2ForMaskedLM(Wav2Vec2PreTrainedModel):

    """
    This class represents a Wav2Vec2 model for Masked Language Modeling (MLM).
    It is deprecated and should be replaced with `Wav2Vec2ForCTC`.

    The `Wav2Vec2ForMaskedLM` class inherits from the `Wav2Vec2PreTrainedModel` class.

    Attributes:
        `wav2vec2`: The underlying Wav2Vec2Model.
        `dropout`: A dropout layer for regularization.
        `lm_head`: A dense layer for language modeling prediction.

    Methods:
        `__init__`: Initializes a new instance of the `Wav2Vec2ForMaskedLM` class.
        `construct`: Constructs the model for masked language modeling.

    Note:
        This class is deprecated and should be replaced with `Wav2Vec2ForCTC`.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes an instance of the 'Wav2Vec2ForMaskedLM' class.

        Args:
            self: The object instance.
            config (Wav2Vec2Config):
                The configuration object containing various hyperparameters for the model.

                - `config` should be an instance of the 'Wav2Vec2Config' class.
                - This parameter is required.

        Returns:
            None

        Raises:
            FutureWarning: Raised if the class `Wav2Vec2ForMaskedLM` is used, as it is deprecated.
                Recommends using `Wav2Vec2ForCTC` instead.
                This warning is raised as a future version may not support the deprecated class.

        Description:
            This method initializes an instance of the 'Wav2Vec2ForMaskedLM' class. It sets up the model architecture
            and initializes the necessary components. The initialization process includes the following steps:

            1. Calls the parent class '__init__' method using 'super()' to initialize the base class.
            2. Raises a 'FutureWarning' to notify users that the class `Wav2Vec2ForMaskedLM` is deprecated and
            recommends using `Wav2Vec2ForCTC` instead.
            3. Initializes the 'wav2vec2' attribute as an instance of 'Wav2Vec2Model' using the provided 'config'.
            4. Initializes the 'dropout' attribute as an instance of 'nn.Dropout' with the dropout probability specified
            in 'config'.
            5. Initializes the 'lm_head' attribute as an instance of 'nn.Dense' with the hidden size and vocabulary
            size specified in 'config'.
            6. Calls the 'post_init' method to perform any additional post-initialization steps.

        Note:
            The 'Wav2Vec2ForMaskedLM' class is deprecated and may not be supported in future versions. It is recommended
            to use the 'Wav2Vec2ForCTC' class instead.
        """
        super().__init__(config)

        warnings.warn(
            "The class `Wav2Vec2ForMaskedLM` is deprecated. Please use `Wav2Vec2ForCTC` instead.", FutureWarning
        )

        self.wav2vec2 = Wav2Vec2Model(config)
        self.dropout = nn.Dropout(p=config.final_dropout)
        self.lm_head = nn.Dense(config.hidden_size, config.vocab_size)

        # Initialize weights and apply final processing
        self.post_init()

    def construct(
        self,
        input_values: Tensor,
        attention_mask: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[Tensor] = None,
    ) -> Union[Tuple, MaskedLMOutput]:
        """
        Args:
            self (Wav2Vec2ForMaskedLM): The instance of the Wav2Vec2ForMaskedLM class.
            input_values (Tensor): The input tensor representing the input audio features. Its shape is
                (batch_size, sequence_length, feature_dim).
            attention_mask (Optional[Tensor]): Optional tensor representing the attention mask for the input.
                If provided, should have the shape (batch_size, sequence_length).
            output_attentions (Optional[bool]): Optional flag to indicate whether to return attentions in the output.
                Defaults to None.
            output_hidden_states (Optional[bool]): Optional flag to indicate whether to return hidden states
                in the output. Defaults to None.
            return_dict (Optional[bool]): Optional flag to indicate whether to return the output as a dictionary.
                If not provided, it defaults to the value specified in the configuration.
            labels (Optional[Tensor]): Optional tensor representing the labels for the masked language modeling task.
                Its shape is (batch_size, sequence_length).

        Returns:
            Union[Tuple, MaskedLMOutput]:
                The return value can be either a tuple or a MaskedLMOutput object.

                - If return_dict is False, it returns a tuple containing the logits and, optionally, the hidden states
                and attentions.
                - If return_dict is True, it returns a MaskedLMOutput object containing the logits,
                hidden states, and attentions.

        Raises:
            None
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.wav2vec2(
            input_values,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = outputs[0]
        hidden_states = self.dropout(hidden_states)
        logits = self.lm_head(hidden_states)

        if not return_dict:
            output = (logits,) + outputs[2:]
            return output

        return MaskedLMOutput(logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForMaskedLM.__init__(config)

Initializes an instance of the 'Wav2Vec2ForMaskedLM' class.

PARAMETER	DESCRIPTION
self	The object instance.
config	The configuration object containing various hyperparameters for the model. config should be an instance of the 'Wav2Vec2Config' class. This parameter is required. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
FutureWarning	Raised if the class Wav2Vec2ForMaskedLM is used, as it is deprecated. Recommends using Wav2Vec2ForCTC instead. This warning is raised as a future version may not support the deprecated class.

Description

This method initializes an instance of the 'Wav2Vec2ForMaskedLM' class. It sets up the model architecture and initializes the necessary components. The initialization process includes the following steps:

1. Calls the parent class 'init' method using 'super()' to initialize the base class.
2. Raises a 'FutureWarning' to notify users that the class Wav2Vec2ForMaskedLM is deprecated and recommends using Wav2Vec2ForCTC instead.
3. Initializes the 'wav2vec2' attribute as an instance of 'Wav2Vec2Model' using the provided 'config'.
4. Initializes the 'dropout' attribute as an instance of 'nn.Dropout' with the dropout probability specified in 'config'.
5. Initializes the 'lm_head' attribute as an instance of 'nn.Dense' with the hidden size and vocabulary size specified in 'config'.
6. Calls the 'post_init' method to perform any additional post-initialization steps.

Note

The 'Wav2Vec2ForMaskedLM' class is deprecated and may not be supported in future versions. It is recommended to use the 'Wav2Vec2ForCTC' class instead.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes an instance of the 'Wav2Vec2ForMaskedLM' class.

    Args:
        self: The object instance.
        config (Wav2Vec2Config):
            The configuration object containing various hyperparameters for the model.

            - `config` should be an instance of the 'Wav2Vec2Config' class.
            - This parameter is required.

    Returns:
        None

    Raises:
        FutureWarning: Raised if the class `Wav2Vec2ForMaskedLM` is used, as it is deprecated.
            Recommends using `Wav2Vec2ForCTC` instead.
            This warning is raised as a future version may not support the deprecated class.

    Description:
        This method initializes an instance of the 'Wav2Vec2ForMaskedLM' class. It sets up the model architecture
        and initializes the necessary components. The initialization process includes the following steps:

        1. Calls the parent class '__init__' method using 'super()' to initialize the base class.
        2. Raises a 'FutureWarning' to notify users that the class `Wav2Vec2ForMaskedLM` is deprecated and
        recommends using `Wav2Vec2ForCTC` instead.
        3. Initializes the 'wav2vec2' attribute as an instance of 'Wav2Vec2Model' using the provided 'config'.
        4. Initializes the 'dropout' attribute as an instance of 'nn.Dropout' with the dropout probability specified
        in 'config'.
        5. Initializes the 'lm_head' attribute as an instance of 'nn.Dense' with the hidden size and vocabulary
        size specified in 'config'.
        6. Calls the 'post_init' method to perform any additional post-initialization steps.

    Note:
        The 'Wav2Vec2ForMaskedLM' class is deprecated and may not be supported in future versions. It is recommended
        to use the 'Wav2Vec2ForCTC' class instead.
    """
    super().__init__(config)

    warnings.warn(
        "The class `Wav2Vec2ForMaskedLM` is deprecated. Please use `Wav2Vec2ForCTC` instead.", FutureWarning
    )

    self.wav2vec2 = Wav2Vec2Model(config)
    self.dropout = nn.Dropout(p=config.final_dropout)
    self.lm_head = nn.Dense(config.hidden_size, config.vocab_size)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForMaskedLM.construct(input_values, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2ForMaskedLM class. TYPE: Wav2Vec2ForMaskedLM
input_values	The input tensor representing the input audio features. Its shape is (batch_size, sequence_length, feature_dim). TYPE: Tensor
attention_mask	Optional tensor representing the attention mask for the input. If provided, should have the shape (batch_size, sequence_length). TYPE: Optional[Tensor] DEFAULT: None
output_attentions	Optional flag to indicate whether to return attentions in the output. Defaults to None. TYPE: Optional[bool] DEFAULT: None
output_hidden_states	Optional flag to indicate whether to return hidden states in the output. Defaults to None. TYPE: Optional[bool] DEFAULT: None
return_dict	Optional flag to indicate whether to return the output as a dictionary. If not provided, it defaults to the value specified in the configuration. TYPE: Optional[bool] DEFAULT: None
labels	Optional tensor representing the labels for the masked language modeling task. Its shape is (batch_size, sequence_length). TYPE: Optional[Tensor] DEFAULT: None

RETURNS	DESCRIPTION
Union[Tuple, MaskedLMOutput]	Union[Tuple, MaskedLMOutput]: The return value can be either a tuple or a MaskedLMOutput object. If return_dict is False, it returns a tuple containing the logits and, optionally, the hidden states and attentions. If return_dict is True, it returns a MaskedLMOutput object containing the logits, hidden states, and attentions.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    input_values: Tensor,
    attention_mask: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[Tensor] = None,
) -> Union[Tuple, MaskedLMOutput]:
    """
    Args:
        self (Wav2Vec2ForMaskedLM): The instance of the Wav2Vec2ForMaskedLM class.
        input_values (Tensor): The input tensor representing the input audio features. Its shape is
            (batch_size, sequence_length, feature_dim).
        attention_mask (Optional[Tensor]): Optional tensor representing the attention mask for the input.
            If provided, should have the shape (batch_size, sequence_length).
        output_attentions (Optional[bool]): Optional flag to indicate whether to return attentions in the output.
            Defaults to None.
        output_hidden_states (Optional[bool]): Optional flag to indicate whether to return hidden states
            in the output. Defaults to None.
        return_dict (Optional[bool]): Optional flag to indicate whether to return the output as a dictionary.
            If not provided, it defaults to the value specified in the configuration.
        labels (Optional[Tensor]): Optional tensor representing the labels for the masked language modeling task.
            Its shape is (batch_size, sequence_length).

    Returns:
        Union[Tuple, MaskedLMOutput]:
            The return value can be either a tuple or a MaskedLMOutput object.

            - If return_dict is False, it returns a tuple containing the logits and, optionally, the hidden states
            and attentions.
            - If return_dict is True, it returns a MaskedLMOutput object containing the logits,
            hidden states, and attentions.

    Raises:
        None
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.wav2vec2(
        input_values,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    hidden_states = outputs[0]
    hidden_states = self.dropout(hidden_states)
    logits = self.lm_head(hidden_states)

    if not return_dict:
        output = (logits,) + outputs[2:]
        return output

    return MaskedLMOutput(logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTraining

Bases: Wav2Vec2PreTrainedModel

Wav2Vec2ForPreTraining

This class represents a pre-training model for Wav2Vec2, which is used for pre-training the Wav2Vec2 model. It includes methods for setting Gumbel softmax temperature, freezing the feature encoder, computing contrastive logits, and constructing the model for pre-training.

METHOD	DESCRIPTION
set_gumbel_temperature	Set the Gumbel softmax temperature to a given value. Only necessary for training.
freeze_feature_extractor	Disable gradient computation for the feature encoder to prevent parameter updates during training.
freeze_feature_encoder	Disable gradient computation for the feature encoder to prevent parameter updates during training.
compute_contrastive_logits	Compute logits for contrastive loss based on cosine similarity between features and apply temperature.
construct	Construct the model for pre-training, including masking features for contrastive loss.

ATTRIBUTE	DESCRIPTION
wav2vec2	Wav2Vec2Model instance for the Wav2Vec2 model.
dropout_features	Dropout layer for feature vectors.
quantizer	Wav2Vec2GumbelVectorQuantizer instance for quantization.
project_hid	Dense layer for projecting hidden states.
project_q	Dense layer for projecting quantized features.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2ForPreTraining(Wav2Vec2PreTrainedModel):

    """Wav2Vec2ForPreTraining

    This class represents a pre-training model for Wav2Vec2, which is used for pre-training the Wav2Vec2 model.
    It includes methods for setting Gumbel softmax temperature, freezing the feature encoder, computing contrastive
    logits, and constructing the model for pre-training.

    Methods:
        set_gumbel_temperature: Set the Gumbel softmax temperature to a given value. Only necessary for training.
        freeze_feature_extractor: Disable gradient computation for the feature encoder to prevent parameter updates
            during training.
        freeze_feature_encoder: Disable gradient computation for the feature encoder to prevent parameter updates
            during training.
        compute_contrastive_logits: Compute logits for contrastive loss based on cosine similarity between features
            and apply temperature.
        construct: Construct the model for pre-training, including masking features for contrastive loss.

    Attributes:
        wav2vec2: Wav2Vec2Model instance for the Wav2Vec2 model.
        dropout_features: Dropout layer for feature vectors.
        quantizer: Wav2Vec2GumbelVectorQuantizer instance for quantization.
        project_hid: Dense layer for projecting hidden states.
        project_q: Dense layer for projecting quantized features.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a new instance of the Wav2Vec2ForPreTraining class.

        Args:
            self: The instance of the Wav2Vec2ForPreTraining class.
            config (Wav2Vec2Config): The configuration object for the Wav2Vec2 model.

        Returns:
            None.

        Raises:
            None
        """
        super().__init__(config)
        self.wav2vec2 = Wav2Vec2Model(config)
        self.dropout_features = nn.Dropout(p=config.feat_quantizer_dropout)

        self.quantizer = Wav2Vec2GumbelVectorQuantizer(config)

        self.project_hid = nn.Dense(config.hidden_size, config.proj_codevector_dim)
        self.project_q = nn.Dense(config.codevector_dim, config.proj_codevector_dim)

        # Initialize weights and apply final processing
        self.post_init()

    def set_gumbel_temperature(self, temperature: int):
        """
        Set the Gumbel softmax temperature to a given value. Only necessary for training
        """
        self.quantizer.temperature = temperature

    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameters will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.wav2vec2.feature_extractor._freeze_parameters()

    @staticmethod
    def compute_contrastive_logits(
        target_features: Tensor,
        negative_features: Tensor,
        predicted_features: Tensor,
        temperature: int = 0.1,
    ):
        """
        Compute logits for contrastive loss based using cosine similarity as the distance measure between
        `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied.
        """
        target_features = ops.cat([target_features, negative_features], axis=0)
        logits = ops.cosine_similarity(predicted_features.float(), target_features.float(), dim=-1).type_as(target_features)
        # apply temperature
        logits = logits / temperature
        return logits

    def construct(
        self,
        input_values: Optional[Tensor],
        attention_mask: Optional[Tensor] = None,
        mask_time_indices: Optional[Tensor] = None,
        sampled_negative_indices: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, Wav2Vec2ForPreTrainingOutput]:
        r"""
        Args:
            mask_time_indices (`Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict
                masked extracted features in *config.proj_codevector_dim* space.
            sampled_negative_indices (`Tensor` of shape `(batch_size, sequence_length, num_negatives)`, *optional*):
                Indices indicating which quantized target vectors are used as negative sampled vectors in contrastive loss.
                Required input for pre-training.

        Returns:
            Union[Tuple, Wav2Vec2ForPreTrainingOutput]

        Example:
            ```python
            >>> import torch
            >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining
            >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices
            >>> from datasets import load_dataset
            ...
            >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
            >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base")
            ...
            >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
            >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values  # Batch size 1
            ...
            >>> # compute masked indices
            >>> batch_size, raw_sequence_length = input_values.shape
            >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item()
            >>> mask_time_indices = _compute_mask_indices(
            ...     shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2
            ... )
            >>> sampled_negative_indices = _sample_negative_indices(
            ...     features_shape=(batch_size, sequence_length),
            ...     num_negatives=model.config.num_negatives,
            ...     mask_time_indices=mask_time_indices,
            ... )
            >>> mask_time_indices = Tensor(data=mask_time_indices, device=input_values.device, dtype=mindspore.int64)
            >>> sampled_negative_indices = Tensor(
            ...     data=sampled_negative_indices, device=input_values.device, dtype=mindspore.int64
            ... )
            ...
            >>> with ops.no_grad():
            ...     outputs = model(input_values, mask_time_indices=mask_time_indices)
            ...
            >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states)
            >>> cosine_sim = ops.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, axis=-1)
            ...
            >>> # show that cosine similarity is much higher than random
            >>> cosine_sim[mask_time_indices.to(mindspore.bool_)].mean() > 0.5
            tensor(True)
            >>> # for contrastive loss training model should be put into train mode
            >>> model = model.train()
            >>> loss = model(
            ...     input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices
            ... ).loss
            ```
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if mask_time_indices is not None:
            mask_time_indices = mask_time_indices.to(mindspore.bool_)

        outputs = self.wav2vec2(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            mask_time_indices=mask_time_indices,
            return_dict=return_dict,
        )

        # 1. project all transformed features (including masked) to final vq dim
        transformer_features = self.project_hid(outputs[0])

        # 2. quantize all (unmasked) extracted features and project to final vq dim
        extract_features = self.dropout_features(outputs[1])

        if attention_mask is not None:
            # compute reduced attention_mask correponding to feature vectors
            attention_mask = self._get_feature_vector_attention_mask(
                extract_features.shape[1], attention_mask, add_adapter=False
            )

        quantized_features, codevector_perplexity = self.quantizer(
            extract_features, mask_time_indices=mask_time_indices
        )
        quantized_features = self.project_q(quantized_features)

        loss = contrastive_loss = diversity_loss = None
        if sampled_negative_indices is not None:
            batch_size, sequence_length, hidden_size = quantized_features.shape

            # for training, we sample negatives
            # 3. sample K negatives (distractors) quantized states for contrastive loss
            # if attention_mask is passed, make sure that padded feature vectors cannot be sampled
            # sample negative quantized vectors BTC => (BxT)C
            negative_quantized_features = quantized_features.view(-1, hidden_size)[
                sampled_negative_indices.long().view(-1)
            ]
            negative_quantized_features = negative_quantized_features.view(
                batch_size, sequence_length, -1, hidden_size
            ).permute(2, 0, 1, 3)

            # 4. compute logits, corresponding to `logs = sim(c_t, [q_t, \sim{q}_t]) / \kappa`
            # of equation (3) in https://arxiv.org/pdf/2006.11477.pdf
            logits = self.compute_contrastive_logits(
                quantized_features[None, :],
                negative_quantized_features,
                transformer_features,
                self.config.contrastive_logits_temperature,
            )

            # 5. if a negative vector is identical to the positive (i.e. when codebook utilization is low),
            # its cosine similarity will be masked
            neg_is_pos = (quantized_features == negative_quantized_features).all(-1)

            if neg_is_pos.any():
                # NOTE: avoid loss NaN
                # float("-inf") => finfo(logits.dtype, 'min') := -3.40282e+38
                logits[1:][neg_is_pos] = -3.40282e+35

            # 6. compute contrastive loss \mathbf{L}_m = cross_entropy(logs) =
            # -log(exp(sim(c_t, q_t)/\kappa) / \sum_{\sim{q}} exp(sim(c_t, \sim{q})/\kappa))
            logits = logits.swapaxes(0, 2).reshape(-1, logits.shape[0])
            target = ((1 - mask_time_indices.long()) * -100).swapaxes(0, 1).flatten()

            contrastive_loss = ops.cross_entropy(logits.float(), target, reduction="sum")
            # 7. compute diversity loss: \mathbf{L}_d
            num_codevectors = self.config.num_codevectors_per_group * self.config.num_codevector_groups
            diversity_loss = ((num_codevectors - codevector_perplexity) / num_codevectors) * mask_time_indices.sum()

            # 8. \mathbf{L} = \mathbf{L}_m + \alpha * \mathbf{L}_d
            loss = contrastive_loss + self.config.diversity_loss_weight * diversity_loss

        if not return_dict:
            if loss is not None:
                return (loss, transformer_features, quantized_features, codevector_perplexity) + outputs[2:]
            return (transformer_features, quantized_features, codevector_perplexity) + outputs[2:]

        return Wav2Vec2ForPreTrainingOutput(
            loss=loss,
            projected_states=transformer_features,
            projected_quantized_states=quantized_features,
            codevector_perplexity=codevector_perplexity,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
            contrastive_loss=contrastive_loss,
            diversity_loss=diversity_loss,
        )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTraining.__init__(config)

Initializes a new instance of the Wav2Vec2ForPreTraining class.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2ForPreTraining class.
config	The configuration object for the Wav2Vec2 model. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a new instance of the Wav2Vec2ForPreTraining class.

    Args:
        self: The instance of the Wav2Vec2ForPreTraining class.
        config (Wav2Vec2Config): The configuration object for the Wav2Vec2 model.

    Returns:
        None.

    Raises:
        None
    """
    super().__init__(config)
    self.wav2vec2 = Wav2Vec2Model(config)
    self.dropout_features = nn.Dropout(p=config.feat_quantizer_dropout)

    self.quantizer = Wav2Vec2GumbelVectorQuantizer(config)

    self.project_hid = nn.Dense(config.hidden_size, config.proj_codevector_dim)
    self.project_q = nn.Dense(config.codevector_dim, config.proj_codevector_dim)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTraining.compute_contrastive_logits(target_features, negative_features, predicted_features, temperature=0.1) staticmethod

Compute logits for contrastive loss based using cosine similarity as the distance measure between [positive_feature, negative_features] and [predicted_features]. Additionally, temperature can be applied.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

@staticmethod
def compute_contrastive_logits(
    target_features: Tensor,
    negative_features: Tensor,
    predicted_features: Tensor,
    temperature: int = 0.1,
):
    """
    Compute logits for contrastive loss based using cosine similarity as the distance measure between
    `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied.
    """
    target_features = ops.cat([target_features, negative_features], axis=0)
    logits = ops.cosine_similarity(predicted_features.float(), target_features.float(), dim=-1).type_as(target_features)
    # apply temperature
    logits = logits / temperature
    return logits

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTraining.construct(input_values, attention_mask=None, mask_time_indices=None, sampled_negative_indices=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
mask_time_indices	Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in config.proj_codevector_dim space. TYPE: `Tensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None
sampled_negative_indices	Indices indicating which quantized target vectors are used as negative sampled vectors in contrastive loss. Required input for pre-training. TYPE: `Tensor` of shape `(batch_size, sequence_length, num_negatives)`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
Union[Tuple, Wav2Vec2ForPreTrainingOutput]	Union[Tuple, Wav2Vec2ForPreTrainingOutput]

Example

>>> import torch
>>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining
>>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices
>>> from datasets import load_dataset
...
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
>>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base")
...
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values  # Batch size 1
...
>>> # compute masked indices
>>> batch_size, raw_sequence_length = input_values.shape
>>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item()
>>> mask_time_indices = _compute_mask_indices(
...     shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2
... )
>>> sampled_negative_indices = _sample_negative_indices(
...     features_shape=(batch_size, sequence_length),
...     num_negatives=model.config.num_negatives,
...     mask_time_indices=mask_time_indices,
... )
>>> mask_time_indices = Tensor(data=mask_time_indices, device=input_values.device, dtype=mindspore.int64)
>>> sampled_negative_indices = Tensor(
...     data=sampled_negative_indices, device=input_values.device, dtype=mindspore.int64
... )
...
>>> with ops.no_grad():
...     outputs = model(input_values, mask_time_indices=mask_time_indices)
...
>>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states)
>>> cosine_sim = ops.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, axis=-1)
...
>>> # show that cosine similarity is much higher than random
>>> cosine_sim[mask_time_indices.to(mindspore.bool_)].mean() > 0.5
tensor(True)
>>> # for contrastive loss training model should be put into train mode
>>> model = model.train()
>>> loss = model(
...     input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices
... ).loss

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    input_values: Optional[Tensor],
    attention_mask: Optional[Tensor] = None,
    mask_time_indices: Optional[Tensor] = None,
    sampled_negative_indices: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, Wav2Vec2ForPreTrainingOutput]:
    r"""
    Args:
        mask_time_indices (`Tensor` of shape `(batch_size, sequence_length)`, *optional*):
            Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict
            masked extracted features in *config.proj_codevector_dim* space.
        sampled_negative_indices (`Tensor` of shape `(batch_size, sequence_length, num_negatives)`, *optional*):
            Indices indicating which quantized target vectors are used as negative sampled vectors in contrastive loss.
            Required input for pre-training.

    Returns:
        Union[Tuple, Wav2Vec2ForPreTrainingOutput]

    Example:
        ```python
        >>> import torch
        >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining
        >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices
        >>> from datasets import load_dataset
        ...
        >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
        >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base")
        ...
        >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
        >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values  # Batch size 1
        ...
        >>> # compute masked indices
        >>> batch_size, raw_sequence_length = input_values.shape
        >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item()
        >>> mask_time_indices = _compute_mask_indices(
        ...     shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2
        ... )
        >>> sampled_negative_indices = _sample_negative_indices(
        ...     features_shape=(batch_size, sequence_length),
        ...     num_negatives=model.config.num_negatives,
        ...     mask_time_indices=mask_time_indices,
        ... )
        >>> mask_time_indices = Tensor(data=mask_time_indices, device=input_values.device, dtype=mindspore.int64)
        >>> sampled_negative_indices = Tensor(
        ...     data=sampled_negative_indices, device=input_values.device, dtype=mindspore.int64
        ... )
        ...
        >>> with ops.no_grad():
        ...     outputs = model(input_values, mask_time_indices=mask_time_indices)
        ...
        >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states)
        >>> cosine_sim = ops.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, axis=-1)
        ...
        >>> # show that cosine similarity is much higher than random
        >>> cosine_sim[mask_time_indices.to(mindspore.bool_)].mean() > 0.5
        tensor(True)
        >>> # for contrastive loss training model should be put into train mode
        >>> model = model.train()
        >>> loss = model(
        ...     input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices
        ... ).loss
        ```
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    if mask_time_indices is not None:
        mask_time_indices = mask_time_indices.to(mindspore.bool_)

    outputs = self.wav2vec2(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        mask_time_indices=mask_time_indices,
        return_dict=return_dict,
    )

    # 1. project all transformed features (including masked) to final vq dim
    transformer_features = self.project_hid(outputs[0])

    # 2. quantize all (unmasked) extracted features and project to final vq dim
    extract_features = self.dropout_features(outputs[1])

    if attention_mask is not None:
        # compute reduced attention_mask correponding to feature vectors
        attention_mask = self._get_feature_vector_attention_mask(
            extract_features.shape[1], attention_mask, add_adapter=False
        )

    quantized_features, codevector_perplexity = self.quantizer(
        extract_features, mask_time_indices=mask_time_indices
    )
    quantized_features = self.project_q(quantized_features)

    loss = contrastive_loss = diversity_loss = None
    if sampled_negative_indices is not None:
        batch_size, sequence_length, hidden_size = quantized_features.shape

        # for training, we sample negatives
        # 3. sample K negatives (distractors) quantized states for contrastive loss
        # if attention_mask is passed, make sure that padded feature vectors cannot be sampled
        # sample negative quantized vectors BTC => (BxT)C
        negative_quantized_features = quantized_features.view(-1, hidden_size)[
            sampled_negative_indices.long().view(-1)
        ]
        negative_quantized_features = negative_quantized_features.view(
            batch_size, sequence_length, -1, hidden_size
        ).permute(2, 0, 1, 3)

        # 4. compute logits, corresponding to `logs = sim(c_t, [q_t, \sim{q}_t]) / \kappa`
        # of equation (3) in https://arxiv.org/pdf/2006.11477.pdf
        logits = self.compute_contrastive_logits(
            quantized_features[None, :],
            negative_quantized_features,
            transformer_features,
            self.config.contrastive_logits_temperature,
        )

        # 5. if a negative vector is identical to the positive (i.e. when codebook utilization is low),
        # its cosine similarity will be masked
        neg_is_pos = (quantized_features == negative_quantized_features).all(-1)

        if neg_is_pos.any():
            # NOTE: avoid loss NaN
            # float("-inf") => finfo(logits.dtype, 'min') := -3.40282e+38
            logits[1:][neg_is_pos] = -3.40282e+35

        # 6. compute contrastive loss \mathbf{L}_m = cross_entropy(logs) =
        # -log(exp(sim(c_t, q_t)/\kappa) / \sum_{\sim{q}} exp(sim(c_t, \sim{q})/\kappa))
        logits = logits.swapaxes(0, 2).reshape(-1, logits.shape[0])
        target = ((1 - mask_time_indices.long()) * -100).swapaxes(0, 1).flatten()

        contrastive_loss = ops.cross_entropy(logits.float(), target, reduction="sum")
        # 7. compute diversity loss: \mathbf{L}_d
        num_codevectors = self.config.num_codevectors_per_group * self.config.num_codevector_groups
        diversity_loss = ((num_codevectors - codevector_perplexity) / num_codevectors) * mask_time_indices.sum()

        # 8. \mathbf{L} = \mathbf{L}_m + \alpha * \mathbf{L}_d
        loss = contrastive_loss + self.config.diversity_loss_weight * diversity_loss

    if not return_dict:
        if loss is not None:
            return (loss, transformer_features, quantized_features, codevector_perplexity) + outputs[2:]
        return (transformer_features, quantized_features, codevector_perplexity) + outputs[2:]

    return Wav2Vec2ForPreTrainingOutput(
        loss=loss,
        projected_states=transformer_features,
        projected_quantized_states=quantized_features,
        codevector_perplexity=codevector_perplexity,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
        contrastive_loss=contrastive_loss,
        diversity_loss=diversity_loss,
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTraining.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.wav2vec2.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTraining.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameters will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTraining.set_gumbel_temperature(temperature)

Set the Gumbel softmax temperature to a given value. Only necessary for training

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def set_gumbel_temperature(self, temperature: int):
    """
    Set the Gumbel softmax temperature to a given value. Only necessary for training
    """
    self.quantizer.temperature = temperature

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTrainingOutput dataclass

Bases: ModelOutput

Output type of [Wav2Vec2ForPreTraining], with potential hidden states and attentions.

PARAMETER	DESCRIPTION
loss	Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the official paper . (classification) loss. TYPE: *optional*, returned when `sample_negative_indices` are passed, `Tensor` of shape `(1,)` DEFAULT: None
projected_states	Hidden-states of the model projected to config.proj_codevector_dim that can be used to predict the masked projected quantized states. TYPE: `Tensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)` DEFAULT: None
projected_quantized_states	Quantized extracted feature vectors projected to config.proj_codevector_dim representing the positive target vectors for contrastive loss. TYPE: `Tensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)` DEFAULT: None
contrastive_loss	The contrastive loss (L_m) as stated in the official paper . TYPE: *optional*, returned when `sample_negative_indices` are passed, `Tensor` of shape `(1,)` DEFAULT: None
diversity_loss	The diversity loss (L_d) as stated in the official paper . TYPE: *optional*, returned when `sample_negative_indices` are passed, `Tensor` of shape `(1,)` DEFAULT: None

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

@dataclass
class Wav2Vec2ForPreTrainingOutput(ModelOutput):
    """
    Output type of [`Wav2Vec2ForPreTraining`], with potential hidden states and attentions.

    Args:
        loss (*optional*, returned when `sample_negative_indices` are passed, `Tensor` of shape `(1,)`):
            Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official
            paper](https://arxiv.org/pdf/2006.11477.pdf) . (classification) loss.
        projected_states (`Tensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`):
            Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked
            projected quantized states.
        projected_quantized_states (`Tensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`):
            Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive
            target vectors for contrastive loss.
        hidden_states (`tuple(Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when
            `config.output_hidden_states=True`):
            Tuple of `Tensor` (one for the output of the embeddings + one for the output of each layer) of
            shape `(batch_size, sequence_length, hidden_size)`.

            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
        attentions (`tuple(Tensor)`, *optional*, returned when `output_attentions=True` is passed or when
            `config.output_attentions=True`):
            Tuple of `Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
            sequence_length)`.

            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
            heads.
        contrastive_loss (*optional*, returned when `sample_negative_indices` are passed, `Tensor` of shape `(1,)`):
            The contrastive loss (L_m) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) .
        diversity_loss (*optional*, returned when `sample_negative_indices` are passed, `Tensor` of shape `(1,)`):
            The diversity loss (L_d) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) .
    """
    loss: Optional[Tensor] = None
    projected_states: Tensor = None
    projected_quantized_states: Tensor = None
    codevector_perplexity: Tensor = None
    hidden_states: Optional[Tuple[Tensor]] = None
    attentions: Optional[Tuple[Tensor]] = None
    contrastive_loss: Optional[Tensor] = None
    diversity_loss: Optional[Tensor] = None

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification

Bases: Wav2Vec2PreTrainedModel

The Wav2Vec2ForSequenceClassification class represents a Wav2Vec2 model for sequence classification tasks. It inherits from the Wav2Vec2PreTrainedModel class. This class provides methods for initializing the model, freezing specific components, and computing the sequence classification output. It also includes methods for handling the feature extractor, feature encoder, and base model. The class supports the construction of the sequence classification output and provides options for setting various parameters such as attention masks, output attentions, output hidden states, and labels.

Deprecated methods such as freeze_feature_extractor and freeze_base_model are included along with their corresponding replacements. The construct method computes the sequence classification/regression loss and handles the classification output based on the input values, attention masks, and labels. The class allows for fine-tuning the model for sequence classification tasks while providing flexibility in handling different components and parameters.

For detailed information about the class and its methods, refer to the individual method docstrings and the base class Wav2Vec2PreTrainedModel for additional context and functionality.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2ForSequenceClassification(Wav2Vec2PreTrainedModel):

    """
    The `Wav2Vec2ForSequenceClassification` class represents a Wav2Vec2 model for sequence classification tasks.
    It inherits from the `Wav2Vec2PreTrainedModel` class. This class provides methods for initializing the model,
    freezing specific components, and computing the sequence classification output. It also  includes methods for
    handling the feature extractor, feature encoder, and base model. The class supports the construction of the sequence
    classification output and provides options for setting various parameters such as attention masks, output attentions,
    output hidden states, and labels.

    Deprecated methods such as `freeze_feature_extractor` and `freeze_base_model` are included along with their
    corresponding replacements. The `construct` method computes the sequence classification/regression loss and handles
    the classification output based on the input values, attention masks, and labels. The class allows for fine-tuning
    the model for sequence classification tasks while providing flexibility in handling different components and
    parameters.

    For detailed information about the class and its methods, refer to the individual method docstrings and the base
    class `Wav2Vec2PreTrainedModel` for additional context and functionality.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a new instance of the Wav2Vec2ForSequenceClassification class.

        Args:
            self: The object itself.
            config (Wav2Vec2Config): An instance of Wav2Vec2Config containing the configuration settings for the model.

        Returns:
            None.

        Raises:
            ValueError: Raised if the 'add_adapter' attribute is set to True in the config, as sequence classification
                does not support the use of Wav2Vec2 adapters.
        """
        super().__init__(config)

        if hasattr(config, "add_adapter") and config.add_adapter:
            raise ValueError(
                "Sequence classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)"
            )
        self.wav2vec2 = Wav2Vec2Model(config)
        num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
        if config.use_weighted_layer_sum:
            self.layer_weights = Parameter(ops.ones(num_layers) / num_layers)
        self.projector = nn.Dense(config.hidden_size, config.classifier_proj_size)
        self.classifier = nn.Dense(config.classifier_proj_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameters will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.wav2vec2.feature_extractor._freeze_parameters()

    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for _, param in self.wav2vec2.parameters_and_names():
            param.requires_grad = False

    def construct(
        self,
        input_values: Optional[Tensor],
        attention_mask: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[Tensor] = None,
    ) -> Union[Tuple, SequenceClassifierOutput]:
        r"""
        Args:
            labels (`Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

        outputs = self.wav2vec2(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        if self.config.use_weighted_layer_sum:
            hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
            hidden_states = ops.stack(hidden_states, axis=1)
            norm_weights = ops.softmax(self.layer_weights, axis=-1)
            hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
        else:
            hidden_states = outputs[0]

        hidden_states = self.projector(hidden_states)
        if attention_mask is None:
            pooled_output = hidden_states.mean(axis=1)
        else:
            padding_mask = self._get_feature_vector_attention_mask(hidden_states.shape[1], attention_mask)
            hidden_states[~padding_mask] = 0.0
            pooled_output = hidden_states.sum(axis=1) / padding_mask.sum(axis=1).view(-1, 1)

        logits = self.classifier(pooled_output)

        loss = None
        if labels is not None:
            labels = labels.astype(mindspore.int32)
            loss = ops.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))

        if not return_dict:
            output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
            return ((loss,) + output) if loss is not None else output

        return SequenceClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification.__init__(config)

Initializes a new instance of the Wav2Vec2ForSequenceClassification class.

PARAMETER	DESCRIPTION
self	The object itself.
config	An instance of Wav2Vec2Config containing the configuration settings for the model. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	Raised if the 'add_adapter' attribute is set to True in the config, as sequence classification does not support the use of Wav2Vec2 adapters.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a new instance of the Wav2Vec2ForSequenceClassification class.

    Args:
        self: The object itself.
        config (Wav2Vec2Config): An instance of Wav2Vec2Config containing the configuration settings for the model.

    Returns:
        None.

    Raises:
        ValueError: Raised if the 'add_adapter' attribute is set to True in the config, as sequence classification
            does not support the use of Wav2Vec2 adapters.
    """
    super().__init__(config)

    if hasattr(config, "add_adapter") and config.add_adapter:
        raise ValueError(
            "Sequence classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)"
        )
    self.wav2vec2 = Wav2Vec2Model(config)
    num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
    if config.use_weighted_layer_sum:
        self.layer_weights = Parameter(ops.ones(num_layers) / num_layers)
    self.projector = nn.Dense(config.hidden_size, config.classifier_proj_size)
    self.classifier = nn.Dense(config.classifier_proj_size, config.num_labels)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification.construct(input_values, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). TYPE: `Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    input_values: Optional[Tensor],
    attention_mask: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[Tensor] = None,
) -> Union[Tuple, SequenceClassifierOutput]:
    r"""
    Args:
        labels (`Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

    outputs = self.wav2vec2(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    if self.config.use_weighted_layer_sum:
        hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
        hidden_states = ops.stack(hidden_states, axis=1)
        norm_weights = ops.softmax(self.layer_weights, axis=-1)
        hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
    else:
        hidden_states = outputs[0]

    hidden_states = self.projector(hidden_states)
    if attention_mask is None:
        pooled_output = hidden_states.mean(axis=1)
    else:
        padding_mask = self._get_feature_vector_attention_mask(hidden_states.shape[1], attention_mask)
        hidden_states[~padding_mask] = 0.0
        pooled_output = hidden_states.sum(axis=1) / padding_mask.sum(axis=1).view(-1, 1)

    logits = self.classifier(pooled_output)

    loss = None
    if labels is not None:
        labels = labels.astype(mindspore.int32)
        loss = ops.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))

    if not return_dict:
        output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
        return ((loss,) + output) if loss is not None else output

    return SequenceClassifierOutput(
        loss=loss,
        logits=logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for _, param in self.wav2vec2.parameters_and_names():
        param.requires_grad = False

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.wav2vec2.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameters will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForXVector

Bases: Wav2Vec2PreTrainedModel

This class represents a Wav2Vec2 model for extracting x-vector embeddings from audio data. It inherits from the Wav2Vec2PreTrainedModel class, and provides methods for freezing specific model components and computing x-vector embeddings from input audio data.

The class contains methods for freezing the feature extractor, freezing the feature encoder, and freezing the base model to disable gradient computation for specific model components. Additionally, it includes methods for computing the output length of the TDNN layers and for constructing x-vector embeddings from input audio data.

The construct method takes input audio data and optional parameters such as attention mask and labels, and returns x-vector embeddings along with optional loss and hidden states. The method also supports outputting hidden states and attentions based on the configuration settings.

This class is designed to be used for x-vector extraction tasks and provides flexibility for customizing the model's behavior and freezing specific components during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2ForXVector(Wav2Vec2PreTrainedModel):

    """
    This class represents a Wav2Vec2 model for extracting x-vector embeddings from audio data. It inherits from the
    Wav2Vec2PreTrainedModel class, and provides methods for freezing specific model components and computing x-vector
    embeddings from input audio data.

    The class contains methods for freezing the feature extractor, freezing the feature encoder, and freezing the base
    model to disable gradient computation for specific model components. Additionally, it includes methods for computing
    the output length of the TDNN layers and for constructing x-vector embeddings from input audio data.

    The construct method takes input audio data and optional parameters such as attention mask and labels, and returns
    x-vector embeddings along with optional loss and hidden states. The method also supports outputting hidden states
    and attentions based on the configuration settings.

    This class is designed to be used for x-vector extraction tasks and provides flexibility for customizing the model's
    behavior and freezing specific components during training.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes an instance of the Wav2Vec2ForXVector class.

        Args:
            self: The instance of the Wav2Vec2ForXVector class.
            config (Wav2Vec2Config): An object of type Wav2Vec2Config containing configuration settings for the model.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)

        self.wav2vec2 = Wav2Vec2Model(config)
        num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
        if config.use_weighted_layer_sum:
            self.layer_weights = Parameter(ops.ones(num_layers) / num_layers)
        self.projector = nn.Dense(config.hidden_size, config.tdnn_dim[0])

        tdnn_layers = [TDNNLayer(config, i) for i in range(len(config.tdnn_dim))]
        self.tdnn = nn.CellList(tdnn_layers)

        self.feature_extractor = nn.Dense(config.tdnn_dim[-1] * 2, config.xvector_output_dim)
        self.classifier = nn.Dense(config.xvector_output_dim, config.xvector_output_dim)

        self.objective = AMSoftmaxLoss(config.xvector_output_dim, config.num_labels)

        self.init_weights()

    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.wav2vec2.feature_extractor._freeze_parameters()

    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for named, param in self.wav2vec2.parameters_and_names():
            param.requires_grad = False

    def _get_tdnn_output_lengths(self, input_lengths: Union[Tensor, int]):
        """
        Computes the output length of the TDNN layers
        """
        def _conv_out_length(input_length, kernel_size, stride):
            # 1D convolutional layer output length formula taken
            # from https://pyops.org/docs/stable/generated/ops.nn.Conv1d.html
            return (input_length - kernel_size) // stride + 1

        for kernel_size in self.config.tdnn_kernel:
            input_lengths = _conv_out_length(input_lengths, kernel_size, 1)

        return input_lengths

    def construct(
        self,
        input_values: Optional[Tensor],
        attention_mask: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[Tensor] = None,
    ) -> Union[Tuple, XVectorOutput]:
        r"""
        Args:
            labels (`Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

        outputs = self.wav2vec2(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        if self.config.use_weighted_layer_sum:
            hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
            hidden_states = ops.stack(hidden_states, axis=1)
            norm_weights = ops.softmax(self.layer_weights, axis=-1)
            hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
        else:
            hidden_states = outputs[0]

        hidden_states = self.projector(hidden_states)

        for tdnn_layer in self.tdnn:
            hidden_states = tdnn_layer(hidden_states)

        # Statistic Pooling
        if attention_mask is None:
            mean_features = hidden_states.mean(axis=1)
            #std_features = hidden_states.std(axis=1)   # NOTE: buggy API
            std_features = ops.std(hidden_states, axis=1, keepdims=True).squeeze(1)
        else:
            feat_extract_output_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(axis=1))
            tdnn_output_lengths = self._get_tdnn_output_lengths(feat_extract_output_lengths)
            mean_features = []
            std_features = []
            for i, length in enumerate(tdnn_output_lengths):
                mean_features.append(hidden_states[i, :length].mean(axis=0))
                std_features.append(hidden_states[i, :length].std(axis=0))
            mean_features = ops.stack(mean_features)
            std_features = ops.stack(std_features)
        statistic_pooling = ops.cat([mean_features, std_features], axis=-1)

        output_embeddings = self.feature_extractor(statistic_pooling)
        logits = self.classifier(output_embeddings)

        loss = None
        if labels is not None:
            labels = labels.astype(mindspore.int32)
            loss = self.objective(logits, labels)

        if not return_dict:
            output = (logits, output_embeddings) + outputs[_HIDDEN_STATES_START_POSITION:]
            return ((loss,) + output) if loss is not None else output

        return XVectorOutput(
            loss=loss,
            logits=logits,
            embeddings=output_embeddings,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForXVector.__init__(config)

Initializes an instance of the Wav2Vec2ForXVector class.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2ForXVector class.
config	An object of type Wav2Vec2Config containing configuration settings for the model. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes an instance of the Wav2Vec2ForXVector class.

    Args:
        self: The instance of the Wav2Vec2ForXVector class.
        config (Wav2Vec2Config): An object of type Wav2Vec2Config containing configuration settings for the model.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)

    self.wav2vec2 = Wav2Vec2Model(config)
    num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
    if config.use_weighted_layer_sum:
        self.layer_weights = Parameter(ops.ones(num_layers) / num_layers)
    self.projector = nn.Dense(config.hidden_size, config.tdnn_dim[0])

    tdnn_layers = [TDNNLayer(config, i) for i in range(len(config.tdnn_dim))]
    self.tdnn = nn.CellList(tdnn_layers)

    self.feature_extractor = nn.Dense(config.tdnn_dim[-1] * 2, config.xvector_output_dim)
    self.classifier = nn.Dense(config.xvector_output_dim, config.xvector_output_dim)

    self.objective = AMSoftmaxLoss(config.xvector_output_dim, config.num_labels)

    self.init_weights()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForXVector.construct(input_values, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). TYPE: `Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    input_values: Optional[Tensor],
    attention_mask: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[Tensor] = None,
) -> Union[Tuple, XVectorOutput]:
    r"""
    Args:
        labels (`Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

    outputs = self.wav2vec2(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    if self.config.use_weighted_layer_sum:
        hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
        hidden_states = ops.stack(hidden_states, axis=1)
        norm_weights = ops.softmax(self.layer_weights, axis=-1)
        hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
    else:
        hidden_states = outputs[0]

    hidden_states = self.projector(hidden_states)

    for tdnn_layer in self.tdnn:
        hidden_states = tdnn_layer(hidden_states)

    # Statistic Pooling
    if attention_mask is None:
        mean_features = hidden_states.mean(axis=1)
        #std_features = hidden_states.std(axis=1)   # NOTE: buggy API
        std_features = ops.std(hidden_states, axis=1, keepdims=True).squeeze(1)
    else:
        feat_extract_output_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(axis=1))
        tdnn_output_lengths = self._get_tdnn_output_lengths(feat_extract_output_lengths)
        mean_features = []
        std_features = []
        for i, length in enumerate(tdnn_output_lengths):
            mean_features.append(hidden_states[i, :length].mean(axis=0))
            std_features.append(hidden_states[i, :length].std(axis=0))
        mean_features = ops.stack(mean_features)
        std_features = ops.stack(std_features)
    statistic_pooling = ops.cat([mean_features, std_features], axis=-1)

    output_embeddings = self.feature_extractor(statistic_pooling)
    logits = self.classifier(output_embeddings)

    loss = None
    if labels is not None:
        labels = labels.astype(mindspore.int32)
        loss = self.objective(logits, labels)

    if not return_dict:
        output = (logits, output_embeddings) + outputs[_HIDDEN_STATES_START_POSITION:]
        return ((loss,) + output) if loss is not None else output

    return XVectorOutput(
        loss=loss,
        logits=logits,
        embeddings=output_embeddings,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForXVector.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for named, param in self.wav2vec2.parameters_and_names():
        param.requires_grad = False

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForXVector.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.wav2vec2.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForXVector.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2GroupNormConvLayer

Bases: Cell

This class represents a group normalization convolutional layer used in the Wav2Vec2 model. It applies a 1D convolution operation followed by group normalization, activation, and layer normalization to the input hidden states.

PARAMETER	DESCRIPTION
config	The configuration object containing the settings for the Wav2Vec2 model. TYPE: Wav2Vec2Config
layer_id	The index of the convolutional layer in the model. Defaults to 0. TYPE: int DEFAULT: 0

ATTRIBUTE	DESCRIPTION
in_conv_dim	The input dimension of the convolutional layer. TYPE: int
out_conv_dim	The output dimension of the convolutional layer. TYPE: int
conv	The 1D convolutional layer used to process the hidden states. TYPE: Conv1d
activation	The activation function applied to the processed hidden states. TYPE: function
layer_norm	The group normalization layer applied to the hidden states. TYPE: GroupNorm

METHOD	DESCRIPTION
construct	Applies the convolutional layer, normalization, activation, and returns the processed hidden states.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2GroupNormConvLayer(nn.Cell):

    """
    This class represents a group normalization convolutional layer used in the Wav2Vec2 model.
    It applies a 1D convolution operation followed by group normalization, activation, and layer normalization to the
    input hidden states.

    Args:
        config (Wav2Vec2Config): The configuration object containing the settings for the Wav2Vec2 model.
        layer_id (int, optional): The index of the convolutional layer in the model. Defaults to 0.

    Attributes:
        in_conv_dim (int): The input dimension of the convolutional layer.
        out_conv_dim (int): The output dimension of the convolutional layer.
        conv (nn.Conv1d): The 1D convolutional layer used to process the hidden states.
        activation (function): The activation function applied to the processed hidden states.
        layer_norm (nn.GroupNorm): The group normalization layer applied to the hidden states.

    Methods:
        construct: Applies the convolutional layer, normalization, activation, and returns the processed hidden states.

    """
    def __init__(self, config: Wav2Vec2Config, layer_id=0):
        """
        Initializes an instance of the Wav2Vec2GroupNormConvLayer class.

        Args:
            self: The current instance of the class.
            config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing configuration settings.
            layer_id (int): The index of the convolutional layer within the configuration. Defaults to 0.

        Returns:
            None.

        Raises:
            ValueError: If the layer_id is less than 0.
            KeyError: If the specified activation function in config is not found in the ACT2FN dictionary.
            ValueError: If the specified pad_mode in the nn.Conv1d function is not 'valid'.
        """
        super().__init__()
        self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
        self.out_conv_dim = config.conv_dim[layer_id]

        self.conv = nn.Conv1d(
            self.in_conv_dim,
            self.out_conv_dim,
            kernel_size=config.conv_kernel[layer_id],
            stride=config.conv_stride[layer_id],
            has_bias=config.conv_bias,
            pad_mode='valid',
        )
        self.activation = ACT2FN[config.feat_extract_activation]
        self.layer_norm = nn.GroupNorm(num_groups=self.out_conv_dim, num_channels=self.out_conv_dim, affine=True)

    def construct(self, hidden_states):
        """
        This method constructs a group normalization convolutional layer for the Wav2Vec2 model.

        Args:
            self (Wav2Vec2GroupNormConvLayer): The instance of the Wav2Vec2GroupNormConvLayer class.
            hidden_states (torch.Tensor): The input tensor representing the hidden states to be processed by the group normalization convolutional layer.

        Returns:
            torch.Tensor: The processed tensor representing the hidden states after passing through the group normalization convolutional layer.

        Raises:
            None.
        """
        hidden_states = self.conv(hidden_states)
        hidden_states = self.layer_norm(hidden_states.unsqueeze(-1)).squeeze(-1)    # tmfix: GroupNorm only support 4D
        hidden_states = self.activation(hidden_states)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2GroupNormConvLayer.__init__(config, layer_id=0)

Initializes an instance of the Wav2Vec2GroupNormConvLayer class.

PARAMETER	DESCRIPTION
self	The current instance of the class.
config	An instance of the Wav2Vec2Config class containing configuration settings. TYPE: Wav2Vec2Config
layer_id	The index of the convolutional layer within the configuration. Defaults to 0. TYPE: int DEFAULT: 0

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the layer_id is less than 0.
KeyError	If the specified activation function in config is not found in the ACT2FN dictionary.
ValueError	If the specified pad_mode in the nn.Conv1d function is not 'valid'.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config, layer_id=0):
    """
    Initializes an instance of the Wav2Vec2GroupNormConvLayer class.

    Args:
        self: The current instance of the class.
        config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing configuration settings.
        layer_id (int): The index of the convolutional layer within the configuration. Defaults to 0.

    Returns:
        None.

    Raises:
        ValueError: If the layer_id is less than 0.
        KeyError: If the specified activation function in config is not found in the ACT2FN dictionary.
        ValueError: If the specified pad_mode in the nn.Conv1d function is not 'valid'.
    """
    super().__init__()
    self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
    self.out_conv_dim = config.conv_dim[layer_id]

    self.conv = nn.Conv1d(
        self.in_conv_dim,
        self.out_conv_dim,
        kernel_size=config.conv_kernel[layer_id],
        stride=config.conv_stride[layer_id],
        has_bias=config.conv_bias,
        pad_mode='valid',
    )
    self.activation = ACT2FN[config.feat_extract_activation]
    self.layer_norm = nn.GroupNorm(num_groups=self.out_conv_dim, num_channels=self.out_conv_dim, affine=True)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2GroupNormConvLayer.construct(hidden_states)

This method constructs a group normalization convolutional layer for the Wav2Vec2 model.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2GroupNormConvLayer class. TYPE: Wav2Vec2GroupNormConvLayer
hidden_states	The input tensor representing the hidden states to be processed by the group normalization convolutional layer. TYPE: Tensor

RETURNS	DESCRIPTION
	torch.Tensor: The processed tensor representing the hidden states after passing through the group normalization convolutional layer.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    """
    This method constructs a group normalization convolutional layer for the Wav2Vec2 model.

    Args:
        self (Wav2Vec2GroupNormConvLayer): The instance of the Wav2Vec2GroupNormConvLayer class.
        hidden_states (torch.Tensor): The input tensor representing the hidden states to be processed by the group normalization convolutional layer.

    Returns:
        torch.Tensor: The processed tensor representing the hidden states after passing through the group normalization convolutional layer.

    Raises:
        None.
    """
    hidden_states = self.conv(hidden_states)
    hidden_states = self.layer_norm(hidden_states.unsqueeze(-1)).squeeze(-1)    # tmfix: GroupNorm only support 4D
    hidden_states = self.activation(hidden_states)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2GumbelVectorQuantizer

Bases: Cell

Vector quantization using gumbel softmax. See `CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX for more information.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2GumbelVectorQuantizer(nn.Cell):
    """
    Vector quantization using gumbel softmax. See `[CATEGORICAL REPARAMETERIZATION WITH
    GUMBEL-SOFTMAX](https://arxiv.org/pdf/1611.01144.pdf) for more information.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a new instance of the Wav2Vec2GumbelVectorQuantizer class.

        Args:
            self: The instance of the Wav2Vec2GumbelVectorQuantizer class.
            config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing configuration parameters
                for the vector quantizer.

                - num_codevector_groups (int): The number of codevector groups.
                - num_codevectors_per_group (int): The number of codevectors per group.
                - codevector_dim (int): The dimension of the codevectors.

        Returns:
            None.

        Raises:
            ValueError: If `config.codevector_dim` is not divisible by `config.num_codevector_groups` for concatenation.
        """
        super().__init__()
        self.num_groups = config.num_codevector_groups
        self.num_vars = config.num_codevectors_per_group

        if config.codevector_dim % self.num_groups != 0:
            raise ValueError(
                f"`config.codevector_dim {config.codevector_dim} must be divisible "
                f"by `config.num_codevector_groups` {self.num_groups} for concatenation"
            )

        # storage for codebook variables (codewords)
        self.codevectors = Parameter(
            ops.zeros((1, self.num_groups * self.num_vars, config.codevector_dim // self.num_groups))
        )
        self.weight_proj = nn.Dense(config.conv_dim[-1], self.num_groups * self.num_vars)

        # can be decayed for training
        self.temperature = 2

    @staticmethod
    def _compute_perplexity(probs, mask=None):
        """
        Compute the perplexity of given probability distribution.

        Args:
            probs (Tensor): The input probability distribution. It should be a tensor of shape (N, D) where N is the
                number of elements and D is the dimensionality of the distribution. mask (Tensor, optional):
                A boolean tensor of the same shape as probs, indicating which elements to include in the computation.
                If provided, only the elements where mask is True will be considered. Defaults to None.

        Returns:
            None: This method does not return anything but updates the internal state of the class.

        Raises:
            ValueError: If the shape of probs and mask do not match.
            ValueError: If the dimensionality of probs is not 2.
        """
        if mask is not None:
            mask_extended = mask.flatten()[:, None, None].expand(probs.shape)
            probs = ops.where(mask_extended, probs, ops.zeros_like(probs))
            marginal_probs = probs.sum(axis=0) / mask.sum()
        else:
            marginal_probs = probs.mean(axis=0)

        perplexity = ops.exp(-ops.sum(marginal_probs * ops.log(marginal_probs + 1e-7), dim=-1)).sum()
        return perplexity

    def construct(self, hidden_states, mask_time_indices=None):
        '''
        Constructs codevectors and computes perplexity for Wav2Vec2GumbelVectorQuantizer.

        Args:
            self: The instance of the Wav2Vec2GumbelVectorQuantizer class.
            hidden_states (tensor): The input hidden states with shape (batch_size, sequence_length, hidden_size).
            mask_time_indices (tensor, optional): A binary mask tensor of shape (batch_size, sequence_length) where
                1s indicate valid time indices and 0s indicate masked time indices. Default is None.

        Returns:
            tuple:
                A tuple containing:

                - codevectors (tensor): The constructed codevectors with shape (batch_size, sequence_length, -1).
                - perplexity (tensor): The computed perplexity.

        Raises:
            ValueError: If the input hidden_states tensor has an invalid shape.
            RuntimeError: If the function encounters a runtime error during computation.
        '''
        batch_size, sequence_length, hidden_size = hidden_states.shape

        # project to codevector dim
        hidden_states = self.weight_proj(hidden_states)
        hidden_states = hidden_states.view(batch_size * sequence_length * self.num_groups, -1)

        if self.training:
            # sample code vector probs via gumbel in differentiateable way
            codevector_probs = ops.gumbel_softmax(
                hidden_states.float(), tau=float(self.temperature), hard=True
            ).type_as(hidden_states)

            # compute perplexity
            codevector_soft_dist = ops.softmax(
                hidden_states.view(batch_size * sequence_length, self.num_groups, -1).float(), axis=-1
            )
            perplexity = self._compute_perplexity(codevector_soft_dist, mask_time_indices)
        else:
            # take argmax in non-differentiable way
            # comptute hard codevector distribution (one hot)
            # NOTE: 把 hidden_states 变成 hardsoftmax(dim=-1) 形式
            codevector_idx = ops.argmax(hidden_states, dim=-1)      # (364) => (364, 1)
            x = hidden_states.new_zeros(hidden_states.shape)    # (364, 320)
            index = codevector_idx.view(-1, 1)
            update = ops.ones_like(index, dtype=hidden_states.dtype)    # fill with onehot
            codevector_probs = ops.tensor_scatter_elements(x, index, update, axis=-1)
            codevector_probs = codevector_probs.view(batch_size * sequence_length, self.num_groups, -1) # (182, 2, 320)

            perplexity = self._compute_perplexity(codevector_probs, mask_time_indices)

        codevector_probs = codevector_probs.view(batch_size * sequence_length, -1)
        # use probs to retrieve codevectors
        codevectors_per_group = codevector_probs.unsqueeze(-1) * self.codevectors
        codevectors = codevectors_per_group.view(batch_size * sequence_length, self.num_groups, self.num_vars, -1)
        codevectors = codevectors.sum(-2).view(batch_size, sequence_length, -1)

        return codevectors, perplexity

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2GumbelVectorQuantizer.__init__(config)

Initializes a new instance of the Wav2Vec2GumbelVectorQuantizer class.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2GumbelVectorQuantizer class.
config	An instance of the Wav2Vec2Config class containing configuration parameters for the vector quantizer. num_codevector_groups (int): The number of codevector groups. num_codevectors_per_group (int): The number of codevectors per group. codevector_dim (int): The dimension of the codevectors. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If config.codevector_dim is not divisible by config.num_codevector_groups for concatenation.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a new instance of the Wav2Vec2GumbelVectorQuantizer class.

    Args:
        self: The instance of the Wav2Vec2GumbelVectorQuantizer class.
        config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing configuration parameters
            for the vector quantizer.

            - num_codevector_groups (int): The number of codevector groups.
            - num_codevectors_per_group (int): The number of codevectors per group.
            - codevector_dim (int): The dimension of the codevectors.

    Returns:
        None.

    Raises:
        ValueError: If `config.codevector_dim` is not divisible by `config.num_codevector_groups` for concatenation.
    """
    super().__init__()
    self.num_groups = config.num_codevector_groups
    self.num_vars = config.num_codevectors_per_group

    if config.codevector_dim % self.num_groups != 0:
        raise ValueError(
            f"`config.codevector_dim {config.codevector_dim} must be divisible "
            f"by `config.num_codevector_groups` {self.num_groups} for concatenation"
        )

    # storage for codebook variables (codewords)
    self.codevectors = Parameter(
        ops.zeros((1, self.num_groups * self.num_vars, config.codevector_dim // self.num_groups))
    )
    self.weight_proj = nn.Dense(config.conv_dim[-1], self.num_groups * self.num_vars)

    # can be decayed for training
    self.temperature = 2

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2GumbelVectorQuantizer.construct(hidden_states, mask_time_indices=None)

Constructs codevectors and computes perplexity for Wav2Vec2GumbelVectorQuantizer.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2GumbelVectorQuantizer class.
hidden_states	The input hidden states with shape (batch_size, sequence_length, hidden_size). TYPE: tensor
mask_time_indices	A binary mask tensor of shape (batch_size, sequence_length) where 1s indicate valid time indices and 0s indicate masked time indices. Default is None. TYPE: tensor DEFAULT: None

RETURNS	DESCRIPTION
tuple	A tuple containing: codevectors (tensor): The constructed codevectors with shape (batch_size, sequence_length, -1). perplexity (tensor): The computed perplexity.

RAISES	DESCRIPTION
ValueError	If the input hidden_states tensor has an invalid shape.
RuntimeError	If the function encounters a runtime error during computation.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states, mask_time_indices=None):
    '''
    Constructs codevectors and computes perplexity for Wav2Vec2GumbelVectorQuantizer.

    Args:
        self: The instance of the Wav2Vec2GumbelVectorQuantizer class.
        hidden_states (tensor): The input hidden states with shape (batch_size, sequence_length, hidden_size).
        mask_time_indices (tensor, optional): A binary mask tensor of shape (batch_size, sequence_length) where
            1s indicate valid time indices and 0s indicate masked time indices. Default is None.

    Returns:
        tuple:
            A tuple containing:

            - codevectors (tensor): The constructed codevectors with shape (batch_size, sequence_length, -1).
            - perplexity (tensor): The computed perplexity.

    Raises:
        ValueError: If the input hidden_states tensor has an invalid shape.
        RuntimeError: If the function encounters a runtime error during computation.
    '''
    batch_size, sequence_length, hidden_size = hidden_states.shape

    # project to codevector dim
    hidden_states = self.weight_proj(hidden_states)
    hidden_states = hidden_states.view(batch_size * sequence_length * self.num_groups, -1)

    if self.training:
        # sample code vector probs via gumbel in differentiateable way
        codevector_probs = ops.gumbel_softmax(
            hidden_states.float(), tau=float(self.temperature), hard=True
        ).type_as(hidden_states)

        # compute perplexity
        codevector_soft_dist = ops.softmax(
            hidden_states.view(batch_size * sequence_length, self.num_groups, -1).float(), axis=-1
        )
        perplexity = self._compute_perplexity(codevector_soft_dist, mask_time_indices)
    else:
        # take argmax in non-differentiable way
        # comptute hard codevector distribution (one hot)
        # NOTE: 把 hidden_states 变成 hardsoftmax(dim=-1) 形式
        codevector_idx = ops.argmax(hidden_states, dim=-1)      # (364) => (364, 1)
        x = hidden_states.new_zeros(hidden_states.shape)    # (364, 320)
        index = codevector_idx.view(-1, 1)
        update = ops.ones_like(index, dtype=hidden_states.dtype)    # fill with onehot
        codevector_probs = ops.tensor_scatter_elements(x, index, update, axis=-1)
        codevector_probs = codevector_probs.view(batch_size * sequence_length, self.num_groups, -1) # (182, 2, 320)

        perplexity = self._compute_perplexity(codevector_probs, mask_time_indices)

    codevector_probs = codevector_probs.view(batch_size * sequence_length, -1)
    # use probs to retrieve codevectors
    codevectors_per_group = codevector_probs.unsqueeze(-1) * self.codevectors
    codevectors = codevectors_per_group.view(batch_size * sequence_length, self.num_groups, self.num_vars, -1)
    codevectors = codevectors.sum(-2).view(batch_size, sequence_length, -1)

    return codevectors, perplexity

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2LayerNormConvLayer

Bases: Cell

This class represents a convolutional layer with layer normalization in the Wav2Vec2 model. It inherits from the nn.Cell class.

ATTRIBUTE	DESCRIPTION
config	The configuration object for the Wav2Vec2 model. TYPE: Wav2Vec2Config
layer_id	The ID of the current layer. TYPE: int

METHOD	DESCRIPTION
__init__	Initializes the Wav2Vec2LayerNormConvLayer with the given configuration and layer ID.
construct	Applies the convolutional layer with layer normalization to the input hidden states.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2LayerNormConvLayer(nn.Cell):

    """
    This class represents a convolutional layer with layer normalization in the Wav2Vec2 model.
    It inherits from the nn.Cell class.

    Attributes:
        config (Wav2Vec2Config): The configuration object for the Wav2Vec2 model.
        layer_id (int): The ID of the current layer.

    Methods:
        __init__:
            Initializes the Wav2Vec2LayerNormConvLayer with the given configuration and layer ID.

        construct:
            Applies the convolutional layer with layer normalization to the input hidden states.

    """
    def __init__(self, config: Wav2Vec2Config, layer_id=0):
        """
        Initialize the Wav2Vec2LayerNormConvLayer.

        Args:
            config (Wav2Vec2Config): The configuration object containing the parameters for the layer.
            layer_id (int, optional): The ID of the layer. Defaults to 0.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
        self.out_conv_dim = config.conv_dim[layer_id]

        self.conv = nn.Conv1d(
            self.in_conv_dim,
            self.out_conv_dim,
            kernel_size=config.conv_kernel[layer_id],
            stride=config.conv_stride[layer_id],
            has_bias=config.conv_bias,
            pad_mode='valid',
        )
        self.layer_norm = nn.LayerNorm(self.out_conv_dim)
        self.activation = ACT2FN[config.feat_extract_activation]

    def construct(self, hidden_states):
        """
        Construct the hidden states using the Wav2Vec2LayerNormConvLayer method.

        Args:
            self (Wav2Vec2LayerNormConvLayer): An instance of the Wav2Vec2LayerNormConvLayer class.
            hidden_states (Tensor): The input hidden states to be processed.
                It should have the shape (batch_size, sequence_length, feature_dim).

        Returns:
            None.

        Raises:
            None.
        """
        hidden_states = self.conv(hidden_states)
        hidden_states = hidden_states.swapaxes(-2, -1)
        hidden_states = self.layer_norm(hidden_states)
        hidden_states = hidden_states.swapaxes(-2, -1)
        hidden_states = self.activation(hidden_states)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2LayerNormConvLayer.__init__(config, layer_id=0)

Initialize the Wav2Vec2LayerNormConvLayer.

PARAMETER	DESCRIPTION
config	The configuration object containing the parameters for the layer. TYPE: Wav2Vec2Config
layer_id	The ID of the layer. Defaults to 0. TYPE: int DEFAULT: 0

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config, layer_id=0):
    """
    Initialize the Wav2Vec2LayerNormConvLayer.

    Args:
        config (Wav2Vec2Config): The configuration object containing the parameters for the layer.
        layer_id (int, optional): The ID of the layer. Defaults to 0.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
    self.out_conv_dim = config.conv_dim[layer_id]

    self.conv = nn.Conv1d(
        self.in_conv_dim,
        self.out_conv_dim,
        kernel_size=config.conv_kernel[layer_id],
        stride=config.conv_stride[layer_id],
        has_bias=config.conv_bias,
        pad_mode='valid',
    )
    self.layer_norm = nn.LayerNorm(self.out_conv_dim)
    self.activation = ACT2FN[config.feat_extract_activation]

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2LayerNormConvLayer.construct(hidden_states)

Construct the hidden states using the Wav2Vec2LayerNormConvLayer method.

PARAMETER	DESCRIPTION
self	An instance of the Wav2Vec2LayerNormConvLayer class. TYPE: Wav2Vec2LayerNormConvLayer
hidden_states	The input hidden states to be processed. It should have the shape (batch_size, sequence_length, feature_dim). TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    """
    Construct the hidden states using the Wav2Vec2LayerNormConvLayer method.

    Args:
        self (Wav2Vec2LayerNormConvLayer): An instance of the Wav2Vec2LayerNormConvLayer class.
        hidden_states (Tensor): The input hidden states to be processed.
            It should have the shape (batch_size, sequence_length, feature_dim).

    Returns:
        None.

    Raises:
        None.
    """
    hidden_states = self.conv(hidden_states)
    hidden_states = hidden_states.swapaxes(-2, -1)
    hidden_states = self.layer_norm(hidden_states)
    hidden_states = hidden_states.swapaxes(-2, -1)
    hidden_states = self.activation(hidden_states)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Model

Bases: Wav2Vec2PreTrainedModel

The Wav2Vec2Model class is a Python class that represents a Wav2Vec2 model for speech recognition. It is a subclass of the Wav2Vec2PreTrainedModel class.

Wav2Vec2Model inherits the following attributes and methods from the parent class:

• config: An instance of the Wav2Vec2Config class, containing the configuration parameters for the model.
• feature_extractor: An instance of the Wav2Vec2FeatureEncoder class, responsible for extracting features from the input waveform.
• feature_projection: An instance of the Wav2Vec2FeatureProjection class, responsible for projecting the extracted features.
• encoder: An instance of the Wav2Vec2Encoder or Wav2Vec2EncoderStableLayerNorm class, responsible for encoding the hidden states.
• adapter: An instance of the Wav2Vec2Adapter class, used to adapt the hidden states (optional).
• post_init(): A method called after the initialization of the model.

The Wav2Vec2Model class also defines the following methods:

• freeze_feature_extractor: Disables the gradient computation for the feature encoder, preventing its parameters from being updated during training.
• freeze_feature_encoder: Disables the gradient computation for the feature encoder, preventing its parameters from being updated during training.
• _mask_hidden_states: Masks extracted features along the time axis and/or the feature axis according to SpecAugment.
• construct: Constructs the model by processing the input values and returns the model outputs.

Please note that the freeze_feature_extractor() method is deprecated and will be removed in Transformers v5. The equivalent freeze_feature_encoder() method should be used instead.

For more information about the Wav2Vec2 model, please refer to the official paper [SpecAugment] (https://arxiv.org/abs/1904.08779).

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2Model(Wav2Vec2PreTrainedModel):

    """
    The `Wav2Vec2Model` class is a Python class that represents a Wav2Vec2 model for speech recognition.
    It is a subclass of the `Wav2Vec2PreTrainedModel` class.

    Wav2Vec2Model inherits the following attributes and methods from the parent class:

    - `config`: An instance of the `Wav2Vec2Config` class, containing the configuration parameters for the model.
    - `feature_extractor`: An instance of the `Wav2Vec2FeatureEncoder` class, responsible for extracting features
    from the input waveform.
    - `feature_projection`: An instance of the `Wav2Vec2FeatureProjection` class, responsible for projecting the
    extracted features.
    - `encoder`: An instance of the `Wav2Vec2Encoder` or `Wav2Vec2EncoderStableLayerNorm` class, responsible for
    encoding the hidden states.
    - `adapter`: An instance of the `Wav2Vec2Adapter` class, used to adapt the hidden states (optional).
    - `post_init()`: A method called after the initialization of the model.

    The `Wav2Vec2Model` class also defines the following methods:

    - `freeze_feature_extractor`: Disables the gradient computation for the feature encoder, preventing its parameters
    from being updated during training.
    - `freeze_feature_encoder`: Disables the gradient computation for the feature encoder, preventing its parameters
    from being updated during training.
    - `_mask_hidden_states`: Masks extracted features along
    the time axis and/or the feature axis according to SpecAugment.
    - `construct`: Constructs the model by processing the input values and returns the model outputs.

    Please note that the `freeze_feature_extractor()` method is deprecated and will be removed in Transformers v5.
    The equivalent `freeze_feature_encoder()` method should be used instead.

    For more information about the Wav2Vec2 model, please refer to the official paper [SpecAugment]
    (https://arxiv.org/abs/1904.08779).
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a new instance of the Wav2Vec2Model class.

        Args:
            self: The instance of the Wav2Vec2Model class.
            config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
                for the model.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not of type Wav2Vec2Config.
            ValueError: If the config parameters mask_time_prob or mask_feature_prob are less than 0.0.
            ValueError: If the config parameter do_stable_layer_norm is not a boolean value.
            ValueError: If the config parameter hidden_size is not defined.
            ValueError: If an error occurs during the initialization process.
        """
        super().__init__(config)
        self.config = config
        self.feature_extractor = Wav2Vec2FeatureEncoder(config)
        self.feature_projection = Wav2Vec2FeatureProjection(config)

        # model only needs masking vector if mask prob is > 0.0
        if config.mask_time_prob > 0.0 or config.mask_feature_prob > 0.0:
            self.masked_spec_embed = Parameter(initializer(Uniform(), (config.hidden_size,), dtype=mindspore.float32))

        if config.do_stable_layer_norm:
            self.encoder = Wav2Vec2EncoderStableLayerNorm(config)
        else:
            self.encoder = Wav2Vec2Encoder(config)

        self.adapter = Wav2Vec2Adapter(config) if config.add_adapter else None

        # Initialize weights and apply final processing
        self.post_init()

    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameters will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.feature_extractor._freeze_parameters()

    def _mask_hidden_states(
        self,
        hidden_states: Tensor,
        mask_time_indices: Optional[Tensor] = None,
        attention_mask: Optional[Tensor] = None,
    ):
        """
        Masks extracted features along time axis and/or along feature axis according to
        [SpecAugment](https://arxiv.org/abs/1904.08779).
        """
        # `config.apply_spec_augment` can set masking to False
        if not getattr(self.config, "apply_spec_augment", True):
            return hidden_states

        # generate indices & apply SpecAugment along time axis
        batch_size, sequence_length, hidden_size = hidden_states.shape

        if mask_time_indices is not None:
            # apply SpecAugment along time axis with given mask_time_indices
            hidden_states[mask_time_indices] = self.masked_spec_embed.to(hidden_states.dtype)
        elif self.config.mask_time_prob > 0 and self.training:
            mask_time_indices = _compute_mask_indices(
                (batch_size, sequence_length),
                mask_prob=self.config.mask_time_prob,
                mask_length=self.config.mask_time_length,
                attention_mask=attention_mask,
                min_masks=self.config.mask_time_min_masks,
            )
            mask_time_indices = Tensor(mask_time_indices, dtype=mindspore.bool_)
            hidden_states[mask_time_indices] = self.masked_spec_embed.to(hidden_states.dtype)

        if self.config.mask_feature_prob > 0 and self.training:
            # generate indices & apply SpecAugment along feature axis
            mask_feature_indices = _compute_mask_indices(
                (batch_size, hidden_size),
                mask_prob=self.config.mask_feature_prob,
                mask_length=self.config.mask_feature_length,
                min_masks=self.config.mask_feature_min_masks,
            )
            mask_feature_indices = Tensor(mask_feature_indices, dtype=mindspore.bool_)
            mask_feature_indices = mask_feature_indices[:, None].expand(-1, sequence_length, -1)
            hidden_states[mask_feature_indices] = 0

        return hidden_states

    def construct(
        self,
        input_values: Optional[Tensor],
        attention_mask: Optional[Tensor] = None,
        mask_time_indices: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, Wav2Vec2BaseModelOutput]:
        """
        Constructs the Wav2Vec2 model for processing input audio data.

        Args:
            self (Wav2Vec2Model): The instance of the Wav2Vec2Model class.
            input_values (Optional[Tensor]): The input audio data values with shape (batch_size, audio_length).
            attention_mask (Optional[Tensor]): The attention mask for the input audio data with shape
                (batch_size, audio_length).
            mask_time_indices (Optional[Tensor]): The mask for time indices with shape (batch_size, audio_length).
            output_attentions (Optional[bool]): Whether to output attentions. Defaults to None.
            output_hidden_states (Optional[bool]): Whether to output hidden states. Defaults to None.
            return_dict (Optional[bool]): Whether to return a dictionary of output. Defaults to None.

        Returns:
            Union[Tuple, Wav2Vec2BaseModelOutput]: The constructed model output, which can be a tuple or a
                Wav2Vec2BaseModelOutput object.

        Raises:
            ValueError: If the input_values and attention_mask have mismatched shapes.
            TypeError: If the input_values or attention_mask is not a Tensor.
            RuntimeError: If the encoder fails to process the input audio data.
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        extract_features = self.feature_extractor(input_values)
        extract_features = extract_features.swapaxes(1, 2)

        if attention_mask is not None:
            # compute reduced attention_mask corresponding to feature vectors
            attention_mask = self._get_feature_vector_attention_mask(
                extract_features.shape[1], attention_mask, add_adapter=False
            )

        hidden_states, extract_features = self.feature_projection(extract_features)
        hidden_states = self._mask_hidden_states(
            hidden_states, mask_time_indices=mask_time_indices, attention_mask=attention_mask
        )

        encoder_outputs = self.encoder(
            hidden_states,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = encoder_outputs[0]

        if self.adapter is not None:
            hidden_states = self.adapter(hidden_states)

        if not return_dict:
            return (hidden_states, extract_features) + encoder_outputs[1:]

        return Wav2Vec2BaseModelOutput(
            last_hidden_state=hidden_states,
            extract_features=extract_features,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
        )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Model.__init__(config)

Initializes a new instance of the Wav2Vec2Model class.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2Model class.
config	An instance of the Wav2Vec2Config class containing the configuration parameters for the model. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not of type Wav2Vec2Config.
ValueError	If the config parameters mask_time_prob or mask_feature_prob are less than 0.0.
ValueError	If the config parameter do_stable_layer_norm is not a boolean value.
ValueError	If the config parameter hidden_size is not defined.
ValueError	If an error occurs during the initialization process.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a new instance of the Wav2Vec2Model class.

    Args:
        self: The instance of the Wav2Vec2Model class.
        config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
            for the model.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not of type Wav2Vec2Config.
        ValueError: If the config parameters mask_time_prob or mask_feature_prob are less than 0.0.
        ValueError: If the config parameter do_stable_layer_norm is not a boolean value.
        ValueError: If the config parameter hidden_size is not defined.
        ValueError: If an error occurs during the initialization process.
    """
    super().__init__(config)
    self.config = config
    self.feature_extractor = Wav2Vec2FeatureEncoder(config)
    self.feature_projection = Wav2Vec2FeatureProjection(config)

    # model only needs masking vector if mask prob is > 0.0
    if config.mask_time_prob > 0.0 or config.mask_feature_prob > 0.0:
        self.masked_spec_embed = Parameter(initializer(Uniform(), (config.hidden_size,), dtype=mindspore.float32))

    if config.do_stable_layer_norm:
        self.encoder = Wav2Vec2EncoderStableLayerNorm(config)
    else:
        self.encoder = Wav2Vec2Encoder(config)

    self.adapter = Wav2Vec2Adapter(config) if config.add_adapter else None

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Model.construct(input_values, attention_mask=None, mask_time_indices=None, output_attentions=None, output_hidden_states=None, return_dict=None)

Constructs the Wav2Vec2 model for processing input audio data.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2Model class. TYPE: Wav2Vec2Model
input_values	The input audio data values with shape (batch_size, audio_length). TYPE: Optional[Tensor]
attention_mask	The attention mask for the input audio data with shape (batch_size, audio_length). TYPE: Optional[Tensor] DEFAULT: None
mask_time_indices	The mask for time indices with shape (batch_size, audio_length). TYPE: Optional[Tensor] DEFAULT: None
output_attentions	Whether to output attentions. Defaults to None. TYPE: Optional[bool] DEFAULT: None
output_hidden_states	Whether to output hidden states. Defaults to None. TYPE: Optional[bool] DEFAULT: None
return_dict	Whether to return a dictionary of output. Defaults to None. TYPE: Optional[bool] DEFAULT: None

RETURNS	DESCRIPTION
Union[Tuple, Wav2Vec2BaseModelOutput]	Union[Tuple, Wav2Vec2BaseModelOutput]: The constructed model output, which can be a tuple or a Wav2Vec2BaseModelOutput object.

RAISES	DESCRIPTION
ValueError	If the input_values and attention_mask have mismatched shapes.
TypeError	If the input_values or attention_mask is not a Tensor.
RuntimeError	If the encoder fails to process the input audio data.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(
    self,
    input_values: Optional[Tensor],
    attention_mask: Optional[Tensor] = None,
    mask_time_indices: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, Wav2Vec2BaseModelOutput]:
    """
    Constructs the Wav2Vec2 model for processing input audio data.

    Args:
        self (Wav2Vec2Model): The instance of the Wav2Vec2Model class.
        input_values (Optional[Tensor]): The input audio data values with shape (batch_size, audio_length).
        attention_mask (Optional[Tensor]): The attention mask for the input audio data with shape
            (batch_size, audio_length).
        mask_time_indices (Optional[Tensor]): The mask for time indices with shape (batch_size, audio_length).
        output_attentions (Optional[bool]): Whether to output attentions. Defaults to None.
        output_hidden_states (Optional[bool]): Whether to output hidden states. Defaults to None.
        return_dict (Optional[bool]): Whether to return a dictionary of output. Defaults to None.

    Returns:
        Union[Tuple, Wav2Vec2BaseModelOutput]: The constructed model output, which can be a tuple or a
            Wav2Vec2BaseModelOutput object.

    Raises:
        ValueError: If the input_values and attention_mask have mismatched shapes.
        TypeError: If the input_values or attention_mask is not a Tensor.
        RuntimeError: If the encoder fails to process the input audio data.
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    extract_features = self.feature_extractor(input_values)
    extract_features = extract_features.swapaxes(1, 2)

    if attention_mask is not None:
        # compute reduced attention_mask corresponding to feature vectors
        attention_mask = self._get_feature_vector_attention_mask(
            extract_features.shape[1], attention_mask, add_adapter=False
        )

    hidden_states, extract_features = self.feature_projection(extract_features)
    hidden_states = self._mask_hidden_states(
        hidden_states, mask_time_indices=mask_time_indices, attention_mask=attention_mask
    )

    encoder_outputs = self.encoder(
        hidden_states,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    hidden_states = encoder_outputs[0]

    if self.adapter is not None:
        hidden_states = self.adapter(hidden_states)

    if not return_dict:
        return (hidden_states, extract_features) + encoder_outputs[1:]

    return Wav2Vec2BaseModelOutput(
        last_hidden_state=hidden_states,
        extract_features=extract_features,
        hidden_states=encoder_outputs.hidden_states,
        attentions=encoder_outputs.attentions,
    )

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Model.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Model.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameters will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2NoLayerNormConvLayer

Bases: Cell

Wav2Vec2NoLayerNormConvLayer is a Python class representing a convolutional layer without layer normalization for the Wav2Vec2 model. This class inherits from nn.Cell and is used for processing audio features.

ATTRIBUTE	DESCRIPTION
config	The configuration object for the Wav2Vec2 model. TYPE: Wav2Vec2Config
layer_id	The index of the convolutional layer. TYPE: int
in_conv_dim	The input dimension of the convolutional layer. TYPE: int
out_conv_dim	The output dimension of the convolutional layer. TYPE: int
conv	The 1D convolutional operation applied to the input. TYPE: Conv1d
activation	The activation function used to process the convolutional output. TYPE: function

METHOD	DESCRIPTION
__init__	Initializes the Wav2Vec2NoLayerNormConvLayer with the provided configuration and layer index.
construct	Applies the convolutional and activation operations to the input hidden_states.

Note

This class is part of the Wav2Vec2 model and is specifically designed for processing audio features without layer normalization.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2NoLayerNormConvLayer(nn.Cell):

    """
    Wav2Vec2NoLayerNormConvLayer is a Python class representing a convolutional layer without layer normalization for
    the Wav2Vec2 model. This class inherits from nn.Cell and is used for processing audio features.

    Attributes:
        config (Wav2Vec2Config): The configuration object for the Wav2Vec2 model.
        layer_id (int): The index of the convolutional layer.
        in_conv_dim (int): The input dimension of the convolutional layer.
        out_conv_dim (int): The output dimension of the convolutional layer.
        conv (nn.Conv1d): The 1D convolutional operation applied to the input.
        activation (function): The activation function used to process the convolutional output.

    Methods:
        __init__: Initializes the Wav2Vec2NoLayerNormConvLayer with the provided configuration and layer index.
        construct: Applies the convolutional and activation operations to the input hidden_states.

    Note:
        This class is part of the Wav2Vec2 model and is specifically designed for processing audio features without
        layer normalization.
    """
    def __init__(self, config: Wav2Vec2Config, layer_id=0):
        """
        __init__(self, config: Wav2Vec2Config, layer_id=0)

        Initializes a new instance of the Wav2Vec2NoLayerNormConvLayer class.

        Args:
            self: The instance of the class.
            config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
                for the Wav2Vec2 model.
            layer_id (int, optional): The index of the layer. Defaults to 0. Specifies the layer for which the
                convolutional layer is initialized.

        Returns:
            None.

        Raises:
            ValueError: If the layer_id is less than 0.
            AttributeError: If the layer_id exceeds the maximum index available in the configuration parameters.
            TypeError: If the provided config parameter is not an instance of the Wav2Vec2Config class.
        """
        super().__init__()
        self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
        self.out_conv_dim = config.conv_dim[layer_id]

        self.conv = nn.Conv1d(
            self.in_conv_dim,
            self.out_conv_dim,
            kernel_size=config.conv_kernel[layer_id],
            stride=config.conv_stride[layer_id],
            has_bias=config.conv_bias,
            pad_mode='valid',
        )
        self.activation = ACT2FN[config.feat_extract_activation]

    def construct(self, hidden_states):
        """
        Constructs the hidden states using convolutional layer and activation function.

        Args:
            self (Wav2Vec2NoLayerNormConvLayer): The instance of the Wav2Vec2NoLayerNormConvLayer class.
            hidden_states (torch.Tensor): The input hidden states tensor.

        Returns:
            torch.Tensor: The constructed hidden states after applying convolution and activation.

        Raises:
            TypeError: If the input hidden_states is not a torch.Tensor.
        """
        hidden_states = self.conv(hidden_states)
        hidden_states = self.activation(hidden_states)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2NoLayerNormConvLayer.__init__(config, layer_id=0)

init(self, config: Wav2Vec2Config, layer_id=0)

Initializes a new instance of the Wav2Vec2NoLayerNormConvLayer class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An instance of the Wav2Vec2Config class containing the configuration parameters for the Wav2Vec2 model. TYPE: Wav2Vec2Config
layer_id	The index of the layer. Defaults to 0. Specifies the layer for which the convolutional layer is initialized. TYPE: int DEFAULT: 0

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the layer_id is less than 0.
AttributeError	If the layer_id exceeds the maximum index available in the configuration parameters.
TypeError	If the provided config parameter is not an instance of the Wav2Vec2Config class.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config, layer_id=0):
    """
    __init__(self, config: Wav2Vec2Config, layer_id=0)

    Initializes a new instance of the Wav2Vec2NoLayerNormConvLayer class.

    Args:
        self: The instance of the class.
        config (Wav2Vec2Config): An instance of the Wav2Vec2Config class containing the configuration parameters
            for the Wav2Vec2 model.
        layer_id (int, optional): The index of the layer. Defaults to 0. Specifies the layer for which the
            convolutional layer is initialized.

    Returns:
        None.

    Raises:
        ValueError: If the layer_id is less than 0.
        AttributeError: If the layer_id exceeds the maximum index available in the configuration parameters.
        TypeError: If the provided config parameter is not an instance of the Wav2Vec2Config class.
    """
    super().__init__()
    self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
    self.out_conv_dim = config.conv_dim[layer_id]

    self.conv = nn.Conv1d(
        self.in_conv_dim,
        self.out_conv_dim,
        kernel_size=config.conv_kernel[layer_id],
        stride=config.conv_stride[layer_id],
        has_bias=config.conv_bias,
        pad_mode='valid',
    )
    self.activation = ACT2FN[config.feat_extract_activation]

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2NoLayerNormConvLayer.construct(hidden_states)

Constructs the hidden states using convolutional layer and activation function.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2NoLayerNormConvLayer class. TYPE: Wav2Vec2NoLayerNormConvLayer
hidden_states	The input hidden states tensor. TYPE: Tensor

RETURNS	DESCRIPTION
	torch.Tensor: The constructed hidden states after applying convolution and activation.

RAISES	DESCRIPTION
TypeError	If the input hidden_states is not a torch.Tensor.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    """
    Constructs the hidden states using convolutional layer and activation function.

    Args:
        self (Wav2Vec2NoLayerNormConvLayer): The instance of the Wav2Vec2NoLayerNormConvLayer class.
        hidden_states (torch.Tensor): The input hidden states tensor.

    Returns:
        torch.Tensor: The constructed hidden states after applying convolution and activation.

    Raises:
        TypeError: If the input hidden_states is not a torch.Tensor.
    """
    hidden_states = self.conv(hidden_states)
    hidden_states = self.activation(hidden_states)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2PositionalConvEmbedding

Bases: Cell

This class represents a positional convolutional embedding layer in the Wav2Vec2 model architecture. It inherits from nn.Cell and is designed to process hidden states through convolutional and activation operations.

ATTRIBUTE	DESCRIPTION
config	Wav2Vec2Config An instance of Wav2Vec2Config containing configuration parameters for the layer.

METHOD	DESCRIPTION
__init__	Initializes the Wav2Vec2PositionalConvEmbedding with the provided configuration.
construct	Applies positional convolutional embedding operations on the input hidden_states and returns the transformed output.

Usage

Instantiate this class by providing a Wav2Vec2Config object as configuration, then call the construct method with hidden states to process them.

Note

This class utilizes a convolutional layer, padding layer, and activation function to process hidden states efficiently.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2PositionalConvEmbedding(nn.Cell):

    """
    This class represents a positional convolutional embedding layer in the Wav2Vec2 model architecture.
    It inherits from nn.Cell and is designed to process hidden states through convolutional and activation operations.

    Attributes:
        config: Wav2Vec2Config
            An instance of Wav2Vec2Config containing configuration parameters for the layer.

    Methods:
        __init__:
            Initializes the Wav2Vec2PositionalConvEmbedding with the provided configuration.

        construct:
            Applies positional convolutional embedding operations on the input hidden_states and returns the
            transformed output.

    Usage:
        Instantiate this class by providing a Wav2Vec2Config object as configuration, then call the construct method
        with hidden states to process them.

    Note:
        This class utilizes a convolutional layer, padding layer, and activation function to process hidden states
        efficiently.
    """
    def __init__(self, config: Wav2Vec2Config):
        """
        Initializes a new instance of the Wav2Vec2PositionalConvEmbedding class.

        Args:
            self: An instance of the Wav2Vec2PositionalConvEmbedding class.
            config (Wav2Vec2Config): The configuration object containing various settings for the Wav2Vec2 model.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.conv = nn.Conv1d(
            config.hidden_size,
            config.hidden_size,
            kernel_size=config.num_conv_pos_embeddings,
            padding=config.num_conv_pos_embeddings // 2,
            pad_mode='pad',
            group=config.num_conv_pos_embedding_groups,
            has_bias=True,
        )

        self.conv = weight_norm(self.conv, name='weight', axis=2)
        self.padding = Wav2Vec2SamePadLayer(config.num_conv_pos_embeddings)
        self.activation = ACT2FN[config.feat_extract_activation]

    def construct(self, hidden_states):
        """
        This method constructs the positional convolutional embedding for the Wav2Vec2 model.

        Args:
            self (Wav2Vec2PositionalConvEmbedding): The instance of the Wav2Vec2PositionalConvEmbedding class.
            hidden_states (array-like): The input hidden states with shape (batch_size, sequence_length, hidden_size).

        Returns:
            None: This method does not return any value. The positional convolutional embedding is applied to the
                input hidden states in place.

        Raises:
            ValueError: If the input hidden_states is not in the expected format or shape.
            RuntimeError: If an error occurs during the convolution or activation process.
        """
        hidden_states = hidden_states.swapaxes(1, 2)
        hidden_states = self.conv(hidden_states)
        hidden_states = self.padding(hidden_states)
        hidden_states = self.activation(hidden_states)
        hidden_states = hidden_states.swapaxes(1, 2)
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2PositionalConvEmbedding.__init__(config)

Initializes a new instance of the Wav2Vec2PositionalConvEmbedding class.

PARAMETER	DESCRIPTION
self	An instance of the Wav2Vec2PositionalConvEmbedding class.
config	The configuration object containing various settings for the Wav2Vec2 model. TYPE: Wav2Vec2Config

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, config: Wav2Vec2Config):
    """
    Initializes a new instance of the Wav2Vec2PositionalConvEmbedding class.

    Args:
        self: An instance of the Wav2Vec2PositionalConvEmbedding class.
        config (Wav2Vec2Config): The configuration object containing various settings for the Wav2Vec2 model.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.conv = nn.Conv1d(
        config.hidden_size,
        config.hidden_size,
        kernel_size=config.num_conv_pos_embeddings,
        padding=config.num_conv_pos_embeddings // 2,
        pad_mode='pad',
        group=config.num_conv_pos_embedding_groups,
        has_bias=True,
    )

    self.conv = weight_norm(self.conv, name='weight', axis=2)
    self.padding = Wav2Vec2SamePadLayer(config.num_conv_pos_embeddings)
    self.activation = ACT2FN[config.feat_extract_activation]

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2PositionalConvEmbedding.construct(hidden_states)

This method constructs the positional convolutional embedding for the Wav2Vec2 model.

PARAMETER	DESCRIPTION
self	The instance of the Wav2Vec2PositionalConvEmbedding class. TYPE: Wav2Vec2PositionalConvEmbedding
hidden_states	The input hidden states with shape (batch_size, sequence_length, hidden_size). TYPE: array - like

RETURNS	DESCRIPTION
None	This method does not return any value. The positional convolutional embedding is applied to the input hidden states in place.

RAISES	DESCRIPTION
ValueError	If the input hidden_states is not in the expected format or shape.
RuntimeError	If an error occurs during the convolution or activation process.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    """
    This method constructs the positional convolutional embedding for the Wav2Vec2 model.

    Args:
        self (Wav2Vec2PositionalConvEmbedding): The instance of the Wav2Vec2PositionalConvEmbedding class.
        hidden_states (array-like): The input hidden states with shape (batch_size, sequence_length, hidden_size).

    Returns:
        None: This method does not return any value. The positional convolutional embedding is applied to the
            input hidden states in place.

    Raises:
        ValueError: If the input hidden_states is not in the expected format or shape.
        RuntimeError: If an error occurs during the convolution or activation process.
    """
    hidden_states = hidden_states.swapaxes(1, 2)
    hidden_states = self.conv(hidden_states)
    hidden_states = self.padding(hidden_states)
    hidden_states = self.activation(hidden_states)
    hidden_states = hidden_states.swapaxes(1, 2)
    return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2PreTrainedModel

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2PreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """
    config_class = Wav2Vec2Config
    base_model_prefix = "wav2vec2"
    main_input_name = "input_values"

    def _init_weights(self, cell):
        """Initialize the weights"""
        # Wav2Vec2ForPreTraining last 2 linear layers need standard Linear init.
        if isinstance(cell, Wav2Vec2ForPreTraining):
            cell.project_hid._is_initialized = True
            cell.project_q._is_initialized = True
        # gumbel softmax requires special init
        elif isinstance(cell, Wav2Vec2GumbelVectorQuantizer):
            cell.weight_proj.weight.set_data(initializer(Normal(1.0), cell.weight_proj.weight.shape, cell.weight_proj.weight.dtype))
            cell.weight_proj.bias.set_data(initializer('zeros', cell.weight_proj.bias.shape, cell.weight_proj.bias.dtype))
            cell.codevectors.set_data(initializer('uniform', cell.codevectors.shape, cell.codevectors.dtype))
        elif isinstance(cell, Wav2Vec2PositionalConvEmbedding):
            cell.conv.weight.set_data(
                initializer(Normal(2 * math.sqrt(1 / (cell.conv.kernel_size[0] * cell.conv.in_channels))),
                            cell.conv.weight.shape, cell.conv.weight.dtype))
            cell.conv.bias.set_data(initializer('zeros', cell.conv.bias.shape, cell.conv.bias.dtype))
        elif isinstance(cell, Wav2Vec2FeatureProjection):
            k = math.sqrt(1 / cell.projection.in_channels)
            cell.projection.weight.set_data(
                initializer(Uniform(k), cell.projection.weight.shape, cell.projection.weight.dtype))
            cell.projection.bias.set_data(
                initializer(Uniform(k), cell.projection.bias.shape, cell.projection.bias.dtype))
        elif isinstance(cell, nn.Dense):
            cell.weight.set_data(initializer(Normal(self.config.initializer_range), cell.weight.shape, cell.weight.dtype))
            if cell.bias is not None:
                cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, (nn.LayerNorm, nn.GroupNorm)):
            cell.weight.set_data(initializer('ones', cell.weight.shape, cell.weight.dtype))
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, nn.Conv1d):
            cell.weight.set_data(initializer('he_normal', cell.weight.shape, cell.weight.dtype))
            if cell.bias is not None:
                k = math.sqrt(cell.group / (cell.in_channels * cell.kernel_size[0]))
                cell.bias.set_data(initializer(Uniform(k), cell.bias.shape, cell.bias.dtype))

    def _get_feat_extract_output_lengths(
        self, input_lengths: Union[Tensor, int], add_adapter: Optional[bool] = None
    ):
        """
        Computes the output length of the convolutional layers
        """
        add_adapter = self.config.add_adapter if add_adapter is None else add_adapter

        def _conv_out_length(input_length, kernel_size, stride):
            # 1D convolutional layer output length formula taken
            # from https://pyops.org/docs/stable/generated/ops.nn.Conv1d.html
            return (input_length - kernel_size) // stride + 1

        for kernel_size, stride in zip(self.config.conv_kernel, self.config.conv_stride):
            input_lengths = _conv_out_length(input_lengths, kernel_size, stride)

        if add_adapter:
            for _ in range(self.config.num_adapter_layers):
                input_lengths = _conv_out_length(input_lengths, 1, self.config.adapter_stride)

        return input_lengths

    def _get_feature_vector_attention_mask(
        self, feature_vector_length: int, attention_mask: Tensor, add_adapter=None
    ):
        """
        This method calculates the attention mask for the feature vectors in a Wav2Vec2 model.

        Args:
            self (Wav2Vec2PreTrainedModel): The instance of the Wav2Vec2PreTrainedModel class.
            feature_vector_length (int): The length of the feature vectors.
            attention_mask (Tensor): The attention mask tensor.
            add_adapter (Optional): An optional parameter to add adapter.

        Returns:
            attention_mask (Tensor): The attention mask tensor for the feature vectors.

        Raises:
            None.
        """
        # Effectively attention_mask.sum(-1), but not inplace to be able to run
        # on inference mode.
        non_padded_lengths = attention_mask.cumsum(axis=-1)[:, -1]

        output_lengths = self._get_feat_extract_output_lengths(non_padded_lengths, add_adapter=add_adapter)
        output_lengths = output_lengths.to(mindspore.int64)

        batch_size = attention_mask.shape[0]

        attention_mask = ops.zeros(
            (batch_size, feature_vector_length), dtype=attention_mask.dtype
        )
        # these two operations makes sure that all values before the output lengths idxs are attended to
        attention_mask[(ops.arange(attention_mask.shape[0]), output_lengths - 1)] = 1
        attention_mask = attention_mask.flip([-1]).cumsum(-1).flip([-1]).bool()
        return attention_mask

    def _get_adapters(self):
        """
        Method _get_adapters in the class Wav2Vec2PreTrainedModel.

        Args:
            self (object): The instance of the class Wav2Vec2PreTrainedModel.

        Returns:
            dict: A dictionary containing adapter weights.
                The keys are composed of the parameter names from the adapter layers and the LM head, and the values are
                the corresponding parameters.

        Raises:
            ValueError: If the 'adapter_attn_dim' attribute in 'config' is not defined, a ValueError is raised with
                a message indicating that the class has no adapter layers and prompting to define
                'config.adapter_attn_dim'.
        """
        if self.config.adapter_attn_dim is None:
            raise ValueError(f"{self.__class__} has no adapter layers. Make sure to define `config.adapter_attn_dim`.")

        adapter_weights = {}
        for name, module in self.parameters_and_names():
            if isinstance(module, Wav2Vec2AttnAdapterLayer):
                for param_name, param in module.parameters_and_names():
                    adapter_weights[".".join([name, param_name])] = param

        if isinstance(self, Wav2Vec2ForCTC):
            for name, param in self.lm_head.parameters_and_names():
                adapter_weights[".".join(["lm_head", name])] = param

        return adapter_weights

    def init_adapter_layers(self):
        """
        (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning
        """
        # init attention adapters
        for module in self.cells():
            if isinstance(module, Wav2Vec2AttnAdapterLayer):
                self._init_weights(module)

        # init lm head
        if isinstance(self, Wav2Vec2ForCTC):
            self._init_weights(self.lm_head)

    def load_adapter(self, target_lang: str, force_load=True, **kwargs):
        r"""
        Load a language adapter model from a pre-trained adapter model.

        Parameters:
            target_lang (`str`):
                Has to be a language id of an existing adapter weight. Adapter weights are stored in the format
                adapter.<lang>.safetensors or adapter.<lang>.bin
            force_load (`bool`, defaults to `True`):
                Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`.
            cache_dir (`Union[str, os.PathLike]`, *optional*):
                Path to a directory in which a downloaded pretrained model configuration should be cached if the
                standard cache should not be used.
            force_download (`bool`, *optional*, defaults to `False`):
                Whether or not to force the (re-)download of the model weights and configuration files, overriding the
                cached versions if they exist.
            resume_download (`bool`, *optional*, defaults to `False`):
                Whether or not to delete incompletely received files. Will attempt to resume the download if such a
                file exists.
            proxies (`Dict[str, str]`, *optional*):
                A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
                'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
            local_files_only(`bool`, *optional*, defaults to `False`):
                Whether or not to only look at local files (i.e., do not try to download the model).
            token (`str` or `bool`, *optional*):
                The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use
                the token generated when running `huggingface-cli login` (stored in `~/.huggingface`).
            revision (`str`, *optional*, defaults to `"main"`):
                The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
                git-based system for storing models and other artifacts on hf-mirror.com, so `revision` can be any
                identifier allowed by git.

                <Tip>

                To test a pull request you made on the Hub, you can pass `revision="refs/pr/<pr_number>".

                </Tip>

            mirror (`str`, *optional*):
                Mirror source to accelerate downloads in China. If you are from China and have an accessibility
                problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety.
                Please refer to the mirror site for more information.

        <Tip>

        Activate the special ["offline-mode"](https://hf-mirror.com/transformers/installation.html#offline-mode) to
        use this method in a firewalled environment.

        </Tip>

        Example:
            ```python
            >>> from transformers import Wav2Vec2ForCTC, AutoProcessor
            ...
            >>> ckpt = "facebook/mms-1b-all"
            >>> processor = AutoProcessor.from_pretrained(ckpt)
            >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng")
            >>> # set specific language
            >>> processor.tokenizer.set_target_lang("spa")
            >>> model.load_adapter("spa")
            ```
        """
        if self.config.adapter_attn_dim is None:
            raise ValueError(f"Cannot load_adapter for {target_lang} if `config.adapter_attn_dim` is not defined.")

        if target_lang == self.target_lang and not force_load: # pylint: disable=access-member-before-definition
            logger.warning(f"Adapter weights are already set to {target_lang}.")
            return

        cache_dir = kwargs.pop("cache_dir", None)
        force_download = kwargs.pop("force_download", False)
        resume_download = kwargs.pop("resume_download", False)
        proxies = kwargs.pop("proxies", None)
        local_files_only = kwargs.pop("local_files_only", False)
        token = kwargs.pop("token", None)
        use_auth_token = kwargs.pop("use_auth_token", None)
        use_safetensors = kwargs.pop("use_safetensors", False)

        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if token is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            token = use_auth_token

        model_path_or_id = self.config._name_or_path
        state_dict = None

        # 1. Let's first try loading a safetensors adapter weight
        if use_safetensors is not False:
            filepath = WAV2VEC2_ADAPTER_SAFE_FILE.format(target_lang)

            try:
                weight_path = cached_file(
                    model_path_or_id,
                    filename=filepath,
                    force_download=force_download,
                    resume_download=resume_download,
                    proxies=proxies,
                    local_files_only=local_files_only,
                    cache_dir=cache_dir,
                )

                # state_dict = safe_load_file(weight_path)
                state_dict = None
            except EnvironmentError:
                if use_safetensors:
                    # Raise any environment error raise by `cached_file`. It will have a helpful error message adapted
                    # to the original exception.
                    raise

            except Exception as exc:
                # For any other exception, we throw a generic error.
                if use_safetensors:
                    raise EnvironmentError(
                        f"Can't load the model for '{model_path_or_id}'. If you were trying to load it"
                        " from 'https://hf-mirror.com/models', make sure you don't have a local directory with the"
                        f" same name. Otherwise, make sure '{model_path_or_id}' is the correct path to a"
                        f" directory containing a file named {filepath}."
                    ) from exc

        # 2. If this didn't work let's try loading a PyTorch adapter weight
        if state_dict is None:
            filepath = WAV2VEC2_ADAPTER_PT_FILE.format(target_lang)

            try:
                weight_path = cached_file(
                    model_path_or_id,
                    filename=filepath,
                    force_download=force_download,
                    resume_download=resume_download,
                    proxies=proxies,
                    local_files_only=local_files_only,
                    cache_dir=cache_dir,
                )

                weights_only_kwarg = {"weights_only": True}
                state_dict = ops.load(
                    weight_path,
                    map_location="cpu",
                    **weights_only_kwarg,
                )

            except EnvironmentError:
                # Raise any environment error raise by `cached_file`. It will have a helpful error message adapted
                # to the original exception.
                raise

            except Exception as exc:
                # For any other exception, we throw a generic error.
                raise EnvironmentError(
                    f"Can't load the model for '{model_path_or_id}'. If you were trying to load it"
                    " from 'https://hf-mirror.com/models', make sure you don't have a local directory with the"
                    f" same name. Otherwise, make sure '{model_path_or_id}' is the correct path to a"
                    f" directory containing a file named {filepath}."
                ) from exc

        adapter_weights = self._get_adapters()
        unexpected_keys = set(state_dict.keys()) - set(adapter_weights.keys())
        missing_keys = set(adapter_weights.keys()) - set(state_dict.keys())

        if len(unexpected_keys) > 0:
            raise ValueError(f"The adapter weights {weight_path} has unexpected keys: {', '.join(unexpected_keys)}.")
        elif len(missing_keys) > 0:
            raise ValueError(f"The adapter weights {weight_path} has missing keys: {', '.join(missing_keys)}.")

        # make sure now vocab size is correct
        target_vocab_size = state_dict["lm_head.weight"].shape[0]
        if target_vocab_size != self.config.vocab_size:
            self.lm_head = nn.Dense(
                self.config.output_hidden_size, target_vocab_size, dtype=self.dtype
            )
            self.config.vocab_size = target_vocab_size

        # make sure that adapter weights are put in exactly the same precision and device placement and overwritten adapter weights
        state_dict = {k: v.to(adapter_weights[k]) for k, v in state_dict.items()}
        self.load_state_dict(state_dict, strict=False)

        # set target language corectly
        self.target_lang = target_lang

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2PreTrainedModel.init_adapter_layers()

(Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def init_adapter_layers(self):
    """
    (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning
    """
    # init attention adapters
    for module in self.cells():
        if isinstance(module, Wav2Vec2AttnAdapterLayer):
            self._init_weights(module)

    # init lm head
    if isinstance(self, Wav2Vec2ForCTC):
        self._init_weights(self.lm_head)

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2PreTrainedModel.load_adapter(target_lang, force_load=True, **kwargs)

Load a language adapter model from a pre-trained adapter model.

PARAMETER	DESCRIPTION
target_lang	Has to be a language id of an existing adapter weight. Adapter weights are stored in the format adapter..safetensors or adapter..bin TYPE: `str`
force_load	Whether the weights shall be loaded even if target_lang matches self.target_lang. TYPE: `bool`, defaults to `True` DEFAULT: True
cache_dir	Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. TYPE: `Union[str, os.PathLike]`, *optional*
force_download	Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. TYPE: `bool`, *optional*, defaults to `False`
resume_download	Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists. TYPE: `bool`, *optional*, defaults to `False`
proxies	A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request. TYPE: `Dict[str, str]`, *optional*
local_files_only(`bool`,	Whether or not to only look at local files (i.e., do not try to download the model). TYPE: *optional*, defaults to `False`
token	The token to use as HTTP bearer authorization for remote files. If True, or not specified, will use the token generated when running huggingface-cli login (stored in ~/.huggingface). TYPE: `str` or `bool`, *optional*
revision	The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on hf-mirror.com, so revision can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/". TYPE: `str`, *optional*, defaults to `"main"`
mirror	Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. TYPE: `str`, *optional*

Activate the special "offline-mode" to use this method in a firewalled environment.

Example

>>> from transformers import Wav2Vec2ForCTC, AutoProcessor
...
>>> ckpt = "facebook/mms-1b-all"
>>> processor = AutoProcessor.from_pretrained(ckpt)
>>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng")
>>> # set specific language
>>> processor.tokenizer.set_target_lang("spa")
>>> model.load_adapter("spa")

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def load_adapter(self, target_lang: str, force_load=True, **kwargs):
    r"""
    Load a language adapter model from a pre-trained adapter model.

    Parameters:
        target_lang (`str`):
            Has to be a language id of an existing adapter weight. Adapter weights are stored in the format
            adapter.<lang>.safetensors or adapter.<lang>.bin
        force_load (`bool`, defaults to `True`):
            Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`.
        cache_dir (`Union[str, os.PathLike]`, *optional*):
            Path to a directory in which a downloaded pretrained model configuration should be cached if the
            standard cache should not be used.
        force_download (`bool`, *optional*, defaults to `False`):
            Whether or not to force the (re-)download of the model weights and configuration files, overriding the
            cached versions if they exist.
        resume_download (`bool`, *optional*, defaults to `False`):
            Whether or not to delete incompletely received files. Will attempt to resume the download if such a
            file exists.
        proxies (`Dict[str, str]`, *optional*):
            A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
            'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
        local_files_only(`bool`, *optional*, defaults to `False`):
            Whether or not to only look at local files (i.e., do not try to download the model).
        token (`str` or `bool`, *optional*):
            The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use
            the token generated when running `huggingface-cli login` (stored in `~/.huggingface`).
        revision (`str`, *optional*, defaults to `"main"`):
            The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
            git-based system for storing models and other artifacts on hf-mirror.com, so `revision` can be any
            identifier allowed by git.

            <Tip>

            To test a pull request you made on the Hub, you can pass `revision="refs/pr/<pr_number>".

            </Tip>

        mirror (`str`, *optional*):
            Mirror source to accelerate downloads in China. If you are from China and have an accessibility
            problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety.
            Please refer to the mirror site for more information.

    <Tip>

    Activate the special ["offline-mode"](https://hf-mirror.com/transformers/installation.html#offline-mode) to
    use this method in a firewalled environment.

    </Tip>

    Example:
        ```python
        >>> from transformers import Wav2Vec2ForCTC, AutoProcessor
        ...
        >>> ckpt = "facebook/mms-1b-all"
        >>> processor = AutoProcessor.from_pretrained(ckpt)
        >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng")
        >>> # set specific language
        >>> processor.tokenizer.set_target_lang("spa")
        >>> model.load_adapter("spa")
        ```
    """
    if self.config.adapter_attn_dim is None:
        raise ValueError(f"Cannot load_adapter for {target_lang} if `config.adapter_attn_dim` is not defined.")

    if target_lang == self.target_lang and not force_load: # pylint: disable=access-member-before-definition
        logger.warning(f"Adapter weights are already set to {target_lang}.")
        return

    cache_dir = kwargs.pop("cache_dir", None)
    force_download = kwargs.pop("force_download", False)
    resume_download = kwargs.pop("resume_download", False)
    proxies = kwargs.pop("proxies", None)
    local_files_only = kwargs.pop("local_files_only", False)
    token = kwargs.pop("token", None)
    use_auth_token = kwargs.pop("use_auth_token", None)
    use_safetensors = kwargs.pop("use_safetensors", False)

    if use_auth_token is not None:
        warnings.warn(
            "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
            FutureWarning,
        )
        if token is not None:
            raise ValueError(
                "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
            )
        token = use_auth_token

    model_path_or_id = self.config._name_or_path
    state_dict = None

    # 1. Let's first try loading a safetensors adapter weight
    if use_safetensors is not False:
        filepath = WAV2VEC2_ADAPTER_SAFE_FILE.format(target_lang)

        try:
            weight_path = cached_file(
                model_path_or_id,
                filename=filepath,
                force_download=force_download,
                resume_download=resume_download,
                proxies=proxies,
                local_files_only=local_files_only,
                cache_dir=cache_dir,
            )

            # state_dict = safe_load_file(weight_path)
            state_dict = None
        except EnvironmentError:
            if use_safetensors:
                # Raise any environment error raise by `cached_file`. It will have a helpful error message adapted
                # to the original exception.
                raise

        except Exception as exc:
            # For any other exception, we throw a generic error.
            if use_safetensors:
                raise EnvironmentError(
                    f"Can't load the model for '{model_path_or_id}'. If you were trying to load it"
                    " from 'https://hf-mirror.com/models', make sure you don't have a local directory with the"
                    f" same name. Otherwise, make sure '{model_path_or_id}' is the correct path to a"
                    f" directory containing a file named {filepath}."
                ) from exc

    # 2. If this didn't work let's try loading a PyTorch adapter weight
    if state_dict is None:
        filepath = WAV2VEC2_ADAPTER_PT_FILE.format(target_lang)

        try:
            weight_path = cached_file(
                model_path_or_id,
                filename=filepath,
                force_download=force_download,
                resume_download=resume_download,
                proxies=proxies,
                local_files_only=local_files_only,
                cache_dir=cache_dir,
            )

            weights_only_kwarg = {"weights_only": True}
            state_dict = ops.load(
                weight_path,
                map_location="cpu",
                **weights_only_kwarg,
            )

        except EnvironmentError:
            # Raise any environment error raise by `cached_file`. It will have a helpful error message adapted
            # to the original exception.
            raise

        except Exception as exc:
            # For any other exception, we throw a generic error.
            raise EnvironmentError(
                f"Can't load the model for '{model_path_or_id}'. If you were trying to load it"
                " from 'https://hf-mirror.com/models', make sure you don't have a local directory with the"
                f" same name. Otherwise, make sure '{model_path_or_id}' is the correct path to a"
                f" directory containing a file named {filepath}."
            ) from exc

    adapter_weights = self._get_adapters()
    unexpected_keys = set(state_dict.keys()) - set(adapter_weights.keys())
    missing_keys = set(adapter_weights.keys()) - set(state_dict.keys())

    if len(unexpected_keys) > 0:
        raise ValueError(f"The adapter weights {weight_path} has unexpected keys: {', '.join(unexpected_keys)}.")
    elif len(missing_keys) > 0:
        raise ValueError(f"The adapter weights {weight_path} has missing keys: {', '.join(missing_keys)}.")

    # make sure now vocab size is correct
    target_vocab_size = state_dict["lm_head.weight"].shape[0]
    if target_vocab_size != self.config.vocab_size:
        self.lm_head = nn.Dense(
            self.config.output_hidden_size, target_vocab_size, dtype=self.dtype
        )
        self.config.vocab_size = target_vocab_size

    # make sure that adapter weights are put in exactly the same precision and device placement and overwritten adapter weights
    state_dict = {k: v.to(adapter_weights[k]) for k, v in state_dict.items()}
    self.load_state_dict(state_dict, strict=False)

    # set target language corectly
    self.target_lang = target_lang

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2SamePadLayer

Bases: Cell

This class represents a layer in the Wav2Vec2 model that performs padding removal.

Wav2Vec2SamePadLayer is a subclass of nn.Cell and is designed to remove padding from hidden states in the Wav2Vec2 model. It is primarily used in the Wav2Vec2 model for speech recognition tasks.

ATTRIBUTE	DESCRIPTION
num_pad_remove	The number of padding elements to remove from the hidden states. TYPE: int

METHOD	DESCRIPTION
__init__	Initializes a new instance of the Wav2Vec2SamePadLayer class.
construct	Removes padding elements from the hidden states.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

class Wav2Vec2SamePadLayer(nn.Cell):

    """
    This class represents a layer in the Wav2Vec2 model that performs padding removal.

    Wav2Vec2SamePadLayer is a subclass of nn.Cell and is designed to remove padding from hidden states in the
    Wav2Vec2 model. It is primarily used in the Wav2Vec2 model for speech recognition tasks.

    Attributes:
        num_pad_remove (int): The number of padding elements to remove from the hidden states.

    Methods:
        __init__: Initializes a new instance of the Wav2Vec2SamePadLayer class.
        construct: Removes padding elements from the hidden states.

    """
    def __init__(self, num_conv_pos_embeddings):
        """
        Initializes an instance of the Wav2Vec2SamePadLayer class.

        Args:
            self (Wav2Vec2SamePadLayer): The current instance of the Wav2Vec2SamePadLayer class.
            num_conv_pos_embeddings (int): The number of convolutional positional embeddings.
                It is used to determine the value of the num_pad_remove attribute.
                The value must be a non-negative integer.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.num_pad_remove = 1 if num_conv_pos_embeddings % 2 == 0 else 0

    def construct(self, hidden_states):
        """
        Constructs the hidden states of the Wav2Vec2SamePadLayer.

        Args:
            self (Wav2Vec2SamePadLayer): An instance of the Wav2Vec2SamePadLayer class.
            hidden_states (torch.Tensor): The hidden states to be processed.
                Expected shape is (batch_size, sequence_length, hidden_size).
                The hidden states are processed based on the `num_pad_remove` value.

        Returns:
            None.

        Raises:
            None.
        """
        if self.num_pad_remove > 0:
            hidden_states = hidden_states[:, :, : -self.num_pad_remove]
        return hidden_states

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2SamePadLayer.__init__(num_conv_pos_embeddings)

Initializes an instance of the Wav2Vec2SamePadLayer class.

PARAMETER	DESCRIPTION
self	The current instance of the Wav2Vec2SamePadLayer class. TYPE: Wav2Vec2SamePadLayer
num_conv_pos_embeddings	The number of convolutional positional embeddings. It is used to determine the value of the num_pad_remove attribute. The value must be a non-negative integer. TYPE: int

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def __init__(self, num_conv_pos_embeddings):
    """
    Initializes an instance of the Wav2Vec2SamePadLayer class.

    Args:
        self (Wav2Vec2SamePadLayer): The current instance of the Wav2Vec2SamePadLayer class.
        num_conv_pos_embeddings (int): The number of convolutional positional embeddings.
            It is used to determine the value of the num_pad_remove attribute.
            The value must be a non-negative integer.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.num_pad_remove = 1 if num_conv_pos_embeddings % 2 == 0 else 0

mindnlp.transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2SamePadLayer.construct(hidden_states)

Constructs the hidden states of the Wav2Vec2SamePadLayer.

PARAMETER	DESCRIPTION
self	An instance of the Wav2Vec2SamePadLayer class. TYPE: Wav2Vec2SamePadLayer
hidden_states	The hidden states to be processed. Expected shape is (batch_size, sequence_length, hidden_size). The hidden states are processed based on the num_pad_remove value. TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/wav2vec2/modeling_wav2vec2.py

def construct(self, hidden_states):
    """
    Constructs the hidden states of the Wav2Vec2SamePadLayer.

    Args:
        self (Wav2Vec2SamePadLayer): An instance of the Wav2Vec2SamePadLayer class.
        hidden_states (torch.Tensor): The hidden states to be processed.
            Expected shape is (batch_size, sequence_length, hidden_size).
            The hidden states are processed based on the `num_pad_remove` value.

    Returns:
        None.

    Raises:
        None.
    """
    if self.num_pad_remove > 0:
        hidden_states = hidden_states[:, :, : -self.num_pad_remove]
    return hidden_states