Speaker-Aware Cross Attention Speaker Extraction Network

Li Zhuo-zhang; Xu Bo-yan; Cai Rui-chu; Hao Zhi-feng

doi:10.12052/gdutxb.230037

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

Speaker-Aware Cross Attention Speaker Extraction Network

更新时间：2026-05-27

- Speaker-Aware Cross Attention Speaker Extraction Network
- Vol. 41, Issue 3, Pages: 91-101(2024)
- 作者机构：
  
  1. 广东工业大学计算机学院,广东,广州,510006
  2. 汕头大学理学院,广东,汕头,515063
- 作者简介：
- 基金信息：
- DOI：10.12052/gdutxb.230037
  CLC： TP391.2
- Published：2024
- 稿件说明：
移动端阅览
李卓璋, 许柏炎, 蔡瑞初, 郝志峰. 说话人感知的交叉注意力说话人提取网络[J]. 广东工业大学学报, 2024, 41(3): 91-101.

Li Zhuo-zhang, Xu Bo-yan, Cai Rui-chu, et al. Speaker-Aware Cross Attention Speaker Extraction Network[J]. 2024, 41(3): 91-101.
李卓璋, 许柏炎, 蔡瑞初, 郝志峰. 说话人感知的交叉注意力说话人提取网络[J]. 广东工业大学学报, 2024, 41(3): 91-101. DOI： 10.12052/gdutxb.230037.

Li Zhuo-zhang, Xu Bo-yan, Cai Rui-chu, et al. Speaker-Aware Cross Attention Speaker Extraction Network[J]. 2024, 41(3): 91-101. DOI： 10.12052/gdutxb.230037.

摘要

目标说话人提取任务的目标是在一段混合音频中提取特定说话人的语音，任务设置上一般会给一段目标说话人注册音频作为辅助信息。现有的研究工作主要有以下不足：(1) 说话人识别的辅助网络无法捕获学习注册音频中的关键信息；(2) 缺乏混合音频嵌入和注册音频嵌入的交互学习机制。以上不足导致了现有研究工作在注册音频和目标音频之间存在较大差异时有说话人混淆问题。为了解决该问题，提出说话人感知的交叉注意力说话人提取网络(Speaker-aware Cross Attention Speaker Extraction Network

SACAN) 。SACAN在说话人识别辅助网络引入基于注意力的说话人聚合模块，有效聚合目标说话人声音特性的关键信息和利用混合音频增强目标说话人嵌入。进一步地，SACAN通过交叉注意力构建交互学习机制促进说话人嵌入与混合音频嵌入融合学习，增强了模型的说话人感知能力。实验结果表明，SACAN相比基准方法在STOI和SI-SDRi分别提高了

0.0133

、

1.0695

dB，并在说话人混淆相关评估和消融实验中验证了不同模块的有效性。

Abstract

Target speaker extraction aims to extract the speech of the specific speaker from mixed audio

which usually treats the enrolled audio of the target speaker as auxiliary information. Existing approaches mainly have the following limitations: the auxiliary network for speaker recognition cannot capture the critical information from enrolled audio

and the second one is the lack of an interactive learning mechanism between mixed and enrolled audio embedding. These

limitations lead to speaker confusion when the difference between the enrolled and target audio is significant. To address this

a speaker-aware cross-attention speaker extraction network (SACAN) is proposed. First

SACAN introduces an attention-based speaker aggregation module in the speaker recognition auxiliary network

which effectively aggregates critical information about target speaker characteristics. Then

it uses mixed audio to enhance target speaker embedding. After that

to promote the integration of speaker embedding and mixed audio embedding

SACAN builds an interactive learning mechanism through cross-attention and enhances the speaker perception ability of the model. The experimental results show that SACAN improves by

0.0133

and

1.0695

in terms of STOI and SI-SDRi when compared with the benchmark model

validating the effectiveness of the proposed module in speaker confusion assessment and ablation experiments.

关键词

Keywords

references

CHERRY E C. Some experiments on the recognition of speech, with one and with two ears [J]. The Journal of the Acoustical Society of America, 1953, 25(5): 975-979.

HERSHEY J R, CHEN Z, LE ROUX J, et al. Deep clustering: discriminative embeddings for segmentation and separation[C]//2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Shanghai: IEEE, 2016: 31-35.

YU D, KOLBK M, TAN Z H, et al. Permutation invariant training of deep models for speaker-independent multi-talker speech separation[C]//2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New Orleans: IEEE, 2017: 241-245.

CHEN Z, LUO Y, MESGARANI N. Deep attractor network for single-microphone speaker separation[C]//2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . New Orleans: IEEE, 2017: 246-250.

LUO Y, MESGARANI N. Conv-tasnet: surpassing ideal time–frequency magnitude masking for speech separation [J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019, 27(8): 1256-1266.

LUO Y, CHEN Z, YOSHIOKA T. Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2020: 46-50.

CHEN J, MAO Q, LIU D. Dual-path transformer network: direct context-aware modeling for end-to-end monaural speech separation[C]//Conference of the International Speech Communication Association. Online: ISCA, 2020: 2642-2646.

ZEGHIDOUR N, GRANGIER D. Wavesplit: end-to-end speech separation by speaker clustering [J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 2840-2849.

LAM M W Y, WANG J, SU D, et al. Sandglasset: a light multi-granularity self-attentive network for time-domain speech separation[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2021: 5759-5763.

SUBAKAN C, RAVANELLI M, CORNELL S, et al. Attention is all you need in speech separation[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2021: 21-25.

DELCROIX M, ZMOLIKOVA K, KINOSHITA K, et al. Single channel target speaker extraction and recognition with speaker beam[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Calgary: IEEE, 2018: 5554-5558.

WAN L, WANG Q, PAPIR A, et al. Generalized end-to-end loss for speaker verification[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Calgary: IEEE, 2018: 4879-4883.

WANG Q, MUCKENHIRN H, WILSON K, et al. VoiceFilter: targeted voice separation by speaker-conditioned spectrogram masking [C]//Conference of the International Speech Communication Association. Graz: ISCA, 2019: 2728-2732.

XU C, RAO W, CHNG E S, et al . Spex: multi-scale time domain speaker extraction network [J ] . IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020, 28: 1370-1384.

GE M, XU C, WANG L, et al. SpEx+: a complete time domain speaker extraction network[C]//Conference of the International Speech Communication Association. Online: ISCA, 2020: 1406-1410.

GE M, XU C, WANG L, et al. Multi-stage speaker extraction with utterance and frame-level reference signals[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Online: IEEE, 2021: 6109-6113.

DENG C, MA S, SHA Y, et al. Robust speaker extraction network based on iterative refined adaptation. [C]// Conference of the International Speech Communication Association. Online: 2021, 3530-3534.

HAO Y, XU J, ZHANG P, et al. Wase: learning when to attend for speaker extraction in cocktail party environments[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2021: 6104-6108.

JI X, YU M, ZHANG C, et al. Speaker-aware target speaker enhancement by jointly learning with speaker embedding extraction[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Online: IEEE, 2020: 7294-7298.

JU Y, RAO W, YAN X, et al. TEA-PSE: tencent-ethereal-audio-lab personalized speech enhancement system for ICASSP 2022 DNS CHALLENGE[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Singapore: IEEE, 2022: 9291-9295.

ZHAO Z, GU R, YANG D, et al. Speaker-aware mixture of mixtures training for weakly supervised speaker extraction [C]//Conference of the International Speech Communication Association. Incheon: ISCA, 2022, 5318-5322.

PANDEY A, WANG D L. Attentive training: a new training framework for talker-independent speaker extraction. [C]//Conference of the International Speech Communication Association. Incheon: ISCA, 2022, 201-205

WANG W, XU C, GE M, et al. Neural speaker extraction with speaker-speech cross-attention network[C]// Conference of the International Speech Communication Association. Online: 2021: 3535-3539.

VASWANI A, SHAZEER N, PARMAR N, et al . Attention is all you need [J ] . Advances in Neural Information Processing Systems, 2017, 30: 5998-6008.

ZHAO Z, YANG D, GU R, et al. Target confusion in end-to-end speaker extraction: analysis and approaches. [C]//Conference of the International Speech Communication Association. Incheon: ISCA, 2022: 5333-5337.

OKABE K, KOSHINAKA T, SHINODA K. Attentive statistics pooling for deep speaker embedding[C]//Conference of the International Speech Communication Association. Hyderabad: ISCA, 2018: 2252-2256.

CHEN L C, PAPANDREOU G, KOKKINOS I, et al . DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848.

LE ROUX J, WISDOM S, ERDOGAN H, et al. SDR–half-baked or well done?[C]//ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Brighton: IEEE, 2019: 626-630.

PANAYOTOV V, CHEN G, POVEY D, et al. Librispeech: an ASR corpus based on public domain audio books[C]//2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Brisbane: IEEE, 2015: 5206-5210.

TAAL C H, HENDRIKS R C, HEUSDENS R, et al. A short-time objective intelligibility measure for time-frequency weighted noisy speech[C]//2010 IEEE International Conference on Acoustics, Speech and Signal Processing. Dallas: IEEE, 2010: 4214-4217.

DELCROIX M, OCHIAI T, ZMOLIKOVA K, et al. Improving speaker discrimination of target speech extraction with time-domain speakerbeam[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2020: 691-695.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Chinese Medical Named Entity Recognition Based on Gated Attention Unit

Single-channel Speech Separation Based on Separated SI-SNR Regression Estimation and Adaptive Frequency Modulation Network

An Algorithm Based on Multi-task Multi-instance Anti-noise Learning

An Optimal Operation Strategy for Community Microgrid Alliances Considering Power Exchange under Multi-package Shared Energy Storage Model

Robust Design for Artificial Noise-assisted UAV Terahertz Secure Communication

Related Author

Wu Xiao-ling

Chen Xiang-wang

Zhan Wen-tao

Ling Jie

Zhang Rui

Lyu Jun

Li Qi-xiang

Xiao Yan-shan

Related Institution

School of Automation, Guangdong University of Technology

School of Computers, Guangdong University of Technology

School of Mathematics and Big Data, Foshan University

AI问答

⁰