说话人感知的交叉注意力说话人提取网络

李卓璋; 许柏炎; 蔡瑞初; 郝志峰

doi:10.12052/gdutxb.230037

欢迎来到能源电力期刊网！

English Version 期刊登录

您当前的位置：

首页 >

文章列表页 >

说话人感知的交叉注意力说话人提取网络

计算机科学与技术 | 更新时间：2026-05-27

- 说话人感知的交叉注意力说话人提取网络
- Speaker-Aware Cross Attention Speaker Extraction Network
- 广东工业大学学报 2024年41卷第3期页码：91-101
- 作者机构：
  
  1. 广东工业大学计算机学院,广东,广州,510006
  2. 汕头大学理学院,广东,汕头,515063
- 作者简介：
  
  蔡瑞初(1983–)，男，教授，博士，主要研究方向为机器学习、因果关系，E-mail：cairuichu@gmail.com
- 基金信息：
  
  科技创新2030-“新一代人工智能”重大项目(2021ZD0111501);国家优秀青年科学基金资助项目 (62122022);国家自然科学基金资助项目(61876043, 61976052, 62206064)
- DOI：10.12052/gdutxb.230037
  中图分类号： TP391.2
- 纸质出版：2024
- 稿件说明：
移动端阅览
李卓璋, 许柏炎, 蔡瑞初, 郝志峰. 说话人感知的交叉注意力说话人提取网络[J]. 广东工业大学学报, 2024, 41(3): 91-101.

Li Zhuo-zhang, Xu Bo-yan, Cai Rui-chu, et al. Speaker-Aware Cross Attention Speaker Extraction Network[J]. 2024, 41(3): 91-101.
李卓璋, 许柏炎, 蔡瑞初, 郝志峰. 说话人感知的交叉注意力说话人提取网络[J]. 广东工业大学学报, 2024, 41(3): 91-101. DOI： 10.12052/gdutxb.230037.

Li Zhuo-zhang, Xu Bo-yan, Cai Rui-chu, et al. Speaker-Aware Cross Attention Speaker Extraction Network[J]. 2024, 41(3): 91-101. DOI： 10.12052/gdutxb.230037.

摘要

目标说话人提取任务的目标是在一段混合音频中提取特定说话人的语音，任务设置上一般会给一段目标说话人注册音频作为辅助信息。现有的研究工作主要有以下不足：(1) 说话人识别的辅助网络无法捕获学习注册音频中的关键信息；(2) 缺乏混合音频嵌入和注册音频嵌入的交互学习机制。以上不足导致了现有研究工作在注册音频和目标音频之间存在较大差异时有说话人混淆问题。为了解决该问题，提出说话人感知的交叉注意力说话人提取网络(Speaker-aware Cross Attention Speaker Extraction Network

SACAN) 。SACAN在说话人识别辅助网络引入基于注意力的说话人聚合模块，有效聚合目标说话人声音特性的关键信息和利用混合音频增强目标说话人嵌入。进一步地，SACAN通过交叉注意力构建交互学习机制促进说话人嵌入与混合音频嵌入融合学习，增强了模型的说话人感知能力。实验结果表明，SACAN相比基准方法在STOI和SI-SDRi分别提高了

0.0133

、

1.0695

dB，并在说话人混淆相关评估和消融实验中验证了不同模块的有效性。

Abstract

Target speaker extraction aims to extract the speech of the specific speaker from mixed audio

which usually treats the enrolled audio of the target speaker as auxiliary information. Existing approaches mainly have the following limitations: the auxiliary network for speaker recognition cannot capture the critical information from enrolled audio

and the second one is the lack of an interactive learning mechanism between mixed and enrolled audio embedding. These

limitations lead to speaker confusion when the difference between the enrolled and target audio is significant. To address this

a speaker-aware cross-attention speaker extraction network (SACAN) is proposed. First

SACAN introduces an attention-based speaker aggregation module in the speaker recognition auxiliary network

which effectively aggregates critical information about target speaker characteristics. Then

it uses mixed audio to enhance target speaker embedding. After that

to promote the integration of speaker embedding and mixed audio embedding

SACAN builds an interactive learning mechanism through cross-attention and enhances the speaker perception ability of the model. The experimental results show that SACAN improves by

0.0133

and

1.0695

in terms of STOI and SI-SDRi when compared with the benchmark model

validating the effectiveness of the proposed module in speaker confusion assessment and ablation experiments.

关键词

Keywords

references

CHERRY E C. Some experiments on the recognition of speech, with one and with two ears [J]. The Journal of the Acoustical Society of America, 1953, 25(5): 975-979.

HERSHEY J R, CHEN Z, LE ROUX J, et al. Deep clustering: discriminative embeddings for segmentation and separation[C]//2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Shanghai: IEEE, 2016: 31-35.

YU D, KOLBK M, TAN Z H, et al. Permutation invariant training of deep models for speaker-independent multi-talker speech separation[C]//2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New Orleans: IEEE, 2017: 241-245.

CHEN Z, LUO Y, MESGARANI N. Deep attractor network for single-microphone speaker separation[C]//2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . New Orleans: IEEE, 2017: 246-250.

LUO Y, MESGARANI N. Conv-tasnet: surpassing ideal time–frequency magnitude masking for speech separation [J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019, 27(8): 1256-1266.

LUO Y, CHEN Z, YOSHIOKA T. Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2020: 46-50.

CHEN J, MAO Q, LIU D. Dual-path transformer network: direct context-aware modeling for end-to-end monaural speech separation[C]//Conference of the International Speech Communication Association. Online: ISCA, 2020: 2642-2646.

ZEGHIDOUR N, GRANGIER D. Wavesplit: end-to-end speech separation by speaker clustering [J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 2840-2849.

LAM M W Y, WANG J, SU D, et al. Sandglasset: a light multi-granularity self-attentive network for time-domain speech separation[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2021: 5759-5763.

SUBAKAN C, RAVANELLI M, CORNELL S, et al. Attention is all you need in speech separation[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2021: 21-25.

DELCROIX M, ZMOLIKOVA K, KINOSHITA K, et al. Single channel target speaker extraction and recognition with speaker beam[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Calgary: IEEE, 2018: 5554-5558.

WAN L, WANG Q, PAPIR A, et al. Generalized end-to-end loss for speaker verification[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Calgary: IEEE, 2018: 4879-4883.

WANG Q, MUCKENHIRN H, WILSON K, et al. VoiceFilter: targeted voice separation by speaker-conditioned spectrogram masking [C]//Conference of the International Speech Communication Association. Graz: ISCA, 2019: 2728-2732.

XU C, RAO W, CHNG E S, et al . Spex: multi-scale time domain speaker extraction network [J ] . IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020, 28: 1370-1384.

GE M, XU C, WANG L, et al. SpEx+: a complete time domain speaker extraction network[C]//Conference of the International Speech Communication Association. Online: ISCA, 2020: 1406-1410.

GE M, XU C, WANG L, et al. Multi-stage speaker extraction with utterance and frame-level reference signals[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Online: IEEE, 2021: 6109-6113.

DENG C, MA S, SHA Y, et al. Robust speaker extraction network based on iterative refined adaptation. [C]// Conference of the International Speech Communication Association. Online: 2021, 3530-3534.

HAO Y, XU J, ZHANG P, et al. Wase: learning when to attend for speaker extraction in cocktail party environments[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2021: 6104-6108.

JI X, YU M, ZHANG C, et al. Speaker-aware target speaker enhancement by jointly learning with speaker embedding extraction[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Online: IEEE, 2020: 7294-7298.

JU Y, RAO W, YAN X, et al. TEA-PSE: tencent-ethereal-audio-lab personalized speech enhancement system for ICASSP 2022 DNS CHALLENGE[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Singapore: IEEE, 2022: 9291-9295.

ZHAO Z, GU R, YANG D, et al. Speaker-aware mixture of mixtures training for weakly supervised speaker extraction [C]//Conference of the International Speech Communication Association. Incheon: ISCA, 2022, 5318-5322.

PANDEY A, WANG D L. Attentive training: a new training framework for talker-independent speaker extraction. [C]//Conference of the International Speech Communication Association. Incheon: ISCA, 2022, 201-205

WANG W, XU C, GE M, et al. Neural speaker extraction with speaker-speech cross-attention network[C]// Conference of the International Speech Communication Association. Online: 2021: 3535-3539.

VASWANI A, SHAZEER N, PARMAR N, et al . Attention is all you need [J ] . Advances in Neural Information Processing Systems, 2017, 30: 5998-6008.

ZHAO Z, YANG D, GU R, et al. Target confusion in end-to-end speaker extraction: analysis and approaches. [C]//Conference of the International Speech Communication Association. Incheon: ISCA, 2022: 5333-5337.

OKABE K, KOSHINAKA T, SHINODA K. Attentive statistics pooling for deep speaker embedding[C]//Conference of the International Speech Communication Association. Hyderabad: ISCA, 2018: 2252-2256.

CHEN L C, PAPANDREOU G, KOKKINOS I, et al . DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848.

LE ROUX J, WISDOM S, ERDOGAN H, et al. SDR–half-baked or well done?[C]//ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Brighton: IEEE, 2019: 626-630.

PANAYOTOV V, CHEN G, POVEY D, et al. Librispeech: an ASR corpus based on public domain audio books[C]//2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Brisbane: IEEE, 2015: 5206-5210.

TAAL C H, HENDRIKS R C, HEUSDENS R, et al. A short-time objective intelligibility measure for time-frequency weighted noisy speech[C]//2010 IEEE International Conference on Acoustics, Speech and Signal Processing. Dallas: IEEE, 2010: 4214-4217.

DELCROIX M, OCHIAI T, ZMOLIKOVA K, et al. Improving speaker discrimination of target speech extraction with time-domain speakerbeam[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Online: IEEE, 2020: 691-695.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于门控注意力单元的中文医学命名实体识别

基于分离结果信噪比估计与自适应调频网络的单通道语音分离技术

基于抗噪声的多任务多示例学习算法研究

面向灾后救援的多无人机通感算一体化资源分配优化方法

多尺度对比嵌入增强的自适应入侵检测模型