Application of reinforcement learning method in flapping airfoil experiment

ZHANG Jin; ZHOU Lei; CAO Bochao

doi:10.7638/kqdlxxb-2023.0071

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

Application of reinforcement learning method in flapping airfoil experiment

更新时间：2026-05-28

- Application of reinforcement learning method in flapping airfoil experiment
- Vol. 41, Issue 9, Pages: 20-29(2023)
- 作者机构：
  
  复旦大学航天航空系,上海,200433
- 作者简介：
- 基金信息：
- DOI：10.7638/kqdlxxb-2023.0071
  CLC： O352;TP181
- Published：2023
- 稿件说明：
移动端阅览
张进, 周雷, 曹博超. 强化学习方法在翼型拍动实验中的应用[J]. 空气动力学学报, 2023, 41(9): 20−29.

ZHANG Jin, ZHOU Lei, CAO Bochao. Application of reinforcement learning method in flapping airfoil experiment[J]. 2023, 41(9): 20-29.
张进, 周雷, 曹博超. 强化学习方法在翼型拍动实验中的应用[J]. 空气动力学学报, 2023, 41(9): 20−29. DOI： 10.7638/kqdlxxb-2023.0071

ZHANG Jin, ZHOU Lei, CAO Bochao. Application of reinforcement learning method in flapping airfoil experiment[J]. 2023, 41(9): 20-29. DOI： 10.7638/kqdlxxb-2023.0071.

摘要

将深度强化学习方法应用于水洞实验，实现了实验室内的自动闭环优化框架，并用该框架优化了雷诺数

= 1.3×10

下纯俯仰运动的NACA0012翼型模型的推进效率。现有的相关研究往往将运动模式限制为某种周期性函数，具有局限性。借助于强化学习方法，实现了在更广的非周期动作空间中的动作搜索。在实验中，模型自动地与水洞环境进行交互，最终学习到了高效推进的非周期运动策略。另外，通过修改奖励函数，实现了在给定推力阈值以上的效率优化。研究结果显示，强化学习模型可以在实验过程中通过不断调整拍动动作的幅度和频率来实现推进效率的持续提升，并且最终通过强化学习方法获得的最优拍动动作均与正弦拍动动作接近，得到的最优推进效率基本位于同等幅度正弦动作效率的上边界。研究展示了强化学习方法用于复杂流动控制问题的可行性。

Abstract

This study presents the application of the deep reinforcement learning (DRL) method in water tunnel experiments

which establishes an automatic closed-loop optimization framework in the laboratory. The framework is used to optimize the propulsion efficiency of a NACA0012 airfoil model under pure pitching motion at a Reynolds number of

= 1.3×10

. Existing related studies often limit the motion patterns to periodic functions

thanks to the DRL method

the optimization process can take place in a broader non-periodic action space. In the experiment

the airfoil automatically interacts with the water tunnel environment

and ultimatel

y learns highly efficient non-periodic motions. By modifying the reward function

the efficiency optimization can be achieved above a given thrust threshold. The present findings demonstrate the DRL model can continuously enhance the propulsion efficiency by adjusting the amplitude and frequency of the flapping motion. Moreover

the optimal flapping motion obtained by DRL is close to the sinusoidal motion

and the achieved optimal propulsion efficiency lies on the upper bound of that of the sinusoidal motion with a similar amplitude. The present study demonstrates the feasibility of using the DRL method for complex flow control problems.

关键词

Keywords

references

BRUNTON S L, NOACK B R. Closed-loop turbulence control: progress and challenges[J]. Applied Mechanics Reviews, 2015, 67(5): 050801. DOI: 10.1115/1.4031175

SUTTON R S, BARTO A G. Reinforcement learning: an introduction[M]. Cambridge: MIT Press, 1998.

GARNIER P, VIQUERAT J, RABAULT J, et al. A review on deep reinforcement learning for fluid mechanics[J]. Computers Fluids, 2021, 225: 104973. DOI: 10.1016/j.compfluid.2021.104973

BRUNTON S L, NOACK B R, KOUMOUTSAKOS P. Machine learning for fluid mechanics[J]. Annual Review of Fluid Mechanics, 2020, 52: 477-508. DOI: 10.1146/annurev-fluid-010719-060214

REN F, GAO C Q, TANG H. Machine learning for flow control: Applications and development trends[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(4): 524686. (in Chinese)

REDDY G, CELANI A, SEJNOWSKI T J, et al. Learning to soar in turbulent environments[J]. Proceedings of the National Academy of Sciences of the United States of America, 2016, 113(33): E4877-E4884. DOI: 10.1073/pnas.1606075113

REDDY G, WONG-NG J, CELANI A, et al. Glider soaring via reinforcement learning in the field[J]. Nature, 2018, 562(7726): 236-239. DOI: 10.1038/s41586-018-0533-0

RABAULT J, KUCHTA M, JENSEN A, et al. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control[J]. Journal of Fluid Mechanics, 2019, 865: 281-302. DOI: 10.1017/jfm.2019.62

FAN D X, YANG L, WANG Z C, et al. Reinforcement learning for bluff body active flow control in experiments and simulations[J]. Proceedings of the National Academy of Sciences of the United States of America, 2020, 117(42): 26091-26098. DOI: 10.1073/pnas.2004939117

LI J C, ZHANG M Q. Reinforcement-learning-based control of confined cylinder wakes with stability analyses[J]. Journal of Fluid Mechanics, 2022, 932: A44. DOI: 10.1017/jfm.2021.1045

YAO Z Y, SHI Z W, DONG Y Z. Deep reinforcement learning for the control of airfoil flow separation[J]. Journal of Experiments in Fluid Mechanics, 2022, 36(3): 55-64. (in Chinese)

GAZZOLA M, HEJAZIALHOSSEINI B, KOUMOUTSAKOS P. Reinforcement learning and wavelet adapted vortex methods for simulations of self-propelled swimmers[J]. SIAM Journal on Scientific Computing, 2014, 36(3): B622-B639. DOI: 10.1137/130943078

ZHU Y, TIAN F B, YOUNG J, et al. A numerical study of fish adaption behaviors in complex environments with a deep reinforcement learning and immersed boundary-lattice Boltzmann method[J]. Scientific Reports, 2021, 11: 1691. DOI: 10.1038/s41598-021-81124-8

LIAO J C. The role of the lateral line and vision on body kinematics and hydrodynamic preference of rainbow trout in turbulent flow[J]. Journal of Experimental Biology, 2006, 209(20): 4077-4090. DOI: 10.1242/jeb.02487

GAZZOLA M, TCHIEU A A, ALEXEEV D, et al. Learning to school in the presence of hydrodynamic interactions[J]. Journal of Fluid Mechanics, 2016, 789: 726-749. DOI: 10.1017/jfm.2015.686

NOVATI G, VERMA S, ALEXEEV D, et al. Synchronisation through learning for two self-propelled swimmers[J]. Bioinspiration Biomimetics, 2017, 12(3): 036001. DOI: 10.1088/1748-3190/aa6311

VERMA S, NOVATI G, KOUMOUTSAKOS P. Efficient collective swimming by harnessing vortices through deep reinforcement learning[J]. Proceedings of the National Academy of Sciences of the United States of America, 2018, 115(23): 5849-5854. DOI: 10.1073/pnas.1800923115

TRIANTAFYLLOU M S, TRIANTAFYLLOU G S, YUE D K P. Hydrodynamics of fishlike swimming[J]. Annual Review of Fluid Mechanics, 2000, 32: 33-53. DOI: 10.1146/annurev.fluid.32.1.33

WU T Y. Fish swimming and bird/insect flight[J]. Annual Review of Fluid Mechanics, 2011, 43: 25-58. DOI: 10.1146/annurev-fluid-122109-160648

SENTURK U, SMITS A J. Numerical simulations of the flow around a square pitching panel[J]. Journal of Fluids and Structures, 2018, 76: 454-468. DOI: 10.1016/j.jfluidstructs.2017.11.001

SENTURK U, SMITS A J. Reynolds number scaling of the propulsive performance of a pitching airfoil[J]. AIAA Journal, 2019, 57(7): 2663-2669. DOI: 10.2514/1.j058371

FLORYAN D, VAN BUREN T, ROWLEY C W, et al. Scaling the propulsive performance of heaving and pitching foils[J]. Journal of Fluid Mechanics, 2017, 822: 386-397. DOI: 10.1017/jfm.2017.302

LAGOPOULOS N S, WEYMOUTH G D, GANAPATHISUBRAMANI B. Universal scaling law for drag-to-thrust wake transition in flapping foils[J]. Journal of Fluid Mechanics, 2019, 872: R1. DOI: 10.1017/jfm.2019.361

LAUDER G V. Fish locomotion: recent advances and new directions[J]. Annual Review of Marine Science, 2015, 7: 521-545. DOI: 10.1146/annurev-marine-010814-015614

FLORYAN D, VAN BUREN T, SMITS A J. Forces and energetics of intermittent swimming[J]. Acta Mechanica Sinica, 2017, 33(4): 725-732. DOI: 10.1007/s10409-017-0694-3

AKOZ E, MOORED K W. Unsteady propulsion by an intermittent swimming gait[J]. Journal of Fluid Mechanics, 2018, 834: 149-172. DOI: 10.1017/jfm.2017.731

CHAO L M, PAN G, ZHANG D, et al. Numerical investigations on the force generation and wake structures of a nonsinusoidal pitching foil[J]. Journal of Fluids and Structures, 2019, 85: 27-39. DOI: 10.1016/j.jfluidstructs.2018.12.002

VAN BUREN T, FLORYAN D, QUINN D, et al. Nonsinusoidal gaits for unsteady propulsion[J]. Physical review fluids, 2017, 2(5): 053101. doi: 10.1103/PhysRevFluids.2.053101

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Rapid design method of rocket aerodynamic shape based on conditional diffusion model

Research progress and considerations on short-term prediction of extreme wind fields based on machine learning

Advances in machine learning for wind-induced fluid-structure interaction of large-scale structures

An inverse design method for three-dimensional morphing wings based on deep reinforcement learning

Machine learning-based study on left atrial appendage hemodynamics in atrial fibrillation patients

Related Author

Lin Jiazhe

Qian Weiqi

Cheng Ming

He Lei

Chen Jiandong

Li Chuan

FAN Yansheng

WANG Hao

Related Institution

State Key Laboratory of Aerodynamics, Mianyang

Computational Aerodynamics Institute of China Aerodynamics Research and Development Center, Mianyang

College of Computer Science, Sichuan University, Chengdu

School of Civil Engineering, Southeast University, Nanjing

Key Laboratory of Concrete and Pre-stressed Concrete Structures of Ministry of Education, Southeast University, Nanjing

AI问答

⁰