PROGRESSES IN SWIMMING STRATEGY OF SMART PARTICLES IN COMPLEX FLOWS
-
摘要: 智能颗粒定义为可以根据环境的变化而主动调整自身在流场中运动的颗粒, 一般用于描述微小的游动物体, 如微生物、浮游生物和微纳机器人. 由于颗粒运动学特性和流场环境的复杂性, 探索智能颗粒的运动策略是一项具有现实意义与挑战性的研究. 近期强化学习算法被应用于智能颗粒的运动策略研究, 并取得一定进展. 本文将讨论强化学习在智能颗粒研究中的应用, 并介绍浮游生物运动策略的相关研究进展, 包括适用于海洋浮游生物的游动颗粒模型, 以及强化学习的基本原理和此类问题的研究框架. 海洋浮游生物的昼夜垂直迁移对其生存和繁衍至关重要, 生物学研究表明某些浮游生物可感知周围的流场信息, 但能否利用这些信息提高垂直方向游动效率仍是一个未知的问题. 基于这一背景, 相关研究考虑了重力沉降和颗粒形状对浮游生物垂直游动策略的影响. 研究发现细长的颗粒能够更加高效地垂直向上运动, 而重力沉降则导致智能游动策略发生较大变化. 在此基础上, 后续研究进一步考虑了局部流体信号的作用, 并讨论了基于局部信息在全局坐标系中定向运动的可能性. 当颗粒只能感受到局部流体信号时, 必须破坏动力学系统的对称性, 否则颗粒无法学到有效的垂向游动策略. 此外, 研究还发现强化学习能够利用流体信号背后的机制, 得到颗粒在二维定常流动和三维非定常湍流中高效垂直运动的策略. 这些智能游动策略依赖于本质的物理规律, 因此这些策略在更加复杂或真实的流动中也可能有优秀的表现.Abstract: Smart particles in the present study refer to the particles in fluid that can actively adjust their motions based on the changing environment, and they are usually used to describe micro-swimmers such as microorganisms, plankton, or micro-robots. Due to the complex dynamics of particles and the flow environment, exploring the swimming strategy of smart particles is challenging but of great practical significance. Recently, reinforcement learning has been adopted for exploring the swimming strategies of smart particles, and certain progress was made. Here, we discuss the application of reinforcement learning in the study of smart particles, and introduce the recent progresses in the swimming strategy of plankton, including the swimming particle model for marine plankton, and the framework of reinforcement learning. The vertical migration is vital to the survival and reproduction of plankton. Biological study suggested that some plankton can perceive information from local fluid environment, but whether this information can be used for accelerating vertical migration still remains unknown. In this context, researchers investigated the influence of gravitational settling and particle shape on the vertical swimming strategy of plankton. Swimmers with slender shape can navigate upward more efficiently, and gravitational settling results in significant changes in smart swimming strategies. Furthermore, successive studies were carried out to investigate the effect of local fluid signals, and to discuss the possibility of navigation in the global frame of reference with only local signals. When swimmers only access to local signals, they cannot learn any effective upward swimming strategy unless the rotational symmetry of the dynamics is broken. Moreover, it was also found that reinforcement learning can make use of the underlying physical mechanism of local signals, and obtain efficient swimming strategies for vertical migration in two-dimensional time-independent flows and three-dimensional turbulent flow. Because the mechanism behind these strategies is essential and robust, these strategies is expected to be effective in more complex and realistic flow environments.
-
Key words:
- smart particles /
- reinforcement learning /
- plankton /
- vertical migration
-
图 2 智能颗粒垂直运动速度vz随形状和重力沉降作用的变化[22]. 颗粒速度以流动的特征速度u0无量纲化. RL: 强化学习得到的智能策略, naive: 简单策略, settle: 考虑沉降作用
Figure 2. Vertical velocity of smart particles vz as functions of the effects of gravitational settling and particle shape [22]. Particle velocity is normalized by velocity scale of the background flow u0. RL: smart strategy found by reinforcement learning. Naive: naive strategy. Settle: the cases considering settling effect
图 3 (a)智能颗粒在二维Taylor-Green旋涡流动中的运动轨迹. 红点表示颗粒的初始位置, 背景为涡量云图. (b)智能颗粒与非智能颗粒当地流体垂直方向速度uy的概率密度分布(PDF)[23]
Figure 3. (a) Trajectories of smart particles in two-dimensional Taylor-Green vortex flow. Red dots represent the initial position of particles. Background contour shows the vorticity of fluid. (b) The probability distribution function (PDF) of the vertical velocity of local fluid uy[23]
4 (a)颗粒在冻结流场(frozen)与随时间变化的流场(DNS)中的垂直方向速度vz, 以及游动速度nzvs, 当地流体速度uz和沉降速度vg,z的贡献. smart 2D:
$ \mathrm{\Delta }{u}_{p} $ 流体信号下的智能策略; naive: 简单策略智能颗粒. (b)不同策略下颗粒在三维各向同性均匀湍流中的运动速度及各部分贡献. naive: 简单策略; 2D: Δup信号下的二维策略; 3D2S: Δup, Δuq信号下的三维策略; 3D4S: Snp, Snq, Δup, Δuq信号下的三维策略. (c)3D4S策略下的颗粒瞬时分布[24]4. (a) Vertical velocity of particles vz in frozen flow (frozen) and time-dependent flow (DNS), with the contributions of swimming velocity nzvs, local fluid velocity uz, and settling velocity vg,z. Smart 2D: smart strategy with signal
$ \mathrm{\Delta }{u}_{p} $ . Naive: naive strategy. (b) Vertical velocity of particles under different swimming strategies in homogeneous isotropic turbulence. Naive: naive strategy. 2D: two-dimensional strategy with Δup signal. 3D2S: two-dimensional strategy with Δup, Δuq signals. 3D4S: two-dimensional strategy with Snp, Snq, Δup, Δuq signals. (c) Instanueous distribution of particles with 3D4S strategy[24]
图 4 (a)颗粒在冻结流场(frozen)与随时间变化的流场(DNS)中的垂直方向速度vz, 以及游动速度nzvs, 当地流体速度uz和沉降速度vg,z的贡献. smart 2D:
$ \mathrm{\Delta }{u}_{p} $ 流体信号下的智能策略; naive: 简单策略智能颗粒. (b)不同策略下颗粒在三维各向同性均匀湍流中的运动速度及各部分贡献. naive: 简单策略; 2D: Δup信号下的二维策略; 3D2S: Δup, Δuq信号下的三维策略; 3D4S: Snp, Snq, Δup, Δuq信号下的三维策略. (c)3D4S策略下的颗粒瞬时分布[24] (续)Figure 4. (a) Vertical velocity of particles vz in frozen flow (frozen) and time-dependent flow (DNS), with the contributions of swimming velocity nzvs, local fluid velocity uz, and settling velocity vg,z. Smart 2D: smart strategy with signal
$ \mathrm{\Delta }{u}_{p} $ . Naive: naive strategy. (b) Vertical velocity of particles under different swimming strategies in homogeneous isotropic turbulence. Naive: naive strategy. 2D: two-dimensional strategy with Δup signal. 3D2S: two-dimensional strategy with Δup, Δuq signals. 3D4S: two-dimensional strategy with Snp, Snq, Δup, Δuq signals. (c) Instanueous distribution of particles with 3D4S strategy[24] (continued)表 1 二维与三维流场中独立流体信号分量
Table 1. Independent fluid signals in two-dimensional (2D) and three-dimensional (3D) flows
Flow signals 2D 3D deformation rate ${S_{nn}},\;\;{S_{np}}$ ${S_{nn}},\;{S_{np}},\;{S_{nq}},\;{S_{pp}},\;{S_{pq}}$ relative angular velocity $\Delta {\varOmega _q}$ $\Delta {\varOmega _n},\;\Delta {\varOmega _p},\;\Delta {\varOmega _q}$ relative velocity $\Delta {u_n},\;\Delta {u_p}$ $\Delta {u_n},\;\Delta {u_p},\;\Delta {u_q}$ 
下载: 导出CSV
-
[1] Berg HC, Brown DA. Chemotaxis in Escherichia coli analysed by three-dimensional tracking. Nature, 1972, 239(5374): 500-504 doi: 10.1038/239500a0 [2] Visser A. Small, Wet & Rational: Individual Based Zooplankton Ecology. Technical University of Denmark, 2011 [3] Nelson BJ, Kaliakatsos IK, Abbott JJ. Microrobots for minimally invasive medicine. Annual Review of Biomedical Engineering, 2010, 12: 55-85 doi: 10.1146/annurev-bioeng-010510-103409 [4] Reynolds CW. Flocks, herds and schools: A distributed behavioral model. ACM SIGGRAPH Computer Graphics, 1987, 21(4): 25-34 [5] Li S, Batra R, Brown D, et al. Particle robotics based on statistical mechanics of loosely coupled components. Nature, 2019, 567(7748): 361-365 doi: 10.1038/s41586-019-1022-9 [6] Durham WM, Climent E, Stocker R. Gyrotaxis in a steady vortical flow. Physical Review Letters, 2011, 106: 238102 doi: 10.1103/PhysRevLett.106.238102 [7] Kessler JO. Individual and collective fluid dynamics of swimming cells. Journal of Fluid Mechanics, 1986, 173: 191-205 doi: 10.1017/S0022112086001131 [8] Durham WM, Climent E, Barry M, et al. Turbulence drives microscale patches of motile phytoplankton. Nature Communications, 2013, 4: 2148 doi: 10.1038/ncomms3148 [9] Borgnino M, Boffetta G, De Lillo F, et al. Gyrotactic swimmers in turbulence: shape effects and role of the large-scale flow. Journal of Fluid Mechanics, 2018, 856: R1 doi: 10.1017/jfm.2018.767 [10] Gustavsson K, Berglund F, Jonsson PR, et al. Preferential sampling and small-scale clustering of gyrotactic microswimmers in turbulence. Physical Review Letters, 2016, 116(10): 108104 doi: 10.1103/PhysRevLett.116.108104 [11] Borgnino M, Gustavsson K, De Lillo F, et al. Alignment of nonspherical active particles in chaotic flows. Physical Review Letters, 2019, 123(13): 138003 doi: 10.1103/PhysRevLett.123.138003 [12] Zhan C, Sardina G, Lushi E, et al. Accumulation of motile elongated micro-organisms in turbulence. Journal of Fluid Mechanics, 2014, 739: 22-36 doi: 10.1017/jfm.2013.608 [13] Qiu J, Cui Z, Climent E, et al. Fluid inertia is an effective gyrotactic mechanism for settling elongated micro-swimmers. ArXiv Preprint ArXiv: 2011.10253.2020 [14] Kiørboe T, Saiz E, Visser AE. Hydrodynamic signal perception in the copepod Acartia tonsa. Marine Ecology Progress Series, 1999, 179: 97-111 doi: 10.3354/meps179097 [15] Kiørboe T, Visser AW. Predator and prey perception in copepods due to hydromechanical signals. Marine Ecology Progress Series, 1999, 179: 81-95 doi: 10.3354/meps179081 [16] Michalec FCCO, Fouxon I, Souissi S, et al. Zooplankton can actively adjust their motility to turbulent flow. Proceedings of the National Academy of Sciences, 2017, 114(52): E11199-E11207 doi: 10.1073/pnas.1708888114 [17] Colabrese S, Gustavsson K, Celani A, et al. Flow navigation by smart microswimmers via reinforcement learning. Physical Review Letters, 2017, 118(15): 158004 doi: 10.1103/PhysRevLett.118.158004 [18] Gustavsson K, Biferale L, Celani A, et al. Finding efficient swimming strategies in a three-dimensional chaotic flow by reinforcement learning. European Physical Journal E, 2017, 40(12): 110 doi: 10.1140/epje/i2017-11602-9 [19] Biferale L, Bonaccorso F, Buzzicotti M, et al. Zermelo's problem: Optimal point-to-point navigation in 2D turbulent flows using reinforcement learning. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2019, 29(10): 103138 doi: 10.1063/1.5120370 [20] Buzzicotti M, Biferale L, Bonaccorso F, et al. Optimal Control of Point-to-Point Navigation in Turbulent Time Dependent Flows using Reinforcement Learning. Springer, 2020: 223-234 [21] Techy L. Optimal navigation in planar time-varying flow: Zermelo's problem revisited. Intelligent Service Robotics, 2011, 4(4): 271-283 doi: 10.1007/s11370-011-0092-9 [22] Qiu J, Huang W, Xu C, et al. Swimming strategy of settling elongated micro-swimmers by reinforcement learning. Science China: Physics, Mechanics & Astronomy, 2020, 63(8): 284711 [23] Qiu J, Mousavi N, Gustavsson K, et al. Navigation of micro-swimmers in steady flow: The importance of symmetries. ArXiv Preprint ArXiv: 2104.11303.2021 [24] Qiu J, Mousavi N, Zhao L, et al. Active gyrotactic stability of microswimmers using hydromechanical signals. ArXiv Preprint ArXiv: 2105.12232.2021 [25] Anderson LW, Sweeney BM. Role of inorganic ions in controlling sedimentation rate of a marine centric diatom ditylum brightwelli 1, 2. Journal of Phycology, 1978, 14(2): 204-214 doi: 10.1111/j.1529-8817.1978.tb02450.x [26] Walsby AE. Gas vesicles. Microbiological Reviews, 1994, 58(1): 94-144 doi: 10.1128/mr.58.1.94-144.1994 [27] Arrieta J, Barreira A, Tuval I. Microscale patches of nonmotile phytoplankton. Physical Review Letters, 2015, 114(12): 128102 doi: 10.1103/PhysRevLett.114.128102 [28] Colabrese S, Gustavsson K, Celani A, et al. Smart inertial particles. Physical Review Fluids, 2018, 3(8): 84301 doi: 10.1103/PhysRevFluids.3.084301 [29] Gazzola M, Tchieu AA, Alexeev D, et al. Learning to school in the presence of hydrodynamic interactions. Journal of Fluid Mechanics, 2016, 789: 726-749 doi: 10.1017/jfm.2015.686 [30] Mirzakhanloo M, Esmaeilzadeh S, Alam M. Active cloaking in Stokes flows via reinforcement learning. Journal of Fluid Mechanics, 2020, 903: A34 [31] Novati G, Mahadevan L, Koumoutsakos P. Controlled gliding and perching through deep-reinforcement-learning. Physical Review Fluids, 2019, 4(9): 93902 doi: 10.1103/PhysRevFluids.4.093902 [32] Huntley M, Brooks ER. Effects of age and food availability on diel vertical migration of Calanus pacificus. Marine Biology, 1982, 71(1): 23-31 doi: 10.1007/BF00396989 [33] Bollens SM, Frost BW. Predator-induced diet vertical migration in a planktonic copepod. Journal of Plankton Research, 1989, 11(5): 1047-1065 doi: 10.1093/plankt/11.5.1047 [34] Titelman J, Kiørboe T. Motility of copepod nauplii and implications for food encounter. Marine Ecology Progress Series, 2003, 247: 123-135 doi: 10.3354/meps247123 [35] Happel J, Brenner H. Low Reynolds Number Hydrodynamics: With Special Applications to Particulate Media. Springer Science & Business Media, 2012 [36] Zhao L, Challabotla NR, Andersson HI, et al. Rotation of nonspherical particles in turbulent channel flow. Physical Review Letters, 2015, 115(24): 244501 doi: 10.1103/PhysRevLett.115.244501 [37] Kim S, Karrila SJ. Microhydrodynamics: Principles and Selected Applications. Courier Corporation, 2013 [38] Jeffery GB. The motion of ellipsoidal particles immersed in a viscous fluid. Proceedings of the Royal Society of London. Series A, 1922, 102(715): 161-179 [39] Sutton RS, Barto AG. Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). Cambridge: MIT Press, 1998 [40] Watkins CJCH, Dayan P. Q-learning. Machine Learning, 1992, 8(3): 279-292 [41] Durham WM, Kessler JO, Stocker R. Disruption of vertical motility by shear triggers formation of thin phytoplankton layers. Science, 2009, 323(5917): 1067-1070 doi: 10.1126/science.1167334 [42] Sengupta A, Carrara F, Stocker R. Phytoplankton can actively diversify their migration strategy in response to turbulent cues. Nature, 2017, 543(7646): 555-558 doi: 10.1038/nature21415 [43] Alageshan JK, Verma AK, Bec J, et al. Machine learning strategies for path-planning microswimmers in turbulent flows. Physical Review E, 2020, 101(4): 43110 doi: 10.1103/PhysRevE.101.043110 [44] Gunnarson P, Mandralis I, Novati G, et al. Learning efficient navigation in vortical flow fields. ArXiv Preprint ArXiv: 2102.10536.2021 [45] Yang Y, Bevan MA, Li B. Efficient navigation of colloidal robots in an unknown environment via deep reinforcement learning. Advanced Intelligent Systems, 2020, 2(1): 1900106 doi: 10.1002/aisy.201900106 [46] Liu Y, Zou Z, Tsang ACH, et al. Mechanical rotation at low Reynolds number via reinforcement learning. Physics of Fluids, 2021, 33(6): 62007 doi: 10.1063/5.0053563 [47] Tsang ACH, Tong PW, Nallan S, et al. Self-learning how to swim at low Reynolds number. Physical Review Fluids, 2020, 5(7): 74101 doi: 10.1103/PhysRevFluids.5.074101 -
下载:
点击查看大图
计量
- 文章访问数: 624
- HTML全文浏览量: 184
- PDF下载量: 193
- 被引次数: 0
