Reinforcement learning for the PMPY swimmer

We use the \(Q\)-\(learning\) and \(Double\) \(Q\)-\(learning\) algorithms to self-learn the PMPY how to swim. The \(\varepsilon\)-greedy scheme will be used in reinforcement learning algorithms for a better exploration of the medium.

The algorithms and the \(\varepsilon\)-greedy were presented in details for the three-sphere swimmer here.

As the total volume of the spheres should be conserved, the swimmer can not freely change the volumes of its spheres. For this reason, the swimmer will change (for e.g.) the length of the left sphere while the volume of the right sphere will be changing according to the left one to guarantee the conservation of the total volume. Hence, the actions taken by the swimmer can be described by the change of :

Let \(v_T\) be the total volume of the spheres. We denote by \(\mathcal{V}_l\) the finite set of possible volumes that the left sphere can take.

\[\mathcal{V}_l = \bigg\{ v_1, v_2, \dots , v_k \big/ k\geq 2\; \text{and}\; 0 <v_i\neq v_{i+1}< v_T\ \bigg\}. \]

Let \(\mathcal{R}_l\) be the set of all the radii that correspond to the volumes of the left sphere given in \(\mathcal{V}_l\). It can be expressed as

\[\mathcal{R}_l = \bigg\{ r_1, r_2, \dots , r_k \big/ r_i = \sqrt{\frac{v_i}{\pi}}\; \text{such that}\; v_i\in\mathcal{V}_l \bigg\}. \]

Let \(L\) be the finite set of the possible length that the arm of the swimmer can take given as follow

\[\mathcal{L} = \bigg\{ \ell_1, \ell_2, \dots , \ell_k \big/ k\geq 2\; \text{and}\;\ell_i\neq\ell_{i+1} \bigg\}. \]

Following these notations, the configurations of the swimmer can be expressed using the radius of the left sphere and the length of its arm. Hence, for \(\mathcal{R}_l\) and \(\mathcal{L}\) given, the space of states can be expressed as follow

\[\mathcal{S}= \mathcal{R}_l\times\mathcal{L}=\bigg\{ (r, \ell) \big/ r\in\mathcal{R}_l\; \text{and}\; \ell\in \mathcal{L} \bigg\}.\]

To move from a state to another, the swimmer changes the length of its arm (to a length given in \(\mathcal{L}\)) or changes the volume of its left sphere to another volume in \(\mathcal{V}_l\). Note that, in our reinforcement learning approach, the swimmer can not change the volume and length of its arm at the same time.

Taking an action in a certain state makes the swimmer move to another position with new configuration. This net displacement will be the rewward that measures the success of taking this action.

The displacement of the swimmer will be computed as follow

\[r = \frac{dx_1+dx_2}{2}, \]

where \(x_1\) and \(x_2\) denote the center of mass of the left and the right spheres respectively.

We note that the swimmer aims to maximize the cumulative reward during the learning process.

In our reinforcement learning implementation, the hydrodynamics interations between the swimmer and the surrounding fluid are simulated with the Feel++ fluid toolbox. At each iteration, a simulation of the swimmer performing the the given action (in a specific state) will be ran automatically and the reward will be recovered from a \(csv\) exported by the toolbox.

Since our model is described with Stokes equations, a pair of action-state always gives the same reward (displacement). Hence, each pair of action-state will be simulated once and the reward will be stored (in a matrix for e.g.) to be used without running the same simulation.

We start with a simple example in which the swimmer’s arm can have only two possible lengths and the volume of its left sphere can take only two values. We put \(\mathcal{R}_l = \{1, 2\}\) and \(\mathcal{L} = \{6, 10\}\).

In this case, the swimmer has only four different configurations illustrated as follow

Note that in this case, the swimmer has only one non-reciprocal motion. Hence, there is only one swimming strategy giving a net displacement (the strategy simulated here).

The two figures below show a typical learning process of the PMPY swimmer using Q-learning and double Q-learning algorithms.here

As can be seen, the swimmer has managed to swim by interacting with the surrounding medium. After accumulating enough knowledge (3O learning steps for QL and 20 for double QL), the swimmer finds the non-reciprocal motion that enables it to swim forward.

To investigate the number of learning steps required to learn this pattern. We performed 100 simulations for both Q-learning and double Q-learning algorithms using the same learning parameters mentioned above.

In this example, we put \(\mathcal{R}_l = \{1, 1.5, 2\}\) and \(\mathcal{L} = \{6, 8, 10\}\). In this case, the swimmer can perform many non-reciprocal motions. We will show that the swimmer can find the optimal swimming strategy that gives the largest net displacement.

Here, the swimmer has nine different configurations and can perform eight different actions. The optimal swimming strategy.

Note that from the net displacement computed in [Avron_pmpu], it is clear that the optimal gait in our case is the closed path in the \(v-\ell\) plane using the extreme values in \(\mathcal{V}_l\) and \(\mathcal{L}\).

*

*

*

*