Dear all,

After watching this very well made video about modeling a dynamical system using RNN and LSTM I decided to model my real system using the same concept. Basically I want to model the dynamics of my real robot to create a “digital twin” of the same. In other words I want to recreate the same robot in a simulator, with virtual physics properties and move it as if it were real.

My robot is controlled by a joystick that sends a float between -1.0 and 1.0 on each axis. I have collected the data (the real robot with sensors already working and implemented). For simplicity, let’s say I want to control the following common coordinate by moving my joystick axis from left to right (Fig. 1).

Image 1

I collected data for an hour and then trained an LSTM with a hidden size of 32 using the following data:

• The input is a combination of the joystick input and the joint coordinate (the state of the robot)
• The goal is representation by the state of the robot in the next step. I simply copied the columns of the state and moved them back one unit. Fig.2 probably shows more than 1000 words.

Image 2

Next I created sequences of 200 and trained my LSTM.
The training went very fast and I was very pleased with the results. But somehow the virtual robot behaves strangely in the virtual environment. It jumps from one position to another with incredible speed and then moves very slowly. So it does not react as the real robot would move (the real one is more fluid during the movement).

Am I missing something important in these types of problems?
What else should I take into account to create a good digital twin of the real robot?

Aside:

• despite the above example I have normalized all movements in the range (-1, 1) or (0, 1)
• all data was collected via an ethernet cable (so no delay due to wireless communication and the like)
• I used the LSTM classes from PyTorch and no custom realization
• The data were collected by generating sinusoidal inputs with different frequencies and covering the entire range of the joint.
• For training, I shuffled the data: a starting index was randomly chosen and a set of 200 elements was cut and used for training.