I created a Vertical Rocket Lander (similar to lunar lander): a continuous 3 action (main engine, side engines, nozzle angle) control problem modeled after SpaceX’s Falcon 9 first stage rocket.
I uploaded the code to GitHub where there’s a bit more documentation. There are also some controllers, including DDPG, MPC, PID, and ES, some of which were inspired by other repos. I hope others benefit from this interesting problem. Would be cool if it’s added to the OpenAI Box2D gym environments for others to notice it.
Compared to the lunar lander, it takes magnitudes of more control effort to successfully land the rocket. Besides having a higher center of gravity, the ratio of the force of the side engines to the main engines is much smaller in the rocket, making it impossible to keep upright unless the nozzle is rotated in a controlled way. Then again, the relatively large longitudinal length of the rocket amplifies small changes in the nozzle angle, making it very sensitive and unstable. Welcome to the problem.