[Paper Review] A Robot to Shape your Natural Plant: The Machine Learning Approach to Model and Control Bio-Hybrid Systems

Mostafa Wahby, Mary Katherine Heinrich+6 more|IT University Of Copenhagen (IT University of Copenhagen)|Apr 18, 2018

Smart Agriculture and AI23 references5 citations

TL;DR

This paper presents a machine learning-driven approach to shape natural plants using robotic control, leveraging an LSTM-based forward model of plant motion and stiffening to evolve neural network controllers for obstacle-avoidance tasks. The method successfully transfers evolved controllers from simulation to real-world experiments, demonstrating collision-free growth around obstacles through 'embodied memory' in plant morphology.

ABSTRACT

Bio-hybrid systems---close couplings of natural organisms with technology---are high potential and still underexplored. In existing work, robots have mostly influenced group behaviors of animals. We explore the possibilities of mixing robots with natural plants, merging useful attributes. Significant synergies arise by combining the plants' ability to efficiently produce shaped material and the robots' ability to extend sensing and decision-making behaviors. However, programming robots to control plant motion and shape requires good knowledge of complex plant behaviors. Therefore, we use machine learning to create a holistic plant model and evolve robot controllers. As a benchmark task we choose obstacle avoidance. We use computer vision to construct a model of plant stem stiffening and motion dynamics by training an LSTM network. The LSTM network acts as a forward model predicting change in the plant, driving the evolution of neural network robot controllers. The evolved controllers augment the plants' natural light-finding and tissue-stiffening behaviors to avoid obstacles and grow desired shapes. We successfully verify the robot controllers and bio-hybrid behavior in reality, with a physical setup and actual plants.

Motivation & Objective

• To develop a bio-hybrid system that combines robotic control with natural plant growth for shape fabrication.
• To address the challenge of controlling plant morphology in dynamic environments using machine learning.
• To close the reality gap by transferring evolved robot controllers from simulation to real plants.
• To model complex plant behaviors—specifically phototropism and stem stiffening—using data-driven techniques.
• To enable scalable, adaptive control of plant growth beyond simple 2D obstacle avoidance.

Proposed method

• Collect experimental data on common bean (Phaseolus vulgaris) motion and stiffening under controlled light stimuli.
• Train a Long Short-Term Memory (LSTM) recurrent neural network to model plant dynamics as a forward model.
• Use the trained LSTM as a simulator in evolutionary robotics to evolve artificial neural network (ANN) controllers for dynamic light stimuli.
• Evolve ANN controllers in simulation to guide plant growth around obstacles while avoiding collisions.
• Validate the evolved controllers in physical reality using real plants and robotic light control systems.
• Introduce the concept of 'embodied memory'—where past motion and stiffening influence final plant shape—into the control framework.

Experimental results

Research questions

• RQ1 Can a data-driven LSTM model accurately capture the temporal dynamics of plant stem stiffening and phototropic motion for use in simulation?
• RQ2 How effectively can evolutionary robotics generate controllers in simulation that transfer successfully to real plant systems?
• RQ3 To what extent does the plant's history of motion and stiffening contribute to its final shape, and can this be leveraged for control?
• RQ4 Can the system achieve collision-free growth around obstacles while guiding the plant toward a target using only light stimuli?
• RQ5 What are the scalability limits of this approach for controlling complex 3D plant morphologies or multiple plants?

Key findings

• The LSTM model successfully captured the complex temporal dynamics of plant motion and stiffening from experimental data, serving as an effective forward model for simulation.
• Evolved robot controllers achieved obstacle-avoidance behavior in real-world experiments, with the plant tip reaching the target area after 198 hours of growth.
• The system demonstrated successful transfer from simulation to reality, validating the reality gap mitigation strategy.
• Fitness values of up to 99.3% were achieved in later experiments, indicating high success in guiding plant growth toward targets.
• The concept of 'embodied memory' was observed, where the plant's past motion and stiffening patterns directly influenced its final morphology.
• The approach enables scalable control of plant growth, with potential for future applications in 3D shape fabrication, plant weaving, and architectural structures.