Skip to content
Search

Shoggoth Mini

lastmod 2025-07-19created 2025-07-164 edits

Soft tentacle robot powered by GPT-4o and RL

About

  • Shoggoth Mini is a soft tentacle robot that uses GPT-4o and reinforcement learning to achieve natural, expressive motion
  • Unlike conventional home robots, it is designed to express internal states and intentions when interacting with humans
  • Describes the whole process in detail, from hardware design to software, camera perception, and the control system
  • Improves usability by intuitively mapping 2D trackpad input to 3D tentacle control, and achieves motion precision and expressiveness by combining modern computer vision and RL techniques
  • The conclusion reflects on the balance between the robot's liveliness, unpredictability, and human-friendliness, and presents ideas for future extensions

Shoggoth_Mini_-_system.gif

Introduction and background

  • Over the past year, robotics has advanced rapidly as it merged with the era of large language models (LLMs)
  • Notable examples like π0.5 and Tesla Optimus understand language commands and perform real actions, but most remain at the level of functional robots
  • Points out that 'expressiveness' is key to natural human-robot interaction and to conveying internal states (intention, attention, confidence, etc.)
  • Inspired by the Pixar-like lamp (Apple ELEGNT) and SpiRobs, which feel strangely 'alive' from simple motions alone, became interested in the difference between deliberately designed expression and accidental liveliness
  • Started building Shoggoth Mini to experiment with this, and shares the design process and insights gained through accidents and experiments

Hardware

  • The first version of the testbed started as a simple structure: 3 motors, a plate to hold the tentacle, and a dome-shaped cover
  • Running out of filament during 3D printing left part of the dome a different color, producing a face shape with cheeks and a mouth; visual mockups were explored with ChatGPT to finalize the form
  • A stereo camera mounted in the dome tracks the tentacle, and the robot eyes' 'prediction error' has the effect of drawing attention
  • The open spool design had the drawback of cables easily slipping off and tangling, so a simple spool cover was added to fix this and speed up iteration
  • A calibration script and extra spare wire length made maintenance and motor tension adjustment quick
  • The backbone (spine) was made moderately thicker to minimize tentacle sagging
  • Both the CAD assembly and STL files for 3D printing are published in the GitHub repository

Manual control

  • Initially, control of the lengths of the 3 tendons was simplified to a 2D control so the tentacle could be moved intuitively with 2D trackpad operation
    • Each tendon has a principal direction on the 2D plane; the cursor direction vector is projected onto each axis to compute the length change needed for the required tension
    • Positive means tendon shortening, negative means lengthening
  • This 2D→3D conversion is reused as the base projection layer in automated control (reinforcement learning, etc.)
  • Limited range of motion but intuitive control, greatly improving head responsiveness and user experience

System design

  • Two-level hierarchical control structure:
    • At the low level, it runs open-loop policies (e.g. <yes>, <shake>) and closed-loop RL policies (e.g. finger tracking); tracking of the tentacle tip/hand position is handled by a stereo vision-based pipeline
    • At the high level, GPT-4o handles real-time voice/text (vision not yet released); video events (hand waving, etc.) are passed to GPT-4o as text cues to decide API calls
  • Designed as a cascaded structure of dedicated vision and lightweight individual controllers, rather than direct end-to-end vision-language-action (VLA) integration based on large models
  • Adjusting the RL observation range, natural homing motion, and an idle (breathing mode) between API calls strengthen the robot's 'alive feeling'
  • Due to practical limitations of VLA (e.g. timing calibration issues), prompt engineering acts as an important factor

Perception

  • Combines MediaPipe for hand tracking with a custom dataset and YOLO model for tentacle tip tracking
    • Uses k-means clustering, Roboflow's auto-labeling/active learning, and Segment Anything for dataset augmentation and precise labeling
    • Trained with Ultralytics YOLO; camera intrinsic/extrinsic parameter estimation and 3D triangulation implemented with a DeepLabCut notebook
  • Real-time 3D hand-tentacle position tracking enables robust closed-loop control

Low-level control API

  • Unlike rigid bodies, inverse kinematics does not work well for soft robots, so the 2D control projection approach is applied consistently
  • Making the backbone thicker also ensures motion reproducibility across sessions
  • Experiments such as object grasping (grab) motions revealed the mechanical robustness unique to soft robots (absorbing variety in the shape/weight of grasped objects)
  • High-frequency fine manipulation is difficult, but basic manipulation is surprisingly stable from mechanical design alone

Reinforcement learning (RL)

  • RL applied starting from simple policies such as tracking the user's finger; dynamics randomization in the MuJoCo simulator (PPO, MLP, frame stacking, mass/friction/damping randomization) improves sim-to-real transfer
  • Initially used tendon lengths directly as the action space, but this led to reward hacking and failed real-world transfer
  • Restricting actions with the 2D projection approach suppresses unrealistic motion (chaos, oscillation, etc.), and curriculum learning is suggested as a path to gradual higher-dimensional extension
  • To counter jitter caused by abrupt action changes, a penalty term was added to the reward, and actions are smoothed with EMA

See also

Favorite site

author Zer0 sitemap