Chapter 2.6: Human-Robot Interaction in Unity
Learning Objectives
By the end of this chapter, students will be able to:
- Design realistic human-robot interaction scenarios in Unity
- Implement gesture and speech recognition for interaction
- Create social behaviors for humanoid robots
- Simulate multi-person interaction environments
- Integrate perception and action for natural interaction
Introduction
Human-robot interaction (HRI) is a critical aspect of humanoid robotics, especially as these robots are designed to operate in human environments and work alongside people. Creating realistic and natural interactions requires understanding both human social behaviors and how robots can appropriately respond to them. Unity provides an excellent platform for simulating these interactions with high-fidelity visualization and realistic environment rendering.
In this chapter, we'll explore how to design and implement human-robot interaction scenarios in Unity, focusing on creating natural and intuitive interactions that bridge the gap between human expectations and robot capabilities. We'll cover gesture recognition, speech interaction, social behaviors, and multi-person scenarios that are essential for humanoid robot development.
Designing Interaction Scenarios
Understanding HRI Principles
Human-robot interaction follows several key principles:
- Predictability: Robot behaviors should be understandable and consistent
- Approachability: Robots should appear non-threatening and open to interaction
- Intuitiveness: Interaction methods should align with human expectations
- Feedback: Robots should provide clear feedback for human actions
Creating Interaction Spaces
Design environments that facilitate natural interaction:
// InteractionSpace.cs - Define areas for human-robot interaction
using UnityEngine;
using System.Collections.Generic;
public class InteractionSpace : MonoBehaviour
{
[Header("Interaction Parameters")]
public float interactionRadius = 2.0f; // Distance for interaction
public float personalSpaceRadius = 0.5f; // Respect personal space
public float approachDistance = 1.0f; // Optimal approach distance
[Header("Interaction Triggers")]
public List<Collider> interactionTriggers = new List<Collider>();
public List<Collider> personalSpaceTriggers = new List<Collider>();
[Header("Visual Feedback")]
public GameObject interactionIndicator;
public Material activeMaterial;
public Material inactiveMaterial;
[Header("Human Detection")]
public LayerMask humanLayer = -1;
private List<Transform> detectedHumans = new List<Transform>();
void Start()
{
SetupInteractionZones();
}
void SetupInteractionZones()
{
// Create interaction trigger colliders
GameObject interactionZone = new GameObject("InteractionZone");
interactionZone.transform.SetParent(transform);
interactionZone.transform.localPosition = Vector3.zero;
SphereCollider interactionCollider = interactionZone.AddComponent<SphereCollider>();
interactionCollider.isTrigger = true;
interactionCollider.radius = interactionRadius;
interactionTriggers.Add(interactionCollider);
// Create personal space collider
GameObject personalSpaceZone = new GameObject("PersonalSpaceZone");
personalSpaceZone.transform.SetParent(transform);
personalSpaceZone.transform.localPosition = Vector3.zero;
SphereCollider personalSpaceCollider = personalSpaceZone.AddComponent<SphereCollider>();
personalSpaceCollider.isTrigger = true;
personalSpaceCollider.radius = personalSpaceRadius;
personalSpaceTriggers.Add(personalSpaceCollider);
// Set up visual indicator
if(interactionIndicator != null)
{
interactionIndicator.SetActive(false);
}
}
void Update()
{
CheckHumanPresence();
UpdateInteractionFeedback();
}
void CheckHumanPresence()
{
detectedHumans.Clear();
// Find humans within interaction radius
Collider[] hitColliders = Physics.OverlapSphere(transform.position, interactionRadius, humanLayer);
foreach(Collider collider in hitColliders)
{
if(collider.CompareTag("Human"))
{
detectedHumans.Add(collider.transform);
}
}
}
void UpdateInteractionFeedback()
{
bool hasHumans = detectedHumans.Count > 0;
if(interactionIndicator != null)
{
interactionIndicator.SetActive(hasHumans);
if(hasHumans)
{
// Use active material when humans are detected
Renderer indicatorRenderer = interactionIndicator.GetComponent<Renderer>();
if(indicatorRenderer != null && activeMaterial != null)
{
indicatorRenderer.material = activeMaterial;
}
}
else
{
// Use inactive material when no humans are detected
Renderer indicatorRenderer = interactionIndicator.GetComponent<Renderer>();
if(indicatorRenderer != null && inactiveMaterial != null)
{
indicatorRenderer.material = inactiveMaterial;
}
}
}
}
public bool IsHumanDetected()
{
return detectedHumans.Count > 0;
}
public Transform GetClosestHuman()
{
if(detectedHumans.Count == 0)
return null;
Transform closest = detectedHumans[0];
float closestDistance = Vector3.Distance(transform.position, closest.position);
foreach(Transform human in detectedHumans)
{
float distance = Vector3.Distance(transform.position, human.position);
if(distance < closestDistance)
{
closest = human;
closestDistance = distance;
}
}
return closest;
}
void OnTriggerEnter(Collider other)
{
if(other.CompareTag("Human"))
{
OnHumanEnterInteractionZone(other.transform);
}
}
void OnTriggerExit(Collider other)
{
if(other.CompareTag("Human"))
{
OnHumanExitInteractionZone(other.transform);
}
}
void OnHumanEnterInteractionZone(Transform human)
{
Debug.Log("Human entered interaction zone: " + human.name);
// Trigger interaction ready state
}
void OnHumanExitInteractionZone(Transform human)
{
Debug.Log("Human exited interaction zone: " + human.name);
// Reset interaction state
}
}
Gesture Recognition and Response
Basic Gesture Detection
Implement gesture recognition using Unity's input system:
// GestureRecognition.cs - Detect and respond to human gestures
using UnityEngine;
using UnityEngine.InputSystem;
using System.Collections.Generic;
public class GestureRecognition : MonoBehaviour
{
[Header("Gesture Detection")]
public float gestureDetectionRadius = 0.3f;
public float minGestureVelocity = 0.1f;
public float gestureTimeout = 2.0f;
[Header("Hand Tracking")]
public Transform leftHand;
public Transform rightHand;
[Header("Gesture Recognition")]
public List<GestureTemplate> gestureTemplates = new List<GestureTemplate>();
[Header("Interaction Feedback")]
public Animator robotAnimator;
public AudioSource audioSource;
private Vector3 lastLeftHandPos;
private Vector3 lastRightHandPos;
private List<Vector3> leftHandPath = new List<Vector3>();
private List<Vector3> rightHandPath = new List<Vector3>();
private float gestureStartTime;
[System.Serializable]
public class GestureTemplate
{
public string name;
public List<Vector3> path; // Normalized gesture path
public float tolerance = 0.1f;
public string responseAction;
}
void Start()
{
if(leftHand != null)
lastLeftHandPos = leftHand.position;
if(rightHand != null)
lastRightHandPos = rightHand.position;
}
void Update()
{
UpdateGestureTracking();
CheckForCompletedGestures();
}
void UpdateGestureTracking()
{
if(leftHand != null)
{
Vector3 currentLeftPos = leftHand.position;
Vector3 leftVelocity = (currentLeftPos - lastLeftHandPos) / Time.deltaTime;
// Check if hand is moving fast enough to be considered a gesture
if(leftVelocity.magnitude > minGestureVelocity)
{
leftHandPath.Add(currentLeftPos);
// Limit path length to prevent memory issues
if(leftHandPath.Count > 100)
{
leftHandPath.RemoveAt(0);
}
lastLeftHandPos = currentLeftPos;
// Start gesture timer if not already started
if(leftHandPath.Count == 1)
{
gestureStartTime = Time.time;
}
}
}
if(rightHand != null)
{
Vector3 currentRightPos = rightHand.position;
Vector3 rightVelocity = (currentRightPos - lastRightHandPos) / Time.deltaTime;
if(rightVelocity.magnitude > minGestureVelocity)
{
rightHandPath.Add(currentRightPos);
if(rightHandPath.Count > 100)
{
rightHandPath.RemoveAt(0);
}
lastRightHandPos = currentRightPos;
if(rightHandPath.Count == 1)
{
gestureStartTime = Time.time;
}
}
}
}
void CheckForCompletedGestures()
{
// Check if gesture timeout has been reached
if(leftHandPath.Count > 10 && Time.time - gestureStartTime > gestureTimeout)
{
RecognizeGesture(leftHandPath, "LeftHand");
leftHandPath.Clear();
}
if(rightHandPath.Count > 10 && Time.time - gestureStartTime > gestureTimeout)
{
RecognizeGesture(rightHandPath, "RightHand");
rightHandPath.Clear();
}
}
void RecognizeGesture(List<Vector3> path, string handName)
{
if(path.Count < 5) return; // Need minimum points for recognition
// Normalize the gesture path
List<Vector3> normalizedPath = NormalizeGesturePath(path);
// Compare with known gesture templates
foreach(GestureTemplate template in gestureTemplates)
{
if(MatchGesture(normalizedPath, template.path, template.tolerance))
{
Debug.Log($"Recognized gesture: {template.name} from {handName}");
RespondToGesture(template.responseAction);
return;
}
}
// If no match, clear the path
path.Clear();
}
List<Vector3> NormalizeGesturePath(List<Vector3> path)
{
// Normalize gesture path for comparison
// This is a simplified approach - real implementation would be more complex
if(path.Count == 0) return new List<Vector3>();
// Find bounding box of the path
Vector3 minPoint = path[0];
Vector3 maxPoint = path[0];
foreach(Vector3 point in path)
{
minPoint = Vector3.Min(minPoint, point);
maxPoint = Vector3.Max(maxPoint, point);
}
// Calculate size of the bounding box
Vector3 size = maxPoint - minPoint;
float maxSize = Mathf.Max(Mathf.Max(size.x, size.y), size.z);
// Normalize to unit size and center at origin
List<Vector3> normalizedPath = new List<Vector3>();
Vector3 center = (minPoint + maxPoint) / 2.0f;
foreach(Vector3 point in path)
{
Vector3 normalizedPoint = (point - center) / maxSize;
normalizedPath.Add(normalizedPoint);
}
return normalizedPath;
}
bool MatchGesture(List<Vector3> path1, List<Vector3> path2, float tolerance)
{
// Simple distance-based matching
// In a real implementation, you'd use more sophisticated algorithms
// like Dynamic Time Warping or Hidden Markov Models
if(path1.Count != path2.Count) return false;
float totalDistance = 0.0f;
for(int i = 0; i < path1.Count; i++)
{
totalDistance += Vector3.Distance(path1[i], path2[i]);
}
float averageDistance = totalDistance / path1.Count;
return averageDistance <= tolerance;
}
void RespondToGesture(string action)
{
switch(action)
{
case "wave":
WaveResponse();
break;
case "point":
PointResponse();
break;
case "beckon":
BeckonResponse();
break;
default:
Debug.Log("Unknown gesture response: " + action);
break;
}
}
void WaveResponse()
{
Debug.Log("Robot responding to wave gesture");
if(robotAnimator != null)
{
robotAnimator.SetTrigger("Wave");
}
if(audioSource != null)
{
// Play greeting sound
audioSource.Play();
}
}
void PointResponse()
{
Debug.Log("Robot responding to point gesture");
if(robotAnimator != null)
{
robotAnimator.SetTrigger("LookAt");
}
}
void BeckonResponse()
{
Debug.Log("Robot responding to beckon gesture");
if(robotAnimator != null)
{
robotAnimator.SetTrigger("Approach");
}
}
}
Speech Interaction
Voice Command Recognition
Implement speech recognition and response:
// SpeechInteraction.cs - Handle speech-based interaction
using UnityEngine;
using UnityEngine.Windows.Speech;
using System.Collections.Generic;
using System.Linq;
public class SpeechInteraction : MonoBehaviour
{
[Header("Speech Recognition")]
public string[] commandKeywords = {
"hello", "goodbye", "follow me", "stop", "help",
"what time is it", "where are you", "come here"
};
[Header("Speech Responses")]
public Dictionary<string, string> responses = new Dictionary<string, string>();
public AudioSource audioSource;
public TextToSpeech textToSpeech; // Custom component for TTS
[Header("Interaction State")]
public bool listening = true;
private KeywordRecognizer keywordRecognizer;
void Start()
{
SetupResponses();
SetupSpeechRecognition();
}
void SetupResponses()
{
// Initialize response dictionary
responses.Add("hello", "Hello! How can I help you today?");
responses.Add("goodbye", "Goodbye! Have a great day.");
responses.Add("follow me", "Okay, I will follow you now.");
responses.Add("stop", "Stopping. What would you like me to do?");
responses.Add("help", "I can follow you, answer questions, or assist with tasks.");
responses.Add("what time is it", "I don't have access to current time, but I can get it for you if connected to the internet.");
responses.Add("where are you", "I am right here, ready to assist you.");
responses.Add("come here", "Coming to you now.");
}
void SetupSpeechRecognition()
{
if(commandKeywords.Length > 0)
{
keywordRecognizer = new KeywordRecognizer(commandKeywords);
keywordRecognizer.OnPhraseRecognized += OnPhraseRecognized;
keywordRecognizer.Start();
}
}
void OnPhraseRecognized(PhraseRecognizedEventArgs args)
{
if(!listening) return;
string recognizedText = args.text.ToLower();
Debug.Log("Recognized: " + recognizedText);
if(responses.ContainsKey(recognizedText))
{
string response = responses[recognizedText];
Debug.Log("Response: " + response);
// Speak the response
if(textToSpeech != null)
{
textToSpeech.Speak(response);
}
else
{
// Fallback: use audio source if TTS is not available
// This would require pre-recorded audio clips
PlayResponseAudio(recognizedText);
}
// Trigger appropriate robot behavior based on command
TriggerRobotBehavior(recognizedText);
}
else
{
// Unknown command - ask for clarification
if(textToSpeech != null)
{
textToSpeech.Speak("I didn't understand that command. Could you please repeat?");
}
}
}
void PlayResponseAudio(string command)
{
// In a real implementation, you would have pre-recorded audio for each response
// For now, we'll just log it
Debug.Log("Playing audio for: " + command);
}
void TriggerRobotBehavior(string command)
{
// Trigger appropriate robot animation or behavior based on command
switch(command)
{
case "hello":
TriggerAnimation("Greet");
break;
case "follow me":
TriggerAnimation("Follow");
break;
case "stop":
TriggerAnimation("Stop");
break;
case "come here":
TriggerAnimation("Approach");
break;
default:
TriggerAnimation("Idle");
break;
}
}
void TriggerAnimation(string animationName)
{
// In a real implementation, this would trigger the appropriate animation
// through the robot's animation controller
Debug.Log("Triggering animation: " + animationName);
}
void OnDisable()
{
if(keywordRecognizer != null && keywordRecognizer.IsRunning)
{
keywordRecognizer.Stop();
}
}
// Method to dynamically add new commands
public void AddCommand(string keyword, string response)
{
if(!commandKeywords.Contains(keyword))
{
// In a real implementation, you'd need to recreate the keyword recognizer
// with the new command added to the array
Debug.Log("Added new command: " + keyword);
}
responses[keyword] = response;
}
// Method to toggle listening state
public void SetListening(bool state)
{
listening = state;
if(listening && keywordRecognizer != null && !keywordRecognizer.IsRunning)
{
keywordRecognizer.Start();
}
else if(!listening && keywordRecognizer != null && keywordRecognizer.IsRunning)
{
keywordRecognizer.Stop();
}
}
}
Social Behaviors Implementation
Social Norms and Behaviors
Implement social behaviors for natural interaction:
// SocialBehavior.cs - Implement social behaviors for humanoid robot
using UnityEngine;
using System.Collections;
public class SocialBehavior : MonoBehaviour
{
[Header("Social Distance")]
public float comfortableDistance = 1.0f;
public float intimateDistance = 0.5f;
public float publicDistance = 3.0f;
[Header("Gaze Behavior")]
public bool maintainEyeContact = true;
public float gazeDuration = 3.0f;
public float gazeVariation = 0.5f;
[Header("Approach Behavior")]
public float approachSpeed = 1.0f;
public float turnSpeed = 90.0f; // degrees per second
[Header("Social Cues")]
public Animator animator;
public SkinnedMeshRenderer meshRenderer;
private Transform targetHuman;
private bool isApproaching = false;
private bool isMaintainingGaze = false;
void Update()
{
if(targetHuman != null)
{
HandleSocialDistance();
HandleGazeBehavior();
}
}
public void SetTargetHuman(Transform human)
{
targetHuman = human;
StartCoroutine(ApproachBehavior());
}
IEnumerator ApproachBehavior()
{
if(targetHuman == null) yield break;
isApproaching = true;
// Calculate approach position (comfortable distance)
Vector3 directionToTarget = (targetHuman.position - transform.position).normalized;
Vector3 targetPosition = targetHuman.position - directionToTarget * comfortableDistance;
// Smooth approach
while(Vector3.Distance(transform.position, targetPosition) > 0.1f)
{
transform.position = Vector3.MoveTowards(
transform.position,
targetPosition,
approachSpeed * Time.deltaTime
);
// Turn to face the human
Vector3 directionToHuman = (targetHuman.position - transform.position).normalized;
Quaternion targetRotation = Quaternion.LookRotation(directionToHuman);
transform.rotation = Quaternion.RotateTowards(
transform.rotation,
targetRotation,
turnSpeed * Time.deltaTime
);
yield return null;
}
isApproaching = false;
// Start maintaining gaze
if(maintainEyeContact)
{
StartCoroutine(MaintainGaze());
}
}
IEnumerator MaintainGaze()
{
if(targetHuman == null) yield break;
isMaintainingGaze = true;
while(targetHuman != null)
{
// Look at the human with some natural variation
Vector3 lookTarget = targetHuman.position;
// Add slight random variation to make gaze more natural
lookTarget += new Vector3(
Random.Range(-gazeVariation, gazeVariation),
Random.Range(-gazeVariation, gazeVariation),
Random.Range(-gazeVariation, gazeVariation)
);
// Use transform lookat or animation for gaze
if(animator != null)
{
// If using animation-based gaze
animator.SetLookAtWeight(1.0f); // Enable lookAt IK
animator.SetLookAtPosition(lookTarget);
}
else
{
// If using transform-based gaze
Vector3 direction = lookTarget - transform.position;
transform.LookAt(lookTarget);
}
yield return new WaitForSeconds(gazeDuration);
}
isMaintainingGaze = false;
}
void HandleSocialDistance()
{
if(targetHuman == null) return;
float distance = Vector3.Distance(transform.position, targetHuman.position);
if(distance < intimateDistance)
{
// Too close - step back
Vector3 direction = (transform.position - targetHuman.position).normalized;
transform.position += direction * approachSpeed * Time.deltaTime;
}
else if(distance > publicDistance)
{
// Too far - but only approach if explicitly told to
// Otherwise maintain current distance
}
}
void HandleGazeBehavior()
{
if(targetHuman == null) return;
// Additional gaze behaviors like blinking, looking away occasionally
// to simulate natural human-like attention patterns
}
// Method to express emotions through facial expressions
public void ShowEmotion(string emotion)
{
if(animator != null)
{
switch(emotion.ToLower())
{
case "happy":
animator.SetTrigger("Happy");
break;
case "sad":
animator.SetTrigger("Sad");
break;
case "surprised":
animator.SetTrigger("Surprised");
break;
case "attentive":
animator.SetTrigger("Attentive");
break;
default:
animator.SetTrigger("Neutral");
break;
}
}
}
// Method to handle turn-taking in conversation
public void HandleConversationTurn()
{
// Implement conversation turn-taking logic
// This would involve listening, waiting for pauses, and responding appropriately
Debug.Log("Handling conversation turn");
}
// Method to show attention to multiple people
public void AttendToMultiplePeople(Transform[] people)
{
if(people.Length == 0) return;
if(people.Length == 1)
{
// Attend to single person
SetTargetHuman(people[0]);
}
else
{
// In a group, attend to the speaker or rotate attention
StartCoroutine(GroupAttentionBehavior(people));
}
}
IEnumerator GroupAttentionBehavior(Transform[] people)
{
int currentIndex = 0;
while(people.Length > 0)
{
// Attend to current person for a duration
SetTargetHuman(people[currentIndex]);
yield return new WaitForSeconds(5.0f); // Attend for 5 seconds
// Move to next person
currentIndex = (currentIndex + 1) % people.Length;
}
}
void OnDrawGizmosSelected()
{
// Visualize social distance zones
Gizmos.color = Color.yellow;
Gizmos.DrawWireSphere(transform.position, comfortableDistance);
Gizmos.color = Color.red;
Gizmos.DrawWireSphere(transform.position, intimateDistance);
Gizmos.color = Color.blue;
Gizmos.DrawWireSphere(transform.position, publicDistance);
}
}
Multi-Person Interaction Scenarios
Managing Group Interactions
Handle scenarios with multiple people:
// GroupInteractionManager.cs - Manage interactions with multiple people
using UnityEngine;
using System.Collections.Generic;
public class GroupInteractionManager : MonoBehaviour
{
[Header("Group Interaction")]
public List<Transform> detectedPeople = new List<Transform>();
public Transform primaryPerson; // The person currently being attended to
public float groupDetectionRadius = 5.0f;
[Header("Interaction Rules")]
public bool allowGroupInteraction = true;
public int maxPeopleInGroup = 5;
public float attentionSwitchInterval = 5.0f;
[Header("Social Priority")]
public SocialPriorityType priorityRule = SocialPriorityType.FirstComeFirstServed;
public enum SocialPriorityType
{
FirstComeFirstServed,
ClosestPerson,
PersonSpeaking,
LeaderDesignated
}
private float lastAttentionSwitchTime;
private SocialBehavior socialBehavior;
void Start()
{
socialBehavior = GetComponent<SocialBehavior>();
lastAttentionSwitchTime = Time.time;
}
void Update()
{
DetectPeople();
UpdateGroupInteraction();
}
void DetectPeople()
{
detectedPeople.Clear();
// Find all people in the environment
Collider[] peopleColliders = Physics.OverlapSphere(
transform.position,
groupDetectionRadius,
LayerMask.GetMask("Human")
);
foreach(Collider col in peopleColliders)
{
if(col.CompareTag("Human"))
{
detectedPeople.Add(col.transform);
}
}
// Limit number of people in group
if(detectedPeople.Count > maxPeopleInGroup)
{
// Keep only the closest people
detectedPeople.Sort((a, b) =>
Vector3.Distance(transform.position, a.position)
.CompareTo(Vector3.Distance(transform.position, b.position))
);
while(detectedPeople.Count > maxPeopleInGroup)
{
detectedPeople.RemoveAt(detectedPeople.Count - 1);
}
}
}
void UpdateGroupInteraction()
{
if(detectedPeople.Count == 0)
{
// No people detected, reset
primaryPerson = null;
return;
}
if(detectedPeople.Count == 1)
{
// Single person interaction
if(primaryPerson != detectedPeople[0])
{
primaryPerson = detectedPeople[0];
OnNewPrimaryPerson(primaryPerson);
}
}
else
{
// Multiple people - manage group interaction
if(allowGroupInteraction)
{
HandleGroupInteraction();
}
else
{
// Select primary person based on priority rule
SelectPrimaryPerson();
}
}
}
void HandleGroupInteraction()
{
// For group interaction, we might want to position the robot
// where it can see all people or focus on the speaker
if(Time.time - lastAttentionSwitchTime >= attentionSwitchInterval)
{
// Switch attention to different person
SwitchGroupAttention();
lastAttentionSwitchTime = Time.time;
}
}
void SelectPrimaryPerson()
{
Transform newPrimary = null;
switch(priorityRule)
{
case SocialPriorityType.FirstComeFirstServed:
newPrimary = detectedPeople[0]; // First in the list
break;
case SocialPriorityType.ClosestPerson:
newPrimary = GetClosestPerson();
break;
case SocialPriorityType.PersonSpeaking:
newPrimary = GetSpeakingPerson();
break;
case SocialPriorityType.LeaderDesignated:
newPrimary = GetLeaderPerson();
break;
}
if(newPrimary != primaryPerson)
{
primaryPerson = newPrimary;
OnNewPrimaryPerson(primaryPerson);
}
}
Transform GetClosestPerson()
{
Transform closest = null;
float closestDistance = float.MaxValue;
foreach(Transform person in detectedPeople)
{
float distance = Vector3.Distance(transform.position, person.position);
if(distance < closestDistance)
{
closest = person;
closestDistance = distance;
}
}
return closest;
}
Transform GetSpeakingPerson()
{
// In a real implementation, this would use audio analysis
// to determine who is speaking
// For now, return the first person as a placeholder
return detectedPeople.Count > 0 ? detectedPeople[0] : null;
}
Transform GetLeaderPerson()
{
// In a real implementation, this would identify the designated leader
// through various cues (position, gestures, etc.)
// For now, return the first person as a placeholder
return detectedPeople.Count > 0 ? detectedPeople[0] : null;
}
void SwitchGroupAttention()
{
if(detectedPeople.Count <= 1) return;
int currentIndex = detectedPeople.IndexOf(primaryPerson);
if(currentIndex < 0) currentIndex = 0;
int nextIndex = (currentIndex + 1) % detectedPeople.Count;
primaryPerson = detectedPeople[nextIndex];
OnNewPrimaryPerson(primaryPerson);
}
void OnNewPrimaryPerson(Transform newPerson)
{
if(newPerson != null && socialBehavior != null)
{
socialBehavior.SetTargetHuman(newPerson);
Debug.Log("Switching attention to: " + newPerson.name);
}
}
// Method to handle specific group interaction scenarios
public void HandleGroupScenario(GroupScenarioType scenario)
{
switch(scenario)
{
case GroupScenarioType.Formation:
HandleFormationScenario();
break;
case GroupScenarioType.Guidance:
HandleGuidanceScenario();
break;
case GroupScenarioType.InformationSharing:
HandleInformationScenario();
break;
}
}
void HandleFormationScenario()
{
// Handle scenarios where the robot should position itself
// appropriately in a group formation
Debug.Log("Handling formation scenario");
}
void HandleGuidanceScenario()
{
// Handle scenarios where the robot is guiding a group
Debug.Log("Handling guidance scenario");
}
void HandleInformationScenario()
{
// Handle scenarios where the robot is sharing information with a group
Debug.Log("Handling information sharing scenario");
}
public enum GroupScenarioType
{
Formation, // Positioning in a group
Guidance, // Guiding a group
InformationSharing // Sharing information with group
}
void OnDrawGizmos()
{
if(allowGroupInteraction)
{
Gizmos.color = Color.cyan;
Gizmos.DrawWireSphere(transform.position, groupDetectionRadius);
}
}
}
Perception and Action Integration
Combining Perception and Action
Integrate perception and action for natural interaction:
// PerceptionActionIntegration.cs - Integrate perception and action
using UnityEngine;
using System.Collections;
public class PerceptionActionIntegration : MonoBehaviour
{
[Header("Perception Components")]
public InteractionSpace interactionSpace;
public GestureRecognition gestureRecognition;
public SpeechInteraction speechInteraction;
public SocialBehavior socialBehavior;
public GroupInteractionManager groupManager;
[Header("Action Components")]
public Animator robotAnimator;
public AudioSource audioSource;
public NavigationAgent navigationAgent; // Custom navigation component
[Header("Interaction State")]
public InteractionState currentState = InteractionState.Idle;
public float interactionTimeout = 30.0f;
[Header("Response Timing")]
public float minResponseDelay = 0.5f;
public float maxResponseDelay = 2.0f;
private float interactionStartTime;
private Coroutine currentResponseRoutine;
public enum InteractionState
{
Idle,
Detecting,
Engaging,
Responding,
Waiting,
Disengaging
}
void Start()
{
interactionStartTime = Time.time;
}
void Update()
{
UpdateInteractionState();
CheckInteractionTimeout();
}
void UpdateInteractionState()
{
switch(currentState)
{
case InteractionState.Idle:
if(interactionSpace.IsHumanDetected())
{
TransitionToState(InteractionState.Detecting);
}
break;
case InteractionState.Detecting:
// Wait for clear signal of intent to interact
if(HasClearInteractionSignal())
{
TransitionToState(InteractionState.Engaging);
}
else if(Time.time - interactionStartTime > 5.0f)
{
// No clear signal after 5 seconds, go back to idle
TransitionToState(InteractionState.Idle);
}
break;
case InteractionState.Engaging:
EngageWithHuman();
TransitionToState(InteractionState.Waiting);
break;
case InteractionState.Waiting:
// Wait for human input
if(HasHumanInput())
{
TransitionToState(InteractionState.Responding);
}
break;
case InteractionState.Responding:
// Response is handled in response routines
break;
case InteractionState.Disengaging:
// Handle disengagement process
if(IsDisengagementComplete())
{
TransitionToState(InteractionState.Idle);
}
break;
}
}
bool HasClearInteractionSignal()
{
// Determine if there's a clear signal of intent to interact
// This could be sustained eye contact, specific gestures, etc.
return gestureRecognition.leftHandPath.Count > 5 ||
gestureRecognition.rightHandPath.Count > 5 ||
speechInteraction.listening;
}
bool HasHumanInput()
{
// Check if there's been recent human input
// This would be implemented based on your specific input detection
return gestureRecognition.leftHandPath.Count > 0 ||
gestureRecognition.rightHandPath.Count > 0;
}
void EngageWithHuman()
{
Transform closestHuman = interactionSpace.GetClosestHuman();
if(closestHuman != null && socialBehavior != null)
{
socialBehavior.SetTargetHuman(closestHuman);
// Play engagement animation
if(robotAnimator != null)
{
robotAnimator.SetTrigger("Greet");
}
// Play engagement sound
if(audioSource != null)
{
// Play "ready to interact" sound
}
}
}
bool IsDisengagementComplete()
{
// Check if disengagement process is complete
// This might involve waiting for human to move away
Transform closestHuman = interactionSpace.GetClosestHuman();
if(closestHuman == null)
{
return true;
}
float distance = Vector3.Distance(transform.position, closestHuman.position);
return distance > interactionSpace.interactionRadius * 1.5f;
}
void TransitionToState(InteractionState newState)
{
// Cancel previous response routine if switching from responding
if(currentState == InteractionState.Responding && currentResponseRoutine != null)
{
StopCoroutine(currentResponseRoutine);
}
currentState = newState;
interactionStartTime = Time.time;
Debug.Log("Transitioned to state: " + newState);
// Handle specific actions for each state transition
switch(newState)
{
case InteractionState.Idle:
OnEnterIdleState();
break;
case InteractionState.Engaging:
OnEnterEngagingState();
break;
case InteractionState.Responding:
currentResponseRoutine = StartCoroutine(HandleResponse());
break;
case InteractionState.Disengaging:
OnEnterDisengagingState();
break;
}
}
void OnEnterIdleState()
{
// Reset all interaction components
if(socialBehavior != null)
{
socialBehavior.enabled = false;
}
if(speechInteraction != null)
{
speechInteraction.SetListening(false);
}
}
void OnEnterEngagingState()
{
// Prepare for interaction
if(socialBehavior != null)
{
socialBehavior.enabled = true;
}
if(speechInteraction != null)
{
speechInteraction.SetListening(true);
}
}
void OnEnterDisengagingState()
{
// Play disengagement animation
if(robotAnimator != null)
{
robotAnimator.SetTrigger("Farewell");
}
}
IEnumerator HandleResponse()
{
// Wait a random delay to make response feel natural
float delay = Random.Range(minResponseDelay, maxResponseDelay);
yield return new WaitForSeconds(delay);
// Determine appropriate response based on current situation
string response = DetermineResponse();
// Execute response
ExecuteResponse(response);
// Transition back to waiting state
TransitionToState(InteractionState.Waiting);
}
string DetermineResponse()
{
// Determine the appropriate response based on:
// - Current interaction state
// - Detected gestures
// - Spoken commands
// - Social context
// For now, return a placeholder
if(gestureRecognition.leftHandPath.Count > 0)
{
return "gesture_response";
}
else
{
return "idle_response";
}
}
void ExecuteResponse(string response)
{
switch(response)
{
case "gesture_response":
if(robotAnimator != null)
{
robotAnimator.SetTrigger("Acknowledge");
}
break;
case "idle_response":
// Continue with idle behavior
if(robotAnimator != null)
{
robotAnimator.SetTrigger("Idle");
}
break;
default:
// Handle other responses
break;
}
}
void CheckInteractionTimeout()
{
if(currentState != InteractionState.Idle &&
Time.time - interactionStartTime > interactionTimeout)
{
TransitionToState(InteractionState.Disengaging);
}
}
// Public method to force state transition (useful for testing)
public void ForceState(InteractionState newState)
{
TransitionToState(newState);
}
// Method to get current interaction status
public string GetInteractionStatus()
{
return $"State: {currentState}, Humans: {interactionSpace.IsHumanDetected()}";
}
}
Best Practices for HRI in Unity
1. Natural Movement and Animation
Create natural-looking robot movements:
// NaturalMovement.cs - Create natural robot movements
using UnityEngine;
public class NaturalMovement : MonoBehaviour
{
[Header("Movement Naturalness")]
public float movementVariation = 0.1f;
public float headSwayIntensity = 0.05f;
public float anticipationIntensity = 0.2f;
[Header("Timing Variation")]
public float timingVariation = 0.2f;
private Animator animator;
private float originalSpeed;
void Start()
{
animator = GetComponent<Animator>();
if(animator != null)
{
originalSpeed = animator.speed;
}
}
// Add subtle variations to make movement look more natural
public void ApplyNaturalMovementVariation()
{
if(animator != null)
{
// Vary animation speed slightly
float speedVariation = Random.Range(1.0f - timingVariation, 1.0f + timingVariation);
animator.speed = originalSpeed * speedVariation;
// Add subtle head sway
Transform head = GetChildByName("Head");
if(head != null)
{
head.localPosition += new Vector3(
Random.Range(-headSwayIntensity, headSwayIntensity),
Random.Range(-headSwayIntensity, headSwayIntensity),
0
);
}
}
}
Transform GetChildByName(string name)
{
foreach(Transform child in transform)
{
if(child.name == name)
return child;
}
return null;
}
}
2. Appropriate Response Timing
Ensure responses feel natural and not robotic:
// ResponseTiming.cs - Manage natural response timing
using UnityEngine;
using System.Collections;
public class ResponseTiming : MonoBehaviour
{
[Header("Response Timing")]
public float minDelay = 0.3f;
public float maxDelay = 1.5f;
public bool varyTiming = true;
public void ExecuteDelayedResponse(System.Action responseAction)
{
float delay = varyTiming ?
Random.Range(minDelay, maxDelay) :
(minDelay + maxDelay) / 2f;
StartCoroutine(DelayedExecution(delay, responseAction));
}
IEnumerator DelayedExecution(float delay, System.Action action)
{
yield return new WaitForSeconds(delay);
action?.Invoke();
}
}
Hands-On Exercise: HRI Scenario Implementation
Objective
Create a complete human-robot interaction scenario in Unity with gesture recognition, speech interaction, and social behaviors.
Prerequisites
- Completed previous chapters
- Unity with robotics setup
- Basic understanding of C# scripting
Steps
- Create a Unity scene with a humanoid robot and human avatars
- Implement the InteractionSpace component for detecting humans
- Add gesture recognition for basic commands (wave, point, beckon)
- Implement speech interaction with voice commands
- Add social behaviors including appropriate distance and gaze
- Create a multi-person interaction scenario
- Integrate perception and action for natural interaction flow
- Test the interaction scenario with different human avatars
Expected Result
Students will have a Unity scene with a humanoid robot that can detect humans, respond to gestures and voice commands, exhibit appropriate social behaviors, and manage multi-person interactions naturally.
Assessment Questions
Multiple Choice
Q1: What is the recommended comfortable distance for human-robot interaction according to proxemics research?
- a) 0.5 meters
- b) 1.0 meters
- c) 2.0 meters
- d) 3.0 meters
Details
Click to reveal answer
Answer: bExplanation: According to proxemics research, the comfortable distance for interaction between familiar people (which applies to human-robot interaction) is typically around 1.0 meter.
Short Answer
Q2: Explain why response timing variation is important in human-robot interaction and how it affects user perception.
Practical Exercise
Q3: Implement a Unity scene with a humanoid robot that can detect human presence, respond to a "wave" gesture with an appropriate response, maintain appropriate social distance, and engage in a simple conversation using voice commands.
Further Reading
- "Human-Robot Interaction: A Survey" - Foundational HRI research
- "Social Robotics" - Principles of social robot design
- "Unity for Simulation and Training" - Robotics simulation in Unity
Summary
In this chapter, we've explored human-robot interaction in Unity, focusing on creating natural and intuitive interactions for humanoid robots. We've covered designing interaction scenarios, implementing gesture and speech recognition, creating social behaviors, managing multi-person interactions, and integrating perception with action for natural interaction flow.
Creating realistic HRI scenarios in Unity is crucial for developing humanoid robots that can operate effectively in human environments. The high-fidelity visualization capabilities of Unity allow for testing and refining interaction behaviors before deployment on physical robots, making the development process more efficient and safer.
With the completion of this chapter, we've covered all aspects of Module 2: The Digital Twin (Gazebo & Unity). In the next module, we'll explore the NVIDIA Isaac platform for AI-powered robotics, where we'll learn to implement advanced perception and control algorithms for humanoid robots.