Sign Language Game
Generate a complete single-file HTML webcam game using MediaPipe to recognize hand gestures and teach vocabulary.
Customize your prompt
Prompt
Act as an expert Creative Technologist and Senior Frontend Developer. I need you to generate a complete, single-file HTML solution (containing HTML, CSS, and JavaScript) for a "Sign Language Word Trainer" game.
**Language Requirement:**
Please ensure all UI text, instructions, and feedback in the app are written in **[TARGET LANGUAGE]**.
**Core Tech Stack:**
- Plain HTML5
- CSS (internal styles)
- Vanilla JavaScript (ES Modules)
- MediaPipe Tasks Vision (specifically HandLandmarker) imported via CDN (version 0.10.0).
**Visual Style & UI:**
- **Theme:** Dark mode. Background: `#0f172a` (Dark Slate), Accents: `#fbbf24` (Amber/Gold), Success: `#2ecc71` (Green).
- **Layout:** - Full-screen viewport.
- Webcam video fills the screen (object-fit: cover), mirrored horizontally (`scaleX(-1)`).
- A Canvas overlay on top of the video for drawing hand landmarks.
- A "Start Screen" overlay with a title, loading text, and a "Start" button (hidden until MediaPipe loads).
- **HUD (Heads Up Display):**
- A bottom bar (`instruction-bar`) with a semi-transparent dark background, blur effect, and border-top.
- Inside the bar: The current target word (huge, uppercase), a text instruction describing the gesture, and a progress bar.
- **Progress Bar:** Fills up based on gesture confidence. If confidence > 95%, it turns green.
- **Status Overlay:** A large text centered on the screen that pops up when a gesture is correct (e.g., "GOOD!"), with a scale/fade animation.
- **Level Counter:** Top right corner (e.g., "1 / 9").
**Game Logic & Architecture:**
1. **Initialization:** Load `FilesetResolver` and `HandLandmarker` (GPU delegate). Show the start button only after loading.
2. **Loop:** On `requestAnimationFrame`, detect hands.
- If no hands are detected, decrease "targetConfidence" rapidly.
- If hands are detected, draw the skeleton (connectors: amber, landmarks: white).
3. **Gesture Recognition Engine:**
- Implement helper functions for geometry:
- `getDist(p1, p2)`: Euclidean distance.
- `isFingerUp(landmarks, tipIdx)`: Checks if the finger tip is vertically higher (lower Y value) than its PIP joint.
- `isThumbOpen(landmarks)`: Checks distance of thumb tip to the palm center (MCP of index finger) to see if it's extended.
- `areFingersUp(landmarks, patternArray)`: Checks the 4 fingers (Index to Pinky) against a pattern (e.g., [1,0,0,0]).
- **Confidence System:** If the geometric check passes, increase `targetConfidence` by 4. If it fails, decrease by 8. Smoothly interpolate the visual `confidence` variable towards `targetConfidence`.
- **Success State:** If `confidence` hits 100%, trigger success animation, wait 2.5 seconds, then switch to the next level.
**The Levels (Implement these specific gestures):**
1. **Word: "HELLO"** -> Instruction: Open hand, wave. (Check: All 5 fingers up/open).
2. **Word: "GOOD"** -> Instruction: Thumbs up. (Check: Thumb open, other 4 fingers closed).
3. **Word: "I LOVE YOU"** -> Instruction: Pinky, Index, Thumb up. (Check: Thumb, Index, Pinky up; Middle & Ring down).
4. **Word: "CALL"** -> Instruction: Phone gesture. (Check: Thumb & Pinky up, others down).
5. **Word: "OK"** -> Instruction: Circle with Thumb and Index. (Check: Distance between Thumb tip and Index tip < 0.1, other 3 fingers up).
6. **Word: "LITTLE BIT"** -> Instruction: Pinch Thumb and Index close. (Check: Thumb & Index close, but other fingers MUST be down).
7. **Word: "PEACE"** -> Instruction: V-sign. (Check: Index & Middle up, others down).
8. **Word: "STOP"** -> Instruction: Flat hand raised. (Same geometry as Hello, but contextually different level).
9. **Word: "POINT"** -> Instruction: Pointing finger. (Check: Index up, others closed).
**Output:**
Provide the full, runnable code in a single code block.Example Output
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Sign Language Trainer</title>
<style>
/* --- CORE & RESET --- */
:root {
--bg-color: #0f172a;
--accent-color: #fbbf24; /* Amber */
--success-color: #2ecc71; /* Green */
--text-color: #f8fafc;
--glass-bg: rgba(15, 23, 42, 0.85);
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
font-family: 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;
background-color: var(--bg-color);
color: var(--text-color);
overflow: hidden; /* Prevent scrolling */
height: 100vh;
width: 100vw;
display: flex;
align-items: center;
justify-content: center;
}
/* --- VIDEO & CANVAS LAYERS --- */
#game-container {
position: relative;
width: 100%;
height: 100%;
background: #000;
}
/* Mirror the video for natural user interaction */
video, canvas {
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
object-fit: cover;
transform: scaleX(-1);
}
/* Canvas needs to be on top of video */
canvas {
z-index: 10;
}
/* --- UI OVERLAYS (Not mirrored) --- */
#ui-layer {
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
z-index: 20;
pointer-events: none; /* Let clicks pass through mostly */
display: flex;
flex-direction: column;
justify-content: space-between;
}
/* --- HUD TOP --- */
.hud-top {
padding: 20px;
display: flex;
justify-content: flex-end;
font-size: 1.2rem;
font-weight: bold;
text-shadow: 0 2px 4px rgba(0,0,0,0.8);
color: var(--accent-color);
}
/* --- HUD BOTTOM (Instruction Bar) --- */
.hud-bottom {
background: var(--glass-bg);
backdrop-filter: blur(10px);
border-top: 2px solid var(--accent-color);
padding: 20px 40px;
text-align: center;
transition: border-color 0.3s ease;
}
.hud-bottom.success {
border-color: var(--success-color);
}
.word-target {
font-size: 3rem;
font-weight: 900;
text-transform: uppercase;
letter-spacing: 2px;
margin-bottom: 8px;
color: #fff;
text-shadow: 0 0 20px rgba(255,255,255,0.2);
}
.instruction-text {
font-size: 1.1rem;
color: #cbd5e1;
margin-bottom: 15px;
}
/* --- PROGRESS BAR --- */
.progress-track {
width: 100%;
height: 10px;
background: rgba(255,255,255,0.1);
border-radius: 5px;
overflow: hidden;
position: relative;
}
.progress-fill {
height: 100%;
width: 0%;
background-color: var(--accent-color);
transition: width 0.1s linear, background-color 0.3s ease;
box-shadow: 0 0 10px var(--accent-color);
}
.progress-fill.complete {
background-color: var(--success-color);
box-shadow: 0 0 15px var(--success-color);
}
/* --- START SCREEN --- */
#start-screen {
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
background: var(--bg-color);
z-index: 50;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
pointer-events: auto;
}
h1 {
font-size: 3rem;
color: var(--accent-color);
margin-bottom: 1rem;
}
.loader {
color: #94a3b8;
margin-bottom: 2rem;
}
button {
background: var(--accent-color);
color: var(--bg-color);
border: none;
padding: 15px 40px;
font-size: 1.5rem;
font-weight: bold;
border-radius: 8px;
cursor: pointer;
opacity: 0.5;
pointer-events: none;
transition: opacity 0.3s, transform 0.2s;
}
button.ready {
opacity: 1;
pointer-events: auto;
box-shadow: 0 0 20px rgba(251, 191, 36, 0.4);
}
button:hover {
transform: scale(1.05);
}
/* --- SUCCESS OVERLAY --- */
#success-overlay {
position: absolute;
top: 50%;
left: 50%;
transform: translate(-50%, -50%) scale(0.5);
font-size: 5rem;
font-weight: 900;
color: var(--success-color);
opacity: 0;
pointer-events: none;
text-shadow: 0 0 30px rgba(46, 204, 113, 0.5);
z-index: 30;
transition: all 0.3s cubic-bezier(0.175, 0.885, 0.32, 1.275);
}
#success-overlay.active {
opacity: 1;
transform: translate(-50%, -50%) scale(1);
}
</style>
</head>
<body>
<div id="game-container">
<video id="webcam" autoplay playsinline></video>
<canvas id="output_canvas"></canvas>
<div id="ui-layer">
<div class="hud-top">
LEVEL <span id="level-indicator" style="margin-left:10px">1 / 9</span>
</div>
<div id="success-overlay">GOOD!</div>
<div class="hud-bottom" id="hud-bottom">
<div class="word-target" id="target-word">LOADING...</div>
<div class="instruction-text" id="instruction-text">Please wait for AI models to load.</div>
<div class="progress-track">
<div class="progress-fill" id="progress-bar"></div>
</div>
</div>
</div>
<div id="start-screen">
<h1>HAND TRAINER</h1>
<div class="loader" id="loader-text">Initializing Vision Engine...</div>
<button id="start-btn">START CAMERA</button>
</div>
</div>
<script type="module">
import { FilesetResolver, HandLandmarker } from "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.0/+esm";
// --- CONFIGURATION ---
const TARGET_FPS = 30;
const CONFIDENCE_INCREASE = 4; // Speed of bar going up
const CONFIDENCE_DECREASE = 8; // Speed of bar going down
const SUCCESS_THRESHOLD = 100;
const SUCCESS_DELAY_MS = 2500;
// --- DOM ELEMENTS ---
const video = document.getElementById('webcam');
const canvasElement = document.getElementById('output_canvas');
const canvasCtx = canvasElement.getContext('2d');
const startScreen = document.getElementById('start-screen');
const startBtn = document.getElementById('start-btn');
const loaderText = document.getElementById('loader-text');
const targetWordEl = document.getElementById('target-word');
const instructionEl = document.getElementById('instruction-text');
const progressBar = document.getElementById('progress-bar');
const hudBottom = document.getElementById('hud-bottom');
const successOverlay = document.getElementById('success-overlay');
const levelIndicator = document.getElementById('level-indicator');
// --- STATE ---
let handLandmarker = undefined;
let webcamRunning = false;
let lastVideoTime = -1;
let currentLevelIdx = 0;
let currentConfidence = 0;
let targetConfidence = 0;
let isLevelLocked = false; // Prevents detection during success animation
// --- LEVELS CONFIG ---
const levels = [
{
word: "HELLO",
instruction: "Open your hand and wave (All 5 fingers up)",
validate: (landmarks) => {
return areFingersUp(landmarks, [1, 1, 1, 1]) && isThumbOpen(landmarks);
}
},
{
word: "GOOD",
instruction: "Thumbs Up (Fist with Thumb extended)",
validate: (landmarks) => {
return areFingersUp(landmarks, [0, 0, 0, 0]) && isThumbOpen(landmarks);
}
},
{
word: "I LOVE YOU",
instruction: "Extend Thumb, Index, and Pinky",
validate: (landmarks) => {
// Index & Pinky UP, Middle & Ring DOWN, Thumb OPEN
return areFingersUp(landmarks, [1, 0, 0, 1]) && isThumbOpen(landmarks);
}
},
{
word: "CALL",
instruction: "Phone gesture (Thumb & Pinky extended)",
validate: (landmarks) => {
return areFingersUp(landmarks, [0, 0, 0, 1]) && isThumbOpen(landmarks);
}
},
{
word: "OK",
instruction: "Touch Thumb and Index tips. Other fingers up.",
validate: (landmarks) => {
const dist = getDist(landmarks[4], landmarks[8]); // Thumb tip to Index tip
// Middle, Ring, Pinky should be UP
return dist < 0.08 && areFingersUp(landmarks, [0, 1, 1, 1]); // Pattern ignores index check here
}
},
{
word: "LITTLE BIT",
instruction: "Pinch Thumb and Index close. Others DOWN.",
validate: (landmarks) => {
const dist = getDist(landmarks[4], landmarks[8]);
return dist < 0.1 && areFingersUp(landmarks, [0, 0, 0, 0]);
}
},
{
word: "PEACE",
instruction: "Index and Middle fingers up (V-sign)",
validate: (landmarks) => {
return areFingersUp(landmarks, [1, 1, 0, 0]);
}
},
{
word: "STOP",
instruction: "Raise flat hand (All fingers up)",
validate: (landmarks) => {
// Geometry is same as Hello, but context differs
return areFingersUp(landmarks, [1, 1, 1, 1]) && isThumbOpen(landmarks);
}
},
{
word: "POINT",
instruction: "Point with Index finger. Others closed.",
validate: (landmarks) => {
return areFingersUp(landmarks, [1, 0, 0, 0]) && !isThumbOpen(landmarks);
}
}
];
// --- INITIALIZATION ---
async function createHandLandmarker() {
try {
const vision = await FilesetResolver.forVisionTasks(
"https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.0/wasm"
);
handLandmarker = await HandLandmarker.createFromOptions(vision, {
baseOptions: {
modelAssetPath: `https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task`,
delegate: "GPU"
},
runningMode: "VIDEO",
numHands: 1 // Single hand focus
});
loaderText.textContent = "System Ready";
startBtn.classList.add('ready');
startBtn.addEventListener('click', enableCam);
} catch (e) {
loaderText.textContent = "Error loading models: " + e.message;
console.error(e);
}
}
function enableCam() {
if (!handLandmarker) {
alert("Please wait for model to load");
return;
}
webcamRunning = true;
startScreen.style.display = 'none';
// Setup Camera
const constraints = { video: { width: 1280, height: 720 } };
navigator.mediaDevices.getUserMedia(constraints).then((stream) => {
video.srcObject = stream;
video.addEventListener("loadeddata", predictWebcam);
loadLevel(0);
});
}
createHandLandmarker();
// --- GAME LOOP ---
async function predictWebcam() {
// Resize canvas to match video
canvasElement.style.width = video.videoWidth;
canvasElement.style.height = video.videoHeight;
canvasElement.width = video.videoWidth;
canvasElement.height = video.videoHeight;
if (isLevelLocked) {
// Keep rendering video but pause detection logic updates
// Just request next frame
window.requestAnimationFrame(predictWebcam);
return;
}
let startTimeMs = performance.now();
if (lastVideoTime !== video.currentTime) {
lastVideoTime = video.currentTime;
const results = handLandmarker.detectForVideo(video, startTimeMs);
canvasCtx.save();
canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
// The canvas is mirrored via CSS, but drawing uses raw coords.
// We must draw naturally (MediaPipe returns normalized 0-1 relative to image).
if (results.landmarks && results.landmarks.length > 0) {
const landmarks = results.landmarks[0]; // Get first hand
// 1. Draw Skeleton
drawHandSkeleton(canvasCtx, landmarks);
// 2. Validate Gesture
const passed = levels[currentLevelIdx].validate(landmarks);
// 3. Update Logic
updateConfidence(passed);
} else {
updateConfidence(false); // Decay if no hand seen
}
canvasCtx.restore();
}
window.requestAnimationFrame(predictWebcam);
}
// --- LOGIC & HELPERS ---
function updateConfidence(passed) {
// Calculate Target
if (passed) {
targetConfidence = Math.min(targetConfidence + CONFIDENCE_INCREASE, 100);
} else {
targetConfidence = Math.max(targetConfidence - CONFIDENCE_DECREASE, 0);
}
// Smooth Interpolation for Visuals
currentConfidence += (targetConfidence - currentConfidence) * 0.1;
// Render UI
renderUI();
// Check Success
if (currentConfidence > 98 && !isLevelLocked) {
triggerSuccess();
}
}
function renderUI() {
progressBar.style.width = `${currentConfidence}%`;
if (currentConfidence > 90) {
progressBar.classList.add('complete');
hudBottom.classList.add('success');
} else {
progressBar.classList.remove('complete');
hudBottom.classList.remove('success');
}
}
function triggerSuccess() {
isLevelLocked = true;
successOverlay.classList.add('active');
// Play sound (optional) or just visual
setTimeout(() => {
successOverlay.classList.remove('active');
nextLevel();
}, SUCCESS_DELAY_MS);
}
function nextLevel() {
currentLevelIdx++;
if (currentLevelIdx >= levels.length) {
// Game Over / Loop
currentLevelIdx = 0;
alert("Congratulations! You completed all words. Restarting...");
}
loadLevel(currentLevelIdx);
targetConfidence = 0;
currentConfidence = 0;
isLevelLocked = false;
// Re-trigger loop explicitly if needed, though RAF usually handles it
window.requestAnimationFrame(predictWebcam);
}
function loadLevel(idx) {
const lvl = levels[idx];
targetWordEl.innerText = lvl.word;
instructionEl.innerText = lvl.instruction;
levelIndicator.innerText = `${idx + 1} / ${levels.length}`;
progressBar.style.width = "0%";
hudBottom.classList.remove('success');
}
// --- GEOMETRY ENGINE ---
// Calculate Euclidean distance between two landmarks (normalized 0-1)
function getDist(p1, p2) {
return Math.sqrt(Math.pow(p1.x - p2.x, 2) + Math.pow(p1.y - p2.y, 2));
}
// Check if a specific finger is "Up" (Extended)
// patternIndex: 0=Index, 1=Middle, 2=Ring, 3=Pinky
function areFingersUp(landmarks, pattern) {
// Tips: 8(Index), 12(Middle), 16(Ring), 20(Pinky)
// PIPs (Joints): 6, 10, 14, 18
const tips = [8, 12, 16, 20];
const pips = [6, 10, 14, 18];
let match = true;
for (let i = 0; i < 4; i++) {
const isUp = landmarks[tips[i]].y < landmarks[pips[i]].y; // Y increases downward
// pattern[i] is 1 for UP, 0 for DOWN
if ((pattern[i] === 1 && !isUp) || (pattern[i] === 0 && isUp)) {
match = false;
break;
}
}
return match;
}
// Check if thumb is "Open" (Extended away from palm)
// Simple heuristic: Distance from ThumbTip(4) to IndexMCP(5)
// Compared to a threshold or relative to hand size
function isThumbOpen(landmarks) {
// Helper: Compare distance of Thumb Tip to Index Base vs Thumb IP to Index Base
// Or simple distance check in X axis relative to Wrist.
// Let's use distance from Tip(4) to IndexMCP(5) vs IP(3) to IndexMCP(5)
const tipToBase = getDist(landmarks[4], landmarks[5]);
const ipToBase = getDist(landmarks[3], landmarks[5]);
// Also check X distance relative to wrist for "outward" extension
// But the prompt asks specifically for "distance of thumb tip to palm center"
// Using IndexMCP(5) as proxy for palm center area.
// Threshold usually around 0.05 in normalized space for "close"
return tipToBase > 0.1; // If distance is large, it's open
}
// --- DRAWING UTILS ---
function drawHandSkeleton(ctx, landmarks) {
const w = ctx.canvas.width;
const h = ctx.canvas.height;
ctx.lineWidth = 4;
ctx.lineCap = "round";
ctx.strokeStyle = "#fbbf24"; // Amber connectors
// Connectors Definition
const connections = [
[0,1], [1,2], [2,3], [3,4], // Thumb
[0,5], [5,6], [6,7], [7,8], // Index
[5,9], [9,10], [10,11], [11,12], // Middle
[9,13], [13,14], [14,15], [15,16], // Ring
[13,17], [17,18], [18,19], [19,20], // Pinky
[0,17] // Wrist to Pinky Base
];
// Draw Lines
connections.forEach(pair => {
const p1 = landmarks[pair[0]];
const p2 = landmarks[pair[1]];
ctx.beginPath();
ctx.moveTo(p1.x * w, p1.y * h);
ctx.lineTo(p2.x * w, p2.y * h);
ctx.stroke();
});
// Draw Landmarks (Nodes)
ctx.fillStyle = "#ffffff";
landmarks.forEach(p => {
ctx.beginPath();
ctx.arc(p.x * w, p.y * h, 5, 0, 2 * Math.PI);
ctx.fill();
});
}
</script>
</body>
</html>