User Guide

Everything you can do once mHumanTrackAI is running.

Accessibility Mode

Accessibility Mode maps hand gestures to real mouse and keyboard input via Win32 SendInput. It includes dwell-click (hover-to-click with a configurable dwell duration) and an adjustable cursor sensitivity multiplier.

Gesture	Detection	Action
Point + move	Index tip position	Move cursor (relative, trackpad-style)
Pinch tap (thumb + index)	Brief contact	Left click
Pinch hold (thumb + index, >300 ms)	Sustained	Left-hold → drag
Thumb + middle	Brief contact	Right click
Thumb + ring	Brief contact	Middle click
Thumb + pinky	Brief contact	Double-click
Index + middle vertical move	Both extended	Scroll
Fist	All curled	Freeze cursor
Both open palms held 2 s	Safety pose	Toggle tracking (panic)

Finger pairing is fixed in code

The gesture editor can toggle mappings on/off and change mode/hand, but which finger pair fires which click is currently fixed in code to preserve the tuned feel. Full finger-pair remapping is deferred.

Gaming Mode

Gaming Mode emits a virtual Xbox 360 controller via ViGEm. Body movement and hand gestures are mapped to controller axes and buttons.

Body input	Controller output
Right pinch (thumb + index)	RT — primary fire
Left pinch (thumb + index)	LT — secondary / aim
Right fist	RB — grab / hold
Left fist	LB — grab / hold
Either hand raised above head	A — jump (edge pulse)
Torso compression (crouch)	B — crouch (held)
Body lean left / right	Left stick X — strafe
Head pan left / right	Right stick X — camera look

Anti-cheat

A ViGEm virtual controller is accepted far more widely than synthetic mouse/keyboard input, but kernel-level anti-cheat can still reject synthetic devices. See Anti-cheat blocks input.

Cameras

The Cameras view lists all connected DirectShow webcams by their friendly name (not just an index). Per camera you can configure:

Resolution and target FPS
FOURCC — use MJPG for the 60 fps fast path
Backend and mirror toggle
ROI (region of interest) and enable/disable

mHumanTrackAI supports multiple cameras. The headline multi-camera case is front + back fusion: a per-joint best-view selector picks whichever camera actually sees each joint, giving full-body occlusion coverage (for example, a hand behind your back).

Body Tracking

Toggle which body parts are tracked. Per-part toggles cover: head, neck, torso, left arm, right arm, left hand (with per-finger thumb / index / middle / ring / pinky toggles), right hand (same), hips, left leg, right leg, left foot, right foot.

Disabled parts are greyed in the skeleton render and excluded from gesture evaluation.
Occlusion prediction (One-Euro filter + constant-velocity Kalman + IK extrapolation) fills in joints no camera can see; predicted joints are rendered dashed/ghosted.

Gestures

The gesture engine is driven by an editable gesture → action table. You can add, remove and edit rows live.

Gesture types: Pinch, Pointer, TwoFingerMove, Fist, Pose, Swipe, Custom
Action kinds: Cursor, Mouse, Key, Scroll, Controller, None
Per row: enabled flag, mode (Accessibility / Gaming / Both), hand (Left / Right / Either / Both), fingers, hold threshold (ms), debounce (ms)

Press Apply to persist changes to the active profile and hot-reload the engine — no restart required.

Recording & Playback

Record the live landmark stream (pose + hands + face, every frame) to a named session file under recordings/.
Play back a saved recording through the full engine pipeline (gestures and input included) — useful for repeatability testing with no camera attached.
Export to CSV (a flat landmark table per frame) or BVH (motion-capture format for animation tools).

The list view shows each recording's name, frame count, duration (s), and file size (KB).

Settings & Profiles

Profiles — load, save-as-new, and delete named profiles. Each profile is a JSON file in profiles/. See the Configuration reference.
Tracking engine — switch between RTM (RTMW whole-body, GPU/CPU) and MediaPipe (CPU fallback) live.
Device — auto, cuda, or cpu. auto picks CUDA when a supported NVIDIA GPU is detected, otherwise CPU.
RTM model tier — balanced (256×192), performance (384×288), or fast.
Additional technologies (NVIDIA GPU only) — an in-app download manager for optional GPU backends (CUDA + cuDNN runtime, ONNX Runtime CUDA, TensorRT). Each download is SHA-256 verified.
Smoothing — One-Euro filter parameters (min-cutoff, beta, d-cutoff).
Engine restart — kills and relaunches the Python engine in place.

Component URLs are placeholders

In the shipped components.json, every download URL is a placeholder (https://REPLACE-ME). Until they are filled in, the Additional technologies window reports that a component's URL is still a placeholder when you try to install it.