adding face tracking to my pico 4
adding face tracking to my pico 4
and how i accidentally got into hardware development
After years of deliberation, I recently pulled the trigger and got<br>myself a second-hand VR headset. Initially, I got a Quest 2, and after<br>hopping on VRChat for a few moments, I immediately fell in love. There's<br>something charming about the experience in VR that playing it on desktop<br>mode just doesn't have.
Within the past year, I've accumulated hundreds of hours in game, took<br>nearly 3,000 photos, and made lots of new friends. Being able to sit<br>next to people and talk to them directly just feels different,<br>in a good way. I get to use my body language to express more than words<br>can.
And I love being able to do that so much to the point that I've spent<br>thousands on full-body tracking equipment. Just so I can swing my legs<br>in VR.
Cute, right? Of course! I love it so much. And if you're still reading<br>this I know you do too. But there's one thing that bothers me about the<br>experience. My avatar has such a cute face, but it's stuck in a<br>perpetual state of expressionless stare. And I know I can change it, but<br>there's only so much winking and blinking I can do with my controllers<br>while I'm trying to have a conversation with someone.
So I looked into face tracking. And my wallet trembled in fear.
the state of face tracking hardware
There's something I forgot to tell you. I actually<br>broke my Quest 2<br>a few months in, and the local repair shops took 2 weeks just to tell me<br>they couldn't repair my headset. All the while I was itching to get back<br>to getting headpats in VR. So I bought a Pico 4 as a replacement.
For $250 SGD second hand, it's a very solid piece of equipment, with<br>pancake lenses, a balanced center of gravity. But it doesn't have eye<br>tracking.
And the cheapest one that does? Probably a second hand Quest Pro that<br>goes for $800, or a Vive Pro Eye which doesn't go around much here in<br>Singapore. But with the VPE, the headset only has eye tracking, so you<br>still need to look for the discontinued face tracker from the second<br>hand market.
The choices are limited and there's not a lot of good options.
open source to the rescue?
So as it turns out there are two projects aiming to provide eye and face<br>tracking, EyeTrackVR and<br>Project Babble. Essentially, they<br>provide the software including pre-trained models that will take in a<br>video feed of your eyes and mouth, and output messages through OSC which<br>will be recognized by VRChat as facial expressions.
Both projects also give out instructions on how to build the hardware<br>that will attach to your VR headset. And so I got all the parts and<br>built one myself.
Yeah, it's not exactly the prettiest. There's wires all over the place,<br>exposed components, multiple cables going towards the headset. It's not<br>something I want to use every day.
However, as messy as this setup is, the tracking results are pretty<br>good.
I was able to blink, move my eyes, control my ears with my frowning,<br>stick out my tongue, and many more. I was quite pleased with the<br>results, but there were a few issues with this mishmashed prototype<br>build that I want to fix.
the issues
The contraption pictured above was a prototype build, and it had a<br>couple of issues.
There were cables everywhere, and things were stuck on with Blu-Tack,<br>which makes handling the headset difficult
The 3D printed plastic mounts were not fitted correctly and the boards<br>would slide around and disconnect mid-session
I'm running two cables to my headset; one for the wired PCVR stream<br>and the other for the eye and face tracking stuffs
I've only added tracking to one eye, which means I can't wink, only<br>blink
I could go on, but those are the big issues preventing me from keeping<br>this setup for longer than was necessary to record some demo videos.
So as a curious fox, I set out to build the perfect solution that will<br>solve all of my problems.
enter: usb hub
One thing I wanted to have is a USB hub attached to my headset, with a<br>looong cable towards my PC. But all of the USB hubs I found on<br>AliExpress are either very dodgy with a short "dongle" USB cable, or<br>heavy duty ones that's too overkill for my use case.
I just wanted a cheap USB hub with a Type-C port to connect to my PC.<br>And nobody's making those that are also cheap.
So I made my own.
With zero experience, I opened KiCad and started designing something<br>resembling a USB hub, with pads for the ESP32 to mount onto. I sent it<br>out to JLCPCB, and a week later, I got my board delivered.
Of course, I plugged it in and... it didn't work! When I plugged it into<br>my PC, dmesg did show something but it kept disconnecting,<br>and it wasn't really working as a USB hub at all. Well, the power did go<br>through, but there's no data flowing.
As it turns out, you can't just draw traces from the USB ports to the<br>hub chip and from the hub chip to the destinations. There are certain<br>rules and constraints that I have to follow to maintain signal<br>integrity. And as a...