Getting a free (as in "it comes with the laptop") IR camera.

My friend

Yesterday my friend arelli asked me if I had ever experimented with the IR camera of my laptop. It just so happens we both have similar models, mine is a Asus Vivobook S16 M5606WA. Most recent laptops have a camera with a bandpass [citation needed] IR filter that’s most commonly used as authentication with Windows hello. That got me curious how to access it from linux.

Exploring

Let’s see if it is directly accessible. Using firefox to access the webcam gives me two options:

1. USB2.0 FHD UVC WebCam: USB2.0 F
2. USB2.0 FHD UVC WebCam: USB2.0 I

that appear to be the same (normal camera) stream so it won’t be that simple :D.

Maybe it is exposed as another webcam! Running lshw, the only webcam or video input device seems to be this:

*-usb
    description: Video
    product: USB2.0 FHD UVC WebCam
    vendor: Shinetech
    physical id: 1
    bus info: usb@1:1
    version: 1.01
    serial: 200901010001
    capabilities: usb-2.01
    configuration: driver=uvcvideo maxpower=500mA speed=480Mbit/s

and all other USB devices seemed to be irrelevant. Hmm…

Another way of looking around for video devices is v4l.

$ sudo v4l2-ctl --list-devices
USB2.0 FHD UVC WebCam: USB2.0 F (usb-0000:63:00.4-1):
	/dev/video0
	/dev/video1
	/dev/video2
	/dev/video3
	/dev/media0
	/dev/media1

This lists the device we saw earlier but thats a lot of paths. Let’s dive deeper.

$ v4l2-ctl -d /dev/video0 --all
Driver Info:
	Driver name      : uvcvideo
	Card type        : USB2.0 FHD UVC WebCam: USB2.0 F
...
Video input : 0 (Camera 1: ok)
Format Video Capture:
	Width/Height      : 1280/720
	Pixel Format      : 'MJPG' (Motion-JPEG)
	Field             : None
	Bytes per Line    : 0
	Size Image        : 1843200
	Colorspace        : sRGB
	Transfer Function : Rec. 709
	YCbCr/HSV Encoding: ITU-R 601
	Quantization      : Default (maps to Full Range)
...
  Frames per second: 30.000 (30/1)
...

$ v4l2-ctl -d /dev/video2 --all
Driver Info:
	Driver name      : uvcvideo
	Card type        : USB2.0 FHD UVC WebCam: USB2.0 I
...
Video input : 0 (Camera 9: ok)
Format Video Capture:
	Width/Height      : 640/400
	Pixel Format      : 'GREY' (8-bit Greyscale)
	Field             : None
	Bytes per Line    : 640
	Size Image        : 256000
	Colorspace        : sRGB
	Transfer Function : Rec. 709
	YCbCr/HSV Encoding: ITU-R 601
	Quantization      : Default (maps to Full Range)
...
	Frames per second: 15.000 (15/1)
...

Hmmm… that’s quite interesting. /dev/video0 provides an Motion-JPEG but /dev/video1 an 8-bit Greyscale stream of low resolution (640/400) and very low framerate (15). That must be it!

Streaming

The simplest way to stream a video device is using ffplay /dev/video2. It works! Raving

But something is up… Everything is flashing! The IR led is visible and I can see it blinking at about 5 fps give or take while the camera is enabled. This is not usable. It’s also heating up after a minute of usage. If we could control it maybe we could disable it, enable it permanently or even PWM it!

Rave strobe lights

Dealing with the issues

Unfortunately I didn’t find a way of controlling it :(. The firmware doesn’t expose any controls:

$ v4l2-ctl -d /dev/video2 --list-ctrls 

User Controls

   region_of_interest_rectangle 0x00981ae1 (rect)   : value=(0,0)/1x1 flags=has-payload, has-min-max
  region_of_interest_auto_ctrls 0x00981ae2 (bitmask): max=0x00000001 default=0x00000001 value=0 flags=has-min-max

Camera Controls

                        privacy 0x009a0910 (bool)   : default=0 value=0 flags=read-only

and no other fps or resolution options are available.

$ v4l2-ctl -d /dev/video2 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'GREY' (8-bit Greyscale)
		Size: Discrete 640x400
			Interval: Discrete 0.067s (15.000 fps)

But going through the output file, I noticed that every second frame is unlit, that means that the LED is flickering at 7.5Hz and lighting the odd frames. I am not sure why they do that, maybe a calibration thing to remove the effect of other IR sources and keep only the one facing the face. Using ffplay /dev/video2 -vf "select=not(mod(n\,2))" Keeping only the odd frames and the lighting looks very consistent albeit the framerate halving.

The party is over

Now the stream is very usable. I am not sure why it is not exposed properly on firefox, maybe firefox under the hood recognises IR streams and falls back to the normal stream for “better” UX even though the media looks properly configured.

$ media-ctl -d /dev/media1 -p
...
- entity 1: USB2.0 FHD UVC WebCam: USB2.0 I (1 pad, 1 link)
            type Node subtype V4L flags 1
            device node name /dev/video2
	pad0: SINK
		<- "Extension 14":1 [ENABLED,IMMUTABLE]
...

It can be manually exported as a webcam though by:

$ sudo modprobe v4l2loopback devices=1 video_nr=10 card_label="IR camera" exclusive_caps=1
$ ffmpeg -f v4l2 -i /dev/video2 -vf "format=yuv420p,select=not(mod(n\,2))" -f v4l2 /dev/video10

Have fun!

Here you can see me turning on the lights and being visible on the IR camera while they are off. The water and the glass pane don’t let as much IR light to pass through. My fleece jacket seems to reflect a lot of IR, decathlon keeps me warm and cozy :}.