QtMultimedia for WebAssembly
QtMultimedia for WebAssembly adds the ability for Qt based apps to use audio, camera and play videos in a web app. Deploy once, run everywhere that has a web browser that supports javascript and WebAssembly.
There have been some improvements in QtMultimedia for WebAssembly lately. Specifically for the Qt 6.12 release and beyond. Probably the biggest one is that it now supports using WebGL for zero-copy VideoFrames.
QML and QSurface::OpenGLSurface surfaces which use webgl canvas behind the scene, can play video and display a camera in a Qt for WebAssembly web app. While QSurface::RasterSurface surfaces like QtWidget will use the previous path with the 2d context canvas that requires the use of copyTo to get the video frame out of the javascript side and into Qt. (Thanks Emscripten!)
There have been improvements in displaying two or more video/cameras in the same app, and ease of switching devices.
This video shows the dual camera capabilities in a QML app in a web browser.
QML app showing a live overlay, showing the same camera stream multiple times:
Next new functionality is QAudioSink and QAudioSource on WebAssembly now use WebAudio Audioworklets API instead of the OpenAL (Open Audio Library) backend that was the first community contribution for webassembly which got this whole wasm ball rolling for QtMultimedia. The new code is more performant, doesn't cause Emscripten (compiler) to use depreciated Javascript API, and allows use of different audio devices, mix and match audio inputs when recording the camera.
Requesting QCameraFormat (video resolution, minFrameRate, maxFrameRate) is now supported, with a caveat - the browser has the final say on camera format.
One nice thing I might as well mention while I am here, is that QtMultimedia for WebAssembly can make use of mobile wireless cameras - use your phone as camera input! As well as seemlessly switch between cameras. This isn't new functionality, I just like pointing it out.
Plus various other fixes and improvements not mentioned here.
Things from the Workbench:
Functionality that may or may not be coming for the next 6.13 release.
audio decoding
spatialaudio demo
I whipped up this spatial audio demo that can split the audio channels into their own 3d space and orbits. It's here in the audio decoding section, but decoding was recently decoupled from spatial audio in QtMultimedia.
All of these demos are running in a browser, of course, using Qt for WebAssembly.
Items still yet to be worked on which again, may or may not be coming, so I cannot demonstrate them here:
- audiobufferoutput
- videoframe input
- audioformat
- videoformat webcodec (challenging due to no mux/demux)
Hopefully at some point soon, WebAssembly can be a fully supported platform for QtMultimedia. As of 6.12, it is still in the 'tech preview' beta stage. At this point, there is only manual testing for singlethreaded, multithreaded and asyncify/JSPI versions of Qt for WebAssembly.
