Debugging Qt WebAssembly - DWARF

 

One of the most tedious tasks a developer will do is debugging a nagging bug. It's worse when it's a web app, and even worse when its a webassembly web app.

The easiest way to debug Qt Webassembly is by configuring using the -g argument, or CMAKE_BUILD_TYPE=Debug . Emscripten embeds DWARF symbols in the wasm binaries. 

NOTE: Debugging wasm files with DWARF works only in the Chrome browser with the help of a browser extension. 

C/C++ DevTools Support (DWARF) browser extension. If you are using Safari or Firefox, or do not want to or cannot install a browser extension, you will need to generate source maps, which I will look at in my next blog post. 

DWARF debugging

You need to also enable DWARF in the browser developer tools settings, but you do not need symlinks to the source directories, as you would need to using source maps, as the binaries are embedded with the full directory path. Like magic!

 Emscripten embeds DWARF symbols into the binaries built with -g by default, so re-building Qt or your application in debug mode is all you need to do.

Qt builds debug libraries by default using the optimized argument -g2, which produces less debugging info, but results in faster link times. To preserve debug symbols you need to build Qt debug using the -g or -g3 argument. Both of these do the same thing.

Using DWARF debugger

Open Chome with the extention mentioned before installed, and open the console tools. Navigate to the Qt for WebAssembly web application you need to debug. Once it opens, it may take a few seconds for all the symbols and files to get parsed. If you are debugging into Qt, this will take quite a few seconds - just keep waiting. 

The javascript console will soon contain file source file paths and sources. You can find your file to debug, and set breakpoints. Just reload the page and once it hits a breakpoint, it will stop execution and highlight the current line in the source view.  It also will show variable names and values.

You can then step though your code as you would debugging a desktop application.

Qt6 Webassembly: QtMultimedia or, How to play a video in a web browser using Qt

QtMultimedia in WebAssembly

Since Qt 6.5.0, QtMultimedia has support for playing video in a QGraphicsVideoItem and QGraphicsScene, as well as recording from a camera. You can use Qt to play video, access the camera from a web browser, thus simplifying deployment.

With anything WebAssembly, there are a few things that work differently from the desktop applications. The javascript function that is used to gather the list of devices,  getUserMedia is asyncronous, returning a javascript Promise, so the list is not readily available. This function can take some amount of time, depending on device and browser. I have found that sometimes, it is inexplicitly long. 

You can connect to the QMediaDevices::audioInputsChanged, QMediaDevices::videoInputsChanged and QMediaDevices::audioOutputsChanged signals to discover when they are available.

QPermission

Browser permissions are required for accessing the microphone and camera, and a secure https context is a prerequisite for granting them. If this is not run or permission is not granted by the user, the list of available devices will be missing the label, or name, of any device not given permission and it will be unavailable for use.

Using the new QPermission API which will be introduced in Qt 6.5 will make sure your app is permissions ready. The example QtMultimedia apps have been changed in Qt 6.6 to use this API.

You do not need to use QPermissions in wasm, because it uses the same function - getUserMedia which causes the browser to ask for permissions.

See https://developer.mozilla.org/en-US/docs/Web/API/Permissions_API

Camera

The camera (and microphone) need user permissions. Something like this will work for granting permissions to use the camera:

    QCameraPermission cameraPermission;

    switch (qApp->checkPermission(cameraPermission)) {

    case Qt::PermissionStatus::Undetermined:

        qApp->requestPermission(cameraPermission, this, &MainWindow::init);

        return;

    case Qt::PermissionStatus::Denied:

        qWarning("Camera permission is not granted!");

        return;

    case Qt::PermissionStatus::Granted:

        break;

    }

What this can look like in the browser:

When you connect to the signal QMediaDevices::videoInputChanged, it will be emitted when the list of camera devices are available. 

   connect(m_mediaDevices, &QMediaDevices::videoInputsChanged, [this]() { doCamera();});

Here is a screenshot of the webcam showing in a test app I used for developing this feature. The sources for this are in qtmultimedia/tests/manual/wasm/camera

You can try it out for yourself! camera-test

(Chrome 114 seems to have issue with VideoFrame, working with Chrome 117) 

Video

Using Javascript's WebCodecs API, we are be able to leverage Javascript's VideoFrame API, which is currently available on Chrome and Safari based browsers.

Permissions are not needed to play video.

Video currently only works with QWidget based apps. Playing video in quick apps will be fixed in an upcoming version of Qt. Here's a screenshot of the test app I used to develop it:

Video recording also works and will save the file to the browsers local filesystem, which has a limited capacity. It can then be played as normal or downloaded to the system filesystem via QFileDialog:SaveFileContent

Known Issues

• List of media devices is not readily available for application.
• Video does not show in QtQuick declarative apps

Qt for WebAssembly on mobile devices

Qt for WebAssembly on mobile devices, specifically phones, has lacked an essential feature - support for the native keyboard. It may or may not have worked. If it worked, it did not work very well. The tricky issue is opening the keyboard when needed and closing when it wasn't. There is no simple API for doing that on any platform we target - iOS, Android and Windows.

Although it feels a bit hacky to me to open the keyboard using javascript , we use Emscripten's C++ interface to create a hidden javascript text input element and set focus to that, which opens the platform keyboard.

Emscripten on Android had one other issue - the usual Emscripten input event API was not working. Nothing being typed on the native virtual keyboard was being handled like on other platforms. I found I could utilize the hidden input element that is used to pop up the keyboard, to listen for input characters and then send them to Qt. 

As it is, this patch adds support for native mobile keyboard for iOS, Android and Windows on Qt for WebAssembly. (now merged into dev as 66a76a5def46d0e4a330f7130ad440c639b87cf7), too late to make it into 6.3.

Other issues on mobile are memory (as usual). Up until recently, Safari limits browser memory to much less than the other browsers. 

Other areas that Qt covers for mobile devices that do not work on Qt for WebAssembly yet are sensors and bluetooth connectivity. Although there is a patch for some sensors support, it has not been merged and probably needs updating. It may not work at all, either.

Bluetooth connectivity for javascript is currently only supported on the Chrome browser and is currently experimental. While I could probably add this support to Qt WebAssembly, it would not get merged and the API might be too changey.

Any areas you find Qt WebAssembly is lacking on mobile, please report to https://bugreports.qt.io/

Qt WebAssembly clipboard

Clipboard use on the desktop platforms is ubiquitous. Most people use it without thinking. Copy, Paste and Cut keyboard strokes are in-grained into muscle memory. 

On the web it can present security issues as someone could read or write to your clipboard without you knowing.

Up until now, Qt for WebAssembly's clipboard was text only and only within the app itself. Qt 6.3 will have better clipboard support between host and app but also adds copy/pasting of images.

WebAssembly is a sandboxed platform like javascript. There are some extra security hurdles in doing some common things such as copy and paste of binary data such as images. One issue is clipboard use between the host platform and the browser sandbox. Allowing the web app to have access to the clipboard in which it could send arbitrary data without the user knowing could be dangerous for the user.

Browsers generally allow clipboard during user generated events such as when a user makes common key sequences such as [ctrl | command] c - the copy keys.

Qt itself has support for programmatically copying text and binary data such as images and works great on the desktop, but which presents issues for web browsers. There are work-arounds, like using a hidden javascript element and the javascript function execCommand to "copy". However, this function has been depreciated. 

By using the asynchronous Clipboard API and making use of javascript clipboard events where possible, we can bring image clipboard support to Qt WebAssembly. The Clipboard API requires a secure context (https) for full feature use. Among other things, the Clipboard API allows image and arbitrary data to be copied and pasted. Whereas before, only text mime types were supported. 

This API is of course implemented in the different browsers in different ways. On Firefox, read() and write() are only partially implemented and is hidden behind about:config settings. As well, copy/paste of arbitrary binary data does not seem to be supported and mostly silently fails.

Here are the ways browsers support Clipboard API:

Firefox 

• write() is available without permission in secure contexts and browser extensions, but only from user-initiated event callbacks.
• Clipboard API
• secure context (https or localhost)
• dom.events.asyncClipboard
• dom.events.asyncClipboard.clipboardItem
• dom.events.asyncClipboard.read
• dom.events.testing.asyncClipboard

Safari

• Clipboard API
• secure context    

Chrome

• Clipboard API
• secure context
• user permissions
• read() only supports
• text/html
• text/png

Also included in the now merged commit f0be152896471aa392bb1b2b649b66feb31480cc  is a clipboard manual test app that can be used on both desktop and webassembly to test clipbpoard use.

You can use the clipboard without a secure https context, but you won't get interaction between host and web app.

Qt Multimedia has a new friend...

 QtMultimedia just got a new platform to run on - Qt for WebAssembly!

I am happy to announce a new platform contribution for Qt Multimedia.

As of change https://codereview.qt-project.org/c/qt/qtmultimedia/+/348150 and very much thanks to Qt contributor, Raffaele Pertile, we now have audio play and record on the WebAssembly platform for Qt 6.2.

This is possible  as Emscripten has built-in support for OpenAL, which this patch uses to push audio data around. So now, all your Qt webAssembly apps can have sound effects! \0/

 

 

  

Qt WebAssembly: prompt on exit

 I sometimes get asked if there is a way to let a Qt app ask the user if they really want to close the app when the browser tab or window gets closed by the user.

The answer is yes and no. Yes, in that there is, and no in that it won't be your own Qt dialog/prompt.

We can use javascript in Qt webassembly apps, so the prompt will be opened by the browsers javascript. 

How?

There are two ways to run javascript. By using emscripten macro EM_ASM, or including emscripten/val.h and using straight c++.

EM_ASM:

The is the easiest way but uses a macro.

#include <emscripten.h>

Just add something like this, in your app constructor function:

    EM_ASM(

           window.onbeforeunload = function (e) {

                e = e || window.event;

                return 'Sure';

            };

    );

C++:

This is a bit more complicated, and involves a global static function.

#include <emscripten/bind.h>

#include <emscripten/val.h>

#include <emscripten.h>

static emscripten::val mybrowserBeforeUnload(emscripten::val e)
{
   return emscripten::val("Sure");
}


EMSCRIPTEN_BINDINGS(app) // must be unique name!
{
    function("myBrowserBeforeUnload", &mybrowserBeforeUnload);
}

and in the c'tor:

 emscripten::val::global("window").set("onbeforeunload", emscripten::val::module_property("myBrowserBeforeUnload"));


The key is to return a string. Any string will do.
and it looks something like this:

(redux) Qt WebAssembly performance enhancement

 In my last post Qt WebAssembly performance enhancement

there were some impressive performance stat speedups. Unfortunately, as my collegue Morten pointed out, both builds were in debug mode.  *sigh*

So I rebuilt them in release mode, and added a few selected benchmarks from the Qt tests/benchmark source directory:

• tst_affectors
• tst_emission
• tst_QGraphicsScene
• tst_QGraphicsView
• tst_QGraphicsWidget
• tst_qanimation
• tst_QMatrix4x4
• BlendBench
• tst_QImageConversion
• tst_DrawTexture
• tst_QPainter

Although not as impressive overall, there is still quite a speed up in the image conversions and QPainter areas, for example:

non-simd:

PASS   : tst_QPainter::drawPixmap(BGR30 on RGB32, (1000x1000), circle)

RESULT : tst_QPainter::drawPixmap():"BGR30 on RGB32, (1000x1000), circle":

     2.3 msecs per iteration (total: 76, iterations: 32)

PASS   : tst_QPainter::drawPixmap(BGR30 on RGB32, (1000x1000), line)

RESULT : tst_QPainter::drawPixmap():"BGR30 on RGB32, (1000x1000), line":

     2.4 msecs per iteration (total: 77, iterations: 32)

PASS   : tst_QPainter::drawPixmap(BGR30 on RGB32, (1000x1000), solidrect)

RESULT : tst_QPainter::drawPixmap():"BGR30 on RGB32, (1000x1000), solidrect":

     2.4 msecs per iteration (total: 78, iterations: 32)

PASS   : tst_QPainter::drawPixmap(BGR30 on RGB32, (1000x1000), alpharect)

RESULT : tst_QPainter::drawPixmap():"BGR30 on RGB32, (1000x1000), alpharect":

     2.4 msecs per iteration (total: 78, iterations: 32)

simd:

RESULT : tst_QPainter::drawPixmap():"BGR30 on RGB32, (1000x1000), circle":

     0.95 msecs per iteration (total: 61, iterations: 64)

PASS   : tst_QPainter::drawPixmap(BGR30 on RGB32, (1000x1000), line)

RESULT : tst_QPainter::drawPixmap():"BGR30 on RGB32, (1000x1000), line":

     0.95 msecs per iteration (total: 61, iterations: 64)

PASS   : tst_QPainter::drawPixmap(BGR30 on RGB32, (1000x1000), solidrect)

RESULT : tst_QPainter::drawPixmap():"BGR30 on RGB32, (1000x1000), solidrect":

     0.92 msecs per iteration (total: 59, iterations: 64)

PASS   : tst_QPainter::drawPixmap(BGR30 on RGB32, (1000x1000), alpharect)

RESULT : tst_QPainter::drawPixmap():"BGR30 on RGB32, (1000x1000), alpharect":

     0.95 msecs per iteration (total: 61, iterations: 64

non-simd:

PASS   : tst_QPainter::drawPixmap(ARGB32_pm on RGB32, (1000x1000), circle)

RESULT : tst_QPainter::drawPixmap():"ARGB32_pm on RGB32, (1000x1000), circle":

     1.7 msecs per iteration (total: 56, iterations: 32)

PASS   : tst_QPainter::drawPixmap(ARGB32_pm on RGB32, (1000x1000), line)

RESULT : tst_QPainter::drawPixmap():"ARGB32_pm on RGB32, (1000x1000), line":

     1.7 msecs per iteration (total: 55, iterations: 32)

PASS   : tst_QPainter::drawPixmap(ARGB32_pm on RGB32, (1000x1000), solidrect)

RESULT : tst_QPainter::drawPixmap():"ARGB32_pm on RGB32, (1000x1000), solidrect":

     1.7 msecs per iteration (total: 55, iterations: 32)

PASS   : tst_QPainter::drawPixmap(ARGB32_pm on RGB32, (1000x1000), alpharect)

RESULT : tst_QPainter::drawPixmap():"ARGB32_pm on RGB32, (1000x1000), alpharect":

     3.6 msecs per iteration (total: 58, iterations: 16)

simd:

PASS   : tst_QPainter::drawPixmap(ARGB32_pm on RGB32, (1000x1000), circle)

RESULT : tst_QPainter::drawPixmap():"ARGB32_pm on RGB32, (1000x1000), circle":

     2.6 msecs per iteration (total: 85, iterations: 32)

PASS   : tst_QPainter::drawPixmap(ARGB32_pm on RGB32, (1000x1000), line)

RESULT : tst_QPainter::drawPixmap():"ARGB32_pm on RGB32, (1000x1000), line":

     4.0 msecs per iteration (total: 64, iterations: 16)

PASS   : tst_QPainter::drawPixmap(ARGB32_pm on RGB32, (1000x1000), solidrect)

RESULT : tst_QPainter::drawPixmap():"ARGB32_pm on RGB32, (1000x1000), solidrect":

     2.2 msecs per iteration (total: 71, iterations: 32)

PASS   : tst_QPainter::drawPixmap(ARGB32_pm on RGB32, (1000x1000), alpharect)

RESULT : tst_QPainter::drawPixmap():"ARGB32_pm on RGB32, (1000x1000), alpharect":

     4.5 msecs per iteration (total: 73, iterations: 16)

and image conversions:

non-simd:

PASS   : tst_QImageConversion::convertGenericInplace(argb32 -> argb32pm -> argb32)

RESULT : tst_QImageConversion::convertGenericInplace():"argb32 -> argb32pm -> argb32":

     6.1 msecs per iteration (total: 98, iterations: 16)

PASS   : tst_QImageConversion::convertGenericInplace(argb32 -> rgb32 -> argb32)

RESULT : tst_QImageConversion::convertGenericInplace():"argb32 -> rgb32 -> argb32":

     2.9 msecs per iteration (total: 94, iterations: 32)

PASS   : tst_QImageConversion::convertGenericInplace(argb32 -> rgba8888 -> argb32)

RESULT : tst_QImageConversion::convertGenericInplace():"argb32 -> rgba8888 -> argb32":

     4.6 msecs per iteration (total: 75, iterations: 16)

simd:

PASS   : tst_QImageConversion::convertGenericInplace(argb32 -> argb32pm -> argb32)

RESULT : tst_QImageConversion::convertGenericInplace():"argb32 -> argb32pm -> argb32":

     4.2 msecs per iteration (total: 68, iterations: 16)

PASS   : tst_QImageConversion::convertGenericInplace(argb32 -> rgb32 -> argb32)

RESULT : tst_QImageConversion::convertGenericInplace():"argb32 -> rgb32 -> argb32":

     0.49 msecs per iteration (total: 63, iterations: 128)

PASS   : tst_QImageConversion::convertGenericInplace(argb32 -> rgba8888 -> argb32)

RESULT : tst_QImageConversion::convertGenericInplace():"argb32 -> rgba8888 -> argb32":

     0.90 msecs per iteration (total: 58, iterations: 64)

But others were slower for the simd build. Probably due to emscripten not fully supporting simd instructions and emulating those where it doesn't support.

For full benchmark results get the zip file