Qt for web (assembly)

So ya, I have been doing work for the up and coming Finnish company Intopalo getting Qt5 for WebAssembly using emscripten up and running. This is the 3rd Finnish company I have done work for, and the 4th Nordic company (Trolltech, Nokia, Jolla, Intopalo)

This is very much a work in progress.

Background:

There was a Qt4 version, emscripten-qt and a related Qt5 for Native Client which do similar things, and we have taken inspiration from both these projects. Thanks!

The gist of it:

We use emscripten to cross compile Qt5 into javascript and/or webassembly. We use 'incoming' from emscripten's git repo. Details how to build emsdk are here.

My working Qt5 repo is here, (the wm branch contains multi window and window decorations) while there is also an outdated WIP gerrit MR  which will get updated at some point.

This is how I configure Qt for emscripten:
 . ~/emsdk/emsdk_env.sh (to get em compiler in path)

~/depot/qt/qt5/qtbase/configure -xplatform emscripten -confirm-license -opensource -nomake tests  -nomake examples -developer-build -no-dbus -no-thread

 (no-thread is a new feature I added for this, as threading in javascript is basically non existent)

and then, use that qmake to compile a Qt app!

To run the app, in the firefox browser
emrun --browser firefox path/to/app.html

This is all bleeding edge from the emscripten compiler to the web assembly support in the browser, so your mileage may vary.

With the 'wm' branch of my github repo, multi windows and window decorations are kind of working. As well as compiling into webassembly and not just javascript asmjs.
Compiling into wasm brings much smaller download sizes, but you need to have a browser that supports it.

There is still heaps of work to do to bring it up to release state. So for now, it is still a research and WIP project, and a lot of things may or may not work. But it can run some Qt widget based apps in your (firefox) browser!

The future:

Currently there is no support for qml/qt quick. That just means we haven't tried it as our focus is currently on widgets and opengl.

Personally, I would like to see location and some sensor events, but that is for the future.

new QtSensor IoT plugins!

I've said it before and I will say it again... one thing I am tired of hearing about, is that people think Qt is only for UI applications.
WRONG!
Qt is great for IoT as well! even without a UI. (IoT is today's buzzword for 'embedded', which really isn't embedded, just small) and you cannot have IoT without sensors!

I just submitted a MR to QtSensors to add support for the SenseHAT sensor board for raspberry pi:

https://codereview.qt-project.org/#/c/193456
For raspberry pi, you will need RTIMULib installed on your rpi (and dev package to compile QtSensors). It's in the default repo, so you don't actually need to compile RTIMULib yourself.

Even though this is submitted for dev branch and should be available in Qt Sensors 5.10, this will compile and work for the Qt 5.3 that comes with the raspbian distribution.

As well, if my spare time allows, I will also get my Matrix Creator QtSensor plugin into shape to submit as well.

I also have planned getting the TI SensorTag code I wrote shoved into a QtSensor plugin. What's special about the SensorTag is that it has (buzzword alert) wireless sensors! This is a bit more tricky as it will use and be dependent on Qt's QLowEnergyService but will work nonetheless. This plugin doesn't run on the device, but requires a host machine which has bluetooth.

Movin' on...

A year has gone by since I started work with Canonical. As it turns out, I must be on my way. Where to? Not real sure at this moment, there seems plenty of companies using Qt & QML these days. \0/

But saying that, I am open to suggestions. LinkedIn
 

Plenty of IoT and devices using sensors around. Heck, even Moto Z phone has some great uses for sensor gestures similar to what I wrote for QtSensors while I was at Nokia.

But a lack of companies that allow freelance or remote work. The last few years I have worked remotely doing work for Jolla and Canonical. Both fantastic companies to work for, which really have it together for working remotely.

I am still surprised that only a handful of companies regularly allow remote work. I do not miss the stuffy non window opening offices and the long daily commute, which sometimes means riding a motorcycle through hail! (I do not suggest this for anyone)

Of course, I am still maintainer for QtSensors, QtSystemInfo for the Qt Project, and Sensor Framework for Mer, and always dreaming up new ways to use sensors. Still keeping tabs on QtNetwork bearer classes.

Although I had to send back the Canonical devices, I still have Ubuntu on my Nexus 4. I still have my Jolla phones and tablet.

That said, I still have this blog here, and besides spending my time looking for a new programming gig, I am (always) preparing to release a new album. http://llornkcor.com

and always willing to work with anyone needing music/audio/soundtrack work.

sensorfw & Qt is not just UI framework! or Without sensors, there's no IoT

Like it says on the Intel IoT developer site, "Without sensors, there's no IoT".

Because I am the maintainer of QtSensors, I like to inquire about  people's use of sensors and if they use QtSensors. Over the years, I have heard quite often something like, 'Qt is thought of as a UI framework'. *sigh*
 
But Qt is more than just a UI framework and it's use is not dependent on widgets or declarative wizardry. It is used in quite a few middleware components without UI elements. One of those middleware frameworks is Sensor Framework.

Sensor framework is a daemon that uses a plugin system written using Qt for reading various sensors such as accelerometer or light sensors. It was originally developed by Nokia for Harmattan and ran on the N9. It was also used in MeeGo and later included in the Mer Project and on Jolla phones and the ill fated tablet. So it has been released onto a few commercial products.

We looked at it when I was working at Nokia on the project that I still cannot name, but we had decided we would come up with our own solution. Looking back, this was the wrong decision, we should have taken the already proven sensor framework and ran with that. Why? Because it existed and works.

I started maintaining it when I was a privateer (contractor) developer for Jolla. No one else had touched it for some time so I grabbed the few not yet merged bug fixes and added support for libhybris/android libhardware adaptors.

Sensor Framework has support for multiple clients with down sampling for different data rates. It uses dbus for control lines (to start and stop, etc) but sends data through a socket. It also has a working backend in QtSensors.

I noticed that Ubuntu's Unity does nothing to respond when I put this into "tablet mode". I have to manually open the virtual keyboard among other things.

So I thought I could use sensorfw on my Dell 2 in 1. It's one of those converged laptop/tablet devices. It has a few sensors - accelerometer, gyroscope, magnetometer, and lid sensors. One problem... sensorfw does not support lid sensors, or a few other sensors that are around today in IoT (which I will add a bit later). Lid "sensor" might be a bit of a misnomer, as they could be switches but I'd like to think it is more like a hal effect sensor that uses magnets. In any case there are event nodes to use.

First one I chose is to add the lid sensor - to detect when this machine is put into tablet mode, so the UI can better deal with it.

I also noticed that this kernel has support for iio sensor interface for the accel and gyro. Sensorfw only supports sysfs, evdev and hybris interfaces, so I also wanted to add support for that.

I worked on adding iio support first. Well... really just wrote a sensor adaptor plugin. My plugin supports accelerometer, gyroscope and magnetometer, which this device seems to have. I will expand this to support other sensors later, as well as clean it up a bit.

Thanks to QtSensors sensor framework backend, I can make a UI app change with the orientation and lid changes. Better yet, I can create a game that uses accelerometer data like a marble maze game. Or I can upload the data to one of those Node.js data visualization web apps.

And since sensor framework is opensource, others can as well.

snappy sensors

Sensors are an important part of IoT. Phones, robots and drones all have a slurry of sensors. Sensor chips are everywhere, doing all kinds of jobs to help and entertain us. Modern games and game consoles can thank sensors for some wonderfully active games.

Since I became involved with sensors and wrote QtSensorGestures as part of the QtSensors team at Nokia, sensors have only gotten cheaper and more prolific.

I used Ubuntu Server, snappy, a raspberry pi 3, and the senseHAT sensor board to create a senseHAT sensors snap. Of course, this currently only runs in devmode on raspberry pi3 (and pi2 as well) .

To future proof this, I wanted to get sensor data all the way up to QtSensors, for future QML access.

I now work at Canonical. Snappy is new and still in heavy development so I did run into a few issues. First up was QFactoryLoader which finds and loads plugins, was not looking in the correct spot. For some reason, it uses $SNAP/usr/bin as it's QT_PLUGIN_PATH. I got around this for now by using a wrapper script and setting QT_PLUGIN_PATH to $SNAP/usr/lib/arm-linux-gnueabihf/qt5/plugins

Second issue was that QSensorManager could not see it's configuration file in /etc/xdg/QtProject which is not accessible to a snap. So I used the wrapper script to set up  XDG_CONFIG_DIRS as $SNAP/etc/xdg

[NOTE] I just discovered there is a part named "qt5conf" that can be used to setup Qt's env vars by using the included command qt5-launch  to run your snap's commands.

Since there is no libhybris in Ubuntu Core, I had to decide what QtSensor backend to use. I could have used sensorfw, or maybe iio-sensor-proxy but RTIMULib already worked for senseHAT. It was easier to write a QtSensors plugin that used RTIMULib, as opposed to adding it into sensorfw. iio-sensor-proxy is more for laptop like machines and lacks many sensors.
RTIMULib uses a configuration file that needs to be in a writable area, to hold additional device specific calibration data. Luckily, one of it's functions takes a directory path to look in. Since I was creating the plugin, I made it use a new variable SENSEHAT_CONFIG_DIR so I could then set that up in the wrapper script.

This also runs in confinement without devmode, but involves a simple sensors snapd interface.
One of the issues I can already see with this is that there are a myriad ways of accessing the sensors. Different kernel interfaces - iio,  sysfs, evdev, different middleware - android SensorManager/hybris, libhardware/hybris, sensorfw and others either I cannot speak of or do not know about.

Once the snap goes through a review, it will live here https://code.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/sensehat, but for now, there is working code is at my sensehat repo.

Next up to snapify, the Matrix Creator sensor array! Perhaps I can use my sensorfw snap or iio-sensor-proxy snap for that.

The other half of the Jolla story

They say there are two sides to every coin, and that holds true for the story of the history leading up to Jolla and it's Sailfish OS. The Jolla story usually starts out with Nokia, but it's really a convergence with Nokia as the center point.

This side of the story starts in Norway, not Finland. Oslo, in fact. Not with Nokia, but with a small company named Trolltech.

I won't start at the very beginning but skip to the part where I join in and include a bit about myself. It was 2001, I was writing a Qt based app called Gutenbrowser. I got an email from A. Kozak at Trolltech, makers of Qt. Saying that Sharp was planning to release a new PDA based on Qt, and wouldn't it be cool if Gutenbrowser would be ported to it? I replied, yes, but as I have no device it might be difficult. He replied back with a name/email of a guy that might be able to help. Sharp was putting on a Developer Symposium where they were going to announce the Zaurus and hand out devices to developers. I jumped at the chance.

It was in California. At that time I was in Colorado. Jason Perlow was working for Sharp at that time, and said he had an extra invite to the Developer symposium. WooHoo! The Zaurus was going to run a Qt based interface originally named QPE, later named Qtopia (and even later renamed Qt Extended). The sdk was released, so I downloaded it and started porting even before I had a device to test it on.

Qtopia was open source, and it was available for developers to tinker with, and put on other devices. There was a community project based on the open source Qtopia called Opie that I became involved with. That turned into me getting a job with Trolltech in Australia, where Qtopia was being developed, as the Qtopia Community Liaison, which luckily later somehow turned into a developer job.

Around the time that Nokia came out with the Maemo tablets, I was putting Qtopia on them. N770, N800, N810, and N900 all got the Qt/Qtopia treatment. (Not to mention the OpenMoko phones I did as well).

Then I was told to flash a Qtopia on an N810 because some Trolls were meeting with Nokia. That became two or three images I had to flash over the coarse of a few weeks. I knew something was up.

Around this time, one of the Brisbane developers (A. Kennedy, I'm looking at you!) had a Creative Friday project to make a dynamic user interface framework using xml. (Creative Friday was something Trolltech did that allowed developers to spend every Friday (unless impending doom of bug fixes/release) of their time on research projects) It was really quite fluid and there was a "prototype" interface running on that N810 as well. It only took a few lines of non c++ code to get dynamic UI's. This would have turned into what the next generation of Qtopia's interface would be made with. It was (and still is) quite amazing.

Then came the news that Nokia was buying Trolltech! Holy cow! A HUGE company that makes zillions of phones wanted to buy little ol' Trolltech. But they already had a Linux based interface - Maemo that was based on Gtk toolkit, and not Qt. WTH!?

Everyone speculated they wanted Trolltech for Qtopia. Wrong. Nokia wanted Qt, and decided to ditch Qtopia. We had a wake for the Qtopia event loop to say our good riddance. All of us in Brissie worried about our jobs.

So our little Trolltech got assimilated into this huge behemoth phone company from Finland. Or was it that Trolltech took over Nokia...? Nokia had plans for Qt that would provide a common toolkit for their massively popular Symbian and new Linux based phones.

The Brisbane office started working on creating the QtMobility API's. Yes, there are parts of Qtopia in QtMobility.

Meanwhile, that creative friday xml interface was still being worked on. It got canceled a few times and also revived a few. That eventually evolved into QML, and QtQuick.
Then came N9 and MeeGo, which was going to use this new fangled dynamic UI. MeeGo was also open source, and it's community version was called Mer and Nemo. Yes, there are parts of Qtopia in MeeGo.

The rest of the story is famous, or rather, infamous now. Nokia made redundant the people working on MeeGo. Later on, all of us Brisbane developers, QA and others were also made redundant. The rest of what I call the Trolltech entity got sold to Digia. The QA server room was packed up and shipped to Digia, who is doing a fantastic job of getting Qt Everywhere!

A few of those guys that were working on MeeGo got together and created a company called Jolla, and created a Linux based mobile OS based on Mer named Sailfish. Yes, there are a few Trolltech Trolls working for Jolla. and yes, there are parts of Qtopia in Sailfish.

state of QSensor and Qt's sensor gestures

I wanted my next post to include some videos demonstrating QSensors and QSensorGestures. I got sidelined by learning BlackBerry Cascades development.


I am happy to report that Qt5's QSensor and QSensorGestures work very well on BB10.

The qt-project, in cooperation with Qt copyright holders Digia, have recently opened up Qt Mobility project and included that code in the Open Governance model that Qt is currently developed under.

This means that I can contribute to qt-mobility again since Nokia closed the Brisbane office.

There are a number of high profile projects that use Qt 4 and Qt Mobility, namely Jolla Mobile and BlackBerry 10.

One of the first community committed patches was me backporting QSensorGesture to Qt Mobility.
You can get the code here:
http://qt.gitorious.org/qt-mobility


There is one change in the signals being generated. I decided to remove the runtime generated signals such as 'shake()' and 'whip()' that use QMetaObjectBuilder. So now there is only 'detected(QString)' signal. The QML API remains the same, with an additional 1.3 version.

On top of that, I added a freefall gesture, as well as moving the irProximity sensor to it's proper place with all the other QSensors.

As for Qt 5...

QSensors unfortunately will not be part of the Qt 5 release package. On top of that, you wont be able to currently find the documentation at qt-project.org nor build QSensors using the normal 'make module-qtsensors'. So, it's a bit like it fell off the face of the Earth.

But fear not, it's still there in git. Still being worked on and improved.