Please following below instructions to see if the error can be resolved:
* A supported native browser must be installed (linux requirements)
* If a supported WebKitGTK and/or XULRunner is installed then it should be found automatically at runtime with no additional effort. For other native Mozilla-based browsers:
    1. Set the environment variable MOZILLA_FIVE_HOME to the native browser's installation folder. e.g. setenv MOZILLA_FIVE_HOME /usr/lib/mozilla
    2. Set the environmnent variable LD_LIBRARY_PATH to include MOZILLA_FIVE_HOME. e.g. setenv LD_LIBRARY_PATH ${MOZILLA_FIVE_HOME}:${LD_LIBRARY_PATH}

And also below are requirements for your linux:
Mozilla 1.4 GTK2 - 1.7.x GTK2, XULRunner 1.8.x - 3.6.x, WebKitGTK+ 1.2.x

Also note that a Firefox release whose contained Gecko version correlates with the Mozilla versions above can also be used (linux only), provided that it has been compiled with linkable Gecko libraries. It is important to note that Firefox downloads from mozilla.org do not satisfy this criteria, but Firefox installations that are included in major Linux distributions often do in the absence of a XULRunner installation. Attempting to use a Firefox install without linkable Gecko libraries will throw an error with message "No more handles [NS_InitEmbedding...error -2147221164]".

Also please refer to below links to know more:

http://www.eclipse.org/swt/faq.php#browserlinux
https://www-304.ibm.com/support/docview.wss?uid=swg21271865

gtk2, xulrunner and webkit (and firefox) are installed

Installed Packages
Name : xulrunner
Arch : x86_64
Version : 13.0
Release : 1.fc16
Size : 29 M
Repo : installed
From repo : updates
Summary : XUL Runtime for Gecko Applications
URL : http://developer.mozilla.org/En/XULRunner
License : MPLv1.1 or GPLv2+ or LGPLv2+
Description : XULRunner is a Mozilla runtime package that can be used to bootstrap XUL+XPCOM
            : applications that are as rich as Firefox and Thunderbird. It provides mechanisms
            : for installing, upgrading, and uninstalling these applications. XULRunner also
            : provides libxul, a solution which allows the embedding of Mozilla technologies
            : in other projects and products.

Installed Packages
Name : gtk2
Arch : i686
Version : 2.24.8
Release : 3.fc16
Size : 13 M
Repo : installed
From repo : updates
Summary : The GIMP ToolKit (GTK+), a library for creating GUIs for X
URL : http://www.gtk.org
License : LGPLv2+
Description : GTK+ is a multi-platform toolkit for creating graphical user
            : interfaces. Offering a complete set of widgets, GTK+ is suitable for
            : projects ranging from small one-off tools to complete application
            : suites.

Installed Packages
Name : webkitgtk
Arch : x86_64
Version : 1.6.1
Release : 1.fc16
Size : 32 M
Repo : installed
From repo : anaconda-0
Summary : GTK+ Web content engine library
URL : http://www.webkitgtk.org/
License : LGPLv2+ and BSD
Description : WebKitGTK+ is the port of the portable web rendering engine WebKit to the
            : GTK+ platform.

Installed Packages
Name : firefox
Arch : x86_64
Version : 13.0
Release : 1.fc16
Size : 25 M
Repo : installed
From repo : updates
Summary : Mozilla Firefox Web browser
URL : http://www.mozilla.org/projects/firefox/
License : MPLv1.1 or GPLv2+ or LGPLv2+
Description : Mozilla Firefox is an open-source web browser, designed for standards
            : compliance, performance and portability.

*Where is Firefox installed to?
$ which firefox
/usr/bin/firefox

but running
$ LD_LIBRARY_PATH="/usr/bin/firefox:$LD_LIBRARY_PATH" MOZILLA_FIVE_HOME=/usr/bin/firefox java -jar traymonitor-x64.jar
leads to error QB-1315
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000000000, pid=9686, tid=140672097302272
#
# JRE version: 6.0_33-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.8-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C 0x0000000000000000 float+0x399c4000
#
# An error report file with more information is saved as:
# /home/ashu/bin/hs_err_pid9686.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

the same error for the Fedora 16 x86 (booted from LiveCD on virtual machine)
but NOT for the Fedora 14 x86 (actually I've migrated to Fedora 16 x64 from Fedora 14 x86 a few days ago)

I think the MOZILLA_FIVE_HOME should be the libs for mozilla, not the binary of mozilla, that is, it should be
/usr/lib/mozilla, or /usr/lib/firefox, instead of /usr/bin/firefox
please check the path in your Fedora box.

And also LD_LIBRARY_PATH is the same case.

You may also try one of below path to see which one can work for MOZILLA_FIVE_HOME, (I have no Fedora box and I'm not sure which is the correct path, sorry):
1. /usr/lib64/firefox
2. /usr/lib/firefox
3. /usr/lib/mozilla
4. /usr/share/firefox
5 /usr/share/mozilla

Firefox libs are in /usr/lib64/firefox
nothing actually changed - error occurs

I suspect xulrunner version makes a difference...
I'll try to upgrade it on Fedora 14 to see whether it will bring a breakage or not

after upgrading Fedora 14 to 15 I've got - xulrunner v12 (Fedora 14 had xulrunner 1.9.15)
error still occurs
I've downgraded xulrunner to v2.0.1 - error occurs

correction: "Fedora 14 had xulrunner 1.9.2.15"

also I've downgraded to xulrunner 1.9.2.24... TrayMonitor is started!
upgrade to v2.0.1 - error...
conclusion I make - it does not work with xulrunner v2.x+

The problem is confirmed. As QB TrayMonitor is using eclipse SWT, and SWT only supports gtk2.x, while xulrunner 2.x uses gtk3.x, so currently, TrayMonitor can only work with xulrunner 1.9.

We will update TrayMonitor after SWT supports gtk3. See development plan for SWT below:
http://www.eclipse.org/swt/R3_8/plan.html