I’ve been thinking about a standards-based client for a mobile environment. XMPP has quite a few strong merits, against it’s competitors such as OMA IMPS and SIP Messaging. For one, it’s a community standard, and it’s actually possible to submit new specs if so required. Secondly, it’s becoming the standard protocol for IM, and it’s emergent in the open Pub/Sub infrastructure, i.e. why-polling-is-bad.

Unfortunately, there are still a couple of key problems with XMPP in a mobile environment – none of them can be solved in a standard way. I wonder if we could harvest a couple of good ideas from OMA IMPS and spec them under XMPP.

Bandwidth

The XMPP stream is verbose, which is not a bad thing by itself, except when you live in a mobile environment. Compression, either at the application (gzip) or the payload level (XEP-138), shifts the problem onto CPU cycles: you still need to decompress first, then decode, which increase latency and shortens battery life. Ideally, you would like to compress the payload onto something that can be decoded and parsed at once without requiring decompression, i.e. WBXML.

Unfortunately it seems like the community has previously slashed against these changes. When Google shifted from XMPP to its custom binary XML protocol for GTalk API, there was a heated discussion about its suitability. But what are folks suggesting that works for the mobile environment, and not just smartphones? I have not seen anything that would work in all phones, hence why WBXML. Please prove me wrong.

Real-time Notifications

The default XMPP stream uses bi-directional TCP/IP connections to allow real-time notifications (messaging, presence, etc.). Open sockets in a mobile environment are a bad idea for quite a few reasons, and although BOSH/HTTP-binding for XMPP solves the TCP/IP persistent connection problem, it is still expensive from a device perspective, as there is always a hanging HTTP request. As far as I can tell, there is currently no XMPP solution to this, and I can see how a extension to XMPP adding WAP-PUSH, WAP-UDP, and perhaps standalone UPD, would make sense.

Again, I’d like to think there is a standard solution for this. And no, iPhone Push notification does not really count.

The Standard Way

We can let folks come up with their own implementations, and then try to refit a spec for the sake of interoperability. But we know how difficult this becomes in other processes like Oasis, JCP, etc. I wonder then, if it is time now to spec this out, (and no, spec jokes, don’t count):

• Binary XMPP
• Push Notifications

I know I said I was not likely to fix pywbxml, but the alternative was even less appealing. I know I have to move away from Java:

1. Continuing to use a mix of Java and PHP to leverage the PHP PECL wbxml extension creates an unnecessary complexity.
2. Moving the PHP code doing wbxml2xml and xml2wbxml transformations to Java is possible but I noticed lack of activity on KML at sourceforge (the project that contains the wbxml library for Java). libwbxml in contrast is owned and maintained by the opensync folks, and it’s only the python bindings that needed some love.
3. The XMPP library I use in Java, Smack uses a model of one-thread-per-socket, which for server-side and maintaining hundreds or thousands of users will require me to have much more memory in the box that I would like to. I rather use an event-based / reactor architectural style.

So I decided to patch the Python bindings for my needs (rather than fixing it properly by exposing the args through the API). The Python bindings seem to be generated out of SWIG, so editing src/pywbxml.pyx is enough to get the dictionary I wanted for IMPS (WBXML_LANG_WV_CSP12). The change is straight forward, simply:

params.lang = WBXML_LANG_WV_CSP12

or the unified patch:

--- pywbxml-0.1/src/pywbxml.pyx 2006-07-28 01:51:57.000000000 +0100
+++ pywbxml-0.1-wv/src/pywbxml.pyx  2009-01-14 21:55:40.000000000 +0000
@@ -14,7 +14,7 @@
     object PyString_FromStringAndSize(char *s, int len)
     int PyString_AsStringAndSize(object obj, char **buffer, int *length)

-class WBXMLParseError:
+class WBXMLParseError(Exception):
     def __init__(self, code):
         self.code = code
         self.description = <char *> wbxml_errors_string(code)
@@ -28,7 +28,7 @@
     cdef WBXMLGenXMLParams params

     params.gen_type = WBXML_GEN_XML_CANONICAL
-    params.lang = WBXML_LANG_AIRSYNC
+    params.lang = WBXML_LANG_WV_CSP12
     params.indent = 0
     params.keep_ignorable_ws = 1

You will notice an additional change in the patch for WBXMLParseError – this comes from the Debian package, and ensures the we are raising a proper exception.

With this applied, compiled, and installed, I ran the same test with an IMPS CSP document, this time successfully:

$ python
Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pywbxml import xml2wbxml, wbxml2xml
>>> xml = """<?xml version="1.0"?><!DOCTYPE WV-CSP-Message PUBLIC "-//OMA//DTD WV-CSP 1.2//EN" "http://www.openmobilealliance.org/DTD/WV-CSP.DTD"><WV-CSP-Message xmlns="http://www.wireless-village.org/CSP1.1"><Session><SessionDescriptor><SessionType>Outband</SessionType></SessionDescriptor><Transaction><TransactionDescriptor><TransactionMode>Request</TransactionMode><TransactionID>nok1</TransactionID></TransactionDescriptor><TransactionContent xmlns="http://www.wireless-village.org/TRC1.1"><Login-Request><UserID>hermes.onesoup</UserID><ClientID><URL>WV:IMPEC01$00001@NOK.S60</URL></ClientID><Password>xxxxxx</Password><TimeToLive>86400</TimeToLive><SessionCookie>wv:nokia.1789505498</SessionCookie></Login-Request></TransactionContent></Transaction></Session></WV-CSP-Message>"""
>>> binary = xml2wbxml(xml)
>>> text = wbxml2xml(binary)
>>> text
'<?xml version="1.0"?><!DOCTYPE WV-CSP-Message PUBLIC "-//OMA//DTD WV-CSP 1.2//EN" "http://www.openmobilealliance.org/DTD/WV-CSP.DTD"><WV-CSP-Message><Session><SessionDescriptor><SessionType>Outband</SessionType></SessionDescriptor><Transaction><TransactionDescriptor><TransactionMode>Request</TransactionMode><TransactionID>nok1</TransactionID></TransactionDescriptor><TransactionContent><Login-Request><UserID>hermes.onesoup</UserID><ClientID><URL>WV:IMPEC01$00001@NOK.S60</URL></ClientID><Password>xxxxxx</Password><TimeToLive>86400</TimeToLive><SessionCookie>wv:nokia.1789505498</SessionCookie></Login-Request></TransactionContent></Transaction></Session></WV-CSP-Message>'

Yes, I am now happier.

I need to be able to receive wbxml (binary XML) and transform it to xml using the IMPS 1.1/1.2/1.3 dictionary in Python. In PHP, I was using the PHP PECL extension wbxml, which uses libwbxml (wbxml2). In python, I need pywbxml. I like MacPorts, and I’ll use that instead of compiling myself the packages.

1. Install MacPorts.
2. Self-update:

   sudo port -v selfupdate
   

3. Some general goodness I can’t live without (wget will pull a long dependency list of things we’ll need for web development):

   sudo port install wget
   

4. Wait.
5. Install wbxml

   sudo port install wbxml2
   

6. And we get an error:

   $ sudo port -v install wbxml2
   --->  Configuring wbxml2
   Error: Target org.macports.configure returned: invalid command name "cd"
   Warning: the following items did not execute (for wbxml2):
   org.macports.activate org.macports.configure org.macports.build
   org.macports.destroot org.macports.install
   Error: Status 1 encountered during processing.
   

So I filed a ticket. It seems like wbxml2 ownership moved to the opensync folks since the port was added. The port builds on 0.9.0 and the latest version is 0.10.1. I don’t much (nothing?) about MacPorts, but I went ahead and I patched it. You can find the patch on the ticket itself.

Finally, moving onto pywbxml. It seems the only maintained pywbxml code is from the synce folks.

1. Download pywbxml from sourceforge.
2. Enter the virtualenv:

   $ source ~/Sites/python/imps/bin/activate
   

3. pywbxml needs pyrex, so we’ll install that first with:

   $ easy_install pyrex
   

4. Install pywbxml:

   $ tar xzfv pywbxml-0.1.tar.gz
   $ cd pywbxml-0.1
   $ ./configure
   $ make
   $ make install
   

5. Optionally, move the libraries to the virtualenv to keep things clean:

   $ mv /Library/Python/2.5/site-packages/pywbxml.* \
       /Users/brunofr/Sites/python/imps/lib/python2.5/site-packages
   

Now, we can test it:

  $ python
  Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
  [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> from pywbxml import xml2wbxml, wbxml2xml
  >>> xml = """<?xml version="1.0"?><!DOCTYPE WV-CSP-Message PUBLIC "-//OMA//DTD WV-CSP 1.2//EN" "http://www.openmobilealliance.org/DTD/WV-CSP.DTD"><WV-CSP-Message xmlns="http://www.wireless-village.org/CSP1.1"><Session><SessionDescriptor><SessionType>Outband</SessionType></SessionDescriptor><Transaction><TransactionDescriptor><TransactionMode>Request</TransactionMode><TransactionID>nok1</TransactionID></TransactionDescriptor><TransactionContent xmlns="http://www.wireless-village.org/TRC1.1"><Login-Request><UserID>hermes.onesoup</UserID><ClientID><URL>WV:IMPEC01$00001@NOK.S60</URL></ClientID><Password>xxxxxx</Password><TimeToLive>86400</TimeToLive><SessionCookie>wv:nokia.1789505498</SessionCookie></Login-Request></TransactionContent></Transaction></Session></WV-CSP-Message>"""
  >>> binary = xml2wbxml(xml)
  >>> text = wbxml2xml(binary)
  >>> binary
  '\x03\x01j\x00Imnp\x80\x19\x01\x01rtv\x80 \x01u\x03nok1\x00\x01\x01s\x00\x01]\x00\x00z\x03hermes.onesoup\x00\x01Jw\x03WV:\x00\x80\x12\x03PEC01$00001@NOK.S60\x00\x01\x01\x00\x01a\x03xxxxxx\x00\x01r\xc3\x03\x01Q\x80\x01p\x03wv:nokia.1789505498\x00\x01\x01\x01\x01\x01\x01'
  >>> text
  '<?xml version="1.0"?><!DOCTYPE AirSync PUBLIC "-//AIRSYNC//DTD AirSync//EN" "http://www.microsoft.com/"><Delete xmlns="http://synce.org/formats/airsync_wm5/airsync"><unknown><unknown><unknown><Truncation/></unknown></unknown><unknown><unknown><unknown><Supported/></unknown><unknown>nok1</unknown></unknown><unknown><Email3Address><unknown>hermes.onesoup</unknown><Fetch xmlns="http://synce.org/formats/airsync_wm5/airsync"><unknown>WV:<CollectionId/>PEC01$00001@NOK.S60</unknown></Fetch><HomeCity>xxxxxx</HomeCity><PagerNumber>\x01Q\x80</PagerNumber><OtherState>wv:nokia.1789505498</OtherState></Email3Address></unknown></unknown></unknown></Delete>'

It seems like it’s not using the WV CSP 1.1/1.2 dictionary (not surprising since I did not specify it anywhere). Looking at the code, in pywbxml.pyx, I can see:

  params.lang = WBXML_LANG_AIRSYNC

So, whereas the PHP PECL wbxml extension is exposing all the parameters in libwbxml, it seems like the Python version pywbxml is not, and it’s hard-coding assumptions.

Lesson learned with my day on python, twisted, wbxml, etc.: spending my time fixing the basics, does not allow me to work on the problems I need to solve. Will I fix pywbxml? I don’t know, but most likely not.

Python 2.5.1 is bundled into Mac OS X 10.5 (Leopard), which comes also with setuptools. The breadth of easy_install packages is available pretty much at your fingertips on Leopard.

On the downside, Leopard’s python ships with old versions of some packages, and we might need to upgrade them. We are left with three choices:

1. Overwrite packages with new ones.
2. Install new ones and set PythonPath.
3. Use virtualenv.

I prefer to use #3. It’s cleaner, it does not change my system, it allows as many environments as we like without one contaminating the other, while at the same time not requiring a fresh python re-install for every working environment.

I want to create an environment for my imps-to-xmpp gateway, for which will be using twisted words. I want to use the latest version (8.2), but the shipped version with leopard is fairly old (2.5):

$ which twistd
/usr/bin/twistd
$ twistd --version
twistd (the Twisted daemon) 2.5.0
Copyright (c) 2001-2006 Twisted Matrix Laboratories.
See LICENSE for details.

So, we’ll start by installing virtualenv using setuptools:

$ easy_install virtualenv

And now, we’ll create a virtual environment on which we can install the updated twisted:

$ cd ~/Sites/
$ mkdir python
$ cd python
$ virtualenv imps
$ source imps/bin/activate
$ easy_install twisted

We can now check that we are running the latest version of twisted (8.2 as of writing):

$ twistd --version
twistd (the Twisted daemon) 8.2.0
Copyright (c) 2001-2008 Twisted Matrix Laboratories.
See LICENSE for details.

AOL has published the protocol powering both AIM and ICQ (code-named OSCAR). Although the protocol was already well known through reverse engineering, even if it’s significantly more complex and chatty than MSN’s or YMSG, this is really good news since it formalizes AOL supporting third party clients, including Yamigo. Additionally, besides publishing the protocol documentation, AOL has released an AIM SDK for C/C++ and Java, and the ability for creating protocol plugins that can even be monetized through revenue-sharing with AOL. Altogether, a good move from AOL.