Operators Don't Want To Get Caught Doing A Barclays Bank

There seems to be a trend among lazy sub-eds to tag anything about mobile payments, wallets etc as "Mobile Banking" - even when the item they're copying doesn't, and they have nothing to do with banking.

The intiative itself appears interesting and has enough momentum to work now that Nokia, Mastercard, Vodafone et al are on board - it's basically just rolling out an internationalized version of the system DoCoMo have been using for some time in Japan - but more interesting is what the operators will do when they hear a lot of people are calling them banks. I bet they'd love to do all sorts of banking related things, but I also bet they really really don't want the banking regulations that come with them. Sticking to payments sounds much more sensible.

There was a lot of discussion about this back when Vodafone launched mPay and other similar initiatives came out, and I seem to remember the consensus opinion was they'd do anything to avoid becoming banks in the regulatory sense of the word, but typically I can't find any of those articles now. Such is life.

(look here if you're confused about the title)

Sony Ericsson in JBed with Esmertec Shocker? We Must be Told

So the Jbed saga continues. 'Guilhem' alleges Sony-Ericsson use the Jbed JVM, something that even Esmertec don't know about - could it be true? Could the most reliable and arguably most advanced MIDP JVM in the market actually be Jbed in disguise? Could the many bugs on the LG Chocolate and the Siemens SF65 purely be down to incompetence at the manufacturers, and nothing to do with the underlying JVM product? They are both fashion phones sold purely for their looks, with questionable insides, after all.

The Case For
The case for is slim. It primarily rests on the word of Guilhem, who doesn't actually give any personal details identifying him further (don't you just hate it when bloggers do that? etc). He'd have to work at SE or Esmertec to have that kind of info, as if SE really do licence Jbed they're much too embarassed to let Esmertec tell anyone about it. Esmertec instead seem to concentrate on no-name Asian OEMs, with a handful of questionable handsets from Siemens and LG as the big names.

You have to look hard for other evidence to support his assertion. The blistering speed of the MIDP2 Sony-Ericssons is down to their use of Jazelle (since 2004), and the performance of the FastBCC variant is much better than the conventional version on eg. the Chocolate - JBenchmark claims FastBCC runs on Jazelle processors too, so there is some connection. However the FastBCC version still has a number of bugs which the SEs don't, in things like the inflate algorithm used in PNG IDATs, so this isn't looking too likely.
(Note: I call this Jazelle version 'fastBCC' purely because the system Profile property reports it as Jbed-FastBCC, and other buggier slower devices don't... I have no idea which devices really use fastBCC vs fastDAC).

When Lenovo signed up to use Jbed the press release claimed "Esmertec's software has been commercially launched within 4 of the 5 leading handset brands worldwide" - on 1/1/'07 that would be Nokia, Moto, Samsung, SE and LG. LG we know about, the others...? Well, we'll consider this again shortly, but the release only says "Esmertec's software" and not explicitly Jbed so it doesn't mean as much as they want you to think it does.

And finally, Sony-Ericsson emulators are just WTK reskins - other Jbed licensees don't ship any emulators, which implies Esmertec can't be bothered to make them, which in turn does suggest a link, but a tenuous one. The SE emulator's 'THIRDPARTYLICENSES.txt', incidentally, does not mention Esmertec/Jbed at all and nor does LicenseAgreement2.pdf.

The Case Against
Firstly, it could just be mistaken identity - the Jazelle is a processor which runs a lot of the JVM bytecode natively on the chip, but it requires support software from ARM called JTEK. Sounds kind of similar, could be confused?

Also it is worth comparing the little things about the JVMs. All Jbed devices declare their Platform system properties as 'Jbed' or some variant - Sony-Ericsson give detailed handset information. SE support every JSR under the sun, with a huge number of filetypes supported under MMAPI etc - traditionally Jbed devices do not, instead focussing on a handful of JSRs and limited media support. Gilhelm suggests this is the sort of customisation a manufacturer must do to the Jbed JVM to make it useful, which SE does and LG can't be bothered with. Possible, but it's a lot of customisation work and surely other licensees would pick up on the JSRs that don't need much native support?
The one massive improvement SE would have had to have made to the standard Jbed as used on all LGs (fastBCC and the rest) is the complete elimination of the multiple compile / jar optimizing steps which take up many seconds on the LGs but are completely missing on Sony-Ericsson devices. Some might call this the only piece of evidence required, but I'll go on.

But the real killer is in the numbers. Esmertec were claiming in that Jan 1st 2007 press release that Jbed has been shipped on 76m devices ever. As a comparison, according to analyst iSuppli Nokia (#1) shipped 348m devices and LG (#5 and Esmertec's biggest acknowledged licensee) shipped 64.4m in 2006. For Nokia, the vast majority of those devices have to have Java; for SE, the percentage is even higher as they sell so few low-end devices. So even if as Esmertec imply 4 of the top 5 manufacturers have shipped at least one device with Jbed in, it was a minority device and could not be their main Java platform.

SE's Java platform has remained incredibly consistent in every way you can examine since their first MIDP2 devices on Jazelle back in 2004, and has been in almost every device except the barely-selling J200 series etc at the low-end and the niche top-end Symbian devices (which run on the rather shaky Monty JVM). So SE's share of those 76m devices would have to be well over half of the Jbed devices shipped if it's core platform (the one I always praise) ran on Jbed - note that iSuppli claim SE shipped more than 42m devices in the second half of 2006, at which point their platform was already over a year old.

Where would that put LG, with those 64.4m devices shipped in 2006? We know that many of their sales come from their home market and from BREW devices shipped in the US, but you could conservatively assume that maybe 1/4 of those devices - 16m - were Java devices running on Jbed (with maybe 1m of those Chocolates - LG don't reveal the split between the Korean, GSM/MIDP and CDMA/BREW versions but we can assume the 1.7m sold before July '06 covers them all). Their pre-2006 devices also shipped Jbed, and have since at least Q2 2003, the launch date of the really really bad G7100 I have in my desk (MIDP1 Jbed, don't go there); LG sold 44m devices in 2004 so overall you could conservatively expect a good 30m of those 76m Jbed devices to be LGs, and my gut says it'd be a higher number. That doesn't leave much room for all those Sony-Ericssons.

So I'm going to have to say that unless Gilhem can pull some pretty amazing evidence out of the bag, I have to assume he's wrong and SE run on their own Java platform based on ARM's Jazelle and JTEK products. Esmertec may have licensed some software component used somewhere in the system by SE and the others in the big 5, but I fail to see what it is and it must be so small as to have little destabilising impact on the rest of the platform. Gilhem, care to prove me wrong?

Clarification
There was the subtle implication in the phrasing of the original comment that I may be biassed against Esmertec/Jbed unduly. This is definitely not the case. I dislike the known confirmed implementations of Jbed purely because they are really really bad, as I outlined in my post. For all I know the company may in every respect except for the Jbed product be the greatest company in the world.

What's Wrong With JBed

In my last post I was fairly blunt in my dislike for JBed, Esmertec's pseudo-JVM for low-end devices such as most of LG's range (including the Chocolate).

Why Is It Different?
When you install content onto a JBed device, after the content has arrived it must go through a compilation stage where it is compiled into a native machine code - it is run native, rather than as interpreted bytecode in a sandboxed JVM. The native code still retains the garbage collection, threading and permissions models of a MIDP/CLDC JVM so code can in theory run just as it would traditionally; I have absolutely no idea if there are any security holes introduced by this process, this could be a fruitful area for a security researcher but I currently have no experience that suggests this is a risk. Array bounds checking and the like appear to still be implemented correctly.

Esmertec claim a performance increase of "up to 20x" from this process. That suggests the underlying hardware this JVM usually runs on must be really poor, because performance is pretty typical for low-end devices with equivalent speed in the non-Java UI. I'm sure there is some advantage gained to the manufacturer in slightly reducing the electronics cost of the device

Why Is It Bad?
The addition of a compilation stage is not, per se, bad - in fact if it really does transparently improve performance without affecting the way the code runs, it's great. Note the qualifiers there though: the reason I dislike it is that sometimes, arbitrarily, it fails to compile the code - code which a few small source changes ago compiled fine - just like the early Symbian MontyThread bugs where a small change would kill your code for the 6600 v3.4 firmware. This sort of unpredictability has been greatly reduced from the early days, when the MIDP1 LGs would just arbitrarily fail things all over the place, but it still exists.

Unpredictable builds are a nightmare, greatly increasing QA time for every tiny change. Do you single out these devices for early and repeated testing so you can trap failures immediately and pin down exactly what change triggered the problem, or is it better to just leave them to the end and accept days of pain if the build happens to fail, tweaking every line until it works? Whenever possible I now opt for the latter as failures can come and go as they please, and I make it clear that these problem devices are troublesome and will only be supported if possible, but this doesn't help when you have to patch code already out in the wild and the problems suddenly surface from nowhere.

I'll repeat, failures at compile time have become much less frequent on more recent MIDP1 devices like the Chocolate, so this is becoming less of an issue - but the unpredictability remains in other areas. Don't try talking to Esmertec about the flaws though, you will be greeted with complete silence and a total lack of useful advice.

ZLib
The ZLib deflate/inflate compression system underpins two major parts of JavaME: the Jar file format used to package applications (which is really just a special case of zip), and the mandatory PNG file format in which almost every image you will use is encoded.

The JBed JVM as implemented on, for example, the Chocolate has problems with both of these file types (it presumably uses the same flawed inflate algorithm for both). With conventional tools, you rarely hit problems but if you use Ken Silverman's extremely useful tools you will hit more problems because they use clever heuristics out on the edges of the deflate spec to improve compression, resulting in Zlib blocks that the normal Zlib will never produce and therefore presumably never appear in the test suites for Esmertec's inflate algorithms (I am assuming that they have test suites).

If you use KZip to create smaller Jars, sometimes they will fail - some files just won't be able to decompress so either the install will fail or some resources will not load correctly at runtime. This seems to go away if you use the conventional Jar tool, so no big problem - your users will pay a little extra for those spare Kilobytes, but it won't be more than 1-2% more.

If you use PNGOut you have larger problems. On many devices, including S40 MIDP2 handsets like the 6230, maybe 0.5% of files run through PNGOut will fail and you just have to resave them in a more conventional tool and they'll work again. These failures can actually be predicted at build time so it's not the end of the world - you aren't going to kill the QA team by making them check every PNG in every game.
On the Chocolate, however, you see more like 10% of the PNGOut-compressed images failing to load, for a wide range of unpredictable reasons (unpredictable in that, without the source code for their inflate algorithm, I really can't be bothered to find the reasons which seem completely aribitrary and random). So, basically, you can't use PNGOut for images intended for the Chocolate - which means the whole device (and others on this platform) have to be treated as special cases with different resources. Because you can't predict when an image will fail, you still have to thoroughly test that every image has loaded every time the resources change, because it may be possible that a PNG created through a normal tool will also occasionally break the loader. I've enough experience with other devices to say this won't happen, but with JBed devices you just can't take the chance.

Not Just ZLib
I've had certain resource files refuse to load completely, even when I deliberately make sure they are not compressed at all inside the jar. They just throw an exception and you end up with nothing. It's impossible to say what causes this, but if you adjust the first few bytes they will suddenly magically load (whatever the compression level of the file). So maybe the file is accidentally colliding with the magic bytes at the front of a JBed-compiled class - because you get a similar error if you try to open a class file. Or maybe it's something completely different, because you just don't know what is happening to the contents of your jar inside this opaque compilation step.

These are my key problems with the device. There are other bugs, but you accept that they exist and work around them just like with any other device on any other JVM. But it's very hard to live with completely unpredictable failure of some builds, sometimes. It annoys the hell out of your testers and your developers, and it makes managing big complicated builds a nightmare. The absence of support from Esmertec just rubs salt in to the wounds, and it's not a huge consolation to know that it's the same salt Nokia et al use when refusing to acknowledge any bug in their forums, because the JBed wounds are that much bigger and more common.

Why Do South Koreans Hate Developers?

I say South Koreans, but of course North Koreans might be equally guilty if the strange (alleged) Bond-loving opera-writing golfing prodigy great leader would stop ravaging their country and allow them to move into the 20th or 21st centuries. But I digress.

South Korea is home to the world's first or second most advanced mobile infrastructure, depending on how you want to measure it, sitting comfortably ahead of everyone but Japan. After a trade dispute with the US, the South Korean people were saved from BREW and instead given their own government-mandated hybrid C/Java "standard" called Wipi - a "standard" which is only published in South Korean and requires you to be a South Korean citizen to download, as far as I can tell, which is I guess in keeping with the government protection traditionally ladled out by South Korea for other industries.


Whilst advanced, the South Korean market is also pretty small (under 50m people) so we should really look to the global handset shipment figures, where the Top 5 handset manufacturers include two South Korean companies, in stark contrast to that other advanced market Japan where a protected domestic market for Japanese manufacturers led to a complete lack of competitiveness abroad.


Top 5 Developer Resources
A comparison of the developer sites for the Top 5 manufacturers:

1. Nokia - Forum Nokia, containing absolutely everything you might possibly want to know
• Detailed spec database with everything you need to know about every handset, hilighting their very public and clear platform strategy;
  
• Docs on every subject, with fully customisable RSS feeds;
• The best emulators in the top 5 (and some of the only true emulators, most being simulators);
• Active detailed forums - OK no-one from Nokia will ever admit to a bug on one of their devices, but you can normally find someone who can answer your questions;
  
• I could go on. The gold standard by which everyone else fails.
2. Motorola - they have MotoCoder dev site, with some things you might want to know
• Spec PDF docs on some of the handsets, updated slowly, and ignoring quite a few but you can usually work out a handset's platform and infer a lot from that;
• Some docs, though not that many and notably absent for some of the JSRs they have led;
• Passable emulators, pretty accurate with a few annoying bugs like images take 4 bytes per pixel not 2 (but the heap size is the same);
• No forums, but you can ask a question and someone from Moto will (probably) answer it.
3. Samsung - yes, well. They do have a site and at least now it works beyond IE.
• New spec database, hilighting the complete lack of standard platforms, with specs always in flux and many handsets never appearing;
• All the old docs seem to have been deleted;
• I've never managed to get the emulators to work for me, but I might just be unlucky;
• They deleted the contents of the old forums, which contained many valuable bug fixes, and replaced them with new empty forums that suck.
4. Sony-Ericsson - nearly as good as Nokia
• Full specs for all handsets making clear their standard platform strategy, in a not-quite-so-convenient browsing format;
• Almost as many docs, with a slightly less flexible RSS feed format;
• Emulators are a bit of a joke, being simple WTK reskins, but you can do on-device debugging etc which is in many ways more valuable;
• Active detailed forums.
5. LG.

The striking thing you'll notice here is that Nokia, Moto and SE all have active developer programmes and platform strategies which are either clear, or easy to guage from publically available information. With the South Korean manufacturers you have two choices - bad or nothing.

LG
You may think I forgot to add anything for LG in the list above, but in fact the omission is theirs. They have no developer support. I did once track down the (Korean) Javadoc for a very old proprietary sound API, which Babelfish kindly translated into pseudo-English for me, but I never got it to work; that's about all I've ever seen.

Despite lots of efforts with the Chocolate series and more recently the Shine, LG still fundamentally make lots of cheap rubbish low-end flip phones, many of which are for the CDMA market - anything in their non-Korean catalogue that is not low-end appears to be for show, selling in very low volumes. Chocolate is the big exception, but it is a deeply flawed device - anyone who has used one will know quite how infuriating it is to watch those sexy soft keys and direction buttons start pressing themselves, sometimes for several minutes as they traverse randomly round menus or totally screw up whichever fuction you were trying to use. I counted a few hundred button presses in 5 minutes just this morning, before I turned the phone off. I sense a YouTube video - watch this space.

This reliance on the low-end seems to have led to their developer strategy - low-end phones struggle with Java, so they fell for the JBed sales pitch and included that trully terrible JVM-replacement on everything but the high(ish)-end U8xxx series. This does some strange precompilation steps to "speed up" your code, which in practice just fail on some classes for no reason. The MIDP2 version is better than the MIDP1, but I have had code which works fine suddenly refuse to precompile and install because I have changed a few lines, and magically start working again later. It also has the flakiest PNG loading on the planet. So no wonder they don't have developer support - they clearly would rather developers didn't develop for the devices and they certainly couldn't afford to handle the huge volumes of support requests and complaints they would get if they tried.

Samsung
Samsung aspire to compete at the high-end. Not long ago they were getting very excited about how they would overtake Motorola and the future was theirs - shortly before the world stopped liking their zillion D500 rip offs and their market share dropped sharply back off, so now they are defending themselves against the solid advance of Sony-Ericsson.

Many of their handsets are on the cutting edge, shipping HSDPA before anyone else and beating Motorola on both the thinness and ease of use fronts (OK, the latter was hardly a challenge). Their Java efforts have always been decidedly middling though - many many bugs, documented on the old forums before they were deleted, and very few APIs beyond the basic MIDP and CLDC. Their LCDUI implementations are usually straight out of the WTK, offering no native UI functions and feeling completely alien to the phones.

The true failing of Samsung, however, has been their lack of platform standardisation. By all accounts they have a large number of research groups basically designing handsets independently - this affects their lineup in a number of ways:

1. The (almost) random model numbering system is highly confusing, and tells you nothing at all except the target region with the last digit;
2. There are a huge number of outwardly identical phones with similar specs but based on different platforms, confusing consumers and probably leading to their recent sales slump;
3. You just can't make any assumptions about them from a developer's perspective - just because your game runs on one MIDP2 176x220 Samsung handset does not mean it will run on any other.

Point 3 is slightly unfair - the developer site used to provide Excel spreadsheets grouping the handsets into devices based on a common platform, suggesting that if your game worked on one in the group it would work on them all. For example, the D500 was in Group 12A and the E800 in Group 4. They categorised maybe two thirds of their European range this way, leaving you to guess the rest.
Then, after at least 6 months of waiting, they launched their new web site. This had a number of amazing innovations:

1. It worked outside IE. The old one could have worked outside IE too, but they chose to implement logins in such a stupid way it didn't;
2. It had a new forum, which was harder to use and now had nothing on it;
3. It had a lot fewer documents;
4. It had a funky new device database.

If the device database had been good, everything else would have been forgiven. It almost is, but is plagues by two things: poor usability and flaky data.

On a usability front, there are a huge number of tiny things that add together to make a really frustrating experience:

• There is no indication of what has been added recently, so you have to manually check a list of 130 handsets in case one has been edited or added - an RSS feed would be ideal, a 'What's New' list would also be good;
• Each device's page shows a massive table of information, but the most important piece - which platform the device belongs to - can only be seen from the index screen;
  
• You can only see 5 devices at once on the index screen - no way to change to show 10/20/100/all - and you have stupid scrolling buttons to page between them. This would be fine if you could use the drop-down of model numbers, but you have to see the entry on the index page to find out the group so you can't;
• Breadcrumbs are shown on the screen, but aren't clickable.
• etc.

Thye flakiness is harder to pin down, but:

• All the old group numbers have been thrown out and the compatibility lists have changed:
  
• A handset now belongs to a Group which is defined as the first handset they released with the same spec such that all handsets in that group should be able to run the same content;
• ie. handsets in a group should share screen size, supported APIs, JVM, memory constraints, underlying hardware etc.
  
• The new platforms appear to change over time, with platforms splitting and devices jumping between them;
• Some data is plain wrong
• eg. the E200 is stated to have a 220x220 screen, but it has a 176x220 screen;
  
• eg. the site currently claims that the E830 and E200
  are part of the E250 group, when they are clearly mid-range devices with a good Java spec and the E250 is a low-end device.
• You get the impression with some groups that the people who define them are being lazy, and creating a new group for every handset
• You get lots of one or two handset groups with basically the same spec from the same teams...

What we should have is a manufacturer clearly stating the widest possible groups of handsets that are internally, from a Java perspective, the same and can reliably run the same builds consistently. What we have instead is a load of changing spec data suggesting handsets should be compatible, but it's so unsure of itself that they move around and anyway as some of the basic specs are wrong, how much do you trust it anyway?

Unless you have a lot of money, you sadly have no choice but to trust it and react when your customers complain.

My constant impression is that the entire dev site is outsourced to basically one guy, SunHo, who valiantly tries to answer everything, help out with bug requests and make sense of platform compatibility with minimal support from Samsungs various departments. If that truly is the case then he does very well considering - but shame on Samsung as a Top 3 vendor for not doing things properly.

Why? Why? WHY?
I wish I could provide some insightful analysis as to why it is only the South Korean members of the Top 5 who behave this way, but presented above are my best guesses. All we can do as developers is plead for more support, and I suspect all they will continue to do as corporate monoliths is to ignore us until the market shows them that 3rd party games and applications are a competitive advantage. I suspect that most people in the mobile space don't dare ask whether the market will ever reach that point, but we certainly aren't there yet.

Crazee Enough for Taptu?

Just saw this amusing job add for mobile search startup Taptu who appear to be after a software engineer with aspirations to be a Web2.0 David Brent. ka ching!! indeed....