Tuesday, October 20
Introduction
If you’re developing commercial software, you’re going to want to sell it globally — for well-localized English-speaking software companies, for instance, I have seen between 20% and 25% of total revenue coming from non-English speaking countries. That’s, like, real money. You’ll want that.
Delicious Monster
Gross Sales by
Language of Country |
|---|
| English | 80.9% | | German | 8.1% | | French | 3.2% | | Japanese | 2.6% | | Unknown / Other | 2.2% | | Dutch | 1.5% | | Spanish | 0.9% | | Norwegian | 0.8% | | Italian | 0.5% | | Danish | 0.5% | | Swedish | 0.5% | | Chinese | 0.3% | | Portuguese | 0.3% | | [Delicious Monster Oct09] |
| | Omni Group
Upgrades by Language |
|---|
| English | 74.4% | | German | 6.0% | | French | 5.3% | | Japanese | 5.0% | | Spanish | 2.3% | | Other | 2.3% | | Italian | 1.9% | | Dutch | 1.4% | | Chinese | 0.7% | | Portuguese | 0.4% | | Norwegian | 0.4% | | [Omni Group Oct09] |
|
Note that the data from the above tables was gathered in different ways: Omni tracks what language a user has set in her preferences
every time she upgrades any of her Omni software, whereas I tracked Delicious Monster’s
total dollar sales by country, and then assigned each country a “primary” language. (For countries like Switzerland, obviously my approach is an approximation.)
Still, you can try to make some educated guesses from these two data sets, although obviously correlations aren’t necessarily causative when analyzing only two samples. But, for example, Omni has Japanese-language support and Japanese-only boxed versions of their products; their Japanese number is twice that of Delicious Monster, which doesn’t have a partnership like this (yet). On the flip side, I have a Norwegian translation in Delicious Library 2, whereas OmniGraffle does not; my Norwegian number is twice that of Omni’s.
My German number is a bit higher, which is surprising because in general Delicious Library doesn’t appear to have penetrated overseas as well as Omni has. Maybe Germans love to organize books? Maybe they love doodads, and they bought my optional Bluetooth scanner, which makes their gross sales number much higher? There’s clearly a whole post to be written about mining this data, but this isn’t that post.
Abstract
In this post I’m going to explain to you what internationalization and localization are, how Apple’s tools handle them by default, and the huge flaws in Apple’s approach. Then I’m going to provide you with the code and tools to do localization in a much, much easier way.
Then you’re going to think, “That will never work, because of blah!” and I’m going to respond,
as if I can read your mind or I’ve already had this argument with a dozen developers, “It already did — I used these tools in Delicious Library and
Delicious Library 2 and they've won three Apple Design Awards between them.”
And you’re going to think, “Stupid show-off… why does he keep mentioning those?” And I’m going to say, “Look, in this case I think it’s justified, because the Design Awards are given based largely on the interface of an application, and I’m merely trying to demonstrate that the compromises I had to make in interface design didn’t have a huge impact on the feel of…” and then we’ll go back and forth for a while in an imaginary argument while I try to convince you to do things my way, dammit, because I’m the daddy.
“Internationalization” vs. “Localization”
What is internationalization, and what is localization? Good question! I’m so glad you asked.
First, you should really read
Apple’s documentation on it. I mean, seriously, this should ALWAYS be your first target of inquiry. Did you read it yet? How about now? Now?
Ok, fine… briefly, ‘internationalization’ is getting your app
ready to be localized for different languages/countries, and ‘localization’ is the process of actually adding a particular language to your app. The two are obviously are very closely related (you can't localize until you internationalize, and there's no point in internationalizing if you're not going to actually do any localizations) but it's vital to understand they are two distinct steps.
For instance, as a developer, you don't need to speak German to get a German localization — you just need to do the internationalization part and find one friendly deutschophone to do the localization, and you're golden. (I mean
literally golden, as in covered in gold — don't ignore the German market! It's 8% of my total sales!)
Apple's Localization Process
Ok, you’re sold on localization. But how do you proceed? Another good question! (Again,
thank you for tossing me all these softballs.)
First off (as I’ve said many-a-time), when you are writing your source code you should surround EVERY string that the user might eventually see in the
NSLocalizedString() macro (or a variant). This will enable you to later automatically pull the English phrases out of your app, for translation by people who actually, like, speak other languages. Errrr… whatchacallums. “Polyglots!”
But then what? Apple’s recommended process is:
- You run Apple’s command-line tool, ‘genstrings’, over your source files (hopefully as part of your build process in Xcode), to pull out all the English phrases you marked with NSLocalizedString() and its cousins. See, I told you that’d be useful! And here’s it’s already paid off, only a couple paragraphs later!
- You give the “.strings” files that genstrings generates, along with any XIBs and image files that contain English words, to a localizer, who is a native speaker of the language you want.
- The localizer creates a new directory of translated “.strings” files, plus XIBs and images with English words translated as well.
- You take this directory, and put in into a language bundle in your project, like “ja.lproj”.
Easy? Sort of. But you can already see problems:
Problem 1: Localizers Can’t Effectively Localize ImagesYou really
really don’t want your localizers editing your image files. I mean, your artists spent a lot of time making your images really nice (since you’re
not an idiot and you didn’t try to do your artwork yourself) and they used things like
layers and
paths and
filters and stuff, and your localizers don’t know how to use Photoshop and aren’t artists, and furthermore you probably gave your localizers the final PNG or JPEG2000 files instead of PSDs and so they don’t have all those layers and paths and filters and they’d do a really crappy job if they even could do it which they can’t.
Also, your artists are going to keep tweaking the images right up until the day you ship (and after — and then they’re going to beg you to delay) and you could spend 100% of your time just sending out updated images to localizers for re-translation, and your localizers are either going to charge you a fortune or get pissed off and quit (depending on if you pay them or not, respectively).
Luckily, there’s a simple and obvious solution to this one: DO NOT EVER PUT WORDS IN YOUR IMAGES.
this is wrong, wrong, so wrong.Yes, I KNOW Apple does it. I think it’s some bizarre hold-over from their Carbon days, when drawing
actual styled text on a 3D button using
actual source code was CRAZY TALK. Plus Apple has a huge in-house art staff, so it’s easy for engineers to say, “Hey, art dudes, give me a button in three different states that says, ‘Buy Album’ in our latest iTunes-only-style-that’s-not-quite-like-Cocoa’s-standards,” and the engineer is done for the day and didn’t have to write any code and can go suck down a mojito.
Never mind that every time Apple’s (separate-but-equal) iTunes UI team changes the look of their buttons, the artists have to redo several
hundred images and then re-localize them, resulting in
thousands of fiddly images. That’s certainly a lot more efficient than the programmer spending a couple hours writing code to draw the button border and inside and the shadow on the text,
once, and then just modifying that code in
one place whenever the look changes, right? RIGHT? (
* see answer at end of post.)
Luckily, you don’t even have the option to do it this stupid way, because you don’t have an art department whose time you’d like to waste. So, use icons whenever possible on buttons, and if you
must use text, then make it localizable. Luckily, this is pretty easy in Cocoa, since the button edges, insides, and the various shines and shadows are
all done for you if you use the standard mechanisms, and they’re easy to subclass, once, if you want a custom look.
For button titles that require text, you can either type English words directly into Interface Builder, or if you’re not on an iPhone you can bind the button’s title some method in your code and use NSLocalizedString() to make sure it’s localized. Since you’re going to have to localize most of your XIBs anyhow (because of menu items, mouseover help text, explanatory text boxes, field titles, window names, etc.), I really only recommend the binding approach if your button’s title changes in a way you can’t model in Interface Builder itself. (That is, don’t write an extra method if you don’t have to. Less code is better.)
Problem 2: Localizers Can’t Effectively Localize XIBsAsking your localizers to modify XIBs directly (using Interface Builder) is a huge pain: First, you limit your pool of potential localizers if they have to have the developer tools installed AND understand how to use them.
hope the localizer finds this "display pattern!"Second, XIB files aren’t flat or self-documenting, so it’s VERY hard for localizers to find EVERY last English word in your XIB. Did they forget a tooltip? What about an alternate title for a button? What about a button’s title binding’s formatter string? Maybe there’s a hidden view somewhere? You’ll have to keep testing the localizations you get for complete coverage, and sending them back to be redone, and then re-testing them again. And again. Yuck.
Sure, there’s a cool command in Interface Builder now, where you can hit control-S and see all the strings in your interface — but that’s ANOTHER thing you have to teach
each localizer to do. In addition to knowing how to use Interface Builder in the first place.
Third, XIBs are like source code: they are written by programmers and contain functional parts. If your localizers happen to delete a button, or disconnect a binding,
your program stops working for that language. Remember, your localizers are NOT coders — they don’t have the same innate fear of changing XIBs that you’ve learned from years of boning yourself. And how fun is it to debug a program that works differently in different languages? Not fun.
Fourth, many language are not as compact as English: the French and Germans are particularly fond of using the descriptions with the lots of the words or compoundwordstodescribeasingleconcept, respectively. You’re asking your localizers to resize your buttons and titles, and then, if those don’t fit in their containers, resize your views and widgets and panels as well.
Let me restate that last point:
you are asking your localizers to design your interface. If you’re making an application that catalogs users’ books, CDs, DVDs, and other physical media, then I urge you to go ahead and do this. For the rest of you, WHAT THE HELL ARE YOU THINKING?
Problem 3: Localizers Can’t Localize from Only Your AppIn the old days, the NIBs we shipped were the same as the NIBs we used to build the app (and to localize), so any polyglot user in the world could just make a new language folder in our app wrapper by copying our English resources, and then start localizing the NIBs and .strings files, and at any time she could launch our app and check her progress. (Which is good, because it’s AMAZINGLY common for localizers to screw up the punctuation in .strings files, and Cocoa’s localization machinery will just silently ignore all strings after any punctuation mistake, resulting in mystery partial localizations.)
Sure, the user had to understand Interface Builder, but at least if she did, she had everything she needed.
Nowadays, Apple doesn’t want people mucking about CHANGING their programs, so we compile XIBs down into read-only NIBs, which means we have to give localizers our original XIB files (as well as our .strings) before they can start work.
Think about the difference between some user, somewhere, just deciding to start editing a folder full of files that she already has, versus her having to write you, you having to bundle up all your XIBs and .strings and send them to her, her having to edit them all without seeing ANY results while she is doing it, then bundle them up and send them back to you, then you have to integrate them into your build system and make a test release, and either look at it yourself or send it to her again.
Lather, rinse, repeat, lather, stab, stab, stab.
Even if this
weren’t a horrible, time-wasting process (it is), this points to the final and biggest problem with localization:
Problem 4: You Have to Maintain Multiple Copies of Each Localized ResourceIf you have nine translations and each has a localized copy of every XIB in your application, then you’ll have to manually maintain nine different versions of each XIB. The strings in XIBs don’t change super-often, but the connections, flags, layout, and other object properties change all the time. And every time you change a flag or layout or connection in one XIB, you have to do the same thing eight more times. And do it perfectly, or you’ll have a language-specific bug.
Of course, you can wait until all your XIBs are fully finished before you localize, but, if you’re like me, you are still fiddling with your XIBs until the day you ship your 1.0 code. So you’d have wait to localize until AFTER shipping 1.0, which stinks, or put off shipping until you’ve finished the product, sent out the localizations, AND gotten them all back — which stinks. And then you find have to change some buggy XIBs for your version 1.2, and you’d still have the exact same problem of having to edit nine XIBs to make one change.
(This is obviously also true of images that have been localized, but hopefully I’ve talked you out of EVER doing that, so I’m not going to discuss that further.)
Now, Apple has provided some tools in an attempt to make this all easier — they have something called “ibtool” (“nibtool” before Leopard) which can pull all the strings from a file, and then put new ones in their place, supposedly. In fact, it’s intended to allow you to look at the geometry and language changes between two localized versions of the same XIB, so you can generate, say, a Japanese version of your Main Menu nib with the latest changes from your English Main Menu nib, except keeping changes that you specifically made in the geometry of objects in the German version (since some words are longer and/or shorter in other languages, and thus XIBs sometimes require relayout).
Apple also has a tool called
AppleGlot which sets up an entire “translation environment.” It apparently dates from Carbon days, and although it’s been updated to work with some versions of Interface Builder last time I tried it, it was a lot more trouble than value to me.
Finally, there are some third-party tools that attempt to simplify some of this: for example
Polyglot, but it appears to have not been updated in a while, and I’m honestly not sure what all it does. (I’m providing the link so you can research it and other products if you want.)
I disagree with the entire approach of storing a bunch of geometry diffs to your XIBs — this seems INCREDIBLY fragile to me. What happens when you move a text field from one view to another? What happens if you even move views around? All those geometry changes turn to gooblity-gook.
The fundamental problem with every tool I’ve seen to fix your XIBs, however,is that
you still have to maintain all the localized versions.
You have nine
extra copies of each XIB in your source code,
you have to make sure they are all synced up, somehow.
Now, if I told you, “Look, it’s easy to localize your Objective-C files — just maintain ten copies of each! Every time you change one, you’ll have to change all the others… but don’t worry! I’ve provided some tools that kind of help with this… until they don’t…” What would you think?
You’d spit in my eye. And I don’t like spitty eyes. So, I came up with a better way, with exactly the same solution Apple used for Objective-C files.
An Almost Ideal Solution
Let’s design, in our heads, the perfect localization system, or as close as we can get without, you know, having to do a lot of work:
• First, we simply promise ourselves we won’t put words in image files. There, that solves a lot. This is usually Apple’s convention as well, although even certain Cocoa teams (-cough-iWeb-cough-) have used fully-rendered images of words like “PUBLISH” instead of using strings — hey, THAT’S not going to be resolution-independent, is it? (I keeed! I keeeeeeed!)
• For XIBS, what we’d like (as programmers) is to have and maintain ONE single English XIB, which will polymorph to the user’s chosen language
at launch time, so we don’t have to ship with ten versions of the same XIB (they’re surprisingly big) or maintain ten versions of the same XIB in our source code (with the concomitant increase in errors as our XIB localizations slowly drift out of sync).
• If we (the programmers) change a string or the whole look of a XIB, we want partial localizations to still work — it’s fine with with us if sometimes we ship with one or two English words “peeking through” a localization, since in most countries (obviously not France) they mix in English words all the time anyhow. It’s certainly better to have a partial localization (as long as it is of high-quality) than either no localization or a corrupted one. (This goes against a programmer’s natural inclination to insist that solutions be provably perfect. Get used to it.)
• To make our localizers’ jobs easier, what we’d like to do is give them ONLY .strings files, so all localization takes is any text editor – localizers don’t have to know how to use Interface Builder or any developer tools.
• We’re forgetful, so we want to generate the strings files from our XIBs and source files every time we do a build, so we never give out-of-date versions to our localizers — if they have a build of the app, then they have the latest strings files, AND the means to test their localizations themselves.
• Since our app is going to ship with the complete set of .strings files needed to localize it, ANY customer who speaks another language can copy our English.lproj to TheirLanguage.lproj, edit the .strings files, and voila create a localized version of our app after a couple hours’ work.
Here's the Code
Now, as you might guess, I've already done this, and, if I may plug myself for a second, this code was already available to clients of
Golden % Braeburn, which is my other company where I give my Delicious store source code (and other miscellaneous helpful code) to Mac developers for a paltry percentage of sales. (Yes, Golden % Braeburn has launched, and yes, one of Golden % Braeburn's clients, Acacia Tree Software, is
currently selling its app with our store.)
The first step to slurp all the English strings out of your XIB files at build time, so your localizers have nice flat strings files to work with. Add two new build phases to your application target in Xcode using “Project ▶ New Build Phase ▶ New Run Script Build Phase”, set the shells to
/bin/zsh, and have them look like this:
|
# -q silences duplicate comments with same key warning
genstrings -q -o ${TARGET_BUILD_DIR}/${UNLOCALIZED_RESOURCES_FOLDER_PATH}/${DEVELOPMENT_LANGUAGE}.lproj ${SRCROOT}/**/*.[hm] |
|
foreach nibFile (${TARGET_BUILD_DIR}/${UNLOCALIZED_RESOURCES_FOLDER_PATH}/**/*.nib)
stringsFilePath=${TARGET_BUILD_DIR}/${UNLOCALIZED_RESOURCES_FOLDER_PATH}/${DEVELOPMENT_LANGUAGE}.lproj/`basename ${nibFile} .nib`.strings
xibFile=`basename ${nibFile} .nib`.xib
xibFilePath=`echo ${SOURCE_ROOT}/**/${xibFile}`
if [[ -e ${xibFilePath} ]] {
ibtool --generate-stringsfile ${stringsFilePath}~ ${xibFilePath}
${BUILT_PRODUCTS_DIR}/xibLocalizationPostprocessor ${stringsFilePath}~ ${stringsFilePath}
rm ${stringsFilePath}~
}
end |
Now comes a slight hitch — under Leopard and beyond, the “ibtool” command
idiotically outputs a new format of .strings file that looks like this:
/* Class = "NSButtonCell"; title = "Get iPhone & iPod Touch App";
ObjectID = "1401603"; */
"1401603.title" = "Get iPhone & iPod Touch App";
/* Class = "NSTextFieldCell"; title = "Remote Libraries:";
ObjectID = "1401604"; */
"1401604.title" = "Remote Libraries:";
Instead of the standard file format (output by the older “nibtool” and the current “genstrings”):
/* Class = "NSButtonCell"; title = "Get iPhone & iPod Touch App";
ObjectID = "1401603"; */
"Get iPhone & iPod Touch App" = "Get iPhone & iPod Touch App";
/* Class = "NSTextFieldCell"; title = "Remote Libraries:";
ObjectID = "1401604"; */
"Remote Libraries:" = "Remote Libraries:";
This new format completely violates the whole .strings file paradigm and obviously makes it impossible for us to use this to localize XIBs on-the-fly, because when we load XIBs on our own,
we don't have those ObjectIDs so we can't match objects up to the strings they need. Nice going, Apple.
So, I wrote a little program to post-process “ibtool”s output to make it like “nibtool” and “genstrings”, you should add this code to your Xcode project and add a new “tool” target for it and make sure your main app depends on this tool being built, and all will be well:
|
// xibLocalizationPostprocessor.m
//
// Created by William Shipley on 4/14/08.
// Copyright © 2005-2009 Golden % Braeburn, LLC.
#import <Cocoa/Cocoa.h>
int main(int argc, const char *argv[])
{
NSAutoreleasePool *autoreleasePool = [[NSAutoreleasePool alloc] init]; {
if (argc != 3) {
fprintf(stderr, "Usage: %s inputfile outputfile\n", argv[0]);
exit (-1);
}
NSError *error = nil;
NSStringEncoding usedEncoding;
NSString *rawXIBStrings = [NSString stringWithContentsOfFile:[NSString stringWithUTF8String:argv[1]] usedEncoding:&usedEncoding error:&error];
if (error) {
fprintf(stderr, "Error reading %s: %s\n", argv[1], error.localizedDescription.UTF8String);
exit (-1);
}
NSMutableString *outputStrings = [NSMutableString string];
NSUInteger lineCount = 0;
NSString *lastComment = nil;
for (NSString *line in [rawXIBStrings componentsSeparatedByString:@"\n"]) {
lineCount++;
if ([line hasPrefix:@"/*"]) { // eg: /* Class = "NSMenuItem"; title = "Quit Library"; ObjectID = "136"; */
lastComment = line;
continue;
} else if (line.length == 0) {
lastComment = nil;
continue;
} else if ([line hasPrefix:@"\""] && [line hasSuffix:@"\";"]) { // eg: "136.title" = "Quit Library";
NSRange quoteEqualsQuoteRange = [line rangeOfString:@"\" = \""];
if (quoteEqualsQuoteRange.length && NSMaxRange(quoteEqualsQuoteRange) < line.length - 1) {
if (lastComment) {
[outputStrings appendString:lastComment];
[outputStrings appendString:@"\n"];
}
NSString *stringNeedingLocalization = [line substringFromIndex:NSMaxRange(quoteEqualsQuoteRange)]; // chop off leading: "blah" = "
stringNeedingLocalization = [stringNeedingLocalization substringToIndex:stringNeedingLocalization.length - 2]; // chop off trailing: ";
[outputStrings appendFormat:@"\"%@\" = \"%@\";\n\n", stringNeedingLocalization, stringNeedingLocalization];
continue;
}
}
NSLog(@"Warning: skipped garbage input line %d, contents: \"%@\"", lineCount, line);
}
if (outputStrings.length && ![outputStrings writeToFile:[NSString stringWithUTF8String:argv[2]] atomically:NO encoding:NSUTF8StringEncoding error:&error]) {
fprintf(stderr, "Error writing %s: %s\n", argv[2], error.localizedDescription.UTF8String);
exit (-1);
}
} [autoreleasePool release];
}
|
Finally, add this class to your main project. Whenever you load a XIB file now, this code will be invoked, and it’ll run through all the displayed strings in your XIB and see if there is a localization in the correspondingly-named strings file. (Eg, if you load “MainMenu.xib”, it’ll automatically be localized with strings from “MainMenu.strings”.)
|
// DMLocalizedNibBundle.m
//
// Created by William Jon Shipley on 2/13/05.
// Copyright © 2005-2009 Golden % Braeburn, LLC. All rights reserved except as below:
// This code is provided as-is, with no warranties or anything. You may use it in your projects as you wish, but you must leave this comment block (credits and copyright) intact. That's the only restriction -- Golden % Braeburn otherwise grants you a fully-paid, worldwide, transferrable license to use this code as you see fit, including but not limited to making derivative works.
#import <Cocoa/Cocoa.h>
#import <objc/runtime.h>
@interface NSBundle (DMLocalizedNibBundle)
+ (BOOL)deliciousLocalizingLoadNibFile:(NSString *)fileName externalNameTable:(NSDictionary *)context withZone:(NSZone *)zone;
@end
@interface NSBundle ()
+ (void)_localizeStringsInObject:(id)object table:(NSString *)table;
+ (NSString *)_localizedStringForString:(NSString *)string table:(NSString *)table;
// localize particular attributes in objects
+ (void)_localizeTitleOfObject:(id)object table:(NSString *)table;
+ (void)_localizeAlternateTitleOfObject:(id)object table:(NSString *)table;
+ (void)_localizeStringValueOfObject:(id)object table:(NSString *)table;
+ (void)_localizePlaceholderStringOfObject:(id)object table:(NSString *)table;
+ (void)_localizeToolTipOfObject:(id)object table:(NSString *)table;
@end
@implementation NSBundle (DMLocalizedNibBundle)
#pragma mark NSObject
+ (void)load;
{
NSAutoreleasePool *autoreleasePool = [[NSAutoreleasePool alloc] init];
if (self == [NSBundle class]) {
method_exchangeImplementations(class_getClassMethod(self, @selector(loadNibFile:externalNameTable:withZone:)), class_getClassMethod(self, @selector(deliciousLocalizingLoadNibFile:externalNameTable:withZone:)));
}
[autoreleasePool release];
}
#pragma mark API
+ (BOOL)deliciousLocalizingLoadNibFile:(NSString *)fileName externalNameTable:(NSDictionary *)context withZone:(NSZone *)zone;
{
NSString *localizedStringsTableName = [[fileName lastPathComponent] stringByDeletingPathExtension];
NSString *localizedStringsTablePath = [[NSBundle mainBundle] pathForResource:localizedStringsTableName ofType:@"strings"];
if (localizedStringsTablePath && ![[[localizedStringsTablePath stringByDeletingLastPathComponent] lastPathComponent] isEqualToString:@"English.lproj"]) {
NSNib *nib = [[NSNib alloc] initWithContentsOfURL:[NSURL fileURLWithPath:fileName]];
NSMutableArray *topLevelObjectsArray = [context objectForKey:NSNibTopLevelObjects];
if (!topLevelObjectsArray) {
topLevelObjectsArray = [NSMutableArray array];
context = [NSMutableDictionary dictionaryWithDictionary:context];
[(NSMutableDictionary *)context setObject:topLevelObjectsArray forKey:NSNibTopLevelObjects];
}
BOOL success = [nib instantiateNibWithExternalNameTable:context];
[self _localizeStringsInObject:topLevelObjectsArray table:localizedStringsTableName];
[nib release];
return success;
} else {
return [self deliciousLocalizingLoadNibFile:fileName externalNameTable:context withZone:zone];
}
}
#pragma mark Private API
+ (void)_localizeStringsInObject:(id)object table:(NSString *)table;
{
if ([object isKindOfClass:[NSArray class]]) {
NSArray *array = object;
for (id nibItem in array)
[self _localizeStringsInObject:nibItem table:table];
} else if ([object isKindOfClass:[NSCell class]]) {
NSCell *cell = object;
if ([cell isKindOfClass:[NSActionCell class]]) {
NSActionCell *actionCell = (NSActionCell *)cell;
if ([actionCell isKindOfClass:[NSButtonCell class]]) {
NSButtonCell *buttonCell = (NSButtonCell *)actionCell;
if ([buttonCell imagePosition] != NSImageOnly) {
[self _localizeTitleOfObject:buttonCell table:table];
[self _localizeStringValueOfObject:buttonCell table:table];
[self _localizeAlternateTitleOfObject:buttonCell table:table];
}
} else if ([actionCell isKindOfClass:[NSTextFieldCell class]]) {
NSTextFieldCell *textFieldCell = (NSTextFieldCell *)actionCell;
// Following line is redundant with other code, localizes twice.
// [self _localizeTitleOfObject:textFieldCell table:table];
[self _localizeStringValueOfObject:textFieldCell table:table];
[self _localizePlaceholderStringOfObject:textFieldCell table:table];
} else if ([actionCell type] == NSTextCellType) {
[self _localizeTitleOfObject:actionCell table:table];
[self _localizeStringValueOfObject:actionCell table:table];
}
}
} else if ([object isKindOfClass:[NSMenu class]]) {
NSMenu *menu = object;
[self _localizeTitleOfObject:menu table:table];
[self _localizeStringsInObject:[menu itemArray] table:table];
} else if ([object isKindOfClass:[NSMenuItem class]]) {
NSMenuItem *menuItem = object;
[self _localizeTitleOfObject:menuItem table:table];
[self _localizeStringsInObject:[menuItem submenu] table:table];
} else if ([object isKindOfClass:[NSView class]]) {
NSView *view = object;
[self _localizeToolTipOfObject:view table:table];
if ([view isKindOfClass:[NSBox class]]) {
NSBox *box = (NSBox *)view;
[self _localizeTitleOfObject:box table:table];
} else if ([view isKindOfClass:[NSControl class]]) {
NSControl *control = (NSControl *)view;
if ([view isKindOfClass:[NSButton class]]) {
NSButton *button = (NSButton *)control;
if ([button isKindOfClass:[NSPopUpButton class]]) {
NSPopUpButton *popUpButton = (NSPopUpButton *)button;
NSMenu *menu = [popUpButton menu];
[self _localizeStringsInObject:[menu itemArray] table:table];
} else
[self _localizeStringsInObject:[button cell] table:table];
} else if ([view isKindOfClass:[NSMatrix class]]) {
NSMatrix *matrix = (NSMatrix *)control;
NSArray *cells = [matrix cells];
[self _localizeStringsInObject:cells table:table];
for (NSCell *cell in cells) {
NSString *localizedCellToolTip = [self _localizedStringForString:[matrix toolTipForCell:cell] table:table];
if (localizedCellToolTip)
[matrix setToolTip:localizedCellToolTip forCell:cell];
}
} else if ([view isKindOfClass:[NSSegmentedControl class]]) {
NSSegmentedControl *segmentedControl = (NSSegmentedControl *)control;
NSUInteger segmentIndex, segmentCount = [segmentedControl segmentCount];
for (segmentIndex = 0; segmentIndex < segmentCount; segmentIndex++) {
NSString *localizedSegmentLabel = [self _localizedStringForString:[segmentedControl labelForSegment:segmentIndex] table:table];
if (localizedSegmentLabel)
[segmentedControl setLabel:localizedSegmentLabel forSegment:segmentIndex];
[self _localizeStringsInObject:[segmentedControl menuForSegment:segmentIndex] table:table];
}
} else
[self _localizeStringsInObject:[control cell] table:table];
}
[self _localizeStringsInObject:[view subviews] table:table];
} else if ([object isKindOfClass:[NSWindow class]]) {
NSWindow *window = object;
[self _localizeTitleOfObject:window table:table];
[self _localizeStringsInObject:[window contentView] table:table];
}
}
+ (NSString *)_localizedStringForString:(NSString *)string table:(NSString *)table;
{
if (![string length])
return nil;
static NSString *defaultValue = @"I AM THE DEFAULT VALUE";
NSString *localizedString = [[NSBundle mainBundle] localizedStringForKey:string value:defaultValue table:table];
if (localizedString != defaultValue) {
return localizedString;
} else {
#ifdef BETA_BUILD
NSLog(@" not going to localize string %@", string);
return string; // [string uppercaseString]
#else
return string;
#endif
}
}
#define DM_DEFINE_LOCALIZE_BLAH_OF_OBJECT(blahName, capitalizedBlahName) \
+ (void)_localize ##capitalizedBlahName ##OfObject:(id)object table:(NSString *)table; \
{ \
NSString *localizedBlah = [self _localizedStringForString:[object blahName] table:table]; \
if (localizedBlah) \
[object set ##capitalizedBlahName:localizedBlah]; \
}
DM_DEFINE_LOCALIZE_BLAH_OF_OBJECT(title, Title)
DM_DEFINE_LOCALIZE_BLAH_OF_OBJECT(alternateTitle, AlternateTitle)
DM_DEFINE_LOCALIZE_BLAH_OF_OBJECT(stringValue, StringValue)
DM_DEFINE_LOCALIZE_BLAH_OF_OBJECT(placeholderString, PlaceholderString)
DM_DEFINE_LOCALIZE_BLAH_OF_OBJECT(toolTip, ToolTip)
@end |
Unresolved Issues
Issue 1: Not all XIB properties are localizedI didn’t have any NSTableViews with standard column headers in Delicious Library, so there’s no code to handle NSTableViews at all in here. I also don’t even attempt to localize bindings strings, I believe. So, that’d be a good thing to do!
I haven’t touched this code in, like, four years. Which says a lot about how well it’s worked for me, but, obviously, there’s room for improving it. So, please pimp *my* code, and send it back to me, and I’ll post your changes. (We’ll all be better off for it!)
Issue 2: You can’t use the same phrase with two different meanings in a single XIBImagine if you have, say, a button labelled “Wind” for “what you do to a watch,” and a text field labelled “Wind” for “moving air.” You obviously can’t localize ‘em with this method. Hasn’t proven to be a big deal with me, but this is clearly what motivated Apple to completely hose string the file format for “ibtool.”
Issue 3: iPhone limitationsI never got around to porting this to work on the iPhone, since
Amazon made me destroy the iPhone version of Delicious Library, but it’d be pretty easy to make this work with hierarchies of UIViews and UIControls instead of their NSCousins.
I don’t think the swap-methods-at-load technique will work on the iPhone, either, so you’d have to manually call the translation method when you loaded a XIB, but I don’t think that’s a huge deal, since on the iPhone you typically are MUCH more conscious of when you load and unload XIBs.
Also, obviously your customers couldn’t simply get the .strings from your app bundle or add their localizations themselves, so you lose those advantages. Still, this is the technique I’ll be using in my future iPhone and apps.
Issue 4: You can’t change the geometry of XIBs depending on the localizationThe method requires that we, as programmers and interface designers, agree to make ALL our text fields and checkbuttons wide enough to allow for the wordiest of languages, since we’re localizing on-the-fly.
Everyone raises this as an objection, but, seriously, the alternatives are MUCH worse — either you maintain ten sets of geometries yourself, OR you let your localizers be your interface designers. Both are horrible.
And, seriously, you’d be surprised how little people notice when you have extra space. With checkboxes, just always stretch ‘em to the edge of the screen. With text fields — well, the user can’t tell HOW wide they are, can she? The only problem one is buttons, and I think those look better with extra space around them. (Update: my German localizer wrote me to tell me he disagreed with this — that it was a huge pain for him to find short enough German words. I apologized to him for not telling him he could request more width.)
I’ve even thought up two ways around this issue: one if obvious, which is you could just store some geometry changes based on the container type & string in a separate file, and merge those changes at run-time. But, that’d be hard to maintain, for reasons I’ve gone over a bunch by now.
The other would be cooler: have your widgets re-lay themselves out at run time based on their resizing properties if their localized titles don’t fit. For example, if button “a” has a button “b” 20 pixels to the right of it, and both buttons have the stretchy spring to their right, then it’s a fair guess that if you grow/shrink button “a”, you need to do so from its right edge, and you need to move “a” by that many pixels, as well. The other combinations of stretchy springs can be assigned values, and all you have to do is use them consistently in your XIBs and you’ll be golden. (You do have to watch for accidentally localizing Apple’s XIBs.)
But, seriously, I haven’t really done either of these yet, and no English speaker has ever said, “Hey, you have too much space around this button!”
And, importantly, I’m able to integrate all thirteen localizations of Delicious Library 2 by myself, and still have time to program and run my company and drink lots of mojitos.
Footnotes
*No, it isn't.
Labels: business, code, mac community
Friday, August 21
First off, I should mention that I don't think I use the word "heuristic" correctly – although in computer science it's grown to replace the word "algorithm" as just a general term for a way to solve a problem, traditionally it has a more narrow definition:
Heuristic (/hjʊˈrɪs.tɪk/) is an adjective for experience-based techniques that help in problem solving, learning and discovery.
(Yay, it's fun to copy and paste from Wikipedia!)
When I use it, I usually am talking about an algorithm that won't
always give the correct solution, but does so often enough that the algorithm is useful. This differs from a classic algorithm, where we struggle mightily to make it
provably correct in
every instance.
But classic computer programming has largely failed, because it failed to copy nature. Nothing in nature works 100% of the time, but it sure works well MOST of the time – and when it fails, well, you die and get replaced. A human being, for instance, is an absolutely amazing machine, and is provably NOT provably correct.
--
To talk about computer heuristics, we'll need to get concrete. Could I get a "for instance?" Heck, yes.
For instance: NSDateFormatter has the following method:
|
- (id)initWithDateFormat:(NSString *)format allowNaturalLanguage:(BOOL)flag;
|
Ignoring for a second that 'flag' should be
really be renamed 'allowNaturalLanguage' (I mean, honestly, if your method body refers to a variable named 'flag' it's not at ALL obvious what you mean, is it? You'd have to look at the method definition every time you saw this 'flag' variable, and that's just poor coding.) uh I lost my train of thought.
Oh! Yes. 'allowNaturalLanguage.' It's really cool! In most places you can enter a textual date in Cocoa (but
not those fiddly NSDatePicker widgets where each number is in its own field), you can enter dates like, "Next Tuesday at noon" or "yesterday at 5:23PM" and Cocoa will magically turn that into a valid NSDate.
Possibly more importantly, you can type "Oct 16, 1969" or "10/16/69" or "10.16.1969" and it'll figure out what you meant my birthday in each case. (You
cannot just type "Wil Shipley's Birthday" but that would be a great extension of their existing heuristic, if you ask me.)
This frees users up from having to figure out what magic combination of digits, dashes, slashes, words, and/or abbreviations comprise a valid date. Without this flag the programmer specifies exactly what format dates must take, like, "mm/dd/yyyy", and if the user doesn't type
exactly that information, she gets an ugly error panel. With the flag, the user doesn't have to learn what the computer wants:
she can continue to do things the way she has done them and the computer will understand her.The latter is the touchstone of great design: we must strive to make our programs require as little learning as possible on the user's part. Each little thing they have to learn about our program is another obstacle to them using it fully, another tiny chunk of enjoyment stripped from their experience.
Now, a few releases of OS X ago (I believe 10.4), 'allowNaturalLanguage' was marked as deprecated; soon to be removed from the APIs. "What what what‽" said I. I filed a bug: "Why?"
The response was, essentially, the current heuristic doesn't work perfectly even in English, and fails badly in foreign languages.
That may seem like a logical reason to remove a piece of API,
if you are a programmer. If you're a user, you're probably thinking, as I did:
this is the worst reasoning in the world.
Let's say 65% of Mac OS X users speak English primarily. They were all enjoying not having to type dates and times the way the computer wanted. 65% percent of the users were just a
little bit more happy with their experience on Mac OS X. And,
crucially, the other 35% who didn't speak English had
no idea what they were missing. It didn't hurt them at all to not have this functionality, it just didn't help them, either.
--
Life isn't fair, and programming is even less fair. Programming is all about picking a certain class of users with a certain specific class of problems, and making their lives much MUCH better. Like, if I didn't listen to music, I wouldn't care about iTunes. If I didn't take photos of my girlfriends naked, iPhoto would add nothing to my life... but that would be OK by me. iTunes and iPhoto don't have to please everyone in order to be good. They just have to please
some people, and should please those people
a lot.We talk a lot about the 80% solution, which can be summarized thusly: Will 80% of your users think this feature / heuristic / bug fix is good? Then do it.
That rule seems obvious, really; the value of this rule is in remembering its obverse: if a feature / heuristic / bug fix is only going to help, say, 20% of your customers, you need to prioritize it lower.
It's easy when programming to get seduced into doing something really super-duper-cool no matter how obscure, but we have to remember our time is finite: spend your time where your users are going to see it. Do a GREAT job on those areas. (Don't do a shitty job on the other areas - skip them entirely.)
--
In Leopard, '-[NSDateFormatter allowNaturalLanguage]' is no longer marked as deprecated – I won that battle. But there's also a new date widget that makes entering dates a much more graphical affair, which democratizes the happiness. Clear graphics trump heuristic input methods – I use the widget now in my programs, unless I'm parsing text files, in which case I use 'allowNaturalLanguage.'
Everyone wins, now.
--
So, we switch to another heuristic, which requires a bit of background:
Amazon recently started requiring that all requests to their product catalog API be digitally signed with the secret password of a registered Amazon associate – one might assume they discovered stealing other people's associate codes is rampant, and they want to crack down on programs and websites that are violating their rules and using their APIs with stolen identities. A reasonable thing.
I rewrote the lookup code for both Delicious Library 1.7 and Delicious Library 2.2 so they will digitally sign requests, so my customers could keep looking up books and DVDs and stuff. (I don't think there are a lot of 1.x users still out there, but even so I didn't want to force them to pay upgrade to 2.x just to keep using my program.)
Now, since I've changed how I look up items, immediately after the release of I asked my support team to immediately prioritize all bugs reporting lookup problems, since you modify some area of code you want to alert your support people to actively look for failures in that particular area, so you can fix them immediately instead of waiting until they become a huge issue.
It turns out there
was a tiny issue (searching for a book by title fails if its title has an apostrophe in it - fixed in 2.2.1) but while looking at user's bugs I discovered something more interesting: a number of users were reporting lookup failures because they were typing in ISBNs the way they see them on the boxes, eg: 978-0-316-01876-0, and they were getting no results, because Amazon stores ISBNs without dashes, eg: 97803780316018760.
Hrm. Now, my first response to this problem (a year ago when I'd just finished 2.0 and was exhausted) was, "Dammit, just don't type the damn dashes. Who looks up things manually by ISBN anyways? There's like twenty easier input methods, including dragging URLs in from Amazon or scanning with the iSight or typing the author's name... Geez." There was actually a lot more cussing than that, actually, but luckily I have Terry as a buffer layer between me and the customers, so it comes out as, "I'm sorry, we don't accept dashes in ISBNs or EANs right now..."
But after seeing bug reported again, I realized I'm violating my cardinal rule: I'm making users learn some picky input method that only exists because of Amazon's particular database formatting. And, worse, there's no good way for them to learn this rule unless they write us.
I should mention that there's only one search field: the user can type in numbers, author names, titles, or even keywords, and we just hand that stuff off to Amazon and let it do a fuzzy search.
So, how do we solve the dashes issue? Let's run through the solutions we think of, until we hit the best one.
|
| Add text near the search field: "Omit dashesWhen entering ISBNs or EANS" |
Advantages of this method: • It's easy for us to add a field to the NIB. • The user can actually learn from this without having to write us.
Disadvantages: • We're teaching the user something that's not generally useful, instead of learning from her. • There's another damn text box on or page, which is one more graphical thing calling for the user's attention, and we've learned from bitter experience that every widget you add to a window is like killing a kitten. • This field requires localization.
|
- (IBAction)findMatchingItems:(id)sender;
{
self.keywordsString = [self.keywordsString stringByReplacingOccurrencesOfString:@"-"
withString:@""];
[...search...]
}
|
Advantages of this method: • One line of code, sweet. • User doesn't have to learn anything: ISBNs and EANs can be entered with or without dashes, work either way.
Disadvantages: • User has lost the ability to search for titles where dashes are meaningful. Eg, "The Mythical Man-Month" would be turned into "The Mythical Manmonth," and Amazon may fail to find THAT, now (actually it does in this instance, but let's not rely on THEIR heuristic working in all cases.).
|
- (IBAction)findMatchingItems:(id)sender;
{
NSString *noDashesString = [self.keywordsString
stringByReplacingOccurrencesOfString:@"-" withString:@""];
BOOL containsOnlyDigits = YES;
for (NSUInteger characterIndex = 0; characterIndex <
noDashesString.length; characterIndex++) {
containsOnlyDigits &= [[NSCharacterSet decimalDigitCharacterSet]
characterIsMember:[noDashesString characterAtIndex:characterIndex]];
if (!containsOnlyDigits)
break;
}
if (containsOnlyDigits)
self.keywordsString = noDashesString;
[...search...]
}
|
Advantages of this method: • User doesn't have to learn anything: ISBNs and EANs can be entered with or without dashes, work either way. • User can still search for titles with dashes in them.
Disadvantages: • User has lost the ability to search for titles that are ONLY digits and dashes, for example, if a user were searching for a book about the latest
quack diet, "10-10-10," she'd end up searching for "101010," and maybe not finding it.
|
- (IBAction)findMatchingItems:(id)sender;
{
NSString * noSpacesOrDashesString = [[self.keywordsString
stringByReplacingOccurrencesOfString:@"-" withString:@""]
stringByReplacingOccurrencesOfString:@" " withString:@""];
BOOL containsOnlyDigits = YES;
for (NSUInteger characterIndex = 0; characterIndex <
noSpacesOrDashesString.length; characterIndex++) {
containsOnlyDigits &= [[NSCharacterSet decimalDigitCharacterSet]
characterIsMember:[noSpacesOrDashesString characterAtIndex:characterIndex]];
if (!containsOnlyDigits)
break;
}
if (containsOnlyDigits) {
switch (noSpacesOrDashesString.length) {
case LIISBNDigitCount: case LIUPCDigitCount: case LIEANDigitCount:
self.keywordsString = noSpacesOrDashesString;
default:
break;
}
}
[...search...]
}
|
(While I was in there, I decided to remove extra spaces if I pass the tests to remove extra dashes, so users can now also type "978 0 316 01876 0" and that will work, as well. It seemed like it might be as common, and it cost me almost nothing to add.)
Advantages of this method: • User doesn't have to learn anything: ISBNs and EANs can be entered with or without dashes, work either way. • User can still search for titles with dashes in them. • User can still search for titles with only decimal digits and dashes in them, as long as the number of digits doesn't happen to form a valid EAN, UPC, or ISBN.
Disadvantages: • Kind of reveals that I am a crazy person.
[Update: several sharp-eyed readers have pointed out I neglect to check for "x" or "X", which is valid as the last digit (the check digit) in the older ISBN-10s. Thank you... I'm human, too!]
[Update update: Reader Ian Stoba wrote me a note and suggested a more clever (and actually less code for me) heuristic, which is to remove the dashes and check the checksum (last digit) to see if the number that remains is a valid ISBN-10, UPC, or EAN. Since I already have written the methods to do these checks, this is very simple, and pretty fail-proof.]
--
So, that is the algorithm I went with. Let's evaluate this in terms of our goals for any good heuristic (like the one from NSDateFormatter):
- It has to help some class of users – it helps users who type the dashes in ISBNs, UPCs, and EANS, and it helps them a lot, because before they had no clues how to proceed when lookups failed.
- It has to not harm other users – it almost never will, because it won't change the input at all unless the user happens to be searching for an author or title that is all numbers and dashes AND has exactly 10, 12, or 13 digits it.
- It shows the user what it's doing, so if the heuristic does fail the user will understand why – in this case, we replace the contents of the text field in which the user just typed her number with our new (dashless) number, and so if she really WERE searching for a book whose title was, say, "1-2-3-4-5-6-7-8-9-10-11," she'd see that was replaced by "1234567891011" when she did the search, and at least have a clue why the search failed.
--
Heuristics are the key to designing programs that work well with humans, that make humans smile. In college computer science classes, we learn all about b*trees and linked lists and sorting algorithms and a ton of crap that I honestly have never, ever used, in 25 years of professional programming. (Except hash tables. Learn those. You'll use them!)
What I do write – every day, every hour – are heuristics that try to understand and intuit what the user is telling me, without her having to learn
my language.
The field of computer interaction is still in its infancy. Computers are too hard to use, they require us to waste our brains learning too many things that aren't REAL knowledge, they're just stupid computer conventions.
It's up to us to fix this.
Labels: code, interface design
Sunday, May 24
This document describes our process for reviewing applications for iPhones and iPods touch submitted to the iTunes App Store. We’ve avoided using legalese in this document so that you’ll actually read the whole thing. Please do so
before starting to write your application, so that you won’t waste a bunch of time writing an application that we cannot publish.
We hate having to reject anyone’s application, so we want to make it clear that we have very compelling motivations behind our acceptance policies, so you can adhere to their spirit. (Oh, our lawyers just reminded us that we should mention that circumstances, times, and social mores change, and we reserve the right to change these policies at any time for any reason, but we’ll do our best to update this document to always reflect the current state of affairs.)
FIRST, you must do no harm. We cannot allow applications that will mess up users’ iPhones, or interfere with their normal operation in any way. The iPhone is a vital communications device, and any downtime on it is unacceptable to Apple and to our customers.
While viruses, malware, and apps that can be remotely exploited are obvious targets of this rule, there are more subtle implications to this. For instance, we cannot allow applications that send too much data through the cellular network; not only would we be breaking our agreements our cellular carriers (who have priced their data plans based on an estimated average data use of customers
without including your application), but overloading the network would make it unusable for
everyone, and thus violates our first rule.
We also do not allow emulators or other computer languages right now, because those kinds of applications are notorious for having subtle security holes, and we must take the cautious path to ensure the iPhone remains stable for our customers.
SECOND, if your application is what a reasonable person would consider offensive, we require it to carry an “M” rating on our app store, which carries with it certain restrictions on who can purchase and use your app.
We do this because, to a certain extent (and despite our best efforts), users do not fully distinguish between third-party authors like yourself and Apple; if something runs on an iPhone and it is ‘bad,’ that is (at least somewhat) considered to be Apple’s fault. Apple abhors censorship in any form, but it also recognizes that even within a single society there are different ideas about what is acceptable and unacceptable, and we would like to warn our customers (yours and Apple’s) who might be more sensitive.
We realize that any attempt to categorize
anything into “offensive” and “inoffensive” is a fool’s errand, especially considering that your application’s audience will encompass thousands of cultures around the world. Thus, we will use our “best efforts” to determine if an app might be found offensive: for example, if it is overtly sexual, if it contains slurs or curse words, if it has violent themes — these are topics reasonable men disagree on, and our goal is to flag anything that might be controversial so that, for instance, parents can review it with
their guidelines before letting their children see it.
RESUBMISSION of your app can be done if you believe that it has been rejected in error — please include in your resubmission a description of what you’ve changed in your app, or why you believe the original judgment was in error.
Because in the end our judgments are made by humans, and humans are variable and fickle creatures, our policy is to always let resubmissions be judged by a different person at Apple, to get another perspective.
REFUNDS are opt-in for developers in the iTunes App Store: you can set the number of days during which Apple will offer a full, no-questions-asked refund to your customers, which number will be displayed prominently in your application’s listing on the iTunes App Store and on the user’s list of installed apps inside iTunes.
While the number is up to you, we generally recommend that the more your application costs, the longer a refund period you offer; if you are making a $20 application and your customers are “done with it” after two days, you may not be offering a good value for money spent. As well, customers will be inclined to trust software that offers a generous refund policy.
Please note that Apple’s checks to you will be delayed by the length of the refund period you offer; we will not act as your creditor to cover refunds.
FINALLY, note that these rules actually exist only in the fevered and every-hopeful imagination of one Wil Shipley. Consult the real iTunes App Store for its actual policies, not a blog. Dur.
Labels: advice, business, mac community, random ideas
Sunday, September 28
Or, I Test-Drove a Tesla and All I Got Was These Lousy Adrenaline Shakes
Short version: Tesla.
--
Long version: I've had insomnia since I was a wee lad, and at night I often calm my brain by thinking up idealized objects. Optimal house cooling, unpowered water purification, the ultimate graphics card, the coolest car
ever. It's a good exercise, I think: if you don't have an idea of what you'd like something to become, you won't ever get it there.
Some of these crazy dreams have come true, which is particularly disturbing. When I was 10 or so I thought up a crazy idea for fonts that would be described by curves, and rendered in real-time by the graphics card at the best possible resolution, instead of having hand-tuned bitmaps. Of course, I was 10, so I had no idea what the math would be, I just figured, THIS would be the coolest, ultimate form of fonts. A few years later PostScript was in printers, and a few years after that Display PostScript hit the desktop, then TrueType, and now nobody even remembers bitmap-only fonts.
Also as a kid I'd try to think up the ultimate car... What if you had no gearbox - you just had one gear and an electric motor that could spin as fast as you wanted directly driving the wheels, so the mapping between the accelerator pedal and your speed would be practically linear — the way kids imagine
all cars work until they actually learn to drive and have to figure out gears and clutches and all that.
But, wait, this ideal car would have so much power, you'd end up jump off the road if you accidentally used the slightest bit too much pressure on the accelerator. Oh, I know, the car would have a computer controlling the wheels, and it'd detect when you started to slip and automatically compensate. But, hold on again, isn't the fun in driving being able to burn out and spin around and stuff? Maybe we need the ability to disable the computer, sometimes.
--
And, now, again:
Tesla. Here it is; the crazy dream-car that a kid might design in his naivety of how the world really works, except some engineers didn't get the memo on what's possible, and went ahead and built it.
Tesla.
It's crazy-fast. It handles like a jet fighter. It gets the equivalent of about 140 mpg. It has no gears. It requires almost no maintenance.e It's gorgeous. It's whisper-quiet. And, in Seattle, runs off hydro power.
--
So, yes, I test-drove a Tesla today, for five laps on a closed course in the parking lot of a defunct K-mart. Then I took my supercharged Lotus Elise around the same course. (The Tesla I drove was an engineering prototype but is said to be very close to the one I'll get next year.)
I expected the Tesla to have more power than the Elise (based on the Tesla's 0-60 of about 4 seconds, as opposed to the supercharged Elise's estimated 4.5-7-ish), and it didn't disappoint.
The first part of the course was a straight, wide-open acceleration towards two tiny cones, which were where you were supposed to brake "as hard as you can" so you didn't end up flying out of the lot and into the street. It turns out they were conservative on this by a bit; by my fifth lap I'd found the Tesla could brake down enough to make the turn in about half the space they'd given me. The car was nimble.
And, compared to the Lotus, the Tesla seemed faster to the cones; at first not by a huge amount, but the huge difference came when the Lotus hit the top of its first gear it started stuttering from the electronic limiter, and I had get out of gear and shift. The Tesla just kept accelerating the whole time, so by the time it hit the cones it was going WAY faster.
Remember: this car is NOT an automatic: there are no damn gears. It's not shifting for you; there's no annoying pauses where the car decides what gear you might want, right in the middle of your burn. The engine is basically hooked up directly to the damn wheels. It's amazing. It has that responsive feeling of when you are driving a normal car in first gear, except it has that no matter how fast you are going.
Taking the 'S' cones in the Tesla, on the second or third lap I was finally able to get the tires to chirp if I turned at the highest speed I could get out of the turn. But the car never slid out of control, even a little bit. The nose was always pointed where I wanted, and the back tires were always ready to accelerate.
This was a huge advantage of the Tesla; I'm not a race-car driver, and I admit that it's pretty hard for me to take a sharp turn at high speed AND be shifting in the Elise. Hell, it's enough hard to hold the damn steering wheel with
two hands at that speed. In the Tesla, at any point during a turn I could tap the accelerator and I was (a) guaranteed to be in gear, and (b) guaranteed to be in the RIGHT gear. Because, you know... there's only one gear, and it was "fast" gear.
Call me a wuss if you like, but I found it a lot more fun to be concentrating on just pointing the wheel and calculating the amount of slide, and NOT trying to shift and steer and clutch all at once. I'm just not that good.
The big surprise from the Tesla was when I took my Elise through the same course. The Elise is several hundred pounds lighter and has stickier tires on the front and the same on the rear.
And the Elise slid around the cones out-of-control. I'd take a cone at high speed and the Elise would start jump-skipping its tires sideways, and steering and acceleration were offline until it'd stop. I'd lose momentum and end up kind of off-course.
I don't really have a good explanation for why the Elise handled worse for me than the heavier Tesla. Maybe having precise control of the accelerator throughout turns enabled me to keep the tires spinning in the road-wise direction? Maybe the Tesla's slightly longer wheelbase or different weight distribution made a big difference? Maybe the traction control on the Tesla is really that good? I don't know.
I do know that, going around the cones as fast as I could, the Elise lost control sliding sideways for a little bit on both laps, where the Tesla merely chirped her tires and drove on.
Normally, features like "traction control" or "no shifting" put me off of a car, because they essentially translate to "low-performance-idiot-mode." You turn them on and you feel like the car is driving and you're a passenger. I've hated 'em in Ferraris and Mercedes SLs.
With the Tesla, these features are put in to allow you to drive
harder and
faster and still feel completely in command. *I* am driving. I feel the road. I feel the wheels.
--
The only drawbacks I found were: the Tesla seemed a little slow immediately off the line - full-throttle starts have a slight pause, and THEN you are pressed back against your seat. I guess this is the price you pay for the One Gear also allowing you to go 100 mph (or so). The other drawback is that, yah, the Tesla really doesn't want to maintain a speed above 100 mph for very long. It just isn't designed for that.
Sure, I don't drive about 100 very
often, but, you know — sometimes it's nice. Not a deal-breaker, though.
--
In Seattle, I've been told a law just got passed where we pay no sales tax on very efficient vehicles, so I avoid the ~9% extra charge. Also, it looks as though the president will sign into law a bill that gives a $7,500 tax
credit (NOT a deduction, but a full CREDIT), to people that buy a car with a battery at least as big as the one in the upcoming Chevy Volt, and the Tesla happens to qualify as well.
This doesn't make the Tesla exactly cheap, but it's sure nice to save $16,000.
--
The Tesla people were uniformly cool and
real and fun. During the open house there wasn't a single question they dodged. They offered up problems they'd found, troubles they'd had getting into production, issues that loom on the horizon.
They test-drove the car themselves because they still all love it so much. One guy turned off traction control (we were not allowed to) and demonstrated how much power the car really has — it was, frankly, daunting, and he got waved down after two laps because the guy in charge thought he was going to power-slide into the defunct K-mart. (A very real possibility.)
My friend commented as we left how nice the Tesla executives were. I am excited about this company, again. Excited to be little tiny part of a team that is so committed to changing the world AND having fun. Excited to have a dealer that is a "company store," where the salespeople aren't on commission, and really
want to help you love their product.
I like to spend my money with companies whose philosophies align with mine: in my life I've only made one purchase at Wal*mart, I don't eat Domino's pizza, I have vegetables delivered from small organic local farms, etc, etc, blah blah blah. My point is, I am *happy* to give these guys my money.
I am happy to have a chance to say, with my dollars, "You guys are doing the right things, and I support you."
--
For sale: Ardent Red 2005 Lotus Elise, with aftermarket supercharger. One driver, excellent condition.
Labels: random ideas
Monday, September 22
Call me a proponent of free markets, but I think Apple needs to have a clearly-documented policy for approving submissions to the iPhone App Store, and it should be:
•
Publish all
software submitted to Apple, as long as the software isn't actively harmful to users, illegal, and does not violate Apple's agreements with cell phone vendors.•
Period.--
The iPhone app store is, at heart, incredible. Every software developer's dream store looks a lot like this:
- 100% of the devices that can run my software have a link directly to this store.
- And only this store.
- And the user can't remove the link to this store from the device.
- There's no other way to buy software, so users are never confused as to whether they should go to some website or physical store or the online store to find software for their devices.
- Users never wonder if there's some other, better software out there - if it's not on the store, it doesn't exist.
- Users can buy with a click.
- Software is instantly installed and enabled for users.
- "Good enough" copy-protection is handled invisibly for all developers.
- Apple hosts the software makes the pretty, professional website.
- Credit card transactions are handled automatically.
- Apple's percentage is MUCH lower than traditional distribution.
- The store actually pays out the money it owes you, unlike the vast majority of physical distributors.
- There's already a market of something around ten million users for iPhone apps.
- The market is increasing by the day.
That's a LOT of plusses. A LOT. And it's working. Developers are reporting making thousands to hundreds of thousands of dollars every MONTH and the store is only a couple months old.
Some of this is likely because of the novelty of the device and the store itself — there's a mini-gold rush effect happening, and already I suspect that if you weren't one of the guys to get rich selling a flashlight app or sudoku, well, you probably shouldn't start writing one now.
So, yay Apple, and yay developers who are rolling in fat, filthy lucre. (I'm not bitter that I didn't get in on that first round. No, no.)
As with any pioneering effort that succeeds (c.f. Twitter's constant whale-fails when it took off), Apple is encountering problems it never anticipated, and having to make up solutions on-the-fly.
Which is fine, and good, except, well... maybe we developers need to give Apple a loving nudge, so the problems are solved in a conscious way that helps everyone, instead of being solved ad-hoc and turning into policies which punish us all.
--
Problem: Most Software is CrapThere's a LOT of crap out there for the iPhone. A LOT. And a bunch of neat apps. How does the the user tell them apart? Should Apple's model be like Nintendo, where Apple only allows software through that meets their rigorous standards for being fun and cool and stable? It sounds nice, except (a) it requires a ton of effort on Apple's part, and Apple's success or failure is determined entirely by the tastes of the people doing the vetting, and (b) it stifles innovation. (Look at the number of titles available for the Nintendo Wii, which has been out for years, vs. the number for the iPhone, whose SDK has been available for months.)
The other problem with Apple vetting apps for quality is that Apple gets blamed if crappy apps slip through the process. Once you appoint yourself censor, you've taken responsibility. If an App Store app crashes, it'll be blamed on Apple. If an App Store app has a bug, it'll be blamed on Apple. If an... well, you see where I'm going.
--
Recently Apple decided to go ice-skating on the slippery slope of censorship by removing the "I am Rich" application from its store. Briefly: some prankster priced an app at $999 that did nothing but show some text and a picture, congratulating the purchaser for being rich and stupid. Apple pulled the app after a few days, citing "not enough functionality" or some such.
Now, this application did point real problems in the
system, but not in the
app. The problems are in the App Store, and they are: it's not really clear how to get refunds, and it's a little TOO easy to click on something that says "$999" without realizing that, seriously, this is a grand you're blowing.
Let's solve the
real problems so that we don't
need to censor apps, and so that developers don't need to guess if their apps are "functional" enough to pass muster with whichever App Store censor they happen to get:
• Apple needs to have a clearly posted refund policy that applies across-the-board. They may already have a policy, but, honestly, I've bought 15 or so apps and I've never seen it, and I'm going to say that if users don't see the policy, you might as well not have it.
I'd suggest something like, "You can get a full refund any time in the first two weeks of ownership of any app." This would solve many problems: if the app turns out to be buggy, or have limited functionality, or insult your mom, or whatever... well, it's not Apple's problem any more. They refund your money and everyone's happy.
• For apps over some threshold ($30? $100?), Apple needs to add a click to the purchase process. Something like, "Note: this is A HUNDRED REAL LIVE SMACKERS, here, so MAKE SURE you really want this, OK?"
--
After that first rejection, there have been two more reports of rejections. I can't verify them myself, of course, but I also have no reason to doubt the reports. One of these applications had 'podcasting' as part of its functionality, and one had fetching mail from Google as part of its functionality.
Both were censored because of a new criterion Apple has invented, which is "duplicates existing functionality." Let me make my position on this perfectly clear: it in unethical and antithetical to the whole IDEA of an App Store for Apple to be censoring applications based on criteria they have
never given to developers, and only told developers
after the developers put in all the work of writing an app.
Even TV network censors produce a "standards and practices" document, so writers can tell if they are pushing the envelope. Apple's censors have acted capriciously and against the interests of all of its developers, its customers, and itself.
This situation is worsened because it's obvious that Apple is only worried about applications duplicating the functionality of
Apple's iPhone applications — there are twenty "sudoku" apps and a dozen "flashlights" and a bunch of pokers and, heck, there's more than one racing game.
But it was only when a developer added functionality that Apple considered sacrosanct to
Apple itself that she was censored. Apple wasn't worried about customer confusion, Apple was worried about getting some competition.
I have to be clear: it simply will not stand for Apple to prevent applications on the iPhone from competing with Apple's own applications. Besides chasing away all decent developers, besides hurting their customers by stifling competition and innovation, besides it simply being evil, it will, shortly, be illegal. This kind of behavior is illegal when you hit a certain point in market saturation for your product; Microsoft was slapped for it constantly in the late '80s. If the iPhone is the success Apple thinks it will be, they will find themselves the target of a huge class-action lawsuit.
--
I can see how the iPhone App Store could be some short-sighted Apple marketing dude's dream: "Hey, we can nip all competing applications in the bud and completely own any market we choose! Imagine how well Final Cut Pro would do without Premiere! Imagine iPhoto without Lightroom! We own it all, baby!"
Those of us who actually write software know that, in fact, killing your competition is a sword that's not just double-edged, but in fact has a blade as its handle, as well.
Without competition there is no innovation. Apple
needs competing apps. As they add features or speed or UI innovations, Apple can copy them and make Apple's apps better.
Competition is how nature has made strong organisms since literally the beginning of time. You simply won't get stronger if you don't have adversity. It is demonstrated in any system you can think of, from virus resistance in operating systems to the relative strength of the huns versus the northern Chinese.
There's a simple proof of why competing apps should exist: (1) If customers use the third-party app, it clearly provides some functionality Apple's version does not, and customers benefit and the platform is stronger. (2) If customer do not use the third-party app, that app withers and dies and nobody is hurt.
But, ignoring how Premiere actually helps Final Cut, let's imagine a world in which Apple DID censor Premiere and Lightroom for "duplication [Apple's] existing functionality." What do you think Adobe would do with Photoshop? Flash? InDesign?
If you voted, "Make those suckers Windows-only," give yourself a gold star. Now think about how not having those applications would have affected where the Mac market is today. (Remember the lag in selling Intel machines until Adobe made Photoshop "Universal?" Imagine if it didn't run
at all.)
Now imagine the next revolutionary application for phones, and what platform it's going to be on if Apple doesn't cut this crap out. (Hint: rhymes with "manbloid.")
--
"What about all the crap-ware? Aren't decent applications getting buried be all the stuff that's just being dumped out there in hopes of a few pity clicks?"
This is actually surprisingly easy to solve. Eventually, there are going to be tens and tens of thousands of apps on the App Store. Just simply paging randomly through applications to find one is already far too onerous to be practical.
The App Store needs to think of itself as two different parts - it already implements these parts, but the people who run the store need to understand that
these two parts are fundamentally separate:
• Part one is a giant warehouse, where every piece of software that is
not actively harmful is kept in case someone wants to buy it (remember, users can always get a refund). This warehouse can be searched with titles and keywords or an item can be directly linked.
• Part two is like a traditional storefront, with limited real estate, so only the best or coolest applications are highlighted. It's a recommendation engine, that highlights popular, highly-rated, or innovative applications.
Everyone can get into the warehouse. Only the select few can get into the storefront.
Customers win because they can choose whatever software they like, regardless of whether Apple "approves" of their choice or not. Apple wins because developers aren't alienated and don't all go develop for Android, and so Apple has the device where all the innovation is happening. And developers win because the obviously cool apps will be featured by Apple and get tons of his, but even if their app isn't "blessed" by Apple, if it's a neat enough idea it'll become popular on its own, through word-of-mouth.
--
It's a huge mistake for Apple to appoint themselves arbitrator of what's cool, or to even
appear to do so. It's an equally huge mistake for Apple to decide that all innovation must come from Apple.
Let's list a handful of cool Apple apps: Safari. iTunes. Preview. Mail. iSync.
Did Apple invent the ideas or protocols behind any of these? Nope. Did Apple write the first implementations? Nope. Did Apple even write the original code they are using for their versions? Nope. (They licensed them all from third parties, except for Mail.)
When the next cool app comes out, and the next one after that, is it going to be on the iPhone, or on Android? It's really Apple's call.
--
UPDATE 9/23: Apple's response is reportedly to
put the rejection letters under nondisclosure, as well. That's, uh, not a solution, guys. In fact, it's the opposite. It makes you look more draconian. And it's a useless gesture.
Do you REALLY think developers are not going to talk among ourselves, or leak info to the press, after we've worked for months on an application and then had it capriciously rejected by Apple? All the press has to say is, "we've heard of several developers who have been rejected" and there's nothing you can do; you can't subpoena people who aren't under NDA, and you won't know who among your NDA'd rejectees talked.
Seriously, Android is open and free. The tighter you try to clench your fists, the more developers you are going to drive away. Yes, you have the nicest frameworks and the prettiest hardware. But that's only the first part of what you need.
Labels: business, mac community
Tuesday, July 29
I guess I should first admit I hate the show
Punk’d. I mean, here’s a guy who is famous for lying about his age so he seems hipper, telling us that his show’s purpose it to deflate the big egos on other stars, and show them what truly matters in life. So he sets up situations where anyone would get upset, and then laughs when he upsets people. I call *cough*bullshit*cough*. (Also *cough*jerkface*cough*.)
So I have to admit I’m not predisposed to like
The Mojave Experiment, where Microsoft took a bunch of “regular folks” XP users who were afraid of Vista, and told them Microsoft was going to show them a secret new operating system — which was actually Vista.
UNSURPRISINGLY, these people mostly said they liked Vista.
Now, if you read this blog, you know I pretty much hate Microsoft, because of their incredibly shady business practices (moreso in the early 1990s) and their shoddy products, most especially their operating systems, whose crappy user experience and programmer interfaces hold back the advance of technology. However, I’m not going to rail on Vista here. Seriously, I’m not.
What I
am going to rail on is this “experiment.” (I use that word advisedly.)
--
I
hate bad science. Hate it. Hate. So let’s look at not one, not two, but FOUR, yes FOUR (ah-ah-ah!) key flaws in this experiment, any single one of which would render its results meaningless:
•
The Placebo Effect: Every time I do a software release, no matter how minor, even if I just change one word, in French, to another French word, someone will send me mail or post on a forum, “Thanks, this release seems a lot faster!” Do I make fun of them? Or videotape them and put it on a blog? No. Because it’s just human nature. If we are told something is new-and-improved, we prime ourselves to believe it (c.f.
Blink by Malcolm Gladwell, which I’ll refer to again in a bit) and make it so in our minds.
This is why we have, for example, blind taste tests: because humans are proven to not be able make dispassionate judgments about subjects they already know about. So, if you say to someone, “Hey, I’m giving you a top-secret peek at a new operating system from Microsoft, you’re incredibly lucky and special, and I really value your opinion!” of COURSE they are going to like it. They almost can’t not like it.
•
The Pepsi Challenge Effect: “The Pepsi Challenge” was a blind taste test that Pepsi overwhelmingly won (again, from Blink). Yet, most people still drink Coke. Why? Gladwell’s thesis is that a
single sip of a soft drink is very different from drinking a
whole can, which is the smallest unit most people imbibe. Pepsi usually wins the challenge because it's a sweeter drink, and initially people respond to this extra sweetness. But after drinking a can, Pepsi becomes cloying.
So, here I am, sat down in front of Mojave-err-Vista, and all I've ever used is XP. Well, look, nobody is doubting the graphics are prettier in Vista. It looks nice compare to XP (it
should — they hired the guy who designed Aqua for Mac OS X).
I play with Mojave, and, yes, some system tasks are easier. Again, nobody doubts there are things that work much better. When I plug my iSight camera into Vista it shows up as a device and offers to let me take pictures in the Vista Explorer thingy. That’s kind of cool! Hey, I kind of like Mojave-nee-Vista!
Except,
those glossy features aren’t why people downgrade from Vista to XP. Those are
not the reason people hate on Vista!
Now, again, look — I don’t use Vista or XP for anything but games. I liked using Vista better, until the new UFO (X-Com) game that I had played great on XP, and wouldn’t launch at all on Vista. Then I bailed. That’s my story. There are apparently hundreds of others.
You, personally, may never have encountered a piece of hardware or an app that didn’t work on Vista, and you might be perfectly happy with it. I’m not going to try to argue you out of that happiness. My point is that
the problems that Vista has become famous for are not the kinds of problems you encounter in a few minutes of playing with it in a controlled environment.Vista is known for people initially
liking it, then after a while discovering it’s not working for them, and “downgrading” to XP. This study has told us exactly what we already knew: that,
initially, people like Vista. (
Initially, people like having sex without condoms, too... it’s simply not a very good criterion all by itself.)
•
The Perfectly Controlled Environment Effect: Microsoft set up the hardware. Microsoft brought the accessories. Microsoft picked the software. Microsoft sat people down with Vista experts driving the mouse, and walked people through Vista. What an INCREDIBLE SHOCKER that in this INCREDIBLY TIGHTLY CONTROLLED ENVIRONMENT Vista performed OK!
Microsoft had set up an environment with a philosophy similar to Apple’s: “Look, we work well with this hardware and software, and too bad if you want something different.” Unfortunately, that’s NOT why people choose Windows. They hack together their own machines, and they want their software to still run.
Did
any of these customers bring in their favorite games and try to play them? Did they bring in their graphics tablets and discover they fail?
Did
any of the test machines ever say, “Oh, I’m sorry, Windows Genuine Advantage has determined that you may be running an invalid copy of Windows, so please jump through these hoops or we’ll disable some of your hardware”? I’m going to guess no. But I’ve seen this message a lot. And I own three valid licenses to Windows.
•
The Personal Tutor Effect: If you sit
anyone down with an expert in a particular program, and the expert walks them through the features and answers their every question, chances are good that person is going to report that she had a good experience with the program. Very good, indeed.
Personal training is so important to customer experience that Apple thinks of it as a key asset of its Apple Stores. But Microsoft doesn’t have Apple Stores in real life. Or any analog. It’s you and a box with a holographic sticker on it. Good luck!
--
Microsoft has managed to prove that if you have a friendly expert on a controlled machine (with Vista pre-installed) showing a carefully selected subset of Vista features to an ignorant XP user for a few minutes, the XP user will often say he finds Vista acceptable. Wow.
This so-called experiment of Microsoft’s is an insult to science, and to our intelligence. And I am
dying to see the out-takes from their shoot. I mean, how many people do you suppose like being told, “Hey, this giant, unpopular monopolistic software company just made an ass out of you! Ha ha! Our leading scienticians just PROVED that you LOVE VISTA and WANT TO MARRY IT. You are TOTALLY GAY for Vista! Haaaaaaa HAAAAAAA!”
Vista may or may not be an upgrade in user experience for most Windows customers. I personally prefer the feel of Vista over XP when the former works as well as the latter, but Vista has failed me on several occasions, and I also don’t enjoy running games MORE slowly than XP.
I've got to imagine that the Microsoft customers who took all the damn time to upgrade their machines to Vista, determined it was unworkable, and then had to take all the time to go BACK to XP,
probably did so for a reason, possibly even a
valid reason, and not because they had been swayed by bad word-of-mouth. I further imagine that these customers are
completely livid at having Microsoft not say, “Oh, sorry, we’ll get right on those bugs,” but, instead, “You’re just stupidly following the crowd, and if you’d just free your mind up, you’ll discover you actually love Vista... hater.”
Is “Our Customers Are Stupid and Have No Idea What They Really Want” really Microsoft’s new mantra?
Again, wow.
Labels: business, mac community, stories
Monday, July 7
Last week, a customer reported a bug to me in
Delicious Library 2: when he first launched version two, his version one data would start to be imported, but after waiting for tens of minutes watching the annoying aqua progress bar creep along, his entire machine would crash. Every time.
His collection was so huge it took me two solid days to download his version one files onto my machine. When I ran Delicious Library 2 on my Air in debug mode, it converted his data for about thirty minutes, then crashed with an exception that said the database file was 'corrupt'. I tried to look where it had crashed, but XCode was reporting that
all my source code files were gone. In fact, so were my iTunes songs, and everything in my Documents folder. I opened Finder and Terminal, and verified that the directories were indeed empty. No error messages, no residual files. Just empty. Panic.
A minute or so later, all these files magically came back. Everything. As if nothing had happened.
After thinking about this overnight (while staying away from the computer), I changed one word in my source code, and the customer's file loaded perfectly, in only twenty minutes.
Part 1: TriageIn my personal pantheon of bugs, this report was triaged at the top. Roughly, my triage order is:
- Data-loss bugs
- Unavoidable crashers
- Functionality-blocking bugs
- Avoidable crashers
- Avoidable bugs
- Misfeatures
- Performance issues
- Feature suggestions
- UI feedback
This is, of course, a rough ordering: obviously if a customer is trying to publish a single item to her website and it's taking a half hour, then I'm obviously going to bump that up the list over someone whose app crashes when she leans on the "a" key for a three hours in the "Actors" field. (For the record, neither of those are real bugs.)
But, note that in the first case, the "performance" issue has made the feature essentially unusable, so it's really a functionality-blocking bug. Further, if publishing is synchronous, then this bug is blocking access to the application for an unacceptable amount of time, and could be considered a crasher of sorts.
Weighing the bug my real user reported, we see that it's The Perfect Storm, a trifecta that crashes the app (and the
machine!), every time (on launch!!), blocks all functionality, and, to add insult to injury, it takes a long time to do so. (
The food wasn't very good, and the portions were very small.)
This bug was suddenly at the top of my list.
Part 2: Machine CrashersHow can machines crash due to user programs? The machine-crashing bug is exceedingly rare in shipping applications; in my experience, there are two primary causes: some system resource gets used up by the program, or the program confuses the window server (the graphics system) and the machine's primary interface locks up.
As an operating system matures, it's usually harder and harder to confuse the window server (I haven't seen such a bug in OS X in a long time), but resource over-allocation problems remain.
One of the goals of the operating system's designers is to not allow programs running in user space to ever crash the entire machine. Some would say it's a primary goal, even. But in real life, most operating systems can be brought to their knees (if not fully knocked-out) by user code.
Consider a simple application that just allocates memory in a tight loop. As the operating system runs this application and starts running out of free physical RAM pages, it starts throwing away the pages used by other applications, in a process that goes kind of like this: oh, you haven't used mail in a few seconds, I'll throw that away for now, and, uh, also this web browser, and, uh, damn, the window server's backing store, and, uh, this essential system font server, and, uh...
Pretty soon you've effectively locked up the system, just because it's going to take so long to page in any particular resource
the windowing system needs to run that by the time it's loaded off the disk your rogue process has caused some
other vital page to be unloaded.
Now, the internet is full of flames about how this is completely unacceptable, and OS designers are 100% to blame and yadda yadda yadda. These are, in my opinion, probably from computer science PhD students who believe in a perfect world of provable programming and the Easter Bunny.
Here in the realm of actually making money, if running your program causes a user's computer to crash, she doesn't care if it's Apple's "fault" — she's going to post all over the interwebs that
your program sucks, and ask
you for her money back. Now, since you're
not a PhD student, you like money, so this is bad.
And, honestly, why blame the OS designer? Your app was written incorrectly. It was going to crash or be killed by the OS anyhow. This is a bug you need to fix either way. And the user would be pissed at you even if the OS catches your app and says, "Sorry, this application has gone crazy, we're killing it."
You can see why OS designers don't spend all their time worrying about such issues. It's not as if a lot of real malware is written to crash machines — like biological viruses, successful software viruses don't
kill the host, they co-exist with the host and use a minimum of resources while replicating themselves. Viruses that kill the host go extinct very quickly.
Further, to allow programmers to get the most out of the hardware, OS designers
need to let us live right on the edge. The more blocks and checks they put between us and the hardware, the slower our programs are going to run. Imagine if the Apple said to programmers: "Ok, in order to not let you dominate the hardware, we will only let you use 50% of the CPU at any one time." Now imagine you're a user, running World of Warcraft on your brand-new $4,000 ultra-lux machine, and you're only getting half the framerate you
should be, and
would be under, say, Windows.
You'd be pissed. And Apple would lose a customer.
InterludeLook, I don't want to seem mean. I'm sure she's a very nice person in real life, and probably quite smart. And, she's a very handsome woman, I'm not denying that. But, honestly, Jessica Alba can't act. It's just not a skill she has.
It's OK. Not everyone was meant to be an actress. But you really have to admit your limitations to yourself. I know, for instance, I'll never be a supermodel. I'm OK with that. I'd make a lousy president, too. I accept it.
It's time to stop, Jessica. You're costing people money. And YOU would be happier doing something you know you're good at, that you could feel good about. There are LOTS of professions for preternaturally beautiful women out there. You could be one of my assistants, for instance.
Part 3: Diagnosis? Delicious!How do you even
begin to fix a bug where things are going great for a while, then your machine crashes and/or all your files disappear?
Remember that the very
first thing you do, when looking at any bug, before you even start thinking about it, and
long before you look at your code, is replicate it. You can't debug what you can't replicate, and user reports are usually lacking in some details that your trained eye will catch.
At this point, I'd already replicated the crash, and I'd seen a valuable clue the user hadn't seen: my files had disappeared. The key clue.
Next, in a case like this, you'd need to make sure you have a recent backup of your machine, because,
damn. I mean, I've been programming for a long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long time, and I've never seen a bug like that.
Ever.
Ok: your machine is crashing, so you suspect that it's running out of some resource. Memory is the most common resource to run out of and the easiest to find and fix, so you run your program under Instruments, with the leak detection tool. You really should have done this before you released your application, but you were totally burnt-out and you'd been in beta for months and didn't seem to have any major issues, so you failed to do everything just right. Now you're paying.
Your code will run slow as hell under Instruments, but in the end, like that hot gypsy lady at the fair, she will magically tell you, "Hey, you leaked this object exactly here and at exactly this time, so cut it out." It's a small miracle, and the team who wrote this has two to go and they're saints, bitches.
Sadly, at this point I had also already run Instruments / leaks weeks before, and although it had found a major image leak in 2.0 (whoopsie!) which had kept some users from loading very large libraries, and released it in 2.0.1, this fix didn't help our guy. In fact, my virtual memory usage was holding steady at 1.26 GB as I loaded his collection, which is really not bad considering every process on my system right now has between 0.5 to 1.18 GB of VM right now, under no load.
Not memory. Hmm. Now you think. Think think think. I think better if I distract myself from the problem and let my hind-brain take over. So I went and played Assassin's Creed, and murdered some fools who had waged war on their fellow man, or made the mistake of calling me peasant, or gotten in my way, or looked funny, or stood around yelling too much about how Salem Ali is a strong man. Honestly, once you start stabbing people in the neck it's hard to stop.
All the while, I'm swishing ideas around in my mouth to see if they taste right. Graphics memory leak? Often a window server will have its own mini-memory manager: was I overloading it as I converted 40,000 cover images from version one? Maybe, but... Instruments had actually found my leak of graphics memory before, so I kind of trusted that it would have found any others I might have had.
Stab. Stab stab. Gurgle.Also, my
files had disappeared. That doesn't
seem like a graphics problem.
Stab. Man, I love jumping from a tall building and landing with my blade in someone's throat. I am so bad-ass.What resources are related to files? Honestly, I don't know a lot about modern file systems; they are kind of a black box to me. When I went to college, we had integers which represented files. And we liked it!
But Mach had introduced a new kind of I/O, called memory-mapped I/O, that I was using extensively. A possible culprit?
Part 4: Memories of Memory MappingBack when NeXTstep 0.8 was first unveiled, one of its key revolutions was it was based on not just plain UNIX, but Mach UNIX, which was a really damn fast and clever re-write of the lowest layers of UNIX, with some extra fun added in, including cheap and fast inter-process communication. Avie Tevanian had been working on Mach as a PhD student at Carnegie Mellon, and Steve Jobs recognized he was a star and hired him straight away. (Microsoft countered by hiring Avie's old advisor to work on NT, which is kind of like Microsoft hiring my mom because I'm a good programmer.)
One of the great things Mach gave us was a new way to read files, which process was
really slow by default under vanilla UNIX, for reasons I don't fully understand. Mach introduced the metaphor of
memory-mapped files, which used the virtual memory system to map a file on disk onto an address somewhere in VM, without actually loading anything from the disk at first. The cool thing was your program could access any byte in this special VM region, and the correct part of the file would be paged in
on-demand, using the exact same mechanism that the VM system used to extend applications' physical memory by mapping their virtual memory spaces to a large reserved area on the disk.
This turned out to be very fast, because it had none of the overhead of normal UNIX I/O, which usually involved reading a byte or so at a time using a UNIX function, which involved an expensive system trap and had a lot of overhead (checking to see if enough bytes have been read in from the disk and if not reading in some more) compared to just fetching a memory location. In the latter case, once a page was fetched in you can fetch all the other bytes on that page for free, no system calls at all, just a standard memory access. Big win.
NeXTstep 1.0 gave us to memory streams, which were awesome, and NeXTStep 2.0 (new capitalization) used memory mapped files by default when you read files using its fancy new NSData class, and all new applications ran a lot faster and life was good.
Well, mostly good. One problem with memory-mapped I/O is that if the user deletes or moves the file on the disk, the OS can't get to it any more, but the programmer never really knows when this might happen, so she don't know whether to just go ahead and make a copy of the entire file into memory or not. So it's not a good idea to keep memory-mapped files open for a long time.
Another, more recent, problem is that virtual memory on 32-bit systems is capped at 4GB, which means the largest file you can memory-map is a lot smaller than 4GB (since your app is going to use, like, at least 1GB of VM itself). Nowadays, it's not uncommon for users to have 6GB movie files in their iTunes. Oops.
So, for a while now, Apple has been moving gently away from memory-mapping I/O. For example, you have to set a flag (CSResourcesFileMapped) in your applications' Info.plist if you want to memory-map the resource files that the AppKit and Foundation frameworks load for you; otherwise they are read using different (unknown) techniques, which may or may not be slower (see below).
Fortuitously, however, I'd just been part of a discussion on a developer mailing list about optimized file system access, and learned an important fact, which gave me the solution.
Part 5: Everything New is Old AgainI have exalted memory-mapped I/O for exactly twenty years now, but just a week or so ago an Apple engineer told me that the latest tests show that Apple's new-fangled "uncached" access method had at least as good of performance as memory-mapped I/O, with neither of the two big drawbacks listed above. It was The New Way.
Here's a bit from the
NSData page:
NSMappedRead- A hint indicating the file should be mapped into virtual memory, if possible.
Available in Mac OS X v10.4 and later.
Declared in NSData.h
NSUncachedRead- A hint indicating the file should not be stored in the file-system caches.
For data being read once and discarded, this option can improve performance.
Available in Mac OS X v10.4 and later.
Declared in NSData.h
Sadly, there's not a lot of guidance there when you should or shouldn't use either option. Also, there's no hint if the options are mutually exclusive. Why would you use NSMappedRead, and why not? Who knows?
But now I'd heard the Word, and It Was Good. I resolved then and there to look over my code, and see when I did and didn't need NSMappedRead. But I didn't want to just naively trust that NSUncachedRead was
always better (APIs so rarely work like that), so I'd need to do timing tests. So, in fact, I was in the middle of these timing tests (on iTunes importing) when I got this new bug report, so I had some experience with memory-mapped vs. cached access.
Part 6: I Love It When a Plan Comes TogetherOk, I'm reading in 40,000 image files as part of the conversion, and storing them in my database. At some point the filesystem goes on vacation and says, to hell with you, I'm not reading any more damn files, even if they are just directories. Hrmmm. Hrmmmmmmmmmm.
"Oh, please, spare my li-- gurgle!"Maybe the problem was there was some huge bug in Apple's Mach, where if you open too many files in a short period of time, the filesystem tried to, like, cache the results, and the cache blew up, and as a result the filesystem incorrectly just would fail to open any more files, instead of flushing the cache. This made a
little sense, since the problem had gone away spontaneously after a minute or so, like a cache problem would when the cache auto-flushed.
One thing I knew, KNEW wasn't the problem was that I had too many file open simultaneously. Because the way my code worked, I'd open the files using NSData's dataWithContentsOfMappedFile:, do a little massaging on the image inside, and then the files would auto-close themselves when the NSData was autoreleased, which I would correctly do every 100 files I read in.
I
knew this. Knew! I've also been around long enough to know that whenever I
know the operating system must be bugged, since
my code is correct, I should take a damn close look at my code. The old adage (not mine) is that 99% of the time operating system bugs are actually bugs in your program, and the other 1% of the time they are still bugs in your program, so look harder, dammit.
So I traced through the steps. First I read the image file into the NSData. Right. It's auto-released, so I'm not leaking it. Then, I set the CoreData object's data field to point to the same NSData. Still no leak. Then, the NSData is autoreleased, correctly, and the system resources should be freed up.
Wait a minute. Wait. It's autoreleased, but that doesn't mean it's
deallocated. It just means the reference count is decremented. But someone else might still have a pointer to it... in fact, I had
just assigned the same NSData to the CoreData object. Knowing it was an immutable data, the CoreData object didn't make a
copy of my data, it just retained the
exact same NSData object. The one that had an open memory-map of the image file!
CoreData's objects persist in memory until you flush them explicitly, which means anything they are holding also persists. So, I was creating 40,000 memory-mapped files and not freeing any of them. That, uh... seemed fishy. Like, five-course-sushi-dinner fishy.
Now, I could flush these cover objects explicitly using CoreData; I'd have to read in a batch of old objects, convert them, save everything, then flush the covers, then read in another batch. In fact, I was already using batches (following
CoreData's best practice guidelines, see "Importing in Batches"), but flushing would add a layer of complexity. I'd have to keep track of the objects I created separately, and loop through and flush them.
Sure, that's easy code to write. But I'm making a 2.0.2 release, and in x.x.y releases, I want to change as little code as possible to fix the bug, because my beta testers weren't going to be excited about testing a release that adds no features. I will test it myself, of course, and I'd ask anyone who reported the bug to test it, but doing something as heavy-duty as suddenly flushing a bunch of objects
could have side effects, and I wouldn't have enough people to test for those. Yes, in theory flushing
shouldn't have side-effects, but in that same world, no code should ever have bugs and communism is a really nifty idea.
But, I'd been told that NSUncachedRead was as good as NSMappedRead, without as many drawbacks. Would it also not tie up the same system resources? Certainly the word "uncached" seemed promising. I'd try it.
I kind of exaggerated on the "one word" in my preamble — it
would have been a one word change, but I'd used the older interface on NSData:
| NSData *imageData = [NSData dataWithContentsOfMappedFile:fullImagePath]; |
I replaced it with this line (which would say NSMappedRead instead of NSUncachedRead to be equivalent to the above):
NSData *imageData = [NSData dataWithContentsOfFile:fullImagePath
options:NSUncachedRead error:&error]; |
Thus removing the memory-mapping.
Importing the large collection worked perfectly. The problem was solved, and now I could upgrade truly HUGE collections.
Part 7: Where Did We Go Wrong?We had tested Delicious Library 2 with importing an enormous sample file of 10,000 Delicious Library 1 items. However, it took so long to build up such a library in version 1 that we ended up building the sample library using a custom program, and as a result that test file didn't have any images in it. But beta testers were enrolled specifically because they had collections of thousands of items, and we had the largest closed beta of any program I've worked on. We also tested with our friends' Delicious Library 1 files, of about 3,300 items.
But this customer had over 4,400 items in Delicious Library 1. That, it turned out, was a few over a limit we had no idea existed. Damn it.
It's a bug we should have caught. We should have spent the time to get the images in the 10,000 item file. I messed up.
Software is written by humans. Humans get tired. Humans become discouraged. They aren't perfect beings. As developers, we want to pretend this isn't so, that our software springs from our head whole and immaculate like the goddess Athena. Customers don't want to hear us admit that we fail.
The measure of a man cannot be whether he ever makes mistakes, because he
will make mistakes. It's what he does in response to his mistakes. The same is true of companies.
We have to apologize, we have to fix the problem, and we have to learn from our mistakes.
Unfortunately, Delicious Monster has so much mail from releasing version 2, we're failing at responding to customers right now. We are a tiny company, and had one support person, and although we're bringing a second up to speed, it takes time to teach someone how to solve our users' problems. Our backlog of mail messages is finally going down — as of today, we're at 1,300, down from 3,000 a week or so after our release. We still get several hundred messages a day, though, and it takes time just to sort through and see which ones are bugs and which ones are customers needing help and which ones are people just saying, "Hey, that's cool, you know what else would be cool?"
When this customer finally got through to me, I apologized to him for the bug and the crummy response time, and elevated his bug to the top. I let him know what I was doing to fix it, and got the fix done in a couple days. I also gave him a free license to Delicious Library 2, since he'd been unable to buy it yet.
I had already known that when I launched 2.0, I couldn't immediately go on vacation; that I'd have to jump in to the 2.0.1 release for any urgent bugs that customers found. What I didn't realize was how much support e-mail would spike. That's the lesson I learned: I need to have extra people ready to do support when I do a major release. I'm not sure how to do this and not be paying people to twiddle their thumbs the rest of the time, but that's something I'll have to figure out. And I apologize to my customers for this screw-up.
The beta for 2.0.2 is out now, at
http://delicious-monster.com/downloads/Delicious%20Library%202%20Beta/DeliciousLibrary2.zip.
It'll load a ton of items.
Delicious Monster
