64 articles and counting

A Sketch of PO LiveEdit

Frédéric Wenzel has a great series of L10n blog posts going and there has been a lot of discussion of L20n work at Mozilla. I ping’d Fred to check in and share an idea with him…

What if you could directly edit the copy on the staging server of a website and have it transparently save your changes in the background without messing with PO files, SVN access, etc

You’d eliminate lots of problems that localizers have… and provide a richer, fun editing environment.

Like any idea worth building… I found out it’s already been invented :) A few months back when I pitched this idea to Stas Malolepszy, he mentioned that Gandalf already thought of the idea and built a prototype.

I’m realizing that this has been sitting in my TODO list for too long and I’m better off doing a brain dump in this blog post. Here’s my take on the idea:

Design Constraints:

Keep po files as the backing store
Work as much as possible with existing gettext tools
Use direct object manipulation for the UI - pick strings directly from page, see live updates

Requirements:

Let localizer view a web page and pick a string to translate
Display an editor widget that sit on top of or beside the page
Display the msgid and msgstr. In practice this is the English String and the Localized version.
As they type, redisplay the page with their updated translation
Provide a revert button
Provide a save button. Save persists this string back to the po file server which commits the change to SVN.

PO LiveEdit at 10,000 feet

POLiveEdit 10'000 foot diagram
PO Enhanced HTML – In PHP or other web code, instead of making calls to gettext methods, a set of wrapper methods would be used which add metadata to the HTML output. This would basically add the msgid to a span tag that surrounds the output of the msgstr. This would extra metadata wouldn’t affect the display, but would capture the semantics of a localized string.
Old PHP code
<p><?= _(‘Be the Difference’); ?></p>
outputs
<p>Fai la differenza</p>

New PHP cde:
<p><?= _w(‘Be the Difference’) ?></p>
outputs
<p><span id=‘md5_58cbc46092296cbe517d748685f52838’>Fai la differenza</span></p>

PO Edit Client - This editor would use the msgid or some other token (such as the md5 hash of the msgid) from the enhanced html, to retrieve the information for an individual string and display it for editing. The comments, English translation, and Current localization string would be displayed. The editor would be aware of the current locale which is a page. It would be a live editor where changes would be reflected in the web page as you type.

PO Edit Web Service – The server manages the collection of messages.po files for a web application. It’s provides read and write access to the PO Edit Client, one string at a time. It manages commits to SVN and conflict management. 

Technology:

So three are three components to build… PHP gettext wrapper libraries, which are pretty trivial and produce the enhanced metadata in the HTML, a Server and a Client which lives within the browser.

Server

Either a stand-alone python server using Stas and Zbigniew’s [https://wiki.mozilla.org/Silme Silme library], or integrated into an existing web based L10n tool like [http://en.wikipedia.org/wiki/Pootle Pootle], it would provide a REST web service for editing po files by locale and msgid.

Client

The client can be built as either:
Pure JavaScript included from the page
Addon – Firefox/Fennec Extension

Pure JS has the benefit of working with any browser (Safari on iPhone, IE 6 in corporate dungeon, etc). It has the negative affect of introducing L10n only code that you wouldn’t want to push to production. It adds another integration point to every app.

The Addon has the benefit of being totally isolated from the specific web application’s codebase, which makes for easier adoption. The client would have the ability to use the PO Edit WS without any same-origin policy limitations. Downsides are that it’s requires Firefox or Fennec and it is an extra step for localizers.

Either way, a simple textarea will be a good start for a prototype. Localizers will be loosing any features of their current text editor (say POEdit), but they will gain rich “preview”. For a prototype, the addon route could be built out of an existing extension, such as Jetpack Slidebar or Ubiquity pageLoad command, since it’s a more rapid development env.

Considerations:

The devil is in the details, here are some design decisions.

By default, does this introduce a new requirement for a L10n preview environment? Or is the web application in a testable state? Adding String metadata changes the web application and invalidates it for QA, unless you ship this metadata in the production version also. The ‘right’ thing to do would be to introduce a localization step into the SDLC, but this might lower adoption. Or you could focus on making the implementation of the String metadata really light weight, so it would have minimal overhead and not need an additional phase, since it could ship to production and other applications can invent new uses for this metadata.

Exposing hard to reach state: Many strings are hidden behind a hard to reach sequence… such as error messages. A common practice in web app development is to expose every part of the app through testing hooks. Many large applications grow these features through time, such as allowing testers special query string parameters for each error condition to simulate the error. Another example is web apps with emails; provide urls so that each email is a click away w/o actually running through the registration process. These kinds of hooks would make translation easier and you’d have to list them somewhere so it’s easy to find them all. These become a requirement for a live web editor, unless you make the user jump through steps to reach all the states of the UI.

Out of Scope(?) but on the Radar

A real-time synchronization between the client and server would be ideal, so that multiple localizers could see who was editing a string and to reduce conflicts. Mike Morgan and Fred have suggested Bespin which is a freaking awesome idea.

Wrapping up

If we had a web based live editor for translating sites, this could be a huge win for localizers and the people of earth. Translating a website is one of the most painful and dangerous workflows I’ve seen. You know we have an amazing community, if they are willing to put up with the draconian toolchain we have today. Throwing everything out, is simply not an option, but if we can make a small evolutionary improvement towards a live preview edit workflow which hides revision control systems and text files…

There are plenty of sticky corner cases and whiz bang features one can dream up, but I think building a prototype for this is a tractable problem. I did some brainstorming with Fred this morning and he had lots of other cool ideas and good observations on how this would work in practice.

9 Responses to “A Sketch of PO LiveEdit”

1
flod - 14/08/09
It's an interesting approach, in particular for single pages or really small sites, but I think that you still need access to the underlying .po file:
* To backup your localization and work offline.
* To perform a find&replace in the whole file (this happens if you change the localization of a single term).
* To check your work (doing QA on a single text file is a lot easier than reading a website online).

P.S. one long line of text in "Considerations:" is messing up the layout of this post and the feed ;-)
2
gandalf - 16/08/09
Hi Austin.

I was recently playing around this concept as well and one of the great technologies I found is translation toolkit used by Gallery3 webapp. It does a lot of what you're proposing together with online syncing and uploading of your localizations and a pretty nice UI.

I would like to take it and mix with my mockup so that it both, allows you to translate the file in the bottom of the screen (a little bit like Firebug UI) or by pointing an element for a click.

We could start from there and take it to the extreme :) I'd love to see such approach if only to experiment with where we can go :)
3
Mozilla Webdev » Blog Archive » Improving Mozilla Web Localization, Part 3: Challenges for Tools - 17/08/09
[...] files, but they do not help you translate the website. One way to change this would be building a “PO live edit” tool, as my colleague Austin King describes in a recent blog [...]
4
ozten - 17/08/09
@flod - Yes, this approach would still let you use existing tools like PO Edit and existing workflows for things like working offline, search & replace, and QA.

This is because SVN and po files are still in use.
5
ozten - 17/08/09
@gandalf - I will have to look into the translation toolkit used by Gallery3. I wonder if Fred has checked it out?
6
Axel Hecht - 17/08/09
There is another system like this, the l10n infrastructure of facebook. Sadly, that's closed source.

Same origin policy isn't a problem with http access headers.

If we try something ourselves, we should talk to the gallery3 guys, and probably investigate writing a web api for that purpose on which we can stack inline-web-ui as well as jetpack sidebars or real extensions.

The association of UI elements to translatable strings does seem to me like it needs to be hooked up into the very system. In particular if you have an element with localizable content, you usually put the po inside the element. To markup the reference, you need to change an attribute on the element. So this indeed will need to be a rather involved stack.

I don't think that svn is the right versioning system, I think we need a special purpose versioning system for those edits, and then some code that takes subsets of contributions and upstreams them into some staging, and then to some backbone versioning system. I'm not sure how either of gallery3 or facebook do this. I talked to Fred about the implications and the datamining here, too.

Should some of this move into .l10n.web for discussion?
7
Fred - 18/08/09
ozten: Yes, gandalf has shown me the Gallery 3 idea. I like its concept (like gandalf says, it's a little firebug-like) but its commit workflow is more targeted to many independent instances of the same software so that part is not easily applicable to Mozilla. Also, Unless I am mistaken, it does not auto-update on the fly without reloading the page.
8
ozten - 18/08/09
@Axel good idea, I've posted a message to mozilla.dev.l10n.web. You are an L10n expert, so you probably see a lot of potential issues that I am missing. Your suggestions go completely against my design constraints... So I'd say this idea for better or for worse, is a (thought) experiment in a different solution space than L20n experiments, such as creating a new string file format or a new version control system.
9
ozten - 18/08/09
@Fred - This sounds promising... I'll check it out. I guess the best outcome of a post like this is to find out what open source code is already out there.