Three Musketeers

Chrome support arrived.

We now have extensions for all 3 major browsers: Chrome, Firefox, Internet Explorer. See the image where I am on this blog page with my avatar and 3 different browsers at the same time.

If my avatar moves in one browser, then it moves in all browsers. If I chat in one browser, I chat in all at the same time. The client logs in to the chat room only once. But the avatar is shown on all browsers.

This is very different from weblin where you could only be on a page with on browser. Other browsers of the same page/domain had no avatars at all.

Initially I wanted to defer Chrome support until after the release. But it is much better to have Chrome support, because I Chrome is my main browser. The client needs lots of day to day testing. Now it is being tested all the time while I am doing my normal web browsing.

By the way: a weblin just private chatted to me, but there is no private chat yet. I just saw it in the logfile, but I could not reply. Do we need private chat early?

IE Plugin, Installer, Setup, VS10, Support Software

Just a short notice what's currently happening: I am working on

• some issues with the IE support. Stability is very good, but avatars were "behind" the browser window
• installer for the IE extension
• simple setup process which starts after downloading/installing, where you can select install browser extensions, select a nickname/avatar.
• I upgraded to MS Visual Studio 2010. There are some migration issues to be solved. But I feel, that it is better to migrate now than during the beta phase.
• There is now a separate code repository for the WebKit SDK. It has 1.5 GB headers, libraries and DLLs, which are needed to build and run the WebView module. Surprisingly the result is not larger than 10 MB.

Still next on the list:

• Finishing Setup and browser extension installers
• Simple Web site
• Installer packaging and final testing

Asynchronous and Messaging

Our extensive unit test suite discovered, that

• if two WebKit instances are loaded simultaneously, which use the same JS file from the local disk, then initializing one of the JS engines sometimes fails.

The average WebKit programmer will never experience this. The average WebKit programmer uses WebKit to make a web browser. And web browsers do not create many instances simultaneously. You press CTRL-T to open a new tab. Then another. Not two at the same time. Especially not from the local file system. Maybe that's the reason why nobody discovered the "feature" before.

Anyway, this means, that creation of WebKits must be serialized. This in turn, introduces all kinds of asynchronous behavior, which must be handled properly in many places. Asynchronous programming is always more challenging than spaghetti coding. Fortunately the client is message based. The image shows events which wait for other things to complete, before they are forwarded. In other words: the display must be loaded before it can show something. As simple as that.

This kept me busy several evenings and made a huge commit. Now I can continue with the setup/install process.

Internet Explorer Plugin, Autostart and System Tray

Three of the remaining six tasks are done:

• IE Support: I implemented an IE browser helper which connects to the client and tells about opening and closing of tabs, about URL navigation and tab switches.
• System Tray Icon: a small fun project. Since the client has no window, we need something else to control it. I created a small system tray icon which can start and stop the client.
• Autostart: this was the easiest task. Just a registry setting, configured by the system tray icon menu.

Now remaining:

• Settings dialog (do we really need this, we could start with config files and add a settings dialog later)
• Simple Web site
• Installer packaging and final testing

Next is the Web site. We need some place to download the installer and a simple way to configure the avatar, e.g. change animations and nickname.

Finally, we have to bundle all files and create an installer for Firefox and IE extensions. Not a simple task, especially the IE installer. And finally, finally there is much testing required because we want the installer to do it's job smoothly.

Translations and Release Packaging

In the last blog post I mentioned a list of remaining tasks:

• Translations
• Release packaging
• Installer testing
• Auto start
• Tray icon
• Settings dialog
• Browser compatibility
• Web portal

I just ticked off 2 tasks: Translations and Release packaging. This is the remaining list in the most likely work order:

• IE Support
• Auto start
• Tray icon
• Settings dialog
• Simple Web site
• Installer testing

Most of these tasks are of medium size. We might defer Google Chrome support until after the first beta release. Maybe someone from the community wants to take this task. We hope to complete these tasks until August 2011.

There are more tasks which should and will be done after the beta release:

• Google Chrome support
• Personal inventory of virtual items
• Virtual Items on Web pages
• Real Web portal with item shop and (avatar) trading
• Facebook, Twitter, OpenID authentication
• More social interactions (wuscheln and more)
• Porting to MacOS

Some of these tasks are big chunks of work. But most will be done until the end of this year.

Chat Window

This is the brand new chat window.

It is a normal chat window as you would expect. But there are few facts worth mentioning:

1.

It is a module. Actually, it is a display module. This means, that while you are in the chat, you will stay in the room, even if the browser is closed. This is a feature which many weblin power

users wanted.

2.

It is HTML based. See the bold user names: a single line CSS style. See the resize bars between chat-out panel, chat-in panel, and participants panel: just a few lines of HTML with the help of jQuery UI Layout.

Do you remember a blog post one year ago about "Starting Avatar Display Development"? This was after the client core and protocol was implemented, but before you could see avatars.

Now the chat is implemented and I can paint a new picture with a blue box around the finished chat module (there is also a grey cross over the IRC protocol. We do not need it now. IRC is just a future option).

All this means, that the client is basically complete.

There are only small things missing, like:

• Translations: there is a translation module, but not all visible texts are translated,
• Installer: there is an installer, but the client must be packaged and tested before release,
• Auto start with windows: currently you have to start as any other program by double clicking the icon,
• Tray icon: would be nice

... and some bigger parts:

• Settings and configuration dialog: there are config files and a config database, but no dialog yet,
• Browser compatibility: currently only Firefox works, but we want IE and Chrome as well,
• Web portal: a portal like weblin.com would be nice where you can change avatar images.

There is still much to do. It is not configured for normal users. It is for people who know what they are doing. It has still less features than weblin. It is an alpha version. But it's cool and it's on the way.

More technical detail: http://blog.wolfspelz.de/2011/06/html-based-chat-window-with-webkit.html

Tabs and Windows

I am working on support for multiple browser tabs. Good old weblin had only one room per browser window. If you changed the tab, then you changed the room. If you switched between tabs very often, then you switched between rooms all the time.

For the OVW client we will support multiple tabs in the same browser window. Each tab will join a chat room. If you change the tab, then you stay in the room. The client will always show the avatars of the visible tab. If the tab is changed, then the next set of avatars will be shown and avatars of the hidden tab will be made invisible.

With weblin there was only one avatar display for each room. If you opened the same page twice in two different browser windows, then you saw avatars only on one of them. The other browser had no avatars. This was sometimes confusing. Now, there will be avatars on all browser windows, even if they show the same page. You can chat in one browser and see the chat also in the other window. You move your avatar in one browser and it magically also moves in the other browser. This is made possible by a real MVC implementation. The avatars will be duplicated. They are just a view of the scene model.

We currently have a Firefox plugin. But of course, we also need IE and Chrome. These plugins are much more complex, than the simple windows message hook of weblin. In other words: they are more work. But they will provide much richer features in the future and ... tabs.

Here is more: http://blog.wolfspelz.de/2011/06/more-ovw-progress.html

Avatar Display

Do you see the difference?

On the left is the new OVW client. On the right is a Weblin display. The OVW client still lacks the points below the nickname. But it has a green glow around the nickname instead. Otherwise, they look almost identical. That's the point.

There is now an avatar display in the new OVW client. It is programmed in HTML and JavaScript with support from the C++ core.

There are still features missing. For example, avatars do not walk yet, they slide. But there has been significant progress. Because of the HTML/JavaScript environment, we can use jQuery for effects and event handlers. This makes programming very quick - and fun.

Just as an example, here is how the nickname is changed. It is a piece of JavaScript code:

  ...
 SetAvatarNickname: function (hParticipant, sNickname)
 {
   $('#' + hParticipant + ' .cNickname').html(EscapeHTML(sNickname));
 },
 ...

Pure JavaScript with jQuery. Very simple.

Windows and Dialogs

Based on Brent's WebKit transparency feature I created 2 modules:

1. a WebView module, which manages transparent browsers of any kind
2. a Dialog module, which shows a dialog frame and the real content inside

1. WebView Module

The WebView module is used for all user interface elements. For a settings dialogs, for a chat window, for avatars on Web pages, for toast windows. Everythng you can see on the screen.

The display of WebView windows is completely created by HTML. Dialogs are designed with HTML, images, CSS, Javascript. The main reason for using WebKit as a user interface engine is portability. Modules and dialogs created with HTML will run on all platforms.

WebKit makes the user interface platform independent. Besides, it is really fun to program the user interface in HTML/Javascript.

2. Dialog Module

The Dialog module configures window border and window controls. The outside is identical for all dialogs. The window content is managed by whatever HTML page is put inside, be it a local page from the file system or a remote Web page.

Examples: an inventory window and the current test dialog.

The window can be styled with CSS. We have themes. I present the beloved - and sometimes passionately hated - wood theme:

Do you prefer the Mac theme?

Remember: it's all HTML. You can tell, that it is HTML from the small (reload) button on the window border. This button is for testing only. It will disappear later.

Next is the avatar display.

Transparent Browser Bounty Awarded

We have the construction!

Brent's changes to the WebKit code have been added to the trunk (http://trac.webkit.org/changeset/84990).

WebKit can now be used as a transparent browser. Brent also made a sample for how to do Javascript calls between the browser engine and C++.

We now have an HTML renderer at our disposal to create a UI with web technologies. Expect UI development soon.

The Open Virtual World project awards the bounty to Brent for step 1 and step 2. We agreed to redefine steps 3 and 4 for more advanced features later. We thank Brent for his contribution to the project. This was really fun.

(We also thank Lupus Labs for sponsoring the OVW project, which made sponsoring of a WebKit feature possible.)

OVW Transparent Browser Bounty About to be Claimed

Brent Fulgham is currently working on the transparency feature for WebKit. Brent is the developer of the WebKit-Cairo port. It looks like the bounty will be claimed soon. Here are first impressions of the test.

What does this mean for the project? We will soon be able to start working on the GUI. The GUI is the last missing part of the new OVW avatar client. Thanks to Brent, the GUI will be purely HTML based. That's the future folks: Web technologies, CSS, HTML, SVG, Canvas, WebGL.

Later we might also incorporate the multi-process rendering and WebGL developments of WebKit. This means, that rendering will be even faster in the future and we can support real 3D avatars.

Also, thanks to Lupus Labs for sponsoring this open source bounty.

OVW Transparent Browser Bounty

The OVW project needs a transparent browser as GUI platform. Transparent browser means, that if I set CSS transparency of the HTML-body to 0.5, then I want to see the desktop partially. All graphics and dialogs of the OVW client will be displayed by this browser. There will be no other native windows or GUI components. There are so many other things to do in the OVW project. We would like to outsource this task to people who have browser library know how. We would like to sponsor your open source work, and work on other OVW parts in the meantime.

The transparent browser will be used by a OVW client plugin module. We will wrap the OVW specific module around the transparent browser library. What we need is a transparent browser DLL, which our GUI module DLL can use. To be able to integrate the transparent browser into our project, we need a MS VisualStudio project which compiles all required components, links them into one or more DLLs and has a driver program which teaches us how to use the transparent browser. We would like to add features later, hence the need for a compileable project, not just a release DLL. Yes, we know that we can build WebKit from SVN, but it's not simple and there is so much else to do.

Key properties of the project:

• transparent browser, no default window background
• in other words: window alpha channel controlled by CSS only
• only libraries which can be distributed with a BSD like license
• Cairo or Skia does not matter, but not CoreGraphics
• Synchronous calls into and out of the browser's Javascript
• normal performance, meaning no bad hacks, which make transparency, but harm performance
• Keyboard input and mouse move, click, scroll
• IFRAME-Support
• max. 20 MB for everything needed before compression
• better no app installation required, no registry prep, just plain DLLs
• preferably as MS Visual Studio 2005 Project, 2008 possible
• Release + Debug builds
• Unicode and CJK-script support, right-to-left script is nice to have
• BSD license

Chromium comes to mind as a base technology because this is where the buzz currently is. There are also Chromium derivatives like Awesomium and Berkelium which could be a starting point. Pure WebKit is also an option. We might migrate to speedy Javascript, out-of-process rendering, and WebGL later. Gecko is also possible.

The bounty is max. 4.000 € in 4 steps:

• 1 k € for a DLL which paints a web page on a transparent window incl. a sample driver program (which has a URL input). We can compile it.
• 1 k € for keyboard/mouse input, IFRAME support, and max 20 MB.
• 1 k € for synchronous calls with arguments between the driver program and the Javascript engine. Int and String type arguments should be enough, Dictionary arguments would be cool. BeforeNavigate, DocumentComplete notifications similar to the MSIE component.
• 1 k € for additional cool stuff to be defined later, e.g. MSVC 2005 because it lets us omit recent C runtime libs making the package smaller, helping us actively to use the thing, shrinking to 10 MB including all support DLLs (beware of the ICU data DLL), WebGL support, DOM API support, helping us to channel HTTP over the OVW client's HTTP component, etc.

More about the Open Virtual World project: http://www.OpenVirtualWorld.com

Animating Avatars

We are still building the avatar display. The most important part of the avatar display are the avatars. There are also chat balloons, nicknames, menus and more GUI stuff, but avatars are the most important part. And we want ths avatars to be animated.

What do we need for animated avatars?

1. a renderer which puts pixels on the screen
2. an animated avatar in whatever format
3. a decoder, that decodes the avatar data

In the previous posting I discussed cairo as the renderer of choice. That's point 1. We now have to decide on point 2, the animation format and choose (or make) a decoder for that format. Of course, the decoder is strongly related to the animation format.

What is an animation format? An avatar can perform multiple actions. It can stand, walk, wave, etc. Therefore, we need not just one, but a set of animations for a single avatar. An animator component will then decode one animation sequence of the set and display frame after frame. If the avatar is supposed to wave, then the decoder will play the "wave" animation. Usually it will just play the "stand" animation. And when the avatar moves, then it should play a "walk" animation. So, the avatar format must provide multiple animations.

Lets review our options for an animation format

• 3D animation formats, e.g. Collada, Doom 3, Quake, blender (?)
  
• Custom formats like config.xml with animated GIFs or PNGs
  

This list is very incomplete. Proposals are welcome.

Real 3D models would be cool, but I am not really a 3D expert. So, we will start by investigating the "config.xml" option.

This means: we need an XML parser (already in the Apollo framework) and a GIF decoder, which is capable of animations. Typical graphics libraries are candidates for the job. We are currently checking which library is capable of loading animated GIFs from memory and from file and be integrated into the project. Getting to know a code library takes some time. So, this will take a while. We are checking:

1. OpenCV (no GIF?)
   
2. CImg (no GIF?)
3. GraphicsMagick (very complex, GIF loader not found)
   
4. DevIL (GIF only from file, but we need decoding also from memory)
   
5. GIL (GIL from Adobe does not support GIF)
   
6. CxImage (GIF works)
   
7. Imlib2 (could not find a Windows MSVC build)
   
8. ...others?
   

The high profile image libraries seem to ignore GIF. This sounds weird, but is logical. GIF has only indexed color with max. 256 colors. It is not suited for real image processing applications.

Only CxImage survived the test. This is not an optimal solution, because CxImage is Windows based. It must be replaced for a Mac/Linux port. Replacing CxImage should not be too difficult, because there are only few lines of code in the application code. Basically loading an image file and iterating through the frames.

We do not want to spend too much time on this. So, we will now continue with the latest CxImage version.

Cairo Test

I am testing the cairo drawing library. I modified the transparency test program from my last blog post so, that it paints an avatar image with nickname instead of the pattern from the last blog.

I can move the avatar around and positioned it on top of a browser window. The result looks like there is an avatar standing on the browser - looks like everything is ready. But it's just a fake avatar. It is static, not animated, not from a chat room. It is just an image painted on a transparent window with the cairo drawing library. But as a cairo test, it is quite successful.

Painting graphics with cairo feels really backwards for us web programmers. We are used to markup: HTML, SVG, and XAML. Programming with "hard coded" functions like cairo_move_to, cairo_set_line_width and cairo_fill feels odd. No CSS, no div, no img-tag. Just good old C++ code with function calls.

I discovered another argument for cairo compared to WebKit. In a blog post (which I do not find anymore) a Google Chrome developer discusses the choice of skia. Skia is not hardware accelerated, which makes skia look like a bad choice. But, the Chrome developer argues, that most of the time the browser spends in HTML code, arranging DOM elements and applying hierarchies of CSS styles. Once the rendering code knows where to paint something, then only a small fraction of the time is spent for the actual drawing. This is a hint, that using WebKit is much worse performance-wise than using plain cairo. When we use cairo directly, then we get rid of all the time that WebKit spends dealing with HTML features. The trade off is, that there is no CSS, no JavaScript, and no iframe in the avatar display. CSS makes applying different styles easy. Styles and themes need much more effort in plain cairo drawing code. But it is should be much faster. And speed counts, because we do not want to slow down your PC.

OK, so, cairo works. The image above is only a fake. It is still a long way to go, but we are on the way.

Starting Avatar Display Development

The client has currently much backend code. It has all the protocol handling, user data, and chat functions we need. But there are no avatars yet. The "green" region is (almost) ready. The "red" region is still missing.

(Another part that is missing is a chat window. But when avatars are done, then the chat is a simple, trust me.)

(The IRC module will not be necessary for chat with avatars on web pages. Supporting the IRC chat protocol in addition to XMPP/Jabber is an option for later.)

So, now it is all about the "red" circle: animated avatars on a transparent layer over the web page.

The data is available, graphics is available, is just has to be painted. This sounds simple. But, it is not that simple. We are programming software that should be portable to different platforms, specifically to MacOS and Linux although the original development (and first release) is on Windows. Painting on different platforms is not as simple as it sounds, because it is not possible to use the painting functions of the operating system. These functions (called drawing API) are not available on other operating systems. And because drawing code is heavy and takes much time to write, it is not good to adapt it for each operating system.

What we need is a cross platform drawing API. Many of us use Firefox or Chrome on Windows, Linux, and Mac. These programs have the same problem. And they solved it by using cross platform drawing APIs. Firefox pains everything you see on the screen using a library called cairo. Chrome uses skia. These libraries are ways to abstract drawing, so that the display can be ported to other platforms with reasonable effort.

There are other ways to paint stuff in a platform independent way. One possibility is to use a HTML-renderer, such as WebKit. We could integrate WebKit and paint avatars and chat balloons with HTML and JavaScript. Actually, this is a good solution, because we intend to integrate WebKit anyway for program windows and dialogs. The only drawback is, that it is not much fun to write lots of drawing code in JavaScript. Also, performance might be a problem, because the drawing code would be interpreted by the JavaScript runtime and every visible element would be in HTML with CSS. This is how web browsers work and it is OK for them. But when you surf the Web and your animated avatar is painted all the time, then you don't want your avatar to take much CPU. A web browser can take much CPU while it paints a page, but after that it is silent. Avatars, however are animated and paint again and again. Very much like a Flash banner on newspaper page, that always draws CPU and slows your PC. We do not want to slow your PC.

On the other hand painting is much easier with WebKit than with graphics libraries like skia and cairo. Speed speaks for skia and cairo. The JavaScript programming issue speaks against WebKit. No easy decision. Do you have any ideas?

I the meantime, I made a simple program that just paints patterns transparently on the background. It is a demo to check how that works on Windows. All we have to do is to use skia, cairo, or WebKit or someting else to paint avatars on such a transparent layer. We will decide soon which library we will use.

If you have comments, let us know.