In the recent years, video games production and distribution have considerably been democratized. Gone is the time when it was mandatory to purchase expensive SDKs and get a license deal to have your games distributed. Web games and mobile games allow for innovations and reinventions of known models.

However, as innovative as this new game market can be, some creators still create games based on classic schemes, and there is nothing wrong with that. For example, even though new styles and sounds were born, rock 'n roll has been around for a while and is here to stay. It may be adapted and reinvented, just like games can be.

Some developers aim to provide their players with a classic feel, but there is a trend that should be put to rest: the on screen gamepad.

On screen gamepad

For those who grew up playing the Nintendo Entertainment System, the Sega Master System and all the following consoles, a gamepad is pretty much with what they learned to play games. Joysticks, trackballs and calculators for which you could swap design were on their way out for home consoles. Over time, gamepads also became more ergonomic and more comfortable, although they were growing more buttons every generation.

Handheld (and thus mobile) game-dedicated devices also made use of these control schemes, from the Game & Watch, the GameBoy, the Game Gear to the following devices.

Games were built around these human-computer interfaces, just like PC games were built for keyboard and mouse interaction.

And then came touch screens.

The Nintendo DS may have been the first device to bring a touch screen to mainstream gamers. Although at first, games were not using the full potential of that screen.

A couple of years later, the iPhone came out and brought games and touch screens to the mainstream populace. Other phones had a touch screen before, but none caught the heart of the customers as much as the iPhone.

No other company claimed they wanted developers to create content for their device, with as much fervor. An affordable SDK and somewhat reasonable profit margin allowed bedroom developers to create games with mainstream reach like never before. And they did.

Growing up with gamepads and side scroller/platformer games, many developers recreated controls they knew and liked. Games like League of Evil, Super Crate Box, Pizza Boy and Pix'n Love Rush are well executed and fun games.

But they do not belong on a mobile phone or tactile screen.

Such games require dexterity and a tactile feedback that only a gamepad can provide. A screen feels like glass and is the same all over, it does not provide any feedback, making it very easy for fingers to slip.

In the case of a phone, your fingers end up covering a huge part of the playable area. Sometimes, like it is the case for Pizza Boy, the playable area is just designed smaller to make way for the controls.

It is a waste of good space, but also a bad practice that tries to reproduce an interaction scheme from an altogether different human-computer interface. Just like the Gravis Gamepad had it wrong when they tried to force a joystick into a gamepad.

Mobile devices add-ons

Realizing that the on screen gamepad caused all sorts of issues, some companies thought of creating hardware that players could add to their mobile devices. These add-ons could be split into two types: those that are affixed to the screen and those that are separate and require an integration into the codebase of the software.

Attaching an add-on to the screen, like the Fling Mini (bottom left in the image above) and the Joystick it (bottom center), solves the issue of the expected physical feedback when playing. Parts of the screen still are hidden though, if not more so in some cases. And what if the game designs the gamepad not in line with the add-on placement?

The other types of add-ons are actually smart. The problem in this case is that the hardware developer expects games devs to use their API so that their add-on can be used. It's not that much of an inconvenient, however there is a multitude of add-ons. The image above samples only a few ones: the iControlPad (top left), the Caliber Advantage (top center), the 8 Bitty (center) and the PhoneJoy (right). And there are more. Game devs would then have to incorporate the API of each of those add-ons to provide support for any eventual add-on their clients might use. On the other hand, if the developers don't, the onus falls on the player to purchase many add-ons to be able to play with a controller, depending on which one the game devs chose.

This is ridiculous.

Let's also remember that the success of gaming on mobile phones relies a lot on the fact that the device is already in your pocket, no need to carry anything else.

The advantage of the gamepad on a console is that the gamepad is standardized and works for all games. Each game does not chose what gamepad it supports.

There are some exceptions, obviously. A couple years back, a hyperactive reinvention of Space Invaders was released. For the DS release, an add-on paddle was created. The paddle was a hommage to the controls that the arcade Space Invaders sported a while back. It was fun  and novel to play for someone like me who grew up with a gamepad in hand.

But ultimately, the gimmick was superfluous. I would carry my DS in my pocket in the subway, but never the add-on. Customers want was is convenient, and carrying more things in their pocket is not. This argument has been laughed at by many, but over time the rise of gaming on mobile phones has been shown to hurt gaming-dedicated handheld devices sales.

On screen interaction

Nintendo has always really been good at making games that explains how to use their innovations, not that all studios or developers were always dedicated enough to apply or explore all those innovative ways.

The Legend of Zelda: Phantom Hourglass was an awesome game that managed to keep its feet rooted in its history while pushing how you use the handheld device in the game. The first thing you notice when playing is that, contrary to previous Zelda games, you move Link by taping holding the stylus where you want him to go. You would also tap directly on the objects and the enemies on screen to interact with them. Sure you can still use the D-pad and the buttons, but that made no sense, the game wasn't built for that.

No on screen gimmick. You want your character to go somewhere? Tap there. Developers should really think of that more often. The A* algorithm deals exactly with that logic.

Another remarkable thing was that the player could write and take personal notes on the map. Let that sink for a moment. It may seem like nothing, but for so long, inventories and maps might have been editable, but always within the confine of the user interface. This game offered the player the option to draw and write whatever they wanted on the maps, to not have to deal with the interface!

At a point during the game, the player has to fold a map to obtain some information about where to go further. So as a player, you tap, you click, you read and still cannot figure what to do. Eventually, you realize that you actually have to close the DS clamshell and reopen it and ta-da! The ink on you map is transposed from the bottom to the top (or the reverse, I forget).

That's smart! This is making use of the device and its specificities, and not trying to recreate old ways.

Fast forward a couple of years later. Lost Winds is a side scroller game that was originally released for the Wii. This game already made use of novel ways to move the character by combining the thumbstick of the Wii's Nunchuck and pointing at the screen with the Wiimote.

When this game was ported to iOS, I was really curious. I was really charmed with the Wii game, but somehow the iOS port is even better. Pointing at the screen with the Wiimote has been easily adapted into swiping the screen with your finger.

How do you move Toku around without the thumbstick? For one thing, there is no on screen game pad. A bit akin to how you would move Link around in the Phantom Hourglass, simply tap in the direction where you want Toku to go. Don't want to tap, tap, tap and tap again? Simply hold the direction for a while, and Toku will move automatically in that direction. See the screenshots above, there is even a visual indication. While he walks, you can swipe and attack monsters. Tap anywhere and he stops walking.

This is a smart adaptation of the on-rails feeling that that you get from endless runners, such as Canabalt. While at moments you may want the character to move automatically, at other times you want more control. This is obviously no way to play a reflex-demanding Mega Man game, but tactile screens are not the proper target for those games.

Create controls for the device you target 

With the advent of new microconsoles like the OUYA and the GameStick, and with the Playstation Vita's affordable SDK, it will be possible for game creators to create games for devices that benefit from an actual game controller, rather than try to recreate those controls. It's too bad that Nintendo is not open to that kind of indie creation, so many game devs would be thrilled to have their games released on Nintendo devices.

There is obviously a demand for side scrollers and adventure games on mobile phones and touch screen devices, but it's about time that developers start exploring ways of controlling their games according to the device onto which the game is published.

A couple of years ago Malcom Gladwell presented what he dubbed the 10,000 hours rule, which explains that for one to become proficient with a skill, 10,000 hours have to be invested into that skill. Oddly, most tech journalists and bloggers seem to take this for gospel. I'm not going to dissect and argue against Gladwell here, but rather I have a different perception of his argument. I believe that if I invest this much of myself into a certain unique skill, I'm going to grow bored of it and I will need to move on.

I have been developing and coding for approximately this long, it's time to move in a different direction.

Oftentimes when trying to explain to clients and colleagues that even though I am good at coding, I do not care for it as much as I care for what the user experiences. Even clients, faced with my technical knowledge, wanted to push me towards back-end coding, which has nothing to do with visuals or interfaces, and thus is of no interest to me.

When I stumbled upon the diagram above, it made so much sense to me! It was so clear that I could then illustrate to people, even those who have no technical knowledge, what I do and where I what I am aiming to do. I need to move towards the left side of that diagram, and not the right side!

Don't let them put you into a box

I have never been a fan of simply being told what to do without seeing the big picture. This is why I always strived to be included in every step of a project, from the brief to the strategy, from the design to the production.

However, not everyone is at ease with that. I once had a manager that was displeased because I wanted to take the time to sketch out storyboards, he thought that our department should only have cared about development, and that storyboarding should only be done by information architects or designers. I think it was short sighted, if I knew the assets I required to build the project and knew how to share the requirements efficiently, why stay confined into the limited description of a job position?

Another time in another place, a director of production had just been hired. He was in charge of the producers/project managers as well as the technical department. From then on, he chose to limit our input to only technical things, we were not to be part of the creative brainstorms. Why would removing the people who build the creative experiences from the creation process be a good idea?

I recently spoke with a friend about expanding or reorienting my service offer, and he suggested that I should choose one specialty, and stick with only this one, rather than many. He thought being technical, creative, strategy- and user-oriented were too many things at once.

A university teacher of mine once said that we should aim to become like the scholars of the Renaissance and try to learn as much as we can from every field. And that is what I want to do, no matter if others feel threatened because they are comfortable with only one specific skill. Too often have we heard stories of specialists, whether they are developers or factory workers, that are rendered useless once a new trend appears.

Updated service offer

With all that in mind, I reorient my service offer towards user experience rather than front end development. Fabricio Teixeira dubbed the person with a multidisciplinary approach to the field the UX chameleon and I find that image quite apt.

As a user experience consultant, I intend to use my technical knowledge and experience to suggest smart strategies in using technologies to appropriately answer the needs of the user, rather than simply pushing for the latest tech buzzword. I can create documents to help others get a clear understanding of the goals to reach. I aim to work more upstream in the creation process: analyze the client's brief and then work with the strategy and the creative teams.

I have already been drawing sketches, creating user flows and building information architecture for a while, now I am pushing this side of my aptitudes as my main service offer.

Even though I have posted screenshots in my portfolio–and I hope to get a video of the kids interacting with it eventually–I thought it would be great to share how we built that project, a case study of sorts.

Navigation

This installation was split between three states of user interaction:

• Attract: At first, catch the user's eye and interest. It's at this point that you invite the user to interact.
• Engage: Once the user interacts, add some information with which the user can play a bit.
• Explore: The user can explore the experience further from there. Offer more content upon the user's request.

These are just general points as to how you get a user to interact with your installation, which we applied for our installation. Let's see how we designed this.

Design and user flow

The original concept was heavily influenced by infographic design, with heavy fonts, flat and bright colors. To be honest, it was a bit of a mess, too many elements at once for the eye to know where to look and how to read. However, the idea was there, we simply needed to refine it into a finished product.

As can be seen in the concept images above, there is a clear disconnect between the design of the map view and the design of the infographic view. There were also lots of elements that needed not be repeated in every state, such as the temperature and other weather information.

I worked with Raed Moussa to streamline the design, keeping only necessary information. He was able to create a cohesive visual environment between the map and the side view.

As for informational elements, we opted to keep them only where they were pertinent:

• The attract view contains animated weather information over a map of the Long Island Sound area, with a clear call to action to invite people to touch the screen.
• The engage view presents people with some additional information overlaid on the same map that they saw from the attract view. All the information is relative to the map they see: water depth, buoy locations, shipwreck locations and historical events.
• The explore view contains a figurative slice view of a coastal region in the same area. Species icons are displayed in locations where said species would usually live, and if the user so desires, more information on each species is available upon touch.

Water depths

One of the scientific information that we had to overlay on the map was the water depths of the area. This information is usually printed directly on the maps or embedded into ships navigational systems. We wanted to display this information in the way topography maps display heights.

The inconvenient was that the available maps did not provide it directly. Somehow, most of the depths variations on the map were not big enough, most were simply noted by numbers.

So we had an idea: highlight all the numbers with different colors in order to create a heat map. And I mean manually highlight! At first, we ordered the physical maps, we planned on sitting at a desk with markers to highlight the numbers. We looked into how to take a high resolution photo of the map, but the depth numbers came out all blurry. It was a no go.

Finally, we were able to get the digital maps from Nautical Charts Online. From then, I took a couple of days in Photoshop with my graphic tablet to highlight all the numbers according to a range we predetermined. We were able to get a good heat map.

It was still necessary to incorporate the information of this heat map into the design. Raed was provided a vectorized topographical render of the heat map and was able to incorporate it smoothly into the map.

Transition from map to slice view

It was a bit tricky to think of a transition between the map view, which is a flat map of the Long Island Sound area, and the side view, which is a slice view of a coastal area. At the time I was watching Avatar: the Last Airbender cartoon. In its intro, there was exactly what I was looking for! Take a look between 0:12 and 0:28.

This was a great inspiration. Being a fan of old Final Fantasy games, I also thought of how the map was tilted back when the characters would hop onto an airship.

With the storyboard above, Francis Dakin-Côté was able to create the perfect transition we needed between the map view and the slice view. This animation is presented when a user chooses to explore the installation more in depth, providing a logical transition between the states. The same animation is played backwards when going from the slice view to the map view.

Development

As I probably mentioned in earlier posts or tweets, Float4 most often uses RealMotion, its own software. The development was then split in two: the RealMotion graph that handles the interaction and embeds the content, built by Bruno Gohier, and the interface and content, which I built.

Akin to how Scaleform allows to embed Flash content, so does RealMotion. Since the content for this installation needed not to live anywhere else, and also since it was mostly a two dimensional multitouch graphical interface, Flash was the most indicated technology to choose here.

As I did for other projects these last years, I built the application with the RobotLegs framework. I also used AS3 Signals rather than events, which is a charm to work with.

I may have said in other posts that RealMotion can communicate touch points to the application via the TUIO protocol. I was using the TUIO AS3 library, although along the way I ended up finding and fixing some of its memory leaks and other issues. Don't get me wrong though, it's still an awesome library.

The development was pretty much straightforward, however I think I haven't done something this demanding on a machine since the adidas Originals Women's Lookbook. This project is in full HD (1920 x 1080) at 60 FPS and was required to work in multitouch.

The transition animation ended up being a lot more work than expected. Using a 60 FPS video would prove impossible, as it would take too much time to start and skip too many frames. I tried scrubbing the video manually at every frame, which ended up being worse. I also tried loading dynamically all 360 frames for the transition and built a class that would display the animation. This worked smoothly and was quite satisfying visually. The issue was that once all the images were loaded, the application would freeze while trying to add the images to the stage. And this would only work while developing in FDT, as soon as I tried in a browser or within the context of RealMotion, a crash would occur. So I ended up using two 30 fps videos, one forward and one backwards, since the user has to be free to come back to the map state.

It was not my first go at multitouch, so I could apply the discoveries I had made there. The main thing was really about keeping track of the touch ids: once an item is touched, no matter how many times other touches disappear from the surface, the item is still considered touched as long as the first id is still touching the item. It would translate into something like this:

ActionScript 3.0:

override protected function onReleased(_:TuioTouchEvent):void
{
	if(_.tuioContainer.sessionID == _touchId)
	{
		super.onReleased(_);
		if(!_isTouched) _released.dispatch(this);
	}
}

Another touch behaviour that I implemented from the slideshow was the "touch and hold to trigger". You can read a bit more about that logic in previous post. In this case it was used in an area where images and videos could be dragged, a bit like a vertical carousel. It then became possible to distinguish between dragging to change images and triggering to start a video.

Because laser touch detection cannot be as precise as touch screens, we wanted to avoid small UI elements as much as possible. We decided that textfields would be draggable, like on mobile devices, also because it allowed us to remove the drag bar and the arrows. More space for content.

The weather data presented in the attract view is pulled from the Yahoo! Weather RSS Feed, whereas the buoy data is obtained from NERACOOS. Actually, if you ever need to use buoy data, contact these guys, there were thrilled to help us use their service. Even before completion of the project, they showcased the installation in their newsletter.

All other historical and animal data is entered in a CMS by the users (students or teachers).

The RealMotion graph contained the Flash content and presented it on screen. Since this software is really strong with visual effects, there was no need to try and implement them in the Flash application. We opted for a water ripple effect, which made sense in the context. As the user touches or swipes the surface, water like waves are created across the screen. In the map view, we also added a mask so that the waves would ricochet off the shores of the coast.

We had to come up with a way to change the states of the waves according to the states of the Flash application. The best way to do so was to send values via the Socket AS3 class. RealMotion contains a module (called "box" in their environment, akin to Max's) that could receive data from a socket. That box can currently only receive bytes, but I believe that it could grow into something more useful where Flash applications and JavaScript applications could send bytes, integers and strings. Even better, if two way communication could be implemented, a new layer of flexibility would be available to content creators.

CMS

In order to manage the content, we provided the client with a CMS (Content Management System). I believe that this project can live long and evolve according to the students' work. Rather than provide the client with a paper manual, I took the landing page of the CMS as an instruction page, and suggested how they could use that tool:

Do not think of this installation simply as a static repository of information, but rather as a dynamic space to display involving and evolving information concerning the Long Island Sound area. Rather than throwing in all the data at once, maybe there could be a theme for each week or each month.

Here are some ideas:

• A student's essay receives a great grade: if a student's essay on a subject that is pertinent to the installation (Long Island Sound area, history or sea life) is well written and worth sharing with the school, present it by adding items in the appropriate sections with excerpts of the essay.
• Shipwreck week (or month): showcase only shipwrecks in the Engage View section by deactivating all items that are not shipwrecks.
• Shellfish week (or month): showcase only shellfish species in the Explore View section by deactivating all items that are not shellfish.

Obviously, these are simple examples, but it's up to you to engage the student in using the installation and maybe write content for it to be shared with fellow students.

As you can see, I believe that as container as well as content creators, we may have a responsibility to inspire our clients on how to use the tools we create for them. Sometimes it's obvious (product display for business), but in this case, I thought it would be best to also guide teachers and educators. These people already have a lot on their plate, let's make sure what we create answer their needs and doesn't complicate their work.

I built the information architecture for the CMS and gave this information to WLAB, the service providers who developed most of the CMS. They used CakePHP, a framework I didn't know then. To be honest, I found this framework too bloated and convoluted for the needs of this project, a fully custom CMS would have been preferable, quicker and more flexible. The providers became overloaded with other projects, so I asked Simon Arame to take over and help us finish the work.

In the end I went through the code to separate CSS and JavaScript from the PHP/HTML. Simon helped me deploy the CMS and the database onto the installation machines.

Installation

The installation itself consists of two separate screens, each with its own computer. One of these computers acts as a server for the CMS, as mentioned above. Each screen is encased with a PQ labs frame, which detects touch points. This hardware is handled by RealMotion.

The installation is somewhere in the high school. As I say at the top of this post, we will eventually film people using the installation. However, as the school is located on the path of hurricane Sandy that devastated a lot of the American east coast recently, we will leave them time to sort their priorities before we push for them to let us come in and film.

Looking ahead

This long project represents a bit how I like to work: touch every bit of the project, do some user experience design and art direction as well as some project management. It gave me a clear view that I'd rather leave hardcore coding aside and put my energy into UX and storyboarding as much as I can.

I'm proud of this project and can't wait to see where the teachers and students of the Marine Science Magnet High School are going to take it!

Even though HTML5 is quite promising, a lot of us realized that the fragmentation of what video and audio codecs are supported is quite an issue.

While making a mobile version of Float4's website, I tested on an Android phone and obtained a message from the embedded Vimeo video that the H264 codec was not supported. Even though each browser vendor publishes its own matrix of what they do and do not support, it's not always easy to find out that information quickly. Worse, not all clients are tech savvy, and they may not easily be able to provide you with the necessary information to figure out what codecs their browser or device support.

It came to me that I should create a simple tool to detect what codecs are supported in a browser. I was inspired by websites like playerversion.com and supportdetails.com to create codecdetect.com. Each website allows any user to find out technical information about their computer, information that they may not be able to know where to find.

So here you go internet, I offer you this tool for you to use to test your browsers and your client's browsers. You may view the JavaScript source all you want to see how I have done it. I will try to maintain and update it to the best of my knowledge, but if you find additional codecs that this tool should test, or better ways to do so, do not hesitate to share!

Link: codecdetect.com

EDIT (2012/11/05): I have added an email details form to make user's life easier (inspired from SupportDetails), and I have also put all the sources available on a Github repository for everyone to see. Enjoy!

Some two weeks ago, I attended the BlackBerry 10 Jam World Tour in Montréal. It was a nice day and I could have done something else, but hey, learning about new technologies is a passion. With such a name, I sort of expected the sessions to be more hands on into coding, but this event was more about introducing the latest OS and its intricacies to developers.

The morning session

Given all the bad press that RIM gets these days, it was understood that the reps had to justify their situation. Without getting too deep into the technical and financial fine points, the company reps boasts that their store grows quickest and offers the most profit per sale.

In fact, RIM is so confident about the future of their upcoming platform that they offer a prize to all certified apps available on their store at the launch of the BlackBerry 10: if the app sold for at least $1,000, but did not exceed the $10,000 mark, they'll cover the difference.

The morning session concluded with Chris Smith who presented the architecture of the BlackBerry 10. He boasted that since the OS is built on Qnix, a system quite robust as he explained, BB10 would be the only OS that could run another OS on top of it. He also claimed that the new OS would be backwards compatible, so a dev could write in the latest OS and deploy across multiple generations of BlackBerry phone. That seems almost too good to be true, I'll believe it when I see it.

Cascades, the new UI,  is built on top of Qt, a C++ framework of which I heard for the first time recently, since the Float4 devs use it to develop their software. This framework functions a bit like AS3's components, or Objective-C's UIKit: it offers pre-made components that respect the OS's UI.

RIM has decided to spend a lot of time to provide many types of developers the librairies to develop with their own tools to target the new BlackBerry 10 platform: C/C++, HTML5, AS3/AIR, Java, even the Android runtime! It is an awesome strategy, however I feel that it may be a late move on RIM's part. It sure feels refreshing compared to Apple's stubborn strategy.

However, it's a bit ironic that during his presentation, Suavek Zajac commented that he did not know if it was worth the effort to care about AS3/AIR since that platform might not have a future...

Native development session

During the afternoon, two sessions were offered: native development and HTML5 development. Since I know already how to develop for the web–and that learning a new JS library shouldn't be too hard–I decided to follow the native presentation.

I honestly believe that Cascades is indeed a beautiful and well though out UI. It's been designed by the Swedish agency Astonishing Tribe, and if I remember correctly, RIM acquired them. I found it interesting that the evangelist giving the presentation on the NDK also took the time to make sure that that core UI concepts were understood, that he was not just there to talk tech.

In an example, a user can delete a photo within the application. Rather than having a button labeled "Cancel" like every other platform, it was labeled "Regret". I thought using good copy like this for call to actions brought a more human side to the interface. Too often are we, developers, designers and copywriters, stuck using terms that have been following us for years.

There was talk that the font of the overall UI could eventually be customized at the application level. This may open the door for really bad design choices. All the time that was put on designing a clean UI might be worthless if developers or designers are allowed to tinker with that. The only reason why this might be useful is for localization, where some specific characters are needed, like Japanese, Chinese, Korean, Cyrillic, and others. Otherwise, I believe the fonts should not be editable. We'll see where that choice of theirs lead.

Obviously, there were tons of tech talk. Zajac started with a really low level explanation of the event system–which kind of scared me by the computer science knowledge required to understand it–but went on to explain Cascades and how it was built on top of Qt. The UI has a dark theme with light blue highlights, quite slick! There seems to be a drag and drop interface builder that comes along the NDK, which reminds me of Flex Builder and XCode. The markup necessary for building such interfaces is QML, which feels like JSON. An interesting note is that JavaScript can be used directly in QML.

In the many examples presented, copy and labels were always hardcoded, which is a bad idea if you've ever had to localize your content. It seems that there is a way to use variables to replace those values. As I understood, it'd be written a bit like how it's done in PHP or Ant.

Many things come out of the box with the NDK, amongst what was mentioned, a couple caught my interest. There seems to be some sort of tweening library, and from the quick peek of the code that was shown, it felt like the concatenation of the animations had a similar syntax to how it's done in jQuery.

There is also an API to allow users to do in-app purchases with the credit card that is already registered to their BlackBerry account. Mind you, I believe that's the same logic that other platforms use, but since I never had to use such an API, I though it was smart to leverage user information already recorded and offload all the security issues to the company platform rather than having to deal with that at the application level.

When an app is installed on a BlackBerry, the app requests authorization to access different things, a bit like how Facebook does it. Having this in iOS might have avoided recent debacles where personal information was tracked without user consent. I thought this was new, but it seems that BlackBerry had that for a while.

The most interesting concept is probably the idea of invocation. The invocation framework allows for an app to register to some actions for all other apps. The simplest example is if someone created a clipboard app, the copy/paste functionality could register to all apps that have editable text, without having to add code to other apps. Another example is a sharing application that could allow you to share photos or text to certain network. The app could register to actions relative to text or images, so that whenever the user has a contextual menu with these elements, the actions of the sharing app could be available as well.

Thoughts on the new platform

Those who attended the event and asked for it in advance were given a Dev Alpha device to be able to develop for the BlackBerry 10 platform before it's publicly released some time at the beginning of next year.

So I got one too, woot!

I was quite surprised by how light this device is! It's practically 150% the size of the iPhone, with a huge 1280 x 720 pixels resolution. It's also exactly the same height as the Playbook, a nice little touch. Actually, it totally looks and feels like a mini Playbook.

But it's definitely an alpha version. The UI is clearly not ready for the device, it's just a port of the Playbook's, no button or label has been resized properly to fit the size that a finger represents on this device. The icons are not ready either, as they seemed stretched and all blocky.

At least, if you develop an app and add a button on screen, it shows with the proper dimensions relative to the target. For example, when I tested an app built with the AIR SDK on the Playbook, the button height was some 44 px, while on the Dev Alpha it ended up at 101 px. It's a nice touch that the UI components handle the DPI themselves.

I am not ready to drop my iPhone anytime soon, nor am I as enthusiastic as some, but I do think that RIM did a good job with their latest OS and communication strategy towards their developers.

Let's keep an eye out for this platform, it might eventually be a good idea to include it in the mobile or tablet suggestions for clients, especially if their markets are not only in North America, as it seems Asia and lots of other markets are quite fond of the BlackBerry universe.

This winter I was working on a slideshow for Float4 to showcase their portfolio inside of their installations. I had posted my designs and some images in my portfolio, but here is a short video. It's easier to grasp the multitouch logic when you see it in action!

Click here to view the video on YouTube.

As I said in the last post, I am currently building a small project to be published onto the BlackBerry Playbook. I am building it in FDT 5.5, using RobotLegs, AS3Signals and TUIO. Above is a short teaser of my work in progress.

Summer is almost at our doors, on the calendar at least. It's been a while since I wrote, I have been busy: I've traveled, I've been coding, and I've been trying out a couple of different things. Let me bring you up to date.

Flash Gaming Summit 2012

In February, GAMES@CODAME held a contest on their Google Plus page to win a ticket to the Flash Gaming Summit 2012 in March in San Francisco. So after a tight race, I ended up winning the ticket! As I said before, video games are something towards which I would like to move, and meeting game devs and hearing them speak of their craft could not have been a better blessing.

I managed to meet with animators, entrepreneurs, devs and all sorts of good people. Some talks were really good, while some others were basically just rants. I personally wished that some of them would have been more technical than business oriented. On that angle, Iain Lobb delivered.

Almost as a fanboy, I made sure that I would meet some public figures in the game devs business, so I chatted a bit with Thibault Imbert, Bruno Fonzi, Iain Lobb, and my compatriot Ryan Creighton.

Later in the day, Kongregate held a party at their offices, where I met other cool devs. When I spoke to Rob Bateman, who had earlier given a quite interesting speech in favor of open source, I had to find a way to brag about my Away3D tutorials. By the way Rob, I'm still interested in writing tutorials for Away3D!

The next day, I was also lucky enough to be invited to visit a game dev studio not far outside of San Francisco. It's quite refreshing to see a team of devs and artists really work together, I hadn't felt a relationship this tight when I was working in an advertising agency.

Working with Float4

As you might have understood from some of my tweets and posts, I have been working with Float4 in Montréal since January. They specialize in video mapping and interactive surfaces, both with HD resolution or higher. It's quite nice to be able to use high quality graphics rather than being restrained to the small dimensions of the web.

Float4 offers a software, RealMotion, that has many purposes, one of which is to display Flash content. So I mostly have been working in ActionScript 3 for their projects. As Stage3D came to be, we did some tests to see if it would be possible to use within the context of RealMotion. To explain shortly, RealMotion does not incorporate the Flash Player directly. An instance of the Flash Player is launched on a thread, and RealMotion reads the visual output created and displays it on a texture. With Stage3D, that became impossible, since the render is actually done on the GPU directly, and there is no other way to get the render but to try and intercept the visuals directly from the graphic card. When I passed by the Flash Gaming Summit, I asked Thibaut Imbert if he had an idea of how to do this, he referred to PIX, a GPU profiler, but it didn't turn out to be of use on a Windows machine. If you have any suggestion, it would be welcome!

Discovering how to handle multitouch was quite new to me, as I was not used to the possibility that many UI elements could be used at the same time on the same screen. Also, I have been writing for installations where neither a mouse nor a touch screen was used, but rather lasers. In this setup, lasers are scanning a surface in order to determine where the user or users touch.

It may seem like it would not affect how to code, but it's actually quite important! While on a desktop and on a mobile surface it's important to listen to the "up" event (mouse up, touch up), it's not quite logical to do so with lasers, since users would not expect to lift their hand to trigger an action. Therefore, a touch down event acts as a click, and triggers something right away.

In the case of a list–like the iPhone contact list–it's not logical to open the contact as soon as it's touched, since it may be required to scroll. That's why on these device, it's the up event that triggers an action. With lasers, I thought it would be best to introduce the logic of holding the position for a while to launch an action. A timer would start and once a certain amount of time is passed, the action is triggered.

RealMotion communicates with Flash via the TUIO protocol, and its AS3 library, TUIO AS3. The library is open source and quite easy to integrate. Rather than using mouse events, the library comes with its own custom TUIO touch events. What is also quite nice is the guy who manages the library is quite open, listens to comments and adapts accordingly.

But how does one test multitouch during development? It does take time to deploy to the installation every time, and let's be honest, it's not always available to the developer. Luckily, it's possible to install an iPhone app and an Android app that can turn your device into a multitouch controller for your project. Simply start up the app before you launch your AS3 project, and then they are connect via TCP.

As for projects, I have been working on an update and a localization of a project for the Canadian government that needed to be displayed in Davos. I hope to obtain some images and footage to add to my portfolio. I also have been working on a slideshow that showcases Float4's portfolio with different AS3 APIs (Vimeo, Flickr, Twitter). You can see a screenshot of that project above this section, I will soon add screenshots to my portfolio. I also have been working on a library that can communicate with RealMotion, so that Float4 can provide clients who want to develop their own content with an API.

An iPhone app on its way

For some reason, I decided to go back to try and learn Objective-C. Since there is no better way to learn something than having an actual project, I will soon release my first iOS app! I finally understood how to link visuals to the code, and how to add and remove event handlers. Despite a lot of idiosyncrasies, XCode is a good IDE for the job. The code completion is marvellous, the interface is Apple-like (ie: fine tuned to the bone), but some features for quick coding and ease are missing. I may try and work more in that environment in the future.

A new toy

During March, BlackBerry had a 50% off deal on their PlayBook tablet. I never really felt the need to own a tablet, and I thought the iPad was way too big and expensive. After attending the Flash Gaming Summit though, I thought that tablets could be an awesome target for games, and that there is no better way to understand how a device works than own and use one.

To my surprise, BlackBerry offers an awesome AIR SDK to develop for the PlayBook with their own UI. Smart! I had some issues while trying to run it on FDT, but a recent post on the BlackBerry forums brings some good news for the future.

I think that I may try to create a TUIO controller app for the PlayBook as my first attempt!

Going back to hacking

In the spirit of exploring many things digital and analog, I decided to go back to hacking electronics, as I have done for school projects before.

First, I will attend a circuit bending workshop given at the Eastern Bloc in Montréal, and see where that takes me. I hope I will play with my Arduino again. I've also been quite interested into looking into the Raspberry Pi, a more than affordable open source computer.

It may get cloudy

Finally, I thought I should leave you with another tease. I've been doodling and thinking of this little game where a cloud has to water plants and make sure to avoid the sun to not evaporate. The full gameplay is not yet defined, and as you can see I am tackling more than one project at a time. I think this project could be a good excuse to look more into haXe and NME.

What have you been up to?

There has been a lot of people who played with Away3D and built awesome things. A lot of said awesome things were built by people who know, or know someone that knows 3D modeling, which is why the quality has been so awesome. However, it may not always be necessary to build a full 3D world, sometimes a project may simply require some 3D elements over a 2D background. But how do you place an image behind all sorts of 3D objects if Away3D's View3D has a black background? I will try and explain the process in this article. Warning: matrices and trigonometry ahead!

Build it and they will come

Usually, if you want to place an image and make it fit your document dimensions, you simply stretch the image accordingly. In 3D however, it is not as simple, as the world dimensions are not necessarily measured in pixels. While working with the guys from Float4, I came to understand that meters are oftentimes used as measurement unit. With that in mind, we cannot directly use the dimensions of our document.

So let's say we want to place an image in the 3D world and make it fill the viewport (the visual space taken by Away3D). At first, we would need an image, and a plane onto which we would like to place the image. As I wrote previously, materials applied onto shapes need to be in dimensions that are a power of 2, so will will need to rescale our image. A Matrix will fulfill this need.

private const WIDTH:uint = 640;
private const HEIGHT:uint = 640;
private const MATERIAL_WIDTH:uint = 1024;
private const MATERIAL_HEIGHT:uint = 1024;
 
[Embed(source="assets/images/escher-relativity.png")]
private const EscherImageClass:Class;
 
private var _imageData:BitmapData;
private var _matrix:Matrix;
private var _material:BitmapMaterial;
private var _plane:Plane;
private var _view3D:View3D;

WIDTH and HEIGHT represent the dimensions of my document, and also the dimensions of the image. The MATERIAL_WIDTH and MATERIAL_HEIGHT represent the dimensions needed for the material, dimensions that are a power of two. Here I will be using an image from Escher, obtained from Wikipedia. There is no need for a Bitmap per se, but we will need the image's BitmapData, hence the _imageData variable.

Let's do this!

Normally, I resize the material dimensions to a value higher than what I need to display; first the material is stretched up, and then visually fitted to the original dimensions of the image. If we were to scale the material to smaller dimensions than that of the image, the plane would stretch up the material afterwards when displaying the image, resulting in possible loss of smoothness or detail.

_matrix = new Matrix();
_matrix.scale(MATERIAL_WIDTH / WIDTH, MATERIAL_HEIGHT / HEIGHT);

Let's draw an instance of the Escher image into the_imageData. This is where _matrix comes in handy. I wonder what pushed Adobe's engineers to make an architectural choice where they thought users would use a ColorTransform, a BlendMode or a clipping Rectangle more often than a flag to smooth the drawing. I mean, putting the flag as a last argument? Anywho, let's add a lot of null parameters...

_imageData = new BitmapData(MATERIAL_WIDTH, MATERIAL_HEIGHT);
_imageData.draw(new EscherImageClass(), _matrix, null, null, null, true);

Next, create the material. As seen previously, let's use a simple trick to assign whether or not the material is mipmapped.

_material = new BitmapMaterial(_imageData, true);
_material.mipmap = (_imageData.width == _imageData.height);

Apply the material to the plane, add the plane to the View3D, and then add the view to the display list.

_plane = new Plane(_material);
_view3D = new View3D();
_view3D.scene.addChild(_plane);
addChild(_view3D);

Do not forget to listen to Event.ENTER_FRAME and call the View3D.render() method at every update.

addEventListener(Event.ENTER_FRAME, updateHandler);

Run it, and what do we get?...

Nothing.

Uh?

By default, the plane is added at (0, 0, 0), so it should be visible...

It is, it turns out that an API choice—which may make sense in a 3D world, but that I do not understand—puts the plane with its face up on instantiation. What this means is when it is created, we see its side, which has no thickness. Just change the yUp parameter to false, so the new plane will then face the camera upon creation.

_plane.yUp = false;

We should get something like this.

If we compare the image and this result, we notice that the image's edges are cropped, which means the plane is too close. Below is a diagram explaining this visually.

Imagine a pyramid drawn from the camera at its top with an angle that is the camera's field of view, up until a certain point. From the camera's point of view, there is always a certain frame that is visible, that is the viewport. However, that pyramid is not infinite, actually what the camera sees is not even a pyramid. The viewing frustum, what is visible within a 3D world, is an area from a certain threshold (the near plane) to another one (the far plane), that's what is visible within a 3D world. What happens with the code above is that the plane is bigger than the frustum, thus being cut.

Below is what we want to achieve:

SOH-CAH-TOA

This is where the fun begins! How do we fit our plane into our viewport? It is possible to simply move the plane on the z axis and eventually position the plane properly to fit our need. But what if the dimensions change? We'd be back onto square one. This problem requires good ol' trigonometry, and the pythagorean theorem.

You may wonder what this title means. It's a mnemonic trick that's been given to me back in high school on how to remember the rules of trigonometry:

Sine of an angle = Opposite side / Hypothenuse

Cosine of an angle = Adjacent side / Hypothenuse

Tangent of an angle = Opposite side / Adjacent side

Then let's consider this as a trigonometry problem, illustrated below. We have a camera, the Camera3D, which is located at (0, 0, -500) by default. Look at the sources to confirm, it is the case with Away3D, but it may be different in other 3D engines. We have a plane with a position yet undefined, but we do want to make sure that its fits perfectly into the viewport.

Where does that 60° angle in the illustration above come from? If we take a look at the Camera3D constructor, we find that it uses a PerspectiveLens by default. In that lens constructor, we can see the default field of view is 60, which is in degrees. It's too bad we need to look at the source to obtain this, it would have been simpler to use a getter to be able to check it, but there is none.

Good, we have an angle, we can calculate dimensions and positions. Let's set an arbitrary z for the plane, we'll see further that it doesn't matter.

var angleY:Number = 60;
var cameraZ:int = _view3D.camera.z;
var planeZ:int = 500;
var distFromCamToPlane:Number = Math.abs(cameraZ) + planeZ;

Since the field of view is vertical, let's measure the height of our plane first. Let's not forget that the field of view covers the whole height, but in order to work with rectangle triangles, we will use only half of that angle. We know the angle, we know the distance from the camera to the plane (the adjacent side), but we do not know the opposite side, so let's use the tangent. The Math.tan() takes radians, so just make sure to convert those degrees to radians.

var planeHeight:int = Math.tan((Math.PI / 180) * (angleY * 0.5)) * distFromCamToPlane;

Since this represents only half of the height, let's correct it for what we need further.

planeHeight *= 2;

To obtain the width of our plane, simply apply the aspect ratio of the image. In this case the image is a square, but it may very well be any kind of rectangle.

var aspectRatio:Number = WIDTH / HEIGHT;
var planeWidth:int = planeHeight * aspectRatio;

And now we are ready to create our plane with the proper dimensions and position.

_plane = new Plane(_material, planeWidth, planeHeight);
_plane.yUp = false;
_plane.z = planeZ;

Let's take a look at the whole process at once.

// get the scale to apply to the image
_matrix = new Matrix();
_matrix.scale(MATERIAL_WIDTH / WIDTH, MATERIAL_HEIGHT / HEIGHT);
 
// create the image data
_imageData = new BitmapData(MATERIAL_WIDTH, MATERIAL_HEIGHT);
_imageData.draw(new EscherImageClass(), _matrix, null, null, null, true);
 
// create the material
_material = new BitmapMaterial(_imageData, true);
_material.mipmap = (_imageData.width == _imageData.height);
 
// create the view
_view3D = new View3D();
 
// camera values
// see away3d.camerasCamera3D's default lens' field of view
// in away3d.cameras.lenses.PerspectiveLens
// it turns out that the field of view is vertical
var angleY:Number = 60;
var cameraZ:int = _view3D.camera.z;
 
// this position is arbitrary as we will see further
// useful to calculate the distance between the camera and the plane
var planeZ:int = 500;
var distFromCamToPlane:Number = Math.abs(cameraZ) + planeZ;
 
// since the field of view is a vertical angle, calculate the height of the plane first
// use pythagorean theorem to calculate
// tip for trigonometry: soh-cah-toa
// toa: tan(angle) = oppositeSide / adjacentSide
var planeHeight:int = Math.tan((Math.PI / 180) * (angleY * 0.5)) * distFromCamToPlane;
 
// and since it was for a rectangle triangle, thus half the size, double the length
planeHeight *= 2;
 
// useful for resizing the image
var aspectRatio:Number = WIDTH / HEIGHT;
 
// use the aspect ratio to calulcate the width of the plane
var planeWidth:int = planeHeight * aspectRatio;
 
// create the plane
_plane = new Plane(_material, planeWidth, planeHeight);
_plane.yUp = false;
_plane.z = planeZ;
 
// add 3D objects to the scene
_view3D.scene.addChild(_plane);
 
// add the away3d view to the display list
addChild(_view3D);
 
// listen to the updates
addEventListener(Event.ENTER_FRAME, updateHandler);

This should produce a result like this, where the image fits perfectly in the viewport.

For additional fun, I added an option to click to rotate it, so that it demonstrates that it is indeed 3D.

Sources

As I always do in my tutorials, you can download the sources. This project has been built for Flash Player 11+ with FDT. You may need to adapt the Flex SDK.

Enjoy!

I've been meaning to write about this for a while. Even though it is not specifically about development, user experience is something that creative developers should take into account when developing websites, games or other interactive experiences. Let's look at a couple of different ways of offering the user localized content, and try to analyze whether these options fall flat on their face or offer enough flexibility to answer your client's and their customers's needs.

Clients that offer services in multiple countries require their content to be localized, and for a good reason: inhabitants of all regions of the world want to be respected and obtain content in their own language. And I agree, it's a simple matter of respect.

The usual suspects

The way localization is most often achieved is by detecting the user's country automatically. This can either be done by localizing the IP address (see an example), soon by simpler means with HTML5's geolocation (see an example) or by asking the user to select a country.

LaCie does just that.

I think that this last option is lazy, both on the company's and on the developer's part. The user should not have to make such choices, especially when there are ways to make such a selection invisible.

What happens from there is that the website content is presented in the language relative to the country. It mostly works, but there are some situations where this logic is not sufficient. Let's take the example of some European countries, like Belgium and Greece. In Belgium, French, Dutch (Flemish) and German are the official languages. In Greece, where obviously Greek is the language, there is also need for a specific alphabet.

From adidas's country selection page, if either Greece or Belgium is chosen, the user has no choice but to visit an English website.

To adidas's credit, there is the option to change country at any time, the site even offers you a choice whether or not you want the cookie to remember your country selection.

Some websites, such a Nike's may offer a simpler solution, only choose your language, not the country.

All these options ask an action from the user before they can even see your products or services, and that small irritant is the first interaction they have with your website.

The mistaken

In all online stores (let's not call these e-commerce no more, please?), there comes a moment where you want to write your shipping address to obtain your purchases. If you have offered your clients a multilingual website, it's just normal that the input fields should support special characters too, regardless of the language the client chose.

When I moved back from Amsterdam to Montréal, I had to change many addresses for many of my services, both physical and online.

Apple requires me to have a valid credit card in the country that I select, as their products availabilities are based on the locale. However, their form failed when I entered my address.

I get informed that the city, that I wrote "Montréal", contains illegal characters. Obviously, this turned out to be that I should not use the accent. This is ridiculous, the user should not be told there is something wrong when in fact there is not. How complicated is it really to handle accented characters? I mean, if you want to provide multilingual services, get your act together and do it everywhere.

The ignorant

There is also the peculiar case of Urban Outfitters. The american company has oftentimes been vilified for stealing designs from indie designers or for shamelessly using native names in their product names, so what I will present here should not come as a surprise.

In Québec, a law states that if a business has a brick and mortar store in the province, the business is responsible to also have a French side to it's website. As mentioned in my introduction, it's a matter of respect to your clients in the locale in which you offer your products and services.

Urban Outfitters did not, and when legally instructed to do so, they decided to simply not offer their clients in Québec with no access to their website whatsoever. Just go to the store.

That is one stupid way to use geolocation! You disrespect a locale and it's language, and at the same time you prevent those in the locale who can speak and read English from having access to your products. Great way to prevent sales!

The road ahead

Of these different approaches I make the following conclusion and add a recommendation for multilingual website and software creators: first, indeed do geolocation to offer the most relevant content to the user. Do give options to change not only the country, but also the language. Paypal is the closest to this idea. If you change your location to Canada, you can then change the language to the available languages is that country, namely French and English.

But here is where I want to get to with this post: why stop there? Indeed, do present the most logical languages relevant to that area, but do not block the other languages from being available, especially since you have them in the database already. I understand that it may be mostly be relevant for expats and travelers, but I strongly believe that this kind of market is growing.

The iTunes Store is built in order to lock you into a language depending on your country. Websites like YesAsia are built with the idea that the content should be available to everyone, even if the clients are not from Hong Kong, Japan or Korea, where their offices are. They offer not only options for locales and languages, but a reference to the value of the currency so the customers may have a general idea of what they will pay.

In the case of Paypal, as I said before languages are chosen by country. They have many more languages, they should offer them. I can understand that there may be issues in doing so. For example, English is not legally bounding in the Netherlands, so important options are not available, not even deleting an account. How about presenting all languages, and just like YesAsia does it, add a note that explains that whatever is said in translations, the version in the official language(s) of the locale prevail?

Ultimately, it's always about how much you really care about your clients. For example, Asian countries dislike it when they are presented with English text rather than localized. Ask your markets to help you make a better localized version of your products, don't just expect your clients to swallow whatever you feed them. This is the kind of stuff metrics will not really show in numbers, this is an analysis you can make from talking with your markets.