Skip to content


Of touch screens and game controllers

Game controller and on screen controller on mobile device

In the recent years, video games production and distribution have considerably been democratized. Gone is the time when it was mandatory to purchase expensive SDKs and get a license deal to have your games distributed. Web games and mobile games allow for innovations and reinventions of known models.

However, as innovative as this new game market can be, some creators still create games based on classic schemes, and there is nothing wrong with that. For example, even though new styles and sounds were born, rock 'n roll has been around for a while and is here to stay. It may be adapted and reinvented, just like games can be.

Some developers aim to provide their players with a classic feel, but there is a trend that should be put to rest: the on screen gamepad.

 

On screen gamepad

On screen gamepad controls

For those who grew up playing the Nintendo Entertainment System, the Sega Master System and all the following consoles, a gamepad is pretty much with what they learned to play games. Joysticks, trackballs and calculators for which you could swap design were on their way out for home consoles. Over time, gamepads also became more ergonomic and more comfortable, although they were growing more buttons every generation.

Handheld (and thus mobile) game-dedicated devices also made use of these control schemes, from the Game & Watch, the GameBoy, the Game Gear to the following devices.

Games were built around these human-computer interfaces, just like PC games were built for keyboard and mouse interaction.

And then came touch screens.

The Nintendo DS may have been the first device to bring a touch screen to mainstream gamers. Although at first, games were not using the full potential of that screen.

A couple of years later, the iPhone came out and brought games and touch screens to the mainstream populace. Other phones had a touch screen before, but none caught the heart of the customers as much as the iPhone.

No other company claimed they wanted developers to create content for their device, with as much fervor. An affordable SDK and somewhat reasonable profit margin allowed bedroom developers to create games with mainstream reach like never before. And they did.

On screen gamepad controls

Growing up with gamepads and side scroller/platformer games, many developers recreated controls they knew and liked. Games like League of Evil, Super Crate Box, Pizza Boy and Pix'n Love Rush are well executed and fun games.

But they do not belong on a mobile phone or tactile screen.

Gravis gamepad

Such games require dexterity and a tactile feedback that only a gamepad can provide. A screen feels like glass and is the same all over, it does not provide any feedback, making it very easy for fingers to slip.

In the case of a phone, your fingers end up covering a huge part of the playable area. Sometimes, like it is the case for Pizza Boy, the playable area is just designed smaller to make way for the controls.

It is a waste of good space, but also a bad practice that tries to reproduce an interaction scheme from an altogether different human-computer interface. Just like the Gravis Gamepad had it wrong when they tried to force a joystick into a gamepad.

 

Mobile devices add-ons

Mobile devices add-ons

Realizing that the on screen gamepad caused all sorts of issues, some companies thought of creating hardware that players could add to their mobile devices. These add-ons could be split into two types: those that are affixed to the screen and those that are separate and require an integration into the codebase of the software.

Attaching an add-on to the screen, like the Fling Mini (bottom left in the image above) and the Joystick it (bottom center), solves the issue of the expected physical feedback when playing. Parts of the screen still are hidden though, if not more so in some cases. And what if the game designs the gamepad not in line with the add-on placement?

The other types of add-ons are actually smart. The problem in this case is that the hardware developer expects games devs to use their API so that their add-on can be used. It's not that much of an inconvenient, however there is a multitude of add-ons. The image above samples only a few ones: the iControlPad (top left), the Caliber Advantage (top center), the 8 Bitty (center) and the PhoneJoy (right). And there are more. Game devs would then have to incorporate the API of each of those add-ons to provide support for any eventual add-on their clients might use. On the other hand, if the developers don't, the onus falls on the player to purchase many add-ons to be able to play with a controller, depending on which one the game devs chose.

This is ridiculous.

Let's also remember that the success of gaming on mobile phones relies a lot on the fact that the device is already in your pocket, no need to carry anything else.

The advantage of the gamepad on a console is that the gamepad is standardized and works for all games. Each game does not chose what gamepad it supports.

DS Paddle

There are some exceptions, obviously. A couple years back, a hyperactive reinvention of Space Invaders was released. For the DS release, an add-on paddle was created. The paddle was a hommage to the controls that the arcade Space Invaders sported a while back. It was fun  and novel to play for someone like me who grew up with a gamepad in hand.

But ultimately, the gimmick was superfluous. I would carry my DS in my pocket in the subway, but never the add-on. Customers want was is convenient, and carrying more things in their pocket is not. This argument has been laughed at by many, but over time the rise of gaming on mobile phones has been shown to hurt gaming-dedicated handheld devices sales.

 

On screen interaction

Legend of Zelda - Phantom Hourglass interaction

Nintendo has always really been good at making games that explains how to use their innovations, not that all studios or developers were always dedicated enough to apply or explore all those innovative ways.

The Legend of Zelda: Phantom Hourglass was an awesome game that managed to keep its feet rooted in its history while pushing how you use the handheld device in the game. The first thing you notice when playing is that, contrary to previous Zelda games, you move Link by taping holding the stylus where you want him to go. You would also tap directly on the objects and the enemies on screen to interact with them. Sure you can still use the D-pad and the buttons, but that made no sense, the game wasn't built for that.

No on screen gimmick. You want your character to go somewhere? Tap there. Developers should really think of that more often. The A* algorithm deals exactly with that logic.

Another remarkable thing was that the player could write and take personal notes on the map. Let that sink for a moment. It may seem like nothing, but for so long, inventories and maps might have been editable, but always within the confine of the user interface. This game offered the player the option to draw and write whatever they wanted on the maps, to not have to deal with the interface!

At a point during the game, the player has to fold a map to obtain some information about where to go further. So as a player, you tap, you click, you read and still cannot figure what to do. Eventually, you realize that you actually have to close the DS clamshell and reopen it and ta-da! The ink on you map is transposed from the bottom to the top (or the reverse, I forget).

That's smart! This is making use of the device and its specificities, and not trying to recreate old ways.

Lost Winds on screen direction control

Fast forward a couple of years later. Lost Winds is a side scroller game that was originally released for the Wii. This game already made use of novel ways to move the character by combining the thumbstick of the Wii's Nunchuck and pointing at the screen with the Wiimote.

When this game was ported to iOS, I was really curious. I was really charmed with the Wii game, but somehow the iOS port is even better. Pointing at the screen with the Wiimote has been easily adapted into swiping the screen with your finger.

How do you move Toku around without the thumbstick? For one thing, there is no on screen game pad. A bit akin to how you would move Link around in the Phantom Hourglass, simply tap in the direction where you want Toku to go. Don't want to tap, tap, tap and tap again? Simply hold the direction for a while, and Toku will move automatically in that direction. See the screenshots above, there is even a visual indication. While he walks, you can swipe and attack monsters. Tap anywhere and he stops walking.

This is a smart adaptation of the on-rails feeling that that you get from endless runners, such as Canabalt. While at moments you may want the character to move automatically, at other times you want more control. This is obviously no way to play a reflex-demanding Mega Man game, but tactile screens are not the proper target for those games.

 

Create controls for the device you target 

With the advent of new microconsoles like the OUYA and the GameStick, and with the Playstation Vita's affordable SDK, it will be possible for game creators to create games for devices that benefit from an actual game controller, rather than try to recreate those controls. It's too bad that Nintendo is not open to that kind of indie creation, so many game devs would be thrilled to have their games released on Nintendo devices.

There is obviously a demand for side scrollers and adventure games on mobile phones and touch screen devices, but it's about time that developers start exploring ways of controlling their games according to the device onto which the game is published.

Posted in Game Design, Game Development, Interaction Design, Mobile Development, Opinion, User Experience, UX.


Moving towards user experience

Digital production positions diagram

A couple of years ago Malcom Gladwell presented what he dubbed the 10,000 hours rule, which explains that for one to become proficient with a skill, 10,000 hours have to be invested into that skill. Oddly, most tech journalists and bloggers seem to take this for gospel. I'm not going to dissect and argue against Gladwell here, but rather I have a different perception of his argument. I believe that if I invest this much of myself into a certain unique skill, I'm going to grow bored of it and I will need to move on.

I have been developing and coding for approximately this long, it's time to move in a different direction.

Oftentimes when trying to explain to clients and colleagues that even though I am good at coding, I do not care for it as much as I care for what the user experiences. Even clients, faced with my technical knowledge, wanted to push me towards back-end coding, which has nothing to do with visuals or interfaces, and thus is of no interest to me.

When I stumbled upon the diagram above, it made so much sense to me! It was so clear that I could then illustrate to people, even those who have no technical knowledge, what I do and where I what I am aiming to do. I need to move towards the left side of that diagram, and not the right side!

 

Don't let them put you into a box

I have never been a fan of simply being told what to do without seeing the big picture. This is why I always strived to be included in every step of a project, from the brief to the strategy, from the design to the production.

However, not everyone is at ease with that. I once had a manager that was displeased because I wanted to take the time to sketch out storyboards, he thought that our department should only have cared about development, and that storyboarding should only be done by information architects or designers. I think it was short sighted, if I knew the assets I required to build the project and knew how to share the requirements efficiently, why stay confined into the limited description of a job position?

Another time in another place, a director of production had just been hired. He was in charge of the producers/project managers as well as the technical department. From then on, he chose to limit our input to only technical things, we were not to be part of the creative brainstorms. Why would removing the people who build the creative experiences from the creation process be a good idea?

I recently spoke with a friend about expanding or reorienting my service offer, and he suggested that I should choose one specialty, and stick with only this one, rather than many. He thought being technical, creative, strategy- and user-oriented were too many things at once.

A university teacher of mine once said that we should aim to become like the scholars of the Renaissance and try to learn as much as we can from every field. And that is what I want to do, no matter if others feel threatened because they are comfortable with only one specific skill. Too often have we heard stories of specialists, whether they are developers or factory workers, that are rendered useless once a new trend appears.

 

Updated service offer

Sketches

With all that in mind, I reorient my service offer towards user experience rather than front end development. Fabricio Teixeira dubbed the person with a multidisciplinary approach to the field the UX chameleon and I find that image quite apt.

As a user experience consultant, I intend to use my technical knowledge and experience to suggest smart strategies in using technologies to appropriately answer the needs of the user, rather than simply pushing for the latest tech buzzword. I can create documents to help others get a clear understanding of the goals to reach. I aim to work more upstream in the creation process: analyze the client's brief and then work with the strategy and the creative teams.

I have already been drawing sketches, creating user flows and building information architecture for a while, now I am pushing this side of my aptitudes as my main service offer.

Posted in Information Architecture, Opinion, Storyboarding, Strategy, UX, Workflow.


Marine Science Magnet High School installation

Last spring, Float4 started building an installation for Marine Science Magnet High School. This organisation, located in Connecticut close to the Long Island Sound, centers its educational program not only on academic knowledge, but also on natural sciences, mainly marine-related science. I was stoked, because I always wanted to participate into more educational projects, whether for schools or museums. Also, since we were targeting a young audience, we could allow ourselves to design interaction without taking them by the hand and show them how interaction works.

Even though I have posted screenshots in my portfolio–and I hope to get a video of the kids interacting with it eventually–I thought it would be great to share how we built that project, a case study of sorts.

 

Navigation

This installation was split between three states of user interaction:

  • Attract: At first, catch the user's eye and interest. It's at this point that you invite the user to interact.
  • Engage: Once the user interacts, add some information with which the user can play a bit.
  • Explore: The user can explore the experience further from there. Offer more content upon the user's request.

These are just general points as to how you get a user to interact with your installation, which we applied for our installation. Let's see how we designed this.

 

Design and user flow

The original concept was heavily influenced by infographic design, with heavy fonts, flat and bright colors. To be honest, it was a bit of a mess, too many elements at once for the eye to know where to look and how to read. However, the idea was there, we simply needed to refine it into a finished product.

Marine Science Magnet Highschool Vanderveil original user flow

As can be seen in the concept images above, there is a clear disconnect between the design of the map view and the design of the infographic view. There were also lots of elements that needed not be repeated in every state, such as the temperature and other weather information.

I worked with Raed Moussa to streamline the design, keeping only necessary information. He was able to create a cohesive visual environment between the map and the side view.

Marine Science Magnet Highschool Vanderveil redesigned user flow

As for informational elements, we opted to keep them only where they were pertinent:

  • The attract view contains animated weather information over a map of the Long Island Sound area, with a clear call to action to invite people to touch the screen.
  • The engage view presents people with some additional information overlaid on the same map that they saw from the attract view. All the information is relative to the map they see: water depth, buoy locations, shipwreck locations and historical events.
  • The explore view contains a figurative slice view of a coastal region in the same area. Species icons are displayed in locations where said species would usually live, and if the user so desires, more information on each species is available upon touch.

 

Water depths

One of the scientific information that we had to overlay on the map was the water depths of the area. This information is usually printed directly on the maps or embedded into ships navigational systems. We wanted to display this information in the way topography maps display heights.

Long Island Sound nautical chart

The inconvenient was that the available maps did not provide it directly. Somehow, most of the depths variations on the map were not big enough, most were simply noted by numbers.

Water depths highlighted values

So we had an idea: highlight all the numbers with different colors in order to create a heat map. And I mean manually highlight! At first, we ordered the physical maps, we planned on sitting at a desk with markers to highlight the numbers. We looked into how to take a high resolution photo of the map, but the depth numbers came out all blurry. It was a no go.

Finally, we were able to get the digital maps from Nautical Charts Online. From then, I took a couple of days in Photoshop with my graphic tablet to highlight all the numbers according to a range we predetermined. We were able to get a good heat map.

Long Island Sound water depths as heat map

It was still necessary to incorporate the information of this heat map into the design. Raed was provided a vectorized topographical render of the heat map and was able to incorporate it smoothly into the map.

Long Island Sound water depths

 

Transition from map to slice view

It was a bit tricky to think of a transition between the map view, which is a flat map of the Long Island Sound area, and the side view, which is a slice view of a coastal area. At the time I was watching Avatar: the Last Airbender cartoon. In its intro, there was exactly what I was looking for! Take a look between 0:12 and 0:28.

Avatar the Last Airbender intro

This was a great inspiration. Being a fan of old Final Fantasy games, I also thought of how the map was tilted back when the characters would hop onto an airship.

Transition storyboard

With the storyboard above, Francis Dakin-Côté was able to create the perfect transition we needed between the map view and the slice view. This animation is presented when a user chooses to explore the installation more in depth, providing a logical transition between the states. The same animation is played backwards when going from the slice view to the map view.

 

Development

As I probably mentioned in earlier posts or tweets, Float4 most often uses RealMotion, its own software. The development was then split in two: the RealMotion graph that handles the interaction and embeds the content, built by Bruno Gohier, and the interface and content, which I built.

Akin to how Scaleform allows to embed Flash content, so does RealMotion. Since the content for this installation needed not to live anywhere else, and also since it was mostly a two dimensional multitouch graphical interface, Flash was the most indicated technology to choose here.

As I did for other projects these last years, I built the application with the RobotLegs framework. I also used AS3 Signals rather than events, which is a charm to work with.

I may have said in other posts that RealMotion can communicate touch points to the application via the TUIO protocol. I was using the TUIO AS3 library, although along the way I ended up finding and fixing some of its memory leaks and other issues. Don't get me wrong though, it's still an awesome library.

The development was pretty much straightforward, however I think I haven't done something this demanding on a machine since the adidas Originals Women's Lookbook. This project is in full HD (1920 x 1080) at 60 FPS and was required to work in multitouch.

The transition animation ended up being a lot more work than expected. Using a 60 FPS video would prove impossible, as it would take too much time to start and skip too many frames. I tried scrubbing the video manually at every frame, which ended up being worse. I also tried loading dynamically all 360 frames for the transition and built a class that would display the animation. This worked smoothly and was quite satisfying visually. The issue was that once all the images were loaded, the application would freeze while trying to add the images to the stage. And this would only work while developing in FDT, as soon as I tried in a browser or within the context of RealMotion, a crash would occur. So I ended up using two 30 fps videos, one forward and one backwards, since the user has to be free to come back to the map state.

It was not my first go at multitouch, so I could apply the discoveries I had made there. The main thing was really about keeping track of the touch ids: once an item is touched, no matter how many times other touches disappear from the surface, the item is still considered touched as long as the first id is still touching the item. It would translate into something like this:

ActionScript 3.0:

  1. override protected function onReleased(_:TuioTouchEvent):void
  2. {
  3. if(_.tuioContainer.sessionID == _touchId)
  4. {
  5. super.onReleased(_);
  6. if(!_isTouched) _released.dispatch(this);
  7. }
  8. }

Another touch behaviour that I implemented from the slideshow was the "touch and hold to trigger". You can read a bit more about that logic in previous post. In this case it was used in an area where images and videos could be dragged, a bit like a vertical carousel. It then became possible to distinguish between dragging to change images and triggering to start a video.

Because laser touch detection cannot be as precise as touch screens, we wanted to avoid small UI elements as much as possible. We decided that textfields would be draggable, like on mobile devices, also because it allowed us to remove the drag bar and the arrows. More space for content.

The weather data presented in the attract view is pulled from the Yahoo! Weather RSS Feed, whereas the buoy data is obtained from NERACOOS. Actually, if you ever need to use buoy data, contact these guys, there were thrilled to help us use their service. Even before completion of the project, they showcased the installation in their newsletter.

All other historical and animal data is entered in a CMS by the users (students or teachers).

The RealMotion graph contained the Flash content and presented it on screen. Since this software is really strong with visual effects, there was no need to try and implement them in the Flash application. We opted for a water ripple effect, which made sense in the context. As the user touches or swipes the surface, water like waves are created across the screen. In the map view, we also added a mask so that the waves would ricochet off the shores of the coast.

We had to come up with a way to change the states of the waves according to the states of the Flash application. The best way to do so was to send values via the Socket AS3 class. RealMotion contains a module (called "box" in their environment, akin to Max's) that could receive data from a socket. That box can currently only receive bytes, but I believe that it could grow into something more useful where Flash applications and JavaScript applications could send bytes, integers and strings. Even better, if two way communication could be implemented, a new layer of flexibility would be available to content creators.

 

CMS

In order to manage the content, we provided the client with a CMS (Content Management System). I believe that this project can live long and evolve according to the students' work. Rather than provide the client with a paper manual, I took the landing page of the CMS as an instruction page, and suggested how they could use that tool:

Do not think of this installation simply as a static repository of information, but rather as a dynamic space to display involving and evolving information concerning the Long Island Sound area. Rather than throwing in all the data at once, maybe there could be a theme for each week or each month.

Here are some ideas:

  • A student's essay receives a great grade: if a student's essay on a subject that is pertinent to the installation (Long Island Sound area, history or sea life) is well written and worth sharing with the school, present it by adding items in the appropriate sections with excerpts of the essay.
  • Shipwreck week (or month): showcase only shipwrecks in the Engage View section by deactivating all items that are not shipwrecks.
  • Shellfish week (or month): showcase only shellfish species in the Explore View section by deactivating all items that are not shellfish.

Obviously, these are simple examples, but it's up to you to engage the student in using the installation and maybe write content for it to be shared with fellow students.

As you can see, I believe that as container as well as content creators, we may have a responsibility to inspire our clients on how to use the tools we create for them. Sometimes it's obvious (product display for business), but in this case, I thought it would be best to also guide teachers and educators. These people already have a lot on their plate, let's make sure what we create answer their needs and doesn't complicate their work.

CMS information architecture

I built the information architecture for the CMS and gave this information to WLAB, the service providers who developed most of the CMS. They used CakePHP, a framework I didn't know then. To be honest, I found this framework too bloated and convoluted for the needs of this project, a fully custom CMS would have been preferable, quicker and more flexible. The providers became overloaded with other projects, so I asked Simon Arame to take over and help us finish the work.

CMS with design

In the end I went through the code to separate CSS and JavaScript from the PHP/HTML. Simon helped me deploy the CMS and the database onto the installation machines.

 

Installation

The installation itself consists of two separate screens, each with its own computer. One of these computers acts as a server for the CMS, as mentioned above. Each screen is encased with a PQ labs frame, which detects touch points. This hardware is handled by RealMotion.

The installation is somewhere in the high school. As I say at the top of this post, we will eventually film people using the installation. However, as the school is located on the path of hurricane Sandy that devastated a lot of the American east coast recently, we will leave them time to sort their priorities before we push for them to let us come in and film.

 

Looking ahead

This long project represents a bit how I like to work: touch every bit of the project, do some user experience design and art direction as well as some project management. It gave me a clear view that I'd rather leave hardcore coding aside and put my energy into UX and storyboarding as much as I can.

I'm proud of this project and can't wait to see where the teachers and students of the Marine Science Magnet High School are going to take it!

Posted in Art Direction, Commercial Works, Development, Information Architecture, Post Mortem, Projects, Storyboarding, UX.

Tagged with , , , , , , , , , , , , , .