Beneath the hood, there are countless methods by which computer programs display information on our monitors. The inner workings of computer applications consist of long, complex lines of code – and most source code is proprietary.
Unlike the World Wide Web, where a common language – HTML – has enabled the mashing-up of different data sources for rich displays, most desktop applications have no common language. If you want to, say, add “play” and “pause” buttons to the Microsoft Word toolbar so you don’t have to switch over to iTunes, you’re out of luck – Microsoft’s and Apple’s proprietary source codes don’t allow for such customization.
But across all the millions of different applications out there, there is one thing in common: They all display information on our monitors using pixels.
And that’s where the research of two University of Washington computer scientists comes in. They have developed software that gives you customization by analyzing not the source code of different applications, but the pixels they display on your screen.
The tool, called Prefab, can identify text, buttons, progress bars, sliders and many other graphical elements that we’re used to seeing in user interfaces. It then can modify what is actually displayed on your monitor.
“Microsoft and Apple aren’t going to open up all their stuff,” James Fogarty, a computer science assistant professor, said in a UW announcement. “But they all create programs that put pixels on the screen. And if we can modify those pixels, then we can change the program’s apparent behavior.”
If you listen closely, you might be able to hear Microsoft’s legal department quickly drafting up lawsuits. Just how pissed off would software companies get over Prefab?
“It’s not a question that’s really been asked, so we don’t know the answer to that,” Fogarty told seattlepi.com in a phone interview. “People embraced the mash-up on the Web.”
He hopes developers would embrace Prefab.
There’s more to the concept than just adding buttons to Word. For one, there is huge potential for Prefab in the realm of computer accessibility, Fogarty said.
In the video above, Prefab is shown highlighting the closest button to a mouse cursor – also known as a “bubble cursor.” This “target-aware pointing” could make using a computer easier for people with movement disabilities such as Parkinson’s disease, cerebral palsy or muscular dystrophy.
Existing accessibility options must be programmed into individual applications. Prefab could cut out the need for that, becoming one usability tool for all applications.
“It’s not up to you, as an application developer, to implement all of those” accessibility options, Fogarty said. “This just runs on top of them.”
What else could Prefab potentially do?
- Translate text without requiring access to source code.
- Take a Web page or app and redesign it for mobile devices.
- Reorder menu items.
- Display multiple image previews in Photoshop.
- Re-skin an entire application or operating system.
- Pretty much whatever visual changes you can imagine.
“This sees the same interface you do,” Fogarty said. “This lets enhancements and modifications see it the same way you do.”
Prefab was developed by Fogarty and UW doctorate student Morgan Dixon. Their research paper was picked as a winner of a Best Paper award for the ACM Conference on Human Factors in Computing Systems (CHI 2010), which will take place April 10 to 15 in Atlanta.
Fogarty said they are working on a way to commercialize Prefab – preferably as downloadable open-source software. He also would like to build a community around it, so people can add to Prefab’s knowledge of the nonstandard buttons, icons, sliders and other GUI elements found in the vast software ecosystem.
And, of course, the researchers hope Prefab sets off a wave of innovation. Especially as Web design, which largely has used a fairly standard range of graphical elements, begins to diversify as well.
“All of these problems are coming back on the Web,” Fogarty said.
Updated 10:50 p.m.: Clarified that for CHI 2010, their paper won a Best Paper award, not the entire conference.