Frame Rates

With today’s interactive entertainment, keeping the needle
at 60 is an absolute must.

A White Paper
from 3dfx Interactive

Much has been said about the importance of frame rate in real-time interactive 3D games.  If frame rates are too low, quality suffers; that much is clear.  But just what is too low?  And how useful are today’s benchmark tests in evaluating game performance? Is an average rate of 100 frames per second (fps) better than a sustained rate of 60fps? 
To answer these questions, industry observers and media professionals should consider the impact of human/machine interaction and latency, the distinctions between average, sustained, and instantaneous frame rates – and, most importantly of all, what this means to game players and other entertainment consumers.  While to many these variables may seem trifling, they actually have a tremendous impact on the performance of games and interactive entertainment.  

Latency Lowers Performance
Over the course of several decades, numerous studies have eyed the effects of lag time, or latency, in simulation applications such as military and flight training.  In one study, investigators calculated the effects of latency on people who were asked to capture a stationary target using a head-mounted system. Subjects placed an aiming device in a target circle as quickly and accurately as possible, and kept it there for at least 350 milliseconds (ms).  The investigators found it took far longer for subjects to achieve the goal when they faced an additional lag of 67ms on a head-coupled system (this is additional to the system’s existing latency of 40ms.  In another study, subjects were required to track a moving target by keeping a ball inside a box, and moving along an unpredictable 3D path.  When testers added additional latency of 40ms to the existing 40ms basic system lag, tracking errors multiplied dramatically.
Studies have also shown that time lags hurt our ability to manipulate and control objects.  For instance, pure lags of 100ms or greater have been shown to degrade the performance of a simple, remotely operated pick-and-place task controlled by a pair of joysticks.  In this test, subjects had to place four balls into appropriately sized boxes. According to the results of the study, increasing the latency caused an approximately linear increase in errors.  (For more information on these and additional studies on the effects of lag, visit www.psicologia.net/pages/hfact.htm.)
If latencies of 100ms or greater is too much, then what is acceptable?  According to other studies, only highly trained or experienced users can detect latencies below 50ms.) For the vast majority of users, latencies between 40ms and 70ms would be virtually undetectable.

Sacrificing Realism For Speed
While these studies investigated virtual reality and other simulation applications, the effects of latency are felt by a far broader audience with interactive 3D games.  One of the two most important ingredients in today’s 3D games is real-time interactivity.  When players are engrossed in their favorite games, they want their actions to be replicated on the computer screen at precisely the same moment they dictate them.  For many gamers, it’s a matter of virtual life or death.  Latency directly and significantly impacts a game’s playability.  It goes to the heart of the consumer’s satisfaction.
The other key factor–and one that is closely related to real-time interactivity–is realism.  Thanks to advances in 3D authoring tools, game developers today are able to create stunning 3D models and scenes that, in some cases, more closely resemble their digital effects counterparts in film and television.  Indeed, game developers face mounting pressure to publish new titles whose graphics rival the jaw-dropping effects of major Hollywood blockbusters such as Pixar’s Toy Story or A Bug’s Life.
In a perfect world, developers would be able to use as many polygons as they want in their 3D models and scenes. Players would be able to interact with these realistic worlds, easily moving about real time.  But this is not a perfect world, and today’s computing power is not nearly that sophisticated. So game authors have to compromise:  They sacrifice realism for speed. 
For instance, developers must diligently work to ensure that their scenes and models are sleek, using as few polygons as necessary to minimize the amount of content that must be generated with every frame.  Meanwhile, they frequently make imaginative use of textures.  For instance, replacing parts of an Apache helicopter model with textures that create the illusion of a complete 3D chopper—limits the number of  polygon and significantly increases performance. More polygons require the system to generate more calculations to produce the bits comprising a frame.  Today, game developers who do not follow these unwritten rules are bound to frustrate consumers by producing an experience that runs so slowly that it is deemed unplayable. 
Game development has for some time been a tug of war between high-end 3D graphics and real-time game play. That dynamic is not likely to change anytime soon.  All of this makes the raging debate over frame rates that much more important, because “fps” may be the single most important variable in game playback performance for several years to come.

Average, Instantaneous, and Sustained Frame Rates
So what is the lowest acceptable frame rate for realistic motion and true real-time interaction?  To determine this, it’s important to understand the importance of average, instantaneous, and sustained frame rates.
Graphics board vendors often use the time demo to calculate the average rate at which frames appear on the computer screen over a period of time—usually 30 seconds.  Time demos are a favorite benchmark of hardware vendors since nearly any graphics card  can achieve an average frame rate of 60fps.  To consumers, an average performance of 60fps sounds great. While the numbers sound impressive, however, they don’t come close to telling the whole story.
Time demos don’t consider instantaneous frame rate, or the rate at which the hardware draws a given frame in the game. These rates can vary dramatically – from a valley (or low) of only a few frames per second in visually complex areas of the time demo, to a peak (or high) of many hundreds of frames per second in a graphically simple scenes.
Work the numbers and the difference becomes clear:  A time demo score of 60fps may mean a peak of 120fps and a valley of 12fps.  If both the PC system and game software are capable of performing at 60fps, but the graphics card can only generate 12fps on a specific set of frames, then latency adds up fast: You’ll acquire 32ms via user input and game, 83ms to render the frame, 13ms to vsync, 10ms to display, and another 10ms from the graphics card’s input buffers.  This results in a total (and unacceptable) latency of 148ms.
With varying sources throughout a system affecting lag, any drop in frame rate to below 60fps may be detectable as lag.  If the frame rate drops below 29fps, latency soars over 100ms, and Tomb Raider’s Lara Croft will suffer a violent and unjust demise.  Whether consumers realize it or not, they care about instantaneous frame rate.
Along with average and instantaneous frame rates, sustained frame rates are also important.  A sustained frame rate is the rate at which a game plays at all times, not just during a highly optimized 30-second game segment.  If a product can sustain 60fps at all times, that means the frame rate never drops below 60fps. This translates into real-time, interactive performance.  With a sustained rate of 60fps, latency drops to 16.6ms per frame (or 81ms behind the application).  This equates to a consistently interactive entertainment environment even for highly trained or experienced players
 
Meaningless Benchmarks? 
High benchmark numbers are seductive in the game business. While they can be used to bring a product’s stellar performance to light, they can also confuse (mislead) gamers about the true performance of a game.  Since we have already determined that 60fps is an appropriate performance goal for a sustained frame rate, a time demo score of 200fps may seem like overkill.  However, an exceedingly high-time demo score could be sending the right signals:  It might say that the overall score was pushed up by a better-than-average valley frame rate.  Since the valley frame rate will always be lower than the average frame rate, this could make the card a good performer.  The problem for consumers is that there is no way to tell, unless manufacturers begin to explicitly address the numbers that matter most – such as sustained frame rate.
Fortunately, at least one widely followed benchmark agrees with this approach.  Game Gauge is designed to test products based on what gamers care about most: real-world performance.  To do this, Game Gauge focuses on frame rate, calculating a minimum, maximum and average frame rate for game titles tested on a particular PC system.  
In testing a particular hardware product, Game Gauge determines the Average FPS score by adding all of the average frame rates of several different titles. Scoring high on one game and low on another will not yield an overall higher score in the average FPS number. By scaling benchmark results across several titles, Game Gauge promotes the creation of 3D accelerator products and helps avoid the common practice of tuning a 3D card to a single game. 
As we have shown, among all the performance measures that should concern this industry, one stands out.  Sustained frame rate defines the very game experience, and is the only measure of a system’s true, lasting performance.  Consumers don’t remember peak performance, and they are not cognizant of a game’s average frame rate.  Sustained frame rate, however, establishes the feel of a game, and 60 fps is the target every graphics technology should strive to achieve.  

--end—

For information, contact:

Susan Carlson, Catapult Partners, Inc.   650-524-1720
susan@catapultinc.com