Eve Online and Virtual Reality

Eve Online is one of the world’s largest gaming universe’s with over 75,000 dedicated online gamers actively subscribed to this award-winning, science fiction MMORPG. This persistent universe where everyone has a role and a place, made of art, technology, and imagination. Players take on the role of pilots and adventure in a huge in depth and hostile galaxy seeking fame and fortune.

Only one problem, its growing too fast! Read on to find out 4000% performance improvement!

Eve Online is one of the world’s largest gaming universe’s with over 75,000 dedicated online gamers actively subscribed to this award-winning, science fiction MMORPG. This persistent universe where everyone has a role and a place, made of art, technology, and imagination. Players take on the role of pilots and adventure in a huge in depth and hostile galaxy seeking fame and fortune.

Only one problem, its growing too fast! Read on to find out about 4000% performance improvement.

The Challenge

Immediate user access and application responsiveness are vital to success in the competitive online gaming market. Sluggish system response to player input is frustrating and may even result in players abandoning the game. In September 2005, Eve Online set a new record of over 15,000 concurrent users. The growing popularity of EVE Online was degrading performance as more players came online simultaneously. Players were beginning to get frustrated as some frequently accessed game features were taking up to 20 seconds to load.

The degrading performance potentially compromised CCP Games’ mission of attracting and retaining customers by providing top quality online entertainment, based in part on maintaining quality of service.

Performance analysis revealed that disk queues were running around 40. This was much too high. Microsoft recommends upgrading to faster hardware when disk queues reach 3 and conventional thinking suggests that adding concurrent users requires more servers. The Eve application itself was running fast and scaling well to accommodate tens of thousands of simultaneous users, and there was adequate computer power available from CCP Games’ infrastructure of IBM eServer xSeries and eServer BladeCenter servers to keep up. However, the system’s storage was not fulfilling requests quickly enough.

Conventional hard disk-based storage is constrained by physical considerations. A magnetic, spinning disk must be written to or read from by a physical arm traveling across the disk. Precious milliseconds go by as one part starts to spin up, then another moves to the appropriate area. Even the fastest hard disk drives have peak performance access times of 5 milliseconds. Multiply this by the billions of operations demanded by users daily, and a considerable lag results. This was the scenario at CCP Games.

The Solution: A Solid State Disk

CCP Games recognized that monolithic RAID, additional servers, adding memory and tuning the underlying SQL Server database were not going to solve the I/O performance bottleneck. “We did consider upgrading to faster disks, but the specs for the solid state disk were so insane that we had to look into it,” said Jörundur Matthíasson, Database Manager for CCP Games. One solid state disk, a Texas Memory Systems RamSan-400, purported to deliver 400,000 I/Os per second, has 3,000 MB of internal bandwidth with latency of less than 15 microseconds. After careful evaluation, the company deployed the RamSan-400 solid state disk, which promised to scale concurrent users by improving server efficiency

Solid state disk is ideal for storing performance-demanding data and accelerating application performance. It is a high performance storage system based on non-volatile solid state disk, ensuring both speed and reliability in accessing data. Solid state disks solve the problem of physical constraints by replacing hard disk drives with high speed circuitry. Instead of a rotating disk, a solid state disk uses memory chips to read and write data, allowing storage to catch up with the rest of the computing world. A solid state disks consistently decrease the response times of demanding applications and are widely used to accelerate enterprise applications like OLTP databases, batch processes, and data warehouses, modeling and video editing.

Although the deployment of a solid state disk to accelerate an online gaming universe was a first, the underlying storage bottleneck problem was a classic Online Transaction Processing (OLTP) one. CCP Games makes 60 million process calls per day with around 1,250 transactions per second at peak hour, so they need to maintain high performance so that the game is smooth.

CCP Games has a multi-tiered architecture with front-end web and proxy servers connecting to the EVE application servers and back-end database servers. The database servers are running SQL Server 2000 and are clustered using Windows 2003 Server Enterprise Edition. Setting up the 64-Gigabyte solid state disk was smooth and simple. From a DBA perspective, the solid state disk presented as just another storage device so there was no learning curve, despite this being CCP’s first experience with solid state disk. CCP moved the most heavily accessed data onto the solid state disk, along with the “tempdb” and the transaction log file. One table alone gets 8 million records added per day plus a huge amount of selects and updates. Less frequently accessed data remains on conventional IBM storage.

The Result: 4000% performance improvement

The solid state disk immediately improved Eve Online’s performance and its ability to support concurrent users. Within four days of installation, EVE Online set a new Peak Concurrent Player (PCU) record with 17,032 playing at the same time.

The solid state disk eliminates virtually all database related lag in EVE Online. It removes the storage access bottleneck that was slowing down Eve Online and is delivering efficiency gains throughout the system and faster response times. Eve Online now has more concurrent users receiving their data at higher speeds than ever before. The wait queue on the solid state disk is not visible, meaning it is below 0.5. Another database performance measurement, “latches/total wait time” went down from 25,000 to 4,000. These technical measurements are indicative of stellar performance – 40 times better than before.

The solid state disk is expandable, so CCP Games can be assured that they can keep pace with increased performance requirements while protecting their storage infrastructure investment.

“The effect of the solid state disk was immediate on both system performance and customer satisfaction,” said Hilmar V. Pétursson, CEO of CCP. “After installing the solid state disk, our player forums quickly lit up with enthusiastic and positive comments. Now we can focus our engineering resources on expanding our game universe and the customer experience without the limitations imposed by hard disk performance. It’s going to be a nice change for our developers.”

Texas Memory Systems manufactures solid state disks. Their RamSan line of solid state storage is used to accelerate enterprise applications like OLTP databases, batch processes, and data warehouses, modeling and video editing.

CCP Games a privately held company, creates challenging, massively multiplayer online games. CCP creates, develops, publishes, and distributes its games, and is one of the few MMOG companies that have experienced continued user growth since its start. CCP Games is headquartered in Reykjavík, Iceland.

EVE Online is a massively multiplayer online role-playing game (MMORPG) and is set in a virtual world of galactic magnitude that is governed by a hyper-capitalistic economy.