We take a closer look at the beating heart of WildStar with Lead Software Engineer Steve Moret.
The game engine is often an unsung hero of modern MMOs. When it works brilliantly we hardly even notice it, focusing instead on the dramatic art style, sublime musical score or excellent narrative. It’s only when problems emerge that we sit up and take notice how our favorite games are presented to us.
But what about the technology that brings it all together? After seeing the upcoming MMO for myself, I wanted to recognize this unsung hero of gaming greatness, and so I asked the folks behind WildStar if they’d shed some light on how their engine works. Whether it’s processing game rules or delivering pixels to our screens, WildStar’s MMO engine is a complex machine that acts as a window into the world of Nexus.
With Carbine’s upcoming MMO entering closed beta, we were eager to take a closer look at this mystical device. Luckily, Lead Software Engineer Steve Moret agreed to give us a rare peek inside the box and explain some of how WildStar’s engine works. From those first lines of code to more recent features, Moret explained what goes into the software that will be sitting on our desktops.
What’s the starting point for something as large as an MMO? Do you begin with something off-the-shelf and build out from there, or do you literally start from scratch?
Step one for us, was to decide what we could use off the shelf, and to see where it would take us. Back in 2005 we knew we had a few years of R&D time, so we evaluated a few off-the-shelf engines and, while these engines would let us show off some fancy graphics in year one, they didn’t look like they’d scale to our needs 5+ years later. We decided to spend a majority of our work home brewing the correct solution for each problem.
Luckily, a majority of our problems were already known. Many of us Carbinites had recently left Blizzard after launching World of Warcraft. To be honest, I had many of our anticipated problems spelled out for me during my interview with Carbine. It was obvious to me from day one that the entire team already knew what we were signing up for. In fact, most were excited and overjoyed to try and start over, this time, not making the same mistakes that might have been made before.
What parts form a modern MMO engine both in terms of what we see as gamers, and what we don’t?
Obviously every game is going to be different; but our technology at Carbine is broken down along 3 boundaries (many of which share components): Clients, Servers and Tools. The clients are things like our patcher, the game client itself and, strangely enough, our UI editor (since the editor is going to be used by players, we treat our UI editor as a Client application and not a Tool). The servers are made up of all the processes necessary to handle the back end of things (David Ray’s informative dev blog covers lots of this), and the tools are all the tools we use to both make the game, and support customers once the game is running (GM tools, log processing, web reporting and game access, etc).
Making up those parts are lots of smaller parts, things like an OS abstraction library, a networking library (for communicating), a graphics library (for drawing pretty things), database libraries for reading / writing data, graph navigation libraries for pathfinding, scripting languages (both server side and client side), physics simulators, input handlers, joystick routines, localization code and more. Most of these aren’t directly recognizable to the end user, certainly they appreciate that their game draws pretty and that their text is in their specific localized language but, for the most part, they’re oblivious to what is happening or why.
Some of the parts I’ve listed above are combined to form even more complex layers. This is where noticeable features begin to emerge. For example our graphics library may handle drawing polygons to the screen but our model system sits atop that and handles animating skinned models, and our costume system sits atop the model system and handles swappable textures and geometry on those models.
Gluing all those little and medium sized parts together while adding in a lot of game content is what the game layer is filled with. This layer contains all the logic to give you XP when you complete a quest, or to make sure a valid path exists between you and the monster that you wish to do a flying leap toward, or to make sure your avatar is missing a weapon when you get disarmed by your opponents.
These game layer systems exist in forms across each of the three components (clients, servers, tools). Sometimes in slightly different ways (for example a server has no idea what language a player is playing in, nor does it even have access to localized text) but some version of the Quest system exists on both the clients, the servers and the tools.
When building an MMO, what comes first – the client or the server?
In my opinion, it better be both. But in our case we had a simple terrain and model renderer first (about a month or two into development); a couple months later we had a basic server that let us run around and see each other (needless to say this was when we received our first cheater). We then started layering in all of our complexities. At Carbine features are always added to both the client and the servers at the same time, usually by the same programmer. Fun fact: we still maintain our standalone server-less Client; our artists use it to preview their art without having to connect to a server.
We take a closer look at the beating heart of WildStar with Lead Software Engineer Steve Moret.
How do you balance what to perform server-side versus client-side?
The general rule at Carbine is that we treat your client as a dumb terminal that is trying to send us malicious data at all times. That means that basically everything has to pass through the server for validation at some point.
At the same time, we have to assume you have tons of latency. I’d really love it if the world had a network infrastructure that could guarantee <10ms pings from any two points on the globe, but thanks to the laws of physics that’s just not gonna happen. We generally assume your client has at least 300ms of lag at all times. This means we have to simulate many things on your client while it is in transit to the server. Sometimes this might mean you play a pre-cast animation for a spell, only to get the message “Invalid Target” because your target is already dead. We’ve found it’s hardly noticeable vs. hitting the attack key a fraction of a second later, but I guess the end user will be the final judge of how well we hide the lag.
What is the development process like? Are there similarities with standard software houses that use an Agile process, or is there more to it?
We’re a mix of Agile principles as well as your traditional waterfall model of software development. While we have the traditional Art, Audio, Design and Programming departments many of us have broken up into Agile style Feature Teams. These teams handle core parts of the game (like Social, Economy and Encounters). They have both long and boring traditional meetings as well as 10 minute daily stand up meetings. We don’t really scrum and sprint but we kind-of do.
What we certainly do a lot of is iterating; features get rewritten and redesigned many times before they’re even shown to anyone outside of a Feature Team.
What stage of development is the engine currently at? Is the core complete, or are you continually adding new features?
Mostly done; yes and yes. While I’d say the core of the engine is done; we are continually prototyping new quest objective types, path missions, adventures, PvP battleground types, arena styles, Warplot components, neat things to put in your house and more. The goal has always been to have a flexible MMO engine that we can continue to support forever (that’s kinda scary to think about now that I say it). We’re always going to need new bosses, with new abilities and new telegraphs, giving the player awesome loot with new effects. All of that will keep us developing forever.
Can you give us a feel for the complexity of WildStar’s engine, and how big a project it is compared to art, content creation and other areas of development?
Well, we have lots of code (99% of which is C++). The engine is roughly 750,000 lines, the Client specific parts are roughly 250,000 lines, the servers are about 500,000 lines, and we have over 1.1 million lines of code in the 130 or so tools we use daily. We’re also pretty good at deleting unused projects and not leaving commented code in our depot, so I’d imagine the 2.6 million lines of code we have are live, active, important lines of code.
Our programming department only makes up 25 of the 210 Carbine employees that I’m aware of. Although, given our job postings those numbers may have slightly changed by the time this article is posted.
What’s been your biggest challenge that’s arisen from a request from the design team?
Luckily that feature was cut, so I can’t really talk about it. But problems regularly arise from Carbine’s over-the-top, pie-in-the-sky, go-buck-WildStar attitude. Some of the best ideas take the most amount of server database storage and there are some hard limits on what we can store if we want to handle the numbers of players we’re hoping to have (i.e. lots). Add to that a desire to not delete old accounts and we end up having to fit everything in 2-3 megs of player storage. Not so great.
Player exploration was a great example of pushing this limit. The size and count of explorable hexes had to be carefully tuned to not cost us too much database storage. Make the hexes too small and we need 100s of megabytes of storage per player, make the hexes too big and you don’t have a reasonable amount of fidelity to show on your map. Hopefully we’ve hit a happy middle ground where players don’t have to worry themselves with the cost per gigabyte of enterprise class storage.
Is there a feature of the WildStar engine that you’re particularly proud of, or which other games don’t have
As a programmer at Carbine, my personal favorite part of our Engine is that math is expressed in typical expression syntax, for example when calculating the barycentric interpolation of a triangle defined by the vectors v1, v2, v3 and scalar co-efficients f and g you can just type:
CVector3 result = v1 + (v2 – v1) * f + (v3 – v1) * g;
In my previous experiences at game development studios, math libraries were written in Intel Assembler Notation and the above would have to be written as:
VectorSubtract(temp1, v2, v1);
VectorScaleFloat(temp2, temp1, f);
VectorAdd(temp3, v1, temp2);
VectorSubtract(temp4, v3, v1);
VectorScaleFloat(temp5, temp4, g);
VectorAdd(result, temp3, temp5);
This lets us express math that looks and feels natural, and we’ve come up with neat techniques using C++ templates to prevent temporary copies by lazily evaluating upon assignment. I certainly hope that other game studios are also taking this path; the old style of writing math with function calls stinks compared to writing clear expressive math expressions.
So, the other programmers have officially said I’m too nerdy, and need to include their favorite Engine features too, so I’d like to also point out that we have an awesome database layer that has a version control system built into it so that you can make local changes and preview them before committing them to the database for others to see (or even suspend and transfer changes to other users). Also, several people said that sockets and plugs (a system we use to replace terrain features for dynamic events) are really awesome in that it is very user facing and really makes the world feel like it is changing as you play. But to be honest, I’m just happy that I move the camera by typing things like:
CVector3 velocity = CVector3::ZAxis() * CMatrix::RotationYawPitchRoll(m_cameraYaw, 0, 0);
m_positionVelocity = velocity.Normal() * (float)cv_CameraCharMovementSpeed;
m_position += m_positionVelocity * secsPerFrame;