[In this reprinted #altdevblogaday in-depth piece, EEDAR's chief information officer Ted Spence offers a step-by-step guide for tuning your server and fixing a program without needing radical surgery or a complete rewrite.] A colleague of mine has a server performance problem. It's a regularly scheduled task that has to work through tons of raw data in a database. The task is reasonably fast except in rare cases. Naturally, over time the volume of data will only increase; so what can we do to make it more reliable and predictable? Okay, first off, I cheated a bit. This problem hasn't just happened to one colleague, it's happened to virtually all of them at one point or another. Slow tasks occur naturally due to the inevitable deterioration of software systems over time – more data accumulates, other features get added, the operating system gets patched, but the task still runs on the same old hardware. When you first write the task it runs great: Compile, build, tune, and it's done. Then you forget about it for a few months and it gets slower and slower every time it runs. It's like the frog in the saucepan – because the change is so gradual, you don't think about it until it reaches the point where it causes pain to everyone. If you have a chance to design these tasks fresh from the start, there are lots of great ideas about how to build them so they can scale. But let's spend today's article talking about what you can do to diagnose and fix a program – one that isn't yours – without radical surgery or rewriting it from the ground up.

Isolate your environment

Before beginning a performance tuning project, you need to isolate your program. It's generally not a good idea to do performance tuning directly on a live server, but most development environments need to be carefully configured in order to provide useful performance tuning work. This may actually be the toughest step in your work: the programs that get ignored (and gradually accumulate performance problems) tend to be the ones with lots of hidden dependencies. So, let's take a few first steps:

With this in place, you have everything you need to start breaking down the problem.

Monitoring is people

Next, you need to know how the task is working in order to know what to improve. You need some statistics; meaningful ones, something better than CPU/Memory/Disk utilization. Here are some useful statistics you might want to gather:

Monitoring can go great with a command line task. I like to build long running server tasks as command line executables which record their progress in the database when they launch and when they finish. I can then use a clever performance monitoring package to check how long the task took to run. I take the task's console output logs and write them to disk and cycle them out after a reasonable period of time – say 90 days.

Fixing performance

So now that you've got a working environment, try drawing up a flowchart of the steps the application goes through. Break it down to meaningful levels and explain it to the guy or girl sitting next to you, even if that person doesn't know anything about the program (in fact, if you have to explain it to somebody new, it's often better – the challenge forces you to clarify your thoughts). With this flowchart in hand, let's start trying to figure out what kind of a performance problem we have.

What happens if you can't map/reduce your data? Don't worry: even if you're not writing for a Hadoop environment you can still split up the work. Consider writing an inherently parallel version of your program that runs multiple items at once. What prevents you from running two threads at once? Ten? Twenty? What prevents you from having dozens of servers doing this work at the same time? The answer is likely to be well understood by anyone who does parallel work: synchronization and bottlenecks. You synchronize your tasks by making sure each server doesn't interfere with the next one, and you identify the bottlenecks by noticing where the pain occurs. Let's start with some simple pseudocode that could be used as a wrapper around your project:

function MyGiganticLoop()
{
    // Retrieve all the cached information I need in order to work
    Setup();
    while (true) {

        // Contact a centralized server to obtain one work item
        var WorkItem = RequestWorkFrom(DISPATCHER_HOST_NAME);
        if (WorkItem != null) {

            // This is my "high intensity" function that takes a lot of
            processing time
            var WorkResults = DoMyTask(WorkItem);

            // Update the work item with my results
            MarkWorkItemComplete(WorkItem, WorkResults, DISPATCHER_HOST_NAME);
        } else {

            // The queue must be empty - I'll sleep for a resonable time and
            check again
            Sleep(1000);
        }
    }
}

This code is useful because it uses a centralized dispatcher server to ensure that each task client continues to work as fast as possible. Ideally, the centralized server keeps a list of all the work items that need to be executed. Each time a client contacts it, the dispatcher marks that item as "in progress" and hands it to a client. If the client crashes or fails to complete a work item in time, the dispatcher can return it into the queue and allow another client to try. More importantly, in this pattern, you have the option to scale pretty much linearly (provided your central dispatcher isn't a performance hog itself). You can simply monitor the work queue and spin up new instances when it falls behind, and shut them down when you catch up. But what if this doesn't resolve the issue? You may find, when you get down to this level, that the function DoMyTask(WorkItem) isn't actually where all the time is being spent. Perhaps there's a database call that is the culprit. But now you have isolated your environment enough to be sure you're on the right track. Oh, and when you're done – finish up your work with a code review of the change. Walk through every line of code you had to modify and explain it. It will take a long time, but it will be worthwhile. Happy tuning! [This piece was reprinted from #AltDevBlogADay, a shared blog initiative started by @mike_acton devoted to giving game developers of all disciplines a place to motivate each other to write regularly about their personal game development passions.]