At Station Casinos, IT meets stringent requirements and tight development schedules by using an application problem resolution system throughout the application life cycle — from development and quality assurance to staged rollout and production support.
IT used to be about supporting business operations. Today, IT drives operations. In many industries, the applications that IT develops are revenue generators and the engines of growth for the company. That’s the case at Station Casinos. Complex applications enable almost every business transaction — from reservations, hotel operations and food service to enabling various gaming services, such as slot machines and sports wagering. So developing and testing quality applications while moving them into production in the shortest possible timeframe is key to maximizing customer satisfaction and enabling revenue growth.
At Station Casinos, we wanted to augment our existing processes and point tools to ensure that new applications are delivered on time, on budget, and meet or exceed functional and transactional performance expectations. We also wanted to effectively coordinate incident and problem management among the service desk, IT operations and the development team, so we could minimize the need to involve developers in solving problems in production systems.
In the gaming industry, quality is both a business and a legal requirement. Like other companies, we must comply with Securities and Exchange Commission (SEC) regulations and legislation such as the Sarbanes-Oxley Act. Because we build wagering systems, however, we also fall under the watchful eye of the Nevada Gaming Control Board (NGCB), which must thoroughly review and approve all systems that affect gaming or gaming devices before they are allowed to go into full production.
An additional complication in the gaming industry is that applications and the computing environment are extremely complex. Sports wagering involves complicated sets of rules based on many years of betting traditions. It’s no easy task to reduce them to mathematics, algorithms and business logic rules. Wagering applications must manage many arcane factors to gain player interest, make the gaming fair and make it a paying proposition for the house. They also have to address the NGCB’s requirements for tracking a transaction — from taking an individual wager all the way to payout. In addition, we have 15 different sites, so we support a combination of centralized and distributed systems that all must work as designed and at a specific level of performance at each location.
We’ve found a way to meet these detailed requirements and still meet the business’s goals for launching critical applications in a timely manner. How? We use an application problem resolution system throughout the application lifecycle, from development and quality assurance (QA) to staged rollout and production support. But the real payoff is that we’ve accomplished this without increasing headcount, and we’ve even been able to accelerate already aggressive release schedules to meet externally imposed deadlines.
Getting Into the Action On Time
The NGCB imposes strict quality requirements on gaming applications. With enough time and enough people, we can meet the NGCB’s stringent requirements, but people and time are often in short supply. In Las Vegas, where Station Casinos is based, there is a serious shortage of skilled technical people. In terms of time, outside factors often drive development schedules — the Christmas holiday season in the retail industry, summer vacations in the hospitality industry and the beginning of the academic year for educational institutions. For Station Casinos, the beginning of the 2005–2006 NFL season was the stake in the ground for completing a new Microsoft .NET Framework-based wagering system called Wagering Information Network (WIN).
From a business standpoint, we needed to get the WIN system deployed by Super Bowl 2006. To meet that goal, we needed to complete our QA process and turn the system over to the NGCB for a mandatory field trial that would begin with the NFL season. Two months into QA testing, we could see that we had a serious problem. Based on past experience, we realized that we would need six to eight months to complete the testing. We knew there was no chance the NFL would postpone the start of its season, so we had to have our application tested and ready for field trial in less than three months.
To tackle this challenge, we could either increase headcount substantially, which would increase our costs and possibly complicate the development process, or adopt advanced technology to accelerate problem diagnosis and resolution. We decided to go the technology route and implemented a third-party solution that helps automate problem resolution throughout the various stages of the application life cycle.
When you’re trying to find and fix an application problem, especially in a complex computing environment, you typically spend 80 percent of your time attempting to identify the root cause and 20 percent of your time fixing it. We estimate that by using Appsight, we can resolve less-complex issues 25 percent faster and more-complex issues at least 75 percent faster. The result? We were able to submit the system to the NGCB by the start of the football season and it was operational by Super Bowl 2006.
Accelerating QA and Testing
Our initial impetus to look for a third-party solution was a critical deadline for getting our system through the QA process so we could deliver it on time to meet the needs of the business — and the solution proved to be invaluable in that role. We are also finding that application problem resolution technology is beneficial throughout the application life cycle, by both accelerating releases and reducing maintenance costs.
Getting such complex applications through QA is difficult at best. In addition to the complicated sets of rules that comprise mathematics, algorithms and business logic to create the application itself, the system must support sports booking services at 15 locations, and must also support phone and Internet wagering. However, by automating problem resolution, we completed the QA process on our WIN system in about three months instead of the six to eight months we originally estimated. We saw the benefits almost immediately. For example, our engineers had struggled for two weeks to solve a problem with the system’s display graphics. Using application problem resolution technology, they were able to track down the problem and resolve it in 45 minutes. According to one of our engineers, they may never have found the problem otherwise.
An automated problem resolution system gives the development and QA teams confidence that they can deliver a system with the robustness, reporting and security that a high-stakes application demands, while meeting an almost impossible deadline. In our case, it also gives the NGCB greater confidence in our WIN application because we can demonstrate that we have a product capable of finding the root cause of problems that conventional methods might miss.
Staged Production Rollout
Application problem resolution technology can also facilitate system improvements during application beta testing and rollout. For the WIN system, we started with a single sports booking site during the NGCB trial, which was, in essence, a beta test. In one instance, WIN was attracting so many pre-kickoff bets that lines were stretching 50 people deep. Bettors were still in line when the game started and all betting had to stop — meaning we were losing potential revenue. We used the application problem resolution system and found that the application ticketing process was executing unnecessary database calls that bogged down the system. Eliminating them cut the ticketing time from six seconds to two, tripling the system’s revenue intake capacity!
Once we realized success with the first site, we expanded it to five. The application problem resolution system was highly beneficial in uncovering problems that occurred when we went to this multi-site environment. For example, we found that certain types of sports bets were taking longer at one of the sites. The system allowed us to drill down into the problem and identify the root cause. Finding and fixing problems such as this one during beta testing will ensure that the application provides superior performance when we put it into full production at all 15 of our sites. Companies that use application problem resolution systems can optimize the processes of developing, testing and maintaining code, and can optimize resource allocation in the application development organization.
Enabling Collaboration
Once your application is in wide-scale production, you can use your application problem resolution system to speed the resolution of field problems and better utilize development resources. For a new application, the initial focus is on getting a solid product out the door. However, development resources are also called on to help support applications in production. When development resources are scarce, it is essential to minimize the time these developers are pulled away from actual development work to resolve production problems.
Automated end-to-end solutions enable our help desk, IT operations and application development teams to collaborate more easily on problems because they have an analysis view of logs with the level of detail that is appropriate to their jobs. The help desk technician, for example, might see only user actions, while a second-tier support person could drill down into application configuration and performance, and a developer could play back recordings at the code level to view function calls, argument values and exceptions. Because the first and second levels of support have more detailed information available, the third level of support — the developers — are called less frequently and, when called, are able to resolve issues more quickly.
The gaming industry tends to have more custom-developed applications than other industries. Fewer problems and faster problem resolution in production means our developers can spend more time working on our next revenue-generating application and less time supporting existing applications. Providing detailed performance information across all systems and applications involved in executing a transaction minimizes the time developers must spend to support applications in production.
A Winning Proposition
WIN and other applications are critical systems that generate revenue and allow Station Casinos to grow its overall business. As is the case in most industries, tracking down problems in the software and eliminating them before putting systems into production is absolutely essential. So is meeting deadlines to launch a system that supports critical business objectives. In our case, the application problem resolution system helped us create a better-performing application and saved us at least 30 percent in QA time and 15 to 20 percent in our overall time to market. And because we’ve also streamlined problem resolution as soon as systems are in production, we have minimized the post-rollout workload of key development resources. And that’s a winning proposition in anyone’s book.
Five Benefits of Application Problem Resolution
• Accelerated QA process for new applications
• Faster resolution of issues
• Reduced maintenance costs
• Collaboration across IT functions
• Better application experience for the customer
Marshall Andrew is VP of IT and CIO for Station Casinos. He has more than 25 years of experience in information technology, having served as director of MIS for INGRES; VP of Engineering at Hotel Information Systems; and VP of IT for Esprit de Corp/Europe. He serves on advisory boards for the University of Nevada, Las Vegas School of Computer Science and School of MIS.
Comments
Post new comment