This case study is courtesy of an IT Architect and Senior DBA working for a large international call and service center. He has managed four separate Database Managed Service Providers (DBMSPs) as a part of keeping mission-critical database systems up and running. This study is based upon his experiences with each.
His company had more than 500 TB of data being used in a mission-critical workflow application, with tight SLAs. The application manages electronics repair from end-to-end — including returns, supply parts, repair/refurbishment, testing, shipping, and environmental disposal.
This study is to help others in considering a DBMSP using the lessons he learned and factors that ultimately became the most important to finding a DBMSP who is a true partner. He determined which criteria are key in choosing the best fit and how best to choose the right database managed service provider (DBMSP) through a process.
As the single database expert on staff, his role was intended to be strategic while interfacing with the managed service provider for all support tasks. Managing the vendor relationship included overseeing deliverables, managing scope and a process for ad-hoc work, and change management. We will look at relationships with four different DBMSPs and how they helped or hindered him in fulfilling that role — plus their ability to promote success for the company.
Case Study Criteria
When comparing one database managed service provider to another, it was crucial to define the list of criteria we use for comparison. We chose the following:
1. Technical proficiency – Do they have the knowledge, breadth, and talent to resolve any issues that arise?
2. Support/SLA – Are they able to meet the SLAs?
3. Proactive stance – Does this vendor check for potential issues such as space or warning messages and resolve them prior to an occurrence, or do they work in a reactionary mode?
4. Security – Do they recommend security improvements? Do you have confidence that their methods of connection will be secure?
5. Relationship – Do they have assigned technicians familiar with your architecture, application, and needs so you can develop an ongoing relationship? Do you feel
confident that the vendor has your best interest at heart? Is there always someone you can call if you have an issue?
6. Language proficiency – Is their proficiency in your primary language strong enough to get through complex issues efficiently?
7. Responsive – Are they responsive to planned and unplanned needs? Do they quickly work with you to solve problems with a positive attitude?
8. Cost – What is the annual cost for 24/7 support?
#1 Original DBMSP Provider
The first provider is the managed services division of a multi-billion-dollar global IT company. The first hardware used for this project was DEC Alpha equipment, which the provider acquired. They migrated from there to Itanium under the same support contract.
There were issues with errors and outages with this provider. For example, there were failed upgrades, which caused longer outages than the SLAs specified. The DBMSP retried one character set upgrade, and it failed monthly for six consecutive months. The DBA resolved the issue by performing the upgrade alone over a weekend.
The company experienced extreme unresponsiveness to issues. Due to the outages, the Service Level Agreements (SLAs) were not fulfilled.
There was little or no proactive problem resolution. The support functioned in a purely reactionary mode.
These servers were not on a public system, so external security access was not a consideration. The security of the vendor was adequate.
Due to the technical problems and the missing proficiency in resolving them, the relationships with the managed service provider were rocky. Because of the size of the company, there were a variety of technicians who responded to issues, which were not conducive to developing a knowledge base of the client site nor a depth in the professional working relationship.
While the DBSMP was responsive, their response to the issues was not always beneficial. This led to frustration and a sense of feeling alone when things weren’t going smoothly.
The annual cost for this system will be considered the baseline throughout the remainder of this comparative study. As such, the annual cost for this solution is equal to or greater than the rest.
Due to errors and outages, utilizing this provider became impracticable. When the hardware was aging out and required a size and speed upgrade, the decision was made to move to Sun SPARC SuperClusters with Oracle Exadata. Since the DBSMP had not met the SLAs proficiently with their own equipment, it seemed implausible for them to manage different hardware. An alternate solution with an equal cost was researched and found.
#2 Second DBMSP Provider
This provider is the advanced managed services division of another multi-billion dollar global IT company. The vendor offered fully outsourced DBMSP services. They were in charge of the runtime environment, including two Sparc Solaris SuperClusters with Exadata nodes, backup devices, gateways to get into data centers, and tape drives. The SuperClusters were full-size Exadata engineered systems for the database nodes. They were serviced by a 100% onshore support team.
Due to the DBMSP internal management cost-cutting changes, an additional layer of support was added, changing the delivery model. Work was coordinated onshore and then performed offshore. Using this 3-tier model, every category considered in this study was affected.
The DBMSP was capable and had excellent, proficient staff to perform any technical tasks required on this platform. In the beginning, the expertise was available, willing, and able to help. Once support was transitioned to the 3-tier support model, it became difficult to gain access to the necessary resources. Communications and hand-offs between onshore and offshore teams had disconnects — leading to errors and downtime. Time zone and language issues became the norm. Any issues, regardless of cause, were subject to time and materials costs, which led to budget overruns.
Originally, the SLA was upheld with excellent response times. Once the offshore team became involved, no effective response was found within the SLA times. Once the SLA times were exceeded, the onshore team would connect and fix the issues. It became faster and less painful for the DBA to fix the problems himself rather than to call support and wait for the SLA to expire.
Ultimately, they weren’t even reactive. It wasn’t uncommon to receive no response at all.
These servers were not on a public system, so external security access was not a consideration. The security of the vendor was adequate.
In the beginning, the relationships were excellent as the service, competence, and responsiveness were excellent. As the service, competence, and responsiveness failed, the relationships became strained. Once the DBA abandoned hope and performed all tasks himself, there was no relationship.
Originally, they were responsive. But, as costs were cut, responsiveness became the most severe issue. Although qualified resources were there, they did not have access to them.
Originally the cost was identical to the previous solution. Once the service was downgraded, a 30% cost reduction was granted.
Not only were the service level agreements for downtime not being met, but additionally, the work had all shifted back to the client. The company was paying for a service and receiving nothing. Intense research and studies were performed to find a database managed service that would live up to their promises.
#3 Third DBMSP Provider: Guardian Eagle
Guardian Eagle provides a DBMSP solution called OnTrackTM. In order to accommodate all of the SuperCluster needs, they partnered with a company that specializes in that area. Guardian Eagle is a right-sized company — small enough to know the system and client needs and large enough to accommodate a diverse skill set. Their 24/7 service is responsive and reliable.
Every issue that arose was resolved in a timely fashion. The expertise was always appropriate and available. All of the work of managing the databases were offloaded to Guardian Eagle — and it did not return. This allowed the DBA to become a strategic DBA again. He was able to approve new applications and designs, manage architecture changes, and perform capacity planning. His role became one of strategic leadership again.
The SLAs were met 100% of the time. Guardian Eagle always delivered on their promise.
Due to the monitoring system installed, early warnings and alerts were captured before they occurred. This allowed them to be resolved seamlessly.
Guardian Eagle is a database security company offering expertise in all manner of database security. Their staff contains relevant certifications, and they offer a few proprietary database security solutions as an add-on to their primary managed services offering, OnTrack.
Since all work was performed well and in a timely fashion, everyone was delighted. The staff was polite and respectful, which ensured working relationships were pleasant and professional.
The responsiveness of this DBSNMP was excellent. They were truly what a partner should be and were always willing and able to help.
This managed service provider delivered an additional 30% reduction in cost. This yielded a cost of (Baseline – 30%) – 30%.
The hardware had reached an age where it needed to be replaced. The business had shifted focus — reducing the size and workload, so fewer CPUs and licenses were required. In order to reduce license fees, the option to move to the Oracle cloud was chosen, allowing the fees to be shifted from licensing to hardware. This created an overall cost reduction for the company. As there was an initiative to decommission the hardware, this solution was a good fit.
#4 Fourth DBMSP Provider: Oracle Cloud with Guardian Eagle
The fourth and current solution is the Oracle Cloud, with Guardian Eagle managing the databases and cloud interface. The Oracle Cloud team is responsible for up/down, patching, and upgrades. Guardian Eagle retains responsibility for managing the database, all troubleshooting, and the interface with Oracle Cloud. Guardian Eagle efficiently handled all configuration and management of the move to the cloud for all layers — including networking, routing, application servers, and databases.
Oracle has the capability to support and maintain its own databases. Especially given the cutting-edge nature of the cloud, their support has been proficient and competent. Guardian Eagle handled the complete transition to the Oracle Cloud without going over budget and is well suited to managing the maintenance and support.
It is well-established that Guardian Eagle fulfills their promises with support and Service Level Agreements. It is also advantageous to offload the handling of the interface with Oracle Cloud to Eagle as well. That offloads all issues with Oracle Cloud adherence to SLAs. Any issues can be monitored, escalated, and advanced by Guardian Eagle, which eliminates any frustration from the company. With Guardian Eagle’s past experience handling Oracle Cloud, they also optimized the cloud experience.
Guardian Eagle has a number of tools available to ensure proactive monitoring — allowing them to capture potential issues and errors. Many issues can be resolved prior to affecting the client. Guardian Eagle has their own monitoring tool known as EagleEye™ and also employs Oracle Cloud Management Services, Oracle Enterprise Manager, and other tools.
Oracle manages all connectivity into the cloud with improved security measures. Connectivity is achieved without accessing any public IPs. Certificates are required, and everything is 100% encrypted. In addition to the security of the previous system, this solution added inherent security items from entering the Oracle cloud.
It is advantageous having Guardian Eagle handle the interface with Oracle. Other companies have reported that the Oracle sales side may not match up with customer needs. Others have run into issues such as the implemented design not meeting the licensing requirements and subsequently adding to the license fees. With a solid contract using fixed fees plus Guardian Eagle handling the configuration and transition to the Oracle Cloud, all of these aggravations were completely avoided or offloaded to the DBMSP. The working relationship with Guardian Eagle has been that of a partner and is highly valued throughout every department of the company.
The service delivery professionals at Guardian Eagle exceeded extremely high standards for support and responsiveness. Whenever their expertise is needed, someone is available to answer the call.
Business had decreased, so the number of databases, storage, and servers were also decreased. This allowed a further drop in cost by 30%. By this step, the cost has been reduced three times by 30%.
Lessons Learned During the Case Study
With careful analysis, we have learned some lessons in choosing the best-fit Database Managed Services Provider. The most high level include:
1. Larger companies have difficulty in thinking like a true partner unless the customer is of equal size. They may mean well, but significant decisions that affect you are being made without consideration to the partnership. Choosing a smaller company with an excellent reputation can yield a true partnership, with much better results than a large multinational company.
2. Document your objectives in bringing on a managed service provider and make sure they can meet them. Then, your comparative evaluation will be meaningful to you.
3. Never get into a contract with Oracle where they have time and materials cost. Make sure all contracts with Oracle outline fixed costs.
4. Double-check that Oracle licensing is appropriate for your design — otherwise you will be facing expensive additional licensing.
5. Choose a managed service provider that truly can handle all the database support tasks you need, so you allow the in-house team time and energy for strategic thinking, personal development, and true leadership.
6. See if you can engage with the team you think you want to go with before signing a long-term contract. Oftentimes, you can get them to do a one-off project so that you see them in action. Make sure it is the same people that you will be working alongside in the database Managed Services engagement.
7. Have a plan for dealing with missed SLAs. It is best if the consequences are included in the contract.
8. When choosing a DBMSP, remember you are choosing a partner, not a vendor.