The Hong Kong branch of an internationally renowned Credit Bureau sought answers from Big Data Architect (BDA) to solve problems related to its analytical tools. The reputable company serves approximately 5 million customers in Hong Kong and owns a database of 70 million credit reports. Consequently, a priority is always given to the efficiency of its IT infrastructure.
Challenges
The Credit Bureau faced multiple challenges which BDA had to address. The biggest challenge was the use of Java as a programming language at ETL (Extract, Transform, and Load) phase. ETL concerns pulling out data from the specific areas of the source and then placing it inside the data warehouse. For this organization, the ETL stage was necessary in order to handle client credit information. However, the process became extremely difficult when the company’s technical staff experienced difficulties navigating through the lines of code using Java.
Furthermore, SAS was being used as a programming language for the analytics of individual credit reporting. Like Java, SAS was extremely hard to navigate and also required users to have a unique skill set of mathematics and IT Knowledge. It was a constant struggle to find qualified staff to oversee and manage this area.
Although Java and SAS were both capable tools, both sets of software’s presented challenges in relation to their complex coding. Adding to the complications, every programmer has a different approach to writing code – making it hard for other users to navigate and comprehend the logic.
Solutions
Before the use of SAS and Java, the Credit Bureau had purchased the IBM DataStage application. DataStage is an ETL tool that allows users to easily understand the workflow of information through a GUI (Graphical User Interface). This allows staff to have a better visual understanding of what is occurring compared to a text-based tool like Java and SAS– where users will have to go through every individual line of code in order to understand the whole picture. Despite DataStage already purchased, the company did not have the available training to teach their technical staffs how to operate the system. When the organization decided that change was needed in order to improve their IT efficiency, they approached BDA to look for the appropriate alternative to replace Java and SAS with DataStage.
Implementing the Solution
To successfully implement a solution, it was important for BDA to equip the company with the necessary skills to become self-sufficient. Thus BDA provided professional services to ensure the migration to run smoothly. The majority of these services included training; besides, consulting and supports were also available.
Before BDA commenced its training program, it was important that they fully understood how the company operated and which features of DataStage were the most demanding. With this information, BDA would be able to create a training program that is designed specifically for the organization. It was decided that the best way to gather this information would be to work with the company to create and implement two new products with the use of DataStage.
Results
Due to DataStage being a GUI system, BDA found it extremely easy to transfer their knowledge to the company. With the information discovered through the implementation of the two new products, training did not encounter any problems and only took one week to complete.
Most companies will find that migrating away from a legacy system does not happen overnight – this Credit Bureau is no exception. SAS and Java are still being used occasionally, however, there is a greater emphasis on DataStage.
The company found that the benefits of the newly implemented DataStage were astonishing. The system gained greater efficiency and needed less maintenance because it is more visually orientated for users. As a result, reduced manpower is required to operate the DataStage, which saves costs with technical staff and training. Overall, the company has found vast advances in efficiency; in other words, the time to complete projects has been significantly reduced.