A technology company producing smart payment cards in Hong Kong experienced significant challenges whilst processing millions of daily transactions. Every day, these transactions generated a value of more than $130 million HKD. With its previous solution, the organization had been able to process its transactional data within two days. The company set out a challenge for IT companies to produce a model that would be able to process the data within one day. This was a challenge that Big Data Architect Limited (BDA) was well prepared and qualified for. Competing against two other vendors, BDA began working on developing its model.
Search for the Solution
The original specification was to process the transactional data within one day. In order to meet this deadline, BDA decided to base its models on 3 very powerful big data tools - IBM Datastage, Cognos and DB2. After testing and fine-tuning its solution, BDA successfully completed the transactional data processing period to an astonishing 8 hours. Overwhelmed, the company chose BDA to win the contract.
DB2 is a database that allows large organizations to deliver powerful solutions without undermining cost, performance or reliability. With IBM Datastage, the organization can have a more efficient ETL (Extract, Transform, and Load) phase. This is made possible due to the powerful Graphical User Interface (GUI) system it has. Users can easily understand the workflow of information in nearly real time – making maintenance an extremely straightforward task. Once the ETL stage had been completed with Datastage, IBM Cognos was used so that information could be visualized –enabling the decision makers of the company to easily understand the data.
Challenges Implementing the Solution
The legacy solution that this organization used was based on an annual leasing model that costs a great deal only to use it. In addition there were also the maintenance and operating costs. BDA made the recommendation to use Datastage as this would reduce operating costs – decreasing the company’s TCO. In order to advance the solution, BDA’s technical team first migrated the data from the legacy system and input it into the newly implemented Datastage system.
The organization that BDA was working with was split into two legal entities. Originally, each of these legal entities shared a database – this was problematic for the organization as it had the risk to mix private client information. BDA’s solution was to design two separate databases so that the legal entities information would no longer be located within a single area. This particular job poses a huge challenge for BDA. However, the task was completed without any major problems. Furthermore, BDA also went beyond the original specification and installed a fault detection mechanism that enhanced the security of the system when it helped separate the data.
The 4.4 billion transactions that passed through the system each month was another additional challenge to BDA. The organization had the requirements to store the data online for 36 months so analytics could be applied.
Due to the large volume of data, the system failed to allow more than one user to access the database efficiently. The organization asked BDA to redesign the database so that more than one user could access the system at any time, which further added to the complexity of the solution. To resolve this problem, BDA separated the database into 36 segments (1 month per segment). By breaking up the database into smaller sections, performance was improved dramatically. This is one of the many reasons why BDA made it possible to reduce the processing time of the transactional data to eight hours.
Results and Benefits
The company’s expectations on BDA were fulfilled beyond as it was now possible to process transactional data within 1 day. The company was also capable of managing its reports in a more systematic way. The Hong Kong Government worked closely with the organization to allow eligible citizens to receive travel discounts using the card payment system. With the implementation of BDA’s solution, reporting to the government in a structured method (within the right time periods) became an easier task to complete.
BDA made the recommendation that the company should move away from its legacy system and adopt the IBM Datastage solution. The technical team found this task extremely challenging. However, after the completion of transferring the data from one system to the next, the organization saved significant amounts of money and predicted that it would gain much more profits within the next 3 years.
The new system introduced by BDA made the reviewing of 36-months of transactional data less complex. This new ability now allows the company to analyze mass transactions through a far more sophisticated and logical process. With this additional data, the company has become more competitive. The insights given by analytics offer the company a superior advantage to continuously grow and generate further revenue.