Author: Mark Hillary

I mentioned that Big Data has progressed far beyond just being a business buzzword. There are entire industries being shaken to their core because a leading player finds a way to analyse their customers better than their rivals. Far from being a management trend, this is a strategy that will fundamentally change many industries.

But have you ever stopped to appreciate just how much data we are creating today?

Some excellent analysis in Business Insider recently explored this question. The problem is that people and companies are just creating so much data – it is increasing at an exponential rate. At the present rate we are doubling the amount of digital data that exists every two years.

But even though we are creating and storing these enormous amounts of data, only 0.5% of it is being analysed. There is so much data out there that companies, governments, and individuals feel swamped, unable to gain insights from it.

The Business Insider article features a comment that cuts to the heart of the Big Data issue: “You have to start with a question and not with the data,” says Andreas Weigend, former Chief Scientist of Amazon, now director of the Social Data Lab and lecturer at UC Berkeley.
Businesses need to start thinking about the insights they could get from their customers, to ask more ‘what-if’ questions. There are solutions out there in the data, but it is impossible to analyse every byte of data.

A typical analogy for the average person might be the difference between email and Twitter. You check every email, even if it is only long enough to decide that it should be deleted. However, you only check Twitter messages that are arriving as you are looking at the news stream, or you use intelligent filters and analysis to ensure that interesting messages are made visible.

Businesses need to start thinking of their Big Data strategy in the same way. How can insights be drawn out from the data they already have?

The Big Data market has transformed how most IT services are bought and sold. Often the customer is a business leader, rather than the CIO, and services are already available, rather than needing to be designed from the ground up.

The Big Data market has transformed how most IT services are bought and sold. Often the customer is a business leader, rather than the CIO, and services are already available, rather than needing to be designed from the ground up.

IT leaders often look at other companies to see what they are doing, it’s usually known as best practice, but what if the entire market is in a state of flux? Some IT companies are now delivering radically different solutions to others. Innovations like the app store concept and cloud based solutions are changing the way that IT services are delivered globally, so how can changes in the Big Data market change how the entire IT industry behaves?

Techrepublic magazine recently outlined some important changes in how IT leaders think about their marketplace. Think of these 5 different changes as examples of what is changing and how it can influence the behaviour of IT leaders:

1.    Off the shelf analytic solutions like Watson from IBM have been customised to be used across many different industry verticals. It’s the same system, but modified for different environments and it can therefore offer immediate results. Everyone is looking for a rapid return on investment today.
2.    Commodity hardware offers a great platform for storage if you are going to keep your data and analytics system internal, rather than in the cloud.
3.    More companies are relying on their suppliers for expertise and innovation today. These are true partnerships now, not just outsourcing arrangements where the cheapest supplier gets the project.
4.    Analytics reporting systems like SAS are still important because they have been tested and developed over so many years, but also because the users are really familiar with the way these systems work. Familiarity is important.
5.    Real time processing systems are changing the way that physical infrastructure is designed and deployed. Look at SAP HANA for an example of how systems using memory in new ways are now feeding back into the design of the physical servers and databases.

What do you think about the lessons from Big Data? Have you seen other ways in which IT is more generally changing because of the way that Big Data is influencing IT strategy?

Challenges

Customer is one of the largest EU-based IT service providers.

Challenge: identify and eliminate vulnerabilities that could be the cause for unauthorized access to employee data before the production release.

Solution highlights

More than 30 vulnerabilities were successfully eliminated from the software to ensure the secure processing of user data.

Key Results

Based on the high value of the project results, Penetration Testing has been successfully integrated into the Customer’s HR Management System development cycle.

Challenges

Customer is a US-based AI writer service provider with significant number of reputation sensitive customers.

Ensure that end customer’s data cannot be compromised with the delivered solution.

Solution highlights

Around 20 vulnerabilities were successfully eliminated from the software before the product was introduced to the market (only 7 out of ~20 had low threat levels).

Key Results

High risk of personal data loss was reduced to a low level.

Challenges

Ensure that a delivered solution does not have security holes that could lead to customer’s sensitive data loss.

Identify vulnerabilities that could bring a high risk of product outage due to hacker attacks.

Solution highlights

Around 28 vulnerabilities were successfully eliminated from the software. 22 out of 28 vulnerabilities had either medium, high or critical threat levels.

Key Results

The customer’s audit was successfully passed after eliminating exploited vulnerabilities.

Challenges

The biggest challenge was using on-premises infrastructure, which brought with it the constant need to support the hardware in the face of increasing data volumes. The bank required new approaches to working with data that the current infrastructure could not support. In light of this, it turned to IBA Group in search of a solution to the several competing challenges it faced.

Background

One of the business units of the customer’s company was previously a stand-alone company, so they used non-standard software to create reports. Employees of this business unit had to use MS Excel to bring their reports to a standard form.

Business Challenge

Automate the generation of standard reports adopted in all business units of the company.

Solution

To automate report generation, we developed SSDM (Security Systems Data Mart) — an integrated data warehouse. This warehouse contains data from their custom systems previously used by this business unit, as well as standard data from the systems of other business units of the company. This data contains information on payments, services and accounting. Automatic reports are generated in Cognos.

The developed system has a complex business logic that is constantly changing and evolving. In the course of the project, there was a migration from one contract management system to another, which has a different data model. Then the system was migrated to interact with clients to another service. These changes were considered in the operation of the data warehouse and the integration was completed successfully.

Result

Most of operations that employees used to do in MS Excel are now performed automatically in SSDM. We configured automatic reporting in Cognos. Also, the developed data warehouse is used by other business units of the company as a data source for reports.

Previously, it took most of the month to generate reports, but with the introduction of the new solution, this time has been reduced to 2-3 days.

Background

А huge global company with a strong technology focus turned to IBA Group with the task of transferring their client’s infrastructure to cloud services. At the time of the beginning of cooperation, the end customer of our client used a lot of data storage for the following internal service tasks:

• production cost tracking;
• dialer’s information;
• providing information to dealers,
• information about clients, employees.

The main part of the infrastructure was based on the Oracle database and the Informatica Power Center ETL tool, as well as many other tools such as Cron, Shell Script, Java, JS.

Several years ago, the company’s management decided to migrate the entire infrastructure to the cloud. During modernization, the company brought production processes to a single standard, increasing its structure, including a unified approach to development, implementing an approved technology stack, and ensuring quality control of development.

Business Challenges

During the modernization process, the end client encountered the following difficulties:

1. The existing services of the company were slow, which impacted all processes — from data analysis to hiring employees.
2. Frequent failures due to errors within the services led to a partial loss of data and the inability to make management decisions, prepare accounting reports, prepare data on the movement of warehouse resources, and more.
3. Permanent delays in providing new services due to the slowdown in development or the impossibility of their implementation, led to a decrease in the company’s attractiveness among customers.
4. Scaling within the framework of the old infrastructure could not be carried out without extra payments, due to the low flexibility of vendor licensing offers.
5. Recruitment of the necessary employees such as full-stack developers presented difficulties due to the huge variety of technologies used; cheaper resources were not of the right quality or up to the high standards of the company.

Key Objective #1

Migrate company services to new infrastructure in the cloud.

Key Objective #2

Tailor AWS vendor migration offerings to company services.

Solution Highlights

The customer formed a separate team that was responsible for the innovation and unification of disparate systems into a single infrastructure. IBA Group provided engineers and architects to form custom teams to participate in the adaptation of infrastructure modernization solutions, developing the top-level structure of the entire ecosystem, as well as general approaches and rules for each system separately.

After giving recommendations for the development and auditing of the process, the IBA Group team directly implemented this migration. To improve the quality of decisions made at the architecture design stage, an AWS vendor was involved.

Due to the business’s specific features, each area had its own architecture, for which separate teams were created that were responsible for its adaption and improvement. They were also involved in the transfer of existing services for their area to cloud technologies — AWS Glue, PySpark, ECS, Kubernetes, EMR + PySpark.

One of the important conditions was a “seamless” transition — teams during the migration process ensured that access to existing services remained uninterrupted.

Project Results

Benefits the client received by moving data management to a cloud service:

1. Increased fault tolerance by fixing bugs and using ready-made cloud-platform services for interaction between different customer services.
2. Continuity of current services during the migration process was maintained.
3. Cost reduction in the form of refusal of some server capacities due to the transition to the cloud service.
4. Flexible planning of financial expenses for scaling due to the pay-as-you-go system provided by the cloud solution.
5. Reduced time for recruiting specialists in the IT department by unifying the requirements for them, defining clear rules, and specifying the technology stack.
6. Reduction of the number of narrowly focused specialists without full-time employment due to their retraining in the technology stack adopted by the company.

Background

Previously, the bank did not use systems to automate the accounting of procurement processes. The documents were prepared in hard copies which were considered/approved at the meetings of the working groups. All documents were stored in folders, employees also made their electronic copies.

Main problems that the bank wanted to address:

• lack of a central repository of all documents related to procurement processes;
• lack of transparency in procurement processes;
• long lead times for the procurement of goods and services;
• lack of reporting on procurement processes and their efficiency;
• lack of budget expenditure reports.

Business Challenge

The Bank submitted a request to automate the accounting of procurement and contracting in the context of its business activities. The new system was supposed to be the last in the trio of applications that were developed at different times. First two applications:

• EDW: Enterprise Data Warehouse and reporting system based on IBM Cognos;
• ABFPS: Automated Budgeting and Financial Planning System.

It was necessary to create a unified accounting system for all documents related to the procurement of goods and services, with their subsequent approval and supervision over the execution of expenses budget. Upon conclusion of the contract, keep records of primary accounting documents and monitor payments. The system was to be built on the basis of the bank’s existing infrastructure based on IBM products (DB2, FileNet, WebSphere, Integration Bus) and integrated with the personnel records system, budgeting system, Enterprise Storage, and the bank’s operating system.

Solution

The IBA Group team has developed a bank business management system based on the terms of reference elaborated jointly with the customer. The system has a client side for working from a browser and a server side for business logic. The approval of documents was organized through a dedicated business process management system. An enterprise service bus was used to integrate with related systems of the bank. All documents created within the system are stored in the document storage powered by FileNet.

Result

A unified system of accounting for all documents in the process of purchasing goods and services has been created. Now all documents are approved in a single system with the ability to access information about the document itself and its approval route. During document approval, the available free budgets for procurement are monitored. Business is able to generate a variety of reports on the procurement processes and execution of expenses budgets. A unified catalog of purchased goods and services has also been created. For all contracts, documents on contractual performance and payments made are available together the capability of automatic payments.

The system has been in commercial operation since January 2019. During this period, about 95,000 documents were created. Now the project is in the support and development stage.

Business Challenge

Reduce preparation time and improve management reporting with IBM Cognos and IBM InfoSphere DataStage platforms.

Solution

We developed and configured an Enterprise Information System (EIS) and an Enterprise Data Warehouse (EDW). The following tasks were completed:

• Development and update of reports required to identify suspicious financial transactions and reports for the Compliance Control Department
• Development of the Market Risk Datamart
• Development of processes for generating data about bank customers from EDW
• Development and configuration of ETL tools for the software package “Unified Register of Customers”
• Development of ETL processes for the archive backup, separation of analytical models (Operational and Archive).

Certified instructors trained the customer’s team to operate the new system and continue to advise the bank’s employees. Also, as part of the project support, IBA Group checks the calculation algorithm in reports, analyzes and eliminates problems in EDW data, makes changes to the EDW model, ETL processes, data marts and reports.

Result

All information from different enterprise systems for analysis and reporting was consolidated in EDW and EIS. Bank employees can generate more detailed financial reports, which assists management in making informed management decisions.

After training, bank employees are able to independently maintain and improve the deployed systems.