MGBI accelerates business data delivery with AWS

Opportunity Driving business with reliable data

MGBI has been transforming data into valuable and reliable business intelligence since 2010. As a technology company, they specialize in processing and analyzing information to empower better decision-making.

MGBI’s data engineering-powered solutions streamline marketing, sales, and debt collection processes, offering tools like comprehensive company and executive databases, bankruptcy lists, financial statement repositories, and custom web scraping solutions, automating data extraction and processing from websites and public databases.

With over 10,000 clients served, MGBI is a trusted partner to Poland’s largest enterprises — particularly in banking, debt collection, and financial services.

In the financial industry, reliability and trust are non-negotiable. MGBI places these principles at the heart of its operations without compromising on innovation — pushing boundaries to provide clients with reliable and always up-to-date data that serves to minimize the risk of high-impact corporate decisions.

In order to be able to continuously provide ever-increasing, high-quality data to its customers, MGBI chose to modernize and migrate its services to AWS. This cloud migration was primarily motivated by an expected increase in development and data processing capability and reliability, alongside time to value reductions. MGBI turned to the trusted expertise of AWS’ partner Chaos Gears to deliver on those expectations.

Solution A future-proof data engineering platform

While most cloud migration projects are conceptually similar, similar does not mean equal — proper methodologies still need to be followed to assert we correctly deal with all potential differences and edge cases.

Some organizations employ their own custom approaches at the core of that process. The argument is typically that a fully custom methodology can be tailored to the exact requirements.

While the benefits of this route may appear tempting at first, a second look at such fully in-house solutions begs one question: where can one find sufficient empirical long-term evidence that it is in fact a good approach? After all, an organization may have successfully built several houses, and they may look nice from the outside — but just because they have not yet collapsed and one can not see the crumbling insides, does not mean they are architected and built well.

When embarking on this journey, it is therefore much more prudent to rely on the collective experience of the industry. Or, to put it simply: to learn from others, instead of repeating their mistakes. Once stringent compliance guidelines are concerned — such as the European Financial Services Addendum in this case — it is not merely prudent to do so: it becomes crucial.

Typically, AWS is discussed for the technical aspects the cloud brings to the table. Yet, a migration to the cloud, in order to be successful, is not purely a matter of technical implementation — it is strongly dependent on an organizational shift. To help facilitate this shift, AWS provides a sea of collectively-shaped knowledge, tools and frameworks that let businesses of any scale create and enact informed strategies.

However, even the best decision-making framework is ultimately contingent on the knowledge driving it. It stands to reason that correct, factual and timely information must therefore be at the core of any decision — if we want it to be the right one, that is.

Given the entire business of MGBI is built on a perfect understanding of those crucial points, our cooperation was off to a well-informed start.

Data-driven decision-making

Proceeding in accordance with these best practices, we worked closely together to prepare a Total Cost of Ownership (TCO) analysis with estimates intended to drive the next steps of the process.

At this early stage, we assessed and evaluated our partner’s environment to determine its cloud readiness, along with discovering and defining the functional workloads happening therein. This data collection then served to outline the future scope of work, and this in turn formed the basis of multi-year cost projections included in our analysis.

We provided those numbers, including an Annual Recurring Revenue (ARR) calculation, to MGBI in a detailed business case — and received a green light to proceed with the actual implementation.

During pilots and PoCs, we showcased various cloud-native data engineering tools like Amazon Managed Workflows for Apache Airflow (MWAA), AWS Glue, and Apache Iceberg. This hands-on approach ensured the customer’s team fully understood the capabilities of the target migrated solution.

It is worth mentioning that while the migration process we follow is a gold standard, it is such because it has been proven globally without prescribing specific solutions. In-house approaches are often biased towards the individual experience of the organization implementing them, instead of being fully dedicated to the customer’s needs. In comparison, the AWS’ best practice guidelines and patterns that Chaos Gears relies on focus exclusively on gathering the best possible information to arrive at the best possible solutions.

Naturally, getting to the green light involved not just numbers, theoretical analyses and raw data — from the very beginning communication and collaboration between Chaos Gears’ and MGBI’s teams had been excellent. Our hallmark open communication helped both parties stay agile in the process and keep all stakeholders aligned on the same goals.

With such a foundation, the technical side of things could then flow smoothly.

Different solutions for different challenges

We proceeded to create a properly configured landing zone for MGBI’s numerous accounts, each of which is monitored through AWS CloudTrail with dedicated AWS Config rules aggregating all pertinent activity logs in separate Amazon S3 buckets.

Our priorities also aligned with regards to security, given MGBI’s emphasis on data security, compliance and strict privacy concerns.

To meet security best practices without sacrificing ergonomics for our client’s teams, we based user authentication on AWS IAM Identity Center synchronized with MGBI’s Google Workspace as a unified and robust single sign-on (SSO) solution, with Cloudflare’s private networking solutions providing access to select cloud services from within MGBI’s premises. This allowed us to enforce the principle of least privilege across the infrastructure without compromising on anything. A move of the client’s sensitive configuration details to AWS Secrets Manager centralized and secured credential handling.

With the foundational work in place and the infrastructure defined as code (IaC), we could now shift our focus from organizational requirements to application requirements.

Before we moved to concrete services, both teams worked closely together to overhaul MGBI's existing CI/CD process. The resulting multistage pipeline is tailored to data processing workloads and streamlines the development flow, enabling a smooth transition to cloud-based service deployment.

Amazon ECS formed the basis of our solution, allowing a two-pronged approach to MGBI’s specific workloads. It allowed us to automate the deployment and management of persistent, processing-heavy services running in Docker containers (EC2) alongside periodic serverless tasks — triggered primarily through Amazon Managed Workflows for Apache Airflow (MWAA) running its tasks via AWS Fargate.

This serverless approach leverages parallelization to increase the throughput of data delivery to the customer's data warehouse and, subsequently, to the customer's contractors. The result was an AWS-native data processing pipeline that seamlessly meets the customer’s requirements for scalability, cost efficiency, and processing speed.

Outcome Reliable data always on time

MGBI’s core services are now built and run on a data platform that is easier to manage and simultaneously more reliable. With better observability and connectivity patterns implemented as a foundational part of the architecture, the company can be bolder with its innovations with fewer risks overall.

The flexible nature of the scheduling architecture we opted for — handling data pipelines separately from containerized services — serves to provide a versatile and easily extendable platform for future expansion.

The ability to quickly create and process ETL jobs — including non-recurring, high-priority datasets — is crucial to any organization primarily dealing in big data, and the solution Chaos Gears delivered is a proven best-in-class approach. It is capable of handling all of the company’s current data engineering needs, and remains easily scalable in both directions, staying in sync with MGBI’s requirements — present and future.

While Chaos Gears had primarily been tasked to migrate a specific set of MGBI’s services and applications to AWS, cooperation between both teams has been close from the beginning and continues beyond the initial scope.

Best practice architectural and organizational patterns — including improved and extensively automated CI/CD pipelines — serve to facilitate a smoother development workflow that works well both internally and externally — such as with Chaos Gears providing data engineering services in this case. Multiple teams can work concurrently, with just the right amount of access and no unnecessary friction.

As MGBI keeps steadily growing thanks to its continuously reliable delivery of the right data at the right time, we are happy to enjoy MGBI’s ongoing trust in our expertise and ability to deliver on promises, and are looking forward to further success stories built together in the future.