LUX MED Group builds a near real-time data analytics platform on AWS

How soon can the business see an event after it happens? For most organizations running batch reporting, the honest answer is: tomorrow. LUX MED Group decided that tomorrow was not fast enough.

LUX MED Group is Poland's leading private healthcare provider and part of the international Bupa Group with more than 30 years of experience, a presence in over 650 cities, 3 million patients under its care and 15,000 corporate clients. Alongside its core healthcare operations, the Group operates a growing health insurance business serving more than one million insured individuals. For this part of the business, we implemented a reliable data platform on AWS, particularly important in insurance, where sensitive personal and financial data, strict regulatory requirements, and high expectations around security and availability demand a robust and resilient data foundation.

The data landscape had grown with the insurance business: more than 30 source databases behind operational systems and products, many of them maintained by external vendors on their own release schedules. Different teams worked with different versions of the same numbers, demand for analytics kept rising, and nightly batch reporting meant analysts always worked on yesterday's data. The Group set the bar deliberately high. Data had to reach the warehouse within single-digit minutes of the event, without adding any load to production systems. Security, GDPR compliance and auditability had to be built in from day one. New sources had to be onboarded in weeks, not months.

The choice of AWS was strategic: pay only for actual usage, scale with business growth, accelerate delivery, and let teams focus on building products instead of running infrastructure. The data platform was the natural next step of a cloud journey the LUX MED Group had already begun.

To turn this ambition into an architecture, LUX MED Group partnered with Chaos Gears, an AWS Premier Tier Services Partner specializing in data platforms and cloud-native architectures.

From the very beginning, this was a joint build: LUX MED Group insurance-arm engineering teams brought deep knowledge of their systems and the regulatory landscape; our data engineers brought the streaming and warehouse expertise.

That collaboration continues today as new systems and use cases come on board.

Solution Every change captured, nothing disturbed

A traditional batch ETL (extract, transform, load) was never going to hit a ten-minute target. A batch job runs on a fixed schedule, so the delay is built in, and the sources here are live databases that cannot take extra query load. We designed the pipeline the other way around: ELT (extract, load, transform), with CDC (change data capture) reading straight from the database logs. Every database already keeps a log of everything it does. Debezium reads that log, so each insert, update and delete operation arrives as an event, with the record before and after the change, and lands in Apache Kafka within seconds. Nothing ever queries the source systems, so they carry almost no extra load. In practice, the systems handling appointments, policies and claims keep running exactly as before, while the data flows out in the background.

Together with LUX MED Group engineers, we weighed every architectural option against scale, control and cost. Instead of MSK Connect, the managed connector service of Amazon Managed Streaming for Apache Kafka (MSK), we run the CDC connectors as containers on Amazon ECS. That setup serves all 35 source databases, saving about $30,000 per year, and allows instant scaling of tasks during initial data loads. In testing, initial data loads ran 18 times faster.

We designed reliability into the same layer. Debezium provides connectors for every engine in the LUX MED Group's estate starting from PostgreSQL, and the connectors are fault tolerant: after a restart they resume exactly where they left off, so no change is lost and none is delivered twice. For a data platform that doubles as an audit record, that guarantee matters as much as the speed.

A data platform with an audit trail by design

We chose Amazon Redshift as the engine for reasons that map directly onto the requirements. Massively parallel processing spreads complex queries across nodes to return results in seconds even at large scale, columnar storage keeps analytical queries fast by reading only the data they need, machine learning based optimization tunes query execution and data layout automatically, reducing manual administration, and RA3 nodes scale compute and storage independently as the estate grows.

Inside the warehouse, the data is organised as a medallion architecture: four layers, each one built from the one below it and rebuildable at any time. Raw CDC events land untouched from Kafka. The bronze layer keeps them immutable and append-only: a full audit trail in which every state of every record is preserved and recoverable. The silver layer deduplicates, validates types and keeps full history. The gold layer serves dimensional models ready for BI and AI.

For a regulated business, the layers double as a security model. Access is role-based per layer: analysts see gold only, engineers see silver and above, and the pipeline writes to bronze only. Everything is encrypted with TLS in transit and at rest under AWS KMS customer-managed keys, and every SQL statement against Redshift is logged. The bronze layer is the compliance record, so when an auditor asks what a record looked like at any point in time, the answer is already stored.

The layered design is also the recovery strategy. A bug in transformation logic is fixed the way software bugs are fixed, by correcting the code and rerunning the build, and the layer is reconstructed from bronze with nothing lost. Analytical convenience never puts source data at risk.

Transformations as code, schemas that evolve themselves

All transformation logic runs as dbt models inside Amazon Redshift: every model is a SQL file in Git, peer-reviewed like application code, with automatic data lineage, and automated execution.

Vendor-maintained source systems change overnight, with no migration scripts and no notice. When a policies table gains new columns, a conventional pipeline stops and queues events until an engineer intervenes. In our case, Debezium absorbs the schema change automatically, new columns flow through with the events, and the pipeline keeps running. Which columns a model actually uses is decided at the dbt layer, where engineers add them like any reviewed code change. Since go-live, schema changes have required zero manual interventions.

Built and delivered like software

The platform is engineered with the same discipline as the data it carries. All infrastructure is defined as code, orchestration runs on managed Apache Airflow (Amazon MWAA), and pipeline DAGs are deployed through their own CI/CD pipeline rather than edited by hand. Environments are reproducible, every change is reviewed before it ships, and the entire data platform can be recreated from the repositories.

The gold layer is the interface

Everything above the gold layer is standard SQL, so any compatible tool connects directly. Amazon QuickSight runs operational dashboards on always-fresh data. Amazon SageMaker can use the warehouse as a direct source for model training, opening the door to predictive underwriting and claims reporting. And for AI driven by Amazon Bedrock LLM models the gold layer is a knowledge base for AI assistants. The platform was built for analysts first, but AI may become just as important a consumer of the gold layer in the coming years.

Outcome One source of truth, minutes not hours

End-to-end latency from a source event to the warehouse dropped from 24 hours to single-digit minutes. Initial loads of new sources run 18 times faster, new sources are onboarded in weeks rather than months, and schema changes that would once have meant broken morning dashboards now require no human intervention at all.

Just as importantly, the whole organization now works from one source of truth. Business teams query the gold layer directly instead of waiting in line for the data team, dashboards run on data that is minutes old, and the same layer already feeds the AI use cases.

The collaboration continues: remaining systems are being onboarded, underwriting and claims teams are getting predictive reporting, and regulatory reporting is moving to the warehouse as a single, auditable, always-fresh source. You can read about the next chapter — modernizing the security foundations of the same platform — in our LUX MED security case study.