CA | ES | EN

Marc Planagumà | SCRM-LIDL Digital Hub

Head of Data Engineering
marc planaguma lidl

Marc Planagumà | SCRM-LIDL Digital Hub

Head of Data Engineering

Biography

Marc Planagumà is head of data engineering at SCM-LIDL Digital hub. Senior Data Engineer expert on building and managing highly scalable platforms and teams for advanced analytics. His current focus of work is design and build the central big data platform for LIDL Group. He also worked at Zurich Insurances as Platform Manager and previously in research institutions like Eurecat, Berlin Big Data Center, Barcelona Digital and Telefonica R&D.

PRESENTATION: HOW TO BUILD AN AGILE AND SCALABLE DATA PLATFORM INSIDE A BIG ENTERPRISE

Create your own data analysis platform is an increasingly common decision in business and productive sectors. Long been carried out practice on research labs level and and digital native companies but relatively recent for large enterprises. Since 90s, IT has been outsourced from large companies but the boom in data analysis has crashed with this strategy.

The datascinece profiles increase in the big companies has made more and more evident the need to have more and better data to analyze. But especially at time to bring advanced analytics solutions in production. It has been shown how the scale of data volumes, customers and requirements from large companies have make obsolete the monolithic and outsourced solutions for data processing.

The big data analytics landscape has been spread into thousands of open source solutions where classic data processing big players providers have ceased from top lead market. Native digital companies have opted in many cases to create their own technology, which they have even released publicly. In the statups case, they are largely committed to designing their own architectures based on integrate different opensource solutions and/or Paas/SaaS services, always with great agility on technological and strategic change. Otherwise, In the case of enterprises, has become unavoidable to build a data platform able to interoperate with all company’s data and provide availability, quality and scalability data services. This brings several challenges for a larges companies:

  • Technological knowledge and data domain knowledge insourcing by own teams and culture creation.

  • Agile culture commitment for design, develop and deploy solutions on fast iterations. Using in lean and fail-fast strategy.

  • Independence and decision-making power as a platform about infrastructure, toolstack, architecture, ALM and even the data product itself.

This presentation we will explain a success story about how we have built a data platform including architecture, teams, methodology and organization within an enterprise such as LIDL. As we have been able to be agile, autonomous and self-sufficient to build in a short period of time a data platform that enables LIDL teams and departments to create scalable and robust data products. As we have created, adapted and transformed methodologies and protocols from LIDL enterprise to be able to be agile as a startup but still providing the robustness and security that a large company demands.

All session by Marc Planagumà | SCRM-LIDL Digital Hub