Marc Planagumà | SCRM-LIDL Digital Hub
Marc Planagumà is head of data engineering at SCM-LIDL Digital hub. Senior Data Engineer expert on building and managing highly scalable platforms and teams for advanced analytics. His current focus of work is design and build the central big data platform for LIDL Group. He also worked at Zurich Insurances as Platform Manager and previously in research institutions like Eurecat, Berlin Big Data Center, Barcelona Digital and Telefonica R&D.
PRESENTATION: HOW TO BUILD AN AGILE AND SCALABLE DATA PLATFORM INSIDE A BIG ENTERPRISE
Create your own data analysis platform is an increasingly common decision in business and productive sectors. Long been carried out practice on research labs level and and digital native companies but relatively recent for large enterprises. Since 90s, IT has been outsourced from large companies but the boom in data analysis has crashed with this strategy.
The datascinece profiles increase in the big companies has made more and more evident the need to have more and better data to analyze. But especially at time to bring advanced analytics solutions in production. It has been shown how the scale of data volumes, customers and requirements from large companies have make obsolete the monolithic and outsourced solutions for data processing.
The big data analytics landscape has been spread into thousands of open source solutions where classic data processing big players providers have ceased from top lead market. Native digital companies have opted in many cases to create their own technology, which they have even released publicly. In the statups case, they are largely committed to designing their own architectures based on integrate different opensource solutions and/or Paas/SaaS services, always with great agility on technological and strategic change. Otherwise, In the case of enterprises, has become unavoidable to build a data platform able to interoperate with all company’s data and provide availability, quality and scalability data services. This brings several challenges for a larges companies:
Technological knowledge and data domain knowledge insourcing by own teams and culture creation.
Agile culture commitment for design, develop and deploy solutions on fast iterations. Using in lean and fail-fast strategy.
Independence and decision-making power as a platform about infrastructure, toolstack, architecture, ALM and even the data product itself.