Senior Data Engineer

The Data & Analytics - Enterprise Data Lake Team has been established to create, operate, and run the Enterprise AI, BI, and Data platforms that facilitate faster time to market for reporting, analytics, and data science teams to run experiments, train models, and generate insights. The Data Factory team sits at the heart of this mission — ingesting data from various sources into the Enterprise Data Lake and performing content-agnostic transformations to provide end-users with clean, reliable, and well-governed datasets. As a Senior Software Engineer , you will take a pivotal role in shaping the future of our data ingestion and transformation solutions within the Enterprise Data Lake. You will lead the development of complex data pipelines and systems, focusing on delivering high-quality, scalable data solutions while leveraging your experience to enhance data accessibility, reliability, and usability. This role requires deep expertise in distributed data systems design, ETL/ELT pipeline development, data modeling, and cloud-based infrastructure, along with the ability to evaluate AI and automation tools for specific data engineering use cases. You'll manage technical debt effectively while designing scalable data ingestion frameworks, transformation pipelines, and reusable data products. About the Role: - Design and implement end-to-end solutions to ingest data from multiple structured, semi-structured, and unstructured sources into the Enterprise Data Lake. Conduct source data analysis in collaboration with data subject matter experts to build robust data pipelines and thoroughly document data transformations. Develop and maintain scalable, sustainable ETL/ELT pipelines for both on-premises and cloud environments, ensuring efficiency, reliability, and performance. Innovate with new approaches for source ingestion, ELT data pipelines, AWS-based frameworks, and stakeholder-facing documentation and usability. Design and build data products that serve downstream consumers including data science, business analytics, and reporting teams. Perform end-to-end deployment of data pipelines using CI/CD processes, ensuring high-quality releases and minimal disruption to data consumers. Collaborate with business analysts, data architects, and cross-functional teams to gather requirements, define data contracts, and deliver solutions aligned with business goals. Effectively communicate and liaise with data management teams embedded across the organization to ensure alignment on data governance standards and best practices. Contribute to improving the overall data consumer experience by delivering clean, well-documented, and easily accessible datasets. Manage and prioritize technical debt while maintaining a high standard of code quality, pipeline reliability, and system scalability. Evaluate and adopt AI-powered tools and frameworks to improve pipeline development, data quality monitoring, and operational efficiency. About You: - You're a fit for the role of Senior Data Engineer, if you bring 7 years of demonstrated expertise across the following areas: Senior Data Engineer - AWS, Python, spark, snowflake, SQL, CICD, data modelling, AI Core Data Engineering Expert in Data Pipeline Development — ETL/ELT tooling for on-premises and cloud environments Expertise in ETL tools, preferably Informatica Working experience building sustainable data pipelines to Data Lakes, Data Warehouses, and Data Marts Expertise in handling semi-structured and unstructured data Big Data experience with distributed processing frameworks Programming & Databases Highly proficient in SQL Proficient in Python and/or Spark for data engineering workloads Highly experienced in working with relational and non-relational databases, including Snowflake Strong knowledge of ingesting data from APIs, RDBMS, SaaS Application ( CRM and ERP ), Social Medial and Realtime/Eventr streaming sources Hands-on experience on DBT Transformation, Shell Scripting, ... (truncated, view full listing at source)