Job Description

Overall Experience: 6 to 12 Years

Job Location: Bangalore/Chennai/Pune

Job Requirements*

Design, develop, and maintain scalable data pipelines for data ingestion, transformation, and processing.
Ensure high standards of data quality, validation, monitoring, and governance across data platforms.
Build, orchestrate, and optimize batch and real-time data pipeline workflows.
Implement and support Medallion Architecture (Bronze, Silver, Gold layers) for efficient data management and analytics.
Develop and integrate APIs for data exchange and system connectivity.
Collaborate with cross-functional teams in an Agile environment to deliver high-quality data solutions.
Follow Git-based development practices, including branching strategies, code reviews, and version control.
Implement and maintain CI/CD pipelines to automate deployment, testing, and release processes.

Key Responsibilities*

Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL), Azure Synapse, Azure Data Factory, and other Azure data services.
Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
Optimize and tune data pipelines for performance and scalability.
Write and optimize complex SQL queries for data extraction, transformation, and analysis.
Use PySpark for large-scale data processing and analytics.
Implement data partitioning, bucketing, z-ordering, liquid clustering and indexing strategies for efficient data retrieval.
Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
Work with APIs, streaming data, and batch processing to ensure seamless data integration.
Implement data governance practices to ensure data quality, consistency, and security.
Monitor and troubleshoot data pipelines to ensure data accuracy and availability.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
Work closely with DevOps teams to deploy and monitor data pipelines in production environments.
Document data pipelines, workflows, and processes for knowledge sharing and future reference.
Maintain up-to-date documentation on data architecture and data models.