Big Data Developer

Experience: 3 – 10 Years

Job Requirements

Core Skills & Expertise

• Strong understanding of concurrent software systems, ensuring scalability, maintainability, and robustness.
• Expertise in designing application solutions within the Hadoop ecosystem.
• Deep knowledge of Hive, HDFS, YARN, Spark, Spark SQL, Scala, and PySpark.
• Proficiency in handling HDFS file formats such as Parquet, ORC, Sequence, and their use cases.
• Solid understanding of data warehousing systems.
• Experience in scripting languages such as Shell and Python.
• Familiarity with HortonWorks distribution and SQL engines (Tez, MR).
• Exposure to Java, RESTful services, and Maven is a plus.

Automation & Deployment

• Experience in Control-M development and resource monitoring using Grafana.
• Ability to create automation scripts in Jenkins for builds, test frameworks, and application configurations.
• Hands-on experience in scalable application deployment using tools like Bitbucket, Jenkins, and Azure DevOps (ADO).

Skill Set

Mandatory Skills:

• Big Data Technologies: Hive, HDFS, Spark, Scala, PySpark

Good to Have:

• Schedulers: Control-M
• ETL Tools: Dataiku
• Scripting & OS: Unix/Shell scripting
• Integration Services: FileIT, MQ, and others
• CI/CD Tools: Jenkins, Jira, ADO DevOps suite