Data Engineer

Data Engineer

Experience Level: Mid

Position Overview: 

This role will be responsible for expanding our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler, who enjoys optimizing data systems and building them from the ground up. 

This position will support our software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. 

Roles and Responsibilities:

• Design, implement, and maintain scalable data pipelines for batch and real-time data processing using tools like Apache Airflow, Apache Kafka, and Azure Data Factory.
• Build ETL/ELT workflows, automate data movement, and integrate data from multiple sources (APIs, databases, third-party systems).
• Work with cloud platforms (e.g., AWS, Azure) and tools like Snowflake, Amazon Redshift, Azure Data Lake, and S3 to manage large-scale data storage.
• Ensure data quality through automated testing, monitoring, and validation of data pipelines.
• Collaborate with Data Analysts, Data Scientists, and other stakeholders to understand and deliver on data needs.
• Optimize and tune data pipelines for performance, scalability, and cost efficiency.
• Maintain documentation and adhere to best practices for data engineering and architecture.
• Continuously stay up-to-date with new technologies and industry trends, improving systems and processes as needed.

Must-Have Skills:

• 5+ years of experience in Data Engineering, with expertise in building data pipelines and managing cloud-based data infrastructure.
• Proficiency in Python, SQL, and Java/Scala for data processing and transformation.
• Experience with cloud platforms (e.g., AWS, Azure) and tools like Snowflake, Redshift, S3, Azure Data Factory.
• Hands-on experience with data orchestration tools like Apache Airflow and Apache Kafka for real-time data workflows.
• Strong knowledge of ETL/ELT processes, data modeling, and data quality practices.
• Ability to troubleshoot, optimize, and maintain high-performance data pipelines.
• Experience working in cross-functional teams and collaborating with Data Scientists and Data Analysts.

Nice-to-Have Skills:

• Familiarity with data governance and privacy regulations (e.g., GDPR, CCPA).
• Experience with NoSQL databases (MongoDB, Cassandra), Hadoop, or Apache Spark.
• Knowledge of machine learning pipelines or integration with MLflow, TensorFlow, PyTorch.
• Experience with BI tools like Power BI, Tableau, or Looker for data visualization.

Qualification: 

• Bachelor / Masters  or equivalent degree in Computer Science or related field. 

Location: 

• Hyderabad