Data Engineer - Azure

• Design develop and maintain real-time data streaming pipelines using Spark Streaming or Azure Functions.

• Load merge and process machine logs from Kafka ensuring efficient data flow and transformation.

• Integrate processed data into Redis cache and send it to the data lake for long-term storage and analysis.

• Implement and optimize data processing solutions using Python.

• Apply software engineering best practices including code reviews version control and continuous integration/deployment (CI/CD).

• Collaborate with cross-functional teams to understand requirements and deliver high-quality data solutions.

Proven experience in real-time data streaming using Spark Streaming or Azure Functions. Strong proficiency in Python for data processing and automation tasks. Hands-on experience with Kafka for data ingestion and message queuing. Familiarity with Redis for caching and fast data retrieval. Knowledge of data lake architectures and best practices for data storage and retrieval. Solid understanding of software engineering principles including design patterns testing and documentation. Strong problem-solving skills and ability to work in a fast-paced collaborative environment.

Preferred Qualifications:

Experience with cloud platforms such as Azur  Knowledge of other programming languages and frameworks related to data engineering. Familiarity with data modeling ETL processes and data lake concepts.