Data Engineer

Introduction
At IBM work is more than a job – it’s a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better but to attempt things you’ve never thought possible. Are you ready to lead in this new era of technology and solve some of the world’s most challenging problems? If so lets talk.

Your Role and Responsibilities
Value Proposition:
Do you love to solve business problems and bring innovative technology to life? Do you love creating exciting features for users? Become an integral part of the IBM CIO’s diverse Marketing IT & Analytics team. We are a fun data driven group working in a fast-paced Agile environment with plenty of room to grow and be creative.

Job Description including responsibilities:
IBM believes in progress — that the application of intelligence reason and science can improve business society and the human condition. We strive to put the client first share expertise and restlessly reinvent to build innovative solutions for our customers. We embrace diversity and value what individuals add to our culture. Our global teams’ solution complex problems through teamwork invention and iteration to create original ideas that drive technology and society forward.
As a Data Engineer you will be integral to the process of invention and innovation. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. You will develop exciting solutions that change the way both IBMers and IBM clients work. You’ll work in a collaborative welcoming environment where new ideas drive technological transformation. We work in small cross-functional teams with big goals so you will need to be adaptive and adept at clear communication. We strive to produce scalable solutions and you will need to be familiar with how an automated framework helps drive them.

High level job responsibilities:

• Develop elegant flexible maintainable and scalable solutions to complex problems by delivering data and analytics.
• Demonstrate experience leading development efforts and coordinating work for junior developers.
• Define and consolidate development and modeling practices within a team.
• Leverage strong collaborative skills to define requirements develop plans and deliver iteratively to support our organization’s mission.
• Leverage strong analytical skills to understand business processes and propose effective data solutions.
• Communicate and manage relationships with business and development teams – provide guidance mentorship and direction as required.
• Translate business needs into technical requirements.
• Learn new tools technologies and processes for continuous improvement.
• As part of a self-directed team take ownership of activities and deliver them on-time and with quality.

Required Technical and Professional Expertise
Above all we value curiosity teamwork and a desire to learn. We are confident that if you possess the right attitude work ethic and skill set that you could succeed in the role even if you do not meet every one of the requirements below.

• 5+ years of experience in Data Engineering with Big Data.
• Comfortable multi-tasking and working as part of a global team as well as providing technical leadership and taking ownership.
• Adaptive to ambiguity and willing to change in a fast-paced environment.
• Advanced proficiency in Python Scala and PySpark.
• Advanced experience with SQL for complex queries and data manipulation.
• Expertise in developing and maintaining data pipelines using Apache Spark and Scala.
• Experience making continuous documentation improvements and maintaining clear and concise technical documentation.
• Strong skills in data modeling including designing complex dimensional data models.
• Strong understanding of Coud platforms particularly IBM Cloud and Cloud Object Storage.
• Strong understanding and application of clean code principles and best development practices.
• Proficiency in data validation testing and ensuring data quality.
• Proficiency in creating and managing workflows with Apache Airflow and Argo Workflows.
• Experience with handling large datasets programmatically and using SQL.
• Experience in test-driven development (TDD) and continuous improvement of code quality.
• Familiarity with CI/CD tools like Jenkins plus experience in setting up and maintaining CI/CD pipelines.

Preferred Technical and Professional Expertise

• Basic knowledge of Kubernetes (K8s) for container orchestration.
• Basic knowledge of business intelligence tools specifically IBM Cognos BI.
• Proficiency in using Jupyter notebooks for data exploration and analysis.