Engineer
TATA Consulting Services - Cincinnati, OH
posted 16 days ago
About the position
We are seeking a skilled Data Engineer with extensive experience in Information Technology, particularly in the design, development, and implementation of large and medium-scale data warehousing applications. The ideal candidate will have a strong background in ETL tools, specifically IBM Infosphere Information Server DataStage, as well as proficiency in SQL and Unix. This role requires end-to-end experience in all stages of the Software Development Life Cycle, including data mapping, build, unit testing, system integration testing, and performance tuning. The candidate should also have a broad portfolio of AWS infrastructure tools and experience with cloud infrastructure management and automation technologies. The Data Engineer will work with various data types, including structured, semi-structured, and unstructured data, and will be exposed to modern data and analytics architecture, including Big Data and Cloud technologies. The role involves collaborating with data analysts, data architects, and software engineers to understand data requirements and deliver relevant data sets for business intelligence. The candidate will be responsible for developing and maintaining data pipelines for efficient ETL processes and building the required infrastructure for optimal data extraction, transformation, and loading using AWS and SQL technologies.
Responsibilities
- Design, develop, and implement large and medium-scale data warehousing applications using ETL tools like IBM Infosphere DataStage.
- Manage end-to-end software development life cycle processes including data mapping, build, unit testing, system integration testing, and performance tuning.
- Develop and maintain data pipelines for efficient data extraction, transformation, and loading (ETL) processes.
- Collaborate with data analysts, data architects, and software engineers to understand data requirements and deliver relevant data sets for business intelligence.
- Build infrastructure for optimal extraction, transformation, and loading of data from various sources using AWS and SQL technologies.
- Perform performance tuning and optimization of ETL jobs and SQL queries in complex report environments.
- Troubleshoot jobs and address production issues, including data issues and performance tuning.
- Develop new solutions to reduce manual intervention and improve existing processes.
Requirements
- 5-7 years of experience with AWS infrastructure tools (EBS, S3, EC2, Elastic IP, Route 53, VPC).
- Strong understanding of Data Warehousing principles, including fact tables, dimension tables, and star/snowflake schema modeling.
- Extensive experience with ETL processes for data warehouses, including data migration and data cleansing.
- Proficient in scripting languages such as shell, Python, and Ruby for monitoring and automation.
- Experience with various databases including Oracle, DB2, SQL Server, and Teradata.
- Strong expertise in writing PL/SQL stored procedures, functions, and packages.
- Proficient in writing Unix shell scripts and working with Unix operating systems.
Nice-to-haves
- Exposure to advanced analytics tools such as Python and R.
- Experience with IBM IIS DataStage Migration projects from planning to execution.
- Familiarity with highly scalable parallel processing infrastructure using parallel jobs and multiple node configuration files.
- Experience in design and development of Decision Support Systems (DSS).
- Experience in various testing methodologies including Unit Testing, Functional Testing, System Testing, Integration Testing, Regression Testing, User Acceptance Testing (UAT), and Performance Testing.
