Data Engineer

Kforce - Madison, WI

Full-time - Mid Level

Madison, WI

Administrative and Support Services

About the position

The Data Engineer will be responsible for designing, implementing, and maintaining data infrastructure using AWS technologies. This role focuses on building robust and scalable data pipelines, ensuring data integrity, quality, and availability, while collaborating with cross-functional teams to deliver comprehensive reporting solutions.

Responsibilities

• Create and optimize ETL processes using AWS services such as Glue, S3, Lambda, and Data Pipelines
• Develop scalable data architectures that support a microservice-based data ecosystem
• Utilize Terraform for infrastructure as code (IaC) to set up and manage resources
• Implement best practices in data modeling, ensuring efficient data structures
• Use PySpark and Python scripting for data transformation and enrichment
• Work with Glue crawlers to discover and catalog data sources
• Build and maintain Lambda functions for serverless data processing
• Leverage Glue jobs for data transformation and orchestration
• Utilize Glue catalog for metadata management
• Implement logging, monitoring, and alerting using AWS CloudWatch and other relevant tools
• Work with Delta Lake and Databricks for advanced analytics and data processing
• Query data stored in S3 data lakes using AWS Athena
• Interface with business stakeholders to gather requirements and deliver complete reporting solutions
• Collaborate with cross-functional teams to ensure data consistency and accuracy
• Stay updated with industry trends and emerging technologies related to data engineering
• Identify opportunities for process optimization and automation

Requirements

• Bachelor's degree in Computer Science, Information Systems, or a related field
• 3+ years of experience in data engineering, preferably with a focus on AWS technologies
• Proficiency in Python and PySpark
• Familiarity with Terraform for infrastructure provisioning
• Experience with Glue jobs, Lambda functions, and Glue crawlers
• Knowledge of Delta Lake, Databricks, and S3 data lake storage
• Strong problem-solving skills and attention to detail
• Excellent communication and teamwork abilities

Benefits

• Medical insurance
• Dental insurance
• Vision insurance
• Health Savings Account (HSA)
• Flexible Spending Account (FSA)
• 401(k) plan
• Life insurance
• Disability insurance
• Paid time off for salaried personnel
• Paid sick leave for hourly employees on Service Contract Act projects