Data Engineer (Application/Solutions Architect)

Southern Company - Atlanta, GA

Full-time - Mid Level

Atlanta, GA

5,001-10,000 employees

Utilities

About the position

The Data Engineer (Application/Solutions Architect) at Southern Company is responsible for transforming data into consumable formats and delivering integrated data solutions across various environments. This role involves building, operationalizing, and maintaining data pipelines for analytics use cases, ensuring alignment with enterprise data management strategies, and collaborating with cross-functional teams to enhance data integration and quality.

Responsibilities

• Designing and developing methods to process structured, semi-structured, and unstructured data using batch and real-time data processing techniques.
• Delivering fast, reliable, and scalable data by incrementally processing it from files or streaming sources like Kafka, DBMS, and NoSQL.
• Developing release pipelines to automate recurring manual tasks such as creating build packages and deploying them to environments.
• Building and maintaining templates like code libraries and pipeline patterns to promote reuse and agility.
• Establishing gatekeeping processes to monitor and control the promotion of successful data processes into production.
• Collaborating with cross-functional teams, including data, business, and technical personas, as well as product owners/managers.
• Advocating for data reusability by breaking down monolithic data delivery processes into modular data product delivery.
• Ensuring data reliability by defining data quality and integrity controls within the pipeline and addressing data quality errors.
• Providing technical guidance and oversight to less experienced data engineers.
• Understanding the usage of performance optimization clusters to minimize data movement in batch and stream data processing.
• Recommending improvements to processes, technology, and interfaces to enhance team effectiveness.
• Participating in enterprise social networking sites to stay updated on new data technologies and best practices.

Requirements

• Bachelor's degree in a technical field such as computer science, engineering, or mathematics; advanced degree preferred.
• 5+ years of hands-on experience designing, developing, testing, deploying, and supporting data engineering and analytics solutions using on-premises tools like MSBI, Informatica, Oracle Golden Gate, SQL, Oracle, and SQL Server.
• 3+ years of hands-on experience with Microsoft cloud-based tools such as Azure Data Lake, Azure Data Factory, Azure Databricks, Python, Azure Synapse, Azure Key Vault, and Power BI.
• Working experience with batch and real-time data processing frameworks.
• Experience in creating functional and technical designs for data engineering and analytics solutions.
• Hands-on experience with big data technologies such as Hadoop, HIVE, and Spark.
• Experience with data quality tools and data modeling techniques.
• Familiarity with containerization methodologies like Docker and OpenShift.
• Experience with Agile and DevOps, CI/CD methodologies.

Nice-to-haves

• Hands-on experience developing and supporting statistical models, R, and/or Python-based AI/ML solutions.
• Experience designing and developing solutions involving data sourcing, enrichment, and delivery using APIs & Web Services.

Benefits

• Health insurance
• Dental insurance
• Vision insurance
• 401k retirement plan
• Paid holidays
• Paid time off
• Flexible scheduling
• Professional development opportunities