WSP Global - New York, NY
posted about 2 months ago
This opportunity at WSP is currently initiating a search for a Data Engineer for our Bridge Inspection Group at the Penn Plaza, New York office. The selected candidate will be involved in projects with our Bridge Inspection Team and will be part of a growing organization that meets our client's objectives and solves their challenges. This role is responsible for the design, deployment, development, and maintenance of data solutions. The Data Engineer will participate in a variety of data-related projects and work closely with business users and other stakeholders to gather requirements and build data pipelines that meet the organization's needs. The Data Engineer will deploy and support data platforms that process and store data from structural engineering, bridge design, and bridge inspection in the required format. This position will contribute to software development methods, tools, and techniques, applying agreed standards and tools to achieve well-engineered outcomes. Responsibilities include designing, developing, and implementing data solutions to meet business requirements and data ingestion needs, facilitating accurate and timely data availability for analysis and decision-making. Key tasks involve extracting, loading, and transforming (ELT/ETL) data from various sources, including on-premises and cloud-based systems, APIs, databases, and files. The Data Engineer will also develop APIs as needed to transform data into the required format from various sources of bridge design and inspection. Writing well-designed, efficient code that adheres to security standards is essential, as is monitoring and troubleshooting data pipelines to identify and resolve issues promptly to minimize disruptions in data processing for both on-premises and cloud environments. Additionally, the role requires implementing data quality checks and validation processes to ensure the accuracy and completeness of data, writing complex queries and scripts to efficiently manipulate, transform, and process raw data from bridge inspection, and creating and executing data validation processes to ensure the reliability and consistency of incoming data. The Data Engineer will continuously optimize data pipelines for performance, scalability, and reliability, create and maintain technical documentation, and contribute to building and maintaining data catalog and lineage. The position also involves designing and developing CI/CD processes that ensure high availability and agility, developing cloud data services provisioning automation with integrated capabilities of IAM, network, security policies as code, and observability. Staying updated with the latest trends and best practices in data engineering, cloud computing, and Azure services to suggest innovative solutions to continually improve the organization's data intelligence capabilities is crucial. Finally, the Data Engineer will monitor and report on supplier performance, customer satisfaction, adherence to security requirements, and market intelligence.