Cloud Resources - Palo Alto, CA
posted 2 months ago
As a Data Architect, you will play a crucial role in designing and managing databases that are essential for our data-driven initiatives. Your primary responsibilities will include working with various database technologies such as Cosmos DB, MongoDB, and PostgreSQL to create efficient and scalable data models. You will ensure data consistency and integrity across all platforms, which is vital for maintaining the quality of our data assets. In addition to database management, you will be responsible for developing and maintaining distributed task processing systems using Celery. This will involve ensuring robust task queue management with tools like Redis or RabbitMQ, which are critical for handling asynchronous tasks effectively. You will also implement and manage Kafka Streams to facilitate real-time data processing and streaming, which is increasingly important in our fast-paced environment. Your expertise in data orchestration will be leveraged as you work with Airflow to manage workflows and automate data pipelines. You will also develop and manage webhooks using FastAPI, integrating them with serverless functions to enhance our data processing capabilities. Familiarity with Iceberg for data lake management and optimization will be beneficial as you work to streamline our data storage solutions. Experience in the healthcare domain will be a significant advantage, as it will help you understand the specific data challenges and requirements in this field.