About This Role
This senior engineering role focuses on protecting the organization's data and systems by managing the ingestion of security telemetry into Elasticsearch and maintaining the underlying Elastic clusters. The position involves owning the end-to-end pipeline for security data, from source integration to ensuring data quality and reliability for detection, incident response, and consulting teams.
Responsibilities
- Build and maintain ingestion of security-relevant data into Elasticsearch (cloud provider audit logs, identity/SaaS activity, endpoint and asset data)
- Integrate with third-party and cloud provider APIs to pull telemetry, handling authentication, pagination, rate limits, and schema changes
- Keep Elastic Cloud on Kubernetes clusters healthy, monitoring and upgrading them regularly
- Update versions and builds, manage capacity and shards, handle index lifecycle management (ILM), and enable cross-cluster search (CCS)
- Use Terraform to manage cloud infrastructure and Elasticsearch resources, including pipelines, index templates, and alerts
- Utilize Kubernetes and Helm to deploy scheduled ingest jobs
- Use AI automation and tooling to reduce toil, including self-healing jobs, health checks, alerting, internal CLIs, and AI or agent-assisted workflows for investigation and operations
- Own data quality and reliability, monitoring backfills, ensuring consistent schemas and fields, and managing costs
- Take ownership of protecting the confidentiality, integrity, and availability of organizational data and systems by following applicable privacy and security policies, standards, and procedures
- Ensure that all individual contributions follow Elastic’s Secure Software Development Framework (SSDF)
- Proactively participate in mandatory role-based training to ensure personal technical execution consistently aligns with the highest standards of data protection, data privacy, and system resilience
Requirements
- Ability to operate Elastic and Elasticsearch in production, including managing ingest pipelines, index templates, mappings, and queries, while ensuring clusters are healthy and upgraded
- Experience with ECK or Elasticsearch on Kubernetes is strongly preferred
- Proven track record of using AI to accelerate development, debug complex systems, and accelerate operations, while still owning the final outcomes
- Kubernetes: deploying and operating workloads (scheduled jobs, Helm charts, operators); troubleshooting pods/jobs in a cluster
- Terraform: managing cloud and Elasticsearch resources as code
- API integration: consuming REST APIs for data ingestion, including authentication, pagination, rate limiting concurrency, and error handling
- Python scripting: able to read, write, and modify ingestion/automation scripts (scripting-level, not full software-engineering depth)