vdartDenver, CO
AgentCore - Evaluation & Observability / AgentOps Architect
AgentCore - Evaluation & Observability / AgentOps Architect
AgentCore - Evaluation & Observability / AgentOps Architect
vdartDenver, CO
Apply for this role (opens in a new tab)today
Job role: AgentCore - Evaluation & Observability / AgentOps Architect
Duration: 3 months contract
Location: RemoteAWS Certification - Mandatory
Active AWS certification is required. Core Skills: AWS AgentCore, AgentOps, AI/ML, Evaluation, Observability, Generative AIWe are seeking a senior AgentCore / AgentOps Architect to design and establish enterprise-grade evaluation, observability, and operational standards for AI agents. The ideal candidate will have strong hands-on experience with AWS AgentCore, Amazon Bedrock, AI/ML, LLM evaluation, agent tracing, observability, and production AI systems.
Key Responsibilities:
Architect and establish AgentOps, Evaluation, and Observability frameworks for AI agents.
Define standards for monitoring:
- Agent execution paths
- Model and tool calls
- Token usage
- LatencyFailures and errors
- Infrastructure and model costs
- Establish evaluation frameworks for:
- Task completion
- CorrectnessFaithfulnessSafetyTool-selection accuracy
- Agent performance and reliability
Design automated evaluation and regression-testing frameworks for AI agents and LLM applications.
Build a continuous Observe → Evaluate → Improve → Validate → Deploy lifecycle.
Establish production monitoring, tracing, alerting, and performance dashboards.
Integrate AgentOps and evaluation capabilities into CI/CD and MLOps pipelines.
Define quality gates and validation standards before AI agents are promoted to production.
Partner with AI/ML, cloud, and engineering teams to operationalize agentic AI solutions.
Drive best practices for AI reliability, safety, governance, and responsible AI.
Required Skills:
Strong hands-on experience with AWS AgentCore.
Strong AI/ML and Generative AI background.
Experience with Amazon Bedrock and LLM-based applications.
Strong experience in AgentOps / AI Agent Observability.
Experience designing LLM/AI evaluation frameworks.
Experience with agent tracing, telemetry, monitoring, and performance analysis.
Strong understanding of LLM/agent metrics including correctness, faithfulness, safety, latency, cost, and tool-selection accuracy.
Experience with automated regression testing for AI/LLM applications.
Strong Python and API development experience.
Experience with cloud architecture, CI/CD, and MLOps.
Also on the board Same function, level within a rung
Level
Lead
Location
Denver, CO
Occupation
Computer Systems Engineers/Architects
Industry
Computer Systems Design Services
Posted
today