FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in building and operating AI-powered workflows and agents, with a strong focus on LLM evaluation frameworks, prompt design, and context engineering. Proficient in implementing metrics and success measures to ensure AI solutions are production-ready and aligned with business outcomes.
Highest-signal resume keywords
AI EngineeringLLM Evaluation FrameworksPrompt DesignRetrieval-Augmented Generation (RAG)Automation Engineering
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
AI-Powered WorkflowsLLM-Based WorkflowsContext EngineeringOrchestration FrameworksTelemetry ImplementationData Quality EnforcementEvaluation Metrics DefinitionBehavioral Signals AnalysisRegression FrameworksHallucination Reduction Strategies
Soft Skills
Analytical AbilitiesClear Communication Skills
Tools & Technologies
SalesforceMicrosoft 365Azure AICopilotGTM ToolingWorkflow Platforms
Industry Keywords
AI SolutionsAutomation SystemsProduction EnvironmentsQuality AssuranceObservability
Tech Stack
Tools & technologiesAzure
About the role
Key responsibilities & impact- Build, test, deploy, and operate AI-powered workflows, agents, and automations within Salesforce, Microsoft 365, Azure AI, and Copilot
- Implement prompt logic, orchestration flows, tool calling, and context retrieval for LLM-based systems
- Translate approved AI solution designs into scalable, maintainable, and secure implementations
- Ensure AI solutions are production-ready, observable, resilient, and aligned with business outcomes
- Design and maintain LLM evaluation frameworks for accuracy, relevance, consistency, and impact
- Implement offline and online evaluations using test datasets, golden answers, regression suites, feedback loops, telemetry, and behavioral signals
- Define evaluation thresholds, quality gates, and launch-readiness criteria
- Design hallucination-reduction strategies using RAG, context filtering, grounding, citations, guardrails, validation, and response constraints
- Monitor AI outputs, confidence signals, and failure modes in live environments
- Investigate incorrect or low-confidence outputs and implement corrective improvements
- Define and instrument metrics covering quality, adoption, latency, reliability, and operational or revenue impact
- Build telemetry connecting AI usage to cycle-time reduction, capacity unlocked, and risk reduction
- Implement context pipelines using structured data, documents, and governed knowledge assets
- Enforce data quality, access controls, grounding standards, and versioning
- Support documentation, testing, auditability, change management, privacy, security, and responsible AI requirements
- Collaborate with Product Managers, AI Solutions Architects, Systems teams, analytics teams, and GTM stakeholders
- Support enablement, adoption, reviews, retrospectives, and continuous post-launch improvement
Requirements
What you’ll need- Strong software, data, or automation engineering background with experience operating production systems
- Hands-on experience building and operating LLM-based workflows and agents
- Demonstrated experience designing and operating LLM evaluation frameworks
- Experience with offline evaluations using test datasets and regression frameworks
- Experience with online evaluations using user feedback, telemetry, and behavioral signals
- Experience reducing hallucinations in production AI systems using grounding, validation, and guardrails
- Experience defining metrics and success measures
- Strong understanding of retrieval-augmented generation (RAG), prompt design, and context engineering
- Familiarity with CRM, GTM tooling, and workflow platforms
- Strong analytical and debugging abilities for complex AI system behavior
- Clear written and verbal communication skills across technical and non-technical audiences
- Experience in AI engineering, applied machine learning, automation engineering, or related roles
- Experience deploying AI or automation systems into production environments
- Experience working with cross-functional product, operations, and systems teams
- Experience supporting AI quality, reliability, and evaluation in live systems
- Experience with LLMs, AI agents, prompt engineering, orchestration frameworks, retrieval-augmented generation (RAG), or workflow automation
- Quality-first approach incorporating testing, reliability, observability, security, and governance
- No degree required; Granicus states it does not have degree requirements for most roles
Benefits
Comp & perks- Remote-first work environment
- Employee Resource Groups
- Coffee with Mark sessions with the CEO
- Microsoft Teams communities focused on wellness, art, pets, family, and parenting
- Special guest sessions addressing issues impacting employees