Job Description

Conversation Quality Analyst

Remote & Contract

Role Summary
We have a large volume of conversation data and need someone who can monitor results, evaluate transcripts, and generate actionable insights to help accelerate our quality program. This is a hands-on human evaluation role - reviewing conversations against defined markers, identifying trends, and providing structured feedback - not a data science or model-building position.

Responsibilities

• Review and evaluate customer-bot conversations on a daily and monthly basis , ensuring assessments are consistent and aligned with predefined quality markers (accuracy, relevance, tone, compliance, and clarity) as well as the ground truth captured.
• Document findings and generate insights and patterns to help teams quickly address recurring issues and improve conversation quality.
• Use tools such as Excel or SQL to organize, analyze, and report on evaluation data, making it easier to track trends and share results with stakeholders.
• Provide structured feedback that directly informs the refinement of conversational AI models, training data, and workflows.
• Collaborate with cross-functional teams (product, data science, engineering) by supplying human evaluation results and insights in a timely manner.
• Maintain strict confidentiality and ensure compliance with IRS Section 7216 requirements for handling sensitive data.

Qualifications

• Must be based in the US (due to IRS Section 7216 compliance).
• Strong attention to detail with the ability to apply evaluation guidelines consistently and objectively.
• Strong analytical skills , including the ability to recognize patterns, interpret data, and produce meaningful insights.
• Proficiency with Excel (pivot tables, formulas, charting) required; exposure to SQL or similar tools is a strong plus.
• Comfortable working with large volumes of text data on a recurring basis.
• Excellent written communication and documentation skills.
• Prior experience in data labeling, quality assurance, conversation review, or customer support is preferred.
• Familiarity with conversational AI/chatbots, evaluation frameworks, or analytics tools is a plus.

Education

• Bachelor's degree required , preferably in business, analytics, computer science, or a related field.

Job Tags

Similar Jobs