💻 Senior Coding Annotator / LLM Evaluation Engineer –$70–$85/hour Remote

By
💻 Senior Coding Annotator / LLM Evaluation Engineer – Remote | Braintrust Are you a senior software engineer with strong Python skills and experience in LLM evaluation, coding annotation, or AI model testing? Braintrust is looking for experienced software engineers to join an advanced LLM evaluation and annotation team working at the intersection of software engineering, artificial intelligence, and model improvement. This is not a junior data annotation role. The position is designed for experienced engineers who can create challenging coding tasks, evaluate AI-generated code, identify model failures, and provide detailed technical feedback. 💼 Job Details Position: Senior Coding Annotator / LLM Evaluation Engineer Category: Engineering Job Type: Freelance / Contract Experience: 5–10 years Rate: 💰 $70–$85/hour Location: 🇫🇷 Paris, France | Europe Remote for strong candidates Timezone: CET/WAT – Full-day overlap Hours: 40 hours/week Duration: Initial 6 months, with potential for long-term engagement BTRST Referral: +14,543 ⚠️ Location Note: Paris-based candidates are preferred. Strong candidates elsewhere in Europe may be considered. Applicants outside the eligible location may have a low chance of receiving an offer. 🧠 What You’ll Do You will: Create challenging coding prompts and reference solutions Design benchmark-style coding tasks similar to SWE-Bench Evaluate LLM outputs for coding, debugging, refactoring, and implementation Identify model failures, edge cases, and reasoning gaps Compare different LLMs through head-to-head evaluations Build or configure coding environments for AI evaluation and reinforcement learning Apply detailed annotation and evaluation guidelines consistently Provide clear technical feedback to improve model performance 💻 Requirements Strong candidates should have: ✅ 5+ years of professional software development experience ✅ Strong Python skills ✅ Experience with at least one additional programming language ✅ 1+ year of coding annotation or LLM evaluation experience ✅ Experience with a major AI lab or AI infrastructure company is valuable ✅ Code review experience is a plus ✅ Strong technical feedback and evaluation skills ✅ Fluent written and spoken English ✅ Team leadership or mentoring experience is an advantage 🤖 Why This Opportunity? This role offers experienced engineers the opportunity to work directly with cutting-edge Large Language Models, using real-world software engineering knowledge to improve AI reasoning, coding ability, reliability, and quality. If you have both strong engineering experience and hands-on LLM evaluation/annotation experience, this could be an excellent long-term technical opportunity. Apply Now