Summary
Overview
Work History
Education
Skills
Timeline
Generic

Jordan Walsh

Santa Maria

Summary

Dynamic LLM Evaluator & RLHF Data Specialist at Surge AI, adept in analytical reasoning and data interpretation. Authored high-fidelity failure examples and refined agent behavior, enhancing tool-use efficiency. Proven track record in performance assessment and project evaluation, driving impactful results in complex environments.

Overview

14
14
years of professional experience

Work History

LLM Evaluator & RLHF Data Specialist (contract)

Surge AI (via DataAnnotation.tech)
Remote Work
11.2024 - Current

Agentic AI Training & Golden Trajectories: Authored and edited multi-step “golden” trajectories and agent behavior for tool-using agents in Surge AI’s Corecraft simulated workplace and other custom-built enterprise environments. Audited tool choice, sequencing, termination behavior, and adherence to complex system/developer instructions; refined sub-optimal tool-call plans to improve tool-use efficiency while preserving outcome correctness.

Adversarial / Naturalistic Domain Evaluation (Legal & STEM): Leveraged STEM and Legal qualifications to engineer challenge prompts designed to surface failures in robust models (e.g., Claude Sonnet-tier and Gemini Pro-tier models), including complex multi-modal prompts to test specific STEM domain reasoning capabilities. Precisely documented jurisdiction-sensitive hallucinations and Law-related reasoning errors to create high-fidelity failure examples; authored corrected “gold” responses and scoring rationales aligned to strict rubric sets used for downstream training.

Synthetic Data & Sandbox Populating: Constructed realistic user personas and scenarios to populate enterprise sandbox environments with realistic simulated datasets (Google Workspace, Slack, Jira, Shopify, WhatsApp-style workflows) for authentic agent testing. Converted large amounts of unstructured natural language inputs into strict JSON-formatted datasets to support logical consistency and tool-use evaluation.

Quality Assurance & Scoring Guidelines: Selected for high-priority Rate & Review workflows due to consistent accuracy. Wrote and maintained scoring guidelines/rubrics for new project types, defining ground-truth standards for agentic behaviors and instruction-following.

Complex Instruction Adherence: Executed tasks requiring adherence to evolving guideline documents (including long-form docs updated weekly), maintaining accuracy across frequent project updates and rapid spec changes.

Property Manager

Nancy G. Scoville
Pismo Beach, CA
01.2015 - Current
  • Managed tenancy and lease renewals proactively: retaining existing tenants at higher rates while minimizing turnover-related expenses, vacancies, and ensuring excellent customer service for existing and prospecting renters
  • Coordinated and supervised the hiring of professionals for necessary repair work in addition to general property maintenance services, maintaining detailed communication with property owners throughout all steps of the process.
  • Evaluated, recommended, and implemented changes in rental pricing to remain competitive in the market.
  • Oversaw all aspects of onboarding and evicting residents- including lease initiation, conducting property walk-throughs before and after tenancy, and conducting background checks on prospective renters.
  • Maintained thorough and accurate financial records regarding rental income, maintenance costs, and repair costs; ensuring clear communication with property owners regarding income and expenses.

Deckhand

Nautilus Sportfishing
San Diego, CA
02.2021 - 01.2023
  • Operated and maintained fishing gear and equipment efficiently during daily excursions.
  • Educated guests on fishing techniques and local marine life to enhance their experience.
  • Contributed to efficient operations by maintaining a clean and organized deck environment.
  • Monitored weather conditions and sea state to optimize fishing strategies.
  • Collaborated with team members to prepare bait, catch, and maintain cleanliness on board.

Freelance Research & Writing Consultant

Self Employed
Remote
01.2017 - 01.2021
  • Produced structured long-form deliverables and research briefs by synthesizing information across many public sources; delivered well-cited references and actionable summaries tailored to client needs and specifications
  • Managed end-to-end delivery: intake, scoping, outlining, drafting, revisions, and final handoff.
  • Handled niche one-off topics requiring rapid domain familiarization and clear explanation for non-expert audiences.
  • Cross-checked claims across multiple sources, flagging uncertainty or conflicting information and documenting assumptions when needed.
  • Evaluated the effectiveness of consultation methods using client feedback surveys, making necessary adjustments as needed for continuous improvement.

Food Server Assistant

Steamers Of Pismo
Pismo Beach, CA
08.2013 - 02.2016
  • Managed order accuracy and timely delivery to enhance guest satisfaction.
  • Maintained, cleaned, and organized restaurant dining areas and service stations in accordance to company policy, and in compliance with local health regulations.
  • Fostered a supportive team environment that contributed to overall restaurant success.
  • Delivered exceptional customer service in fast-paced dining environment.
  • Trained and mentored new team members on service standards and procedures.
  • Demonstrated strong multitasking skills by managing several tables at once while maintaining high standards of service quality.

Specimen Processing Technician

Ward's Science
San Luis Obispo, CA
06.2012 - 01.2014
  • Processed and shipped biological specimens in compliance with safety and regulatory standards.
  • Conduct regular audits and upkeep of specimen storage conditions to maintain integrity and viability.
  • Collaborated effectively with laboratory staff to ensure smooth communication between departments, enhancing overall productivity.
  • Collaborate with laboratory personnel to troubleshoot issues related to specimen integrity and processing workflows.
  • Expedited critical test results by promptly notifying appropriate medical personnel of any abnormal findings or discrepancies.

Education

No Degree - Undeclared

San Diego State University
San Diego, CA

No Degree - General Studies

Cuesta College
San Luis Obispo, CA
05-2010

Skills

  • Analytical reasoning
  • Data interpretation
  • Performance assessment
  • Project evaluation

Timeline

LLM Evaluator & RLHF Data Specialist (contract)

Surge AI (via DataAnnotation.tech)
11.2024 - Current

Deckhand

Nautilus Sportfishing
02.2021 - 01.2023

Freelance Research & Writing Consultant

Self Employed
01.2017 - 01.2021

Property Manager

Nancy G. Scoville
01.2015 - Current

Food Server Assistant

Steamers Of Pismo
08.2013 - 02.2016

Specimen Processing Technician

Ward's Science
06.2012 - 01.2014

No Degree - Undeclared

San Diego State University

No Degree - General Studies

Cuesta College
Jordan Walsh