Building an HSK Speaking Test AI: Real-time Tone Grading with Gemini

I built a free Mandarin speaking assessment tool that grades tone + grammar in real time. Here's the engineering behind it.

The Problem

HSK (Chinese proficiency test) has a speaking component (HSKK), but most learners can't self-assess their level. Online tutors are expensive. Generic AI conversation tools don't grade tones.

So I built ToneTutor: a 3-minute spoken-HSK test that estimates your speaking level and identifies weak points.

The Tech Stack

Frontend:

• Web Audio API (record user voice → PCM → LINEAR16)


• React + TypeScript (real-time transcript display)

Backend:

• FastAPI (Python) on Google Cloud Run


• Gemini 2.5 Flash (real-time conversation + transcript grading)


• Firestore (user sessions + results)

The Challenge:

Web Audio API records as WebM. Gemini expects LINEAR16 (WAV). iOS Safari doesn't support WebM. So:

1. Transcode WebM → PCM in browser (Web Audio context)


2. Send raw PCM bytes to backend


3. Backend wraps PCM in WAV header → sends to Gemini Speech-to-Text


4. Gemini analyzes transcript + provides HSK level estimate

The Grading Loop

python

async def grade_session(transcript: str):

    prompt = """

    Rate this Mandarin response on HSK 1-6 scale.

    Assess: tone accuracy, grammar, vocabulary range.

    Provide: level estimate + weak points.

    """

    response = await gemini.generate_content(prompt, stream=True)

    return parse_hsk_level(response)



Results



- 3-min test

- Real-time feedback

- Shareable HSK score card

- Free (limited sessions)



Open source coming soon. Built because I'm a native speaker + voice actor frustrated with generic tools.



Try it: tonetutor.tefusiang.com (free for 3 sessions)



Curious about the speech-to-text pipeline or tone grading logic? Ask below.