What 18 months building a self-hosted media server taught me about playback

Project: https://quven.tv/

Security model: https://quven.tv/security/

For the last 18 months, I have been building Quven, a self-hosted media server for personal movie, TV, and documentary libraries.

I started with a seemingly simple goal: let people keep their media on their own hardware while giving them a polished client experience.

Playback quickly became the hardest part.

A media server does not simply send a video file to a screen. It has to understand the source, the client, the network, and the user's choices, then select a playback path without making any of that complexity feel visible.

These are some of the lessons I learned.

1. "Can this file play?" is the wrong question

The real question is whether a particular client can play a particular combination of:

• container;


• video codec and profile;


• audio codec and channel layout;


• subtitle format;


• resolution, bitrate, and frame rate;


• HDR format;


• network conditions.

A client might support the video codec but not the audio track. A browser might decode the video but require a different container. Enabling an image-based subtitle can turn an otherwise direct-playable file into a video transcode.

Playback compatibility is therefore not a boolean property of a file. It is a negotiation between the source and the active client.

2. Direct play should be the preferred outcome, not a promise

Direct play preserves the original file and avoids unnecessary server work. When the client supports the selected combination, it is usually the best path.

But forcing direct play at all costs produces a worse experience. A high-bitrate file may technically be supported while still exceeding the available connection. A selected subtitle might require burning into the video. A television may accept a container while rejecting one of its audio formats.

The practical hierarchy I settled on is:

1. Direct play when the complete source is compatible.


2. Remux when the streams are compatible but the container is not.


3. Transcode only what must change.

That ordering sounds obvious. Making it reliable across native applications and browsers is not.

3. The client is part of the media pipeline

It is tempting to treat the client as a thin interface over a powerful server. I think that leads to mediocre playback software.

The client knows things the server cannot infer reliably on its own:

• which tracks the user selected;


• whether the display supports HDR;


• which codecs the playback engine actually accepts;


• whether the application is local or remote;


• whether playback should prioritize original quality or bandwidth;


• how interruptions and resume behavior should feel.

This is one reason I focused Quven on native clients instead of treating every platform as the same web application in a different wrapper.

A browser, a native desktop player, and a mobile application have different playback capabilities and lifecycle constraints. They can share product semantics, but they should not be forced through an imaginary identical engine.

4. Transcoding is resource management, not just an FFmpeg command

Starting FFmpeg is easy. Operating it safely is the real work.

A media server has to prevent background analysis, concurrent streams, thumbnail generation, and metadata work from exhausting the host. Cancellation must stop the correct process. Failed work must release capacity. Hardware acceleration needs a software fallback. A remote viewer must not make the machine unusable for someone watching locally.

This changes how transcoding should be designed. Process limits, cancellation, timeouts, fallback behavior, and cleanup are part of the product contract, not implementation details.

The most dangerous failures are often not crashes. They are degraded states that still appear to work: excessive buffering, repeated retries, resource starvation, or a fallback loop that never stabilizes.

5. Remote playback changes the failure model

LAN playback has comparatively simple assumptions: low latency, stable addressing, and no managed intermediary.

Remote playback adds authentication, expiring tokens, reconnection, bandwidth variation, and partial failures between multiple systems.

A stream can still be playing while the control connection reconnects. A server can remain online while a media path expires. Retrying every failure can be worse than failing once because retries may multiply load during an already degraded session.

The lesson for me was to design remote playback as an explicit transport mode, not as "LAN playback through a longer URL."

6. Observability must not become surveillance

A media server needs enough diagnostics to explain failures involving codecs, processes, operating systems, and network paths.

At the same time, media libraries are private. File paths, account identifiers, tokens, and library contents should not casually leave the device just because a developer wants a convenient stack trace.

Useful diagnostics require deliberate boundaries:

• structured events instead of dumping arbitrary objects;


• scrubbing paths, emails, and tokens;


• recording technical classifications rather than media titles;


• allowing diagnostics to be disabled;


• treating telemetry failure as non-fatal.

The harder a system is to debug, the more tempting excessive logging becomes. Privacy therefore has to be designed before production failures happen.

7. Transparency matters when the product is proprietary

Quven is proprietary rather than open source. I know that creates a trust barrier in the self-hosted community.

My response cannot be to hide that distinction. It has to be clear about the boundary: media files and folder layouts remain on the user's hardware, LAN playback stays local, and self-managed remote access remains possible without using the managed relay.

I have also used AI coding agents extensively during implementation, testing, debugging, reviews, and documentation. I own the architecture and product decisions and personally validate releases, but presenting the project as if it were developed without substantial AI assistance would be misleading.

Both choices deserve direct disclosure because trust is earned through verifiable behavior, not positioning language.

Where the project is now

Quven currently has public Windows and macOS Server + Desktop bundles and Linux x64 and ARM64 server packages. The hosted web client supports managed remote access. Native Android and iPhone/iPad clients are built but are still completing their store release gates.

I am now looking for blunt technical and product feedback, particularly around this question:

What would a new proprietary self-hosted media server need to demonstrate before you would trust it with your library?