Music data platform

I built this platform to collect public music data. The sources are charts and catalogs, such as Billboard and MusicBrainz, and playlists from music platforms. The code is on GitHub, with all rights reserved.

The matching is strict. Each record is linked to a MusicBrainz recording by ISRC, the standard ID code for a recording. Two of the ways of making that link carry a minimum confidence score for each platform, and the rule for changing one is to check it against a reviewed sample first. Unclear matches wait for review.

One method on one platform fell short of the target of 97 right matches in every 100, so its minimum score is set to 1.01. The scale stops at 1.0, which means nothing from that method gets through until a reviewed sample says otherwise.

Playlists get the same caution. Some sources show only the top of a list, so when a song isn't in that slice, it might have been dropped, or it might just be further down where nobody looked. A partial look can show a song is there, but it can't show it's gone. A removal goes on the record only after a full read comes back without the song, and it's logged as happening somewhere between two reads. A day nobody looked at stays unknown instead of counting as zero.

How it's built

A Python service does the collecting. Every request goes through one path that logs it, and every stored record says which request and which run produced it. If a collector tries to open its own network connection, a Python audit hook stops it before a byte goes out.

Each page of results also has to balance. What the collector saw must equal what it kept, plus what it rejected, plus what it skipped for a reason declared up front. A podcast episode tucked into a playlist is a declared skip, and so is an air break in a radio station's play log. From that log, the DJs' names and comments never get stored. Any other reason for a skip gets refused.

Fields it doesn't recognize are kept to one side, so if a provider changes its format, that shows up instead of rows quietly going missing.

Every join and every total happens in dbt, so the model graph shows where two sources meet. A retry reads the same frozen inputs as the first try and picks up where the work stopped. Each row the API serves carries flags for what its data may be used for.

A TypeScript control plane manages the jobs and retries, and it shows each run in an operator console. It also generates a typed SDK for the served tables, and an R package reads them for analysis.

Half the sources in the platform's catalog are built and ship switched off, and the README asks anyone running it to check each provider's terms before turning one on. The whole stack starts on a laptop from one script, loaded with a synthetic week: seven fixed days of made-up observations, with no network calls.

Stack

Languages

TypeScriptRPython

Data

dbtPostgres

Infrastructure

DockerFly.io