Hi all, I'm a UX architect with a past as a web developer.
I started digging for dubby, minimal and ambient electronic records many years ago, and as you may imagine I literally spent hours on Discogs.
For a crate digger like myself the most powerful thing is the credits: which label a release is on, who engineered, mastered or remixed something. The connections between these things is what allows you to discover new stuff.
Doing that on Discogs is definitely possible, but it comes with a lot of friction.
Dub Digger is a personal, side project and it's my attempt at a tool to remove that friction. It's essentially the tool I would use to dig out records.
You search an artist, label or release and follow the credits from one name to the next.
I needed something focussed on a core of specific genres, so I built a curated database centered on techno, deep, ambient and experimental labels and artists, based on Discogs data.
Each artist and label has a relevance grade based on how close each entity is to the core of the database, from very high down to extremely low.
On top of that I added a manual curation layer to mitigate the issues and inconsistencies the automated rules were showing. I am also constantly adding manual overrides, to promote or demote specific artists I think should sit at a different level of relevance from where the raw Discogs data places them.
It currently has about 305k artists, 109k labels and 1M releases.
One interesting thing is that I tried to avoid using user data as much as possible and I did my best to make it truly accessible.
How I built it may be of interest
Source: Discogs' monthly XML dumps (CC0). The releases file is over 100 GB unpacked. No calls to the Discogs API to keep it light and with as few dependencies as possible.
Ingest: a Node/TypeScript script streams the XML, keeps only the fields the tool needs and goes through about 19M releases.
Two stages:
Stage 1 - it takes records tagged dub techno, deep techno, dub, ambient or minimal, with some rules added on top (for example dub only if tagged along with electronic , otherwise you get the whole reggae catalogue). An artist is in the core only if at least 10% of their releases have those tags.
Labels work in the same way. A label belongs to the core if at least half of the artists on it are core artists.
Stage 2 - one step out: releases with a core artist tagged on them, and releases on core labels. It stops there, as I was afraid a further step would pull in most of techno.
Precomputing: relevance grades, counts, rankings and the search index are all calculated offline. The output is one read-only SQLite file (about 1.3 GB).
Serving: Astro with server-side rendering, on a small nginx VPS, reading the SQLite file directly. No database server and no live API calls from the server. Covers and 30-second previews come from Apple: I match them offline with the iTunes Search API, and the browser loads them straight from Apple (the previews only when the user press play).
Updates: I re-run the pipeline on a new monthly dump and reupload the file.
Curation: The rules worked well only to an extent. The data were patchy so I had to add my personal curation layer. As a starting point I defined nine "traditions/influences" in the config, among them roots dub, reggae, Detroit techno, afrobeat, dubstep and UK garage. Belonging to one of them raises an artist to a fixed grade. All of this is paired with manual overrides.
Client: no accounts, cookies or tracking. “My List” uses localStorage. Aims for WCAG 2.1 AA ( I checked it myself, no certifications from any third parties).
It’s a beta and a passion project. And of course is far from perfect. I’d love some (kind, please!) feedback on how it feels to navigate.
Wild. I just happened to be searching for CC0 Dub Techno resources and your post is 6 hours old.
It would be good to have copyright information of the artists. If any provide their music under a libre/free license (CC0, CC-BY, CC-BY-SA) that would make it attractive to developers to use.
Some folks also want to search for artists to invite to shows, so being able to filter by location might be beneficial.
Great to hear you found this post useful. Also the timing factor is pretty cool.
Just to clarify, the CC0 part is about the Discogs data (credits, labels, releases), not the music itself. Dub Digger doesn't host any audio, the previews come from Apple.
Your points are really good and I agree it would be super useful finding a way to add those info. So I checked the dump straight away.
Long story short, there is a field for who owns the copyright, but nothing about licences.
CC licencing only shows up in a patchy way in free-text fields. It means it's not standardised in any way (no consistency, no structure...). That makes it impossible to filter on it decently.
Location is the same story: there's no location field for artists, only free text in some profiles.
Release country is there, the problem is that it's the country of release and it's not necessarily linked to where the artist is based.
Yeah, well, one reason the copyright information would be useful is because it's hard to come by in a standard way.
Soundcloud occasionally has copyright information tied to an individual track. Bandcamp has a structured field that lists the copyright.
A long time ago, I scraped Bandcamp metadata and put it in a SQLlite file [0]. From that you can filter on bands by copyright (among other things).
Maybe a stripped down LLM could handle the free form text and figure out licensing issues. It looks like your list is pretty reasonably sized, so that might not be such a big job.
Hi all, I'm a UX architect with a past as a web developer.
I started digging for dubby, minimal and ambient electronic records many years ago, and as you may imagine I literally spent hours on Discogs.
For a crate digger like myself the most powerful thing is the credits: which label a release is on, who engineered, mastered or remixed something. The connections between these things is what allows you to discover new stuff.
Doing that on Discogs is definitely possible, but it comes with a lot of friction.
Dub Digger is a personal, side project and it's my attempt at a tool to remove that friction. It's essentially the tool I would use to dig out records.
You search an artist, label or release and follow the credits from one name to the next. I needed something focussed on a core of specific genres, so I built a curated database centered on techno, deep, ambient and experimental labels and artists, based on Discogs data. Each artist and label has a relevance grade based on how close each entity is to the core of the database, from very high down to extremely low.
On top of that I added a manual curation layer to mitigate the issues and inconsistencies the automated rules were showing. I am also constantly adding manual overrides, to promote or demote specific artists I think should sit at a different level of relevance from where the raw Discogs data places them.
It currently has about 305k artists, 109k labels and 1M releases.
One interesting thing is that I tried to avoid using user data as much as possible and I did my best to make it truly accessible.
How I built it may be of interest
Source: Discogs' monthly XML dumps (CC0). The releases file is over 100 GB unpacked. No calls to the Discogs API to keep it light and with as few dependencies as possible.
Ingest: a Node/TypeScript script streams the XML, keeps only the fields the tool needs and goes through about 19M releases. Two stages: Stage 1 - it takes records tagged dub techno, deep techno, dub, ambient or minimal, with some rules added on top (for example dub only if tagged along with electronic , otherwise you get the whole reggae catalogue). An artist is in the core only if at least 10% of their releases have those tags. Labels work in the same way. A label belongs to the core if at least half of the artists on it are core artists. Stage 2 - one step out: releases with a core artist tagged on them, and releases on core labels. It stops there, as I was afraid a further step would pull in most of techno.
Precomputing: relevance grades, counts, rankings and the search index are all calculated offline. The output is one read-only SQLite file (about 1.3 GB).
Serving: Astro with server-side rendering, on a small nginx VPS, reading the SQLite file directly. No database server and no live API calls from the server. Covers and 30-second previews come from Apple: I match them offline with the iTunes Search API, and the browser loads them straight from Apple (the previews only when the user press play).
Updates: I re-run the pipeline on a new monthly dump and reupload the file.
Curation: The rules worked well only to an extent. The data were patchy so I had to add my personal curation layer. As a starting point I defined nine "traditions/influences" in the config, among them roots dub, reggae, Detroit techno, afrobeat, dubstep and UK garage. Belonging to one of them raises an artist to a fixed grade. All of this is paired with manual overrides.
Client: no accounts, cookies or tracking. “My List” uses localStorage. Aims for WCAG 2.1 AA ( I checked it myself, no certifications from any third parties).
It’s a beta and a passion project. And of course is far from perfect. I’d love some (kind, please!) feedback on how it feels to navigate.
Cheers
Wild. I just happened to be searching for CC0 Dub Techno resources and your post is 6 hours old.
It would be good to have copyright information of the artists. If any provide their music under a libre/free license (CC0, CC-BY, CC-BY-SA) that would make it attractive to developers to use.
Some folks also want to search for artists to invite to shows, so being able to filter by location might be beneficial.
Great to hear you found this post useful. Also the timing factor is pretty cool.
Just to clarify, the CC0 part is about the Discogs data (credits, labels, releases), not the music itself. Dub Digger doesn't host any audio, the previews come from Apple.
Your points are really good and I agree it would be super useful finding a way to add those info. So I checked the dump straight away.
Long story short, there is a field for who owns the copyright, but nothing about licences.
CC licencing only shows up in a patchy way in free-text fields. It means it's not standardised in any way (no consistency, no structure...). That makes it impossible to filter on it decently.
Location is the same story: there's no location field for artists, only free text in some profiles.
Release country is there, the problem is that it's the country of release and it's not necessarily linked to where the artist is based.
I'm keeping on digging the issue, though!
Yeah, well, one reason the copyright information would be useful is because it's hard to come by in a standard way.
Soundcloud occasionally has copyright information tied to an individual track. Bandcamp has a structured field that lists the copyright.
A long time ago, I scraped Bandcamp metadata and put it in a SQLlite file [0]. From that you can filter on bands by copyright (among other things).
Maybe a stripped down LLM could handle the free form text and figure out licensing issues. It looks like your list is pretty reasonably sized, so that might not be such a big job.
[0] https://archive.org/details/thisonetime