peaseagee 1 day ago

Congrats on training on open and public domain recordings. Thanks for proving this kind of work can be done in a more ethical way.

  • piltdownman 1 day ago

    TBF it's by far the best corpus for this use case anyway.

    Classical Music as an academic concern is very pedantic in the best way, and has the lions share of the research hours and funding compared to any other genre. As a result the best 1:1 approximations of multi-instrument Audio into multi-channel MIDI files is done simply by OCR of the established Gesamtausgabe in each case. Computer based scoring and playback has been solid since Finale/Sibelius/Score came on the market, and democratized at a wide consumer level somewhere between Amiga mod-trackers and Macintosh Quicktime 2.0.

gw32 1 day ago

This is fantastic. I love the commitment to train only on PD recordings. In my opinion, that is an essential to maintaining the favor of musicians - building technology not to take from their work and replace them, but rather to celebrate human-written music by making it accessible.

This is why I believe (AMT, OMR) and MIR-adjacent projects to be so much more meaningful than generative models (e.g. Suno). I'm looking forward to hearing more from kobi music - congrats on the launch!

  • xp84 5 hours ago

    I think it’s very cool that the author did that, and preferable. But also if I built two opaque black rooms, and two masked teams entered the rooms and trained equally powerful AI models, one using someone’s whole Napster/KaZaA collection and the other using a ton of public domain music, both models now exist and do the same thing. I see only philosophical differences, no differences that change anyone’s outcomes.

mewse-hn 20 hours ago

Impressive, I ran nine inch nails - march of the pigs through this, which is an extremely messy, noisy song with random tempo changes, and it came up with a remarkable accurate 26 KB(!!) midi file that sounds like the original song using only 6 instruments. Crazy.

lachlan_gray 22 hours ago

Wow! Gave it a try on this https://www.youtube.com/watch?v=8XD4m_l8bgY

And it pretty much nailed it note for note, including drums. Only fault is some mix up of instruments, e.g. electric bass turns into electric guitar in higher registers which is totally excusable.

epiccoleman 1 day ago

That is awesome! I'll definitely have to check it out. I was very impressed with muscriptor when I used it in a recent project (a conference talk I gave). It was an AI talk with a musical component, and the final "scene" in the talk involved my own guitar playing. I used Muscriptor to transcribe that playing for categorization as part of a "guitar solo generator" that recut a new solo out of my stock phrases each time. Really fun little demo for a bigger project.

Anyway, it would be awesome to be able to do those sorts of transcriptions better and faster - so I'll definitely give this a whirl. Maybe I can dig up the pipeline scripts from processing my licks for that talk and see how dettato slots in.

thih9 1 day ago

Congrats on the launch!

The top illustration looks like an Angry Birds character with an ear. Fine for a one time blog post but if you plan to reuse it then you may want to change the design.

keymasta 18 hours ago

This is encouraging! I've been watching the space for some number of years. I remember rigging up a system that used librosa which has some nice built-in algorithms which got me pretty close!

I've got a use-case for this, but I'm wondering how "stuck" in this format this is. By format I mean, basically, python runtime, etc. Would it be possible to extract into something more usable by a consumer, like into a standalone rust module? Can anyone poitn me to how this would be easy?

AlexITC 10 hours ago

Looks amazing! I gave it a try with a music track from a game while also trying the 2 linked apps that produce sheet music from audio. It worked surprisingly well and way better than one of those apps.

alin23 1 day ago

What a coincidence! I was just in the process of integrating MuScriptor on a personal macOS app [0] I work on and off for helping me practice my guitar, kaval and trumpet skills.

I'm so glad to see a smaller model being so accurate, MuScriptor is especially bad on the Romanian kaval, can't wait to try this in-app. I do it with already separated stems through BS-RoFormer so I'm hoping this can improve accuracy a bit.

[0] https://github.com/FuzzyIdeas/Muskey

rogerdickey 1 day ago

all the examples seem to be classical music or largely instrumental stuff. how well does it work for things like pop music, rap?

  • hidude562 23 hours ago

    Rap I haven't tried. For pop it works decent, but vocals can get mushy

    • xp84 5 hours ago

      Can you elaborate? How can vocals get mushy in a MIDI? What do the vocals become?

  • lachlan_gray 22 hours ago

    I tried it on some pretty aggressive jazz and it was solid, I'm guessing it'll do well on other stuff

dekhn 1 day ago

I tried this on a Phish guitar track that I have and while GPU failed with an error, CPU eventually finished and it did a pretty good job! The recording is 45 minutes long, so getting GPU working is my next interest.

Unfortunately, Phish live recordings are copyrighted, so it's unlikely you can put them in the training corpus. Demonstrating that you can deconvolute and transcribe multiple different instruments with highly overlapping frequencies and sounds faster than an expert human would be a great demonstrate of the capabilities.

  • tough 1 day ago

    could you have personal use fine-tunes for copyrighted material as long as you don't publish it

    • dekhn 22 hours ago

      I think you need labelled data to fine-tune this model, which is prohibitevly expensive.

hdb2 1 day ago

I just signed up and fed it a piano solo from a Dixie Dregs tune, and it nailed it. this is very impressive for players like myself to transcribe things very, very quickly.

dimmuborgir 1 day ago

Is there a way to force it to a single instrument? I uploaded audio files of pure piano music and the output midi files are split into piano, guitar and drum parts.

  • hidude562 23 hours ago

    Nope, you'd have to edit the instruments in-post

  • ldkjhg 21 hours ago

    I have local program that you can use to output to a single track. It was done a month ago because I needed to extract different compositions. It can use various models, and if this new model works well, then I'll add to it. I've fed it real recordings and also the Doom original tracks, and they came out ok. Piano was an easy task for it.

    https://github.com/ld9999999999/AudioToMidi

oever 1 day ago

Audio-to-MIDI sounds very ambitious. How do you verify that the midi fits the audio? Does it result in useful beat detection? E.g. I'd very much like to have a way to map audio offset in a recording to labels in sheet music. But even beat detection in fairly standard jazz recordings fails often.

Does the model know when a piece (whole piece, not a section) is 3/4 or 4/4 th and can it give the onsets of all measures? Or all the midi notes just based on time and not organized rhythmically?

  • gunalx 1 day ago

    Its definetly a problem where generating the trivial version of synthetic data is easy though.

  • sed-i 6 hours ago

    This is very close to what I'm working on! I'm working on ensomi, a system that helps human find beats interactively and incrementally. Beat tracking has a long history in MIR, and there is some fundamental difficulties in 100% reconstructing the tempo map. There has been tempomap AIs but a very common problem is they confidently give out bad estimations even provided human hints. If you're interested in any detail, feel free to ask me! Our model is private for now but will publish as open source in the future.

pier25 1 day ago

Very cool. This is the sort of stuff I want to see from AI.

jsphweid 1 day ago

I've been out of the loop since Magenta's MT3 model. This feels a lot better and significantly smaller. Very impressive, dude. And thank you!

bryzaguy 19 hours ago

> we trained dettato on a mix of synthetic instrument recordings

can you go into more detail here? how did you make synthetic instrument recordings? with virtual instruments or generative AI?

  • hidude562 17 hours ago

    Using virtual instruments, also mixed with public domain recordings (like URMP), no gen ai. Using the ~30 sound banks I have I use a method I published (check out the soundfont babies blog) for mixing them together. For cc recordings I split and stretch them to play target midi notes.

mycall 1 day ago

I'm still waiting for someone to convert beatboxing to midi in realtime. That would be a game changer for that art.

  • maxgraf96 1 day ago

    You should check out https://vochlea.com/ :)

    • mycall 1 day ago

      > Trained on your own percussive sounds

      This is probably where I would find it limiting range of sounds can be huge, much more than just 6 sounds (10000+) and not all of them percussive. Worth an eval though, thanks!

randyrand 17 hours ago

Shows we still have a ways to go. In the Grandioso of Stars and Stripes (great song!) it completely misses the flute part.

That said, this is awesome.

locusofself 1 day ago

Very cool, I have been working on a DAW for several months that is getting close to a beta release. I'd love to incorporate this.

As a guitarist who can't play keys well, I'd love to use something like this to help me transcribe string arrangements.

  • bolangi 17 hours ago

    What's your DAW about, with so many others? I ask as a DAW author myself, written in, ohmygod, perl :-). Mine is in at https://github.com/bolangi/nama, offering an accessible, terminal interface with a simple Tk GUI as well.

    • locusofself 16 hours ago

      I wanted to see just how performant I could make a DAW for Apple Silicon, particularly to see if I can track reliably through plugins at ultra low latency without external DSP. I have used REAPER for 17+ years and found despite it's reputation for being really "light on CPU" it is in fact not, and so I switched to Logic Pro for a while, and that was fine but also bugged me in some ways. So its like a super light weight, REAPER-inspired DAW for M-series macs only, primarily for recording real instruments/ voices through plugins and really good editing features.

      • bolangi 13 hours ago

        So, you take audio streams from, say, one or more microphones, run them through plugins, and record them to disk? Is that also with live monitoring?

        What do you use for the audio engine? I see that ardour, developed for linux, also runs on MacOS, although it is less tested on your platform.

        Probably you're well along already, and it may not be the time to discuss architecture, but...

        Ecasound, which I use for audio processing, compiles on MacOS. It is a very efficient library for audio processing, however it is limited to internal, LADSPA and LV2 plugins. If you use the JACK audio framework, there are hosts such as Carla that accommodate VST and VST3 plugins, however I can't say how well tested they are on MacOS.

        When you include MIDI, automation, etc. there is a huge feature set. Probably you can get away with the minimum necessary for your use case.

        Interested to hear whatever you'd like to share.

        Cheers and good luck!

    • xp84 5 hours ago

      That’s awesome that you did it in Perl! I didn’t expect that.

raffraffraff 1 day ago

Interesting. Aside from conversion of tracks to midi, it looks like i could run my music collection through this and use the output to create a recommendation system (along with existing metadata and lyrics)

SoleilAbsolu 1 day ago

Curious what's the performance for both note duration, and note pitch? How does it do for tunings/temperaments outside 12TET? Very refreshing to not violate copyright in this day and age amongst all the greedy grabbers!

  • hidude562 23 hours ago

    The full benchmarks can be checked out on the GitHub. The strongest model point is onset - which is similar to pitch. The weakest being 'offset', which is length. Both scores still edge out Mirelo's closed source model, and significantly beat MuScriptor.

    I haven't tried anything outside of 12TET, but probably not good (it predicts in 12TET). Alternative tunings should work fine (it re-tunes to a440 for the model).

gpugreg 1 day ago

Nice! Worked surprisingly well.

You might want to implement deduplication for your website so impatient users do not convert the same mp3 multiple times (at multiple times the price) when they click "run audio to midi" and the server takes a bit longer to respond.

  • cocodill 1 day ago

    I guess impatient users could run audio to midi local.

    • xp84 5 hours ago

      Sure, they can, but the server still needs to be protected. If nothing else, the UI hopefully prevents many clicks causing duplicate requests. A few people’s “rageclicks” can turn into a DDOS.

ch_sm 1 day ago

Just tried it with some home-recorded songs (pop/rock) — this works great!

gregzem 3 days ago

Congrats with SOTA model. How good is it in recognition of the instruments which are overlapping by frequencies? Does is provide note offset and velocity in the transcribed MIDI? Thanks!

  • hidude562 23 hours ago

    I would say it performs okay, but not perfect by any means. In music it tends to flip on and off similar sounding tracks.

    Yes! Offset and velocity both are provided.

  • thomspoon 23 hours ago

    I started playing with it to make a song to GP5 converter and it works pretty well on solo instruments, but definitely gets confused when multiple instruments are played at the same time. Detatto does provide offset and velocity in the midi.

JodieBenitez 1 day ago

I need this in Bitwig !

  • thatmf 1 day ago

    my exact thought. hmm. weekend project?

  • pier25 1 day ago

    yeah I was thinking that DAWs should be implementing this

simedw 1 day ago

Very cool, I wonder if my MIDI generation model would improve if I trained it on the MIDI files your model generates.

awinter-py 1 day ago

at long last we have solved karaoke

purrkey 8 hours ago

cool! is it fast enough for near-real-time?

eduardoagarcia 1 day ago

super interesting, I'll have to check this out tonight when I get home!

pikann22 1 day ago

crazy how a 20m param model actually beats the 1,3b muscriptor. awesome work!

IAmGraydon 1 day ago

Excellent! I'll try this tonight. The timing couldn't be better as I've been researching audio->MIDI models for some personal projects recently.

th1nhng0 1 day ago

just in time when I need it. Thank you so much

hongyong 1 day ago

nice, this is what i've been looking for

smrtinsert 1 day ago

Very excited to try this. Been looking for something exactly like this for my composition hobby