Learning to hear

The question

Generating music is a solved problem in the sense that matters here: a model runs on the lab’s own card, a text description goes in, a track comes out. That part worked from the first week. What did not work was everything that follows. Is this the genre it was asked for. Is it in time. Is it too loud. Is the second run better than the first, or does it only feel better because it is newer.

Those are questions a person answers by listening. The lab’s founder listens for a living, more or less, and has done for years. So for a while the arrangement was simple: the machine produced, the human judged, and the judgement was written down next to the run.

That arrangement broke as soon as the volume grew. There are 96 distinctly named pieces across the lab’s music folders today, and in the main generation folder alone 469 rendered audio files behind them. Nobody is going to listen to that twice. The question this page answers is what happened when the lab tried to build instruments that listen instead, and how far it got.

Three machines, three dialects

MiniMax Music 3 arrived in August, four days after its release, and runs on the lab’s own card in two separate ways: through a Python pipeline and through a dedicated instance of a node based workflow tool on its own port, deliberately isolated from the image generation stack that was already there. It wants a structured caption, written in sections, with global metadata and vocal details and arrangement spelled out. It produces whole songs with sung vocals, to a maximum of 360 seconds, with no ability to extend. On a quiet machine it needs 4.83 seconds of compute per second of music.

YuE2 arrived in September, about a week after its release. It is about nine times faster on the same hardware, and through the workflow tool it is faster than real time: 131 seconds of music in 67 seconds including model loading, which is 0.51 seconds of compute per second of music against the older model’s 4.83. It wants something entirely different. Its own documentation is explicit: the style is one comma separated line of about twenty five words, language then genre then voice then tempo then instruments then character, and the lyrics field holds only the words that are sung. There is no field for a negative prompt, no field for tempo, no field for a reference recording.

A pattern language is the third machine and does not generate anything. It is code that describes rhythm and melody, and it turns out to run without a browser and without an audio context at all, which means a pattern can be queried for a time range and written straight out as standard MIDI. From there it lands as an editable track in a digital audio workstation. This is the route where nothing is guessed: what comes out is exactly what was written.

The interesting part is not the list. It is what the lab did with it.

What was built

A genre meter. An audio classifier over 527 sound classes, run across a whole track, producing a score per class. On its own that is a number without meaning. What makes it an instrument is the calibration: it was run over 120 techno tracks and 57 reggae tracks from a bought, professionally tagged collection, and the tenth, fiftieth and ninetieth percentile of each axis were stored as a corridor. A new track is then not scored in the abstract but placed against that corridor.

A tempo tracker that separates drums. The first version measured the onset energy of the whole low end and called the strongest recurring point the first beat. The second version measures the kick band and the snare band separately, because the first version was wrong in a way that is invisible unless you already know the answer.

A key and chord reader. Chroma analysis with a standard profile for the key, a chord per bar for the harmony. Both were checked against test material with a known answer before being used on anything real.

A level meter. A worklet on the audio path, one meter per effect bus and one on the master, so that a set can be watched while it plays rather than reconstructed afterwards from complaints.

A conductor. A live performance system that holds a timeline, crossfades between blocks, and serves a control page. It exists because the off-the-shelf route sent updates only when an editor window happened to have focus, which meant updates were silently lost and the human had to start things by hand three times in a row.

What was measured

• 30 August. Four techno pieces and two roots reggae pieces came back from a rented card and missed their genre entirely. The founder’s verdict was immediate and the machine had nothing to say about it. This is the day the genre meter was commissioned.
• 30 August. The calibration run finished. On the techno axis, the bought collection sits at a median of 0.162 with a tenth percentile of 0.043. On the reggae axis the same collection sits at a median of 0.0005. The separation is a factor of about 320, which is what makes the corridor usable as a test rather than a decoration.
• 30 August. The length of the text in the lyrics channel controls the duration of the output, even for instrumentals where that text is never sung. 114 characters produced 52 and 140 seconds. 292 characters produced 156 and 169. 1,299 characters produced 254, 230 and 247 seconds across three seeds, a spread of 24 seconds. Structure tags alone did not do this.
• 19 September. The automatic first-beat detector was checked against a funk track and had landed on the snare, which put the kick on beats two and four. Measuring the kick band separately found the real first beat at 9.015 seconds and the tempo at 96.08.
• 19 September. The two generative models were compared on the same box. Clearing the machine of other work removes the older model’s spread upwards but does not move its floor of 4.83, so that figure is the hardware rather than the competition. The newer one produced 240 seconds of music in 153, and through the workflow tool reached half of real time. That retired the question of whether a rented card was needed for it.
• 19 September. The division of labour between the two was settled by listening, not by measurement, and it held: one model for groove and band feel, the other for emotion and voice. No instrument in this room can produce that sentence.
• 20 September. The bought collection was inventoried and turned out to be a document in its own right: 2,204 tracks, every one of them tagged with genre, tempo and key. The median tempo is exactly 128. 98 percent of the collection is in a minor key, 2,170 tracks out of 2,204. Techno and its subgenres account for more than two thirds. A calibration corridor is only as good as what it was calibrated on, so this is not trivia: it is the reason the genre meter can say what techno is and has nothing to say about a ballad.
• 20 September. The pattern language turns out to run without a browser and without an audio context at all. Querying a time range returns events with exact times, which can be written straight to standard MIDI. A pattern became 64 notes and landed as a track in a digital audio workstation, natively.
• 21 September. The level meter was checked against a completely separate recording tool measuring the same window. Both reported -13.4 dB. A self test at -6 and -9 dB came back exact.
• 21 September. The tempo tracker reported 122.93 for a track the lab had previously recorded as 123.399. The founder found the error by looking at the waveform: over 200 seconds the difference between the two figures is 1.56 beats, which is most of a bar. The machine had been confidently wrong for two days and nothing in the machine noticed.

What went wrong

The second model was fed the first model’s dialect for a fortnight. The structured caption that MiniMax wants is long: the one in use ran to 396 words with a section by section timeline. YuE2 was given the same thing, because it was there and it had worked before. Its documentation does not provide for that format at all. The results were judged as a property of the model: the voice was wrong, a female voice was asked for and a male voice came back, the emotional pieces fell flat. Some of that may well be true. None of it was ever tested against the format the authors actually specify, and the lab had been reading a model’s behaviour through its own bad habit. The corrective test, same text and a short style line with the voice at the front, is written down and not yet run. This is the single largest open item on the page.

A seed is not an address. Reproducing the YuE2 authors’ own demo from their own published seed produced a different piece: 211 seconds against their 203, with an envelope correlation of 0.14. The same holds for MiniMax, where changing the caption while keeping the seed gave a correlation of 0.004. A seed fixes a starting point on one machine with one configuration. It does not name a piece of music, and treating it as a citation is a mistake that looks like rigour.

The licence on the repackaged weights disagrees with the licence on the original. For one model the convenient repackaged copy carries a tag suggesting a permissive open licence and ships no licence file of its own, while the original repository carries a bespoke licence with a labelling obligation for public release. For the other model the direction is reversed: the copy shipped with the weights reads as noncommercial with no exception, while the project’s own separate licence file grants individual creators a waiver that the copy never mentions. Both were read at the source rather than taken from a summary, and the load bearing one was verified twice through independent channels. Anyone reasoning about either model from the convenient copy alone would reach the wrong conclusion in both directions.

Eleven pieces were deleted and are gone. A synchronisation script carried a delete flag and removed everything from the generation folder that did not exist on the other side. The other side was a rented machine that had already been shut down. About two hours of generation, unrecoverable. The script is now additive, and its cleanup mode requires a dry run and a typed confirmation.

Five variables were changed at once. When the genre finally landed, the caption format, the caption length, the device names in the text, the machine and the duration had all changed together. Which of them carried the result was never isolated. It is written down as unresolved rather than claimed as a finding.

A whole album variant was wrong and nobody had measured it. Eight entries were written, five of them generated, and then the variant was rejected outright by the only listener who mattered, genre and lyrics together. The instruments at that point could confirm the genre was as requested. They had nothing to say about whether it was good.

Three separate reasons for silence in one evening. The sound card was on the wrong profile, one mixer channel was muted, and the workstation had taken exclusive control of the audio device. The volume control appeared to do nothing because the listening path ran over an optical output the analogue controls do not touch. None of this was audible as a fault. It was audible as nothing at all.

The verdict

The instruments work and they are honest, in the narrow sense that each of them was checked against something outside itself before being trusted: the genre meter against a bought collection, the tempo tracker against a track whose first beat had been located by a second, independent method, the level meter against a second recorder, the chord reader against test chords with a known answer. That is the standard this lab holds itself to and it was met here.

What they do not do is judge. The genre meter can say that a piece is not techno. It cannot say that a piece is boring. The tempo tracker can say the grid is wrong. It cannot say the arrangement drags. Every real verdict in this room over the last two months came from a person listening. There has been exactly one occasion where an instrument and the listener disagreed outright, and the listener was right. That is a thin record to draw conclusions from, and it points at something more awkward than a scoreboard: in every other case there was no instrument that could have disagreed at all. The person was not winning an argument. The person was the only witness.

There is also a lesson here that has nothing to do with audio. The most expensive mistake of these two months was not a bug and not a wrong number. It was a habit: a caption format that worked for one model, applied to a second model without checking whether that model wanted it, and then the result read as a fact about the model rather than about the lab. Nothing in the instruments catches that. It was caught by going back and reading the documentation of a tool that had already been in use for a week.

So the honest position is this: the measuring instruments have removed a class of error, the class where something is confidently wrong and nothing notices. They have not removed the listener, and there is currently no plan that would. Two questions stay open. Whether the corridors can be extended from genre towards something closer to taste, without turning into a machine that simply agrees with whoever calibrated it. And whether the second model, addressed in its own language for once, sounds like a different instrument than the one this lab spent a fortnight judging.