Author Topic: Best framework to encode entire tree of music recursively?  (Read 3196 times)

0 Members and 1 Guest are viewing this topic.

Offline Surly73

  • Thread Starter
  • Posts: 425
Best framework to encode entire tree of music recursively?
« on: Thu, 05 January 2012, 13:36:51 »
Yes, some place like hydrogenaudio probably has more experty experts, but I'm not a regular participant there and I bet there's enough HA types here to give me some ideas.

I'm finally looking at doing a multiple format music library scenario that's easier to manage, with lossless FLACs where possible as well as high quality MP3s (LAME -V 2 probably).  Right now all of my ripped CDs are in a directory structure compressed using an older version LAME with -V 4 I think it was.

I've used all2lame on a number of occasions to convert various formats into LAME MP3s in smaller quantities, like an album at a time, and it does a fine job.  My issue is that I'd like to have a utility which could do what I need recursively that I could re-encode my entire lossless collection in an automated fashion while I'm sleeping.

e.g.:

Say I have a source directory structure like D:\AUDIO\FLAC\%Artist%\%Album%\%Track---%.FLAC, I would like a hands-off, walk-away utility that I could point to D:\AUDIO\FLAC\ and it would process all files and duplicate the structure into D:\AUDIO\LAMEV2\%Artist%\%Album%\%Track---%.mp3.  My preference would be that it would preserve the file tree and not make up it's own new directories based on the tags (just in case tags are missing or inaccurate).  Something that would automatically launch multiple LAMEs on my quad-core processor would be delightful too.

I know that foobar and mediamonkey can do re-encodes, but brief playing didn't seem to display the level of whole-collection re-encoding functionality I'm looking for.

In the end I'll also need to look into having multiple libraries within a tool like mediamonkey.  I thought I read that MM was good at understanding that you might have the same music in multiple formats and dealing with that, but I haven't seen that behaviour in the version I'm running now.

Thanks in advance,

Offline cheebs

  • Posts: 83
Best framework to encode entire tree of music recursively?
« Reply #1 on: Thu, 05 January 2012, 14:06:06 »
Why not just write a shell script for this ?

Offline Surly73

  • Thread Starter
  • Posts: 425
Best framework to encode entire tree of music recursively?
« Reply #2 on: Thu, 05 January 2012, 14:30:20 »
Quote from: cheebs;483968
Why not just write a shell script for this ?

Well, valid question.  My top-of-head answers:

1/ I may or may not have all the skills and I don't have the time or entertainment value from tinkering with scripts.  This isn't a hobby or profession I just want to listen to my music.  I have a family, home, commute and day job to take care of and I'm not looking for more time sinks.

2/ someone else has probably already made a tool, I just don't know what it is or I'm not configuring what I already have properly, or just need a foobar add-in or....

3/ I wouldn't know how to manage the launch of n threads in parallel and manage that throughout the process since LAME isn't multi-threaded last time I checked.

4/ I'd be beating my head against the desk over the handling of all kinds of corner cases like filenames with "!" in it, paths >255 and God knows what else for a long time trying to get my tool perfect.  See #1 and #2.

5/ Windows platform - scripting's not quite as fun

Offline IvanIvanovich

  • Mr. Silk Underwear
  • Posts: 8199
  • Location: USA
Best framework to encode entire tree of music recursively?
« Reply #3 on: Thu, 05 January 2012, 16:01:41 »
Take a look at dbpoweramp batch converter. Pretty sure it does exactly what you want, as long as the right options are chosen.

Offline godly_music

  • Posts: 255
Best framework to encode entire tree of music recursively?
« Reply #4 on: Fri, 06 January 2012, 03:22:44 »
You don't need to do this recursively or in fact touch the command line at all.

I would recommend trying it with foobar2000. Just point the program at your encoders once (lame.exe and flac.exe and whatever else). Then you can pull the entire directory structure into a playlist and tell it to encode it all, given a certain template.

Output Path is set to specify folder. You put in: D:\AUDIO\LAMEV2\

Output Style is set to convert each track to an individual file, with a pattern like: [%artist%\][%album%\][Track %tracknumber%]

This does not account for multiple disc albums, though. I would recommend instead to use: [%artist%\][%album%\][CD %discnumber% ][Track %tracknumber%]

It's really easy to modify as long as you know the different %tags%.

Offline Surly73

  • Thread Starter
  • Posts: 425
Best framework to encode entire tree of music recursively?
« Reply #5 on: Fri, 06 January 2012, 07:36:10 »
Quote from: godly_music;484490
You don't need to do this recursively or in fact touch the command line at all.

I would recommend trying it with foobar2000. Just point the program at your encoders once (lame.exe and flac.exe and whatever else). Then you can pull the entire directory structure into a playlist and tell it to encode it all, given a certain template.

Output Path is set to specify folder. You put in: D:\AUDIO\LAMEV2\

Output Style is set to convert each track to an individual file, with a pattern like: [%artist%\][%album%\][Track %tracknumber%]

This does not account for multiple disc albums, though. I would recommend instead to use: [%artist%\][%album%\][CD %discnumber% ][Track %tracknumber%]

It's really easy to modify as long as you know the different %tags%.


OK, I'll take another look at foobar's re-encoding methods.  This looks like it builds the output directory structure based on the tags instead of the input structure which is one of the things I didn't want but I'll play with it.  I have some cases where I have some old LPs I've captured myself - like some old traditional Christmas music which was my grandfather's - and they all might be by "Various Artists" but I keep them in a unique directory tree for several reasons instead of being lumped in with all of the other "Various Artists" on disc (I know they would be in the library view)

Also - how well would this handle tagging album art into the destination files and that sort of thing.  Now that I write this, I'm not sure if FLAC tags can store album art in every track.  I've usually seen the album art as a standalone JPG in the directory with the FLAC files and most software seems to handle it properly when transcoding or importing.   More discovery required I guess.  The point is that I guess I want to be sure that I just need to be a fusspot with all of the tags and metadata once on the FLAC side and the transcoding using foobar will preserve everything possible (where it translates from FLAC to ID3 tags of course).  I guess LAME would replaygain each track, and then I would just tell foobar to do album replaygain on the entire library using tags to build albums and adjust the tags.

The reason I want the library re-encoding to handle as much as possible is that I don't want to touch or tweak the destination files or tags at all if possible beyond telling a tool to apply album replaygain if it didn't at encode time - that is also recursive and hands-off.  I might end up with FLAC for home, LAME -V2 for work listening where I have quality DAC and headphone amps, LAME -V4 or -V6 for car use, and who knows what - some lower bitrate AAC for a mini-portable where space is very limited and the DAC/amps suck anyways.  If I'm going to the trouble to re-rip all of my music into lossless one of the goals is to build the flexibility to produce new copies of my library with whatever encoding I want for a particular purpose.  This encludes re-encoding simply because there's been some kind of huge codec breakthrough.  I ripped most of my music back in the 2003 time frame with LAME best practices at the time.  LAME is even better now, and I feel somewhat inclined to re-encode just because of that.  It would be great if I could just point a tool at FLACs, set my new settings and click "go".  I'm sure that desire's not unique to me, of course.

I don't have a whole lot of reason to have a linux box around any more.  I suppose otherwise I might start building some scripts and a makefile so I could just 'make' from my FLAC directory and have everything do it's thing.  I still use it for NAS and tape backup but a lot of my other uses have simply been replaced by other technology or are no longer needed.  It's still an Athlon 2600XP while my windows box is an i5-2500K.  It's pretty clear to me where I want to do my encoding, retagging using mp3tag etc...
« Last Edit: Fri, 06 January 2012, 07:48:21 by Surly73 »

Offline alaricljs

  • I be WOT'ing all day...
  • ** Moderator Emeritus
  • Posts: 3715
  • Location: NE US
Best framework to encode entire tree of music recursively?
« Reply #6 on: Fri, 06 January 2012, 08:32:08 »
You can replaygain the FLAC since it's just more tags in the file... I don't know what LAME might do with it, but I use Vorbis so it just copies the tags... FLAC (and Vorbis) use the same tag structure which does allow for embedded images.  I don't use that feature tho.

If you were running Linux I could give you my transcoding script which duplicates the directory structure from the source to the destination.  I also do similar to you and have Various Artists as well as Christmas and Soundtracks.
Filco w/ Imsto thick PBT
Ducky 1087XM PCB+Plate, w/ Matias "Quiet Click" spring-swapped w/ XM Greens

Offline Surly73

  • Thread Starter
  • Posts: 425
Best framework to encode entire tree of music recursively?
« Reply #7 on: Fri, 06 January 2012, 09:57:14 »
Quote from: ripster;484600
Keep your music in FLAC and have MediaMonkey reencode on the fly as you load onto portable devices.

My collection doesn't work that way, unfortunately.  I don't manage my portable devices (ipods and blackberries mostly) with any manager I just drag and drop folders.  For work whatever I have on me lives on a Truecrypt SD card for various reasons too long to get into.  If MM could understand VolIDs and paths as "portable devices" I might be able to use that method though....

Offline godly_music

  • Posts: 255
Best framework to encode entire tree of music recursively?
« Reply #8 on: Fri, 06 January 2012, 11:03:08 »
Quote
OK, I'll take another look at foobar's re-encoding methods.  This looks like it builds the output directory structure based on the tags instead of the input structure which is one of the things I didn't want but I'll play with it.  I have some cases where I have some old LPs I've captured myself - like some old traditional Christmas music which was my grandfather's - and they all might be by "Various Artists" but I keep them in a unique directory tree for several reasons instead of being lumped in with all of the other "Various Artists" on disc (I know they would be in the library view)


Yes, it builds directories based on tags. One other option would be to select 'Source Track Folder' and remove the artist and album folders from Output Style. Then you would have to use a tool or some commandline magic to cut and paste those mp3s and create an exact folder tree duplicate. Probably not that hard if you know how. For the unique directory tree, you can select all those albums you don't want to put together with Various Artists, and put something else in their Album Artist tag. That'll still put them all into the same artist folder.

Quote
Also - how well would this handle tagging album art into the destination files and that sort of thing.  Now that I write this, I'm not sure if FLAC tags can store album art in every track.  I've usually seen the album art as a standalone JPG in the directory with the FLAC files and most software seems to handle it properly when transcoding or importing.   More discovery required I guess.  The point is that I guess I want to be sure that I just need to be a fusspot with all of the tags and metadata once on the FLAC side and the transcoding using foobar will preserve everything possible (where it translates from FLAC to ID3 tags of course).  I guess LAME would replaygain each track, and then I would just tell foobar to do album replaygain on the entire library using tags to build albums and adjust the tags.


Tags are always carried over when the target file is able to store them. And I would always recommend to keep album art in separate files and to let foobar2000 just work through your music collection, either the FLAC one or the mp3 one later, and have the ReplayGain info written into the tags instead of having the volume level altered during encode, for example. But that's just my personal sense of 'order', since I have various devices that should play my files. But since I don't have any album art, someone else might be able to expand on this better.

Quote
The reason I want the library re-encoding to handle as much as possible is that I don't want to touch or tweak the destination files or tags at all if possible beyond telling a tool to apply album replaygain if it didn't at encode time - that is also recursive and hands-off.  I might end up with FLAC for home, LAME -V2 for work listening where I have quality DAC and headphone amps, LAME -V4 or -V6 for car use, and who knows what - some lower bitrate AAC for a mini-portable where space is very limited and the DAC/amps suck anyways.  If I'm going to the trouble to re-rip all of my music into lossless one of the goals is to build the flexibility to produce new copies of my library with whatever encoding I want for a particular purpose.  This encludes re-encoding simply because there's been some kind of huge codec breakthrough.  I ripped most of my music back in the 2003 time frame with LAME best practices at the time.  LAME is even better now, and I feel somewhat inclined to re-encode just because of that.  It would be great if I could just point a tool at FLACs, set my new settings and click "go".  I'm sure that desire's not unique to me, of course.


The beauty of having your stuff tagged perfectly is that, in foobar2000's example, it will always produce a perfect directory structure as well. And you can save these encoding settings (and in exactly the same way, copy / move settings) and have everything done with a few mouse clicks. This means though that there is the initial investment of fixing all your tags to follow your desired scheme, and of adjusting these encode / copy / move settings exactly how you want them.

I used to have my whole libary in the mp3 format as well and was always reasonably happy with it. That was until I heard a game soundtrack in Ogg Vorbis and it just sounded incredibly good to me. I checked, and the score was encoded at a measly 64kbps. So yeah.. I don't wanna preach or anything, but while you still have time before the big encoding happens, have a look at the more recent codecs too.

Offline Surly73

  • Thread Starter
  • Posts: 425
Best framework to encode entire tree of music recursively?
« Reply #9 on: Fri, 06 January 2012, 11:54:41 »
Quote from: godly_music;484722
Yes, it builds directories based on tags. One other option would be to select 'Source Track Folder' and remove the artist and album folders from Output Style. Then you would have to use a tool or some commandline magic to cut and paste those mp3s and create an exact folder tree duplicate. Probably not that hard if you know how. For the unique directory tree, you can select all those albums you don't want to put together with Various Artists, and put something else in their Album Artist tag. That'll still put them all into the same artist folder.

"Various Artists" was an illustration.  There may be several reasons I want to keep files separate.  Christmas music, for instance, is in its own file hierarchy since it's limited in use and if it gets copied somewhere it's usually ALL copied (and then ALL deleted a few weeks later).  The point is that I can't bank on "Various Artists" in a tag field to duplicate file tree preservation.  Music that is of digital origin is stored in different directories than that which I've ripped from CD, or captured from LP.

Quote
Tags are always carried over when the target file is able to store them. And I would always recommend to keep album art in separate files and to let foobar2000 just work through your music collection, either the FLAC one or the mp3 one later, and have the ReplayGain info written into the tags instead of having the volume level altered during encode, for example. But that's just my personal sense of 'order', since I have various devices that should play my files. But since I don't have any album art, someone else might be able to expand on this better.

I agree with all of this...  I want RG data tagged with no modification or normalization during or after encode.  I guess I'm not sure if FLAC tags can hold all the "stuff" I would tend to put in MP3 ID3 tags.  I'll have to look into it more.  I've only recently starting including album art.  It's another one of those things that if I am bothering to re-do my entire library, I'd like to get it right including things like art.


Quote
The beauty of having your stuff tagged perfectly is that, in foobar2000's example, it will always produce a perfect directory structure as well. And you can save these encoding settings (and in exactly the same way, copy / move settings) and have everything done with a few mouse clicks. This means though that there is the initial investment of fixing all your tags to follow your desired scheme, and of adjusting these encode / copy / move settings exactly how you want them.

Yep - some up front investment in properly tagging the lossless (or whatever format might be your "master") is worth it down the road.  Whether a directory structure created from tags is "perfect" or not is up for discussion :D  I get what you're driving at and agree.

Quote
I used to have my whole libary in the mp3 format as well and was always reasonably happy with it. That was until I heard a game soundtrack in Ogg Vorbis and it just sounded incredibly good to me. I checked, and the score was encoded at a measly 64kbps. So yeah.. I don't wanna preach or anything, but while you still have time before the big encoding happens, have a look at the more recent codecs too.

I was a Vorbis guy until I got my first iPod as a gift.  Back in the day I had no trouble hearing that Vorbis was better even without headphone amps and high end cans. Up until that point I was just playing on PCs using Winamp or something.  I determined that for my own sanity I'd switch to an excellent MP3 codec like LAME instead, just so that I could be sure I had "support" anywhere I'd want to play these files instead of being the guy swimming against the current feeling that he "knows better" and using generally unsupported codecs.  Restructuring everything would allow me to easily create an ogg branch of my collection too assuming that I get my conversion automation figured out satisfactorily.

Offline alaricljs

  • I be WOT'ing all day...
  • ** Moderator Emeritus
  • Posts: 3715
  • Location: NE US
Best framework to encode entire tree of music recursively?
« Reply #10 on: Fri, 06 January 2012, 12:04:07 »
Vorbis's tag structure (and by extension Flac's) is designed to hold any arbitrary string of data as a key/value pair.  The exception to this is for images stored in a comment named METADATA_BLOCK_PICTURE which allows you to embed multiple images.

Were you to want to script your own encoding solution custom tags/comments could be used for any purpose on that end of the process in addition to the usual Artist/Album/Track/Genre/...
Filco w/ Imsto thick PBT
Ducky 1087XM PCB+Plate, w/ Matias "Quiet Click" spring-swapped w/ XM Greens

Offline Surly73

  • Thread Starter
  • Posts: 425
Best framework to encode entire tree of music recursively?
« Reply #11 on: Sun, 12 February 2012, 07:21:54 »
I know this thread has been idle for a while, but I've started fooling around with getting this project rolling.

I've been playing with the "Convert.." function of foobar2000 and it gets things fairly close, actually.  Even though it says that it rebuilds directory structure based on tag data, it has also preserved some directory names which don't exactly match tags (which is good).  You can tell it specifically to leave the filenames alone which I like.

It has some problems, though....

First, it does not pass the replaygain tags directly from FLAC to LAME.  Luckily it has a checkbox to solve that problem and just re-scan after encoding - it's wasted CPU since the FLAC files already had replaygain data but at least it works and it isn't manual.  

It doesn't, however, have a solution for other tag types that don't get passed like cover art.  I believe that this is because LAME itself doesn't support these types of tags so it can't build them into the new file.  Unless there's an ID3-related option (like forcing different versions or something) which I just have to add to a custom command line in foobar2000.  Any advice on getting more tag types to transit through the conversion process?

I know that all2lame uses an external tag manipulation program tag.exe, which seems to have better support than what LAME alone can do so a scripted approach with yet another program might end up working, but then it won't be multi-threaded (foobar2000 handles making multiple calls to get all 4 cores working).

I know that some have debated folder art and how to do it (folder.jpg vs in the tags).  I want it in the tags for each file.  I won't always copy entire structures to smaller capacity devices, or I might be shuffling tracks etc...  If I'm going to all the work to reorganize everything then I want to get this right too.  I haven't looked at Vorbis at all yet.

Next would be looking at having something smart enough to only re-encode what's changed in the master lossless collection instead of having to do the whole thing.

Right now I'm using free MediaMonkey.  I do like the multi-collection concept (e.g. my Christmas music is a whole other collection), and maybe I could use the re-encode on the fly functionality to solve my whole issue.  I'm trying to do it first using free software though.