Mark D. Smucker, James Allan, and Ben Carterette. 2007. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management (CIKM ’07). ACM, New York, NY, USA, 623-632. DOI=10.1145/1321440.1321528 http://doi.acm.org/10.1145/1321440.1321528

Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. InProceedings of the 18th conference on Computational linguistics – Volume 2, Vol. 2. Association for Computational Linguistics, Morristown, NJ, USA, 947-953. DOI=10.3115/992730.992783 http://dx.doi.org/10.3115/992730.992783

The method seemed like exactly what I was looking for, unfortunately I could not find an implementation that was easily assessable and usable.  So I rolled my own.  If you find this useful please let me know.  More importantly, let me know if you find any errors, or if you have any suggestions.

You can download the code at: RandomPermutation.php

I already have a wifi router that does all these things well. So my task was to set up the Westell modem so that it passes off all duties (other than connecting to Verizon) to my router.

I played around with some of the modem settings to no avail.  I googled and found a posting saying that a similar Westell modem model, with similar problems, could be set to “bridge mode”.  While in that mode it should pass all duties to whatever internet device it is connected to.  This too did not work for me.  Verizon technical support was useless (no one I spoke to even knew what dynamic DNS is).

Finally, out of desperation, I meticulously went through all of the Westell’s configuration options.  One setting did the trick.  It has an option to set up a connected device as a DMZ host.  Luckily all external connection requests get passed to that device AND that device is assigned the external IP address.  In my case that is everything I needed.

Problem

That task done, the next task is the ensure consistent metadata across my entire music collection. The rippers I just mentioned all support pulling metadata down from multiple online databases, but as anyone as anal as I am about their metadata knows, automatically populated audio metadata is often incorrect and never consistent.  For example, is the artist field in classical music for composer or performer?  Do you prefer hip-hop, rap, or r&b?  Do you want to use Rock & Roll or RockNRoll or Rock and Roll.  And just cause you know they are all the same does not mean that iTunes (or whatever you use) does.

So, to fix audio metadata I resort to using the following tools depending on the task; iTunes + GimmeSomeTune, PicardTagger and Mp3tag.

Solution 1: iTunes + GimmeSomeTune

I use iTunes to play my music and to sync with my iPhone and iPod, so its to be expected that I do some of my music tagging work in iTunes.  If it is a simple, manual task like renaming a couple tracks names, modifying an album title or artist, etc… Then I just use the tag editing functions built into iTunes.  This also works fine for manually adding album art.

GimmeSomeTune is a nice little iTunes plug in that automagically adds metadata to tracks that iTunes plays.  I have mine configured to add lyrics and album art if they are missing.  I would estimate that this works around 75% of the time.  The other 25% is a mixture of GimmeSomeTunes not finding the lyrics or album art for my music and finding the wrong lyrics or art.  In the later case I manually go in and fix the error or resort to using one of the two following applications.

One thing to note is that the GimmeSomeTune website says that the app is being rewritten from the ground up.

Solution 2: PicardTagger

PicardTagger is nice because it works onall platforms and it can try and discover what a track is based solely on the music file (sorta like how shazam).  It can also use whatever metadata is already in a track (album name, artist, track number, track name) and search for the rest of the data.  The problem with Picard is that it only works against the MusicBrainz database, which does not always have what you want, or may not be able to find your music because of the way that the database is set up for searching.

Another bonus is that PicardTagger has a build for all platforms.

Solution 3:Mp3tag

This is (for me) the big dog.  The best music tagger I’ve seen. Some notable features that I have use are:

• It searches multiple music dat abases (Amazon, discogs, musicbranz and others),
• it supports every audio format I can think of, it automatically downloads music art,
• it supports all tag types, it can export and import music data from text and html,
• it can be used to infer metadata from filenames or rename files based on metadata,
• text editing and replacement features

And all these functions can be be run in batch over subsets or your music collection or over the whole thing.  Basically, if Mp3tag doesn’t do it, its time to start writing code.

The one drawback is that Mp3tag only has a windows version.

One last thing

Using either Picard or Mp3tag would not work if iTunes didn’t scan audio files for changes.  Thankfully it does. Right before playing a file iTunes checks if it has been modified, if so it reads the files metadata and updates the iTunes music database.  Apple, please never take this away!

Happy Tagging…

I recently (like yesterday) wanted to convert some YouTube videos to audio tracks that I could listen to on my iPod.  A quick google search gives tons of possible solutions, some like vixy.net take a URL and convert it in the cloud and give you a link for download.  This would work fine if I had only a couple videos and they were kinda short.  However I wanted to convert a bunch of videos each of which were over an hour in length.  So I decided to search for a command-line solution so I could write a shell script to do the conversions.

Solution:

So, basically I needed to download the YouTube video, and then convert that video to mp3.  For the first step there is a script called youtube-dl.  It downloads YouTube videos and saves them as flash video files.  It works PERFECTLY.  For converting flash video files to mp3s I used ffmpeg.  Again, it works PERFECTLY!

I did my Youtube downloading on Ubuntu where there is a youtube-dl package, and I did my ffmpeg audio converting on a mac where there is a macports package available.  But both tools are cross platform and should work on whatever you have.

Usage:

Once both tools are installed, the following is all you need to do to first download from youtube and then convert to mp3.  Lets say the link to the youtube video is http://www.youtube.com/watch?v=aFIPeFvPICs , and you want an mp3 file named audio.mp3 .  All you need do is on your command line (or in your shell script) run the following:

> youtube-dl -o video.flv http://www.youtube.com/watch?v=aFIPeFvPICs
> ffmpeg -i video.flv -ar 44100 audio.mp3

Both tools have a bunch of options.  The only additional options that I had to use were to include a username and password for Youtube because they were required to download the video I wanted.  In that case you run youtube-dl as follows:

> youtube-dl -u USERNAME -p PASSWORD -o video.flv http://www.youtube.com/watch?v=aFIPeFvPICs

After some digging and testing here is what I came up with. I’m sure there are numerous other (possibly simpler) ways of getting this done, but this works just fine.  Admittedly not a common problem, but hopefully useful to someone.

# save the old splitting char
SaveIFS=$IFS
# set the splitting char to linux split
IFS=":"
# split the classpath
declare -a SPLITPATH=($CLASSPATH)
# restore IFS
IFS=$SaveIFS
# iterate through the path elements
for (( i = 0 ; i < ${#SPLITPATH[@]} ; i++ ))
do
   # get rid of symbolic links
   new_single=`readlink -f "${SPLITPATH[i]}"`
   # convert from cygwin paths to windows paths
   new_single=`cygpath -w "$new_single"`
   # append to the new path
   new_path=$new_path";"$new_single
done
export CLASSPATH=$new_path

<?php
	// A quick dirty CSV viewer

	$USAGE = "USAGE: csvfilename [ column_width ]";
	if ($argc < 2) {
		echo $USAGE . "\n";
		exit(1);
	}

	$csvFilename = $argv[1];
	$colWidth = ($argc == 3) ? $argv[2] - 1 : 5;

	$lines = file($csvFilename);
	foreach($lines as $line) {
		$cols = explode(",", $line);
		$newCols = array();
		foreach($cols as $col) {
			$col = trim($col);
			if (strlen($col) > $colWidth) {
				// trim the column
				$col = substr( $col, 0, $colWidth);
			}

			if (strlen($col) < $colWidth) {
				// pad the col
				$col = str_pad($col, $colWidth, " ", STR_PAD_LEFT);
			}

			$newCols[] = $col;
		}

		echo implode($newCols, ", ") . "\n";
	}
?>

My new music collection maintenance recommendation is GimmeSomeTune .  For what it does it works very well.  GimmeSomeTune runs in the background and monitors the music currently playing in iTunes.  When a new track is played it can check to if the track has lyrics and/or artwork associated to it in iTunes.  If it does not it can automatically pull down lyrics and artwork and add it to the track.  In the case of artwork it appears to apply whatever it finds to all tracks of the album that playing track is part of.  I’m not sure where its getting artwork or lyrics, but it works pretty well on more popular stuff, while (not surprisingly) it is less accurate on more obscure stuff.  The artwork and lyrics are typically of pretty good quality. Added bonus is that it is “donationware” (i.e. free to use, donate what you wish).

It can do other things like integrate with LastFM, and act as a mini controller for iTunes, but while those functions seem to work ok, the thing that makes GimmeSomeTune special is the automatic lyric and coverart updating.

Only problems I’ve encountered are that the current version does not have a function that allows you to scan and update a bunch of tracks at once.  There are a couple ways around this.  I will likely write an applescript that plays the first couple seconds of all tracks that I want automatically updated, that should give GimmeSomeTune the opportunity to update what it can.  The second (more troubling) problem is that for anything that GimmeSomeTune gets wrong, it will consistently get it wrong.  So if it uses the wrong coverart or lyrics for a track, and you go in and remove the bad info, the next time you play that track GimmeSomeTune will again add the same incorrect info (unless presumably the source that it is getting its information from with better metadata).  The only way around this seems to be using a placeholder for artwork or lyrics so GimmeSomeTune will leave the track alone.

Anyways, I’m still on the lookout for something better, so let me know if you know of anything.

for a in 1 2 3; do echo $a;  done ;

Note that the semicolons can be carriage returns instead.  In fact it seems that everything prior to the first semi-colon can appear on the same line, and do can appear on the first line or by itself on the second line or on the same line as the first command in the list of commands to execute (in the case of the example that is echo).

There are 3 types of ripping/encoding configuration settings that are important to me (and should be to you) when ripping CDs; the rip settings, the encode settings and the output metadata settings.  When ripping I want the highest quality rip I can get. For that I configure Max to use cdparanoia set to Full paranoia mode.  For encoding I want both mp3 files and FLAC files.  MP3 files are the most portable, while FLAC files are lossless and have the highest possible quality.  Max allows you to encode to multiple formats simultaneously, so for encoding CDs I enable and configure both.  I use the default FLAC settings.  For MP3 I use the Transparent setting, which produces the highest quality VBR (variable bit rate) MP3 files appropriate for the audio.

If there is any weakness to Max it is in the management of metadata.  Since Max is concerned with ripping and encoding its not too much of a negative that metadata management is lacking, but it is something of an inconvenience.  Max attempts to pulls tag information (artist, album, track title, ets…) from MusicBrainz, which in my experience has maybe a 70% hit rate.  Even if it pulls the correct information it does not automatically pull album art.  Max does allow you to pull art from Amazon, but if the tag information does not match with Amazon, or if no tag information is available from MusicBrainz then you’ll have to manually correct your tags before querying Amazon.  I’d estimate that getting a valid hit in MusicBrainz and having the tags match with Amazon for album are is about 50% for me.  For the other 50% you can manually enter all the album info prior to ripping and Max will place the metadata into the encoded music files.  That’s too much of a hassle for me, so instead I just enter the artist and album name.  For compilations I just enter the album name.  Later I use an app like mp3tag (sadly only available on PC) to correct tags and get album art in bulk.

In addition to tags you should make sure that the filename convention works for you.  I like having a folder full of artist names and in each folder have all music tracks with a naming convention of:

ALBUMNAME-TRACKNUMBER-TRACKTITLE

If the album is a compilation it will be placed into a top level folder named Various Artists.

The only thing to add is that on PC the best I’ve seen (and I’ve used it extensively as well) is Exact Audio Copy (EAC).

> find . -name ‘FILENAME’  -exec tar –append -vf ~/TAR_FILE.tar {} \;