John's Programs

The Importance of Bug Lists

2010-02-06T21:44:00.000-08:00

Status of the Music DB program

The first phase of my Music Database Project is done. The program I have allows me to slip a CD into my computer and will determine a disc ID and track lengths from which it can go out and get the name of the album, principal artist(s), and track titles.

This data is presented to the user along with empty boxes into which he can supply the names of the performers for each track, composers, lyricists, various URLs (such as the URL to a video of the performer performing the music on the track), and notes.

When the user has completed the form, there are checks to try to make sure that the data is as clean as possible, namely, that there are no "typos" that might result in duplicate entries for the same artist or the same piece of music.

Once the user accepts the data, it is placed in an object-oriented database from which it will be retrieved in the second phase of the program.

Lessons learned

There have been many lessons learned but in this blog entry, I want to champion the use of bug lists.

As with any software project, once you get to the point where you have software that is working (although not complete), you find that there are many things that are still incorrect. Develop a "bug list". I use a simple text editor in which each bug I find is placed as another new line. For example:

• Artist ambiguities not complete.
>>Fixed.

This particular bug was easily fixed and I just noted that it had been fixed.

Other bugs are difficult. Here's one that proved to be tough to resolve:

• The online search for track titles fails and produces no output for the user for the album: "Dr. T" with Billy Taylor.

This particular bug involved the online search in the freedb web site for the particular album. In my program this involves a CGI Perl script that uses telnet to communicate with the freedb site. There are a number of pieces to the software that could contain the bug.

Instead of just reporting that I had fixed the bug when that happened (eventually), it's really important to document what you did along the way. Someday there will be a similar bug but by then you will have forgotten how you resolved this one. So I keep good records of what things I do and their results. Here is a piece of that log (which I insert into the bug list as I go as additional lines after the original bug description):

(1) go back to the command line access to the online database to see what's going on.
At the terminal window, entered the command (after inserting the CD):
cd-discid /dev/rdisk2

Here's the reply:
730d480a 10 150 27962 75475 102750 120057 148635 168450 190332 202657 236085 3402

Now I typed:
telnet freedb.freedb.org 8880

Here's the reply:
Trying 195.214.216.38...
Connected to freedb.freedb.org.

This also serves as a record of those things you try and the results. This documentation can be used in other bug fix situations where cutting and pasting may save you a lot of time.

Prioritize the bugs

Not all bugs are equally important. Some can be ignored for a long time simply because they don't affect the program significantly. Others have the quality that if they are not fixed, the program simply will not work properly.

I keep two sections in my bug list file: the bugs still not resolved (at the top) and the solved bugs (below the unsolved ones).

I prioritize bugs in the bug list by placing the most important bug at the top. As I resolve one bug, I move the entry down into the solved bug area of the file. In the course of solving one bug, I may find that other bugs become more critical since they may impact the bug I'm working on.

In the end, the bug list provides an important tool in the software development process.

Data Validation

2010-01-20T14:28:00.000-08:00

The purpose of validation

The idea behind validation is to minimize the number of errors in the database. The blog entry on September 3rd lays out the plan for data validation. Implementing those plans required some changes.

Suppose the user enters the name of a principal artist that is not currently in the database. Is this really a new artist or is there a typo or perhaps the user is using a different version of the name. For example suppose the artist is Johann Sebastian Bach but the user enters J S Bach. We would like to avoid having two artists in the database that should really just be one.

What to do if we suspect that there is ambiguity about an entry

For each new artist name, we check the name against that of the existing artists in the database. Suppose we find that there is not an exact match. We then look for names that have a sufficient similarity that they could be the artist. Once we have this list of names we want to indicate that there is the possibility of a duplication and let the user decide which one to keep.

I started by focusing on the mechanism for keeping the multiple names. Remember that the user is in the process of entering the data for an album and is presented with a screen such as the following. (Click on the image to see a larger version).

In the same space that contains an input field, we want to display the choices available to fill that field. The obvious choice is a pull-down menu containing the list of artists. But how would the user be quickly alerted to the fields where there were potential ambiguities? The choice here was to change the background color for that field (see the image below)

and to have an alert message displayed when the page was first opened

.
Internally, how do we represent the ambiguity. An Artist object, for example, has only one name field yet we want to somehow convey the multiple artists that are in contention for a particular data field. One possibility was to create an abstract (Artist) class, for example, and then have one subclass for the "real" Artist class and another that carried the ambiguity in the name. This approach proved to be unwieldy and, in the end, I chose to have parallel classes to the existing data classes. These parallel classes offered the advantage that the data fields could be maintained as String objects which is convenient for working with the form data on the JSPs.

At the top there is a PrelimAlbum class with data fields:


    private String title;
    private String recordingLabel;
    private String yearRecorded;
    private String coverArtFile;
    private String playTime;
    private String discID;
    private String physicalLocation;
    private String notes;
    private List principalArtistList;
    private List trackList;

Notice that all the data fields are Strings and that the principal artist field and the list of tracks are represented by lists of new classes PrelimArtist and PrelimTrack.

The new PrelimArtist class has data fields:


    private List names;
    private String birthDate;
    private String wikipediaArticle;
    private List URLs;

Now we can accommodate any ambiguity in the list of names. If there is more than one name in the list, then there is a potential ambiguity.

Likewise, the PrelimTrack class parallels the Track Class and has fields:


    private List URLs;
    private String audioFileName;
    private String playTime;
    private PrelimPieceOfMusic piece;
    private List performerList;

There is no ambiguity carried directly in these fields but indirectly the possibility of a track from another album having the same piece of music led to the creation of the PrelimPieceOfMusic class which parallels a corresponding (new) class PieceOfMusic.

The Performer class has its parallel class PrelimPerformer with fields:


    private PrelimArtist artist;
    private String instrument;

Notice that any ambiguities are carried in the PrelimArtist class field.

The PrelimPieceOfMusic class carries any potential ambiguities in a similar manner to the PrelimArtist class. Here are its fields:


    private List titleList; 
    private List composerList;
    private List lyricistList;
    private String leadSheet;

Ambiguities for composers and lyricists are carried in the PrelimComposer and PrelimLyricist classes. Note that the lists in the PrelimPieceOfMusic class represent the fact that a piece of music may have several composers or lyricists regardless of ambiguities. It's the PrelimCompser and PrelimLyricist classes that have a list of names. [The leadSheet field represents a new field that I added to the database to allow saving lead sheets (sometimes called "fake sheets") with the chord changes for many pieces of jazz music.]

When we check the database for existing entries as we validate the information, we have a small problem. Suppose we find an exact match for an artist in the database. We place that artist's name in the JSP shown to the user with the validated data and then later when the user commits to placing the data in the database, we need to know which object was associated with that artist. Remember that all we have in the JSP is the name of the artist. We need to know, when we process the data for entry into the database, that this is associated with an existing object.

The mechanism I chose was to assign a random number to each object and to make it a part of that object's instance variables. Now, when we pick a name from an existing Artist object to display in the validation page, we append the id of the object to the name (as in "Stan Getz[46265]"). In the post-processing of the validated data, I use that id to locate the appropriate Artist object. This is facilitated with the use of a hashtable for each of the object classes that can incorporate amibiguities (Artist, Composer, Lyricist, PieceOfMusic).

There is a second small problem. What if, in the course of processing the data for an album, we locate a new artist not in the database and for which there are no "nearby" names that could signal an ambiguity? We create a new Artist object but it isn't in the database yet. Later in the processing of the data we encounter another artist not yet in the database. We must not only check the database but we must check any new Artist objects that have been created in previous processing but are not yet in the database. This is facilitated by another hashtable that uses the artist's name as the key to locate the corresponding Artist object.

The same problems occur for composers, lyricists, and pieces of music. These are treated in the same manner.

Lessons Learned

1. Trying to prevent duplicate objects in the database is not trivial. I had to develop a method for detecting such duplicates and I had to develop a way to handle such potential ambiguities in a web environment. This involved algorithm development for locating potential duplicates, data structure utilization for marking and quickly accessing objects that are potential duplicates of objects associated with names in a form.

2. Storing the information for an object (e.g. Artist) in the presence of ambiguities required developing parallel classes that allowed me to encapsulate the ambiguities in a list of things (e.g. names). Having parallel classes also allowed me to store the information in String form whenever possible which then made working with forms easier in other parts of the program.

Progress and a review of purpose

2010-01-13T07:18:00.000-08:00

Why the Blog

The reason I started writing this blog was to document this project I was working on. Over time I've been re-thinking that purpose. My teaching roots push me in the direction of wanting to make this more than just a documentation. What if I were to record lessons learned and to focus on the larger picture rather than the nitty gritty details of one or more Java classes? So, this is the direction I hope to move.

Progress to-date

Despite the fact that the last post was almost 4 months ago, I have been working on the project. I've reached a point where I can insert a CD into my computer and have the program fetch information about the CD such as its title, the principal artist, number of tracks, track titles, and track playtimes. The program will check to see if an artist or composer or lyricist is already in the database so that a duplicate entry is avoided. There have been changes to accommodate more fields as seemed appropriate and there are many more classes that resulted from these changes.

Some of the changes that I made were the result of entering data into early versions of the program. These exposed some problems that I could correct at an early stage rather than trying to make modifications after the program was completely written.

Changes

The Album class was modified so that instead of just one principal artist, I could have a list of them. Sometimes an album has several artists that are equally prominent. The second change was to add a notes field. Sometimes, there are some interesting facts that the database user should know but that don't fall into the category of another field. For example, in the liner notes for one album, I found out that there were two piano players on the tracks but that the particular one on a particular track was not listed. This is something I would like to know if I'm looking at a particular track on this album.

The Track class was modified more extensively. I was thinking of how a user may want to look for other occurrences of a particular piece of music and I realized that the structure that I had would probably require that I search through all tracks on all albums. The problem, I decided, was that I needed to separate a piece of music from a track. In particular, a piece of music has a title, a composer and perhaps a lyricist. This is separate from a track in which a particular piece of music is played by an artist using some instrument. The result was the introduction of a new class: PieceOfMusic with a title, a list of composers (after all there may be more than one), and a list of lyricists. There are a couple of other fields I'll talk about later.

This simplified the Track class so that it now looks like:


   Track
       piece           PieceOfMusic
       URLs            List
       audiFileName    File
       performerList   List
       playTime        PlayTime

The other classes (Artist, Performer, Instrument, PlayTime) remain essentially the same.

Things learned

The modifications that I have made were the result of several things that represent things learned. The first thing is to maintain flexibility as long as possible. Once we commit to the fields in the database and begin placing data in the database, we soon have a very strong incentive to avoid making changes. Changes, however are often necessary and as long as things are fluid, changes are relatively easy to make.

The second thing that I learned is that we should simulate the data entry process. This may be done in a "dry run" way or it may involve actual execution of preliminary versions of the program. Many of the changes I made were made as a result of entering data and realizing that there were cases that didn't fit the design. For example I found that some albums had more than one principal artist. Another example was that I found (as mentioned above) that sometimes one wants to place a note with the album to account for some issue that a data field itself can't resolve.

The third thing I learned was that its a lot easier to debug a program that's mostly working rather than a program that isn't yet working. So I started by getting a program that just displayed the entries I was making but didn't yet place them in the database. Then I added more fields as necessary or I modified the layout on the screen to accomodate what I had so far. Most IDEs let you step through the execution of the program when you are debugging. For a web-based application like this part of the project, I have not found a way to do this. That may be partly due to the fact that I have some Perl (the CGI component at the beginning) as well as a Servlet and JSPs. Fortunately, the old-fashioned insertion of print statments in strategic locations results in output showing up in the Glassfish output which is nicely integrated with the NetBeans IDE.

One more thing

2009-09-18T07:18:00.000-07:00

As the user enters data in the CD entry mode, he adds more rows of track data as necessary. This is done by clicking on a button at the bottom of the block of data corresponding to the track. What happens as soon as the user clicks on the button is that a new page is presented with the new row of blank input boxes but the window was positioned at the top and the user had to scroll down to the track for which he was entering data. A friend pointed out that it would be much nicer if the new page would be positioned on the track for which the new row had been created. This was a lot harder than it seems.

The first thought is to use the HTML <a> tag with the name attribute. You attach a name (a bookmark, really) to a location on the page with the <a> tag (example: <a name="track3">) and then reference that name in the <a> tag in another document (example: <a href="dataUpdate.html#track3">). Unfortunately, these are not static HTML documents but JSPs which are assembled on the fly to produce the HTML pages delivered to the browser. In other words, there is no <a href="..."> tag to work with that will allow you to use the bookmark mechanism.

I spent some time following links on the web to see if I could find an answer to this puzzle. The answer is to use Javascript. In Javascript one can issue commands to tell the browser to scroll the page to a particular point (e.g. y pixels from the top of the page). To do this you have to determine the number of pixels from the top of the page to the location you want. Here's what you do. Use the fact that you can attach an id to an HTML element. In particular, the table row element that starts each track section is given an id:


<tr id="row_<% out.print(j); %>">

The index j is a loop index that cycles through the tracks. The result is that in the generated HTML document you get (for j = 5 here):


<tr id="row_5">

This id associated with this tag can now be used with a Javascript function to find the offset from the top of the window so that we can scroll the window in such a way that the displayed portion starts at the row corresponding to the track of interest.

Here is the appropriate Javascript code:


<script type="text/javascript">
    function scroller(obj)
    {
        var curleft = curtop = 0;
        curleft += obj.offsetLeft;
        curtop += obj.offsetTop;
        obj = obj.offsetParent;
        curleft += obj.offsetLeft;
        curtop += obj.offsetTop;
        window.scrollTo(curleft, curtop);
    }
</script>

This is placed in the <head> portion of the HTML document. Then we modify the <body> tag using the following code:


<%! int numberOfTracks; %>
<% numberOfTracks = Integer.parseInt(request.getParameter("numberOfTracks")); 
    String addFields = null, trackId = "row_1";
    for (int t = 0; t < numberOfTracks; t++)
       {
           addFields = request.getParameter("addInputDataFields"+t);
           if (addFields != null)
           {
               trackId="'row_"+t+"'";
               break;
           }
       } %>

<body onload="scroller(document.getElementById(<% out.print(trackId); %>))">

Remember, the Java portions of this code are executed at the time the JSP engine is creating the HTML document. The user has just clicked on one of the "More input fields" buttons. The first thing we need to do is to find which of those buttons was clicked. When we emerge from the for loop, the trackId string will be (in the case of track 5 as an example again) 'row_5'. (The single quotes are necessary because in the generated HTML, this string is embedded in an outer double quoted string).

The "onload" piece of the <body> tag refers to a Javascript "event". In this case, the event is the loading of the page by the browser. We are telling the browser to execute the Javascript function scroller when this event occurs. A good place to start to learn about this stuff is the w3schools website.

Briefly, the scroller function is a modification of a more comprehensive version that is available from QuirksMode.org. For some reason, I was never able to get the more comprehensive version to work for my application. In the function we use the offsetLeft and offsetRight properties associated with HTML elements. Because the table row is embedded in a table and the offset properties only give you the offset relative to the parent, we have to walk up the chain of nestings to get the total offset. When we have the total offset (both from the left and the top of the window), we call the scrollTo function to position the page right where we want it.

Now, we are ready to go on to the validation of the CD data.

Plan for Data Validation Phase

2009-09-03T20:39:00.000-07:00

Once the user is satisfied with the data for an album, the next step is to check the data against what is currently in the database.

First of all, we can check if the album has previously been entered. This is an easy check by comparing the disc ID. The next question is what to check and how to check it. Here's what we are going to check:
• the name of the principal artist or artists;
• the title of each track;
• the name of each artist on each track.

The way we will check is to use string comparison with n-grams. There's a paper describing this algorithm by Jeremy A. Hylton in the context of bibliographic data.

Each item that we're checking in the album data will be compared to all the current entries in the database. For example, each principal artist will be compared to all the artists currently in the database and a list of very similar names will be generated (if such a list exists). Similarly, a list of very similar track title names will be collected by comparing each track title against those in the database.

What happens if we find very similar names of titles? The user is presented with the data that was entered in the same layout. For those fields for which there are very similar (but different) entries already in the database, instead of the actual entry the user will see a SELECT HTML element with the users entry displayed but with the alternative entries showing in the drop-down selection. The user can choose to leave things as they are or to change the entry to one of those available.

The mechanics will be as follows. We will make the Album class, the Artist class, and the Track class into JavaBeans. This will allow us to use JSP technology. JavaBeans are described in the
wikipedia article. When we collect the data from the user we will create an appropriate Album object and fill in the appropriate fields (including creating any necessary new objects). In particular here is the code to build the Album object:


private Album makeAlbum(HttpServletRequest request)
{
    Album newAlbum = new Album(request.getParameter("albumTitle"));

    //--Pick up the number of principal artist entry fields (which
    //--may be more than the number of principal artists).
    int numberOfPrincipalArtistFields =
            Integer.parseInt(request.getParameter("numberOfPrincipalArtistFields"));

    //--Determine the principal artists for this album and add them
    //--to the list.
    for (int i = 1; i <= numberOfPrincipalArtistFields; i++)
    {
        String principalArtist = request.getParameter("principalArtist_"+i);
        //--Add the artist if the field is not null or blank.
        if (!(principalArtist == null || principalArtist.equals("")))
        {
            newAlbum.addPrincipalArtist(new Artist(principalArtist));
        }
    }

    //--Fill in the other non-track fields.
    String recordingLabel = request.getParameter("recordingLabel");
    newAlbum.setRecordingLabel(recordingLabel);
    String yearRecorded = request.getParameter("year");
    newAlbum.setYearRecorded(yearRecorded);
    String coverArtFileName = request.getParameter("coverArtFileName");
    File coverArtFile = new File(MusicDBPreferences.coverArtFolder+coverArtFileName);
    newAlbum.setCoverArtFile(coverArtFile);
    String totalPlayTime = request.getParameter("totalPlayTime");
    PlayTime playTime = PlayTime.parsePlayTime(totalPlayTime);
    newAlbum.setPlayTime(playTime);
    String discID = request.getParameter("discID");
    newAlbum.setDiscID(discID);
    String location = request.getParameter("location");
    newAlbum.setPhysicalLocation(location);

    //--Now take care of the tracks.
    int numberOfTracks = Integer.parseInt(request.getParameter("numberOfTracks"));
    System.out.println("makeAlbum-- numberOfTracks: " + numberOfTracks);
    newAlbum.setNumberOfTracks(numberOfTracks);
    for (int j = 1; j <= numberOfTracks; j++)
    {
        Track track = makeTrack(request, j);
        newAlbum.addTrack(track);
    }
    return newAlbum;
}

    private Track makeTrack(HttpServletRequest request, int j) 
    {
        String title = request.getParameter("trackTitle"+j);
        Track newTrack = new Track(title);
        
        int numberOfInputDataFields = 
                Integer.parseInt(request.getParameter("numberOfInputDataFields"+j));

        //--Pick up the performers.
        for (int i = 1; i <= numberOfInputDataFields; i++)
        {
            String performerName = request.getParameter("performer"+j+"_"+i);
            String instrument = request.getParameter("instrument"+j+"_"+i);
            if (!(performerName == null || performerName.equals("")))
            {
                Performer performer = new Performer(new Artist(performerName),
                        Instrument.getInstance(instrument));
                newTrack.addPerformer(performer);
            }
        }

        //--Pick up the links associated with this track.
        for (int i = 1; i <= numberOfInputDataFields; i++)
        {
            String linkName = request.getParameter("links"+j+"_"+i);
            if (!(linkName == null || linkName.equals("")))
            {
                newTrack.addURI(linkName);
            }
        }

        //--Pick up the composers associated with this track.
        for (int i = 1; i <= numberOfInputDataFields; i++)
        {
            String composer = request.getParameter("composer"+j+"_"+i);
            if (!(composer == null || composer.equals("")))
            {
                newTrack.addComposer(composer);
            }
        }

        //--Pick up the lyricists associated with this track.
        for (int i = 1; i <= numberOfInputDataFields; i++)
        {
            String lyricist = request.getParameter("lyricist"+j+"_"+i);
            if (!(lyricist == null || lyricist.equals("")))
            {
                newTrack.addLyricist(lyricist);
            }
        }

       //--Pick up the play time for this track.
        String playTimeString = request.getParameter("playtime"+j);
        PlayTime playTime = PlayTime.parsePlayTime(playTimeString);
        newTrack.setPlayTime(playTime);
        return newTrack;
    }
}

This code is found in the servlet that gets called by each of the genereated web pages (either the Perl scrip, or the update data entry jsp, or the validation jsp). The servlet is the traffic cop, directing traffic to the appropriate JSP.

The first JSP (updateDataEntryForm.jsp) allows the user to fill in the data missing after the Perl script fills in what it can. This JSP will allow the user to add more fields as necessary by clicking on an add button.

The second JSP (dataValidation.jsp) takes care of the case when the user is satisfied that all the information for the CD album has been entered. This JSP is the one that will communicate with the database (through the JavaBeans) to identify possible mis-spellings and errors that we mentioned above. For the appropriate fields, the JSP will call on the appropriate JavaBean to check the database. If there are near-misses, then they are collected into an array of Strings. These will be placed in a SELECT HTML data construct that will display the value supplied by the user with "nearby" values listed as pull-down items.

We will describe this in some detail on the next blog entry.

Data Classes

2009-07-10T20:11:00.000-07:00

I have put off the writing of the data classes for as long as I could. This was not a matter of procrastenation. As I've been working on the data entry and trying things out on CDs that I currently own, I find that I need to consider new fields or revise the ones that I had decided on earlier. Once the data classes have been established and I begin to enter data, it will be much more difficult to change things. For example, how will I convert the old data already entered into the new data with additions and modifications in the structure?

So here is the latest version of the data classes:


Classes fields            type
   Album
        title             String
        principalArtist   Artist
        recordingLabel    String
        yearRecorded      int
        coverArtFile      File
        playTime          PlayTime
        discID            String
        physicalLocation  String
        trackList         List
   Track
        title             String
        performerList     List
        URLs              List
        composerList      List
        lyricistList      List
        audioFileName     File
        playTime          PlayTime
   Artist
        name              String
        birthDate         Calendar
        wikipediaArticle  URL
        URLs              List
   Performer
        artist            Artist
        instrument        Instrument
   Instrument
        name              String
   PlayTime
        min               int
        sec               int

A few remarks: The PlayTime class allows me to get at the minutes and seconds component without having to parse a String object to get those elements any time I want them. I overrode the toSting method to provide the String version of the PlayTime object. Notice that I have a number of fields that contain objects of the URL class. This will allow my program to let the user view certain videos (on youtube, for example) or articles (such as wikipedia). These articles or videos will enhance the database.

There are other classes, of course, that will manage the logic flow of the program. There is the data to save to the database, there is validation of the data to try to catch typos or minor variations on the title of a tune. For now this will give me a blueprint with which to work.

From Tomcat and Eclipse to Glassfish and NetBeans and the Overall Plan

2009-07-05T08:58:00.000-07:00

This is a process of programmer development as well as program development. After spending a significant amount of time trying to integrate Tomcat 5.5 and Eclipse 3.4 (see this article), I gave up. The problem probably was related to the fact that I wanted to have an independent Tomcat installation running outside of the Eclipse "box" and not just withing the IDE environment. I was resigned to letting Textmate manage the code components from diverse locations in the file system and then manually starting and stopping Tomcat during debugging. This is not an unacceptable way to go except that an IDE can save so much effort during the development of a project like this.

Then I decided to revisit some of the technologies such as JavaServer Faces (JSF). As I worked through some of the tutorials (such as kickstart). Again I had problems getting Tomcat to work (this time with JSF). Finally I came to the thought that the Java technologies come from Sun Microsystems and they use NetBeans for an IDE and Glassfish for the application server (that Tomcat provides). So why am I trying to get Tomcat and Eclipse working together for this project?

I started by downloading GlassFish, JavaEE, Tools (incl. Netbeans) from downloads (Mac OS X and PPC version). I customized the install to leave out Portlet stuff and worked through the tutorial: quickstart. Not all is exactly as it says but it seemed much easier to fix minor inconsistencies than it had with the Eclipse IDE. NetBeans handles the launching of Glassfish easily. One issue is that NetBeans is quite a memory hog. I decided to invest in a memory upgrade from 512MB to 1GB ($60) and things loosened up considerably.

I worked through another tutorial (jAstrologer intro) to learn about the special tags that are the characteristic of JSF. Since the graphical elements of the user interface will have to be more extensive than just the form field stuff, I decided to work through a tutorial on Applets and NetBeans (web and applets). Now I'm seeing the utility of NetBeans, especially in the tools that let you build a form page graphically and then go back and supply the code that provides the appropriate response to events.

I'm sold. Here's the current plan––the access of the online CD database (freeDB) and the ability to manually add information to the album data is done (the boxes in blue in the diagram). Validating the data involves placing the data into the database and trying to make sure there are very few errors. In particular, I want to check existing track titles against the current track titles to discover and eliminate inconsequential variants. The same applies to artists, instruments, etc.

Another part of the program involves collecting and placing cover art into the database. This may involve just incorporating images that the user has scanned or actually searching the web for cover art images.

Still another part of the program will convert from aiff format files (existing CDs) to MP3 format that allows storing (and playing) these files in compressed format. And another part is an editor that will allow the user to modify an existing record or to add information to the database.

Rounding out the program will be the browser itself. This browser is where the user can ask to see music with particular characteristics involving any of the data fields. This part of the program will involve a more substantial use of graphics in with some animation involving the cover art images.

Music Database Program: Tomcat

2009-06-15T10:02:00.000-07:00

From Apache to Tomcat
Last time, I showed the initial part of the program to extract the data from a CD and present it to the user so that additional data can be added for a particular CD and eventually placed in the database. In the previous blog entry, I described how to use the Apache web server that comes as part of Mac OS X to provide the underlying structure for our application to obtain title and track information for an audio CD.

This information will become the start of the information that we collect for each CD in building up our Music Database. The information that we collect from the FreeDB web site (freedb.org) contains CD album titles, track titles and some artist information. That's only a portion of the data I want to have for each CD in our collection. Much of that information will have to be entered in by the owner of the database. The next phase of the project addresses this issue.

JavaServer Pages (JSP) builds on top of Java Servlet technology to provide for full Java applications on the server side of the client-server internet connection. Apache-Tomcat (usually just known as tomcat) is the server software that manages the connection and provides the bridge to the Java software that supplies the content that the user sees. Once I have the FreeDB data I can work completely in the Java language instead of having to revert to Perl to provide the Telnet service that I needed. This is the path I will take.

Installing Tomcat
First of all, you have to install Tomcat on your machine. Here are the steps I took.

Downloaded apache-tomcat-5.5.27 from the
tomcat-apache web site.

Placed the unzipped folder into /usr/local

Created a symbolic link (from the terminal application)

ln -s /usr/local/apache-tomcat-5.5.27 /Library/Tomcat

(this sets the path to the tomcat home directory to be known as /Library/Tomcat)

There is a very helpful web site with
contnued directions. Following the directions in that web site I then issued (in the terminal application):

cd /Library/Tomcat/bin

and issued the following (to set the appropriate files to be executable):

chmod 755 *.sh

Next I placed the following in my .tcshrc file (which is located in your home folder):

# Tomcat stuff

setenv TOMCAT_HOME /Library/Tomcat

and issued (this will make the information you just place into the .tcshrc file immediately available in yout terminal window. Since each time you start up a terminal window this file is read, this information will be available to your terminal sessions from now on):

source .tcshrc

Finally, I started up Tomcat by issuing (again from the terminal window):

/Library/Tomcat/bin/startup.sh

Here was the response:
```
Using CATALINA_BASE:   /Library/Tomcat
Using CATALINA_HOME:   /Library/Tomcat
Using CATALINA_TMPDIR: /Library/Tomcat/temp
Using JRE_HOME:       /Library/Java/Home
```
In the browser, I entered the url:

http://localhost:8080

and I got the Tomcat home page. This meant that I now had the tomcat software installed and that it was listennng on port 8080 for any incoming requests.

As an aid in the development process, I added the following three lines to
the .tcshrc file in my home folder:


alias startTomcat '${TOMCAT_HOME}/bin/startup.sh'
alias stopTomcat '${TOMCAT_HOME}/bin/shutdown.sh'
alias restartTomcat 'startTomcat; stopTomcat'

Now, when I want to start, stop, or restart tomcat, I can just enter the commands


startTomcat
stopTomcat
restartTomcat

in the terminal window which I keep open during development.

JSP, Servlets, and the MusicDB program
The plan is to pick up from the flow diagram of April 10. We need to let the user expand the number of copies of certain fields as necessary. In particular, there may be more than one principal artist for an album or more than the three possible artist/instrument pairs boxes that I supply in the first view that the user sees (see the May 4th entry). Next, I must validate the data to try to remove typos, discrepancies, and other errors. All this is done with Java.

I used the Model-View-Controller pattern in which a controller servlet gets the incoming data and decides which view to send back the user based on the (data entry) model I have put together. The servlet is just Java code that ties in with the tomcat software. The views are supplied by Java Server Pages (JSPs) that consist of a mixture of Java statements and HTML statements. I think that it would be taking this too far afield to try to explain the elements of JSP and servlets. The book I found most useful is Core Servlets & JavaServer Pages by Hall and Brown.

In this blog entry I'm just looking at the box in the diagram here labeled "Process request for more copies of certain fields". As I built the JSP and the servlet, I uncovered some bugs in the original Perl script (the part that generated the HTML). I've fixed these bugs which generally had to do with neglecting to provide some hidden fields that really were necessary for processing the data (such as the number of artist/instrument fields for each track). The data is first sent to a servlet. If the servlet determines that a request for more fields is present (more principal artist fields or more track fields) then the JSP that is invoked is the one that adds more field entry boxes but otherwise just keeps the fields that have so far been entered.

The mixture of HTML and Java code in the JSP makes for some difficult debugging. Here's an example of a piece of the JSP that places previous prinicipal artist fields into the proper position and keeps the field entry values that have already been filled.


<%! String moreArtists;
    int i;
       String bgColor = ""; %>
<%-- Include all previous principal Artist fields (if more than one) --%>
<% i = 2;
   while (i <= numberOfPrincipalArtistFields)
   { 
      if ((i-1)%3 == 0)
         bgColor = "blue"; 
      if ((i-1)%3 == 1)
         bgColor = "orange"; 
      if ((i-1)%3 == 2)
         bgColor = "red";
    %>
      <tr>
         <td></td>
         <td bgcolor="<% out.print(bgColor); %>">
            <input name="principalArtist_<% out.print(i); %>" 
               value="<% out.print(request.getParameter("principalArtist_"+i));%>"  
                  size="30" type="text"></td>
         <td colspan="5"></td>
      </tr>
<%   i++;
   } %>
<%-- i is now numberOfPrincipalArtistFields + 1 --%>
<% moreArtists = request.getParameter("addPrincipalArtist");
   if (moreArtists != null)  //--We are adding a new principal artist field.
   {  
      numberOfPrincipalArtistFields++;

      if ((i-1)%3 == 0)
         bgColor = "blue"; 
      if ((i-1)%3 == 1)
         bgColor = "orange"; 
      if ((i-1)%3 == 2)
         bgColor = "red";
 %>
      <tr>
         <td></td>
         <td bgcolor="<% out.print(bgColor); %>">
            <input name="principalArtist_<% out.print(i); %>" 
               value="" size="30" type="text"></td>
         <td colspan="5"></td>
      </tr>
<% } %>

In the next entry I'll talk about replacing Apace with Tomcat altogether for this project.

Music Database Program: Using Apache

2009-05-04T06:58:00.000-07:00

The program we have developed so far is a web applicaton. It uses telnet to communicate with the freeDB server. Instead of building a GUI front end to the perl script, we can use readily available tools, in particular the browser, to provide the front end. Mac OS X comes with the web server software known as Apache. This open source software is designed to deliver web pages and documents to clients who have requested them. It is also capable of communicating with other applications (such as our perl script). Our initial data entry program will utilize this technology.

You have to make sure that the Apache web server is running. To check, select System Preferences... from the Apple menu then click on the Sharing icon. Select the Services tab and make sure the Personal Web Sharing box is checked. If not, then click the checkbox followed by the Start button to the right. This should also insure that Apache will launch at startup whenever you reboot.

Apache has a folder in which it first looks for documents that you request. Since we are doing everything locally, the url for the document we want is something like

http://localhost/MusicDB/CDdataEntry.html

On the Mac, the location of the file CDdataEntry.html is located in a folder named MusicDB. That folder, in turn, is located in the base folder for web documents that Apache can deliver. That folder is /Library/WebServer/Documents. The base folder we can leave alone, the folder MusicDB and any other folders we want to create are up to us. Generally, I like to keep each project in its own folder, perhaps creating other folders inside these project folders as I see fit.

Here is the picture of the simple web page that serves as the front end of the data entry program (click on the picture to see it in full size).

As the page shows, the user is asked to insert a CD and to click on the button. Clicking on the button has to cause the Apache web server software to launch the perl script we have constructed. This mechanism uses a protocol called CGI (for common gateway interface). Basically, it dessribes a mechanism that lets the application know how to get the data it needs from Apache and it tells Apache how to obtain the data from the cgi-application.

Here is the text version of the HTML document:


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
   "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
<head>
   <meta http-equiv="content-type" content="text/html; charset=utf-8">
   <title>Music CD DB Data Entry</title>
   <meta name="generator" content="BBEdit 7.1.3">
   <style type="text/css">
      body
      { background: url("Images/dataFetchBackground.JPG") }
      table
      { border: 0}
      p.right
      {
         position:fixed;
         left:280px;
         top:60px;
      }
   </style>
</head>
<body>
<h2>Music DB Data Entry</h2>
<form action="http://localhost/cgi-bin/musicDataFetch.pl" method="post">
<p>
<b>Insert a CD into your drive and click the "Get Data" button.</b>
</p>
<p class="right">
<br><br><br>
<input type="submit" value="Get Data">
</p>
</form>
</body>
</html>

You can see that we use the form tag and that the action attribute names a URL.

In our case, clicking on the button sends information to the perl script which we've named musicDataFetch.pl. That script must be placed in a specific folder just as in the case of web documents. Here that folder is /Library/WebServer/CGI-Executables. Several points to make:

The perl script file must have execute permissions. In the terminal applicaton you can do this by executing the commands

cd /Library/WebServer/CGI-Executables
followed by

chmod 755 musicDataFetch.pl

The telnet module used by the perl script: Telnet.pm must be placed in a folder named Net which is placed, in turn, in a folder named lib placed in the CGI-Executables folder.

In an earlier blog I showed the major portion of the perl script. Here, now is the reamining generateHTML subroutine that generates the returning HTML document that Apace will deliver. The CGI protocol dictates that the output of the CGI application must include two leading lines in the output followed by whatever else is being delivered. In our case, because we are returning an HTML document, the first line must be
Content-type: text/html
The second line must be blank.

Here is the generateHTML subroutine:


sub generateHTML
{
##--Front end stuff
print("Content-type: text/html\n");
print("\n");
print("<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\"\n");
print("   \"http://www.w3.org/TR/html4/loose.dtd\">\n");
print("<html lang=\"en\">\n");
print("<head>\n");
print("   <meta http-equiv=\"content-type\" content=\"text/html; charset=utf-8\">\n");
print("   <title>Music DB Data Entry</title>\n");
print("   <meta name=\"generator\" content=\"BBEdit 7.1.3\">\n");
print("   <style type=\"text/css\">\n");
print("      body\n");
print("      {   background: url(\"/MusicDB/Images/dataFetchBackground.JPG\") }\n");
print("      table\n");
print("      { border: 0}\n");
print("   </style>\n");
print("</head>\n");
print("<body>\n");
print("<h2>Music DB Data Entry</h2>\n");
print("<form action=\"cgi-bin/dataEntryScript.pl\" method=\"post\">\n");

##--First table with the CD-specific elements
print("<table>\n");
print("   <tr>\n");
print("      <th>Album Title</th>\n");
print("      <th>Principal Artist</th>\n");
print("      <th>[+]</th>\n");
print("      <th>Recording Label</th>\n");
print("      <th>Year</th>\n");
print("      <th>Cover Art file name</th>\n");
print("      <th>Play time</th>\n");
print("      <th>disc ID</th>\n");
print("   </tr>\n");
print("   <tr>\n");
if ($codeMessage ne "OK") 
{   
   $cdTitleField = $codeMessage; 
   $principalArtistField = "";
   $coverArtFileName = ""; 
}
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"albumTitle\" \n");
print("         value=\"",$cdTitleField,"\" size=\"35\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"principalArtist\" \n");
print("         value=\"",$principalArtistField,"\" size=\"30\"></td>\n");
print("      <td><input type=\"checkbox\" name=\"addPrincipalArtist\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"recordingLabel\" \n");
print("         value=\"\" size=\"15\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"year\" \n");
print("         value=\"\" size=\"6\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"coverArtFileName\" \n");
print("         value=\"",$coverArtFileName,"\" size=\"30\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"TotalPlayTime\" \n");
print("         value=\"",$playingTime,"\" size=\"10\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"discID\" \n");
print("         value=\"",$discid,"\" size=\"11\"></td>\n");
print("   </tr>\n");
print("   <tr>\n");
print("      <td align=\"center\"><b>Physical location</b></td>\n");
print("      <td colspan=\"7\"></td>\n");
print("   </tr>\n");
print("   <tr>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"location\" size=\"35\"></td>\n");
print("      <td colspan=\"7\"></td>\n");
print("   </tr>\n");
print("<table>\n");
print("<br><br><br>\n");

##--Now the second table with the track elements
print("<table>\n");
print("   <tr>\n");
print("      <th>Track</th>\n");
print("      <th>Track title</th>\n");
print("      <th>Artist/Instrument</th>\n");
print("      <th>[+]</th>\n");
print("      <th>Comment</th>\n");
print("      <th>Composer</th>\n");
print("      <th>Lyricist</th>\n");
print("      <th>Play time</th>\n");
print("   </tr>\n");
for ($index = 1; $index <= $noTracks; $index++)
{
   if ($codeMessage ne "OK")
   {
      $trackTitle[$index] = "";
   }
print("   <tr>\n");
print("      <td align=\"center\"><b>",$index,"</b></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"trackTitle",$index,"\" size=\"40\"\n");
print("         value=\"",$trackTitle[$index],"\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"artistInstrument",$index,"a\" size=\"30\"></td>\n");
print("      <td></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"comment",$index,"\" size=\"20\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"composer",$index,"\" size=\"20\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"lyricist",$index,"\" size=\"20\"></td>\n");
print("      <td bgcolor=\"blue\"><input type=\"text\" name=\"playtime",$index,"\" size=\"8\"\n");
print("         value=\"",$trackTime[$index],"\"></td>\n");
print("   </tr>\n");
print("   <tr>\n");
print("      <td colspan=\"2\"></td>\n");
print("      <td bgcolor=\"orange\"><input type=\"text\" name=\"artistInstrument",$index,"b\" size=\"30\"></td>\n");
print("      <td colspan=\"5\"></td>\n");
print("   </tr>\n");
print("   <tr>\n");
print("      <td colspan=\"2\"></td>\n");
print("      <td bgcolor=\"red\"><input type=\"text\" name=\"artistInstrument",$index,"c\" size=\"30\"></td>\n");
print("      <td><input type=\"checkbox\" name=\"addArtistInstrument",$index,"\"></td>\n");
print("      <td colspan=\"4\"></td>\n");
print("   </tr>\n");
}
print("</table>\n");
print("</form>\n");
print("</body>\n");
print("</html>\n");

}

As an example of the resulting web document we have the following image showing a portion of the document delivered by Apache (click the image for full size view).

Music Database Program: A bug-fix

2009-04-29T08:32:00.000-07:00

In the previous entry, I had made an assumption that proved to be false. The program needs to read the CD in order to compute the disc-id. Based on the experience with my computer, I had assumed this would always be identified with the device file /dev/rdisk1. Well, that's not quite true. The device file name that is assigned to the CD drive depends on the order in which the system discovers the drive. If you happen to have connected an external drive (as I had) and then ran this program. The system will have already given out /dev/rdisk1.

The device will be /dev/rdisk- where - will be replaced by a number. There are a number of issues that arise when you try to find out which rdisk number corresponds to the CD ROM drive. The clue is the acroynm: ROM (read-only-memory). I look for a drive that is not writeable. If you have more than one such drive, this will only work if the CD ROM drive is the first one discovered.

The program starts with rdisk1, then rdisk2, up to rdisk5 looking for a drive that is not writeable. If none is found, an error condition exists. If one is found, it is assumed to be the CD drive. Perl has file test operators that will determine if a file is writeable. Unfortunately, writeable or not-writeable means "by the person running the program". If the perl script is executing as a cgi application in a web server environment, then the account under which the execution is taking place is usually a special account set up in the system. This account, for security purposes, has little or no privileges. In particular, it may not have write privilege to a disk even if the disk is writeable. So, here's what I did. I look to see if the drive /dev/rdisk1 (or whichever number I'm checking) exists. If it does I then execute a system command ls -l /dev/rdisk1 and capture the output to a temporary file.

Here's an example of what the output should look like:

cr--r-----   1 avila  operator   14,   6 Apr 29 08:11 /dev/rdisk1

The important part are the first four characters. The c identifies this "device file" as a character special file (that's not particularly important here). The next three characters identify the permissions set for the owner of the file. The first (r) says the owner has read permissions. The second and third characters identify the write and execute permissions. In this case both are the character - which means that even the owner does not have write or execute permissions. This is what we are looking for. If even the owner cannot write to the file, then the device file is not writeable and we will take that to mean this is identifies a CD drive.

Here's the code:


##--Look for a device that is not writeable. It is assumed that 
##--is the CD drive
$diskfile = '/dev/rdisk';
$done = 0;
$found = 0;
$diskNumber = 1;
while ($done != 1)
{
 $CDdiskFile = $diskfile.$diskNumber;

 ##--Check if we have found a non-writeable drive
 ##--Actually it must be non-writeable by the owner. Since 
 ##--the owner is not the id of the web server, we have to be
 ##--more clever to find out if the owner can write or not.
 sleep 5 while !-e $CDdiskFile;
 
 $command = 'ls -l '.$CDdiskFile." >".$listOutput;
 system($command);
 open (LISTFILE, $listOutput) or die "Cannot open ls output\n";
 $permissionString = ;
 
 if (substr($permissionString, 0, 3) eq "cr-") 
 { 
  $found = 1; $done = 1;
 }
 else 
 {
  $diskNumber++;
  ##--Only look for 5 drive numbers
  if ($diskNumber > 5){$done = 1;}
 }
 ##--In either case, remove the list file output.
 $command = 'rm -f '.$listOutput;
 system($command);
}

##--As we exit the while loop check if we actually found the CD drive
##--if not, exit.
if (! $found)
{
 print "Cannot locate the CD drive\n";
 exit 1;
}

A second "bug"

If you are behind a firewall then you have to worry about a second issue. Some firewalls have been set to prevent the use of telnet to access non-local sites. If this is the case, then the program will not be able to access the database from which to extract the album title, track titles, etc.

At least for the moment, I have chosen to ignore that problem.

Music Database Program: The first big step toward a data entry program

2009-04-10T10:33:00.000-07:00

It took a number of things to get to the first step of the data entry program. The idea was to create a web application that would read an audio CD, determine appropriate information about that CD, connect to the freeDB web server to get the album title and track title information, and create a form that the user can use to add data to the music database--with the available data already filled in.

The task of obtaining the necessary information from which we can get the track titles, playtimes, etc. is handed off to the C program cd-discid. In the previous blog entry, I indicated how one can download and install the program using the information on the Darwin Ports web site. The cd-discid program must be able to read the CD once it is inserted into the CD drive. The information that is extracted is the byte offsets into the CD for each track. With this information the disc id can be computed and with the disc id, the freeDB server can be queried to obtain an album title, principal artist, and track title information. This information is then inserted into an HTML document that is the start of the data entry program. The user will be able to add information and/or modify existing information. At the point where the user is satisfied with the data, he clicks on a "submit data" button which will then take the data and begin the process of adding it to the database. I'll have more to say about this later.

The diagram shows the flow of the process described in the previous paragraph. The connector at the end represents the next stage where the user modifies or adds to the data in preparation to placing it in the database.

Walking through the script

Using cd-discid to read frame offsets from the CD

The beginning of the Perl script has to check to make sure a CD has been inserted into the disk drive. On the Mac, once a disc is inserted, the file /dev/rdisk1 is created and is associated with the CD. If you have used the Darwin Ports procedure to install cd-discid, you will find it located in the directory /opt/local/bin. The command shown will take the output of the cd-discid program and place it in a temporary file which we place in the /tmp directory. One critical item that took me some time to debug is that cd-discid needs to have the uid bit set. I'll show you how to do that in a minute. What this does is allow the program to read the CD. By default, the CD "belongs" to the user who inserted it into the drive. The uid issue has to do with permissions. By setting what's called the uid bit, we give permission to the cd-discid program to read anyone's CD.

Here's how to do it. Open a terminal window and issue the command:

chmod u=rws,g=rs,o=rx /opt/local/bin/cd-discid

then issue the command:

ls -l /opt/local/bin/cd-discid

The permission elements at the beginning of the line that is displayed should agree with the corresponding elements in the following line:

-rwSr-Sr-x   2 root  admin  13432 Mar 16 11:38 cd-discid

Here's the script so far:


$diskfile = '/dev/rdisk1';
$cdDiscIdOutput = "/tmp/discidOutput";

##--Wait until the disk has been inserted.
sleep 5 while !-e $diskfile;

##--Execute the cd-discid program to compute the discid and 
##--obtain frame offsets.
$command = '/opt/local/bin/cd-discid /dev/rdisk1 1> '. $cdDiscIdOutput;
system ($command);

Using telnet to communicate with the freeDB server

Now that we have the track offsets and the disc id, we have the information we need to request the available information from the freeDB server. The Telnet package has the Perl software necessary to establish a connection to the freeDB server using telnet. It requires that we create an object which represents the connection. We need to specify two temporary files to collect the telnet commands that we generate as well as the responses. These we create in the /tmp directory as above.

We open a telnet connection to the freeDB telnet server on port 8880 and wait until we receive a reply from the server that contains the numerical code "201". This tells us that the connection is now made. Next we issue a command to the server that starts with "cddb". This is the first command and represents the "handshake" protocol. If you look at the Perl code you see "userMe mySite.com". This needs to be replaced by your e-mail account at the location where your e-mail account is located (e.g. smith yahoo.com). If all is well, the response will start with the code "200". (So far, I have not decided what to do if something goes wrong with establishing the connection.)

If all is well to this point then I have to read the file with the file that I got from cd-discid. There is some fairly standard Perl to pull apart the data string from the temporary file and to grab the portions that we need. Even before we query the freeDB server, we can already compute the playtime for each track. If we know the number of frames on the track, we can use the fact that there are 75 frames per second to determine the length of each track. The subroutine sec2minSec converts the track length data into a formatted version of the form [mm:ss]. The final piece of this part of the script is to form the command to the freeDB server to seek a match for a CD with the given disc id and track offsets that we have extracted from the given CD.


$telnetDumpFile = "/tmp/telnetDumpFile";
$telnetInputLogFile = "/tmp/telnetInputLogFile";

##--Establish an object to communicate via telnet
use Net::Telnet ();
$t = new Net::Telnet (Dump_Log => $telnetDumpFile, 
 Input_log => $telnetInputLogFile);
$t->open(Host => "freedb.freedb.org", Port => "8880");

##--Wait for the connection to be established.
$t->waitfor('/201.*$/');

##--Send the first command to establish a connection to the 
##--database server.
$t->cmd(String => "cddb hello userMe mySite.com telnet v0.0",
 Prompt => '/200.*$/');

##--Read the file into which cd-discid placed the discid and 
##--offsets
open(discidAndOffsets, $cdDiscIdOutput) || 
 die "Can't open discidAndOffsets";
$discidPlus = <discidAndOffsets>;

## Grab the discid. This will be used to query the database.
@discData = split /\s+/, $discidPlus;
$discid = $discData[0];

$noTracks = $discData[1];

##--Compute the total number of minutes, seconds for the album
$timeTotal = $discData[$noTracks + 2];
@minSec = sec2minSec($timeTotal);
$playTimeTotalMin = $minSec[0];
$playTimeTotalSec = $minSec[1];

##--Now compute the play time for each track.
$playTime[$noTracks] = $timeTotal - $discData[$noTracks + 1]/75;
@minSec = sec2minSec($playTime[$noTracks]);
$playTimeMin[$noTracks] = $minSec[0];
$playTimeSec[$noTracks] = $minSec[1];

for ($index = 1; $index < $noTracks; $index++)
{
 $playTime[$index] = ($discData[$index+2] - $discData[$index+1])/75;
 @minSec = sec2minSec($playTime[$index]);
 $playTimeMin[$index] = $minSec[0];
 $playTimeSec[$index] = $minSec[1];
 if ($debug) {print $index, ": [", $minSec[0],":",$minSec[1],"]\n";}
}

##--Query the database with the discid and frame offsets.
$cddbQuery = "cddb query ".$discidPlus;

$ok = $t->cmd(String => "$cddbQuery", Prompt => '/2[0,1][0-2].*$/');
## Since there is no prompt character issued by the host
## this is all we can do to capture the returning output
$cddbOutput1 = $t->last_prompt;

Extracting the data from the query reply

The output from the last command shown at the end of the previous segment of the script will contain all the information that we want. Our job is to extract the information using a lot of string manipulation to do it. The code is self-explanatory. There are some possibilities in the response to the query that might happen that I haven't yet dealt with as you can see. Also, the last thing we do is generate an HTML document to contain the information that the user will see.

I'll start here next time and describe how the data entry program will function.


@discInfo = split /\s+/, $cddbOutput1;
$code = $discInfo[0];
$genre = $discInfo[1];

if ($code eq "202")  ## no match
{
 $t->close;
 print "No match\n";
 exit 202;
}
elsif ($code eq "211")  ## fuzzy match 
{
 $t->close; 
 print "Fuzzy match\n";
 exit 211;
}

## If we fall through, then there is at least one exact match.
## Knowing the genre and discid, read the data for this disc
$cddbQuery = "cddb read ".$genre." ".$discid;
@cddbOutput2 = $t->cmd(String => $cddbQuery, Errmode => "return", 
 Prompt => '/\.$/');

##--Gather the principal artist and album title
$trackNumber = 0;
foreach $line (@cddbOutput2)
{
 chop $line;
 if (index($line, "TTITLE") != -1)  #--Found a track title line
 {
  $trackNumber++;
  $marker = index($line, "=");
  $trackTitle[$trackNumber] = substr($line, $marker+1);
  $trackTime[$trackNumber] = " [".$playTimeMin[$trackNumber].
   ":".$playTimeSec[$trackNumber]."]";
 }
 elsif (index($line, "DTITLE") != -1) #--Found the disc title line
 {
  $marker = index($line, "=");
  $fields = substr($line, $marker+1);
  ($principalArtistField, $cdTitleField) = split /\//, $fields;
  $playingTime = "[".$playTimeTotalMin.":".$playTimeTotalSec."]";
  ##--remove white space and certain other characters 
  ##  from the title string
  @cdTitleWords = split /[ .,:;()]/,$cdTitleField;  #/
  $compactTitle = join '', @cdTitleWords;
  $coverArtFileName = $compactTitle."_".$discid.".png";
 }
}
$t->close;

##--Cleanup
if (! $debug)
{
 $command = 'rm -f '.$cdDiscIdOutput;
 system($command);
 $command = 'rm -f '.$telnetInputLogFile;
 system($command);
 $command = 'rm -f '.$telnetDumpFile;
 system($command);
}

generateHTML();
exit 0;

Music Database Program: The Elephant in the Room

2009-03-28T21:18:00.000-07:00

Even though we are still working on the design and attempting to determine how to utilize existing available software (such as NeoDatis), the large object in the middle of the room is the data entry element of the database. The utility of the database will require that we have the most comprehensive set of music data that we can supply. Yet, if we have to enter every field for every musical element, no one will want to use this software.

So, in this entry, we want to look at how to automate, as much as possible, the task of entering data into the database. First of all, there are already existing (and free) online CD databases. This is where we will start. Generally, we have in our database many more elements than just the track titles, times, and some artist information for each album. Nevertheless, if we can start with the data that is available online, we will only have to enter those elements that extend that data.

There is a database called freeDB which contains a large repository of information on music CDs. In particular, the data available consists of album titles and principal artists, track titles and playing times, and musical genre. In order to identify the CD, you need some information from the disc itself.

The basic identifier for a CD is something called the disc id. This number is obtained by an algorithm that utilizes the information available on the CD. This consists primarily of the numbers that represent the frame offsets to the individual tracks on the CD. You can find out more about this at the MusicBrainz web site. There's even a C program called cd-discid that you can download and install that will read an audio CD and generate the disc id. If you are on a Mac, you can find what you need to install it at the Darwin Ports web site.

There are a number of ways of obtaining the data from the freedb database. freedb.org even provides the ability to obtain the information using a simple front end. I want to automate the search so that I can insert a CD, the software will automatically pick off the data on the CD that I need to query the freedb database, make the query, collect the results and place the results in appropriate fields, leaving me to fill in missing fields or to modify the pre-filled data if necessary. The manual interaction with the database using the freedb.org front end won't suffice.

There is another mechanism to access the freedb database and that is to use telnet to query the database. One establishes a connection to the freedb database server and interacts with the server using the cddb protocol. There is a very useful document that you can get from the freedb website by clicking on the freedb howto v1.07 link on the freedb website.

This is the mechanism that seemed the most reasonable; utilize available tools and software as much as possible and choose methods that can be automated. It would be my preference to be able to provide a Java program to do this but for starters cd-discid is in C and the mechanisms in Java to communicate with C are arcane. The most sensible is to use a scripting language. Perl is what I chose. In a Perl script, I can call the cd-discid program. Using telnet from within a Perl script requires some tricky programming involving sockets. Fortunately, there is a telnet library which I was able to use.

So now I have a script that first waits until the CD has been inserted into the drive, then it calls the cd-discid program and captures the output of that program. At this point telnet is used to connect to the freedb database server and a series of queries yields the desired data.

There is still work to do on the script. What if there is no match with the discid? What if there are multiple matches? What if there are no matches but there are "close misses"?

Next, I want to turn the Perl script into a cgi application that will allow me to interact with a browser to let me view the existing data, add missing fields, modify any of the data items that are incorrect and then save the resulting data for the CD into my object-oriented database.

I'll post more when I've put these elements together.

Music Database Program: A NeoDatis Test

2009-03-09T13:16:00.000-07:00

We will need to be able to store and retrieve data from our database. Moreover, we want to be able to store the data as objects and not have to convert the object data into tables as in a relational database model. The NeoDatis object database engine appears to be a good candidate in our quest for support for this music database program. In reality, our plan is to use the software more as a "persistence" engine. We will be able to store the objects of our program in a way that allows us to restore them at a later time while preserving the references between objects in our model. But first, we had to be able to see how this might work and if, in fact, it worked as advertized.

In our tests we had to create a number of objects, store them, come back after exiting the program to see if the data had, in fact, been stored in a persistent manner. Moreover, we had to test to make sure the connection (references) between objects was preserved.

As part of this test we added another class to those described earlier. This class is the Album class. It's similar to the earlier classes in that it represents one of the data objects that make up our model. We've supplied a number of fields for this class but there may be more later. We have also gone back and made some changes to the classes we previously defined. This, too, will be described later. Generally, this is part of the process of discovery as we design and build the program.

There were two Neodatis database software tests. In the first test, we created and stored a number of objects. In the second test, we retrieved the objects created in the first test and demonstrated that they were in fact whole; that is, that they behave as they did before they were stored and later reconstituted. One of the great features of the NeoDatis software is that when it stores an object, all objects that are referenced by that object (and all those referenced by those, recursively) will be stored as well. This greatly simplifies the job of saving the data.

One of the modifications to the Artist class was to add a field of type LinkedList that contains a list of all the albums on which that particular artist appears. When we retrieve the Artist objects, we should be able to display the information about the albums associated with any artist and to see the tracks on that album. All this should be possible without any additional work on our part in storing the objects.

In the first test we created two Album objects. The Albums contain references to Artists. The Artist objects contain lists that reference the albums. Since NeoDatis will traverse an object that it has been asked to store and automatically store all the objects referenced within the object (recursively), we should be able to store the album objects and get everything else stored along the way. This is because the Album objects contain references to track objects which contain references the performer objects which contain references to the Artist and Instrument objects. All this by just storing two Album objects!

The second test required that we ask NeoDatis to recover the Album objects. NeoDatis makes this easy by allowing us to ask for all objects of a particular class. Just to make sure the Artist objects have been retrieved, we display the name of each artist. Next we retrieved the two Album objects. From this collection of albums, we chose one and then displayed the titles of each of the tracks on that album.

The good news is that once we discovered a basic error in the NeoDatis documentation, we were able to carry out the tests successfully.

Here are some of the details:

This is now the set of data fields for the Artist class:


    private String name = null;
    private LinkedList<Album> albums = null;

The new Album class has the fields:


    private String name = null;    //--the name of the album
    private Artist[] principalArtist = null;
    private String location = null;    //--Where to find the physical album
    private int numberOfTracks = 0;
    private String label = null;
    private List<Track> tracks = null;

There are the usual constructors, getter and setter methods.

Here is the code that stores the data into the NeoDatis database...


    //--Open the database
    ODB odb = ODBFactory.open("MyMusicDB"); 

    //--Store the two albums in this test
    odb.store(album1);
    odb.store(album2);

    //--Close the database 
    odb.close();

This is it! That's all that's required to store everything!

And here is the retrieval of the data elements we want to check from the second test:


//--Open the database.
ODB odb = null; 
try
{ 
    //--Open the database 
    odb = ODBFactory.open("MyMusicDB"); 
        
    //--Get all object of type Artist 
    Objects<Artist> objects = odb.getObjects(Artist.class); 
    System.out.println(objects.size() + "  artist(s)"); 
        
    //--display each object 
    while(objects.hasNext())
    { 
        Artist a = (Artist)objects.next();
        System.out.println("    " + a.getName()); 
    } 
    
    //--Get the albums, pick out album2 and display its tracks.
    Objects<Album> albumObj = odb.getObjects(Album.class);
    
    while (albumObj.hasNext())
    {
        Album album = (Album)albumObj.next();
        if (album.getName().equals("The Essential Stan Getz"))
        {
            System.out.println("Album/Tracks");
            List<Track> tracks = album.getTracks();
            ListIterator<Track> iter = tracks.listIterator();
            while(iter.hasNext())
            {
                System.out.println("    " + iter.next().getTitle());
            }
        }
    }
}finally
{ 
    //--Closes the database 
    if(odb!=null)
    { 
        odb.close(); 
    } 
}

Music Database Program: Four Data Classes

2009-02-25T20:14:00.000-08:00

It's been too long since I have had a chance to work on this project. Too many other projects. The "friction losses" are always larger than you think they will be when you have been away from something for this long. In this entry we are looking to create enough of the classes with enough of the detail so that we can check out the Neodatis database program.

Here are the first few data classes: the Instrument class, the Artist class, the Performer class, and the Track class. If you compare this post to the previous one, you will see that there have already been some changes reflecting more thought about the program. For example we decided to keep an artist and the instrument separate so that an artist could appear on a track with one instrument and on another track with a different instrument.

The Instrument class represents an instrument such as a trumpet or piano. If we had the names of all the instrument types we would ever use, then we would probably use an enumerated type. This would make it less likely that we would have two classes associated with a single instrument type. Unfortunately, we don't know that there won't be some instrument that we had not anticipated. Doing things this way lets us add instruments at run-time if we need to.

The Artist class has to represent the elements of an artist that we want to preserve in our database. It is tempting to associate an artist with a particular instrument but unfortunately, many artists use multiple instruments. It's easier to have an artist consist of the name of the artist and some background information (such as a picture, a biography, discography). Perhaps we will have a link to the Wikepedia article about an artist (if one exists). The Performer class represents the pair of artist and the instrument that are used on a particular track. It would be tempting to make the Performer class a subclass of Artist. The reason for NOT doing that is that we want to have a single Artist object for each artist regardless of how many instrument he or she plays

Finally, the Track class represents one track on a CD or record. It will contain several pieces of information related to the track such as the name of the tune on the track, the running time in minutes and seconds, and a list of performers on the track. In the final version of this class, we may add other fields such as the composer and any other pieces of information we want to store related to a particular track. For now we will just keep it simple.

In each of these classes we have included a static main method which gives us the ability to test each class as we develop the program.

The Instrument class...



public class Instrument
{
    private String instrumentName;
    
    /**
     * 
     * @param name is the name of the instrument (e.g. Piano)
     */
    public Instrument(String name)
    {
        instrumentName = name;
    }

    public String getInstrumentName()
    {
        return instrumentName;
    }
    
    /**
     * @param args
     */
    public static void main(String[] args)
    {
        //--Test the Instrument class
        
        Instrument piano = new Instrument("Piano");
        Instrument trumpet = new Instrument("Trumpet");
        
        System.out.println("piano: " + piano.getInstrumentName());
        System.out.println("trumpet: " + trumpet.getInstrumentName());

    }

}

The Artist class...



public class Artist
{
        
    private String artistName = null;
    
    public Artist(String name)
    {
        artistName = name;
    }
    
    /**
     * Get method
     * @return the name of the artist
     */
    public String getName()
    {
        return artistName;
    }
    
    /**
     * @param args
     */
    public static void main(String[] args)
    {
        /*
         * Create an object of type Artist
         */
        Artist bill = new Artist("Bill Evans");
        
        /*
         * Print the contents of the new object using the GET methods
         */
        System.out.println("Artist name = " + bill.getName());
    }

}

The Performer class...


/**
 * A performer will consist of an artist and an instrument.
 * 
 * @author avila
 *
 */
public class Performer
{
    private Artist artist = null;
    private Instrument instrument = null;
    
    /**
     * 
     * @param artist 
     * @param instrument
     */
    public Performer(Artist artist, Instrument instrument)
    {
        this.artist = artist;
        this.instrument = instrument;
    }
    
    /**
     * 
     * @return the artist
     */
    public Artist getArtist()
    {
        return artist;
    }
    
    /**
     * Getter method
     * @return the instrument associated with this performer
     */
    public Instrument getInstrument()
    {
        return instrument;
    }
    
    /**
     * @param args
     */
    public static void main(String[] args)
    {
        //--Create a few Instrument objects that will be used for the Music DB
        Instrument piano = new Instrument("Piano");
        Instrument bass = new Instrument("Bass");
        Instrument drums = new Instrument("Drums");
        
        //--Create some Artist objects
        Artist billEvans = new Artist("Bill Evans");
        Artist scottLaFaro = new Artist("Scott La Faro");
        Artist paulMotion = new Artist("Paul Motion");
   
        //--Some Performer objects
        Performer performer1 = new Performer(billEvans, piano);
        Performer performer2 = new Performer(scottLaFaro, bass);
        Performer performer3 = new Performer(paulMotion, drums);
        
        //--Display the contents of the generated Performer objects
        System.out.println("Performer performer1: " + performer1.getArtist().getName() +
                performer1.getInstrument().getInstrumentName() );
        System.out.println("Performer performer2: " + performer2.getArtist().getName() +
                performer2.getInstrument().getInstrumentName() );
        System.out.println("Performer performer3: " + performer3.getArtist().getName() +
                performer3.getInstrument().getInstrumentName() );
        

    }

}

The Track class...


import java.util.*;

public class Track
{
    private String title;
    private String length;
    private List performers;
    
    public Track(String title, String length, 
            List<Performer> performers)
    {
        this.title = title;
        this.length = length;
        this.performers = performers;
    }
    
    /**
     * @return track title
     */
    public String getTitle()
    {
        return title;
    }
    
    /**
     * @return track length (min:sec)
     */
    public String getLength()
    {
        return length;
    }
    
    /**
     * @return list of performers on this track
     */
    public List<Performer> getPerformers()
    {
        return performers;
    }
    
    /**
     * @param args
     */
    public static void main(String[] args)
    {
        /*
         * Test the Track class
         */
        
        /**
         * and the Instrument objects we need
         */
        Instrument piano = new Instrument("Piano");
        Instrument bass = new Instrument("Bass");
        Instrument drums = new Instrument("Drums");
        
        /**
         * Create some artists
         */
        Artist billEvans = new Artist("Bill Evans");
        Artist scottLaFaro = new Artist("Scott La Faro");
        Artist paulMotion = new Artist("Paul Motion");
        
        /*
         * Create the list of performers for this track
         */
        List performers = new LinkedList();
        performers.add(new Performer(billEvans, piano));
        performers.add(new Performer(scottLaFaro, bass));
        performers.add(new Performer(paulMotion, drums));
        
        /*
         * Create the Track object
         */
        Track one = new Track("Israel", "6:08", performers);
        
        /*
         * Now test the get methods
         */
        String trackTitle = one.getTitle();
        System.out.println("Track title: " + trackTitle);
        
        String trackLength = one.getLength();
        System.out.println("Track length: " + trackLength);
        
        List trackPerformers = one.getPerformers();
        System.out.println("Performers/Instruments: ");
        ListIterator<Performer> iter = trackPerformers.listIterator();
        while (iter.hasNext())
        {
            Performer performer = iter.next();
            System.out.println(performer.getArtist().getName() + " : " +
                    performer.getInstrument().getInstrumentName());
        }
    }

}

Music Database Program: Setting up an early design test

2008-11-22T22:12:00.000-08:00

Let’s imagine the database elements. We have albums. They have a title, a principal artist or group, a list of tracks and additional information such as recording dates and where the album is located (so that we can actually find it physically). We may discover additional items of information that we need as we develop the design. The tracks each have a title, the length of the track, a list of artists, the instruments they play, the composer, and perhaps additional information we will find we want.

At this point we are trying to develop a design. That is, we want to determine the objects in our program and their relationships with each other. As we develop the design, we will build some software elements that we will use to test ideas and perhaps to form the start of the program itself.

Let’s try to identify some of the classes of objects we will be working with. The idea here is not to nail down all the detail but rather to discover as many classes as we can and to think carefully about the inter-relationship of these classes.

We will need an Album object. That object contains a list of (references to) Track objects. Track objects will carry the information for a single track on a particular album. An Album will have a reference to an Artist object (the principal artist). Here we will have to think about how to handle groups of artists as a single object (e.g. the New York Philharmonic). Each Track object will have a reference to a list of artists that perform on that particular track.

The query: “find all occurrences of the title: ‘Waltz for Debby’” will result in searching the titles of each track on each album for that string. We should get a list of Album, Track pairs as the response to the query so that we can not only display each track but display the album information including the physical location of the album. I can see a screen showing the cover art for one album with a list of the tracks with the relevant one highlighted. In addition, there are Next and Previous buttons that allow us to move forward or backward in the list. If we want information about the album (such as its physical location) we double-click the cover art and a separate window will let us choose what we want to see about the album.

A first step in sorting some of this out is to build a very small set of objects in a preliminary version of our music database and test to see how we would use NeoDatis as our search engine. We’ll start with a description of a few classes. Then in the next blog entry we will describe the testing of these objects in the NeoDatis context.

The Track class:
Data:
Title (String)
Length (String -- min:sec)
Performers (List of Artist objects)

Methods:
Get/Set methods for the data

The Artist class:
Data:
Name (String)
Instruments (List of instruments (enum type) played on this track)

Methods:
Get/Set methods for the data

The Album class:
Data:
Album name (String)
Principal Artist (Artist object)
List of tracks (List of Track objects)
Length of Album (String -- min:sec (computed from track info))
Physical location of album (String)

Methods:
Get/Set methods for the data

A couple of final notes: the Set methods will be used to initialize database elements. Also, the fact that this is a database application means there is no need of methods (at least at this point in the design) other than the Get/Set methods.

Music Database Program: An Object Oriented Database

2008-11-19T17:08:00.000-08:00

Before we commit to a design, however, we have a lot more work to do. If I build the object-oriented database myself, I will be relying on object serialization to store the objects on disk in between uses of the program. The reading of serialized data each time the program is run will be time consuming. There are OODBs already out there that are open source. Maybe it would be better to build on top of such software where many of these issues have been dealt with.

I did some online research and found what may be an answer: NeoDatis - a native object database. The fact that this is an opensource project makes it attractive. One can learn a lot from the source code. Presumably, some of the issues have been resolved already about accessing the data base in an efficient manner.

A first look at NeoDatis (the five minute tutorial on their web site) makes it appear that the OODB has been written to look a lot like a relational database. In particular the queries appear as method calls that look like SQL queries. I need to think about this and see if I can map the music DB program into a form for which this works.

Music Database Program: Introduction

2008-11-17T17:54:00.000-08:00

Over the years I’ve accumulated many CDs. In fact, the collection has grown large enough that I’ve lost track of exactly what I have. Often I’ll want to find something that I remember having but I don’t remember where the CD is located. Sometimes I may remember something about the track that I want to hear but can’t remember the artist or the title of the CD. Well, isn’t that what databases are supposed to be for?

Of course, the easy answer to the “problem” is to rip the music collection to MP3 files and use iTunes (or a similar application) to handle the data (and the playing of it). That is an option, but then I wouldn’t have the fun of putting the software together and learning some things along the way. There is also the fact that some of the queries don’t map easily into the choices I have in iTunes.

The usual approach for the use of databases is to use one of the freely available high-quality relational database systems (such as mySQL) and then write a front-end to the database system with a nice graphical user interface. I thought it would more interesting to develop instead an object-oriented database (OODB). The idea is to tie the data together in a set of inter-related objects in such a way as to be able to locate the information you want in a few steps.

For example, suppose I want to find a track that has Bill Evans playing “Waltz for Debby”. One object will contain a list of all CDs in which Bill Evans plays one or more tracks. Another object contains a list of all CDs on which “Waltz for Debby” is played. Assuming these lists are relatively small (how many versions of “Waltz for Debby” could I have?), the intersection of the elements in these two lists can be computed in O(n²) for a small value of n.

Well, that seems like a good start but let’s explore the kinds of queries I want to make of this database. Suppose I want to find anything written by George Gershwin. According to the model, we have an object that contains a list of all CDs with at least one track composed by Gershwin. Suppose I want something recorded in the last 5 years originally written by Gershwin. Now the problem is a little more difficult because the value of 5 years is arbitrary and perhaps tomorrow I will want that number to be 3 years or 10 years. I’ll need a way to get a list of recordings based on the date they were recorded. The idea is the same (an object that contains a list of CDs that meet the criterion) but with the additional aspect that the list is ordered by the recording date. Now we take the intersection of the Gershwin list with the list of CDs recorded in the last 5 years.

The concept is to associate each property (some aspect of the database that provides a criterion for searching) with an object that contains a list of those CDs with that property.

One choice that I’ve made that deserves more scrutiny is the use of the CD as the atomic element of the database and not the track. It seems that most of the tracks on a CD will share the same properties (artists, instrument, time of recording, etc.) so that keeping the CD together avoids scattering the more fine-grained data (the tracks) into an assortment of objects. The price of extracting the relevant tracks once we have narrowed the search down to CDs seems small. Besides, ultimately, I will need to locate the CD if I want to play any of its tracks.