DEMO

How I Did It

Part I: The Data

===================

Parsing the data

Unfortunately, as of May 2010 the data was provided in a PDF format [example PDF] and had to be converted into a spreadsheet/database before geo-coding could be done. Using Adobe Acrobat’s Export to Plain Text feature I was able to produce a file that I could parse (the HTML and DOC exports were unable to produce consistent tables due to the lack of vertical lines on the tables). Luckily the text file was formatted so that each row was on a separate line and columns (for the most part) were delimited by a tab. Using the following regular expression and some PHP I was able to convert 15 months worth of PDFs into a MySQL table.

// REGEX
^(.+?)  (.+?)(?:10|09)-.+?  (.+?)  (.+?)(?:Active|Inactive|Arrest|Unfounded).*?  (.*)

Cleaning the data

After looking through the data that was collected I noticed that the format used for displaying the location had changed sometime during the year. Locations were sometimes written BUILDING NAME – ADDRESS while other times listed as ADDRESS (BUILDING NAME). In order to geo-code the data I needed to separate the address from the building name. I decided to add a `building` field to my MySQL table which I would populate by parsing the location field for each record. I used the following PHP and regular expression to do so:

$l = 'Location From the DB'

if(strpos($l,'(') !== false){
    $loc = preg_replace('/\s*\(.+?\)\s*/im', '', $l);

    if (preg_match('/\((.+?)\)/im', $l, $regs))
        $name = $regs[1];
}
elseif (strpos($l,'-') !== false) {
    $c = explode('-',$l);
    $name = trim($c[0]);
    $loc = trim($c[1]);
}

While this does not take into account hyphenated building names and other edge cases, it provided adequate results for the project and was definitely preferred to manually parsing the data.

Geo-coding the data

The next step of the process was to geo-code (i.e. determine the latitude and longitude) for each of the crimes. I turned to Google for this task and wrote a PHP script to query the Google Geocoding API and save the resulting data to the MySQL table. By only geocoding DISTINCT locations and issuing an UPDATE on all crimes that shared a particular location I was able to drastically cut down on the number of geocoder lookups needed.

For each lookup Google provides an accuracy value that indicates how sure Google is about the result. By saving this value to the database I was able to determine which locations needed more attention. Some of the entries were misspelled or corrupt, while others didn’t include adequate location information. In particular, many locations just had a BUILDING NAME without an address. In addition to this many buildings are located in the middle of campus (not near a road). To remedy these issues I turned to the Vanderbilt Map which makes use of a GML file containing the latitude/longitude coordinates for each building on campus (I discovered the file using the Firefox Tamper Data plugin). After a few tweaks to the file, PHP was able to parse it and provided me the geocoding data for all buildings on campus. I saved this parsed data into my database and used it to update my crimes table.

<?PHP

$buildings = simplexml_load_string(file_get_contents("vu.gml"));

foreach($buildings as $building) {
  $building_name = $building->facilities->FACILITY_NAME;
  $coords = $building->facilities->msGeometry->Polygon->outerBoundaryIs->LinearRing->coordinates;
}

?>

Spreading out the data

Since there exact locations that appear more than once, there are also multiple latitude/longitude pairs that are the same. This becomes problematic when visualizing the data on a map because all of crimes that occurred at the same location are plotted on top of one another are covered by the top marker. In order to provide a more useful visualization I moved all pins that were at the same lat/lon by a random amount (between 0 and a hundred feet) in a random direction. I accomplished this using the below SQL query.

$result = mysql_query("SELECT count(*), lat, lon FROM `crimes` GROUP BY lat, lon HAVING count(*) > 1 ORDER BY count(*) desc");

while($r = mysql_fetch_assoc($result)){
    mysql_query("UPDATE crimes SET lat = (lat + (RAND() * 0.0002516197121987) - 0.00012580985609934), lon = (lon + (RAND() * 0.0002516197121987) - 0.00012580985609934) WHERE lat=".$r['lat']." AND lon=".$r['lon'];
}

Each team was assigned a single workstation in a classroom with 5-7 other teams. The workstation was not connected to the internet and the use of other electronic devices was not allowed. The competition stipulates that solutions may only be submitted in Java, C, or C++. Fortunately, Eclipse with code assist and Javadoc was available, making the precious minutes we actually got to program very enjoyable. When the packet of problems was handed out we quickly scanned through them to determine which could be solved quickly and which would require a longer time commitment. We did this by assigning certain problems to each member of our 3 person team so we could each screen for difficulty, and code-up a solution for any really easy ones we found. We used modular arithmetic to divide the problems. I was assigned those that the problem number mod 3 equaled 1 (1, 4 and 7). Since we had to share a single terminal, most of our time was spent writing out code (pseudo-code really) on paper and stacking it up to be programmed when the PC became available.

Each program could be run locally against a set of test inputs to determine if the output was correct and once the problem was submitted to the judges, the program was tested against a much larger set of inputs to verify correct functionality.  A response was usually received by the judges within about 30 seconds stating whether or not the submitted solution was correct. I submitted the Gnome problem (#1) within the first 15 minutes (after waiting my turn for the computer) and worked the remainder of the time predominantly on the Cell problem (#4) which involved calculating the signal strength of cellular towers at precise intervals located along an arbitrary route.

After the competition we devoured lots of free pizza and sodas before heading to the auditorium to hear the results. Each member of each team was presented with a certificate and the top 3 teams were awarded prizes. For first place, my team received IBM water powered clocks.

Check out the article published by the Vanderbilt School of Engineering:

“Vanderbilt computer science team places first in regional round of IBM’s Battle of the Brains”

My first response was to search the Settings menu for a backlight option. Unfortunately changing the backlight setting to “Never” was useless. So my options were either to remove the Zune from the dock or try to cover up the display to reduce the light. Because Microsoft failed to include a sleep timer, the first option doesn’t make sense if you’d like to wake up to a charged mp3 player. So while I waited for months on end for Microsoft to release updated firmware that would remedy my dilemma, I found myself scavenging each night for a book or other dense objects that could block out the Zune’s display from lighting up my otherwise dark room.

As the weeks turned into months, my hope that Microsoft would fix this bug/feature diminished. Because physically blocking the screen wasn’t completely effective, I continued searching for workarounds. Using my programmer mentality I looked into writing my own software for the Zune that would cut off the display. This was another dead end as the Zune’s API doesn’t support that. My next move was to load a solid black image into the Zune photo gallery and display it fullscreen when going to sleep. That worked fairly well and was a great alternative to having the album art displayed.

Recently after trying to download an open source sleep timer to my Zune, I discovered the best workaround yet — the TV output mode. Apparently Microsoft thought it was necessary to have the Zune capable of being displayed on a TV using special cords or a dock. While I doubt I’d ever want to hook up my Zune to a TV, the TV output mode has an interesting property — it redirects the video from the Zune’s display to the TV. Luckily the Zune doesn’t check to see if it’s actually connected to a TV, and shuts off its display regardless. Finally a way to preserve the pitch black state of my room without sacrificing the Zune’s battery or my sanity. Now if Microsoft would only provide a sleep timer, I’d be set.