fixedd.com Full Site Feed

Jyte Statistics 1: Text-Based

2008-08-07T00:54:02Z

I was asked by some of the users of my Jyte Stats/Charts if I would share the source code for my different Jyte Statistics projects. In lue of that I've decided to write an article explaining what I did. You're free to do whatever you want with the code in the articles, but I'd ask that you credit me if you make use of it.

For the all of the articles I'm going to use the following conventions:

Black Backgrounds represents commands that is to be typed into a shell (command line) prompt. There will be one command per line.
Yellow Backgrounds indicates that the enclosed text is a filename.
Blue Backgrounds represent actual source-code.

For the first article in the Jyte Statistics series we're going to cover the text based statistics as most of the concepts that the other charts use as demonstrated here. As I'm still pretty new to Rails myself I'm not going to say that anything I show you below is the best way to do things, it's just how I did it.

Before we get started I'm going to assume a few things;

That you have Ruby and Rails installed.
That you have some basic understanding of both Ruby and Rails.
That you have SQLite3 (prefered for simplicity) or some other database available and setup.
That you can configure Rails to use the aforementioned database (if it isn't SQLite3).
That you're using a version of Rails that is at least 2.0.0.
That you can either perform this exercise in a Linux environment or figure out how to translate it to your environment.

Now, let's begin.

Create a new Rails app named myapp and kill the pesky ./public/index.html file.

rails myapp
cd myapp
rm ./public/index.html

I don't want to be constantly hitting the Jyte servers when the stats don't really change all that quickly so I'm going to store everyone's data in a database and only refresh it every 15 minutes.

To access the data we'll need to create a new model named Jyter:

./script/generate model Jyter

Make the generated class file (/app/models/jyter.rb) look like this:

class Jyter < ActiveRecord::Base
    validates_uniqueness_of(:identifier)
    validates_numericality_of(:claims, :comments, :agree, :disagree, :reverse, :total_votes)
    validates_presence_of(:identifier, :name, :claims, :comments, :agree, :disagree, :reverse, :total_votes)

    # this will return the first (and supposidly only) record for the supplied identifier
    def self.find_by_openid(identifier)
        return find(
            :first,
            :conditions => {:identifier => identifier}
        )
    end
end

and edit the generated migration (./db/migrate/001_create_jyters.rb) so it reads:

class CreateJyters < ActiveRecord::Migration
    def self.up
        create_table :jyters do |t|
            t.timestamps

            t.column :identifier,        :string
            t.column :name,         :string
            t.column :claims,        :integer
            t.column :comments,        :integer
            t.column :agree,        :integer
            t.column :disagree,        :integer
            t.column :reverse,        :integer
            t.column :total_votes,         :integer
        end
    end

    def self.down
        drop_table :jyters
    end
end

Now Rails (or more specifically, Rake) knows how to migrate the Jyters table.

Next tell Rails how to talk to your database (I'm using SQLite3 here) by editing ./config/database.yml (this will be the default for many people):

development:
  adapter: sqlite3
  database: db/development.sqlite3
  timeout: 5000
test:
  adapter: sqlite3
  database: db/test.sqlite3
  timeout: 5000
production:
  adapter: sqlite3
  database: db/production.sqlite3
  timeout: 5000

Now that Rails/Rake know how to talk to the database and how to build our table we tell them to do it:

rake db:migrate

Ok... that's it for the Model/database. On to the Controller:

Firstly I built a number of methods into the ApplicationController class in ./app/controllers/application.rb. Some of these didn't really need to be seperate methods, but I thought it handy since I reuse this code for all the Jyte Statistics projects. I'll briefly describe them now:

This one's dead simple, it simply divides the first number by the second to give us our ratio:

def get_ratio(first, second)
    return (first.to_f / second.to_f)
end

The next method formats the numbers so they'll look pretty. It makes sure everything only has 2 digits after the decimal and it handles infinite and division by zero conditions:

def jyte_format(to_format)
    to_format = 0.00 if to_format.nan? # make sure it's a valid number
    to_format = "Infinity" if to_format.infinite? # make sure we catch infinites
    to_format = sprintf('%0.02f', to_format) if to_format.is_a?(Float) # format the float to only have 2 decimal places
    return to_format
end

Here's where we're starting to get into a little logic. This method asks the database if we have a record for this user that's less than 15 minutes old and if we don't then it updates.

def get_jyte_stats_hash(identifier = nil)
    return nil if identifier.nil?

    user = Jyter.find_by_openid(identifier)

    # do we have a record, or a recent enough record, for the user to use?  if not create it!
    if (user.nil?) or (user.updated_at < (Time.now - 15.minutes))
        # create/update the user
        user = record_jyte_profile(identifier)
    end 

    return user
end

This is where the fun takes place. The short description for what's going on is that we're grabbing a user's profile page from Jyte and extracting the information we want from it. I've tried to include enough comments that you'll be able to understand what's going on.

def record_jyte_profile(identifier = nil)
    return nil if identifier.nil?

    # if the user already exists, grab them
    user = Jyter.find_by_openid(identifier)
        
    # pull in the library we'll use for making the HTTP connection to Jyte
    require 'net/http'
    
    data_from_jyte = nil
    
    # get the user page from Jyte
    response = Net::HTTP.get_response(URI.parse('http://jyte.com/profile/' + identifier))
    case response
        # on Success continue
        when Net::HTTPSuccess 
            resp = response.body    
        
            # grab the user's current name (alias)
            if resp =~ /title>Jyte - Profile for (.+)</title/
                name = $1
            
                # pull out all the numberic data
                if resp =~ /has made (d+) claims?, +(d+) comments?, +agreed with (d+) claims?, +disagreed with (d+), and changed positions +(d+) times?/m
                    # if we already have a record for that user in the DB update them then return the updated record
                    if user
                        return Jyter.update(
                            user[:id],
                            {
                                :name         => name,
                                :identifier        => identifier,
                                :claims        => $1.to_i,
                                :comments        => $2.to_i,
                                :agree        => $3.to_i,
                                :disagree        => $4.to_i,
                                :reverse        => $5.to_i,
                                :total_votes     => ($3.to_i + $4.to_i).to_i
                            }
                        )
                    # if we don't have a record for that user in the DB create one then return the new record
                    else
                        return Jyter.create(
                            :name         => name,
                            :identifier        => identifier,
                            :claims        => $1.to_i,
                            :comments        => $2.to_i,
                            :agree        => $3.to_i,
                            :disagree        => $4.to_i,
                            :reverse        => $5.to_i,
                            :total_votes     => ($3.to_i + $4.to_i).to_i
                        )
                    end
                end
            end    
        # on anything other that Success we fail out
        else
            return nil
    end
end

Ok, that's all the heavy lifting. No we just need to get that info to the user.

First, we create a charts controller.

./script/generate controller charts

Inside the charts controller class file (./app/controllers/charts_controller.db) we need to create an index action. Make your file look like the following:

class ChartsController < ApplicationController
    def index
        # if we've gotten an openid identifier from the form then gather the data and display it to the user
        if params.include? :identifier
            # get the data from the database (remember that this will also update the DB if necessary)
            @data = get_jyte_stats_hash(params[:identifier])
            @data[:comments_to_claims]     = jyte_format( get_ratio(@data[:comments],     @data[:claims]))
            @data[:votes_to_claims]     = jyte_format( get_ratio(@data[:total_votes],     @data[:claims]))
            @data[:agree_to_disagree]     = jyte_format( get_ratio(@data[:agree],         @data[:disagree]))
            @data[:votes_per_switch]     = jyte_format( get_ratio(@data[:total_votes],     @data[:reverse]))

            # show the output view
            render :action => :jyte_stats_output
        # we didn't get any identifiers so we need to show the form
        else
            # show the form view
            render :action => :jyte_stats_form
        end
    end
end

You may have noticed the render statements from the block above. For those we'll need to create two files. We'll need a page with a form for them to input the data into and a page for the output.

First, the form. Create ./app/views/charts/jyte_stats_form.rhtml and make it look like this:

<% form_tag({:action => :index}, {:method => :get}) do %>
    <label for="jyte_lookup_identifier">Please input the user you wish to look up: http://jyte.com/profile/</label>
    <%= text_field_tag(:identifier, 'fixedd.com', {:id => 'jyte_lookup_identifier'}) %>
    <%= submit_tag('check') %>
<% end %>

Next, the output page. Create ./app/views/charts/jyte_stats_output.rhtml and make it look like this:

<h3>Results:</h3>
<h4>As of <%= Date.today %> <%= @data[:name] %> has the following stats:</h4>
<h5>Ratios:</h5>
<dl>
    <dt>Comments-to-Claims</dt>
    <dd><%= @data[:comments_to_claims] %> (<%= @data[:comments] %> / <%= @data[:claims] %>)</dd>

    <dt>Votes-to-Claims</dt>
    <dd><%= @data[:votes_to_claims] %> (<%= @data[:total_votes] %> / <%= @data[:claims] %>)</dd>

    <dt>Agreed-to-Disagreed</dt>
    <dd><%= @data[:agree_to_disagree] %> (<%= @data[:agree] %> / <%= @data[:disagree] %>)</dd>

    <dt>Votes-Per-Switches</dt>
    <dd><%= @data[:votes_per_switch] %> (<%= @data[:total_votes] %> / <%= @data[:reverse] %>)</dd>
</dl>

<h5>Raw Stats:</h5>
<p>This is everything that the user has done.</p>
<dl>
    <dt>Claims</dt>
    <dd><%= @data[:claims] %></dd>

    <dt>Comments</dt>
    <dd><%= @data[:comments] %></dd>

    <dt>Agreed</dt>
    <dd><%= @data[:agree] %></dd>

    <dt>Disagreed</dt>
    <dd><%= @data[:disagree] %></dd>

    <dt>Reversals</dt>
    <dd><%= @data[:reverse] %></dd>
</dl>

That's it, you're done. To test it out simply run:

./script/server

Then navigate your browser to http://0.0.0.0:3000/charts and you should see your app.

Now that you've done all that by hand I'm going to be mean and offer the code for download: Jyte Statistics 1: Text-Based Source Files.

AT&T Wireless DNS Correction Script

2008-08-07T00:54:02Z

Getting my Laptop Connect PCMCIA PC-Card working under Linux was time-consuming, at best. Even after I got it working AT&T would still set the DNS incorrectly about 1/3 of the time. This script corrects that.

Site Scanner

2008-08-07T00:54:02Z

A script to scan a website looking for dead links.

Jyte Stats

2008-08-07T00:54:02Z

A quick-and-dirty way to view any Jyte users' statistics.

Jyte Charts (Flash-based)

2008-08-07T00:54:02Z

This project allows you to generate a set of Flash-based charts (that illustrate your general statistics) for your Jyte profile page. It makes use of the SWF Charts library.

Jyte Charts (Image-based)

2008-08-07T00:54:02Z

This project allows you to generate a set of PNG charts (that illustrate your general statistics) for your Jyte profile page. It makes use of the Google Charts API.

Microsoft Points Converter

2008-08-07T00:54:02Z

A quick-and-dirty way to convert the XBox 360's Microsoft Points to real money.

Phone Number Info Scraper

2008-08-07T00:54:02Z

This is a script I worked up for a relative that needed to look a large number of phone numbers as quickly as possible. It scrapes the data from the SuperPages.com web site and saves it to a CSV file.

Jyte Cred Bar

2008-08-07T00:54:02Z

This little widget thingy allows you to embed an image into your Jyte profile that shows your current cred standings.

Jyte Cred Deficit

2008-08-07T00:54:02Z

This project page allows you to check to see if a specific Jyte user is running a cred deficit.

Jyte API library for PHP 5: Services_Jyte

2008-08-07T00:54:02Z

Services_Jyte is a PHP implementation of the Jyte API.

Security Through Obscurity is Neither

2008-08-07T00:54:02Z

Recently one of my friends was working with some commercial PHP software where the code had been heavily obfuscated. He needed to customize it, so he, obviously, needed access to the source code. The original authors had run it through a long line of things like gzdeflate(), base64_encode(), etc then wrapped it in an eval. Ultimately it ended up looking something like this (I deleted about 300 lines of rubbish for sanity):

<?php eval(gzinflate(base64_decode('FZzHjuNAlkV/ZXbTDS7oHWa6G/SkSNH7zYDee8+v
HyVQmypAylBEvHvPUUn5n3//73/mev6v4kz6f1RvM5Z9shf/SJOtILD/y4tsyot//LcYv3K3mUzn
2aCdFj5T5ZhIHBazYw7aUwimQShr8vBDjt8oithczMM4xboabM8SJGmU8GOwFwui5E1gQMaka0xq
puNjzgR6pMUho0UgA0Eh1sN3LDSpvJhTQ3jcWFYeel+swFlCnk66rfGhueJ9Fu8hOFA246x7lA2W
AN6lYxI0wzhRDVU2GnhJ6HES6jwOZPsYMeJ24CgakRzn6FZjvYxZNpk4TIl4NPHICot4u4ioC28v
OQFdtMldIgk/IvSM2VoZRpU593jzFjSAfvIgI8f9qQWehgrwHQpo4ySQQVNgk5MBd1jDLDw+QpUn
A/jVL7wDzrpFpsluE7fvnr/coyqIrtXUyYh+TFJ5uSfcdz5PnQncr0ewMJc1vFxrDVWRt+Rv'))) ?>

The problem was that every time you ran an eval() on it you were left with another obfuscated layer. After decoding down 2-3 layers I quickly realized that this was a losing battle and I needed to let code do my work for me, a good thing too. On the first file I decoded there were 122 layers of obfuscation.

Here's the code I wrote to do the work:

<?php
    // this wants to be run like this:
    //     $php decoder.php input_file.php

    // open the input file
    $fh = fopen($argv[1], 'r');
    $encoded_script = fread($fh, filesize($argv[1]));
    fclose($fh);
    
    // decode
    while (true) {
        if (preg_match('/^\??>?<\?php\s+eval(.+)\s+\?><?\??/s', $encoded_script, $matches))
            $encoded_script = eval('return ' . $matches[1] . ';');
        else
            break;
    }
    
    // write the output file
    $fh = fopen($argv[1] . '.decoded', 'w') or die("can't open file");
    fwrite($fh, $encoded_script);
    fclose($fh);
?>

This simply starts up a loop that runs the eval() as many times as it takes to get to the real source. It then writes the decoded source to a new file. This is pretty specific to the source files in question, but I'm sure it could prove useful as a starting point if you ever come across anything like this.

Random Array Sorting is Tricky

2008-08-07T00:54:02Z

I was asked in an interview recently how I would go about randomizing the order of an array if the language/framework didn't supply any functions/methods to do it for me (I don't think he really cared what my answer was... just wanted to see how I thought). When I was answering the question it was a psuedo-code, but I'm going to use PHP here since that's what the job has me writing most the time (he offered me the job and I took it).

My solution came out something like this:

<?php
    // this loops over the source array "popping" a random element out of it 
    // and placing it in the new array until the source array is empty
    function randomize_array($array_to_randomize) {
        $new_array = array();
        while (count($array_to_randomize) > 0) {
            $rand_num = rand(0, count($array_to_randomize)-1);
            $extracted = array_splice($array_to_randomize, $rand_num, 1);
            $new_array[] = $extracted[0];
        }
        return $new_array;
    }
?>

I was mostly okay with that solution because it 1) worked and 2) didn't seem to waste too much time, I mean, it was just moving the items over one random piece at a time.

Well, the interviewer (my new boss) suggested a different solution (not suggesting it as any more correct, just as an alternative) and as soon as he got the first word out of his mouth I had one of those "ah ha!" moments where I knew what he was getting at before he said it because the same idea had been knocking around the back of my head already (just hadn't worked it's way to the forefront). His suggestion was something like:

<?php
    // this uses the custom sorting mechanism of arrays to generate a fake order
    function randomize_array($array_to_randomize) {
        usort($array_to_randomize, "rand_sort");
        return $array_to_randomize;
    }
    function rand_sort($a, $b) {
        return rand(-1, 1);
    }
?>

I ran some time comparisons and (on my machine) sorting a 26-element array 100,000 times using both methods yielded a time difference of about 0.08 seconds (the second method was the fastest). In light of this I asked my programmer buddies what they thought and they had some really great responses (it pays to have smart friends). The gist was that in most modern scripting languages the first method gives you a truly random sorting whereas the second method only gives you the illusion of randomizization. It basically has to do with the fact that they don't compare every element to every other... put another (incredibly simplistic) way, the optimizations do things like: if you have three elements in your array; A, B, and C then it assumes that if A > B and B > C then A > C.

Well... I decided to do some tests (inspired by Paul). I ran both methods a million times and tallied the results. Here's the output:

Method 1:
    [3, 2, 1] => 166342
    [3, 1, 2] => 165804
    [2, 3, 1] => 167113
    [1, 2, 3] => 167414
    [2, 1, 3] => 166929
    [1, 3, 2] => 166398

Method 2:
    [3, 2, 1] => 444010
    [1, 3, 2] => 147944
    [1, 2, 3] => 259064
    [3, 1, 2] => 74641
    [2, 1, 3] => 24776
    [2, 3, 1] => 49565

As you can see Method 1 has an almost perfect distribution whereas the difference in the most-to-least common in Method 2 is almost 18 times. What that means is that, using Method 2 you'll get [3, 2, 1] 18 times more often than you'll get [2, 1, 3]. I'm liking my first-instinct more all the time.