I mean, the sad part is I actually understand why they likely implemented this instead. At this point in Visual Studio’s lifecycle, it’s easier just to poll to check if the UI is unresponsive than to actually root out the cause of the unresponsiveness. It’s sad, actually. And the fun part is, in preparation for this blog article I didn’t even have to try very hard to get that little balloon to pop up. Took me about 30 seconds (and no I’m not going through the trouble of reporting it to Microsoft because it’s not entirely repeatable and I’m skeptical of what the results would be anyway). Go Visual Studio.

I’ll start this off with a quick question for you to ponder the answer to before I go into the details — What is a “property” on an object other than essentially just a variable defined only within the scope of that object? Rather than answer that question just yet, here’s the problem I was scratching my head over. In SSIS, they have some nifty pre-defined events which you can easily attach event handlers to. Thanks to Jamie Thomson’s blog post entitled “Custom Logging Using Event Handlers” I’ve been happily coding away logging routines using the OnPreExecute and OnPostExecute event handlers (among a few others). But, I soon ran into a problem that wasn’t addressed in the blog post, and that was that every event that fires propagates up the call chain, regardless of whether or not the event is handled^[*]. That is to say, if I have an SSIS package that has (”contains”) a task which loads a file into a database, and I define an event handler for OnPreExecute that simply writes a row to an event log that says “Started: “, then once it gets to the file load task, the database will contain the following:

1. Started: Main package
2. Started: File load
3. Started: Main package

Obviously that last row is a bit silly because the main package didn’t just start, it’s in the middle of executing all of its tasks. That’s because the PreExecute event propagated up from the file load task. Now, one might argue that propagating that particular event doesn’t make a lot of sense because you only ever PreExecute once for each task, and it certainly has nothing to do with when the child tasks do anything. But, for the sake of consistency I can forgive that. If that was the wart, then that’d just be a “huh, thats strange” kind of wart. After all, you certainly do want Error and Warning events to propagate up so you can just define event handlers for those in the parents.

So, at this point I figure there has to be some way to prevent the upward propagation of some events, even if I have to specify it per event handler. The first place I looked was in the properties for the event handler. The only thing that looked mildly promising was the LoggingMode property, but that was limited to “Use Parent Setting”, “Enabled”, or “Disabled” - nothing about “handle and stop propagating”. So, the next place I checked was the “expressions” for the event handler. Expressions in SSIS are basically ways of setting properties that are dynamic (ie, can change at/during runtime), and they’re very useful and generally contain most of the properties anyway, plus an occasional extra tidbit. Well, no extra helpful tidbits in this case, though I was positive that was where it would be.

What does any good programmer do when they’ve exhausted their available local resources? Google it, of course. Unfortunately no matter what search string I used (and my Google-fu is pretty good), I couldn’t find anything that would tell me how to do what I want. Plus, the only other SME on SSIS in the company that I’m aware of was unavailable to ask, and I’m about to outgrow using him as a reference anyway. Then I remembered the aforementioned book, and sure enough one of the first things in the index under events was “bubbling”. And in less than 2 minutes of reading, I had my solution.

Now for the solution and then a quick revisit to the question I kicked this story off with. The solution: there is a “system variable” defined at the error handler scope level entitled Propagate, which if you set it to False will not propagate the event to its parent(s). How the heck is a system variable defined only within the scope of a single object (and obviously only useful within that scope) a better choice than a property? Properties are displayed in a nice little grid that’s easily accessible. System variables are hidden behind several clicks in the interface. I’d wager that 95% or more of SSIS packages never have to muck about with them, and none of the dozen or more packages I’ve worked on have ever needed to do so (until now!), in spite of all the functionality I’ve included. Thankfully, I’m not the only one who had such an issue (I didn’t find that blog post until I started writing this one, and he doesn’t have the solution in his post, but Jamie Thomson posted a response which he found).

And to make matters worse I just realized that this means I’ll have to define that variable for all children within a package otherwise I’ll get a lot of “Started: Main package” messages, one for each sub-task (which is a lot, including checking for input, truncating tables, etc.). According to the blog post and response linked at the end of the last paragraph, there’s no way to tell an event handler to not listen to propagated events either. Sigh.

^[*] - it also doesn’t help that Microsoft’s documentation has a very misleading/incorrect diagram showing event propagation stopping at the first handled event

First I’d have to do some definition/demystifying up front: answering the who/what/why’s and all that, because this is a heavy Microsoft crowd, though not so much that they are fanboys or anything like that (well, some might be but they likely won’t attend) it’s just what they know and that’s what the clients run on. I know at least a few people, including some of the partners, are genuinely interested in open source not only as a business strategy but as a community. So I think the first few big talking points in this section would be:

• What is open source? (I’d like to stick mostly with definitions here and not delve too much into history and/or the fractious issue of what constitues “free” or “open source” software)
• What open source projects they might be aware of, even if they don’t know they are open source (Linux, Mozilla, Java (yes, Java), PHP, Subversion, etc.)
• What open source projects affect them that they are likely unaware of (Apache, Python, MySQL, Mediawiki (wikipedia))
• Why open source? (Why do developers give away code? Why use it?)

And that would be the bulk of the presentation. From there I can come up with a more directed approach talking about how my company can leverage the advantages of open source to help increase productivity/add value/etc. Really I just wanted to hear what people thought about the first part and whether or not I was missing anything or perhaps should take some stuff out. Though if anyone has any feedback for what open source could mean for a small IT consulting firm, then by all means, fire away

On the current client I’m designing stuff for, we’re using MS SQL Server 2005’s SSIS toolchain, which is a great idea with mediocre execution. For those who are unfamiliar with SSIS, it’s basically the evolution of their DTS package that came with SQL Server 2000, which is designed to simplify the ETL process of getting data from a source (or various sources) into the destination database after all sorts of cleansing and transformation. The DTS package that shipped with SQL Server 2000 was apparently pretty warty, a trait that is common amongst first-gen Microsoft products, and so SSIS is apparently notably cleaner and nicer. But, as most of us are aware, that doesn’t mean it’s wart-free, and I’ve found a nasty wart or two.

So, for the product I’m working on we have to load lots of files into the database, sometimes batches at a time. They’re segregated into different folders based on what kind of input files they are, and you need to be able to load all of the files in the folder that match a specific file pattern. Great, Microsoft provides a nice “control flow item” named the “Foreach Loop Container” which does exactly that. It looks just like a box with a little label on it where you can place other “control flow items”, and in my case I just use the “Data Flow Task” control item, because that’s what takes data from one place and transforms it and ships it to another. So, since Foreach Loop Containers are necessarily looping over files in a given folder (really it’s not as generic as the name sounds, it’s a “Foreach File in a Folder Container” really), you’d figure that there’d be a sensible way to hook up any file sources within it to, you know, the current file being processed. Well, there’s not. To get to it, you have to define a local variable for the container in the container definitions, and that will be re-set each time it starts a new file. That by itself wouldn’t be so bad if there was a straightforward way to make a data source that simply looked at that variable and was usable that way, but there’s not. In fact, to make it even worse, there is a specific “connection” type called the “Multiple File Connection”, which would work perfectly with this if it were designed to work cooperatively with the Foreach container, but no. The way Microsoft wants you to accomplish it is to set that local variable and then configure a flat file connection pointing to any random file (you have to point it to a file first to configure the output columns), and then tell it to ignore the actual file location you used in setup by overriding the ConnectionString property of the connection object. Of note is that if you should decide that you want to go back and change some of the output column formatting you have to once again go through the whole process of picking one of the files (even though it doesn’t actually use that file, it uses the ConnectionString property which we’ve overriden), then modifying it and then saving it. Luckily you don’t have to re-edit the property again.

All this discussion over what should be a pretty simple task. And another quick, but equally ugly wart is: in SSIS you can’t have a delimited input file that has a variable number of columns that will get parsed nicely for each line (and fill in blank/null outputs for the columns it fails to see). So if I have a CSV file with only 4 columns on the first line and 8 on the next, then my only option in SSIS is to set the number of columns to 4 and then write a script to process the lines that have separators still left in them.

Most of the initial work was doable within an hour or two, just laying out the models and such. The built in admin interface is really nice, though there was one case where I thought I could use the edit_inline property on some of the ForeignKey fields I had, but it doesn’t look like I can flex it like that. Basically, I was hoping the admin interface would allow me to add in the relevant game results/statistics for this model:

class Game(models.Model):
    week = models.ForeignKey(Week)
    home = models.ForeignKey(TeamGame, related_name='home')
    away = models.ForeignKey(TeamGame, related_name='away')

with the TeamGame class being the individual representation of how a team performed in a given game:

class TeamGame(models.Model):
    team = models.ForeignKey(Team)
    points = models.IntegerField()
    yards = models.IntegerField()
    turnovers = models.IntegerField()

I used a similar data model for the poker league tracking site I wrote in Django about a year ago, only since it was a less-strict “many-to-one” (in that case, PlayerGame to Game) relationship instead of the strict one home, one away “two-to-one” relationship we have here, it makes using edit_inline unworkable, at least the way I understand it and the way I used it on the other site. But, writing a custom view to add the data was relatively simple, and I think I could have even just used the generic CRUD views provided, but I just wrote my own instead.

What makes me wonder if this data model/schema is the “right way” to do it is the fact that extracting some of the necessary data to do the rankings gets kind of cumbersome, probably because of all the normalization (though really normalization is supposed to be the big win here, remember). For example, the next page I wrote after the “add a game’s stats” page was one to view the year-to-date statistics of a given team for each week. And it’s not as clean and pretty as I’d like it to be. In fact, it’s kind of warty-looking, though it’s not too bad in the actual ORM syntax, even though I did resort to using the Q() method, because remember, I want the opponent info as well as the actual team’s info from each game that they’re in, which is what the Game objects are pulled for (and then I assign the current team’s TeamGame for that particular Game in the my_tg variable, and the opponent’s TeamGame in opp_tg):

def display_team(request, team_id):
    team = get_object_or_404(Team, pk=team_id)
    teamgames = TeamGame.objects.filter(team=team)
    games = Game.objects.filter(Q(home__in=teamgames) | Q(away__in=teamgames))
    tmp_games = []
    totals = {'pts_for': 0, 'pts_against': 0,
              'yds_for': 0, 'yds_against': 0,
              'to_margin': 0}
    for game in games:
        if team == game.home.team:
            opp_tg = game.away
            my_tg = game.home
        else:
            opp_tg = game.home
            my_tg = game.away
        if my_tg.points > opp_tg.points:
            result = 'W'
        else:
            result = 'L'
        tmp_games.append({'opponent': opp_tg.team,
                          'result': '%s, %s - %s' % (result, my_tg.points,
                                                      opp_tg.points),
                          'pts_for': my_tg.points,
                          'pts_against': opp_tg.points,
                          'yds_for': my_tg.yards,
                          'yds_against': opp_tg.yards,
                          'to_margin': opp_tg.turnovers - my_tg.turnovers,})
        totals['pts_for'] += my_tg.points
        totals['pts_against'] += opp_tg.points
        totals['yds_for'] += my_tg.yards
        totals['yds_against'] += opp_tg.yards
        totals['to_margin'] += (opp_tg.turnovers - my_tg.turnovers)
    num_games = float(games.count())
    averages = {'pts_for': totals['pts_for']/num_games,
                'pts_against': totals['pts_against']/num_games,
                'yds_for': totals['yds_for']/num_games,
                'yds_against': totals['yds_against']/num_games,}
    return render_to_response('rankings/team.html',
                              {'team': team, 'games': tmp_games,
                               'totals': totals, 'averages': averages})

Granted, a good portion of that is all that dict assignment junk so that I can have a much cleaner looking template file with less “business logic” in it and more pure iteration over results. But it just seems like an unhealthy amount of logic for what is pretty darn simple. But it does make for a pretty simple/clean template:

<h2>{{ team }}</h2>

Games
<table>
    <thead>
        <tr>
            <td>Opponent</td>
            <td>Result</td>
            <td>Yards For</td>
            <td>Yards Against</td>
            <td>TO Margin</td>
        </tr>
    </thead>
    <tfoot>
        <tr>
            <td>TOTALS</td>
            <td>{{ totals.pts_for }} - {{ totals.pts_against }}</td>
            <td>{{ totals.yds_for }}</td>
            <td>{{ totals.yds_against }}</td>
            <td>{{ totals.to_margin }}</td>
        </tr>
        <tr>
            <td>AVERAGES</td>
            <td>{{ averages.pts_for }} - {{ averages.pts_against }}</td>
            <td>{{ averages.yds_for }}</td>
            <td>{{ averages.yds_against }}</td>
            <td></td>
        </tr>
    </tfoot>
    <tbody>
    {% for game in games %}
        <tr>
            <td>{{ game.opponent }}</td>
            <td>{{ game.result }}</td>
            <td>{{ game.yds_for }}</td>
            <td>{{ game.yds_against }}</td>
            <td>{{ game.to_margin }}</td>
        </tr>
        </tr>
    {% endfor %}
    </tbody>
</table>

So I like the result on the template side, but seeing as how I had to do something very similar with my poker site in order to get all the game data for the “display player info” page on that site (that is, aggregate stuff into a temp list or dict and churn a lot of data basically), I’m wondering if there’s a better way to do the view without making the template code crazy complex.

But, Google has plenty of great web designers at work there, so they don’t have horribly malformed HTML or tons of inline style crap. Most other websites out there aren’t quite in the same boat. In fact, looking at the source of the vast majority of sites will make most competent web designers wince if not cry. Even well-formed HTML is not that easy to parse, and folks often resort to imprecise string matching or nasty regular expressions to get the job done. If only there was a way to get that nasty HTML into a more nicely-parseable format…

Well, I saw this blog post on programming.reddit.com (highly recommended, btw), and apparently he has set up a service that will fetch webpages and transmogrify them into either (presumably well-formed) XML or JSON output, two flavors of output that have become popular with the rise of AJAX. Unfortunately, you still get a lot of crap because, hey, Garbage In, Garbage Out. But, at least it’s crap that’s in a somewhat prettier outfit. Or if not prettier, at least, easier to dig through. However, take heed in the authors plea at the end and don’t hammer the crap out of this.

To wit, I did write a Yahoo search plugin that people have used and like using. Thankfully they actually actively embrace Python programmers using their APIs and just provide a nice RESTful way of doing things without any domain specific junk. It’s really quite a good plugin and a fairly good search engine and I encourage Supybot users to use it if they don’t have a Google SOAP key.

However, that said, people still want their Google. Unfortunately simply “updating” the Supybot Google plugin isn’t as simple as transliterating things from one API to the other, you see. Instead of providing a nice API like Yahoo did, Google replaced the SOAP API with an AJAX one. That’s fantastic for embedding search boxes on web pages and for ease of rolling out updates (just update the script that everyone’s <script> element points to), but really crappy for the IRC bot plugin authors! Fortunately, behind the scenes of the nasty obfuscated JavaScript there is a RESTful service as well, and it too (like Yahoo’s) returns a JSON-formatted resultset (with a catch that I’ll get to later).

So, what to do if you want to use this nice (invisible to most) service? Here’s the skinny.

The “new” search system is called “UDS” for “User Distributed Search”. I don’t know what that’s supposed to mean, but man am I tired of seeing the UDS acronym all over that JavaScript file I linked to above. I’m actually in the process of untangling/de-obfuscating it at the moment, but that’s only tangentially relevant to this post. Really all you need to know to use this service is how to construct the proper URL and what the results mean. But first things first, you’re going to need one of the AJAX keys to do it so go grab one first.

Now, the URL structure is pretty simple, the “base URL” is just:

http://www.google.com/uds/GwebSearch?

In fact, that in and of itself is a valid search URL. Go ahead and try it if you want, and you’ll see an example of an error result. For real application uses, you’ll want to construct the proper query string that goes next. Here’s what you need to put in:

• callback = for now just use GwebSearch.RawCompletion here, though once I detangle the JavaScript I may figure out other possible values here (required)
• context = not sure about this yet either, but using 0 as the value has worked for me pretty well (required)
• hl = the “locale” you want to search in, for English-speaking folks, it’s en. For any other locale I’m guessing it matches this listing. Entries that aren’t on that listing seem to use some sensible default (likely en).
• key = this is the big API key you got when you signed up (required)
• v = the “version” code, I just use 1.0 (required)
• q = the actual query string, formatted for URLs (replace spaces with + signs, etc.) (required)
• rsz = the “result size” you want, the valid values I see are large and small, with the default being small. Small returns 4 results, large returns 8 results. (optional)

There are various other arguments that you can add, but those are the most relevant and the ones I understand best. So, a quick example URL for a search for “moo” would be:

http://www.google.com/uds/GwebSearch?callback=GwebSearch.RawCompletion&context=0&hl=en&key=<key>&v=1.0&q=moo&rsz=large

Obviously, you’d put your key in instead of just <key>, but the rest is all exactly what you’d put in for a search for “moo” where you wanted the most results. One important thing to note though is that the order of these arguments does not matter. As long as they are all key=value pairs separated by & there should be no problem.

Great, so now what? Well, the result you get back from a GET (note: this is important, it must be a GET and not a POST) on that URL, it looks something like this:

GwebSearch.RawCompletion('0',{"results": <stuff>}, 200, null, 200)

Of course, it’ll be much longer because <stuff> will actually be the important stuff we’re looking for. But this is the catch I spoke of earlier. Yes, this is a properly-formatted JSON response. But, most JSON parsers will choke on it, because while it does represent a JavaScript object, it doesn’t know what the heck a GwebSearch.RawCompletion is without a little context! Fortunately, basically all we are interested in is the dictionary “results”, and we can just parse that out and give that to our JSON parser instead.

So, let’s look at what the “results” dictionary looks like for this search:


{"results":
[
{"GsearchResultClass":"GwebSearch",
"unescapedUrl":"http://www.moo.com/",
"url":"http://www.moo.com/",
"visibleUrl":"www.moo.com",
"cacheUrl":"http://www.google.com/search?q\u003Dcache:DsfEzWoDcLoJ:www.moo.com",
"title":"\u003Cb\u003EMOO\u003C/b\u003E | We love to print",
"titleNoFormatting":"MOO | We love to print",
"content":"\u003Cb\u003EMOO\u003C/b\u003E Print Limited, registered in England; company number 5121723; registered office 6 Bakers Yard, Bakers Row, London, EC1R 3DD \u003Cb\u003E...\u003C/b\u003E"
},
{"GsearchResultClass":"GwebSearch",
"unescapedUrl":"http://en.wikipedia.org/wiki/MOO",
"url":"http://en.wikipedia.org/wiki/MOO",
"visibleUrl":"en.wikipedia.org",
"cacheUrl":"http://www.google.com/search?q\u003Dcache:O-LNoqQLso8J:en.wikipedia.org",
"title":"\u003Cb\u003EMOO\u003C/b\u003E - Wikipedia, the free encyclopedia",
"titleNoFormatting":"MOO - Wikipedia, the free encyclopedia",
"content":"A \u003Cb\u003EMOO\u003C/b\u003E (MUD object oriented) is a type of MUD and is a text-based online virtual reality system to which multiple users are connected at the same time. \u003Cb\u003E...\u003C/b\u003E"
},
<6 more similar results here, snipped for brevity>
],
“cursor”:{}
}


So basically it’s a dictionary with two elements: results and cursor. We’re not interested in the cursor, just the results, which maps to a list of dictionaries itself. In each of those dictionaries you’ve got several fields:

• GSearchResultClass = for web searches, it looks like this will always be GwebSearch
• unescapedUrl = it is what it sounds like
• url = also what it sounds like
• visibleUrl = not entirely sure what precisely this is supposed to be other than the main site URL
• cacheUrl = a link to the URL for the search result as cached by Google, very cool
• title = duh
• titleNoFormatting = duh (see what I did there?)
• content = The standard 2-line-or-so block of text describing the URL

Now, with this in mind, with any handy JSON parsing library, you can code up your very own Google search app that uses the new AJAX keys instead of the SOAP API keys. In an upcoming post I’ll do a Python example.

The specific example this time is in dealing with what statistics to track and how to score them. Most popular fantasy leagues only track a very small subset of the available stats on a given sport. In football, it’s typically a list like:

• QB: pass yds, pass TDs, interceptions
• RB/WR/TE: rec yds, rec TDs
• QB/RB/WR/TE: rush yds, rush TDs, fumbles lost, 2pt conversions
• K: FG made/missed in specified ranges (0-19 yds, 20-29 yds, … up to 50 yards or more)
• DEF: fumble recoveries, defensive touchdowns, sacks, interceptions

But there are still plenty of other stats out there, especially in leagues that use different positions (like ones that use IDPs instead of team defenses). So, my ultimate end would be to have a flexible place to store each statistic and its info, though for now I’m happy with something that just the ones I need to run my leagues.

The question is, how exactly would such a structure look in the database? Keeping in mind that some of these stats are fundamentally different, and some have some scoring quirks, I’ve come up with three separate stat “types”: the Range Stat, the Rate Stat, and the Rate Stat With Bonuses.

The Range Stat would be something like “Points Allowed” for a team defense or “FG Made” for a kicker. I think, rather than having specific stats like “FG1-19″ (number of field goals made that were between 1 and 19 yards), I should just be able to say “he made an 18 yard field goal” and let the scoring rules for each individual league adjust it accordingly, instead of shoe-horning it into the FG1-19 stat for one league and the FG1-29 for another. Same thing with defensive points allowed, I’d like to just be able to enter each team’s scores for a game and then have it slot accordingly.

Rate Stats are the bread and butter of fantasy. They are the “6 pts per rush TD” and “0.1 pts per rush yd” stats. There’s not much magic here, really.

Rate Stats With Bonuses are less common, but useful. Some leagues choose to offer a bonus for big games like instead of just “0.1 pts per rush yd” they’ll have “0.1 pts per rush yd, +5 pts at 100 yds”. Not a whole lot of magic here either, as the bonus is applied during the score evaluation and not stored in the database itself.

But the question is, how do I represent these in the DB without having static columns for each possible stat? Is that possible? I’d like to be able to potentially add new stats without having to alter table structures. I’d rather not have to just have a huge table with nothing but default values for things that I’m not using at the moment. I’m wondering if there’s some sort of generic pattern for doing this or maybe I just haven’t thought hard enough about it. I’ll post details about the current schema in a later post, perhaps.