Monday, December 29, 2008

Unit testing Google App Engine

Was having a massive pain with unit testing with my test development work..well nose plugin and blogging to the rescue.

Will hopefully start making faster progress (code, develop, run test manually, is slower now than tdd).

Sunday, December 28, 2008

Google App Engine

Been playing with this lately. The framework Google provides to access their app engine service is written in python and largely Django from what I can see.

You can supposedly run whatever you want on top of it that uses WSGI with some modifications.

I think I may give this pylons a try at the same time with appengine-monkey . I fear that may be enough steps when using a new framework that idea may be doa but we'll see how much time I have over the next couple of days.

Friday, December 26, 2008

The great dead horse of ORM vs Sproc

I'm not the first to dive in this at all, but here is my pov. I'm only mentioning this because I still see it cropping up a lot on various blogs and message boards as if there was an actual valid controversy.

Let me recap the basic sides of the argument:

ORM side
  • This is the haven of coders
  • Automatically creates your sql for you so you don't have to worry about handcrafting sql
  • uses normal source code so you get easy refactoring and source control
Sproc side
  • Position of pretty much all DBA's, some coders typically from a more traditional development software development model.
  • Been recommended model for development for a couple of decades (since stored procs came into existence )
  • Optimal performance

Now with those points in place if you were a neutral observer (which is a mythical creature on this subject), you would think i'd come out on the side of sprocs with the benefits in performance and it's proven history...and you'd be wrong.

The above is how I'd say a majority of people I've talked to involved in software see things, but reality is I'd put refactoring and source control as being OVERWHELMINGLY important and performance as being secondary and rarely your limiter.

Finally, I think it's intellectually lazy to just go with what's "been done in the past" regardless of existing realities and what changes today brings.

So lets go blow by blow:

Source Control

Without source control we're left never making changes unless we absolutely have to. This has subtle awful effects on software being reused and growing to match what end users want and need to use.

Source control is one tool that helps software change be safe and thoughtfully done. Sprocs while you can keep the scripts in source control to make or alter the sprocs, this means the dba or whoever is editing them has to be manually disciplined or a third party utility has to make the process seamless.

In my experience this does not work without constantly standing on someones head (including your own) that you always check in sproc changes. In the end, everyone is reluctant to change the application.


So lets say a customer or end user comes to you and requests a simple name change of a column in the database. There may be many reasons for this, sometimes it's for a report, sometimes its compliance with a regulation, sometimes it's someone with power who is whim driven. Regardless you now have several ways of dealing with this.

With the orm approach, you renamed the property in your orm class and since most IDE's have integrated mass rename where all references are updated in one click, all you're left with is renaming the table column (or just mapping the new column name in your orm class).

With sprocs you're in a whole different world. Now there is no auto-renaming tool I'm aware of for sql servers (Redgate may have something expensive) and then I doubt there is one for mysql, mssql, db2, etc. So not only do you to renamed code that you've got to change and test your udf's now that may depend on that column name.

In some cases this could be very quick, but in some practical cases I've worked with this just leads to no one being willing to enact such a change or they just cheat the system a bit and only rename the property or column in one place that makes it appear it was changed.

Things get worse as the issues become more complex however I'll assume you get the idea, again this leads to being reluctant to change the application.


Performance is nice, but if you REALLY need it, sprocs will not be what makes the difference. I say this with a couple of caveats, depending on your ORM library bulk updates, deletes, and inserts are going to be slow..very slow, but they have to be REALLY bulk. Otherwise scalability in the truest sense is gotten by application design thought through at a systems point of view using messaging systems and growing horizontally.

Lets say sprocs were somehow 25% more performant than hand sql/orm, they're not, but if they were imagine say you had an widget seller app that could process 700 transactions a second.

The widget seller company comes out with a buy one get one free coupon, now transactions are regularly over 100,000 a second. If you were using sprocs or not, you're done for.

However, if you've written an app that uses a messaging protocol and can be spread out over multiple datasources (or even one datasource that's clustered), with several web servers and several middle ware maybe actually able to scale to that 100,000 a second transactions regardless of if you use sprocs or not.

Scaling out and the use of asynchronous transactions are the actual key way to achieve scalability...everything else is a fake wedge issue.

Most Importantly

Generally speaking anything that gets in the way of giving the customer what they want is very very bad. It leads to applications getting scrapped repeatedly because either the customer has replaced you.

If you're lucky, you've convinced them that yet again they're wrong and did not properly know at the very beginning exactly what they wanted, please deposit another 40k in development to generate the app you desire...this time it'll be perfect we promise.

Heavy use of sprocs are hard to justify when it comes to the maintenance side of things.


Ultimately I think this does come down to a real valid pull between what dba's have been taught is there job and what agile developers needs are. With the proliferation of agile growing daily (and I'm certain the quality of the agile attempts decreasing as it becomes more and more "the thing to do") it's natural and logical to see this red herring happen over and over again.

Are there other things to consider in this argument? Oh definitely but I've yet to find anything that tops maintainability and adaptability as key reasons to avoid sprocs.

This argument I'm certain will continue indefinitely, and I'll in the near future have to call some 2000 line business logic heavy stored proc that takes 10 minutes to read and then setup on a test database to make sure i'm understanding what my changes will do, but I'll go out of my way to document and then see if i can move it all to unit tested, source controlled, good old boring bog standard code.

Sunday, December 21, 2008

Snake Guice

So I was looking at making my own IoC container in python using inspiration from Ayende's post on rolling your own IoC in a pinch (though I may run into this again when I start playing with ruby) and I came up what I thought would be an ingenious idea to use decorators on methods and classes to do type resolution.
Something like so:

class Foo(object):

@ioc(booable = ghost, mooable = cow)
def __init__(self, booable, moooable): = boo
self.moo = moo

I'm sure it wouldn't be popular with your average python dev, but I again really want autowiring of dependencies, and didn't want to make a bunch of factory classes to tie together a bunch of related dependency chains.

Well someone already went with that idea with snake-guice an IoC container inspiried by google guice which is written in java and makes heavy use of annotations the python equivalent of decorators and not concidentally my inspiration for my attempt at an IoC container.

I'll update with more information later as I get to play with snake-guice if I can grok the concepts and code, perhaps I'll dare venture into OSS and see if I can help contribute.

Sunday, December 14, 2008

IoC in ruby, python and lack of autowiring

In my last post I waxed joyously about my love of IoC containers which are third party libs that track all my dependencies.

Now I've been trying to play around with Python and a bit of Ruby lately, and while I just adore the lack of getting decoupling for free (duck-typing gives you this), I enjoy the lack of compiling and I enjoy getting to have a fresh look as a programmer at languages I used to use as an admin for simple scripts.

However, the love fest and productivity ends when I realize I've lost my beloved auto-wiring of dependency management I get with things like Castle Windsor.

Most of the ruby and python IoC containers I found were no longer in active development and were started back when IoC was taking off in Java back in 2004 or so, namely Needle, Copland and py-container.

So I've looked at Spring Python which is active, looks nice and all, but I have to explicitly map components myself. You do get Aspects, service location, and some framework helpers like security, and transaction managment, but without auto-wiring ..... I have to be honest may not be worth it to me at all in a dynamic language.

So this leaves me in a bind. I've been convinced that dynamic languages are the future, staticly compiled languages keep moving that way, but now that i've learned all these awful tricks to make static languages more flexible, and in the meantime found the joy of IoC autowiring....seems like I'm stuck being most productive still in statically compiled languages.

The only alternative I forsee right now if I keep developing in Ruby or Python is build alot of factory classes to tie together large junks of dependency injected code.

Hopefully, I can find an autowiring IoC container for one of my dynamic languages of choice but I dont see how easy it'll be.

IoC and Dependecy Injection

IoC (Inversion Of Control) is one of many programming concepts that when I was first approached with it I wondered why on earth anyone would have need of such and thing and now I can't imagine a world without it.

The 10,000 foot simple view of it goes something like this:
  1. Anything in your code that requires a resource or code from somewhere else is a "dependency". Examples: database access, another class that provides a method call or service,a web service, etc, etc.
  2. Give all your dependencies an interface or an abstract underclass. have all of your code depend on these instead.
  3. Make all your dependencies come into your class via the constructor, a method parameter, or a setter property. Your end goal is to never have "new" called on anything unless you're working with a class that only contains data . Also using static methods to get around new is cheating and still hard coding a dependency.
  4. Use a third party solution or wire up your own IoC container, that looks at all your dependencies and plays match maker (oh this class needs widget a,b, and c , let me call new widget a, b,c, and add them to this class when called). This replaces all your "Factory" style classes.
Short example of this in code is as follows:

public interface DrawWidget{

void Draw();

public class ColorWidget: DrawWidget
public ColorWidget()
//imagine this connects and initializes connection to printer

public void Draw()

// imagine color widget active code here

public interface WidgetData

public string[] GetStrings();
public class WidgetDb : WidgetData
public WidgetDB()
// imagine this sets up connections strings and makes db connection.

public string[] GetStrings()
//imagine db code here


traditional approach:

public class WidgetCombiner

public CombinedWidget WidgetPayload()


WidgetPayLoad payload = new WidgetPayload();
payload.Draw = new ColorWidget();//connection to printer must be alive
payload.Data = new WidgetDb(); //connection to db must be alive

return payload;



Dependency Injection Approach

public class WidgetCombiner{

private DrawWidget draw;
private WidgetData data;

public WidgetCombiner( DrawWidget draw, WidgetData data)
this.draw = draw; = data;

public CombinedWidget WidgetPayload()

WidgetPayLoad payload = new WidgetPayload();
payload.Draw = this.draw;
payload.Data =;

return payload;

The key difference is WidgetCombiner in the first "traditional" implementation it depends on the printer being live and the database connection functioning. The second implementation does not care at all, it's dependency is dynamic and easily changed to another database, another printer, or just a fake test environment.

Thursday, December 4, 2008

YAGNI and when

I struggle with YAGNI frequently (You Ain't Gonna Need It). But, recently I had some success with just being draconian about. Not adding features unless I could justifiably say "this is adding value X for Y need".

Today, another developer I was working with gave me a bit of a conundrum. We were expecting one of two values from a column in a database (Say "S" and "E"). However, in very very rare cases we could get another couple of different values (Say "X" and "C"). They would likely never come up into our code, and if they did, the rest of the corresponding data would be completely incorrect, and the data would go through the normal paths of incorrect data.

So in this case, should I adjust the code to handle extreme corner cases, that don't effect the actual outcome of the application, but could give some fidelity into logging if an "X" or a "C" actually showed up?

I think YAGNI in this case is telling me not to worry about the C and the X, as they don't actually effect the end result and we have no need for that level of logging fidelity at this time(and changing the where block to exclude the bad possibilities should be enough). But it did give me pause to just leave those corner cases floating out there untouched.

It made me think a bit how in reality a YAGNI style approach probably helped contribute to the justification of y2k problems, and to a degree this was the same issue (which was this other developers contention).

However, I'm pretty certain with modern languages and patterns one could stick to YAGNI and avoid a y2k style mess.

Tuesday, December 2, 2008

Mako Templates

Ok so tired of the goofy stuff with Kid even though I did find out a way to debug templates but I was still upset enough that I wanted to go away from an XML based template engine entirely.

Enter Mako. Mako is newish but the preferred engine of Pylons and it works on Turbogears as well. Instead of being xml based it's a run of the mill view engine using python as it's scripting language. This struck me as the easiest and most straight forward for my skill set(especially since I use the pythonic Brail in Monorail).

Turbogears and Mako integration take some hunting. I had to resort to just giving it a stab how I wanted it to work, googling my errors, and then bit by bit piecing together the sections of configuration i needed.

Finally I groked the concept thanks to Wesley Chun the author of Core Python Programming who'd run into a similar conundrum (at least it makes me feel less stupid) .

Key excerpt here:

2. another problem that people have is that in your controller, your @expose decorator call tends to have more of a "full-path" to the template when porting from Kid, i.e.,

@expose(template="tghello.templates.hello") def hello(self): :

when switching to Mako, because of your mako.directories list, you no longer have to give the entire path, meaning that the failures are happening because it's trying to find that full name as a template file name, which isn't going to work.

Finally, played with this briefly feel much more at home with this templating style.

Castle Active Record and ActiveRecordMediator

the ActiveRecord pattern as it's typically implemented works like so.
step 1 - Take a base class with all the persistence methods you need (save, update, delete, find, etc)

public class ActiveRecordBase{

public static object Find(ICriteria criteria)
//return object

public void Save()
//save object to db here
// rest hidden

step 2 - take all your db classes and have them inherit from them (optionally passing on column configuration or whatever you'll make available).

public class Account : ActiveRecordBase

public int Id{get;set;}
public string Name{get;set;}
public AcctType Type {get;set;}


client code ends up looking like so

public void static Main(string[] args)
Account acct = new Account();
acct.Name = "Max Earnings";
acct.AcctType = AcctType.Checking;

Account[] acctsfromdb = Account.FindAll();


Now alot of people like this pattern. They'll attach behavior or methods to the properties or even add their services to the same class (in the above example maybe adding a method that calculates current interest rate against a lookup table).

I however have always had issues with unit testing this way. You end up testing against a database directly (which is substantially slower), and your class becomes decorated with all these persistence methods. Plus I've never personally enjoyed mixing data classes and method classes (called services in Domain Driven Design), this confuses the picture for me and muddies things up (what do you do when you have an interaction between multiple classes for example?)

So how do you get the simplicity of db access with ActiveRecord but not have to worry about having database tied classes? Well you can use an orm that's persistent ignorant like Nhibernate (or several others), which involves alot of by hand configuration (I say this watching the fluent nhibernate project closely), session, transaction handling, and generally gives you more control than you need for alot of starting projects. If only you could find a nice middle ground.

Enter Castle ActiveRecord and ActiveRecordMediator class. ActiveRecordMediator is NHibernate for dummies (or lazys). So back to our earlier example

public class Account
public int Id{get;set;}

public string Name{get;set;}

public AcctType Type {get;set;}


now client code looks like so

public static void Main(string[] args){

Account acct = new Account();
acct.Name = "Max Earnings";
acct.AcctType = AcctType.Checking;

Account[] acctsfromdb = ActiveRecordMediator<account>.FindAll();


So recap the difference in code is small. Lines of code is about the same..but now I can decouple the use of my objects away from database...but HOW you ask?

Ahh yes let me carry you that last step.

public interface IRepository<T>{

public void Save(T t);
public T[] FindAll();
public void Update(T t);
//etc etc

public class DbRepsository<T>:IRepository<T>where T : class

public void Save(T t){


public T[] FindAll()
var records = ActiveRecordMediator<T>.FindAll();
return records;
//etc, etc

public class Client{
private IRepository<Account> _db;

public Client(IRepository<Account>() db)
_db = db


public void AddAccount(Account acct)


public class FakeDb<T>:IRepository<T> where T : class
private List<T> records = new List<T>();

public void Save(T t){
public T[] FindAll(){
return records.ToArray();
//etc etc


public class TestClientCode{

public void should_save_record_to_repository(){

var memdb = new FakeDb<Account>();
var client = new Client(memdb);
Account acct = new Account();
acct.Name = "Max Earnings";
acct.AcctType = AcctType.Checking;


var fromdb = memdb.FindAll();


Final Recap if you look at the test we can now easily write unit tests (with no db to worry about). With the simple example provided here there isn't much obvious gain, but start adding your persistence classes to controllers, services and any other usage you can think of and this pays back the initial work we did here 10 fold.

Monday, December 1, 2008

Issues with Kid templating

Was working on my little Turbogears app I'm building and ran into some issues with the Kid template system that's default with turbogears 1.0.

Now I've wasted several hours just messing around with master and layout files. I can build my own master file and have it work quite well, but no one wants to see my websites so I grabbed a template off OSWD and it was even in strict xhtml.

First, I try the master approach. This is to me a pretty noisy process (have to specify match parameters in html, head, and what i want to be the content area, i learned through trial and error this does not seem to be optional) but at least it looks like a complete xhtml file to a web designer.

Now the specific pages just need a py:extends="'sitemaster'" attribute in the html tag and blows up with an odd error (
Template file "'sitemaster.kid'" not found
while processing layout="'sitemaster.kid'"

I dive into the kid language specification and find the py:layout="'sitemaster'" option. this looks much better. far less noise and at least somewhat closer to what you'd find in NVelocity or Brail.
However in the end once I've made the same changes, done a test case that works with some hand thrown together html. When trying to load up my 3rd party template it blows up with same odd error.

Long story short, I narrowed enough down to find there is something in an WC3 xhtml strict compliant doc that throws the kid parser for a loop. I can delete all the html in the body and it works.

I believe the way kid ends up approaching things is ultimately to blame here, and I'd have to change the template output to strict xhtml or something to get this all to work..but ultimately I shouldn't have to think about this at all.

It is my belief that view engines should stay out of the way of the html entirely, they shouldnt barf on it, and if they do they should tell you what element is causing you the problem, or where the parser gives up at the horrible mess you've sent it.

I get that Kid is trying really hard to make a designer friendly view engine, but in this instance if I were working with a designer and we got this error when he made an html change..and he actually did it 3 days ago, and he did a bunch more changes since...we'd be struggling for hours to figure out what was the offending piece of html he added.

Now I'm sure experienced users of Kid would chastise me, and that I'm supposed to learn all the ins and outs and then it'll all be worth it (it may very well be, but I'm not convinced yet).

I had a look at genshi and it looks like the same model but better in some ways (extension is .html..this alone makes it worthwhile for dealing with skiddish designers). Maybe it'll tell me where this html blows up...regardless I'll say it again..view engines should not care what html I throw it.